Hi,
This patchset is the latest version of soft offline rework patchset
targetted for v5.9.
Since v5, I dropped some patches which tweak refcount handling in
madvise_inject_error() to avoid the "unknown refcount page" error.
I don't confirm the fix (that didn't reproduce with v5 in my environment),
but this change surely call soft_offline_page() after holding refcount,
so the error should not happen any more.
Dropped patches
- mm,madvise: call soft_offline_page() without MF_COUNT_INCREASED
- mm,madvise: Refactor madvise_inject_error
- mm,hwpoison: remove MF_COUNT_INCREASED
- mm,hwpoison: remove flag argument from soft offline functions
Thanks,
Naoya Horiguchi
Quoting cover letter of v5:
----
Main focus of this series is to stabilize soft offline. Historically soft
offlined pages have suffered from racy conditions because PageHWPoison is
used to a little too aggressively, which (directly or indirectly) invades
other mm code which cares little about hwpoison. This results in unexpected
behavior or kernel panic, which is very far from soft offline's "do not
disturb userspace or other kernel component" policy.
Main point of this change set is to contain target page "via buddy allocator",
where we first free the target page as we do for normal pages, and remove
from buddy only when we confirm that it reaches free list. There is surely
race window of page allocation, but that's fine because someone really want
that page and the page is still working, so soft offline can happily give up.
v4 from Oscar tries to handle the race around reallocation, but that part
seems still work in progress, so I decide to separate it for changes into
v5.9. Thank you for your contribution, Oscar.
---
Previous versions:
v1: https://lore.kernel.org/linux-mm/[email protected]/
v2: https://lore.kernel.org/linux-mm/[email protected]/
v3: https://lore.kernel.org/linux-mm/[email protected]/
v4: https://lore.kernel.org/linux-mm/[email protected]/
v5: https://lore.kernel.org/linux-mm/[email protected]/T/#t
---
Summary:
Naoya Horiguchi (5):
mm,hwpoison: cleanup unused PageHuge() check
mm, hwpoison: remove recalculating hpage
mm,hwpoison-inject: don't pin for hwpoison_filter
mm,hwpoison: introduce MF_MSG_UNSPLIT_THP
mm,hwpoison: double-check page count in __get_any_page()
Oscar Salvador (7):
mm,hwpoison: Un-export get_hwpoison_page and make it static
mm,hwpoison: Kill put_hwpoison_page
mm,hwpoison: Unify THP handling for hard and soft offline
mm,hwpoison: Rework soft offline for free pages
mm,hwpoison: Rework soft offline for in-use pages
mm,hwpoison: Refactor soft_offline_huge_page and __soft_offline_page
mm,hwpoison: Return 0 if the page is already poisoned in soft-offline
include/linux/mm.h | 3 +-
include/linux/page-flags.h | 6 +-
include/ras/ras_event.h | 3 +
mm/hwpoison-inject.c | 18 +--
mm/madvise.c | 5 -
mm/memory-failure.c | 307 +++++++++++++++++++++------------------------
mm/migrate.c | 11 +-
mm/page_alloc.c | 60 +++++++--
8 files changed, 203 insertions(+), 210 deletions(-)
From: Naoya Horiguchi <[email protected]>
Another memory error injection interface debugfs:hwpoison/corrupt-pfn
also takes bogus refcount for hwpoison_filter(). It's justified
because this does a coarse filter, expecting that memory_failure()
redoes the check for sure.
Signed-off-by: Naoya Horiguchi <[email protected]>
Signed-off-by: Oscar Salvador <[email protected]>
---
mm/hwpoison-inject.c | 18 +++++-------------
1 file changed, 5 insertions(+), 13 deletions(-)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/hwpoison-inject.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/hwpoison-inject.c
index e488876b168a..1ae1ebc2b9b1 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/hwpoison-inject.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/hwpoison-inject.c
@@ -26,11 +26,6 @@ static int hwpoison_inject(void *data, u64 val)
p = pfn_to_page(pfn);
hpage = compound_head(p);
- /*
- * This implies unable to support free buddy pages.
- */
- if (!get_hwpoison_page(p))
- return 0;
if (!hwpoison_filter_enable)
goto inject;
@@ -40,23 +35,20 @@ static int hwpoison_inject(void *data, u64 val)
* This implies unable to support non-LRU pages.
*/
if (!PageLRU(hpage) && !PageHuge(p))
- goto put_out;
+ return 0;
/*
- * do a racy check with elevated page count, to make sure PG_hwpoison
- * will only be set for the targeted owner (or on a free page).
+ * do a racy check to make sure PG_hwpoison will only be set for
+ * the targeted owner (or on a free page).
* memory_failure() will redo the check reliably inside page lock.
*/
err = hwpoison_filter(hpage);
if (err)
- goto put_out;
+ return 0;
inject:
pr_info("Injecting memory failure at pfn %#lx\n", pfn);
- return memory_failure(pfn, MF_COUNT_INCREASED);
-put_out:
- put_hwpoison_page(p);
- return 0;
+ return memory_failure(pfn, 0);
}
static int hwpoison_unpoison(void *data, u64 val)
--
2.17.1
From: Oscar Salvador <[email protected]>
Place the THP's page handling in a helper and use it
from both hard and soft-offline machinery, so we get rid
of some duplicated code.
Signed-off-by: Oscar Salvador <[email protected]>
Signed-off-by: Naoya Horiguchi <[email protected]>
---
mm/memory-failure.c | 48 +++++++++++++++++++++------------------------
1 file changed, 22 insertions(+), 26 deletions(-)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
index 6853bf3a253d..276c1588f82a 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
@@ -1103,6 +1103,25 @@ static int identify_page_state(unsigned long pfn, struct page *p,
return page_action(ps, p, pfn);
}
+static int try_to_split_thp_page(struct page *page, const char *msg)
+{
+ lock_page(page);
+ if (!PageAnon(page) || unlikely(split_huge_page(page))) {
+ unsigned long pfn = page_to_pfn(page);
+
+ unlock_page(page);
+ if (!PageAnon(page))
+ pr_info("%s: %#lx: non anonymous thp\n", msg, pfn);
+ else
+ pr_info("%s: %#lx: thp split failed\n", msg, pfn);
+ put_page(page);
+ return -EBUSY;
+ }
+ unlock_page(page);
+
+ return 0;
+}
+
static int memory_failure_hugetlb(unsigned long pfn, int flags)
{
struct page *p = pfn_to_page(pfn);
@@ -1325,21 +1344,8 @@ int memory_failure(unsigned long pfn, int flags)
}
if (PageTransHuge(hpage)) {
- lock_page(p);
- if (!PageAnon(p) || unlikely(split_huge_page(p))) {
- unlock_page(p);
- if (!PageAnon(p))
- pr_err("Memory failure: %#lx: non anonymous thp\n",
- pfn);
- else
- pr_err("Memory failure: %#lx: thp split failed\n",
- pfn);
- if (TestClearPageHWPoison(p))
- num_poisoned_pages_dec();
- put_page(p);
+ if (try_to_split_thp_page(p, "Memory Failure") < 0)
return -EBUSY;
- }
- unlock_page(p);
VM_BUG_ON_PAGE(!page_count(p), p);
}
@@ -1850,19 +1856,9 @@ static int soft_offline_in_use_page(struct page *page, int flags)
int mt;
struct page *hpage = compound_head(page);
- if (!PageHuge(page) && PageTransHuge(hpage)) {
- lock_page(page);
- if (!PageAnon(page) || unlikely(split_huge_page(page))) {
- unlock_page(page);
- if (!PageAnon(page))
- pr_info("soft offline: %#lx: non anonymous thp\n", page_to_pfn(page));
- else
- pr_info("soft offline: %#lx: thp split failed\n", page_to_pfn(page));
- put_page(page);
+ if (!PageHuge(page) && PageTransHuge(hpage))
+ if (try_to_split_thp_page(page, "soft offline") < 0)
return -EBUSY;
- }
- unlock_page(page);
- }
/*
* Setting MIGRATE_ISOLATE here ensures that the page will be linked
--
2.17.1
From: Oscar Salvador <[email protected]>
After commit 4e41a30c6d50 ("mm: hwpoison: adjust for new thp refcounting"),
put_hwpoison_page got reduced to a put_page.
Let us just use put_page instead.
Signed-off-by: Oscar Salvador <[email protected]>
Signed-off-by: Naoya Horiguchi <[email protected]>
---
include/linux/mm.h | 1 -
mm/memory-failure.c | 30 +++++++++++++++---------------
2 files changed, 15 insertions(+), 16 deletions(-)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/include/linux/mm.h v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/linux/mm.h
index 8f742373a46a..371970dfffc4 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/include/linux/mm.h
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/linux/mm.h
@@ -2985,7 +2985,6 @@ extern int memory_failure(unsigned long pfn, int flags);
extern void memory_failure_queue(unsigned long pfn, int flags);
extern void memory_failure_queue_kick(int cpu);
extern int unpoison_memory(unsigned long pfn);
-#define put_hwpoison_page(page) put_page(page)
extern int sysctl_memory_failure_early_kill;
extern int sysctl_memory_failure_recovery;
extern void shake_page(struct page *p, int access);
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
index 405c9bef6ffb..6853bf3a253d 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
@@ -1144,7 +1144,7 @@ static int memory_failure_hugetlb(unsigned long pfn, int flags)
pr_err("Memory failure: %#lx: just unpoisoned\n", pfn);
num_poisoned_pages_dec();
unlock_page(head);
- put_hwpoison_page(head);
+ put_page(head);
return 0;
}
@@ -1336,7 +1336,7 @@ int memory_failure(unsigned long pfn, int flags)
pfn);
if (TestClearPageHWPoison(p))
num_poisoned_pages_dec();
- put_hwpoison_page(p);
+ put_page(p);
return -EBUSY;
}
unlock_page(p);
@@ -1389,14 +1389,14 @@ int memory_failure(unsigned long pfn, int flags)
pr_err("Memory failure: %#lx: just unpoisoned\n", pfn);
num_poisoned_pages_dec();
unlock_page(p);
- put_hwpoison_page(p);
+ put_page(p);
return 0;
}
if (hwpoison_filter(p)) {
if (TestClearPageHWPoison(p))
num_poisoned_pages_dec();
unlock_page(p);
- put_hwpoison_page(p);
+ put_page(p);
return 0;
}
@@ -1630,9 +1630,9 @@ int unpoison_memory(unsigned long pfn)
}
unlock_page(page);
- put_hwpoison_page(page);
+ put_page(page);
if (freeit && !(pfn == my_zero_pfn(0) && page_count(p) == 1))
- put_hwpoison_page(page);
+ put_page(page);
return 0;
}
@@ -1683,7 +1683,7 @@ static int get_any_page(struct page *page, unsigned long pfn, int flags)
/*
* Try to free it.
*/
- put_hwpoison_page(page);
+ put_page(page);
shake_page(page, 1);
/*
@@ -1692,7 +1692,7 @@ static int get_any_page(struct page *page, unsigned long pfn, int flags)
ret = __get_any_page(page, pfn, 0);
if (ret == 1 && !PageLRU(page)) {
/* Drop page reference which is from __get_any_page() */
- put_hwpoison_page(page);
+ put_page(page);
pr_info("soft_offline: %#lx: unknown non LRU page type %lx (%pGp)\n",
pfn, page->flags, &page->flags);
return -EIO;
@@ -1715,7 +1715,7 @@ static int soft_offline_huge_page(struct page *page, int flags)
lock_page(hpage);
if (PageHWPoison(hpage)) {
unlock_page(hpage);
- put_hwpoison_page(hpage);
+ put_page(hpage);
pr_info("soft offline: %#lx hugepage already poisoned\n", pfn);
return -EBUSY;
}
@@ -1726,7 +1726,7 @@ static int soft_offline_huge_page(struct page *page, int flags)
* get_any_page() and isolate_huge_page() takes a refcount each,
* so need to drop one here.
*/
- put_hwpoison_page(hpage);
+ put_page(hpage);
if (!ret) {
pr_info("soft offline: %#lx hugepage failed to isolate\n", pfn);
return -EBUSY;
@@ -1779,7 +1779,7 @@ static int __soft_offline_page(struct page *page, int flags)
wait_on_page_writeback(page);
if (PageHWPoison(page)) {
unlock_page(page);
- put_hwpoison_page(page);
+ put_page(page);
pr_info("soft offline: %#lx page already poisoned\n", pfn);
return -EBUSY;
}
@@ -1794,7 +1794,7 @@ static int __soft_offline_page(struct page *page, int flags)
* would need to fix isolation locking first.
*/
if (ret == 1) {
- put_hwpoison_page(page);
+ put_page(page);
pr_info("soft_offline: %#lx: invalidated\n", pfn);
SetPageHWPoison(page);
num_poisoned_pages_inc();
@@ -1814,7 +1814,7 @@ static int __soft_offline_page(struct page *page, int flags)
* Drop page reference which is came from get_any_page()
* successful isolate_lru_page() already took another one.
*/
- put_hwpoison_page(page);
+ put_page(page);
if (!ret) {
LIST_HEAD(pagelist);
/*
@@ -1858,7 +1858,7 @@ static int soft_offline_in_use_page(struct page *page, int flags)
pr_info("soft offline: %#lx: non anonymous thp\n", page_to_pfn(page));
else
pr_info("soft offline: %#lx: thp split failed\n", page_to_pfn(page));
- put_hwpoison_page(page);
+ put_page(page);
return -EBUSY;
}
unlock_page(page);
@@ -1931,7 +1931,7 @@ int soft_offline_page(unsigned long pfn, int flags)
if (PageHWPoison(page)) {
pr_info("soft offline: %#lx page already poisoned\n", pfn);
if (flags & MF_COUNT_INCREASED)
- put_hwpoison_page(page);
+ put_page(page);
return -EBUSY;
}
--
2.17.1
From: Oscar Salvador <[email protected]>
When trying to soft-offline a free page, we need to first take it off
the buddy allocator.
Once we know is out of reach, we can safely flag it as poisoned.
take_page_off_buddy will be used to take a page meant to be poisoned
off the buddy allocator.
take_page_off_buddy calls break_down_buddy_pages, which splits a
higher-order page in case our page belongs to one.
Once the page is under our control, we call page_handle_poison to set it
as poisoned and grab a refcount on it.
Signed-off-by: Oscar Salvador <[email protected]>
Signed-off-by: Naoya Horiguchi <[email protected]>
---
ChangeLog v4 -> v5:
- fix comile error
ChangeLog v2 -> v3:
- use add_to_free_list() instead of add_to_free_area()
- use del_page_from_free_list() instead of del_page_from_free_area()
- add fast return
- move extern definition to header file as warned by checkpatch.pl
---
include/linux/page-flags.h | 1 +
mm/memory-failure.c | 18 ++++++----
mm/page_alloc.c | 68 ++++++++++++++++++++++++++++++++++++++
3 files changed, 81 insertions(+), 6 deletions(-)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/include/linux/page-flags.h v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/linux/page-flags.h
index 6be1aa559b1e..9fa5d4e2d69a 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/include/linux/page-flags.h
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/linux/page-flags.h
@@ -423,6 +423,7 @@ PAGEFLAG(HWPoison, hwpoison, PF_ANY)
TESTSCFLAG(HWPoison, hwpoison, PF_ANY)
#define __PG_HWPOISON (1UL << PG_hwpoison)
extern bool set_hwpoison_free_buddy_page(struct page *page);
+extern bool take_page_off_buddy(struct page *page);
#else
PAGEFLAG_FALSE(HWPoison)
static inline bool set_hwpoison_free_buddy_page(struct page *page)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
index 276c1588f82a..0e619012e050 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
@@ -65,6 +65,13 @@ int sysctl_memory_failure_recovery __read_mostly = 1;
atomic_long_t num_poisoned_pages __read_mostly = ATOMIC_LONG_INIT(0);
+static void page_handle_poison(struct page *page)
+{
+ SetPageHWPoison(page);
+ page_ref_inc(page);
+ num_poisoned_pages_inc();
+}
+
#if defined(CONFIG_HWPOISON_INJECT) || defined(CONFIG_HWPOISON_INJECT_MODULE)
u32 hwpoison_filter_enable = 0;
@@ -1879,14 +1886,13 @@ static int soft_offline_in_use_page(struct page *page, int flags)
static int soft_offline_free_page(struct page *page)
{
- int rc = dissolve_free_huge_page(page);
+ int rc = -EBUSY;
- if (!rc) {
- if (set_hwpoison_free_buddy_page(page))
- num_poisoned_pages_inc();
- else
- rc = -EBUSY;
+ if (!dissolve_free_huge_page(page) && take_page_off_buddy(page)) {
+ page_handle_poison(page);
+ rc = 0;
}
+
return rc;
}
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/page_alloc.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/page_alloc.c
index efe2e94c45f5..aab89f7db4ac 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/page_alloc.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/page_alloc.c
@@ -8776,6 +8776,74 @@ bool is_free_buddy_page(struct page *page)
}
#ifdef CONFIG_MEMORY_FAILURE
+/*
+ * Break down a higher-order page in sub-pages, and keep our target out of
+ * buddy allocator.
+ */
+static void break_down_buddy_pages(struct zone *zone, struct page *page,
+ struct page *target, int low, int high,
+ int migratetype)
+{
+ unsigned long size = 1 << high;
+ struct page *current_buddy, *next_page;
+
+ while (high > low) {
+ high--;
+ size >>= 1;
+
+ if (target >= &page[size]) {
+ next_page = page + size;
+ current_buddy = page;
+ } else {
+ next_page = page;
+ current_buddy = page + size;
+ }
+
+ if (set_page_guard(zone, current_buddy, high, migratetype))
+ continue;
+
+ if (current_buddy != target) {
+ add_to_free_list(current_buddy, zone, high, migratetype);
+ set_page_order(current_buddy, high);
+ page = next_page;
+ }
+ }
+}
+
+/*
+ * Take a page that will be marked as poisoned off the buddy allocator.
+ */
+bool take_page_off_buddy(struct page *page)
+{
+ struct zone *zone = page_zone(page);
+ unsigned long pfn = page_to_pfn(page);
+ unsigned long flags;
+ unsigned int order;
+ bool ret = false;
+
+ spin_lock_irqsave(&zone->lock, flags);
+ for (order = 0; order < MAX_ORDER; order++) {
+ struct page *page_head = page - (pfn & ((1 << order) - 1));
+ int buddy_order = page_order(page_head);
+
+ if (PageBuddy(page_head) && buddy_order >= order) {
+ unsigned long pfn_head = page_to_pfn(page_head);
+ int migratetype = get_pfnblock_migratetype(page_head,
+ pfn_head);
+
+ del_page_from_free_list(page_head, zone, buddy_order);
+ break_down_buddy_pages(zone, page_head, page, 0,
+ buddy_order, migratetype);
+ ret = true;
+ break;
+ }
+ if (page_count(page_head) > 0)
+ break;
+ }
+ spin_unlock_irqrestore(&zone->lock, flags);
+ return ret;
+}
+
/*
* Set PG_hwpoison flag if a given page is confirmed to be a free page. This
* test is performed under the zone lock to prevent a race against page
--
2.17.1
From: Oscar Salvador <[email protected]>
Since get_hwpoison_page is only used in memory-failure code now,
let us un-export it and make it private to that code.
Signed-off-by: Oscar Salvador <[email protected]>
Signed-off-by: Naoya Horiguchi <[email protected]>
---
include/linux/mm.h | 1 -
mm/memory-failure.c | 3 +--
2 files changed, 1 insertion(+), 3 deletions(-)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/include/linux/mm.h v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/linux/mm.h
index 5e76bb4291e6..8f742373a46a 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/include/linux/mm.h
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/linux/mm.h
@@ -2985,7 +2985,6 @@ extern int memory_failure(unsigned long pfn, int flags);
extern void memory_failure_queue(unsigned long pfn, int flags);
extern void memory_failure_queue_kick(int cpu);
extern int unpoison_memory(unsigned long pfn);
-extern int get_hwpoison_page(struct page *page);
#define put_hwpoison_page(page) put_page(page)
extern int sysctl_memory_failure_early_kill;
extern int sysctl_memory_failure_recovery;
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
index f8d200417e0f..405c9bef6ffb 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
@@ -925,7 +925,7 @@ static int page_action(struct page_state *ps, struct page *p,
* Return: return 0 if failed to grab the refcount, otherwise true (some
* non-zero value.)
*/
-int get_hwpoison_page(struct page *page)
+static int get_hwpoison_page(struct page *page)
{
struct page *head = compound_head(page);
@@ -954,7 +954,6 @@ int get_hwpoison_page(struct page *page)
return 0;
}
-EXPORT_SYMBOL_GPL(get_hwpoison_page);
/*
* Do all that is necessary to remove user space mappings. Unmap
--
2.17.1
From: Oscar Salvador <[email protected]>
Currently, there is an inconsistency when calling soft-offline from
different paths on a page that is already poisoned.
1) madvise:
madvise_inject_error skips any poisoned page and continues
the loop.
If that was the only page to madvise, it returns 0.
2) /sys/devices/system/memory/:
When calling soft_offline_page_store()->soft_offline_page(),
we return -EBUSY in case the page is already poisoned.
This is inconsistent with a) the above example and b)
memory_failure, where we return 0 if the page was poisoned.
Fix this by dropping the PageHWPoison() check in madvise_inject_error,
and let soft_offline_page return 0 if it finds the page already poisoned.
Please, note that this represents a user-api change, since now the
return error when calling soft_offline_page_store()->soft_offline_page()
will be different.
Signed-off-by: Oscar Salvador <[email protected]>
Signed-off-by: Naoya Horiguchi <[email protected]>
---
mm/madvise.c | 5 -----
mm/memory-failure.c | 4 ++--
2 files changed, 2 insertions(+), 7 deletions(-)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/madvise.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/madvise.c
index a16dba21cdf6..5fa5f66468b3 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/madvise.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/madvise.c
@@ -910,11 +910,6 @@ static int madvise_inject_error(int behavior,
*/
size = page_size(compound_head(page));
- if (PageHWPoison(page)) {
- put_page(page);
- continue;
- }
-
if (behavior == MADV_SOFT_OFFLINE) {
pr_info("Soft offlining pfn %#lx at process virtual address %#lx\n",
pfn, start);
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
index 3a2cd094b77f..61e20b7dd81c 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
@@ -1800,7 +1800,7 @@ static int __soft_offline_page(struct page *page)
unlock_page(page);
put_page(page);
pr_info("soft offline: %#lx page already poisoned\n", pfn);
- return -EBUSY;
+ return 0;
}
if (!PageHuge(page))
@@ -1904,7 +1904,7 @@ int soft_offline_page(unsigned long pfn, int flags)
pr_info("soft offline: %#lx page already poisoned\n", pfn);
if (flags & MF_COUNT_INCREASED)
put_page(page);
- return -EBUSY;
+ return 0;
}
get_online_mems();
--
2.17.1
From: Naoya Horiguchi <[email protected]>
memory_failure() is supposed to call action_result() when it handles
a memory error event, but there's one missing case. So let's add it.
I find that include/ras/ras_event.h has some other MF_MSG_* undefined,
so this patch also adds them.
Signed-off-by: Naoya Horiguchi <[email protected]>
Signed-off-by: Oscar Salvador <[email protected]>
---
include/linux/mm.h | 1 +
include/ras/ras_event.h | 3 +++
mm/memory-failure.c | 5 ++++-
3 files changed, 8 insertions(+), 1 deletion(-)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/include/linux/mm.h v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/linux/mm.h
index 371970dfffc4..442921a004a2 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/include/linux/mm.h
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/linux/mm.h
@@ -3024,6 +3024,7 @@ enum mf_action_page_type {
MF_MSG_BUDDY,
MF_MSG_BUDDY_2ND,
MF_MSG_DAX,
+ MF_MSG_UNSPLIT_THP,
MF_MSG_UNKNOWN,
};
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/include/ras/ras_event.h v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/ras/ras_event.h
index 36c5c5e38c1d..0bdbc0d17d2f 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/include/ras/ras_event.h
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/ras/ras_event.h
@@ -361,6 +361,7 @@ TRACE_EVENT(aer_event,
EM ( MF_MSG_POISONED_HUGE, "huge page already hardware poisoned" ) \
EM ( MF_MSG_HUGE, "huge page" ) \
EM ( MF_MSG_FREE_HUGE, "free huge page" ) \
+ EM ( MF_MSG_NON_PMD_HUGE, "non-pmd-sized huge page" ) \
EM ( MF_MSG_UNMAP_FAILED, "unmapping failed page" ) \
EM ( MF_MSG_DIRTY_SWAPCACHE, "dirty swapcache page" ) \
EM ( MF_MSG_CLEAN_SWAPCACHE, "clean swapcache page" ) \
@@ -373,6 +374,8 @@ TRACE_EVENT(aer_event,
EM ( MF_MSG_TRUNCATED_LRU, "already truncated LRU page" ) \
EM ( MF_MSG_BUDDY, "free buddy page" ) \
EM ( MF_MSG_BUDDY_2ND, "free buddy page (2nd try)" ) \
+ EM ( MF_MSG_DAX, "dax page" ) \
+ EM ( MF_MSG_UNSPLIT_THP, "unsplit thp" ) \
EMe ( MF_MSG_UNKNOWN, "unknown page" )
/*
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
index 61e20b7dd81c..bed4b6aac9a0 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
@@ -583,6 +583,7 @@ static const char * const action_page_types[] = {
[MF_MSG_BUDDY] = "free buddy page",
[MF_MSG_BUDDY_2ND] = "free buddy page (2nd try)",
[MF_MSG_DAX] = "dax page",
+ [MF_MSG_UNSPLIT_THP] = "unsplit thp",
[MF_MSG_UNKNOWN] = "unknown page",
};
@@ -1373,8 +1374,10 @@ int memory_failure(unsigned long pfn, int flags)
}
if (PageTransHuge(hpage)) {
- if (try_to_split_thp_page(p, "Memory Failure") < 0)
+ if (try_to_split_thp_page(p, "Memory Failure") < 0) {
+ action_result(pfn, MF_MSG_UNSPLIT_THP, MF_IGNORED);
return -EBUSY;
+ }
VM_BUG_ON_PAGE(!page_count(p), p);
}
--
2.17.1
From: Naoya Horiguchi <[email protected]>
hpage is never used after try_to_split_thp_page() in memory_failure(),
so we don't have to update hpage. So let's not recalculate/use hpage.
Suggested-by: "Aneesh Kumar K.V" <[email protected]>
Signed-off-by: Naoya Horiguchi <[email protected]>
Signed-off-by: Oscar Salvador <[email protected]>
Reviewed-by: Mike Kravetz <[email protected]>
---
mm/memory-failure.c | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
index 3d2d61f1c6e9..f8d200417e0f 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
@@ -1342,7 +1342,6 @@ int memory_failure(unsigned long pfn, int flags)
}
unlock_page(p);
VM_BUG_ON_PAGE(!page_count(p), p);
- hpage = compound_head(p);
}
/*
@@ -1414,11 +1413,8 @@ int memory_failure(unsigned long pfn, int flags)
/*
* Now take care of user space mappings.
* Abort on fail: __delete_from_page_cache() assumes unmapped page.
- *
- * When the raw error page is thp tail page, hpage points to the raw
- * page after thp split.
*/
- if (!hwpoison_user_mappings(p, pfn, flags, &hpage)) {
+ if (!hwpoison_user_mappings(p, pfn, flags, &p)) {
action_result(pfn, MF_MSG_UNMAP_FAILED, MF_IGNORED);
res = -EBUSY;
goto out;
--
2.17.1
From: Naoya Horiguchi <[email protected]>
Soft offlining could fail with EIO due to the race condition with
hugepage migration. This issuse became visible due to the change by
previous patch that makes soft offline handler take page refcount
by its own. We have no way to directly pin zero refcount page, and
the page considered as a zero refcount page could be allocated just
after the first check.
This patch adds the second check to find the race and gives us
chance to handle it more reliably.
Reported-by: Qian Cai <[email protected]>
Signed-off-by: Naoya Horiguchi <[email protected]>
---
mm/memory-failure.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
index bed4b6aac9a0..e6f6559d573e 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
@@ -1700,6 +1700,9 @@ static int __get_any_page(struct page *p, unsigned long pfn, int flags)
} else if (is_free_buddy_page(p)) {
pr_info("%s: %#lx free buddy page\n", __func__, pfn);
ret = 0;
+ } else if (page_count(p)) {
+ /* raced with allocation */
+ ret = -EBUSY;
} else {
pr_info("%s: %#lx: unknown zero refcount page type %lx\n",
__func__, pfn, p->flags);
@@ -1716,6 +1719,9 @@ static int get_any_page(struct page *page, unsigned long pfn, int flags)
{
int ret = __get_any_page(page, pfn, flags);
+ if (ret == -EBUSY)
+ ret = __get_any_page(page, pfn, flags);
+
if (ret == 1 && !PageHuge(page) &&
!PageLRU(page) && !__PageMovable(page)) {
/*
--
2.17.1
From: Naoya Horiguchi <[email protected]>
Drop the PageHuge check, which is dead code since memory_failure() forks
into memory_failure_hugetlb() for hugetlb pages.
memory_failure() and memory_failure_hugetlb() shares some functions like
hwpoison_user_mappings() and identify_page_state(), so they should properly
handle 4kB page, thp, and hugetlb.
Signed-off-by: Naoya Horiguchi <[email protected]>
Signed-off-by: Oscar Salvador <[email protected]>
Reviewed-by: Mike Kravetz <[email protected]>
---
mm/memory-failure.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
index fe53768e0793..3d2d61f1c6e9 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
@@ -1382,10 +1382,7 @@ int memory_failure(unsigned long pfn, int flags)
* page_remove_rmap() in try_to_unmap_one(). So to determine page status
* correctly, we save a copy of the page flags at this time.
*/
- if (PageHuge(p))
- page_flags = hpage->flags;
- else
- page_flags = p->flags;
+ page_flags = p->flags;
/*
* unpoison always clear PG_hwpoison inside page lock
--
2.17.1
From: Oscar Salvador <[email protected]>
This patch changes the way we set and handle in-use poisoned pages.
Until now, poisoned pages were released to the buddy allocator, trusting
that the checks that take place prior to hand the page would act as a
safe net and would skip that page.
This has proved to be wrong, as we got some pfn walkers out there, like
compaction, that all they care is the page to be PageBuddy and be in a
freelist.
Although this might not be the only user, having poisoned pages
in the buddy allocator seems a bad idea as we should only have
free pages that are ready and meant to be used as such.
Before explaining the taken approach, let us break down the kind
of pages we can soft offline.
- Anonymous THP (after the split, they end up being 4K pages)
- Hugetlb
- Order-0 pages (that can be either migrated or invalited)
* Normal pages (order-0 and anon-THP)
- If they are clean and unmapped page cache pages, we invalidate
then by means of invalidate_inode_page().
- If they are mapped/dirty, we do the isolate-and-migrate dance.
Either way, do not call put_page directly from those paths.
Instead, we keep the page and send it to page_set_poison to perform the
right handling.
page_set_poison sets the HWPoison flag and does the last put_page.
This call to put_page is mainly to be able to call __page_cache_release,
since this function is not exported.
Down the chain, we placed a check for HWPoison page in
free_pages_prepare, that just skips any poisoned page, so those pages
do not end up in any pcplist/freelist.
After that, we set the refcount on the page to 1 and we increment
the poisoned pages counter.
We could do as we do for free pages:
1) wait until the page hits buddy's freelists
2) take it off
3) flag it
The problem is that we could race with an allocation, so by the time we
want to take the page off the buddy, the page is already allocated, so we
cannot soft-offline it.
This is not fatal of course, but if it is better if we can close the race
as does not require a lot of code.
* Hugetlb pages
- We isolate-and-migrate them
After the migration has been successful, we call dissolve_free_huge_page,
and we set HWPoison on the page if we succeed.
Hugetlb has a slightly different handling though.
While for non-hugetlb pages we cared about closing the race with an
allocation, doing so for hugetlb pages requires quite some additional
code (we would need to hook in free_huge_page and some other places).
So I decided to not make the code overly complicated and just fail
normally if the page we allocated in the meantime.
Because of the way we handle now in-use pages, we no longer need the
put-as-isolation-migratetype dance, that was guarding for poisoned pages
to end up in pcplists.
Signed-off-by: Oscar Salvador <[email protected]>
Signed-off-by: Naoya Horiguchi <[email protected]>
---
include/linux/page-flags.h | 5 -----
mm/memory-failure.c | 45 ++++++++++++++------------------------
mm/migrate.c | 11 +++-------
mm/page_alloc.c | 28 ------------------------
4 files changed, 19 insertions(+), 70 deletions(-)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/include/linux/page-flags.h v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/linux/page-flags.h
index 9fa5d4e2d69a..d1df51ed6eeb 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/include/linux/page-flags.h
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/include/linux/page-flags.h
@@ -422,14 +422,9 @@ PAGEFLAG_FALSE(Uncached)
PAGEFLAG(HWPoison, hwpoison, PF_ANY)
TESTSCFLAG(HWPoison, hwpoison, PF_ANY)
#define __PG_HWPOISON (1UL << PG_hwpoison)
-extern bool set_hwpoison_free_buddy_page(struct page *page);
extern bool take_page_off_buddy(struct page *page);
#else
PAGEFLAG_FALSE(HWPoison)
-static inline bool set_hwpoison_free_buddy_page(struct page *page)
-{
- return 0;
-}
#define __PG_HWPOISON 0
#endif
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
index 0e619012e050..95bf8aa44a9a 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
@@ -65,8 +65,12 @@ int sysctl_memory_failure_recovery __read_mostly = 1;
atomic_long_t num_poisoned_pages __read_mostly = ATOMIC_LONG_INIT(0);
-static void page_handle_poison(struct page *page)
+static void page_handle_poison(struct page *page, bool release)
{
+ if (release) {
+ put_page(page);
+ drain_all_pages(page_zone(page));
+ }
SetPageHWPoison(page);
page_ref_inc(page);
num_poisoned_pages_inc();
@@ -1756,19 +1760,13 @@ static int soft_offline_huge_page(struct page *page, int flags)
ret = -EIO;
} else {
/*
- * We set PG_hwpoison only when the migration source hugepage
- * was successfully dissolved, because otherwise hwpoisoned
- * hugepage remains on free hugepage list, then userspace will
- * find it as SIGBUS by allocation failure. That's not expected
- * in soft-offlining.
+ * We set PG_hwpoison only when we were able to take the page
+ * off the buddy.
*/
- ret = dissolve_free_huge_page(page);
- if (!ret) {
- if (set_hwpoison_free_buddy_page(page))
- num_poisoned_pages_inc();
- else
- ret = -EBUSY;
- }
+ if (!dissolve_free_huge_page(page) && take_page_off_buddy(page))
+ page_handle_poison(page, false);
+ else
+ ret = -EBUSY;
}
return ret;
}
@@ -1807,10 +1805,8 @@ static int __soft_offline_page(struct page *page, int flags)
* would need to fix isolation locking first.
*/
if (ret == 1) {
- put_page(page);
pr_info("soft_offline: %#lx: invalidated\n", pfn);
- SetPageHWPoison(page);
- num_poisoned_pages_inc();
+ page_handle_poison(page, true);
return 0;
}
@@ -1841,7 +1837,9 @@ static int __soft_offline_page(struct page *page, int flags)
list_add(&page->lru, &pagelist);
ret = migrate_pages(&pagelist, alloc_migration_target, NULL,
(unsigned long)&mtc, MIGRATE_SYNC, MR_MEMORY_FAILURE);
- if (ret) {
+ if (!ret) {
+ page_handle_poison(page, true);
+ } else {
if (!list_empty(&pagelist))
putback_movable_pages(&pagelist);
@@ -1860,27 +1858,16 @@ static int __soft_offline_page(struct page *page, int flags)
static int soft_offline_in_use_page(struct page *page, int flags)
{
int ret;
- int mt;
struct page *hpage = compound_head(page);
if (!PageHuge(page) && PageTransHuge(hpage))
if (try_to_split_thp_page(page, "soft offline") < 0)
return -EBUSY;
- /*
- * Setting MIGRATE_ISOLATE here ensures that the page will be linked
- * to free list immediately (not via pcplist) when released after
- * successful page migration. Otherwise we can't guarantee that the
- * page is really free after put_page() returns, so
- * set_hwpoison_free_buddy_page() highly likely fails.
- */
- mt = get_pageblock_migratetype(page);
- set_pageblock_migratetype(page, MIGRATE_ISOLATE);
if (PageHuge(page))
ret = soft_offline_huge_page(page, flags);
else
ret = __soft_offline_page(page, flags);
- set_pageblock_migratetype(page, mt);
return ret;
}
@@ -1889,7 +1876,7 @@ static int soft_offline_free_page(struct page *page)
int rc = -EBUSY;
if (!dissolve_free_huge_page(page) && take_page_off_buddy(page)) {
- page_handle_poison(page);
+ page_handle_poison(page, false);
rc = 0;
}
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/migrate.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/migrate.c
index 2c809ffcf0e1..d7a9379c343b 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/migrate.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/migrate.c
@@ -1222,16 +1222,11 @@ static int unmap_and_move(new_page_t get_new_page,
* we want to retry.
*/
if (rc == MIGRATEPAGE_SUCCESS) {
- put_page(page);
- if (reason == MR_MEMORY_FAILURE) {
+ if (reason != MR_MEMORY_FAILURE)
/*
- * Set PG_HWPoison on just freed page
- * intentionally. Although it's rather weird,
- * it's how HWPoison flag works at the moment.
+ * We release the page in page_handle_poison.
*/
- if (set_hwpoison_free_buddy_page(page))
- num_poisoned_pages_inc();
- }
+ put_page(page);
} else {
if (rc != -EAGAIN) {
if (likely(!__PageMovable(page))) {
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/page_alloc.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/page_alloc.c
index aab89f7db4ac..e4896e674594 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/page_alloc.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/page_alloc.c
@@ -8843,32 +8843,4 @@ bool take_page_off_buddy(struct page *page)
spin_unlock_irqrestore(&zone->lock, flags);
return ret;
}
-
-/*
- * Set PG_hwpoison flag if a given page is confirmed to be a free page. This
- * test is performed under the zone lock to prevent a race against page
- * allocation.
- */
-bool set_hwpoison_free_buddy_page(struct page *page)
-{
- struct zone *zone = page_zone(page);
- unsigned long pfn = page_to_pfn(page);
- unsigned long flags;
- unsigned int order;
- bool hwpoisoned = false;
-
- spin_lock_irqsave(&zone->lock, flags);
- for (order = 0; order < MAX_ORDER; order++) {
- struct page *page_head = page - (pfn & ((1 << order) - 1));
-
- if (PageBuddy(page_head) && page_order(page_head) >= order) {
- if (!TestSetPageHWPoison(page))
- hwpoisoned = true;
- break;
- }
- }
- spin_unlock_irqrestore(&zone->lock, flags);
-
- return hwpoisoned;
-}
#endif
--
2.17.1
From: Oscar Salvador <[email protected]>
Merging soft_offline_huge_page and __soft_offline_page let us get rid of
quite some duplicated code, and makes the code much easier to follow.
Now, __soft_offline_page will handle both normal and hugetlb pages.
Note that move put_page() block to the beginning of page_handle_poison()
with drain_all_pages() in order to make sure that the target page is
freed and sent into free list to make take_page_off_buddy() work properly.
Signed-off-by: Oscar Salvador <[email protected]>
Signed-off-by: Naoya Horiguchi <[email protected]>
---
ChangeLog v4 -> v5:
- use "char const *msg_page[]" instead of "const char *msg_page[]"
- move adding drain_all_pages() to 12/16
ChangeLog v2 -> v3:
- use page_is_file_lru() instead of page_is_file_cache(),
- add description about put_page() and drain_all_pages().
- fix coding style warnings by checkpatch.pl
---
mm/memory-failure.c | 183 ++++++++++++++++++++------------------------
1 file changed, 83 insertions(+), 100 deletions(-)
diff --git v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
index 95bf8aa44a9a..3a2cd094b77f 100644
--- v5.8-rc7-mmotm-2020-07-27-18-18/mm/memory-failure.c
+++ v5.8-rc7-mmotm-2020-07-27-18-18_patched/mm/memory-failure.c
@@ -65,15 +65,33 @@ int sysctl_memory_failure_recovery __read_mostly = 1;
atomic_long_t num_poisoned_pages __read_mostly = ATOMIC_LONG_INIT(0);
-static void page_handle_poison(struct page *page, bool release)
+static bool page_handle_poison(struct page *page, bool hugepage_or_freepage, bool release)
{
if (release) {
put_page(page);
drain_all_pages(page_zone(page));
}
+
+ if (hugepage_or_freepage) {
+ /*
+ * Doing this check for free pages is also fine since dissolve_free_huge_page
+ * returns 0 for non-hugetlb pages as well.
+ */
+ if (dissolve_free_huge_page(page) || !take_page_off_buddy(page))
+ /*
+ * We could fail to take off the target page from buddy
+ * for example due to racy page allocaiton, but that's
+ * acceptable because soft-offlined page is not broken
+ * and if someone really want to use it, they should
+ * take it.
+ */
+ return false;
+ }
+
SetPageHWPoison(page);
page_ref_inc(page);
num_poisoned_pages_inc();
+ return true;
}
#if defined(CONFIG_HWPOISON_INJECT) || defined(CONFIG_HWPOISON_INJECT_MODULE)
@@ -1718,63 +1736,52 @@ static int get_any_page(struct page *page, unsigned long pfn, int flags)
return ret;
}
-static int soft_offline_huge_page(struct page *page, int flags)
+static bool isolate_page(struct page *page, struct list_head *pagelist)
{
- int ret;
- unsigned long pfn = page_to_pfn(page);
- struct page *hpage = compound_head(page);
- LIST_HEAD(pagelist);
+ bool isolated = false;
+ bool lru = PageLRU(page);
+
+ if (PageHuge(page)) {
+ isolated = isolate_huge_page(page, pagelist);
+ } else {
+ if (lru)
+ isolated = !isolate_lru_page(page);
+ else
+ isolated = !isolate_movable_page(page, ISOLATE_UNEVICTABLE);
+
+ if (isolated)
+ list_add(&page->lru, pagelist);
- /*
- * This double-check of PageHWPoison is to avoid the race with
- * memory_failure(). See also comment in __soft_offline_page().
- */
- lock_page(hpage);
- if (PageHWPoison(hpage)) {
- unlock_page(hpage);
- put_page(hpage);
- pr_info("soft offline: %#lx hugepage already poisoned\n", pfn);
- return -EBUSY;
}
- unlock_page(hpage);
- ret = isolate_huge_page(hpage, &pagelist);
+ if (isolated && lru)
+ inc_node_page_state(page, NR_ISOLATED_ANON +
+ page_is_file_lru(page));
+
/*
- * get_any_page() and isolate_huge_page() takes a refcount each,
- * so need to drop one here.
+ * If we succeed to isolate the page, we grabbed another refcount on
+ * the page, so we can safely drop the one we got from get_any_pages().
+ * If we failed to isolate the page, it means that we cannot go further
+ * and we will return an error, so drop the reference we got from
+ * get_any_pages() as well.
*/
- put_page(hpage);
- if (!ret) {
- pr_info("soft offline: %#lx hugepage failed to isolate\n", pfn);
- return -EBUSY;
- }
-
- ret = migrate_pages(&pagelist, new_page, NULL, MPOL_MF_MOVE_ALL,
- MIGRATE_SYNC, MR_MEMORY_FAILURE);
- if (ret) {
- pr_info("soft offline: %#lx: hugepage migration failed %d, type %lx (%pGp)\n",
- pfn, ret, page->flags, &page->flags);
- if (!list_empty(&pagelist))
- putback_movable_pages(&pagelist);
- if (ret > 0)
- ret = -EIO;
- } else {
- /*
- * We set PG_hwpoison only when we were able to take the page
- * off the buddy.
- */
- if (!dissolve_free_huge_page(page) && take_page_off_buddy(page))
- page_handle_poison(page, false);
- else
- ret = -EBUSY;
- }
- return ret;
+ put_page(page);
+ return isolated;
}
-static int __soft_offline_page(struct page *page, int flags)
+/*
+ * __soft_offline_page handles hugetlb-pages and non-hugetlb pages.
+ * If the page is a non-dirty unmapped page-cache page, it simply invalidates.
+ * If the page is mapped, it migrates the contents over.
+ */
+static int __soft_offline_page(struct page *page)
{
- int ret;
+ int ret = 0;
unsigned long pfn = page_to_pfn(page);
+ struct page *hpage = compound_head(page);
+ char const *msg_page[] = {"page", "hugepage"};
+ bool huge = PageHuge(page);
+ LIST_HEAD(pagelist);
struct migration_target_control mtc = {
.nid = NUMA_NO_NODE,
.gfp_mask = GFP_USER | __GFP_MOVABLE | __GFP_RETRY_MAYFAIL,
@@ -1787,98 +1794,74 @@ static int __soft_offline_page(struct page *page, int flags)
* so there's no race between soft_offline_page() and memory_failure().
*/
lock_page(page);
- wait_on_page_writeback(page);
+ if (!PageHuge(page))
+ wait_on_page_writeback(page);
if (PageHWPoison(page)) {
unlock_page(page);
put_page(page);
pr_info("soft offline: %#lx page already poisoned\n", pfn);
return -EBUSY;
}
- /*
- * Try to invalidate first. This should work for
- * non dirty unmapped page cache pages.
- */
- ret = invalidate_inode_page(page);
+
+ if (!PageHuge(page))
+ /*
+ * Try to invalidate first. This should work for
+ * non dirty unmapped page cache pages.
+ */
+ ret = invalidate_inode_page(page);
unlock_page(page);
+
/*
* RED-PEN would be better to keep it isolated here, but we
* would need to fix isolation locking first.
*/
- if (ret == 1) {
+ if (ret) {
pr_info("soft_offline: %#lx: invalidated\n", pfn);
- page_handle_poison(page, true);
+ page_handle_poison(page, false, true);
return 0;
}
- /*
- * Simple invalidation didn't work.
- * Try to migrate to a new page instead. migrate.c
- * handles a large number of cases for us.
- */
- if (PageLRU(page))
- ret = isolate_lru_page(page);
- else
- ret = isolate_movable_page(page, ISOLATE_UNEVICTABLE);
- /*
- * Drop page reference which is came from get_any_page()
- * successful isolate_lru_page() already took another one.
- */
- put_page(page);
- if (!ret) {
- LIST_HEAD(pagelist);
- /*
- * After isolated lru page, the PageLRU will be cleared,
- * so use !__PageMovable instead for LRU page's mapping
- * cannot have PAGE_MAPPING_MOVABLE.
- */
- if (!__PageMovable(page))
- inc_node_page_state(page, NR_ISOLATED_ANON +
- page_is_file_lru(page));
- list_add(&page->lru, &pagelist);
+ if (isolate_page(hpage, &pagelist)) {
ret = migrate_pages(&pagelist, alloc_migration_target, NULL,
(unsigned long)&mtc, MIGRATE_SYNC, MR_MEMORY_FAILURE);
if (!ret) {
- page_handle_poison(page, true);
+ bool release = !huge;
+
+ if (!page_handle_poison(page, true, release))
+ ret = -EBUSY;
} else {
if (!list_empty(&pagelist))
putback_movable_pages(&pagelist);
- pr_info("soft offline: %#lx: migration failed %d, type %lx (%pGp)\n",
- pfn, ret, page->flags, &page->flags);
+
+ pr_info("soft offline: %#lx: %s migration failed %d, type %lx (%pGp)\n",
+ pfn, msg_page[huge], ret, page->flags, &page->flags);
if (ret > 0)
ret = -EIO;
}
} else {
- pr_info("soft offline: %#lx: isolation failed: %d, page count %d, type %lx (%pGp)\n",
- pfn, ret, page_count(page), page->flags, &page->flags);
+ pr_info("soft offline: %#lx: %s isolation failed: %d, page count %d, type %lx (%pGp)\n",
+ pfn, msg_page[huge], ret, page_count(page), page->flags, &page->flags);
}
return ret;
}
-static int soft_offline_in_use_page(struct page *page, int flags)
+static int soft_offline_in_use_page(struct page *page)
{
- int ret;
struct page *hpage = compound_head(page);
if (!PageHuge(page) && PageTransHuge(hpage))
if (try_to_split_thp_page(page, "soft offline") < 0)
return -EBUSY;
-
- if (PageHuge(page))
- ret = soft_offline_huge_page(page, flags);
- else
- ret = __soft_offline_page(page, flags);
- return ret;
+ return __soft_offline_page(page);
}
static int soft_offline_free_page(struct page *page)
{
- int rc = -EBUSY;
+ int rc = 0;
- if (!dissolve_free_huge_page(page) && take_page_off_buddy(page)) {
- page_handle_poison(page, false);
- rc = 0;
- }
+ if (!page_handle_poison(page, true, false))
+ rc = -EBUSY;
return rc;
}
@@ -1929,7 +1912,7 @@ int soft_offline_page(unsigned long pfn, int flags)
put_online_mems();
if (ret > 0)
- ret = soft_offline_in_use_page(page, flags);
+ ret = soft_offline_in_use_page(page);
else if (ret == 0)
ret = soft_offline_free_page(page);
--
2.17.1
On Thu, Aug 06, 2020 at 06:49:11PM +0000, [email protected] wrote:
> Hi,
>
> This patchset is the latest version of soft offline rework patchset
> targetted for v5.9.
>
> Since v5, I dropped some patches which tweak refcount handling in
> madvise_inject_error() to avoid the "unknown refcount page" error.
> I don't confirm the fix (that didn't reproduce with v5 in my environment),
> but this change surely call soft_offline_page() after holding refcount,
> so the error should not happen any more.
With this patchset, arm64 is still suffering from premature 512M-size hugepages
allocation failures.
# git clone https://gitlab.com/cailca/linux-mm
# cd linux-mm; make
# ./random 1
- start: migrate_huge_offline
- use NUMA nodes 0,1.
- mmap and free 2147483648 bytes hugepages on node 0
- mmap and free 2147483648 bytes hugepages on node 1
madvise: Cannot allocate memory
[ 292.456538][ T3685] soft offline: 0x8a000: hugepage isolation failed: 0, page count 2, type 7ffff80001000e (referenced|uptodate|dirty|head)
[ 292.469113][ T3685] Soft offlining pfn 0x8c000 at process virtual address 0xffff60000000
[ 292.983855][ T3685] Soft offlining pfn 0x88000 at process virtual address 0xffff40000000
[ 293.271369][ T3685] Soft offlining pfn 0x8a000 at process virtual address 0xffff60000000
[ 293.834030][ T3685] Soft offlining pfn 0xa000 at process virtual address 0xffff40000000
[ 293.851378][ T3685] soft offline: 0xa000: hugepage migration failed -12, type 7ffff80001000e (referenced|uptodate|dirty|head)
The fresh-booted system still had 40G+ memory free before running the test.
Reverting the following commits allowed the test to run succesfully over and over again.
"mm, hwpoison: remove recalculating hpage"
"mm,hwpoison-inject: don't pin for hwpoison_filter"
"mm,hwpoison: Un-export get_hwpoison_page and make it static"
"mm,hwpoison: kill put_hwpoison_page"
"mm,hwpoison: unify THP handling for hard and soft offline"
"mm,hwpoison: rework soft offline for free pages"
"mm,hwpoison: rework soft offline for in-use pages"
"mm,hwpoison: refactor soft_offline_huge_page and __soft_offline_page"
i.e., it is not enough to only revert,
mm,hwpoison: double-check page count in __get_any_page()
mm,hwpoison: introduce MF_MSG_UNSPLIT_THP
mm,hwpoison: return 0 if the page is already poisoned in soft-offline
>
> Dropped patches
> - mm,madvise: call soft_offline_page() without MF_COUNT_INCREASED
> - mm,madvise: Refactor madvise_inject_error
> - mm,hwpoison: remove MF_COUNT_INCREASED
> - mm,hwpoison: remove flag argument from soft offline functions
>
> Thanks,
> Naoya Horiguchi
>
> Quoting cover letter of v5:
> ----
> Main focus of this series is to stabilize soft offline. Historically soft
> offlined pages have suffered from racy conditions because PageHWPoison is
> used to a little too aggressively, which (directly or indirectly) invades
> other mm code which cares little about hwpoison. This results in unexpected
> behavior or kernel panic, which is very far from soft offline's "do not
> disturb userspace or other kernel component" policy.
>
> Main point of this change set is to contain target page "via buddy allocator",
> where we first free the target page as we do for normal pages, and remove
> from buddy only when we confirm that it reaches free list. There is surely
> race window of page allocation, but that's fine because someone really want
> that page and the page is still working, so soft offline can happily give up.
>
> v4 from Oscar tries to handle the race around reallocation, but that part
> seems still work in progress, so I decide to separate it for changes into
> v5.9. Thank you for your contribution, Oscar.
>
> ---
> Previous versions:
> v1: https://lore.kernel.org/linux-mm/[email protected]/
> v2: https://lore.kernel.org/linux-mm/[email protected]/
> v3: https://lore.kernel.org/linux-mm/[email protected]/
> v4: https://lore.kernel.org/linux-mm/[email protected]/
> v5: https://lore.kernel.org/linux-mm/[email protected]/T/#t
> ---
> Summary:
>
> Naoya Horiguchi (5):
> mm,hwpoison: cleanup unused PageHuge() check
> mm, hwpoison: remove recalculating hpage
> mm,hwpoison-inject: don't pin for hwpoison_filter
> mm,hwpoison: introduce MF_MSG_UNSPLIT_THP
> mm,hwpoison: double-check page count in __get_any_page()
>
> Oscar Salvador (7):
> mm,hwpoison: Un-export get_hwpoison_page and make it static
> mm,hwpoison: Kill put_hwpoison_page
> mm,hwpoison: Unify THP handling for hard and soft offline
> mm,hwpoison: Rework soft offline for free pages
> mm,hwpoison: Rework soft offline for in-use pages
> mm,hwpoison: Refactor soft_offline_huge_page and __soft_offline_page
> mm,hwpoison: Return 0 if the page is already poisoned in soft-offline
>
> include/linux/mm.h | 3 +-
> include/linux/page-flags.h | 6 +-
> include/ras/ras_event.h | 3 +
> mm/hwpoison-inject.c | 18 +--
> mm/madvise.c | 5 -
> mm/memory-failure.c | 307 +++++++++++++++++++++------------------------
> mm/migrate.c | 11 +-
> mm/page_alloc.c | 60 +++++++--
> 8 files changed, 203 insertions(+), 210 deletions(-)
On Mon, Aug 10, 2020 at 11:22:55AM -0400, Qian Cai wrote:
> On Thu, Aug 06, 2020 at 06:49:11PM +0000, [email protected] wrote:
> > Hi,
> >
> > This patchset is the latest version of soft offline rework patchset
> > targetted for v5.9.
> >
> > Since v5, I dropped some patches which tweak refcount handling in
> > madvise_inject_error() to avoid the "unknown refcount page" error.
> > I don't confirm the fix (that didn't reproduce with v5 in my environment),
> > but this change surely call soft_offline_page() after holding refcount,
> > so the error should not happen any more.
>
> With this patchset, arm64 is still suffering from premature 512M-size hugepages
> allocation failures.
>
> # git clone https://gitlab.com/cailca/linux-mm
> # cd linux-mm; make
> # ./random 1
> - start: migrate_huge_offline
> - use NUMA nodes 0,1.
> - mmap and free 2147483648 bytes hugepages on node 0
> - mmap and free 2147483648 bytes hugepages on node 1
> madvise: Cannot allocate memory
>
> [ 292.456538][ T3685] soft offline: 0x8a000: hugepage isolation failed: 0, page count 2, type 7ffff80001000e (referenced|uptodate|dirty|head)
> [ 292.469113][ T3685] Soft offlining pfn 0x8c000 at process virtual address 0xffff60000000
> [ 292.983855][ T3685] Soft offlining pfn 0x88000 at process virtual address 0xffff40000000
> [ 293.271369][ T3685] Soft offlining pfn 0x8a000 at process virtual address 0xffff60000000
> [ 293.834030][ T3685] Soft offlining pfn 0xa000 at process virtual address 0xffff40000000
> [ 293.851378][ T3685] soft offline: 0xa000: hugepage migration failed -12, type 7ffff80001000e (referenced|uptodate|dirty|head)
>
> The fresh-booted system still had 40G+ memory free before running the test.
As I commented over v5, this failure is expected and it doesn't mean kernel
issue. Once we successfully soft offline a hugepage, the memory range
covering the hugepage will never participate in hugepage because one of the
subpages is removed from buddy. So if you iterate soft offlining hugepages,
all memory range are "holed" finally, and no hugepage will be available in
the system.
Please fix your test program to properly determine nubmer of loop (NR_LOOP)
so that you can assume that you can always allocate hugepage during testing.
For example, if you can use 40G memory and hugepage size is 512MB, NR_LOOP
should not be larger than 80.
>
> Reverting the following commits allowed the test to run succesfully over and over again.
>
> "mm, hwpoison: remove recalculating hpage"
> "mm,hwpoison-inject: don't pin for hwpoison_filter"
> "mm,hwpoison: Un-export get_hwpoison_page and make it static"
> "mm,hwpoison: kill put_hwpoison_page"
> "mm,hwpoison: unify THP handling for hard and soft offline"
> "mm,hwpoison: rework soft offline for free pages"
> "mm,hwpoison: rework soft offline for in-use pages"
> "mm,hwpoison: refactor soft_offline_huge_page and __soft_offline_page"
I'm still not sure why the test succeeded by reverting these because
current mainline kernel provides similar mechanism to prevent reuse of
soft offlined page. So this success seems to me something suspicious.
To investigate more, I want to have additional info about the page states
of the relevant pages after soft offlining. Could you collect it by the
following steps?
- modify random.c not to run hotplug_memory() in migrate_huge_hotplug_memory(),
- compile it and run "./random 1" once,
- to collect page state with hwpoisoned pages, run "./page-types -Nlr -b hwpoison",
where page-types is available under tools/vm in kernel source tree.
- choose a few pfns of soft offlined pages from kernel message
"Soft offlining pfn ...", and run "./page-types -Nlr -a <pfn>".
Thanks,
Naoya Horiguchi
>
> i.e., it is not enough to only revert,
>
> mm,hwpoison: double-check page count in __get_any_page()
> mm,hwpoison: introduce MF_MSG_UNSPLIT_THP
> mm,hwpoison: return 0 if the page is already poisoned in soft-offline
>
> On Aug 10, 2020, at 11:11 PM, HORIGUCHI NAOYA(堀口 直也) <[email protected]> wrote:
>
> I'm still not sure why the test succeeded by reverting these because
> current mainline kernel provides similar mechanism to prevent reuse of
> soft offlined page. So this success seems to me something suspicious.
Even if we call munmap() on the range, it still can’t be be reused? If so, how to recover those memory then?
>
> To investigate more, I want to have additional info about the page states
> of the relevant pages after soft offlining. Could you collect it by the
> following steps?
Do you want to collect those from the failed or succeed kernel?
On Mon, Aug 10, 2020 at 11:45:36PM -0400, Qian Cai wrote:
>
>
> > On Aug 10, 2020, at 11:11 PM, HORIGUCHI NAOYA($BKY8}!!D>Li(B) <[email protected]> wrote:
> >
> > I'm still not sure why the test succeeded by reverting these because
> > current mainline kernel provides similar mechanism to prevent reuse of
> > soft offlined page. So this success seems to me something suspicious.
>
> Even if we call munmap() on the range, it still can$B!G(Bt be be reused? If so, how to recover those memory then?
No, it can't, because soft offline isolates the physical page.
so even after calling munmap(), the side effect remains on the page.
In your random.c, memory online/offline resets the status of hwpoison.
So you can reallocate hugepages in another run of the program.
>
> >
> > To investigate more, I want to have additional info about the page states
> > of the relevant pages after soft offlining. Could you collect it by the
> > following steps?
>
> Do you want to collect those from the failed or succeed kernel?
I'd like to check on the succeeded kernel.
Sorry for the lack of information.
Thanks,
Naoya Horiguchi
On Tue, Aug 11, 2020 at 03:11:40AM +0000, HORIGUCHI NAOYA(堀口 直也) wrote:
> I'm still not sure why the test succeeded by reverting these because
> current mainline kernel provides similar mechanism to prevent reuse of
> soft offlined page. So this success seems to me something suspicious.
>
> To investigate more, I want to have additional info about the page states
> of the relevant pages after soft offlining. Could you collect it by the
> following steps?
>
> - modify random.c not to run hotplug_memory() in migrate_huge_hotplug_memory(),
> - compile it and run "./random 1" once,
> - to collect page state with hwpoisoned pages, run "./page-types -Nlr -b hwpoison",
> where page-types is available under tools/vm in kernel source tree.
> - choose a few pfns of soft offlined pages from kernel message
> "Soft offlining pfn ...", and run "./page-types -Nlr -a <pfn>".
# ./page-types -Nlr -b hwpoison
offset len flags
99a000 1 __________B________X_______________________
99c000 1 __________B________X_______________________
99e000 1 __________B________X_______________________
9a0000 1 __________B________X_______________________
ba6000 1 __________B________X_______________________
baa000 1 __________B________X_______________________
Every single one of pfns was like this,
# ./page-types -Nlr -a 0x99a000
offset len flags
99a000 1 __________B________X_______________________
# ./page-types -Nlr -a 0x99e000
offset len flags
99e000 1 __________B________X_______________________
# ./page-types -Nlr -a 0x99c000
offset len flags
99c000 1 __________B________X_______________________
On Tue, Aug 11, 2020 at 01:39:24PM -0400, Qian Cai wrote:
> On Tue, Aug 11, 2020 at 03:11:40AM +0000, HORIGUCHI NAOYA($BKY8}(B $BD>Li(B) wrote:
> > I'm still not sure why the test succeeded by reverting these because
> > current mainline kernel provides similar mechanism to prevent reuse of
> > soft offlined page. So this success seems to me something suspicious.
> >
> > To investigate more, I want to have additional info about the page states
> > of the relevant pages after soft offlining. Could you collect it by the
> > following steps?
> >
> > - modify random.c not to run hotplug_memory() in migrate_huge_hotplug_memory(),
> > - compile it and run "./random 1" once,
> > - to collect page state with hwpoisoned pages, run "./page-types -Nlr -b hwpoison",
> > where page-types is available under tools/vm in kernel source tree.
> > - choose a few pfns of soft offlined pages from kernel message
> > "Soft offlining pfn ...", and run "./page-types -Nlr -a <pfn>".
>
> # ./page-types -Nlr -b hwpoison
> offset len flags
> 99a000 1 __________B________X_______________________
> 99c000 1 __________B________X_______________________
> 99e000 1 __________B________X_______________________
> 9a0000 1 __________B________X_______________________
> ba6000 1 __________B________X_______________________
> baa000 1 __________B________X_______________________
Thank you. It only shows 6 lines of records, which is unexpected to me
because random.c iterates soft offline 2 hugepages with madvise() 1000 times.
Somehow (maybe in arch specific way?) other hwpoisoned pages might be cleared?
If they really are, the success of this test is a fake, and this patchset
can be considered as a fix.
>
> Every single one of pfns was like this,
>
> # ./page-types -Nlr -a 0x99a000
> offset len flags
> 99a000 1 __________B________X_______________________
>
> # ./page-types -Nlr -a 0x99e000
> offset len flags
> 99e000 1 __________B________X_______________________
>
> # ./page-types -Nlr -a 0x99c000
> offset len flags
> 99c000 1 __________B________X_______________________
On Wed, Aug 12, 2020 at 04:32:01AM +0900, Naoya Horiguchi wrote:
> On Tue, Aug 11, 2020 at 01:39:24PM -0400, Qian Cai wrote:
> > On Tue, Aug 11, 2020 at 03:11:40AM +0000, HORIGUCHI NAOYA(堀口 直也) wrote:
> > > I'm still not sure why the test succeeded by reverting these because
> > > current mainline kernel provides similar mechanism to prevent reuse of
> > > soft offlined page. So this success seems to me something suspicious.
> > >
> > > To investigate more, I want to have additional info about the page states
> > > of the relevant pages after soft offlining. Could you collect it by the
> > > following steps?
> > >
> > > - modify random.c not to run hotplug_memory() in migrate_huge_hotplug_memory(),
> > > - compile it and run "./random 1" once,
> > > - to collect page state with hwpoisoned pages, run "./page-types -Nlr -b hwpoison",
> > > where page-types is available under tools/vm in kernel source tree.
> > > - choose a few pfns of soft offlined pages from kernel message
> > > "Soft offlining pfn ...", and run "./page-types -Nlr -a <pfn>".
> >
> > # ./page-types -Nlr -b hwpoison
> > offset len flags
> > 99a000 1 __________B________X_______________________
> > 99c000 1 __________B________X_______________________
> > 99e000 1 __________B________X_______________________
> > 9a0000 1 __________B________X_______________________
> > ba6000 1 __________B________X_______________________
> > baa000 1 __________B________X_______________________
>
> Thank you. It only shows 6 lines of records, which is unexpected to me
> because random.c iterates soft offline 2 hugepages with madvise() 1000 times.
> Somehow (maybe in arch specific way?) other hwpoisoned pages might be cleared?
> If they really are, the success of this test is a fake, and this patchset
> can be considered as a fix.
The test was designed to catch a previous bug (the latest patchset fixed that)
where kernel will be enterting into an endless loop.
https://lore.kernel.org/lkml/[email protected]/
However, I don't understand why mmap() does not return ENOMEM in the first
place where overcommit_memory == 0 instead of munmap() or/and madvise()
returning ENOMEM. I suppose that is the price to pay with heuristic, and I
can't easily confirm if it is related to this patchset or not.
addr = mmap(NULL, length, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);
if (addr == MAP_FAILED) {
if (i == 0 || errno != ENOMEM) {
perror("mmap");
return 1;
}
usleep(1000);
continue;
}
memset(addr, 0, length);
code = madvise(addr, length, MADV_SOFT_OFFLINE);
if(safe_munmap(addr, length))
return 1;
/* madvise() could return >= 0 on success. */
if (code < 0 && errno != EBUSY) {
perror("madvise");
return 1;
}
Otherwise, our test will keep running and ignore ENOMEM correctly. I did also
confirm that this patchset has a higher success rate of soft-offlining
("page-types" shows 400+ lines) which changes the existing assumption (looks
like in a good way in this case).
On Thu, Aug 06, 2020 at 06:49:23PM +0000, [email protected] wrote:
> From: Naoya Horiguchi <[email protected]>
>
> Soft offlining could fail with EIO due to the race condition with
> hugepage migration. This issuse became visible due to the change by
> previous patch that makes soft offline handler take page refcount
> by its own. We have no way to directly pin zero refcount page, and
> the page considered as a zero refcount page could be allocated just
> after the first check.
>
> This patch adds the second check to find the race and gives us
> chance to handle it more reliably.
>
> Reported-by: Qian Cai <[email protected]>
> Signed-off-by: Naoya Horiguchi <[email protected]>
Reviewed-by: Oscar Salvador <[email protected]>
--
Oscar Salvador
SUSE L3
On 2020-08-06 20:49, [email protected] wrote:
> From: Oscar Salvador <[email protected]>
>
> This patch changes the way we set and handle in-use poisoned pages.
> Until now, poisoned pages were released to the buddy allocator,
> trusting
> that the checks that take place prior to hand the page would act as a
> safe net and would skip that page.
>
> This has proved to be wrong, as we got some pfn walkers out there, like
> compaction, that all they care is the page to be PageBuddy and be in a
> freelist.
> Although this might not be the only user, having poisoned pages
> in the buddy allocator seems a bad idea as we should only have
> free pages that are ready and meant to be used as such.
>
> Before explaining the taken approach, let us break down the kind
> of pages we can soft offline.
>
> - Anonymous THP (after the split, they end up being 4K pages)
> - Hugetlb
> - Order-0 pages (that can be either migrated or invalited)
>
> * Normal pages (order-0 and anon-THP)
>
> - If they are clean and unmapped page cache pages, we invalidate
> then by means of invalidate_inode_page().
> - If they are mapped/dirty, we do the isolate-and-migrate dance.
>
> Either way, do not call put_page directly from those paths.
> Instead, we keep the page and send it to page_set_poison to perform
> the
> right handling.
>
> page_set_poison sets the HWPoison flag and does the last put_page.
> This call to put_page is mainly to be able to call
> __page_cache_release,
> since this function is not exported.
>
> Down the chain, we placed a check for HWPoison page in
> free_pages_prepare, that just skips any poisoned page, so those pages
> do not end up in any pcplist/freelist.
>
> After that, we set the refcount on the page to 1 and we increment
> the poisoned pages counter.
>
> We could do as we do for free pages:
> 1) wait until the page hits buddy's freelists
> 2) take it off
> 3) flag it
>
> The problem is that we could race with an allocation, so by the time
> we
> want to take the page off the buddy, the page is already allocated,
> so we
> cannot soft-offline it.
> This is not fatal of course, but if it is better if we can close the
> race
> as does not require a lot of code.
>
> * Hugetlb pages
>
> - We isolate-and-migrate them
>
> After the migration has been successful, we call
> dissolve_free_huge_page,
> and we set HWPoison on the page if we succeed.
> Hugetlb has a slightly different handling though.
>
> While for non-hugetlb pages we cared about closing the race with an
> allocation, doing so for hugetlb pages requires quite some additional
> code (we would need to hook in free_huge_page and some other places).
> So I decided to not make the code overly complicated and just fail
> normally if the page we allocated in the meantime.
>
> Because of the way we handle now in-use pages, we no longer need the
> put-as-isolation-migratetype dance, that was guarding for poisoned
> pages
> to end up in pcplists.
>
> Signed-off-by: Oscar Salvador <[email protected]>
> Signed-off-by: Naoya Horiguchi <[email protected]>
Hi Andrew,
I just found out yesterday that the patchset Naoya sent has diverged
from mine in some aspects that lead to some bugs [1].
This was due to a misunderstanding so no blame here.
So, patch#8 and patch#9 need a little tweak [2].
I was wondering what do you prefer?
1) I paste the chunks that need to be changed in the offending patches
(this and patch#9)
2) I just send them as standalone patches and you applied them on top
I am asking this because although patches had hit -next, we might still
have time to change them.
If not, do not worry, I will send them as standalone.
[1] https://patchwork.kernel.org/comment/23622881/
[2] https://patchwork.kernel.org/comment/23622985/
I will go ahead and paste the chunks here, in case you lean towards
option#1:
Patch#8:
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index f68cb5e3b320..4ffaaa5c2603 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -67,11 +67,6 @@ atomic_long_t num_poisoned_pages __read_mostly =
ATOMIC_LONG_INIT(0);
static bool page_handle_poison(struct page *page, bool
hugepage_or_freepage, bool release)
{
- if (release) {
- put_page(page);
- drain_all_pages(page_zone(page));
- }
-
if (hugepage_or_freepage) {
/*
* Doing this check for free pages is also fine since
dissolve_free_huge_page
@@ -89,6 +84,12 @@ static bool page_handle_poison(struct page *page,
bool hugepage_or_freepage, boo
}
SetPageHWPoison(page);
+
+ if (release) {
+ put_page(page);
+ drain_all_pages(page_zone(page));
+ }
+
page_ref_inc(page);
num_poisoned_pages_inc();
return true;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 0d9f9bd0e06c..8a6ab05f074c 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1173,6 +1173,16 @@ static __always_inline bool
free_pages_prepare(struct page *page,
trace_mm_page_free(page, order);
+ if (unlikely(PageHWPoison(page)) && !order) {
+ /*
+ * Untie memcg state and reset page's owner
+ */
+ if (memcg_kmem_enabled() && PageKmemcg(page))
+ __memcg_kmem_uncharge_page(page, order);
+ reset_page_owner(page, order);
+ return false;
+ }
+
/*
* Check tail pages before head page information is cleared to
* avoid checking PageCompound for order-0 pag
Patch#9:
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index c3b96ca5c86d..a1bc573d91d5 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -1835,7 +1835,7 @@ static int __soft_offline_page(struct page *page)
if (!ret) {
bool release = !huge;
- if (!page_handle_poison(page, true, release))
+ if (!page_handle_poison(page, huge, release))
ret = -EBUSY;
} else {
if (!list_empty(&pagelist)
Thanks ans sorry for the inconvenience.
On Fri, 18 Sep 2020 09:58:22 +0200 [email protected] wrote:
> I just found out yesterday that the patchset Naoya sent has diverged
> from mine in some aspects that lead to some bugs [1].
> This was due to a misunderstanding so no blame here.
> So, patch#8 and patch#9 need a little tweak [2].
>
> I was wondering what do you prefer?
Well. I (and I suspect the rest of the world) have lost track of
what's going on here.
So please let's have a full resend of the whole series?
On 2020-09-19 02:23, Andrew Morton wrote:
> On Fri, 18 Sep 2020 09:58:22 +0200 [email protected] wrote:
>
>> I just found out yesterday that the patchset Naoya sent has diverged
>> from mine in some aspects that lead to some bugs [1].
>> This was due to a misunderstanding so no blame here.
>> So, patch#8 and patch#9 need a little tweak [2].
>>
>> I was wondering what do you prefer?
>
> Well. I (and I suspect the rest of the world) have lost track of
> what's going on here.
>
> So please let's have a full resend of the whole series?
Sure, I will resend a new version squeezing all changes into it on
Monday.