free_pcppages_bulk() has taken two passes through the pcp lists since
commit 0a5f4e5b4562 ("mm/free_pcppages_bulk: do not hold lock when picking
pages to free") due to deferring the cost of selecting PCP lists until
the zone lock is held. Now that list selection is simplier, the main
cost during selection is bulkfree_pcp_prepare() which in the normal case
is a simple check and prefetching. As the list manipulations have cost
in itself, go back to freeing pages in a single pass.
The series up to this point was evaulated using a trunc microbenchmark
that is truncating sparse files stored in page cache (mmtests config
config-io-trunc). Sparse files were used to limit filesystem interaction.
The results versus a revert of storing high-order pages in the PCP lists is
1-socket Skylake
5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2
Min elapsed 540.00 ( 0.00%) 530.00 ( 1.85%) 530.00 ( 1.85%)
Amean elapsed 543.00 ( 0.00%) 530.00 * 2.39%* 530.00 * 2.39%*
Stddev elapsed 4.83 ( 0.00%) 0.00 ( 100.00%) 0.00 ( 100.00%)
CoeffVar elapsed 0.89 ( 0.00%) 0.00 ( 100.00%) 0.00 ( 100.00%)
Max elapsed 550.00 ( 0.00%) 530.00 ( 3.64%) 530.00 ( 3.64%)
BAmean-50 elapsed 540.00 ( 0.00%) 530.00 ( 1.85%) 530.00 ( 1.85%)
BAmean-95 elapsed 542.22 ( 0.00%) 530.00 ( 2.25%) 530.00 ( 2.25%)
BAmean-99 elapsed 542.22 ( 0.00%) 530.00 ( 2.25%) 530.00 ( 2.25%)
2-socket CascadeLake
5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2
Min elapsed 510.00 ( 0.00%) 500.00 ( 1.96%) 500.00 ( 1.96%)
Amean elapsed 529.00 ( 0.00%) 521.00 ( 1.51%) 510.00 * 3.59%*
Stddev elapsed 16.63 ( 0.00%) 12.87 ( 22.64%) 11.55 ( 30.58%)
CoeffVar elapsed 3.14 ( 0.00%) 2.47 ( 21.46%) 2.26 ( 27.99%)
Max elapsed 550.00 ( 0.00%) 540.00 ( 1.82%) 530.00 ( 3.64%)
BAmean-50 elapsed 516.00 ( 0.00%) 512.00 ( 0.78%) 500.00 ( 3.10%)
BAmean-95 elapsed 526.67 ( 0.00%) 518.89 ( 1.48%) 507.78 ( 3.59%)
BAmean-99 elapsed 526.67 ( 0.00%) 518.89 ( 1.48%) 507.78 ( 3.59%)
The original motivation for multi-passes was will-it-scale page_fault1
using $nr_cpu processes.
2-socket CascadeLake (40 cores, 80 CPUs HT enabled)
5.17.0-rc3 5.17.0-rc3
vanilla mm-highpcpopt-v2
Hmean page_fault1-processes-2 2694662.26 ( 0.00%) 2695780.35 ( 0.04%)
Hmean page_fault1-processes-5 6425819.34 ( 0.00%) 6435544.57 * 0.15%*
Hmean page_fault1-processes-8 9642169.10 ( 0.00%) 9658962.39 ( 0.17%)
Hmean page_fault1-processes-12 12167502.10 ( 0.00%) 12190163.79 ( 0.19%)
Hmean page_fault1-processes-21 15636859.03 ( 0.00%) 15612447.26 ( -0.16%)
Hmean page_fault1-processes-30 25157348.61 ( 0.00%) 25169456.65 ( 0.05%)
Hmean page_fault1-processes-48 27694013.85 ( 0.00%) 27671111.46 ( -0.08%)
Hmean page_fault1-processes-79 25928742.64 ( 0.00%) 25934202.02 ( 0.02%) <--
Hmean page_fault1-processes-110 25730869.75 ( 0.00%) 25671880.65 * -0.23%*
Hmean page_fault1-processes-141 25626992.42 ( 0.00%) 25629551.61 ( 0.01%)
Hmean page_fault1-processes-172 25611651.35 ( 0.00%) 25614927.99 ( 0.01%)
Hmean page_fault1-processes-203 25577298.75 ( 0.00%) 25583445.59 ( 0.02%)
Hmean page_fault1-processes-234 25580686.07 ( 0.00%) 25608240.71 ( 0.11%)
Hmean page_fault1-processes-265 25570215.47 ( 0.00%) 25568647.58 ( -0.01%)
Hmean page_fault1-processes-296 25549488.62 ( 0.00%) 25543935.00 ( -0.02%)
Hmean page_fault1-processes-320 25555149.05 ( 0.00%) 25575696.74 ( 0.08%)
The differences are mostly within the noise and the difference close to
$nr_cpus is negligible.
Signed-off-by: Mel Gorman <[email protected]>
Reviewed-by: Vlastimil Babka <[email protected]>
---
mm/page_alloc.c | 56 +++++++++++++++++++------------------------------
1 file changed, 21 insertions(+), 35 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 635a4e0f70b4..68e2132717c5 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1455,8 +1455,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,
unsigned int order;
int prefetch_nr = READ_ONCE(pcp->batch);
bool isolated_pageblocks;
- struct page *page, *tmp;
- LIST_HEAD(head);
+ struct page *page;
/*
* Ensure proper count is passed which otherwise would stuck in the
@@ -1467,6 +1466,13 @@ static void free_pcppages_bulk(struct zone *zone, int count,
/* Ensure requested pindex is drained first. */
pindex = pindex - 1;
+ /*
+ * local_lock_irq held so equivalent to spin_lock_irqsave for
+ * both PREEMPT_RT and non-PREEMPT_RT configurations.
+ */
+ spin_lock(&zone->lock);
+ isolated_pageblocks = has_isolate_pageblock(zone);
+
while (count > 0) {
struct list_head *list;
int nr_pages;
@@ -1489,7 +1495,11 @@ static void free_pcppages_bulk(struct zone *zone, int count,
nr_pages = 1 << order;
BUILD_BUG_ON(MAX_ORDER >= (1<<NR_PCP_ORDER_WIDTH));
do {
+ int mt;
+
page = list_last_entry(list, struct page, lru);
+ mt = get_pcppage_migratetype(page);
+
/* must delete to avoid corrupting pcp list */
list_del(&page->lru);
count -= nr_pages;
@@ -1498,12 +1508,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
if (bulkfree_pcp_prepare(page))
continue;
- /* Encode order with the migratetype */
- page->index <<= NR_PCP_ORDER_WIDTH;
- page->index |= order;
-
- list_add_tail(&page->lru, &head);
-
/*
* We are going to put the page back to the global
* pool, prefetch its buddy to speed up later access
@@ -1517,36 +1521,18 @@ static void free_pcppages_bulk(struct zone *zone, int count,
prefetch_buddy(page, order);
prefetch_nr--;
}
- } while (count > 0 && !list_empty(list));
- }
- /*
- * local_lock_irq held so equivalent to spin_lock_irqsave for
- * both PREEMPT_RT and non-PREEMPT_RT configurations.
- */
- spin_lock(&zone->lock);
- isolated_pageblocks = has_isolate_pageblock(zone);
+ /* MIGRATE_ISOLATE page should not go to pcplists */
+ VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
+ /* Pageblock could have been isolated meanwhile */
+ if (unlikely(isolated_pageblocks))
+ mt = get_pageblock_migratetype(page);
- /*
- * Use safe version since after __free_one_page(),
- * page->lru.next will not point to original list.
- */
- list_for_each_entry_safe(page, tmp, &head, lru) {
- int mt = get_pcppage_migratetype(page);
-
- /* mt has been encoded with the order (see above) */
- order = mt & NR_PCP_ORDER_MASK;
- mt >>= NR_PCP_ORDER_WIDTH;
-
- /* MIGRATE_ISOLATE page should not go to pcplists */
- VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
- /* Pageblock could have been isolated meanwhile */
- if (unlikely(isolated_pageblocks))
- mt = get_pageblock_migratetype(page);
-
- __free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE);
- trace_mm_page_pcpu_drain(page, order, mt);
+ __free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE);
+ trace_mm_page_pcpu_drain(page, order, mt);
+ } while (count > 0 && !list_empty(list));
}
+
spin_unlock(&zone->lock);
}
--
2.31.1
On Thu, Feb 17, 2022 at 12:22:26AM +0000, Mel Gorman wrote:
> free_pcppages_bulk() has taken two passes through the pcp lists since
> commit 0a5f4e5b4562 ("mm/free_pcppages_bulk: do not hold lock when picking
> pages to free") due to deferring the cost of selecting PCP lists until
> the zone lock is held. Now that list selection is simplier, the main
> cost during selection is bulkfree_pcp_prepare() which in the normal case
> is a simple check and prefetching. As the list manipulations have cost
> in itself, go back to freeing pages in a single pass.
>
> The series up to this point was evaulated using a trunc microbenchmark
> that is truncating sparse files stored in page cache (mmtests config
> config-io-trunc). Sparse files were used to limit filesystem interaction.
> The results versus a revert of storing high-order pages in the PCP lists is
>
> 1-socket Skylake
> 5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
> vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2
> Min elapsed 540.00 ( 0.00%) 530.00 ( 1.85%) 530.00 ( 1.85%)
> Amean elapsed 543.00 ( 0.00%) 530.00 * 2.39%* 530.00 * 2.39%*
> Stddev elapsed 4.83 ( 0.00%) 0.00 ( 100.00%) 0.00 ( 100.00%)
> CoeffVar elapsed 0.89 ( 0.00%) 0.00 ( 100.00%) 0.00 ( 100.00%)
> Max elapsed 550.00 ( 0.00%) 530.00 ( 3.64%) 530.00 ( 3.64%)
> BAmean-50 elapsed 540.00 ( 0.00%) 530.00 ( 1.85%) 530.00 ( 1.85%)
> BAmean-95 elapsed 542.22 ( 0.00%) 530.00 ( 2.25%) 530.00 ( 2.25%)
> BAmean-99 elapsed 542.22 ( 0.00%) 530.00 ( 2.25%) 530.00 ( 2.25%)
>
> 2-socket CascadeLake
> 5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
> vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2
> Min elapsed 510.00 ( 0.00%) 500.00 ( 1.96%) 500.00 ( 1.96%)
> Amean elapsed 529.00 ( 0.00%) 521.00 ( 1.51%) 510.00 * 3.59%*
> Stddev elapsed 16.63 ( 0.00%) 12.87 ( 22.64%) 11.55 ( 30.58%)
> CoeffVar elapsed 3.14 ( 0.00%) 2.47 ( 21.46%) 2.26 ( 27.99%)
> Max elapsed 550.00 ( 0.00%) 540.00 ( 1.82%) 530.00 ( 3.64%)
> BAmean-50 elapsed 516.00 ( 0.00%) 512.00 ( 0.78%) 500.00 ( 3.10%)
> BAmean-95 elapsed 526.67 ( 0.00%) 518.89 ( 1.48%) 507.78 ( 3.59%)
> BAmean-99 elapsed 526.67 ( 0.00%) 518.89 ( 1.48%) 507.78 ( 3.59%)
>
> The original motivation for multi-passes was will-it-scale page_fault1
> using $nr_cpu processes.
>
> 2-socket CascadeLake (40 cores, 80 CPUs HT enabled)
> 5.17.0-rc3 5.17.0-rc3
> vanilla mm-highpcpopt-v2
> Hmean page_fault1-processes-2 2694662.26 ( 0.00%) 2695780.35 ( 0.04%)
> Hmean page_fault1-processes-5 6425819.34 ( 0.00%) 6435544.57 * 0.15%*
> Hmean page_fault1-processes-8 9642169.10 ( 0.00%) 9658962.39 ( 0.17%)
> Hmean page_fault1-processes-12 12167502.10 ( 0.00%) 12190163.79 ( 0.19%)
> Hmean page_fault1-processes-21 15636859.03 ( 0.00%) 15612447.26 ( -0.16%)
> Hmean page_fault1-processes-30 25157348.61 ( 0.00%) 25169456.65 ( 0.05%)
> Hmean page_fault1-processes-48 27694013.85 ( 0.00%) 27671111.46 ( -0.08%)
> Hmean page_fault1-processes-79 25928742.64 ( 0.00%) 25934202.02 ( 0.02%) <--
> Hmean page_fault1-processes-110 25730869.75 ( 0.00%) 25671880.65 * -0.23%*
> Hmean page_fault1-processes-141 25626992.42 ( 0.00%) 25629551.61 ( 0.01%)
> Hmean page_fault1-processes-172 25611651.35 ( 0.00%) 25614927.99 ( 0.01%)
> Hmean page_fault1-processes-203 25577298.75 ( 0.00%) 25583445.59 ( 0.02%)
> Hmean page_fault1-processes-234 25580686.07 ( 0.00%) 25608240.71 ( 0.11%)
> Hmean page_fault1-processes-265 25570215.47 ( 0.00%) 25568647.58 ( -0.01%)
> Hmean page_fault1-processes-296 25549488.62 ( 0.00%) 25543935.00 ( -0.02%)
> Hmean page_fault1-processes-320 25555149.05 ( 0.00%) 25575696.74 ( 0.08%)
>
> The differences are mostly within the noise and the difference close to
> $nr_cpus is negligible.
I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
servers: CascadeLake and CooperLaker and will let you know the result
once it's out.
I'm using 'https://github.com/hnaz/linux-mm master' and doing the
comparison with commit c000d687ce22("mm/page_alloc: simplify how many
pages are selected per pcp list during bulk free") and commit 8391e0a7e172
("mm/page_alloc: free pages in a single pass during bulk free") there.
The kernel for each commit will have to be fetched and built and then
the job will be run for 3 times for each commit. These servers are also
busy with other jobs so it may take a while.
Regards,
Aaron
On Thu, Feb 17, 2022 at 09:53:08AM +0800, Aaron Lu wrote:
> On Thu, Feb 17, 2022 at 12:22:26AM +0000, Mel Gorman wrote:
> > free_pcppages_bulk() has taken two passes through the pcp lists since
> > commit 0a5f4e5b4562 ("mm/free_pcppages_bulk: do not hold lock when picking
> > pages to free") due to deferring the cost of selecting PCP lists until
> > the zone lock is held. Now that list selection is simplier, the main
> > cost during selection is bulkfree_pcp_prepare() which in the normal case
> > is a simple check and prefetching. As the list manipulations have cost
> > in itself, go back to freeing pages in a single pass.
> >
> > The series up to this point was evaulated using a trunc microbenchmark
> > that is truncating sparse files stored in page cache (mmtests config
> > config-io-trunc). Sparse files were used to limit filesystem interaction.
> > The results versus a revert of storing high-order pages in the PCP lists is
> >
> > 1-socket Skylake
> > 5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
> > vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2
> > Min elapsed 540.00 ( 0.00%) 530.00 ( 1.85%) 530.00 ( 1.85%)
> > Amean elapsed 543.00 ( 0.00%) 530.00 * 2.39%* 530.00 * 2.39%*
> > Stddev elapsed 4.83 ( 0.00%) 0.00 ( 100.00%) 0.00 ( 100.00%)
> > CoeffVar elapsed 0.89 ( 0.00%) 0.00 ( 100.00%) 0.00 ( 100.00%)
> > Max elapsed 550.00 ( 0.00%) 530.00 ( 3.64%) 530.00 ( 3.64%)
> > BAmean-50 elapsed 540.00 ( 0.00%) 530.00 ( 1.85%) 530.00 ( 1.85%)
> > BAmean-95 elapsed 542.22 ( 0.00%) 530.00 ( 2.25%) 530.00 ( 2.25%)
> > BAmean-99 elapsed 542.22 ( 0.00%) 530.00 ( 2.25%) 530.00 ( 2.25%)
> >
> > 2-socket CascadeLake
> > 5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
> > vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2
> > Min elapsed 510.00 ( 0.00%) 500.00 ( 1.96%) 500.00 ( 1.96%)
> > Amean elapsed 529.00 ( 0.00%) 521.00 ( 1.51%) 510.00 * 3.59%*
> > Stddev elapsed 16.63 ( 0.00%) 12.87 ( 22.64%) 11.55 ( 30.58%)
> > CoeffVar elapsed 3.14 ( 0.00%) 2.47 ( 21.46%) 2.26 ( 27.99%)
> > Max elapsed 550.00 ( 0.00%) 540.00 ( 1.82%) 530.00 ( 3.64%)
> > BAmean-50 elapsed 516.00 ( 0.00%) 512.00 ( 0.78%) 500.00 ( 3.10%)
> > BAmean-95 elapsed 526.67 ( 0.00%) 518.89 ( 1.48%) 507.78 ( 3.59%)
> > BAmean-99 elapsed 526.67 ( 0.00%) 518.89 ( 1.48%) 507.78 ( 3.59%)
> >
> > The original motivation for multi-passes was will-it-scale page_fault1
> > using $nr_cpu processes.
> >
> > 2-socket CascadeLake (40 cores, 80 CPUs HT enabled)
> > 5.17.0-rc3 5.17.0-rc3
> > vanilla mm-highpcpopt-v2
> > Hmean page_fault1-processes-2 2694662.26 ( 0.00%) 2695780.35 ( 0.04%)
> > Hmean page_fault1-processes-5 6425819.34 ( 0.00%) 6435544.57 * 0.15%*
> > Hmean page_fault1-processes-8 9642169.10 ( 0.00%) 9658962.39 ( 0.17%)
> > Hmean page_fault1-processes-12 12167502.10 ( 0.00%) 12190163.79 ( 0.19%)
> > Hmean page_fault1-processes-21 15636859.03 ( 0.00%) 15612447.26 ( -0.16%)
> > Hmean page_fault1-processes-30 25157348.61 ( 0.00%) 25169456.65 ( 0.05%)
> > Hmean page_fault1-processes-48 27694013.85 ( 0.00%) 27671111.46 ( -0.08%)
> > Hmean page_fault1-processes-79 25928742.64 ( 0.00%) 25934202.02 ( 0.02%) <--
> > Hmean page_fault1-processes-110 25730869.75 ( 0.00%) 25671880.65 * -0.23%*
> > Hmean page_fault1-processes-141 25626992.42 ( 0.00%) 25629551.61 ( 0.01%)
> > Hmean page_fault1-processes-172 25611651.35 ( 0.00%) 25614927.99 ( 0.01%)
> > Hmean page_fault1-processes-203 25577298.75 ( 0.00%) 25583445.59 ( 0.02%)
> > Hmean page_fault1-processes-234 25580686.07 ( 0.00%) 25608240.71 ( 0.11%)
> > Hmean page_fault1-processes-265 25570215.47 ( 0.00%) 25568647.58 ( -0.01%)
> > Hmean page_fault1-processes-296 25549488.62 ( 0.00%) 25543935.00 ( -0.02%)
> > Hmean page_fault1-processes-320 25555149.05 ( 0.00%) 25575696.74 ( 0.08%)
> >
> > The differences are mostly within the noise and the difference close to
> > $nr_cpus is negligible.
>
> I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
> servers: CascadeLake and CooperLaker and will let you know the result
> once it's out.
The test on CooperLake has finished and the result is in noise range,
i.e. no obvious performance change due to this patch. Feel free to add
my tested-by tag.
There is a problem with the CascadeLake server, I can't test there. But
I don't expect a surprising result either :-)
BTW, the lock contention on the 4s CooperLake is only 46%, with 14% from
page allocator allocation path and 4% from the free path, others are on
LRU path. I remember the lock contention used to be in the 80% percent
in my previous testing, and I found this old mail where page_fault1 test
on 2s skylake saw 80% lock contention:
https://lore.kernel.org/linux-mm/[email protected]/
So I also did the test on a 2s CascadeLake(failed to find a skylake server)
and found that the lock contention is about 70%, with 12% coming from page
allocator allocation path and 1.6% from free path, others are on LRU path.
I guess something has happend in the page allocator and zone lock contention
is much better now, I don't know what exactly, I'll take a look at this later.
And since free path's lock contention is small, I wonder how much
value of the prefetch_done_on_free_path still has. I'll also give it a
test later.
Regards,
Aaron
>
> I'm using 'https://github.com/hnaz/linux-mm master' and doing the
> comparison with commit c000d687ce22("mm/page_alloc: simplify how many
> pages are selected per pcp list during bulk free") and commit 8391e0a7e172
> ("mm/page_alloc: free pages in a single pass during bulk free") there.
>
> The kernel for each commit will have to be fetched and built and then
> the job will be run for 3 times for each commit. These servers are also
> busy with other jobs so it may take a while.
>
> Regards,
> Aaron
On Thu, Feb 17, 2022 at 09:53:08AM +0800, Aaron Lu wrote:
> > 2-socket CascadeLake (40 cores, 80 CPUs HT enabled)
> > 5.17.0-rc3 5.17.0-rc3
> > vanilla mm-highpcpopt-v2
> > Hmean page_fault1-processes-2 2694662.26 ( 0.00%) 2695780.35 ( 0.04%)
> > Hmean page_fault1-processes-5 6425819.34 ( 0.00%) 6435544.57 * 0.15%*
> > Hmean page_fault1-processes-8 9642169.10 ( 0.00%) 9658962.39 ( 0.17%)
> > Hmean page_fault1-processes-12 12167502.10 ( 0.00%) 12190163.79 ( 0.19%)
> > Hmean page_fault1-processes-21 15636859.03 ( 0.00%) 15612447.26 ( -0.16%)
> > Hmean page_fault1-processes-30 25157348.61 ( 0.00%) 25169456.65 ( 0.05%)
> > Hmean page_fault1-processes-48 27694013.85 ( 0.00%) 27671111.46 ( -0.08%)
> > Hmean page_fault1-processes-79 25928742.64 ( 0.00%) 25934202.02 ( 0.02%) <--
> > Hmean page_fault1-processes-110 25730869.75 ( 0.00%) 25671880.65 * -0.23%*
> > Hmean page_fault1-processes-141 25626992.42 ( 0.00%) 25629551.61 ( 0.01%)
> > Hmean page_fault1-processes-172 25611651.35 ( 0.00%) 25614927.99 ( 0.01%)
> > Hmean page_fault1-processes-203 25577298.75 ( 0.00%) 25583445.59 ( 0.02%)
> > Hmean page_fault1-processes-234 25580686.07 ( 0.00%) 25608240.71 ( 0.11%)
> > Hmean page_fault1-processes-265 25570215.47 ( 0.00%) 25568647.58 ( -0.01%)
> > Hmean page_fault1-processes-296 25549488.62 ( 0.00%) 25543935.00 ( -0.02%)
> > Hmean page_fault1-processes-320 25555149.05 ( 0.00%) 25575696.74 ( 0.08%)
> >
> > The differences are mostly within the noise and the difference close to
> > $nr_cpus is negligible.
>
> I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
> servers: CascadeLake and CooperLaker and will let you know the result
> once it's out.
>
Thanks, 4 sockets and a later generation would be nice to cover.
> I'm using 'https://github.com/hnaz/linux-mm master' and doing the
> comparison with commit c000d687ce22("mm/page_alloc: simplify how many
> pages are selected per pcp list during bulk free") and commit 8391e0a7e172
> ("mm/page_alloc: free pages in a single pass during bulk free") there.
>
The baseline looks fine. It's different to what I used but the page_alloc
shouldn't have much impact.
When looking at will-it-scale, please pay attention to lower CPU counts
as well and take account changes in standard deviation. Looking at the
old commit (which I acked so I've no excuse), I think it's important to
look at cases other than the fully utilised case because it's the best
case for something like will-it-scale pf but it's also an unlikely case
(all CPUs all faulting continuously).
I expect there will be different good/bad points based on looking at
Zen1 results (8 nodes, varying distances, 64 cores with 128 CPUs HT
enabled)
5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2
Hmean page_fault1-threads-2 2985366.46 ( 0.00%) 2984649.41 ( -0.02%) 3028407.35 ( 1.44%)
Hmean page_fault1-threads-5 3491833.63 ( 0.00%) 3500237.35 ( 0.24%) 3489971.99 ( -0.05%)
Hmean page_fault1-threads-8 3254335.58 ( 0.00%) 3277515.51 * 0.71%* 3234275.28 * -0.62%*
Hmean page_fault1-threads-12 5101504.72 ( 0.00%) 5390649.46 * 5.67%* 5162047.68 ( 1.19%)
Hmean page_fault1-threads-21 7714265.64 ( 0.00%) 7714763.10 ( 0.01%) 7854367.65 * 1.82%*
Hmean page_fault1-threads-30 10034561.94 ( 0.00%) 9865446.68 ( -1.69%) 9746368.76 * -2.87%*
Hmean page_fault1-threads-48 12571351.99 ( 0.00%) 13257508.23 * 5.46%* 12160897.07 * -3.27%*
Hmean page_fault1-threads-79 11124387.46 ( 0.00%) 10641145.82 * -4.34%* 10677656.39 * -4.02%*
Hmean page_fault1-threads-110 11980424.12 ( 0.00%) 10778220.84 * -10.03%* 10354249.62 * -13.57%* <-- close to nr_cpus
Hmean page_fault1-threads-141 9727528.73 ( 0.00%) 9966965.70 ( 2.46%) 9656148.13 ( -0.73%) <-- close to nr_cpus
Hmean page_fault1-threads-172 11807964.92 ( 0.00%) 10335576.64 * -12.47%* 10443310.45 * -11.56%*
Hmean page_fault1-threads-203 9471961.29 ( 0.00%) 9749857.24 * 2.93%* 11890019.87 * 25.53%*
Hmean page_fault1-threads-234 11322381.78 ( 0.00%) 9163162.66 ( -19.07%) 9141561.16 ( -19.26%)
Hmean page_fault1-threads-265 7956982.52 ( 0.00%) 7774650.20 ( -2.29%) 8292405.57 * 4.22%*
Hmean page_fault1-threads-296 7892153.88 ( 0.00%) 8272671.84 * 4.82%* 7907026.20 ( 0.19%)
Hmean page_fault1-threads-327 7957124.50 ( 0.00%) 8078297.34 ( 1.52%) 8129776.79 ( 2.17%)
Hmean page_fault1-threads-358 7847563.90 ( 0.00%) 8202303.36 ( 4.52%) 8139027.38 ( 3.71%)
Hmean page_fault1-threads-389 7928386.47 ( 0.00%) 8104732.41 ( 2.22%) 8022002.73 ( 1.18%)
Hmean page_fault1-threads-420 7690107.89 ( 0.00%) 7587821.54 ( -1.33%) 7783777.95 ( 1.22%)
Hmean page_fault1-threads-451 7683132.29 ( 0.00%) 7979578.21 ( 3.86%) 7693067.13 ( 0.13%)
Hmean page_fault1-threads-482 7720646.31 ( 0.00%) 7597453.65 ( -1.60%) 7870063.90 ( 1.94%)
Hmean page_fault1-threads-512 7353458.45 ( 0.00%) 7584407.14 ( 3.14%) 8119539.24 ( 10.42%)
Stddev page_fault1-processes-2 4086.39 ( 0.00%) 1698.11 ( 58.44%) 1488.13 ( 63.58%)
Stddev page_fault1-processes-5 1448.69 ( 0.00%) 1616.59 ( -11.59%) 1567.37 ( -8.19%)
Stddev page_fault1-processes-8 1828.29 ( 0.00%) 2628.59 ( -43.77%) 2701.96 ( -47.79%)
Stddev page_fault1-processes-12 14073.12 ( 0.00%) 1575.18 ( 88.81%) 4880.93 ( 65.32%)
Stddev page_fault1-processes-21 4368.35 ( 0.00%) 7865.27 ( -80.05%) 3778.03 ( 13.51%)
Stddev page_fault1-processes-30 5348.13 ( 0.00%) 11751.43 (-119.73%) 3240.22 ( 39.41%)
Stddev page_fault1-processes-48 23687.16 ( 0.00%) 7803.01 ( 67.06%) 2635.85 ( 88.87%)
Stddev page_fault1-processes-79 12779.16 ( 0.00%) 4311.60 ( 66.26%) 22539.03 ( -76.37%)
Stddev page_fault1-processes-110 21031.04 ( 0.00%) 15115.36 ( 28.13%) 12136.54 ( 42.29%)
Stddev page_fault1-processes-141 589804.99 ( 0.00%) 1335519.71 (-126.43%) 19560.01 ( 96.68%)
Stddev page_fault1-processes-172 7033.94 ( 0.00%) 7147.71 ( -1.62%) 11366.64 ( -61.60%)
Stddev page_fault1-processes-203 6322.20 ( 0.00%) 5035.55 ( 20.35%) 4043.45 ( 36.04%)
Stddev page_fault1-processes-234 12046.53 ( 0.00%) 24208.37 (-100.96%) 9159.91 ( 23.96%)
Stddev page_fault1-processes-265 11869.43 ( 0.00%) 13528.26 ( -13.98%) 8943.99 ( 24.65%)
Stddev page_fault1-processes-296 8918.50 ( 0.00%) 16130.54 ( -80.87%) 5211.80 ( 41.56%)
Stddev page_fault1-processes-327 101102.64 ( 0.00%) 845864.70 (-736.64%) 16238.99 ( 83.94%)
Stddev page_fault1-processes-358 2102190.38 ( 0.00%) 11316.00 ( 99.46%) 7508.57 ( 99.64%)
Stddev page_fault1-processes-389 61012.79 ( 0.00%) 121446.55 ( -99.05%) 18279.64 ( 70.04%)
Stddev page_fault1-processes-420 2305208.40 ( 0.00%) 2347564.71 ( -1.84%) 3202.77 ( 99.86%)
Stddev page_fault1-processes-451 20214.37 ( 0.00%) 173800.17 (-759.79%) 492258.35 (-2335.19%)
Stddev page_fault1-processes-482 236881.21 ( 0.00%) 330501.32 ( -39.52%) 15307.31 ( 93.54%)
Stddev page_fault1-processes-512 201354.82 ( 0.00%) 207019.93 ( -2.81%) 4900536.90 (-2333.78%)
This is showing there was a impact around the nr_cpus (110 and 141
processes measured) but the standard deviation around 141 was particularly
high in the baseline case taking two passes through lists. It's also
interesting to note that in most cases that standard deviation is reduced
by the series even though it's not universally true.
As a side-note, there is also a fair amount of NUMA balancing that takes
place during this test which further muddies the waters. This is a slightly
surprising result and I suspect what's happening is that processes are
getting migrated cross-node as the number of processes exceed a local
nodes capacity due to load balancing. It might be highlighting a weakness
in the test itself where it ends up measuring more than one thing (not
just fault capacity but load balancing effects as individual nodes CPU
capacity approaches fully busy).
My main concern when writing this patch was the basic case of one CPU doing
a lot of frees (exiting, large truncate, large unmap, anything hammering
on release_pages for a large region etc) suffered from taking two loops
through lists with all the associated cost of the list manipulations. I
worried that by trying to optimise for a corner case (all CPUs allocating
simultaneously), we missed a basic case (one CPU doing a large amount
of allocating/freeing).
If possible, it would be nice if you could add something like
configs/config-io-trunc from mmtests to lkp if it doesn't exist already
to consider the simple case. As its most basic, all it's doing is
---8<---
#!/bin/bash
for i in {1..10}; do
dd if=/dev/zero of=sparse_file-$i bs=1 count=0 seek=1G &>/dev/null
cat sparse_file-$i > /dev/null
done
sync
# Primary metric
time rm sparse_file*
---8<---
The main difference is that the mmtests will report the time to fault the
sparse files (bulk simple allocate inserting into page cache) as well as
the bulk truncate (bulk simple release of page cache).
Thanks.
--
Mel Gorman
SUSE Labs
On Thu, Feb 17, 2022 at 09:31:13AM +0000, Mel Gorman wrote:
> On Thu, Feb 17, 2022 at 09:53:08AM +0800, Aaron Lu wrote:
> > > 2-socket CascadeLake (40 cores, 80 CPUs HT enabled)
> > > 5.17.0-rc3 5.17.0-rc3
> > > vanilla mm-highpcpopt-v2
> > > Hmean page_fault1-processes-2 2694662.26 ( 0.00%) 2695780.35 ( 0.04%)
> > > Hmean page_fault1-processes-5 6425819.34 ( 0.00%) 6435544.57 * 0.15%*
> > > Hmean page_fault1-processes-8 9642169.10 ( 0.00%) 9658962.39 ( 0.17%)
> > > Hmean page_fault1-processes-12 12167502.10 ( 0.00%) 12190163.79 ( 0.19%)
> > > Hmean page_fault1-processes-21 15636859.03 ( 0.00%) 15612447.26 ( -0.16%)
> > > Hmean page_fault1-processes-30 25157348.61 ( 0.00%) 25169456.65 ( 0.05%)
> > > Hmean page_fault1-processes-48 27694013.85 ( 0.00%) 27671111.46 ( -0.08%)
> > > Hmean page_fault1-processes-79 25928742.64 ( 0.00%) 25934202.02 ( 0.02%) <--
> > > Hmean page_fault1-processes-110 25730869.75 ( 0.00%) 25671880.65 * -0.23%*
> > > Hmean page_fault1-processes-141 25626992.42 ( 0.00%) 25629551.61 ( 0.01%)
> > > Hmean page_fault1-processes-172 25611651.35 ( 0.00%) 25614927.99 ( 0.01%)
> > > Hmean page_fault1-processes-203 25577298.75 ( 0.00%) 25583445.59 ( 0.02%)
> > > Hmean page_fault1-processes-234 25580686.07 ( 0.00%) 25608240.71 ( 0.11%)
> > > Hmean page_fault1-processes-265 25570215.47 ( 0.00%) 25568647.58 ( -0.01%)
> > > Hmean page_fault1-processes-296 25549488.62 ( 0.00%) 25543935.00 ( -0.02%)
> > > Hmean page_fault1-processes-320 25555149.05 ( 0.00%) 25575696.74 ( 0.08%)
> > >
> > > The differences are mostly within the noise and the difference close to
> > > $nr_cpus is negligible.
> >
> > I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
> > servers: CascadeLake and CooperLaker and will let you know the result
> > once it's out.
> >
>
> Thanks, 4 sockets and a later generation would be nice to cover.
>
> > I'm using 'https://github.com/hnaz/linux-mm master' and doing the
> > comparison with commit c000d687ce22("mm/page_alloc: simplify how many
> > pages are selected per pcp list during bulk free") and commit 8391e0a7e172
> > ("mm/page_alloc: free pages in a single pass during bulk free") there.
> >
>
> The baseline looks fine. It's different to what I used but the page_alloc
> shouldn't have much impact.
>
> When looking at will-it-scale, please pay attention to lower CPU counts
> as well and take account changes in standard deviation. Looking at the
I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.
> old commit (which I acked so I've no excuse), I think it's important to
> look at cases other than the fully utilised case because it's the best
> case for something like will-it-scale pf but it's also an unlikely case
> (all CPUs all faulting continuously).
I see.
>
> I expect there will be different good/bad points based on looking at
> Zen1 results (8 nodes, varying distances, 64 cores with 128 CPUs HT
> enabled)
>
> 5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
> vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2
> Hmean page_fault1-threads-2 2985366.46 ( 0.00%) 2984649.41 ( -0.02%) 3028407.35 ( 1.44%)
> Hmean page_fault1-threads-5 3491833.63 ( 0.00%) 3500237.35 ( 0.24%) 3489971.99 ( -0.05%)
> Hmean page_fault1-threads-8 3254335.58 ( 0.00%) 3277515.51 * 0.71%* 3234275.28 * -0.62%*
> Hmean page_fault1-threads-12 5101504.72 ( 0.00%) 5390649.46 * 5.67%* 5162047.68 ( 1.19%)
> Hmean page_fault1-threads-21 7714265.64 ( 0.00%) 7714763.10 ( 0.01%) 7854367.65 * 1.82%*
> Hmean page_fault1-threads-30 10034561.94 ( 0.00%) 9865446.68 ( -1.69%) 9746368.76 * -2.87%*
> Hmean page_fault1-threads-48 12571351.99 ( 0.00%) 13257508.23 * 5.46%* 12160897.07 * -3.27%*
> Hmean page_fault1-threads-79 11124387.46 ( 0.00%) 10641145.82 * -4.34%* 10677656.39 * -4.02%*
> Hmean page_fault1-threads-110 11980424.12 ( 0.00%) 10778220.84 * -10.03%* 10354249.62 * -13.57%* <-- close to nr_cpus
> Hmean page_fault1-threads-141 9727528.73 ( 0.00%) 9966965.70 ( 2.46%) 9656148.13 ( -0.73%) <-- close to nr_cpus
I have never tested thread mode, because I think the heavy loaded
thread mode is more about testing the mmap_sem contention than page
allocator's performance? It's surprising this patch caused a
performance change.
> Hmean page_fault1-threads-172 11807964.92 ( 0.00%) 10335576.64 * -12.47%* 10443310.45 * -11.56%*
> Hmean page_fault1-threads-203 9471961.29 ( 0.00%) 9749857.24 * 2.93%* 11890019.87 * 25.53%*
> Hmean page_fault1-threads-234 11322381.78 ( 0.00%) 9163162.66 ( -19.07%) 9141561.16 ( -19.26%)
> Hmean page_fault1-threads-265 7956982.52 ( 0.00%) 7774650.20 ( -2.29%) 8292405.57 * 4.22%*
> Hmean page_fault1-threads-296 7892153.88 ( 0.00%) 8272671.84 * 4.82%* 7907026.20 ( 0.19%)
> Hmean page_fault1-threads-327 7957124.50 ( 0.00%) 8078297.34 ( 1.52%) 8129776.79 ( 2.17%)
> Hmean page_fault1-threads-358 7847563.90 ( 0.00%) 8202303.36 ( 4.52%) 8139027.38 ( 3.71%)
> Hmean page_fault1-threads-389 7928386.47 ( 0.00%) 8104732.41 ( 2.22%) 8022002.73 ( 1.18%)
> Hmean page_fault1-threads-420 7690107.89 ( 0.00%) 7587821.54 ( -1.33%) 7783777.95 ( 1.22%)
> Hmean page_fault1-threads-451 7683132.29 ( 0.00%) 7979578.21 ( 3.86%) 7693067.13 ( 0.13%)
> Hmean page_fault1-threads-482 7720646.31 ( 0.00%) 7597453.65 ( -1.60%) 7870063.90 ( 1.94%)
> Hmean page_fault1-threads-512 7353458.45 ( 0.00%) 7584407.14 ( 3.14%) 8119539.24 ( 10.42%)
> Stddev page_fault1-processes-2 4086.39 ( 0.00%) 1698.11 ( 58.44%) 1488.13 ( 63.58%)
> Stddev page_fault1-processes-5 1448.69 ( 0.00%) 1616.59 ( -11.59%) 1567.37 ( -8.19%)
> Stddev page_fault1-processes-8 1828.29 ( 0.00%) 2628.59 ( -43.77%) 2701.96 ( -47.79%)
> Stddev page_fault1-processes-12 14073.12 ( 0.00%) 1575.18 ( 88.81%) 4880.93 ( 65.32%)
> Stddev page_fault1-processes-21 4368.35 ( 0.00%) 7865.27 ( -80.05%) 3778.03 ( 13.51%)
> Stddev page_fault1-processes-30 5348.13 ( 0.00%) 11751.43 (-119.73%) 3240.22 ( 39.41%)
> Stddev page_fault1-processes-48 23687.16 ( 0.00%) 7803.01 ( 67.06%) 2635.85 ( 88.87%)
> Stddev page_fault1-processes-79 12779.16 ( 0.00%) 4311.60 ( 66.26%) 22539.03 ( -76.37%)
> Stddev page_fault1-processes-110 21031.04 ( 0.00%) 15115.36 ( 28.13%) 12136.54 ( 42.29%)
> Stddev page_fault1-processes-141 589804.99 ( 0.00%) 1335519.71 (-126.43%) 19560.01 ( 96.68%)
> Stddev page_fault1-processes-172 7033.94 ( 0.00%) 7147.71 ( -1.62%) 11366.64 ( -61.60%)
> Stddev page_fault1-processes-203 6322.20 ( 0.00%) 5035.55 ( 20.35%) 4043.45 ( 36.04%)
> Stddev page_fault1-processes-234 12046.53 ( 0.00%) 24208.37 (-100.96%) 9159.91 ( 23.96%)
> Stddev page_fault1-processes-265 11869.43 ( 0.00%) 13528.26 ( -13.98%) 8943.99 ( 24.65%)
> Stddev page_fault1-processes-296 8918.50 ( 0.00%) 16130.54 ( -80.87%) 5211.80 ( 41.56%)
> Stddev page_fault1-processes-327 101102.64 ( 0.00%) 845864.70 (-736.64%) 16238.99 ( 83.94%)
> Stddev page_fault1-processes-358 2102190.38 ( 0.00%) 11316.00 ( 99.46%) 7508.57 ( 99.64%)
> Stddev page_fault1-processes-389 61012.79 ( 0.00%) 121446.55 ( -99.05%) 18279.64 ( 70.04%)
> Stddev page_fault1-processes-420 2305208.40 ( 0.00%) 2347564.71 ( -1.84%) 3202.77 ( 99.86%)
> Stddev page_fault1-processes-451 20214.37 ( 0.00%) 173800.17 (-759.79%) 492258.35 (-2335.19%)
> Stddev page_fault1-processes-482 236881.21 ( 0.00%) 330501.32 ( -39.52%) 15307.31 ( 93.54%)
> Stddev page_fault1-processes-512 201354.82 ( 0.00%) 207019.93 ( -2.81%) 4900536.90 (-2333.78%)
>
> This is showing there was a impact around the nr_cpus (110 and 141
> processes measured) but the standard deviation around 141 was particularly
~~~~~~~~~
Did you mean threads?
> high in the baseline case taking two passes through lists. It's also
> interesting to note that in most cases that standard deviation is reduced
> by the series even though it's not universally true.
>
> As a side-note, there is also a fair amount of NUMA balancing that takes
> place during this test which further muddies the waters. This is a slightly
> surprising result and I suspect what's happening is that processes are
> getting migrated cross-node as the number of processes exceed a local
> nodes capacity due to load balancing. It might be highlighting a weakness
> in the test itself where it ends up measuring more than one thing (not
> just fault capacity but load balancing effects as individual nodes CPU
> capacity approaches fully busy).
Makes sense.
>
> My main concern when writing this patch was the basic case of one CPU doing
> a lot of frees (exiting, large truncate, large unmap, anything hammering
> on release_pages for a large region etc) suffered from taking two loops
> through lists with all the associated cost of the list manipulations. I
> worried that by trying to optimise for a corner case (all CPUs allocating
> simultaneously), we missed a basic case (one CPU doing a large amount
> of allocating/freeing).
I see.
>
> If possible, it would be nice if you could add something like
> configs/config-io-trunc from mmtests to lkp if it doesn't exist already
> to consider the simple case. As its most basic, all it's doing is
>
> ---8<---
> #!/bin/bash
>
> for i in {1..10}; do
> dd if=/dev/zero of=sparse_file-$i bs=1 count=0 seek=1G &>/dev/null
> cat sparse_file-$i > /dev/null
> done
> sync
>
> # Primary metric
> time rm sparse_file*
> ---8<---
>
> The main difference is that the mmtests will report the time to fault the
> sparse files (bulk simple allocate inserting into page cache) as well as
> the bulk truncate (bulk simple release of page cache).
Thanks for the suggestion.
vm-scalability has a similar test called case-truncate which LKP already uses:
https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-truncate
except in case-truncate, the rm is done concurrently and only the
truncate time is reported. I'll modify the case to make it do the rm in
sequential mode and also report the fault time.
Regards,
Aaron
On Thu, Feb 17, 2022 at 12:22:26AM +0000, Mel Gorman wrote:
... ...
> ---
> mm/page_alloc.c | 56 +++++++++++++++++++------------------------------
> 1 file changed, 21 insertions(+), 35 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 635a4e0f70b4..68e2132717c5 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1455,8 +1455,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,
> unsigned int order;
> int prefetch_nr = READ_ONCE(pcp->batch);
> bool isolated_pageblocks;
> - struct page *page, *tmp;
> - LIST_HEAD(head);
> + struct page *page;
>
> /*
> * Ensure proper count is passed which otherwise would stuck in the
> @@ -1467,6 +1466,13 @@ static void free_pcppages_bulk(struct zone *zone, int count,
> /* Ensure requested pindex is drained first. */
> pindex = pindex - 1;
>
> + /*
> + * local_lock_irq held so equivalent to spin_lock_irqsave for
> + * both PREEMPT_RT and non-PREEMPT_RT configurations.
> + */
> + spin_lock(&zone->lock);
> + isolated_pageblocks = has_isolate_pageblock(zone);
> +
> while (count > 0) {
> struct list_head *list;
> int nr_pages;
> @@ -1489,7 +1495,11 @@ static void free_pcppages_bulk(struct zone *zone, int count,
> nr_pages = 1 << order;
> BUILD_BUG_ON(MAX_ORDER >= (1<<NR_PCP_ORDER_WIDTH));
> do {
> + int mt;
> +
> page = list_last_entry(list, struct page, lru);
> + mt = get_pcppage_migratetype(page);
> +
> /* must delete to avoid corrupting pcp list */
> list_del(&page->lru);
> count -= nr_pages;
> @@ -1498,12 +1508,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
> if (bulkfree_pcp_prepare(page))
> continue;
>
> - /* Encode order with the migratetype */
> - page->index <<= NR_PCP_ORDER_WIDTH;
> - page->index |= order;
> -
> - list_add_tail(&page->lru, &head);
> -
> /*
> * We are going to put the page back to the global
> * pool, prefetch its buddy to speed up later access
> @@ -1517,36 +1521,18 @@ static void free_pcppages_bulk(struct zone *zone, int count,
> prefetch_buddy(page, order);
> prefetch_nr--;
> }
The comment above 'if (prefetch_nr)' says: "We are going to put the page
back to the global pool, prefetch its buddy to speed up later access
under zone->lock..." will have to be modified as the prefetch is now
done inside the lock.
I remember prefetch_buddy()'s original intent is to fetch the buddy
page's 'struct page' before acquiring the zone lock to speed up
operations inside the locked region. Now that the zone lock is acquired
early, whether to still keep the prefetch_buddy() inside the lock
becomes questionable.
After the nr_task=4/16/64 tests finished, I'll also test the effect of
removing prefetch_buddy() here.
Thanks,
Aaron
> - } while (count > 0 && !list_empty(list));
> - }
>
> - /*
> - * local_lock_irq held so equivalent to spin_lock_irqsave for
> - * both PREEMPT_RT and non-PREEMPT_RT configurations.
> - */
> - spin_lock(&zone->lock);
> - isolated_pageblocks = has_isolate_pageblock(zone);
> + /* MIGRATE_ISOLATE page should not go to pcplists */
> + VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
> + /* Pageblock could have been isolated meanwhile */
> + if (unlikely(isolated_pageblocks))
> + mt = get_pageblock_migratetype(page);
>
> - /*
> - * Use safe version since after __free_one_page(),
> - * page->lru.next will not point to original list.
> - */
> - list_for_each_entry_safe(page, tmp, &head, lru) {
> - int mt = get_pcppage_migratetype(page);
> -
> - /* mt has been encoded with the order (see above) */
> - order = mt & NR_PCP_ORDER_MASK;
> - mt >>= NR_PCP_ORDER_WIDTH;
> -
> - /* MIGRATE_ISOLATE page should not go to pcplists */
> - VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
> - /* Pageblock could have been isolated meanwhile */
> - if (unlikely(isolated_pageblocks))
> - mt = get_pageblock_migratetype(page);
> -
> - __free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE);
> - trace_mm_page_pcpu_drain(page, order, mt);
> + __free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE);
> + trace_mm_page_pcpu_drain(page, order, mt);
> + } while (count > 0 && !list_empty(list));
> }
> +
> spin_unlock(&zone->lock);
> }
>
> --
> 2.31.1
>
On Fri, Feb 18, 2022 at 12:20:03PM +0800, Aaron Lu wrote:
> > The baseline looks fine. It's different to what I used but the page_alloc
> > shouldn't have much impact.
> >
> > When looking at will-it-scale, please pay attention to lower CPU counts
> > as well and take account changes in standard deviation. Looking at the
>
> I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.
>
Thanks.
> > I expect there will be different good/bad points based on looking at
> > Zen1 results (8 nodes, varying distances, 64 cores with 128 CPUs HT
> > enabled)
> >
> > 5.17.0-rc3 5.17.0-rc3 5.17.0-rc3
> > vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2
> > Hmean page_fault1-threads-2 2985366.46 ( 0.00%) 2984649.41 ( -0.02%) 3028407.35 ( 1.44%)
> > Hmean page_fault1-threads-5 3491833.63 ( 0.00%) 3500237.35 ( 0.24%) 3489971.99 ( -0.05%)
> > Hmean page_fault1-threads-8 3254335.58 ( 0.00%) 3277515.51 * 0.71%* 3234275.28 * -0.62%*
> > Hmean page_fault1-threads-12 5101504.72 ( 0.00%) 5390649.46 * 5.67%* 5162047.68 ( 1.19%)
> > Hmean page_fault1-threads-21 7714265.64 ( 0.00%) 7714763.10 ( 0.01%) 7854367.65 * 1.82%*
> > Hmean page_fault1-threads-30 10034561.94 ( 0.00%) 9865446.68 ( -1.69%) 9746368.76 * -2.87%*
> > Hmean page_fault1-threads-48 12571351.99 ( 0.00%) 13257508.23 * 5.46%* 12160897.07 * -3.27%*
> > Hmean page_fault1-threads-79 11124387.46 ( 0.00%) 10641145.82 * -4.34%* 10677656.39 * -4.02%*
> > Hmean page_fault1-threads-110 11980424.12 ( 0.00%) 10778220.84 * -10.03%* 10354249.62 * -13.57%* <-- close to nr_cpus
> > Hmean page_fault1-threads-141 9727528.73 ( 0.00%) 9966965.70 ( 2.46%) 9656148.13 ( -0.73%) <-- close to nr_cpus
>
> I have never tested thread mode, because I think the heavy loaded
> thread mode is more about testing the mmap_sem contention than page
> allocator's performance?
You're right, I meant to paste in the processes figures and used
processes for the stddev
Hmean page_fault1-processes-2 3087765.27 ( 0.00%) 3040255.24 * -1.54%* 3026943.42 * -1.97%*
Hmean page_fault1-processes-5 3630079.14 ( 0.00%) 3644005.83 * 0.38%* 3641029.26 * 0.30%*
Hmean page_fault1-processes-8 3435519.22 ( 0.00%) 3440525.39 * 0.15%* 3430091.10 * -0.16%*
Hmean page_fault1-processes-12 7060647.54 ( 0.00%) 7078730.32 * 0.26%* 7066516.90 ( 0.08%)
Hmean page_fault1-processes-21 10529603.15 ( 0.00%) 10543342.71 * 0.13%* 10529619.72 ( 0.00%)
Hmean page_fault1-processes-30 13919518.76 ( 0.00%) 13916089.66 ( -0.02%) 13911735.60 * -0.06%*
Hmean page_fault1-processes-48 20655910.65 ( 0.00%) 20680704.25 * 0.12%* 20634196.53 * -0.11%*
Hmean page_fault1-processes-79 27154979.79 ( 0.00%) 27200579.85 * 0.17%* 27111810.79 * -0.16%*
Hmean page_fault1-processes-110 26456190.23 ( 0.00%) 26498119.30 * 0.16%* 26414120.14 * -0.16%*
Hmean page_fault1-processes-141 25741741.47 ( 0.00%) 25377519.19 ( -1.41%) 26020885.64 ( 1.08%)
Hmean page_fault1-processes-172 26029813.28 ( 0.00%) 26107861.43 * 0.30%* 26011987.83 * -0.07%*
Hmean page_fault1-processes-203 26005230.37 ( 0.00%) 26114882.22 * 0.42%* 25999181.70 ( -0.02%)
Hmean page_fault1-processes-234 26021903.34 ( 0.00%) 26123727.47 * 0.39%* 26000412.62 * -0.08%*
Hmean page_fault1-processes-265 26019386.67 ( 0.00%) 26139301.80 * 0.46%* 26014073.54 ( -0.02%)
Hmean page_fault1-processes-296 26014579.15 ( 0.00%) 26101018.62 * 0.33%* 26009459.16 ( -0.02%)
Hmean page_fault1-processes-327 26059483.56 ( 0.00%) 26279026.62 ( 0.84%) 25990821.88 ( -0.26%)
Hmean page_fault1-processes-358 19604338.34 ( 0.00%) 26115341.28 * 33.21%* 25995281.86 * 32.60%*
Hmean page_fault1-processes-389 26084730.88 ( 0.00%) 26058850.78 ( -0.10%) 26007661.51 * -0.30%*
Hmean page_fault1-processes-420 25358929.58 ( 0.00%) 25097140.75 ( -1.03%) 26005923.68 ( 2.55%)
Hmean page_fault1-processes-451 26172808.51 ( 0.00%) 26439611.24 * 1.02%* 26078355.47 ( -0.36%)
Hmean page_fault1-processes-482 26848297.49 ( 0.00%) 26722385.24 ( -0.47%) 26171033.04 * -2.52%*
> It's surprising this patch caused a
> performance change.
>
The figures say it meakes little difference. I wasn't really
concentrating on will-it-scale-pf as such when writing the patch. I
included pf because it was the original justification for deferring
the zone lock acquisition until after pages had been taken off the PCP.
> > Hmean page_fault1-threads-234 11322381.78 ( 0.00%) 9163162.66 ( -19.07%) 9141561.16 ( -19.26%)
> > Hmean page_fault1-threads-265 7956982.52 ( 0.00%) 7774650.20 ( -2.29%) 8292405.57 * 4.22%*
> > Hmean page_fault1-threads-296 7892153.88 ( 0.00%) 8272671.84 * 4.82%* 7907026.20 ( 0.19%)
> > Hmean page_fault1-threads-327 7957124.50 ( 0.00%) 8078297.34 ( 1.52%) 8129776.79 ( 2.17%)
> > Hmean page_fault1-threads-358 7847563.90 ( 0.00%) 8202303.36 ( 4.52%) 8139027.38 ( 3.71%)
> > Hmean page_fault1-threads-389 7928386.47 ( 0.00%) 8104732.41 ( 2.22%) 8022002.73 ( 1.18%)
> > Hmean page_fault1-threads-420 7690107.89 ( 0.00%) 7587821.54 ( -1.33%) 7783777.95 ( 1.22%)
> > Hmean page_fault1-threads-451 7683132.29 ( 0.00%) 7979578.21 ( 3.86%) 7693067.13 ( 0.13%)
> > Hmean page_fault1-threads-482 7720646.31 ( 0.00%) 7597453.65 ( -1.60%) 7870063.90 ( 1.94%)
> > Hmean page_fault1-threads-512 7353458.45 ( 0.00%) 7584407.14 ( 3.14%) 8119539.24 ( 10.42%)
> > Stddev page_fault1-processes-2 4086.39 ( 0.00%) 1698.11 ( 58.44%) 1488.13 ( 63.58%)
> > Stddev page_fault1-processes-5 1448.69 ( 0.00%) 1616.59 ( -11.59%) 1567.37 ( -8.19%)
> > Stddev page_fault1-processes-8 1828.29 ( 0.00%) 2628.59 ( -43.77%) 2701.96 ( -47.79%)
> > Stddev page_fault1-processes-12 14073.12 ( 0.00%) 1575.18 ( 88.81%) 4880.93 ( 65.32%)
> > Stddev page_fault1-processes-21 4368.35 ( 0.00%) 7865.27 ( -80.05%) 3778.03 ( 13.51%)
> > Stddev page_fault1-processes-30 5348.13 ( 0.00%) 11751.43 (-119.73%) 3240.22 ( 39.41%)
> > Stddev page_fault1-processes-48 23687.16 ( 0.00%) 7803.01 ( 67.06%) 2635.85 ( 88.87%)
> > Stddev page_fault1-processes-79 12779.16 ( 0.00%) 4311.60 ( 66.26%) 22539.03 ( -76.37%)
> > Stddev page_fault1-processes-110 21031.04 ( 0.00%) 15115.36 ( 28.13%) 12136.54 ( 42.29%)
> > Stddev page_fault1-processes-141 589804.99 ( 0.00%) 1335519.71 (-126.43%) 19560.01 ( 96.68%)
> > Stddev page_fault1-processes-172 7033.94 ( 0.00%) 7147.71 ( -1.62%) 11366.64 ( -61.60%)
> > Stddev page_fault1-processes-203 6322.20 ( 0.00%) 5035.55 ( 20.35%) 4043.45 ( 36.04%)
> > Stddev page_fault1-processes-234 12046.53 ( 0.00%) 24208.37 (-100.96%) 9159.91 ( 23.96%)
> > Stddev page_fault1-processes-265 11869.43 ( 0.00%) 13528.26 ( -13.98%) 8943.99 ( 24.65%)
> > Stddev page_fault1-processes-296 8918.50 ( 0.00%) 16130.54 ( -80.87%) 5211.80 ( 41.56%)
> > Stddev page_fault1-processes-327 101102.64 ( 0.00%) 845864.70 (-736.64%) 16238.99 ( 83.94%)
> > Stddev page_fault1-processes-358 2102190.38 ( 0.00%) 11316.00 ( 99.46%) 7508.57 ( 99.64%)
> > Stddev page_fault1-processes-389 61012.79 ( 0.00%) 121446.55 ( -99.05%) 18279.64 ( 70.04%)
> > Stddev page_fault1-processes-420 2305208.40 ( 0.00%) 2347564.71 ( -1.84%) 3202.77 ( 99.86%)
> > Stddev page_fault1-processes-451 20214.37 ( 0.00%) 173800.17 (-759.79%) 492258.35 (-2335.19%)
> > Stddev page_fault1-processes-482 236881.21 ( 0.00%) 330501.32 ( -39.52%) 15307.31 ( 93.54%)
> > Stddev page_fault1-processes-512 201354.82 ( 0.00%) 207019.93 ( -2.81%) 4900536.90 (-2333.78%)
> >
> > This is showing there was a impact around the nr_cpus (110 and 141
> > processes measured) but the standard deviation around 141 was particularly
> ~~~~~~~~~
>
> Did you mean threads?
>
I meant processes both times and based the reasoning on processes and
pasted the wrong thing. I'm going to split this config into threads
versions and processes versions because they measure different things
and considering them together in the context of the same test is hazardous.
> > If possible, it would be nice if you could add something like
> > configs/config-io-trunc from mmtests to lkp if it doesn't exist already
> > to consider the simple case. As its most basic, all it's doing is
> >
> > ---8<---
> > #!/bin/bash
> >
> > for i in {1..10}; do
> > dd if=/dev/zero of=sparse_file-$i bs=1 count=0 seek=1G &>/dev/null
> > cat sparse_file-$i > /dev/null
> > done
> > sync
> >
> > # Primary metric
> > time rm sparse_file*
> > ---8<---
> >
> > The main difference is that the mmtests will report the time to fault the
> > sparse files (bulk simple allocate inserting into page cache) as well as
> > the bulk truncate (bulk simple release of page cache).
>
> Thanks for the suggestion.
>
> vm-scalability has a similar test called case-truncate which LKP already uses:
> https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-truncate
> except in case-truncate, the rm is done concurrently and only the
> truncate time is reported.
This is still a valid test except you may also be measuring LRU lock
contention so it'll be less clear for evaluating this series unless the
scale factor is 1.
> I'll modify the case to make it do the rm in
> sequential mode and also report the fault time.
>
Thanks.
--
Mel Gorman
SUSE Labs
On Fri, Feb 18, 2022 at 09:47:16AM +0000, Mel Gorman wrote:
> On Fri, Feb 18, 2022 at 02:07:42PM +0800, Aaron Lu wrote:
> > > @@ -1498,12 +1508,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
> > > if (bulkfree_pcp_prepare(page))
> > > continue;
> > >
> > > - /* Encode order with the migratetype */
> > > - page->index <<= NR_PCP_ORDER_WIDTH;
> > > - page->index |= order;
> > > -
> > > - list_add_tail(&page->lru, &head);
> > > -
> > > /*
> > > * We are going to put the page back to the global
> > > * pool, prefetch its buddy to speed up later access
> > > @@ -1517,36 +1521,18 @@ static void free_pcppages_bulk(struct zone *zone, int count,
> > > prefetch_buddy(page, order);
> > > prefetch_nr--;
> > > }
> >
> > The comment above 'if (prefetch_nr)' says: "We are going to put the page
> > back to the global pool, prefetch its buddy to speed up later access
> > under zone->lock..." will have to be modified as the prefetch is now
> > done inside the lock.
> >
>
> Yes, that was my understanding.
>
> > I remember prefetch_buddy()'s original intent is to fetch the buddy
> > page's 'struct page' before acquiring the zone lock to speed up
> > operations inside the locked region. Now that the zone lock is acquired
> > early, whether to still keep the prefetch_buddy() inside the lock
> > becomes questionable.
> >
>
> I agree. I wanted to take it out but worried it might stall (drumroll)
> the rest of the series as evaluating prefetch is machine specific. Before
Understood.
> this series I thought it was possible that the prefetched lines would be
> flushed if the lists were large enough. Due to free_factor, it's possible
> we are 10's of thousands of pages and the prefetched pages would be
> evicted. It would require a fairly small cache though.
Makes sense.
>
> There are still two reasons why I thought it should go away as a
> follow-up to the series.
>
> 1. There is a guaranteed cost to calculating the buddy which definitely
> has to be calculated again. However, as the zone lock is held and
> there is no deferring of buddy merging, there is no guarantee that the
> prefetch will have completed when the second buddy calculation takes
> place and buddies are being merged. With or without the prefetch, there
> may be further stalls depending on how many pages get merged. In other
> words, a stall due to merging is inevitable and at best only one stall
> might be avoided at the cost of calculating the buddy location twice.
>
> 2. As the zone lock is held, prefetch_nr makes less sense as once
> prefetch_nr expires, the cache lines of interest have already been
> merged.
>
> It's point 1 that was my main concern. We are paying a guaranteed cost for
> a maybe win if prefetching is fast enough and it would be very difficult to
> spot what percentage of prefetches actually helped. It was more clear-cut
> when the buddy freeing was deferred as there was more time for the prefetch
> to complete.
Both points make sense to me.
I'm also thinking since zone lock contention is much better
now(presumbly due to your free_factor patchset) than before, these
techniques(pick pages to free before acquiring lock and prefetch buddy
on free path) make less sense now.
>
> > After the nr_task=4/16/64 tests finished, I'll also test the effect of
> > removing prefetch_buddy() here.
> >
>
> I'd appreciate it. I think the patch is this (build tested only);
>
Looks good to me, thanks!
> --8<--
> mm/page_alloc: Do not prefetch buddies during bulk free
>
> free_pcppages_bulk() has taken two passes through the pcp lists since
> commit 0a5f4e5b4562 ("mm/free_pcppages_bulk: do not hold lock when picking
> pages to free") due to deferring the cost of selecting PCP lists until
> the zone lock is held.
>
> As the list processing now takes place under the zone lock, it's less
> clear that this will always benefit for two reasons.
>
> 1. There is a guaranteed cost to calculating the buddy which definitely
> has to be calculated again. However, as the zone lock is held and
> there is no deferring of buddy merging, there is no guarantee that the
> prefetch will have completed when the second buddy calculation takes
> place and buddies are being merged. With or without the prefetch, there
> may be further stalls depending on how many pages get merged. In other
> words, a stall due to merging is inevitable and at best only one stall
> might be avoided at the cost of calculating the buddy location twice.
>
> 2. As the zone lock is held, prefetch_nr makes less sense as once
> prefetch_nr expires, the cache lines of interest have already been
> merged.
>
> The main concern is that there is a definite cost to calculating the
> buddy location early for the prefetch and it is a "maybe win" depending
> on whether the CPU prefetch logic and memory is fast enough. Remove the
> prefetch logic on the basis that reduced instructions in a path is always
> a saving where as the prefetch might save one memory stall depending on
> the CPU and memory.
>
> Suggested-by: Aaron Lu <[email protected]>
> Signed-off-by: Mel Gorman <[email protected]>
> ---
> mm/page_alloc.c | 24 ------------------------
> 1 file changed, 24 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index de9f072d23bd..2d5cc098136d 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1432,15 +1432,6 @@ static bool bulkfree_pcp_prepare(struct page *page)
> }
> #endif /* CONFIG_DEBUG_VM */
>
> -static inline void prefetch_buddy(struct page *page, unsigned int order)
> -{
> - unsigned long pfn = page_to_pfn(page);
> - unsigned long buddy_pfn = __find_buddy_pfn(pfn, order);
> - struct page *buddy = page + (buddy_pfn - pfn);
> -
> - prefetch(buddy);
> -}
> -
> /*
> * Frees a number of pages from the PCP lists
> * Assumes all pages on list are in same zone.
> @@ -1453,7 +1444,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
> int min_pindex = 0;
> int max_pindex = NR_PCP_LISTS - 1;
> unsigned int order;
> - int prefetch_nr = READ_ONCE(pcp->batch);
> bool isolated_pageblocks;
> struct page *page;
>
> @@ -1508,20 +1498,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
> if (bulkfree_pcp_prepare(page))
> continue;
>
> - /*
> - * We are going to put the page back to the global
> - * pool, prefetch its buddy to speed up later access
> - * under zone->lock. It is believed the overhead of
> - * an additional test and calculating buddy_pfn here
> - * can be offset by reduced memory latency later. To
> - * avoid excessive prefetching due to large count, only
> - * prefetch buddy for the first pcp->batch nr of pages.
> - */
> - if (prefetch_nr) {
> - prefetch_buddy(page, order);
> - prefetch_nr--;
> - }
> -
> /* MIGRATE_ISOLATE page should not go to pcplists */
> VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
> /* Pageblock could have been isolated meanwhile */
On Fri, Feb 18, 2022 at 02:07:42PM +0800, Aaron Lu wrote:
> > @@ -1498,12 +1508,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
> > if (bulkfree_pcp_prepare(page))
> > continue;
> >
> > - /* Encode order with the migratetype */
> > - page->index <<= NR_PCP_ORDER_WIDTH;
> > - page->index |= order;
> > -
> > - list_add_tail(&page->lru, &head);
> > -
> > /*
> > * We are going to put the page back to the global
> > * pool, prefetch its buddy to speed up later access
> > @@ -1517,36 +1521,18 @@ static void free_pcppages_bulk(struct zone *zone, int count,
> > prefetch_buddy(page, order);
> > prefetch_nr--;
> > }
>
> The comment above 'if (prefetch_nr)' says: "We are going to put the page
> back to the global pool, prefetch its buddy to speed up later access
> under zone->lock..." will have to be modified as the prefetch is now
> done inside the lock.
>
Yes, that was my understanding.
> I remember prefetch_buddy()'s original intent is to fetch the buddy
> page's 'struct page' before acquiring the zone lock to speed up
> operations inside the locked region. Now that the zone lock is acquired
> early, whether to still keep the prefetch_buddy() inside the lock
> becomes questionable.
>
I agree. I wanted to take it out but worried it might stall (drumroll)
the rest of the series as evaluating prefetch is machine specific. Before
this series I thought it was possible that the prefetched lines would be
flushed if the lists were large enough. Due to free_factor, it's possible
we are 10's of thousands of pages and the prefetched pages would be
evicted. It would require a fairly small cache though.
There are still two reasons why I thought it should go away as a
follow-up to the series.
1. There is a guaranteed cost to calculating the buddy which definitely
has to be calculated again. However, as the zone lock is held and
there is no deferring of buddy merging, there is no guarantee that the
prefetch will have completed when the second buddy calculation takes
place and buddies are being merged. With or without the prefetch, there
may be further stalls depending on how many pages get merged. In other
words, a stall due to merging is inevitable and at best only one stall
might be avoided at the cost of calculating the buddy location twice.
2. As the zone lock is held, prefetch_nr makes less sense as once
prefetch_nr expires, the cache lines of interest have already been
merged.
It's point 1 that was my main concern. We are paying a guaranteed cost for
a maybe win if prefetching is fast enough and it would be very difficult to
spot what percentage of prefetches actually helped. It was more clear-cut
when the buddy freeing was deferred as there was more time for the prefetch
to complete.
> After the nr_task=4/16/64 tests finished, I'll also test the effect of
> removing prefetch_buddy() here.
>
I'd appreciate it. I think the patch is this (build tested only);
--8<--
mm/page_alloc: Do not prefetch buddies during bulk free
free_pcppages_bulk() has taken two passes through the pcp lists since
commit 0a5f4e5b4562 ("mm/free_pcppages_bulk: do not hold lock when picking
pages to free") due to deferring the cost of selecting PCP lists until
the zone lock is held.
As the list processing now takes place under the zone lock, it's less
clear that this will always benefit for two reasons.
1. There is a guaranteed cost to calculating the buddy which definitely
has to be calculated again. However, as the zone lock is held and
there is no deferring of buddy merging, there is no guarantee that the
prefetch will have completed when the second buddy calculation takes
place and buddies are being merged. With or without the prefetch, there
may be further stalls depending on how many pages get merged. In other
words, a stall due to merging is inevitable and at best only one stall
might be avoided at the cost of calculating the buddy location twice.
2. As the zone lock is held, prefetch_nr makes less sense as once
prefetch_nr expires, the cache lines of interest have already been
merged.
The main concern is that there is a definite cost to calculating the
buddy location early for the prefetch and it is a "maybe win" depending
on whether the CPU prefetch logic and memory is fast enough. Remove the
prefetch logic on the basis that reduced instructions in a path is always
a saving where as the prefetch might save one memory stall depending on
the CPU and memory.
Suggested-by: Aaron Lu <[email protected]>
Signed-off-by: Mel Gorman <[email protected]>
---
mm/page_alloc.c | 24 ------------------------
1 file changed, 24 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index de9f072d23bd..2d5cc098136d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1432,15 +1432,6 @@ static bool bulkfree_pcp_prepare(struct page *page)
}
#endif /* CONFIG_DEBUG_VM */
-static inline void prefetch_buddy(struct page *page, unsigned int order)
-{
- unsigned long pfn = page_to_pfn(page);
- unsigned long buddy_pfn = __find_buddy_pfn(pfn, order);
- struct page *buddy = page + (buddy_pfn - pfn);
-
- prefetch(buddy);
-}
-
/*
* Frees a number of pages from the PCP lists
* Assumes all pages on list are in same zone.
@@ -1453,7 +1444,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
int min_pindex = 0;
int max_pindex = NR_PCP_LISTS - 1;
unsigned int order;
- int prefetch_nr = READ_ONCE(pcp->batch);
bool isolated_pageblocks;
struct page *page;
@@ -1508,20 +1498,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
if (bulkfree_pcp_prepare(page))
continue;
- /*
- * We are going to put the page back to the global
- * pool, prefetch its buddy to speed up later access
- * under zone->lock. It is believed the overhead of
- * an additional test and calculating buddy_pfn here
- * can be offset by reduced memory latency later. To
- * avoid excessive prefetching due to large count, only
- * prefetch buddy for the first pcp->batch nr of pages.
- */
- if (prefetch_nr) {
- prefetch_buddy(page, order);
- prefetch_nr--;
- }
-
/* MIGRATE_ISOLATE page should not go to pcplists */
VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
/* Pageblock could have been isolated meanwhile */
On Fri, Feb 18, 2022 at 12:20:03PM +0800, Aaron Lu wrote:
> On Thu, Feb 17, 2022 at 09:31:13AM +0000, Mel Gorman wrote:
> > On Thu, Feb 17, 2022 at 09:53:08AM +0800, Aaron Lu wrote:
> > > > 2-socket CascadeLake (40 cores, 80 CPUs HT enabled)
> > > > 5.17.0-rc3 5.17.0-rc3
> > > > vanilla mm-highpcpopt-v2
> > > > Hmean page_fault1-processes-2 2694662.26 ( 0.00%) 2695780.35 ( 0.04%)
> > > > Hmean page_fault1-processes-5 6425819.34 ( 0.00%) 6435544.57 * 0.15%*
> > > > Hmean page_fault1-processes-8 9642169.10 ( 0.00%) 9658962.39 ( 0.17%)
> > > > Hmean page_fault1-processes-12 12167502.10 ( 0.00%) 12190163.79 ( 0.19%)
> > > > Hmean page_fault1-processes-21 15636859.03 ( 0.00%) 15612447.26 ( -0.16%)
> > > > Hmean page_fault1-processes-30 25157348.61 ( 0.00%) 25169456.65 ( 0.05%)
> > > > Hmean page_fault1-processes-48 27694013.85 ( 0.00%) 27671111.46 ( -0.08%)
> > > > Hmean page_fault1-processes-79 25928742.64 ( 0.00%) 25934202.02 ( 0.02%) <--
> > > > Hmean page_fault1-processes-110 25730869.75 ( 0.00%) 25671880.65 * -0.23%*
> > > > Hmean page_fault1-processes-141 25626992.42 ( 0.00%) 25629551.61 ( 0.01%)
> > > > Hmean page_fault1-processes-172 25611651.35 ( 0.00%) 25614927.99 ( 0.01%)
> > > > Hmean page_fault1-processes-203 25577298.75 ( 0.00%) 25583445.59 ( 0.02%)
> > > > Hmean page_fault1-processes-234 25580686.07 ( 0.00%) 25608240.71 ( 0.11%)
> > > > Hmean page_fault1-processes-265 25570215.47 ( 0.00%) 25568647.58 ( -0.01%)
> > > > Hmean page_fault1-processes-296 25549488.62 ( 0.00%) 25543935.00 ( -0.02%)
> > > > Hmean page_fault1-processes-320 25555149.05 ( 0.00%) 25575696.74 ( 0.08%)
> > > >
> > > > The differences are mostly within the noise and the difference close to
> > > > $nr_cpus is negligible.
> > >
> > > I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
> > > servers: CascadeLake and CooperLaker and will let you know the result
> > > once it's out.
> > >
> >
> > Thanks, 4 sockets and a later generation would be nice to cover.
> >
> > > I'm using 'https://github.com/hnaz/linux-mm master' and doing the
> > > comparison with commit c000d687ce22("mm/page_alloc: simplify how many
> > > pages are selected per pcp list during bulk free") and commit 8391e0a7e172
> > > ("mm/page_alloc: free pages in a single pass during bulk free") there.
> > >
> >
> > The baseline looks fine. It's different to what I used but the page_alloc
> > shouldn't have much impact.
> >
> > When looking at will-it-scale, please pay attention to lower CPU counts
> > as well and take account changes in standard deviation. Looking at the
>
> I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.
>
For the record, these tests don't show any visible performance changes
on CooperLake.
On Mon, Feb 21, 2022 at 09:38:22PM +0800, Aaron Lu wrote:
> On Fri, Feb 18, 2022 at 12:20:03PM +0800, Aaron Lu wrote:
> > On Thu, Feb 17, 2022 at 09:31:13AM +0000, Mel Gorman wrote:
> > > On Thu, Feb 17, 2022 at 09:53:08AM +0800, Aaron Lu wrote:
> > > > > 2-socket CascadeLake (40 cores, 80 CPUs HT enabled)
> > > > > 5.17.0-rc3 5.17.0-rc3
> > > > > vanilla mm-highpcpopt-v2
> > > > > Hmean page_fault1-processes-2 2694662.26 ( 0.00%) 2695780.35 ( 0.04%)
> > > > > Hmean page_fault1-processes-5 6425819.34 ( 0.00%) 6435544.57 * 0.15%*
> > > > > Hmean page_fault1-processes-8 9642169.10 ( 0.00%) 9658962.39 ( 0.17%)
> > > > > Hmean page_fault1-processes-12 12167502.10 ( 0.00%) 12190163.79 ( 0.19%)
> > > > > Hmean page_fault1-processes-21 15636859.03 ( 0.00%) 15612447.26 ( -0.16%)
> > > > > Hmean page_fault1-processes-30 25157348.61 ( 0.00%) 25169456.65 ( 0.05%)
> > > > > Hmean page_fault1-processes-48 27694013.85 ( 0.00%) 27671111.46 ( -0.08%)
> > > > > Hmean page_fault1-processes-79 25928742.64 ( 0.00%) 25934202.02 ( 0.02%) <--
> > > > > Hmean page_fault1-processes-110 25730869.75 ( 0.00%) 25671880.65 * -0.23%*
> > > > > Hmean page_fault1-processes-141 25626992.42 ( 0.00%) 25629551.61 ( 0.01%)
> > > > > Hmean page_fault1-processes-172 25611651.35 ( 0.00%) 25614927.99 ( 0.01%)
> > > > > Hmean page_fault1-processes-203 25577298.75 ( 0.00%) 25583445.59 ( 0.02%)
> > > > > Hmean page_fault1-processes-234 25580686.07 ( 0.00%) 25608240.71 ( 0.11%)
> > > > > Hmean page_fault1-processes-265 25570215.47 ( 0.00%) 25568647.58 ( -0.01%)
> > > > > Hmean page_fault1-processes-296 25549488.62 ( 0.00%) 25543935.00 ( -0.02%)
> > > > > Hmean page_fault1-processes-320 25555149.05 ( 0.00%) 25575696.74 ( 0.08%)
> > > > >
> > > > > The differences are mostly within the noise and the difference close to
> > > > > $nr_cpus is negligible.
> > > >
> > > > I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
> > > > servers: CascadeLake and CooperLaker and will let you know the result
> > > > once it's out.
> > > >
> > >
> > > Thanks, 4 sockets and a later generation would be nice to cover.
> > >
> > > > I'm using 'https://github.com/hnaz/linux-mm master' and doing the
> > > > comparison with commit c000d687ce22("mm/page_alloc: simplify how many
> > > > pages are selected per pcp list during bulk free") and commit 8391e0a7e172
> > > > ("mm/page_alloc: free pages in a single pass during bulk free") there.
> > > >
> > >
> > > The baseline looks fine. It's different to what I used but the page_alloc
> > > shouldn't have much impact.
> > >
> > > When looking at will-it-scale, please pay attention to lower CPU counts
> > > as well and take account changes in standard deviation. Looking at the
> >
> > I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.
> >
>
> For the record, these tests don't show any visible performance changes
> on CooperLake.
One thing I just noticed is that, zone lock contention increased to some
extent. I'm not sure if this is worrisome so I suppose I should at least
mention it here.
The nr_task=100% test on the 4 sockets Cooper Lake showed that zone lock
contention increased from 13.56% to 20.16% and for nr_task=16, it
increased from 4.75% to 6.18%.
The reason is probably due to more code are now inside the lock and when
there is contention, it will make things worse. I'm aware of that
nr_task=100% is a rare case and this patchset is meant to improve things
when there is very little contention, which should be the common case.
So I guess that's just the tradeoff we have to make...
Here are the results on performance metric and zone lock metrics:
nr_task=100%
=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/nr_task/mode/test/thp_enabled/cpufreq_governor:
lkp-cpl-4sp1/will-it-scale/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/100%/process/page_fault1/never/performance
commit/ucode:
8391e0a7e1728d74faecebf096b446ac5d0a5709/0x7002302 (mm/page_alloc: free pages in a single pass during bulk free)
c000d687ce22252c8ea96e47b4a2add592fbad6c/0x7002302 (mm/page_alloc: simplify how many pages are selected per pcp list during bulk free)
7decb609034044e56cffd1c9971738878467ee96/0x7002402 (mm/page_alloc: Do not prefetch buddies during bulk free)
8391e0a7e1728d74 c000d687ce22252c8ea96e47b4a 7decb609034044e56cffd1c9971
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
11807831 -0.5% 11750578 -0.3% 11778047 will-it-scale.144.processes
15.44 ? 10% -4.9 10.58 ? 8% +0.6 16.01 ? 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.rmqueue_bulk.get_page_from_freelist.__alloc_pages
4.72 ? 8% -1.7 2.98 -0.1 4.63 ? 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_pcppages_bulk.free_unref_page_list.release_pages
nr_task=16
=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/nr_task/mode/test/thp_enabled/cpufreq_governor/ucode:
lkp-cpl-4sp1/will-it-scale/debian-10.4-x86_64-20200603.cgz/x86_64-rhel-8.3/gcc-9/16/process/page_fault1/never/performance/0x7002402
commit:
8391e0a7e1728d74faecebf096b446ac5d0a5709 (mm/page_alloc: free pages in a single pass during bulk free)
c000d687ce22252c8ea96e47b4a2add592fbad6c (mm/page_alloc: simplify how many pages are selected per pcp list during bulk free)
7decb609034044e56cffd1c9971738878467ee96 (mm/page_alloc: Do not prefetch buddies during bulk free)
8391e0a7e1728d74 c000d687ce22252c8ea96e47b4a 7decb609034044e56cffd1c9971
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
3410615 +0.2% 3416565 +0.2% 3415846 will-it-scale.16.processes
4.83 ? 3% -1.1 3.76 ? 9% -0.4 4.40 ? 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.rmqueue_bulk.get_page_from_freelist.__alloc_pages
1.35 ? 9% -0.4 0.99 ? 14% -0.2 1.17 ? 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_pcppages_bulk.free_unref_page_list.release_pages
Regards,
Aaron
On Wed, Feb 23, 2022 at 07:30:52PM +0800, Aaron Lu wrote:
> On Mon, Feb 21, 2022 at 09:38:22PM +0800, Aaron Lu wrote:
> > On Fri, Feb 18, 2022 at 12:20:03PM +0800, Aaron Lu wrote:
> > > On Thu, Feb 17, 2022 at 09:31:13AM +0000, Mel Gorman wrote:
> > > > On Thu, Feb 17, 2022 at 09:53:08AM +0800, Aaron Lu wrote:
> > > > > > 2-socket CascadeLake (40 cores, 80 CPUs HT enabled)
> > > > > > 5.17.0-rc3 5.17.0-rc3
> > > > > > vanilla mm-highpcpopt-v2
> > > > > > Hmean page_fault1-processes-2 2694662.26 ( 0.00%) 2695780.35 ( 0.04%)
> > > > > > Hmean page_fault1-processes-5 6425819.34 ( 0.00%) 6435544.57 * 0.15%*
> > > > > > Hmean page_fault1-processes-8 9642169.10 ( 0.00%) 9658962.39 ( 0.17%)
> > > > > > Hmean page_fault1-processes-12 12167502.10 ( 0.00%) 12190163.79 ( 0.19%)
> > > > > > Hmean page_fault1-processes-21 15636859.03 ( 0.00%) 15612447.26 ( -0.16%)
> > > > > > Hmean page_fault1-processes-30 25157348.61 ( 0.00%) 25169456.65 ( 0.05%)
> > > > > > Hmean page_fault1-processes-48 27694013.85 ( 0.00%) 27671111.46 ( -0.08%)
> > > > > > Hmean page_fault1-processes-79 25928742.64 ( 0.00%) 25934202.02 ( 0.02%) <--
> > > > > > Hmean page_fault1-processes-110 25730869.75 ( 0.00%) 25671880.65 * -0.23%*
> > > > > > Hmean page_fault1-processes-141 25626992.42 ( 0.00%) 25629551.61 ( 0.01%)
> > > > > > Hmean page_fault1-processes-172 25611651.35 ( 0.00%) 25614927.99 ( 0.01%)
> > > > > > Hmean page_fault1-processes-203 25577298.75 ( 0.00%) 25583445.59 ( 0.02%)
> > > > > > Hmean page_fault1-processes-234 25580686.07 ( 0.00%) 25608240.71 ( 0.11%)
> > > > > > Hmean page_fault1-processes-265 25570215.47 ( 0.00%) 25568647.58 ( -0.01%)
> > > > > > Hmean page_fault1-processes-296 25549488.62 ( 0.00%) 25543935.00 ( -0.02%)
> > > > > > Hmean page_fault1-processes-320 25555149.05 ( 0.00%) 25575696.74 ( 0.08%)
> > > > > >
> > > > > > The differences are mostly within the noise and the difference close to
> > > > > > $nr_cpus is negligible.
> > > > >
> > > > > I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
> > > > > servers: CascadeLake and CooperLaker and will let you know the result
> > > > > once it's out.
> > > > >
> > > >
> > > > Thanks, 4 sockets and a later generation would be nice to cover.
> > > >
> > > > > I'm using 'https://github.com/hnaz/linux-mm master' and doing the
> > > > > comparison with commit c000d687ce22("mm/page_alloc: simplify how many
> > > > > pages are selected per pcp list during bulk free") and commit 8391e0a7e172
> > > > > ("mm/page_alloc: free pages in a single pass during bulk free") there.
> > > > >
> > > >
> > > > The baseline looks fine. It's different to what I used but the page_alloc
> > > > shouldn't have much impact.
> > > >
> > > > When looking at will-it-scale, please pay attention to lower CPU counts
> > > > as well and take account changes in standard deviation. Looking at the
> > >
> > > I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.
> > >
> >
> > For the record, these tests don't show any visible performance changes
> > on CooperLake.
>
> One thing I just noticed is that, zone lock contention increased to some
> extent. I'm not sure if this is worrisome so I suppose I should at least
> mention it here.
>
> The nr_task=100% test on the 4 sockets Cooper Lake showed that zone lock
> contention increased from 13.56% to 20.16% and for nr_task=16, it
> increased from 4.75% to 6.18%.
>
> The reason is probably due to more code are now inside the lock and when
> there is contention, it will make things worse. I'm aware of that
> nr_task=100% is a rare case and this patchset is meant to improve things
> when there is very little contention, which should be the common case.
> So I guess that's just the tradeoff we have to make...
>
I think it's a reasonable tradeoff. The page_fault1 will-it-scale is
an extreme case that exercises severe contention for the zone lock with
both allocators and freeing contending for the lock at the same time. I
think it's reasonable to optimise for the common case of completing the
bulk freeing as quickly as possible. If anything, I think will-it-scale
would benefit more if zone->lock was split to cover regions within a zone
instead of protecting an entire zone which could be hundreds of GB in size.
--
Mel Gorman
SUSE Labs
On Wed, 2022-02-23 at 13:05 +0000, Mel Gorman wrote:
> On Wed, Feb 23, 2022 at 07:30:52PM +0800, Aaron Lu wrote:
> > On Mon, Feb 21, 2022 at 09:38:22PM +0800, Aaron Lu wrote:
> > > On Fri, Feb 18, 2022 at 12:20:03PM +0800, Aaron Lu wrote:
> > > > On Thu, Feb 17, 2022 at 09:31:13AM +0000, Mel Gorman wrote:
> > > > > On Thu, Feb 17, 2022 at 09:53:08AM +0800, Aaron Lu wrote:
> > > > > > > 2-socket CascadeLake (40 cores, 80 CPUs HT enabled)
> > > > > > > 5.17.0-rc3 5.17.0-rc3
> > > > > > > vanilla mm-highpcpopt-v2
> > > > > > > Hmean page_fault1-processes-2 2694662.26 ( 0.00%) 2695780.35 ( 0.04%)
> > > > > > > Hmean page_fault1-processes-5 6425819.34 ( 0.00%) 6435544.57 * 0.15%*
> > > > > > > Hmean page_fault1-processes-8 9642169.10 ( 0.00%) 9658962.39 ( 0.17%)
> > > > > > > Hmean page_fault1-processes-12 12167502.10 ( 0.00%) 12190163.79 ( 0.19%)
> > > > > > > Hmean page_fault1-processes-21 15636859.03 ( 0.00%) 15612447.26 ( -0.16%)
> > > > > > > Hmean page_fault1-processes-30 25157348.61 ( 0.00%) 25169456.65 ( 0.05%)
> > > > > > > Hmean page_fault1-processes-48 27694013.85 ( 0.00%) 27671111.46 ( -0.08%)
> > > > > > > Hmean page_fault1-processes-79 25928742.64 ( 0.00%) 25934202.02 ( 0.02%) <--
> > > > > > > Hmean page_fault1-processes-110 25730869.75 ( 0.00%) 25671880.65 * -0.23%*
> > > > > > > Hmean page_fault1-processes-141 25626992.42 ( 0.00%) 25629551.61 ( 0.01%)
> > > > > > > Hmean page_fault1-processes-172 25611651.35 ( 0.00%) 25614927.99 ( 0.01%)
> > > > > > > Hmean page_fault1-processes-203 25577298.75 ( 0.00%) 25583445.59 ( 0.02%)
> > > > > > > Hmean page_fault1-processes-234 25580686.07 ( 0.00%) 25608240.71 ( 0.11%)
> > > > > > > Hmean page_fault1-processes-265 25570215.47 ( 0.00%) 25568647.58 ( -0.01%)
> > > > > > > Hmean page_fault1-processes-296 25549488.62 ( 0.00%) 25543935.00 ( -0.02%)
> > > > > > > Hmean page_fault1-processes-320 25555149.05 ( 0.00%) 25575696.74 ( 0.08%)
> > > > > > >
> > > > > > > The differences are mostly within the noise and the difference close to
> > > > > > > $nr_cpus is negligible.
> > > > > >
> > > > > > I have queued will-it-scale/page_fault1/processes/$nr_cpu on 2 4-sockets
> > > > > > servers: CascadeLake and CooperLaker and will let you know the result
> > > > > > once it's out.
> > > > > >
> > > > >
> > > > > Thanks, 4 sockets and a later generation would be nice to cover.
> > > > >
> > > > > > I'm using 'https://github.com/hnaz/linux-mm master' and doing the
> > > > > > comparison with commit c000d687ce22("mm/page_alloc: simplify how many
> > > > > > pages are selected per pcp list during bulk free") and commit 8391e0a7e172
> > > > > > ("mm/page_alloc: free pages in a single pass during bulk free") there.
> > > > > >
> > > > >
> > > > > The baseline looks fine. It's different to what I used but the page_alloc
> > > > > shouldn't have much impact.
> > > > >
> > > > > When looking at will-it-scale, please pay attention to lower CPU counts
> > > > > as well and take account changes in standard deviation. Looking at the
> > > >
> > > > I'll also test nr_task=4/16/64 on the 4sockets CooperLake(nr_cpu=144) then.
> > > >
> > >
> > > For the record, these tests don't show any visible performance changes
> > > on CooperLake.
> >
> > One thing I just noticed is that, zone lock contention increased to some
> > extent. I'm not sure if this is worrisome so I suppose I should at least
> > mention it here.
> >
> > The nr_task=100% test on the 4 sockets Cooper Lake showed that zone lock
> > contention increased from 13.56% to 20.16% and for nr_task=16, it
> > increased from 4.75% to 6.18%.
> >
> > The reason is probably due to more code are now inside the lock and when
> > there is contention, it will make things worse. I'm aware of that
> > nr_task=100% is a rare case and this patchset is meant to improve things
> > when there is very little contention, which should be the common case.
> > So I guess that's just the tradeoff we have to make...
> >
>
> I think it's a reasonable tradeoff. The page_fault1 will-it-scale is
> an extreme case that exercises severe contention for the zone lock with
> both allocators and freeing contending for the lock at the same time. I
> think it's reasonable to optimise for the common case of completing the
> bulk freeing as quickly as possible. If anything, I think will-it-scale
> would benefit more if zone->lock was split to cover regions within a zone
> instead of protecting an entire zone which could be hundreds of GB in size.
>
Thanks for the explanation and suggestion on how to deal with zone lock
contention.