If GFP_ATOMIC allocations will start failing soon because the amount of
free memory is substantially under per-zone min watermarks, it is better
to oom kill a process rather than continue to reclaim.
This intends to significantly reduce the number of page allocation
failures that are encountered when the demands of user and atomic
allocations overwhelm the ability of reclaim to keep up. We can see this
with a high ingress of networking traffic where memory allocated in irq
context can overwhelm the ability to reclaim fast enough such that user
memory consistently loops. In that case, we have reclaimable memory, and
reclaiming is successful, but we've fully depleted memory reserves that
are allowed for non-blockable allocations.
Commit 400e22499dd9 ("mm: don't warn about allocations which stall for
too long") removed evidence of user allocations stalling because of this,
but the situation can apply anytime we get "page allocation failures"
where reclaim is happening but per-zone min watermarks are starved:
Node 0 Normal free:87356kB min:221984kB low:416984kB high:611984kB active_anon:123009936kB inactive_anon:67647652kB active_file:429612kB inactive_file:209980kB unevictable:112348kB writepending:260kB present:198180864kB managed:195027624kB mlocked:81756kB kernel_stack:24040kB pagetables:11460kB bounce:0kB free_pcp:940kB local_pcp:96kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:105616kB min:225568kB low:423716kB high:621864kB active_anon:122124196kB inactive_anon:74112696kB active_file:39172kB inactive_file:103696kB unevictable:204480kB writepending:180kB present:201326592kB managed:198174372kB mlocked:204480kB kernel_stack:11328kB pagetables:3680kB bounce:0kB free_pcp:1140kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0
Without this patch, there is no guarantee that user memory allocations
will ever be successful when non-blockable allocations overwhelm the
ability to get above per-zone min watermarks.
This doesn't solve page allocation failures entirely since it's a
preemptive measure based on watermarks that requires concurrent blockable
allocations to trigger the oom kill. To complete solve page allocation
failures, it would be possible to do the same watermark check for non-
blockable allocations and then queue a worker to asynchronously oom kill
if it finds watermarks to be sufficiently low as well.
Signed-off-by: David Rientjes <[email protected]>
---
mm/page_alloc.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4372,11 +4372,21 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
ac->nodemask) {
unsigned long available;
unsigned long reclaimable;
+ unsigned long free;
unsigned long min_wmark = min_wmark_pages(zone);
bool wmark;
+ free = zone_page_state_snapshot(zone, NR_FREE_PAGES);
+ /*
+ * If this zone is approaching the point where even order-0
+ * GFP_ATOMIC allocations will fail, stop considering reclaim.
+ */
+ if (!__zone_watermark_ok(zone, 0, min_wmark, ac_classzone_idx(ac),
+ alloc_flags | ALLOC_HIGH, free))
+ continue;
+
available = reclaimable = zone_reclaimable_pages(zone);
- available += zone_page_state_snapshot(zone, NR_FREE_PAGES);
+ available += free;
/*
* Would the allocation succeed if we reclaimed all
On 2020/04/25 5:48, David Rientjes wrote:
> @@ -4372,11 +4372,21 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
> ac->nodemask) {
> unsigned long available;
> unsigned long reclaimable;
> + unsigned long free;
> unsigned long min_wmark = min_wmark_pages(zone);
> bool wmark;
>
> + free = zone_page_state_snapshot(zone, NR_FREE_PAGES);
> + /*
> + * If this zone is approaching the point where even order-0
> + * GFP_ATOMIC allocations will fail, stop considering reclaim.
> + */
> + if (!__zone_watermark_ok(zone, 0, min_wmark, ac_classzone_idx(ac),
> + alloc_flags | ALLOC_HIGH, free))
> + continue;
This is to trigger the OOM killer more aggressively, isn't it? But where is the
guarantee that this is an allocation request which can trigger the OOM killer?
If this is an allocation request which cannot trigger the OOM killer, wouldn't
this cause premature allocation failures?
> +
> available = reclaimable = zone_reclaimable_pages(zone);
> - available += zone_page_state_snapshot(zone, NR_FREE_PAGES);
> + available += free;
>
> /*
> * Would the allocation succeed if we reclaimed all
>
On Fri, 24 Apr 2020 13:48:06 -0700 (PDT) David Rientjes <[email protected]> wrote:
> If GFP_ATOMIC allocations will start failing soon because the amount of
> free memory is substantially under per-zone min watermarks, it is better
> to oom kill a process rather than continue to reclaim.
>
> This intends to significantly reduce the number of page allocation
> failures that are encountered when the demands of user and atomic
> allocations overwhelm the ability of reclaim to keep up. We can see this
> with a high ingress of networking traffic where memory allocated in irq
> context can overwhelm the ability to reclaim fast enough such that user
> memory consistently loops. In that case, we have reclaimable memory, and
"user memory allocation", I assume? Or maybe "blockable memory
allocatoins".
> reclaiming is successful, but we've fully depleted memory reserves that
> are allowed for non-blockable allocations.
>
> Commit 400e22499dd9 ("mm: don't warn about allocations which stall for
> too long") removed evidence of user allocations stalling because of this,
> but the situation can apply anytime we get "page allocation failures"
> where reclaim is happening but per-zone min watermarks are starved:
>
> Node 0 Normal free:87356kB min:221984kB low:416984kB high:611984kB active_anon:123009936kB inactive_anon:67647652kB active_file:429612kB inactive_file:209980kB unevictable:112348kB writepending:260kB present:198180864kB managed:195027624kB mlocked:81756kB kernel_stack:24040kB pagetables:11460kB bounce:0kB free_pcp:940kB local_pcp:96kB free_cma:0kB
> lowmem_reserve[]: 0 0 0 0
> Node 1 Normal free:105616kB min:225568kB low:423716kB high:621864kB active_anon:122124196kB inactive_anon:74112696kB active_file:39172kB inactive_file:103696kB unevictable:204480kB writepending:180kB present:201326592kB managed:198174372kB mlocked:204480kB kernel_stack:11328kB pagetables:3680kB bounce:0kB free_pcp:1140kB local_pcp:0kB free_cma:0kB
> lowmem_reserve[]: 0 0 0 0
>
> Without this patch, there is no guarantee that user memory allocations
> will ever be successful when non-blockable allocations overwhelm the
> ability to get above per-zone min watermarks.
>
> This doesn't solve page allocation failures entirely since it's a
> preemptive measure based on watermarks that requires concurrent blockable
> allocations to trigger the oom kill. To complete solve page allocation
> failures, it would be possible to do the same watermark check for non-
> blockable allocations and then queue a worker to asynchronously oom kill
> if it finds watermarks to be sufficiently low as well.
>
Well, what's really going on here?
Is networking potentially consuming an unbounded amount of memory? If
so, then killing a process will just cause networking to consume more
memory then hit against the same thing. So presumably the answer is
"no, the watermarks are inappropriately set for this workload".
So would it not be sensible to dynamically adjust the watermarks in
response to this condition? Maintain a larger pool of memory for these
allocations? Or possibly push back on networking and tell it to reduce
its queue sizes? So that stuff doesn't keep on getting oom-killed?
On 2020/04/26 9:27, Andrew Morton wrote:
> Well, what's really going on here?
>
> Is networking potentially consuming an unbounded amount of memory? If
> so, then killing a process will just cause networking to consume more
> memory then hit against the same thing. So presumably the answer is
> "no, the watermarks are inappropriately set for this workload".
Maybe somebody is abusing __GFP_MEMALLOC allocations. David, please include whole
context (e.g. which function from which process) instead of only memory stat lines.
On Sat, 25 Apr 2020, Andrew Morton wrote:
> > If GFP_ATOMIC allocations will start failing soon because the amount of
> > free memory is substantially under per-zone min watermarks, it is better
> > to oom kill a process rather than continue to reclaim.
> >
> > This intends to significantly reduce the number of page allocation
> > failures that are encountered when the demands of user and atomic
> > allocations overwhelm the ability of reclaim to keep up. We can see this
> > with a high ingress of networking traffic where memory allocated in irq
> > context can overwhelm the ability to reclaim fast enough such that user
> > memory consistently loops. In that case, we have reclaimable memory, and
>
> "user memory allocation", I assume? Or maybe "blockable memory
> allocatoins".
>
"user memory allocations consistently loop", yeah. Thanks.
> > reclaiming is successful, but we've fully depleted memory reserves that
> > are allowed for non-blockable allocations.
> >
> > Commit 400e22499dd9 ("mm: don't warn about allocations which stall for
> > too long") removed evidence of user allocations stalling because of this,
> > but the situation can apply anytime we get "page allocation failures"
> > where reclaim is happening but per-zone min watermarks are starved:
> >
> > Node 0 Normal free:87356kB min:221984kB low:416984kB high:611984kB active_anon:123009936kB inactive_anon:67647652kB active_file:429612kB inactive_file:209980kB unevictable:112348kB writepending:260kB present:198180864kB managed:195027624kB mlocked:81756kB kernel_stack:24040kB pagetables:11460kB bounce:0kB free_pcp:940kB local_pcp:96kB free_cma:0kB
> > lowmem_reserve[]: 0 0 0 0
> > Node 1 Normal free:105616kB min:225568kB low:423716kB high:621864kB active_anon:122124196kB inactive_anon:74112696kB active_file:39172kB inactive_file:103696kB unevictable:204480kB writepending:180kB present:201326592kB managed:198174372kB mlocked:204480kB kernel_stack:11328kB pagetables:3680kB bounce:0kB free_pcp:1140kB local_pcp:0kB free_cma:0kB
> > lowmem_reserve[]: 0 0 0 0
> >
> > Without this patch, there is no guarantee that user memory allocations
> > will ever be successful when non-blockable allocations overwhelm the
> > ability to get above per-zone min watermarks.
> >
> > This doesn't solve page allocation failures entirely since it's a
> > preemptive measure based on watermarks that requires concurrent blockable
> > allocations to trigger the oom kill. To complete solve page allocation
> > failures, it would be possible to do the same watermark check for non-
> > blockable allocations and then queue a worker to asynchronously oom kill
> > if it finds watermarks to be sufficiently low as well.
> >
>
> Well, what's really going on here?
>
> Is networking potentially consuming an unbounded amount of memory? If
> so, then killing a process will just cause networking to consume more
> memory then hit against the same thing. So presumably the answer is
> "no, the watermarks are inappropriately set for this workload".
>
> So would it not be sensible to dynamically adjust the watermarks in
> response to this condition? Maintain a larger pool of memory for these
> allocations? Or possibly push back on networking and tell it to reduce
> its queue sizes? So that stuff doesn't keep on getting oom-killed?
>
No - that would actually make the problem worse.
Today, per-zone min watermarks dictate when user allocations will loop or
oom kill. should_reclaim_retry() currently loops if reclaim has succeeded
in the past few tries and we should be able to allocate if we are able to
reclaim the amount of memory that we think we can.
The issue is that this supposes that looping to reclaim more will result
in more free memory. That doesn't always happen if there are concurrent
memory allocators.
GFP_ATOMIC allocators can access below these per-zone watermarks. So the
issue is that per-zone free pages stays between ALLOC_HIGH watermarks
(the watermark that GFP_ATOMIC allocators can allocate to) and min
watermarks. We never reclaim enough memory to get back to min watermarks
because reclaim cannot keep up with the amount of GFP_ATOMIC allocations.
In production scnearios, this results in userspace processes going out to
lunch because they are constantly looping in the page allocator reclaiming
only for the benefit of GFP_ATOMIC allocations.
In fact, when we hit ALLOC_HIGH watermarks and we start getting "page
allocation failures" in the kernel log, there is also no guarantee that
kswapd's reclaim will outpace GFP_ATOMIC allocations. Thus, an oom kill
is really the best policy at this point to provide an actual guarantee of
net positive memory freeing.
This isn't a matter of any specific networking stack; the scope of
allocations that can trigger this is the set of all GFP_ATOMIC (or
GFP_MEMALLOC) allocations in the kernel.
Tetsuo: the specific allocation that triggers a page allocation failure is
not interesting; we have tens of thousands of examples. Each example is
simply the unlucky last GFP_ATOMIC allocation that fails; the interesting
point is the amount of free memory. In other words, when free memory is
below ALLOC_HIGH watermarks, we assume that we have depleted memory
reserves *faster* than when user allocations started to fail. In the
interest of userspace being responsive, we should oom kill here.
On 2020/04/27 12:12, David Rientjes wrote:
> Tetsuo: the specific allocation that triggers a page allocation failure is
> not interesting; we have tens of thousands of examples. Each example is
> simply the unlucky last GFP_ATOMIC allocation that fails; the interesting
> point is the amount of free memory. In other words, when free memory is
> below ALLOC_HIGH watermarks, we assume that we have depleted memory
> reserves *faster* than when user allocations started to fail. In the
> interest of userspace being responsive, we should oom kill here.
My interest is, which function (and which process if process context) is [ab]using
GFP_ATOMIC (or __GFP_MEMALLOC) allocations enough to hit memory allocation failure.
GFP_NOWAIT (or __GFP_NOMEMALLOC) could be used if that allocation can't sleep and
can't shortly recover free memory.
On 4/26/20 2:27 AM, Andrew Morton wrote:
> On Fri, 24 Apr 2020 13:48:06 -0700 (PDT) David Rientjes <[email protected]> wrote:
>
>> If GFP_ATOMIC allocations will start failing soon because the amount of
>> free memory is substantially under per-zone min watermarks, it is better
>> to oom kill a process rather than continue to reclaim.
>>
>> This intends to significantly reduce the number of page allocation
>> failures that are encountered when the demands of user and atomic
>> allocations overwhelm the ability of reclaim to keep up. We can see this
>> with a high ingress of networking traffic where memory allocated in irq
>> context can overwhelm the ability to reclaim fast enough such that user
>> memory consistently loops. In that case, we have reclaimable memory, and
> "user memory allocation", I assume? Or maybe "blockable memory
> allocatoins".
>
>> reclaiming is successful, but we've fully depleted memory reserves that
>> are allowed for non-blockable allocations.
>>
>> Commit 400e22499dd9 ("mm: don't warn about allocations which stall for
>> too long") removed evidence of user allocations stalling because of this,
>> but the situation can apply anytime we get "page allocation failures"
>> where reclaim is happening but per-zone min watermarks are starved:
>>
>> Node 0 Normal free:87356kB min:221984kB low:416984kB high:611984kB active_anon:123009936kB inactive_anon:67647652kB active_file:429612kB inactive_file:209980kB unevictable:112348kB writepending:260kB present:198180864kB managed:195027624kB mlocked:81756kB kernel_stack:24040kB pagetables:11460kB bounce:0kB free_pcp:940kB local_pcp:96kB free_cma:0kB
>> lowmem_reserve[]: 0 0 0 0
>> Node 1 Normal free:105616kB min:225568kB low:423716kB high:621864kB active_anon:122124196kB inactive_anon:74112696kB active_file:39172kB inactive_file:103696kB unevictable:204480kB writepending:180kB present:201326592kB managed:198174372kB mlocked:204480kB kernel_stack:11328kB pagetables:3680kB bounce:0kB free_pcp:1140kB local_pcp:0kB free_cma:0kB
>> lowmem_reserve[]: 0 0 0 0
>>
>> Without this patch, there is no guarantee that user memory allocations
>> will ever be successful when non-blockable allocations overwhelm the
>> ability to get above per-zone min watermarks.
>>
>> This doesn't solve page allocation failures entirely since it's a
>> preemptive measure based on watermarks that requires concurrent blockable
>> allocations to trigger the oom kill. To complete solve page allocation
>> failures, it would be possible to do the same watermark check for non-
>> blockable allocations and then queue a worker to asynchronously oom kill
>> if it finds watermarks to be sufficiently low as well.
>>
> Well, what's really going on here?
>
> Is networking potentially consuming an unbounded amount of memory? If
> so, then killing a process will just cause networking to consume more
> memory then hit against the same thing. So presumably the answer is
> "no, the watermarks are inappropriately set for this workload".
>
> So would it not be sensible to dynamically adjust the watermarks in
> response to this condition? Maintain a larger pool of memory for these
> allocations? Or possibly push back on networking and tell it to reduce
> its queue sizes? So that stuff doesn't keep on getting oom-killed?
>
I think I seen similar issues when dma-buf allocate a lot. But that is on older kernels and out of tree.
So networking is maybe not the only cause. dma-buf are used a lot for camera stuff in android.
On Fri 24-04-20 13:48:06, David Rientjes wrote:
> If GFP_ATOMIC allocations will start failing soon because the amount of
> free memory is substantially under per-zone min watermarks, it is better
> to oom kill a process rather than continue to reclaim.
>
> This intends to significantly reduce the number of page allocation
> failures that are encountered when the demands of user and atomic
> allocations overwhelm the ability of reclaim to keep up. We can see this
> with a high ingress of networking traffic where memory allocated in irq
> context can overwhelm the ability to reclaim fast enough such that user
> memory consistently loops. In that case, we have reclaimable memory, and
> reclaiming is successful, but we've fully depleted memory reserves that
> are allowed for non-blockable allocations.
>
> Commit 400e22499dd9 ("mm: don't warn about allocations which stall for
> too long") removed evidence of user allocations stalling because of this,
> but the situation can apply anytime we get "page allocation failures"
> where reclaim is happening but per-zone min watermarks are starved:
>
> Node 0 Normal free:87356kB min:221984kB low:416984kB high:611984kB active_anon:123009936kB inactive_anon:67647652kB active_file:429612kB inactive_file:209980kB unevictable:112348kB writepending:260kB present:198180864kB managed:195027624kB mlocked:81756kB kernel_stack:24040kB pagetables:11460kB bounce:0kB free_pcp:940kB local_pcp:96kB free_cma:0kB
> lowmem_reserve[]: 0 0 0 0
> Node 1 Normal free:105616kB min:225568kB low:423716kB high:621864kB active_anon:122124196kB inactive_anon:74112696kB active_file:39172kB inactive_file:103696kB unevictable:204480kB writepending:180kB present:201326592kB managed:198174372kB mlocked:204480kB kernel_stack:11328kB pagetables:3680kB bounce:0kB free_pcp:1140kB local_pcp:0kB free_cma:0kB
> lowmem_reserve[]: 0 0 0 0
>
> Without this patch, there is no guarantee that user memory allocations
> will ever be successful when non-blockable allocations overwhelm the
> ability to get above per-zone min watermarks.
We have never had any guarantee and we will not have any after this
patch either. The fundamental problem is that direct reclaim doesn't
provide any guarantee that the reclaimed memory is going to be used for
the reclaimer. You can see the same if the memory demand is higher
than the reclaim. GFP_ATOMIC is only different in that aspect that they
are not throttled by reclaiming and consume what is availble right away
which makes the problem worse.
> This doesn't solve page allocation failures entirely since it's a
> preemptive measure based on watermarks that requires concurrent blockable
> allocations to trigger the oom kill. To complete solve page allocation
> failures, it would be possible to do the same watermark check for non-
> blockable allocations and then queue a worker to asynchronously oom kill
> if it finds watermarks to be sufficiently low as well.
I do not think this is the right approach. This patch is also quite
dangerous as pointed out by Tetsuo. You are effectively allowing a
remote DoS via OOM killer. If the reclaim is making progress then the
issue seem to be more on the configuration side I believe. Have you
tried to tune watermarks resp. watermark_scale_factor? Another potential
problem might be that the kswapd is not making sufficient progress
because it is blocked on something.
> Signed-off-by: David Rientjes <[email protected]>
> ---
> mm/page_alloc.c | 12 +++++++++++-
> 1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -4372,11 +4372,21 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
> ac->nodemask) {
> unsigned long available;
> unsigned long reclaimable;
> + unsigned long free;
> unsigned long min_wmark = min_wmark_pages(zone);
> bool wmark;
>
> + free = zone_page_state_snapshot(zone, NR_FREE_PAGES);
> + /*
> + * If this zone is approaching the point where even order-0
> + * GFP_ATOMIC allocations will fail, stop considering reclaim.
> + */
> + if (!__zone_watermark_ok(zone, 0, min_wmark, ac_classzone_idx(ac),
> + alloc_flags | ALLOC_HIGH, free))
> + continue;
> +
> available = reclaimable = zone_reclaimable_pages(zone);
> - available += zone_page_state_snapshot(zone, NR_FREE_PAGES);
> + available += free;
>
> /*
> * Would the allocation succeed if we reclaimed all
--
Michal Hocko
SUSE Labs
On Sun, 26 Apr 2020 20:12:58 -0700 (PDT) David Rientjes <[email protected]> wrote:
> > > blockable allocations and then queue a worker to asynchronously oom kill
> > > if it finds watermarks to be sufficiently low as well.
> > >
> >
> > Well, what's really going on here?
> >
> > Is networking potentially consuming an unbounded amount of memory? If
> > so, then killing a process will just cause networking to consume more
> > memory then hit against the same thing. So presumably the answer is
> > "no, the watermarks are inappropriately set for this workload".
> >
> > So would it not be sensible to dynamically adjust the watermarks in
> > response to this condition? Maintain a larger pool of memory for these
> > allocations? Or possibly push back on networking and tell it to reduce
> > its queue sizes? So that stuff doesn't keep on getting oom-killed?
> >
>
> No - that would actually make the problem worse.
>
> Today, per-zone min watermarks dictate when user allocations will loop or
> oom kill. should_reclaim_retry() currently loops if reclaim has succeeded
> in the past few tries and we should be able to allocate if we are able to
> reclaim the amount of memory that we think we can.
>
> The issue is that this supposes that looping to reclaim more will result
> in more free memory. That doesn't always happen if there are concurrent
> memory allocators.
>
> GFP_ATOMIC allocators can access below these per-zone watermarks. So the
> issue is that per-zone free pages stays between ALLOC_HIGH watermarks
> (the watermark that GFP_ATOMIC allocators can allocate to) and min
> watermarks. We never reclaim enough memory to get back to min watermarks
> because reclaim cannot keep up with the amount of GFP_ATOMIC allocations.
But there should be an upper bound upon the total amount of in-flight
GFP_ATOMIC memory at any point in time? These aren't like pagecache
which will take more if we give it more. Setting the various
thresholds appropriately should ensure that blockable allocations don't
get their memory stolen by GPP_ATOMIC allocations?
I took a look at doing a quick-fix for the
direct-reclaimers-get-their-stuff-stolen issue about a million years
ago. I don't recall where it ended up. It's pretty trivial for the
direct reclaimer to free pages into current->reclaimed_pages and to
take a look in there on the allocation path, etc. But it's only
practical for order-0 pages.
On Mon, 27 Apr 2020, Andrew Morton wrote:
> > No - that would actually make the problem worse.
> >
> > Today, per-zone min watermarks dictate when user allocations will loop or
> > oom kill. should_reclaim_retry() currently loops if reclaim has succeeded
> > in the past few tries and we should be able to allocate if we are able to
> > reclaim the amount of memory that we think we can.
> >
> > The issue is that this supposes that looping to reclaim more will result
> > in more free memory. That doesn't always happen if there are concurrent
> > memory allocators.
> >
> > GFP_ATOMIC allocators can access below these per-zone watermarks. So the
> > issue is that per-zone free pages stays between ALLOC_HIGH watermarks
> > (the watermark that GFP_ATOMIC allocators can allocate to) and min
> > watermarks. We never reclaim enough memory to get back to min watermarks
> > because reclaim cannot keep up with the amount of GFP_ATOMIC allocations.
>
> But there should be an upper bound upon the total amount of in-flight
> GFP_ATOMIC memory at any point in time? These aren't like pagecache
> which will take more if we give it more. Setting the various
> thresholds appropriately should ensure that blockable allocations don't
> get their memory stolen by GPP_ATOMIC allocations?
>
Certainly if that upper bound is defined and enforced somewhere we would
not have run into this issue causing all userspace to become completely
unresponsive. Do you have links to patches that proposed enforcing this
upper bound? It seems like it would have to be generic to
__alloc_pages_slowpath() because otherwise multiple different GFP_ATOMIC
allocators, all from different sources, couldn't orchestrate their memory
allocations amongst themselves to enforce this upper bound. They would
need to work together to ensure they don't conspire to cause this
depletion. I'd be happy to take a look if there are links to other
approaches.
On Mon, 27 Apr 2020 16:03:56 -0700 (PDT) David Rientjes <[email protected]> wrote:
> On Mon, 27 Apr 2020, Andrew Morton wrote:
>
> > > No - that would actually make the problem worse.
> > >
> > > Today, per-zone min watermarks dictate when user allocations will loop or
> > > oom kill. should_reclaim_retry() currently loops if reclaim has succeeded
> > > in the past few tries and we should be able to allocate if we are able to
> > > reclaim the amount of memory that we think we can.
> > >
> > > The issue is that this supposes that looping to reclaim more will result
> > > in more free memory. That doesn't always happen if there are concurrent
> > > memory allocators.
> > >
> > > GFP_ATOMIC allocators can access below these per-zone watermarks. So the
> > > issue is that per-zone free pages stays between ALLOC_HIGH watermarks
> > > (the watermark that GFP_ATOMIC allocators can allocate to) and min
> > > watermarks. We never reclaim enough memory to get back to min watermarks
> > > because reclaim cannot keep up with the amount of GFP_ATOMIC allocations.
> >
> > But there should be an upper bound upon the total amount of in-flight
> > GFP_ATOMIC memory at any point in time? These aren't like pagecache
> > which will take more if we give it more. Setting the various
> > thresholds appropriately should ensure that blockable allocations don't
> > get their memory stolen by GPP_ATOMIC allocations?
> >
>
> Certainly if that upper bound is defined and enforced somewhere we would
> not have run into this issue causing all userspace to become completely
> unresponsive. Do you have links to patches that proposed enforcing this
> upper bound?
There is no such enforcement and there are no such patches, as I'm sure
you know.
No consumer of GFP_ATOMIC memory should consume an unbounded amount of
it. Subsystems such as networking will consume a certain amount and
will then start recycling it. The total amount in-flight will vary
over the longer term as workloads change. A dynamically tuning
threshold system will need to adapt rapidly enough to sudden load
shifts, which might require unreasonable amounts of headroom.
Michal asked relevant questions regarding watermark tuning - an ansewr
to those would be interesting. To amplify that, is it possible to
manually tune this system so that the problem no longer exhibits? If
so, then why can't that tuning be performed automatically?
On Mon 27-04-20 16:35:58, Andrew Morton wrote:
[...]
> No consumer of GFP_ATOMIC memory should consume an unbounded amount of
> it.
> Subsystems such as networking will consume a certain amount and
> will then start recycling it. The total amount in-flight will vary
> over the longer term as workloads change. A dynamically tuning
> threshold system will need to adapt rapidly enough to sudden load
> shifts, which might require unreasonable amounts of headroom.
I do agree. __GFP_HIGH/__GFP_ATOMIC are bound by the size of the
reserves under memory pressure. Then allocatios start failing very
quickly and users have to cope with that, usually by deferring to a
sleepable context. Tuning reserves dynamically for heavy reserves
consumers would be possible but I am worried that this is far from
trivial.
We definitely need to understand what is going on here. Why doesn't
kswapd + N*direct reclaimers do not provide enough memory to satisfy
both N threads + reserves consumers? How many times those direct
reclaimers have to retry?
We used to have the allocation stall warning as David mentioned in the
patch description and I have seen it triggering without heavy reserves
consumers (aka reported free pages corresponded to the min watermark).
The underlying problem was usually kswapd being stuck on some FS locks,
direct reclaimers stuck in shrinkers or way too overloaded system with
dozens if not hundreds of processes stuck in the page allocator each
racing with the reclaim and betting on luck. The last problem was the
most annoying because it is really hard to tune for.
--
Michal Hocko
SUSE Labs
On 4/27/20 10:30 PM, Andrew Morton wrote:
> On Sun, 26 Apr 2020 20:12:58 -0700 (PDT) David Rientjes <[email protected]> wrote:
>
>>
>> GFP_ATOMIC allocators can access below these per-zone watermarks. So the
>> issue is that per-zone free pages stays between ALLOC_HIGH watermarks
>> (the watermark that GFP_ATOMIC allocators can allocate to) and min
>> watermarks. We never reclaim enough memory to get back to min watermarks
>> because reclaim cannot keep up with the amount of GFP_ATOMIC allocations.
>
> But there should be an upper bound upon the total amount of in-flight
> GFP_ATOMIC memory at any point in time? These aren't like pagecache
If it's a network receive path, then this is effectively bounded by link speed
vs ability to deal with the packets quickly and free the buffers. And the bursts
of incoming packets might be out of control of the admin. With my "enterprise
kernel support" hat on, it's it's annoying enough to explain GFP_ATOMIC failures
(usually high-order) in dmesg every once in a while (the usual suggestion is to
bump min_free_kbytes and stress that unless they are frequent, there's no actual
harm as networking can defer the allocation to non-atomic context). If there was
an OOM kill as a result, that could not be disabled, I can well imagine we would
have to revert such patch in our kernel as a result due to the DOS (intentional
or not) potential.
> which will take more if we give it more. Setting the various
> thresholds appropriately should ensure that blockable allocations don't
> get their memory stolen by GPP_ATOMIC allocations?
I agree with the view that GFP_ATOMIC is only a (perhaps more visible) part of
the problem that there's no fairness guarantee in reclaim, and allocators can
steal from each other. GFP_ATOMIC allocations just have it easier thanks to
lower thresholds.
> I took a look at doing a quick-fix for the
> direct-reclaimers-get-their-stuff-stolen issue about a million years
> ago. I don't recall where it ended up. It's pretty trivial for the
> direct reclaimer to free pages into current->reclaimed_pages and to
> take a look in there on the allocation path, etc. But it's only
> practical for order-0 pages.
FWIW there's already such approach added to compaction by Mel some time ago, so
order>0 allocations are covered to some extent. But in this case I imagine that
compaction won't even start because order-0 watermarks are too low.
The order-0 reclaim capture might work though - as a result the GFP_ATOMIC
allocations would more likely fail and defer to their fallback context.
On Tue, 28 Apr 2020, Vlastimil Babka wrote:
> > I took a look at doing a quick-fix for the
> > direct-reclaimers-get-their-stuff-stolen issue about a million years
> > ago. I don't recall where it ended up. It's pretty trivial for the
> > direct reclaimer to free pages into current->reclaimed_pages and to
> > take a look in there on the allocation path, etc. But it's only
> > practical for order-0 pages.
>
> FWIW there's already such approach added to compaction by Mel some time ago,
> so order>0 allocations are covered to some extent. But in this case I imagine
> that compaction won't even start because order-0 watermarks are too low.
>
> The order-0 reclaim capture might work though - as a result the GFP_ATOMIC
> allocations would more likely fail and defer to their fallback context.
>
Yes, order-0 reclaim capture is interesting since the issue being reported
here is userspace going out to lunch because it loops for an unbounded
amount of time trying to get above a watermark where it's allowed to
allocate and other consumers are depleting that resource.
We actually prefer to oom kill earlier rather than being put in a
perpetual state of aggressive reclaim that affects all allocators and the
unbounded nature of those allocations leads to very poor results for
everybody.
I'm happy to scope this solely to an order-0 reclaim capture. I'm not
sure if I'm clear on whether this has been worked on before and patches
existed in the past?
Somewhat related to what I described in the changelog: we lost the "page
allocation stalls" artifacts in the kernel log for 4.15. The commit
description references an asynchronous mechanism for getting this
information; I don't know where this mechanism currently lives.
On 2020/04/29 6:48, David Rientjes wrote:
> Somewhat related to what I described in the changelog: we lost the "page
> allocation stalls" artifacts in the kernel log for 4.15. The commit
> description references an asynchronous mechanism for getting this
> information; I don't know where this mechanism currently lives.
The asynchronous mechanism I meant in that commit is
https://lkml.kernel.org/r/[email protected] .
We are still forced to live in a world where we cannot get a clue for stalled allocations. :-(
On 4/28/20 11:48 PM, David Rientjes wrote:
> On Tue, 28 Apr 2020, Vlastimil Babka wrote:
>
> Yes, order-0 reclaim capture is interesting since the issue being reported
> here is userspace going out to lunch because it loops for an unbounded
> amount of time trying to get above a watermark where it's allowed to
> allocate and other consumers are depleting that resource.
>
> We actually prefer to oom kill earlier rather than being put in a
> perpetual state of aggressive reclaim that affects all allocators and the
> unbounded nature of those allocations leads to very poor results for
> everybody.
Sure. My vague impression is that your (and similar cloud companies) kind of
workloads are designed to maximize machine utilization, and overshooting and
killing something as a result is no big deal. Then you perhaps have more
probability of hitting this state, and on the other hand, even an occasional
premature oom kill is not a big deal?
My concers are workloads not designed in such a way, where premature oom kill
due to temporary higher reclaim activity together with burst of incoming network
packets will result in e.g. killing an important database. There, the tradeoff
looks different.
> I'm happy to scope this solely to an order-0 reclaim capture. I'm not
> sure if I'm clear on whether this has been worked on before and patches
> existed in the past?
Andrew mentioned some. I don't recall any, so it might have been before my time.
> Somewhat related to what I described in the changelog: we lost the "page
> allocation stalls" artifacts in the kernel log for 4.15. The commit
> description references an asynchronous mechanism for getting this
> information; I don't know where this mechanism currently lives.
>
On 4/28/20 9:43 AM, Michal Hocko wrote:
> On Mon 27-04-20 16:35:58, Andrew Morton wrote:
> [...]
>> No consumer of GFP_ATOMIC memory should consume an unbounded amount of
>> it.
>> Subsystems such as networking will consume a certain amount and
>> will then start recycling it. The total amount in-flight will vary
>> over the longer term as workloads change. A dynamically tuning
>> threshold system will need to adapt rapidly enough to sudden load
>> shifts, which might require unreasonable amounts of headroom.
> I do agree. __GFP_HIGH/__GFP_ATOMIC are bound by the size of the
> reserves under memory pressure. Then allocatios start failing very
> quickly and users have to cope with that, usually by deferring to a
> sleepable context. Tuning reserves dynamically for heavy reserves
> consumers would be possible but I am worried that this is far from
> trivial.
>
> We definitely need to understand what is going on here. Why doesn't
> kswapd + N*direct reclaimers do not provide enough memory to satisfy
> both N threads + reserves consumers? How many times those direct
> reclaimers have to retry?
Was this not supposed to be avoided with PSI, user-space should
a fair change to take actions before it goes bad in user-space?
> We used to have the allocation stall warning as David mentioned in the
> patch description and I have seen it triggering without heavy reserves
> consumers (aka reported free pages corresponded to the min watermark).
> The underlying problem was usually kswapd being stuck on some FS locks,
> direct reclaimers stuck in shrinkers or way too overloaded system with
> dozens if not hundreds of processes stuck in the page allocator each
> racing with the reclaim and betting on luck. The last problem was the
> most annoying because it is really hard to tune for.
On Wed 29-04-20 10:31:41, peter enderborg wrote:
> On 4/28/20 9:43 AM, Michal Hocko wrote:
> > On Mon 27-04-20 16:35:58, Andrew Morton wrote:
> > [...]
> >> No consumer of GFP_ATOMIC memory should consume an unbounded amount of
> >> it.
> >> Subsystems such as networking will consume a certain amount and
> >> will then start recycling it. The total amount in-flight will vary
> >> over the longer term as workloads change. A dynamically tuning
> >> threshold system will need to adapt rapidly enough to sudden load
> >> shifts, which might require unreasonable amounts of headroom.
> > I do agree. __GFP_HIGH/__GFP_ATOMIC are bound by the size of the
> > reserves under memory pressure. Then allocatios start failing very
> > quickly and users have to cope with that, usually by deferring to a
> > sleepable context. Tuning reserves dynamically for heavy reserves
> > consumers would be possible but I am worried that this is far from
> > trivial.
> >
> > We definitely need to understand what is going on here. Why doesn't
> > kswapd + N*direct reclaimers do not provide enough memory to satisfy
> > both N threads + reserves consumers? How many times those direct
> > reclaimers have to retry?
>
> Was this not supposed to be avoided with PSI, user-space should
> a fair change to take actions before it goes bad in user-space?
Yes, PSI is certainly a tool to help userspace make actions on heavy
reclaim. And I agree that if there is a desire to trigger the oom killer
early as David states elsewhere in the thread then this approach should
be considered.
--
Michal Hocko
SUSE Labs
On Wed 29-04-20 09:51:39, Vlastimil Babka wrote:
> On 4/28/20 11:48 PM, David Rientjes wrote:
> > On Tue, 28 Apr 2020, Vlastimil Babka wrote:
> >
> > Yes, order-0 reclaim capture is interesting since the issue being reported
> > here is userspace going out to lunch because it loops for an unbounded
> > amount of time trying to get above a watermark where it's allowed to
> > allocate and other consumers are depleting that resource.
> >
> > We actually prefer to oom kill earlier rather than being put in a
> > perpetual state of aggressive reclaim that affects all allocators and the
> > unbounded nature of those allocations leads to very poor results for
> > everybody.
>
> Sure. My vague impression is that your (and similar cloud companies) kind of
> workloads are designed to maximize machine utilization, and overshooting and
> killing something as a result is no big deal. Then you perhaps have more
> probability of hitting this state, and on the other hand, even an occasional
> premature oom kill is not a big deal?
>
> My concers are workloads not designed in such a way, where premature oom kill
> due to temporary higher reclaim activity together with burst of incoming network
> packets will result in e.g. killing an important database. There, the tradeoff
> looks different.
Completely agreed! The in kernel OOM killer is to deal with situations
when memory is desperately depleted without any sign of a forward
progress. If there is a reclaimable memory then we are not there yet.
If a workload can benefit from early oom killing based on response time
then we have facilities to achieve that (e.g. PSI).
--
Michal Hocko
SUSE Labs
On 2020/04/29 18:04, Michal Hocko wrote:
> Completely agreed! The in kernel OOM killer is to deal with situations
> when memory is desperately depleted without any sign of a forward
> progress. If there is a reclaimable memory then we are not there yet.
> If a workload can benefit from early oom killing based on response time
> then we have facilities to achieve that (e.g. PSI).
Can PSI work even if userspace process cannot avoid reclaimable memory
allocations (e.g. page fault, file read) is already stalling? I'm not sure
whether PSI allows responding quickly enough to "keep reclaimable memory
allocations not to reclaim" despite there is still reclaimable memory...
On Wed 29-04-20 19:45:07, Tetsuo Handa wrote:
> On 2020/04/29 18:04, Michal Hocko wrote:
> > Completely agreed! The in kernel OOM killer is to deal with situations
> > when memory is desperately depleted without any sign of a forward
> > progress. If there is a reclaimable memory then we are not there yet.
> > If a workload can benefit from early oom killing based on response time
> > then we have facilities to achieve that (e.g. PSI).
>
> Can PSI work even if userspace process cannot avoid reclaimable memory
> allocations (e.g. page fault, file read) is already stalling?
The userspace itself would have to be careful and use mlock of course.
But collecting the psi information itself should be pretty independent
on memory allocations as monitoring the system memory state is one of
the main usecases.
> I'm not sure
> whether PSI allows responding quickly enough to "keep reclaimable memory
> allocations not to reclaim" despite there is still reclaimable memory...
PSI is supposed to monitor time spent in the memory allocator (among
other things) and report the tendency. This should be a sufficient
metric to judge that a large part of the userspace is not making forward
progress.
--
Michal Hocko
SUSE Labs