by Michal Hocko

[permalink] [raw]

Subject: Re: [PATCH] mm, oom: report compaction/migration stats for higher order requests

On Sun 21-08-16 23:19:50, Arkadiusz Miskiewicz wrote:
> On Friday 19 of August 2016, Vlastimil Babka wrote:
> > On 08/18/2016 08:49 PM, Arkadiusz Miskiewicz wrote:
> > > On Wednesday 17 of August 2016, Michal Hocko wrote:
> > >> On Wed 17-08-16 10:34:54, Arkadiusz Miśkiewicz wrote:
> > >> [...]
> > >>
> > >>> With "[PATCH] mm, oom: report compaction/migration stats for higher
> > >>> order requests" patch:
> > >>> https://ixion.pld-linux.org/~arekm/p2/ext4/log-20160817.txt
> > >>>
> > >>> Didn't count much - all counters are 0
> > >>> compaction_stall:0 compaction_fail:0 compact_migrate_scanned:0
> > >>> compact_free_scanned:0 compact_isolated:0 pgmigrate_success:0
> > >>> pgmigrate_fail:0
> > >>
> > >> Dohh, COMPACTION counters are events and those are different than other
> > >> counters we have. They only have per-cpu representation and so we would
> > >> have to do
> > >> + for_each_online_cpu(cpu) {
> > >> + struct vm_event_state *this = &per_cpu(vm_event_states,
> > >> cpu); + ret += this->event[item];
> > >> + }
> > >>
> > >> which is really nasty because, strictly speaking, we would have to do
> > >> {get,put}_online_cpus around that loop and that uses locking and we do
> > >> not want to possibly block in this path just because something is in the
> > >> middle of the hotplug. So let's scratch that patch for now and sorry I
> > >> haven't realized that earlier.
> > >>
> > >>> two processes were killed by OOM (rm and cp), the rest of rm/cp didn't
> > >>> finish
> > >>>
> > >>> and I'm interrupting it to try that next patch:
> > >>>> Could you try to test with
> > >>>> patch from
> > >>>> http://lkml.kernel.org/r/20160816031222.GC16913@js1304-P5Q-DELUXE
> > >>>> please? Ideally on top of linux-next. You can add both the compaction
> > >>>> counters patch in the oom report and high order atomic reserves patch
> > >>>> on top.
> > >>>
> > >>> Uhm, was going to use it on top of 4.7.[01] first.
> > >>
> > >> OK
> > >
> > > So with
> > > http://lkml.kernel.org/r/20160816031222.GC16913@js1304-P5Q-DELUXE OOM no
> > > longer happens (all 10x rm/cp processes finished).
> >
> > Is it on top of 4.7 then?
>
> Yes, it was on top of 4.7.0.
>
> > That's a bit different from the other reporter
> > who needed both linux-next and this patch to avoid OOM.
> > In any case the proper solution should restrict this disabled heuristic
> > to highest compaction priority, which needs the patches from linux-next
> > anyway.
> >
> > So can you please also try linux-next with the patch from
> > http://marc.info/?l=linux-mm&m=147158805719821 ?
>
> https://ixion.pld-linux.org/~arekm/p2/ext4/log-20160819.txt
> https://ixion.pld-linux.org/~arekm/p2/ext4/log-trace_pipe-20160819.txt.gz
>
> rm/cp -al x10 succeeded without any OOM
>
> so the question is - which solution is "the one" for stable/4.7.x ?

I will send an email later today with other people reporting pre-mature
OOMs later today and will make sure you are on the CC list as well.

Thanks for the testing!
--
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

2016-08-23 02:20:56

by Dave Chinner

[permalink] [raw]

Subject: Re: 4.7.0, cp -al causes OOM

On Sun, Aug 14, 2016 at 12:50:49PM +0200, Michal Hocko wrote:
> On Sat 13-08-16 11:42:59, Dave Chinner wrote:
> > On Fri, Aug 12, 2016 at 09:44:55AM +0200, Michal Hocko wrote:
> > However, we throttle the rate at which we dirty pages to prevent
> > filling memory with unreclaimable dirty pages as that causes
> > spurious OOM situations to occur. The same spurious OOM situations
> > occur when memory is full of dirty inodes, and so allocation rate
> > throttling is needed for large scale inode cache intersive workloads
> > like this as well....
>
> Is there any generic way to do this throttling or every fs has to
> implement its own way?

tl;dr: no obvious generic way - every filesystem has different
reclaim requirements and behaviour.

Keep in mind that the inode cache shrinker tries to avoid dirty
inodes on the LRU, so it never blocks on known dirty inodes. Hence
if the LRU is full of dirty inodes, it won't reclaim any inodes, and
it won't block waiting for inodes to come clean. This feeds back to
the shrinker infrastructure in tht total number of inodes freed by a
shrinker pass (i.e. scanned vs freed ratio).

XFS is quite different. It only marks inodes as having dirty pages,
never as being metadata dirty. We don't even implement
->write_inode, because it is never correct for the VFS to write an
XFS inode directly. Hence, for XFS, VFS reclaim only skips inodes
that are still waiting for dirty page writeback to complete. These
inodes can't be immediately reclaimed, anyway, and page reclaim
should block on them if we are hitting near-OOM conditions in the
first place.

Hence, for XFS, inodes that are just metadata dirty (as is the case
of rm -rf, or cp -al), the VFS only sees clean inodes and so
immediately evicts them. XFS inode reclaim is aware of the dirty
status of inodes marked for reclaim, and optimises for it being a
common case.

When the XFS inode shrinker is run from the superblock shrinker, it
first kicks background reclaim threads - that's where most of the
XFS inode reclaim occurs. It runs async, lockless, non-blocking, and
scans the inode cache in IO-optimal order, enabling reclaim to scan,
clean and reclaim hundreds of thousands of dirty inodes per second.

Meanwhile, after kicking background reclaim, the XFS inode shrinker
picks up a "shrinker reclaim cursor" and starts off from where that
points to. It then cleans and blocking on any dirty inodes it
encounters before reclaiming them; clean inodes are immediately
reclaimed without blocking. This is what throttles memory
reclaim - if background reclaim is keeping up, the shrinker will only
see clean inodes and so it will be fast and non-blocking. If background
reclaim is not keeping up, the shrinker will encounter dirty inodes
and hence it blocks dropping the direct reclaim rate to that at
which inodes are being cleaned. When background reclaim catches up,
the shrinker stops throttling on IO....

Further to that, direct reclaim can trigger massive shrinker
concurrency. This happens when lots of processes are all hitting the
memory allocator and we are low on memory. If we allow all of these
direct reclaimers to issue IO to clean dirty inodes, reclaim turns
into small random write IO and the reclaim rate tanks. Badly. OOM
occurs pretty much as soon as this happens. To avoid this, we use
reclaim cursors, and have a limited number of them available. The
shrinker first has to get a reclaim cursor (which is how it knows
where to start reclaim from), and if none are available the shrinker
will block waiting for a cursor. This forms the high level direct
reclaim concurrency control, and it allows the background reclaim
threads to have priority access to the reclaim queues. Hence even
under extreme memory pressure, we get fast, efficient dirty inode
reclaim behaviour rather than overwhelming the IO subsystem and
tanking....

I'm not sure there's anything generic you can get from this; it's
all based around avoiding using the generic VFS dirty inode tracking
to begin with...

Cheers,

Dave.
--
Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>