Received: by 2002:a25:868d:0:0:0:0:0 with SMTP id z13csp166000ybk; Tue, 12 May 2020 18:56:35 -0700 (PDT) X-Google-Smtp-Source: APiQypILjJsDp29qEh88Z+IN6JNHyuJKJagT877qPBOx6a57yv71RKd6nySEjBjrHyYcenj/PJkP X-Received: by 2002:a17:906:454a:: with SMTP id s10mr20433929ejq.141.1589334995273; Tue, 12 May 2020 18:56:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1589334995; cv=none; d=google.com; s=arc-20160816; b=wtHdM+WYM71vBZkvr9P0E/U8zgeRxP7k8rhRfDTruQ2sN8v0Ne/Ck0f6fPYz4/nH3o H3o2Hh8/rx3R4pMM/haGfFcw5DzzNoI+HhfTcT4LlL5jeW2n1aTUNOyQUvkqySsXmQdr 5wu5qynd6mLchpKya9UX6dqU3hnwi8ne4kaamAszdSyYl9opK64ofbDYREB6nguKPO0Q ZdEmcqOo6NiIHlAFS64TC8xdCq20YXNP4BHzqjTVE/ttpmi58E5jc+QECksvD000ecOl 5oEBPnEFNEehdjnhflZ1+GIs9BqWLyK5h/KMy3aYuesHSADOnYyHLt5zbkrgeHWi0v5J mj9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date; bh=YrriXCuEhnP4aYZzt4wWyqEXAF+LK32P6nBLNv76W9s=; b=DeBzPicztxI5uQx2jCyfX3xZLou10RdJi+btyzAfVkK0DBz6/LZAcvpvXY54LbnH+T T4DzAvGPhnLbSWqdUC+wd5g8rMYWUfzcP1EdyuQCVHGAaBi6gzJTckKQP/s1hA7P172t +84N9dbdAvxIZ+YKQ/yA8WK5U4bzJaKU2Xnd3CQy0CroYjI3pxSsUjbpan8krHdqShZ+ X/FUtKVAlYMOdKlHP8syUwjgLoXUFuDqmIh9aiUoHtte8Kbo062qA1dwfcd8TvXukzN7 z7615nNhPZz+8W9EgnZ6bXR9cQhvQRbI5TS7VgGWdAtBqYWgUn7r7i9fMCv/XbZ3AJLb 5BOg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id x17si9685068edi.409.2020.05.12.18.56.12; Tue, 12 May 2020 18:56:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728887AbgEMByu (ORCPT + 99 others); Tue, 12 May 2020 21:54:50 -0400 Received: from mail109.syd.optusnet.com.au ([211.29.132.80]:32862 "EHLO mail109.syd.optusnet.com.au" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726028AbgEMByu (ORCPT ); Tue, 12 May 2020 21:54:50 -0400 X-Greylist: delayed 1313 seconds by postgrey-1.27 at vger.kernel.org; Tue, 12 May 2020 21:54:48 EDT Received: from dread.disaster.area (pa49-195-157-175.pa.nsw.optusnet.com.au [49.195.157.175]) by mail109.syd.optusnet.com.au (Postfix) with ESMTPS id E353ED7B1C6; Wed, 13 May 2020 11:32:40 +1000 (AEST) Received: from dave by dread.disaster.area with local (Exim 4.92.3) (envelope-from ) id 1jYgGE-0001cR-CO; Wed, 13 May 2020 11:32:38 +1000 Date: Wed, 13 May 2020 11:32:38 +1000 From: Dave Chinner To: "Paul E. McKenney" Cc: Konstantin Khlebnikov , Johannes Weiner , Andrew Morton , rcu@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com, mingo@kernel.org, jiangshanlai@gmail.com, dipankar@in.ibm.com, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com, joel@joelfernandes.org, viro@zeniv.linux.org.uk Subject: Re: [PATCH RFC tip/core/rcu] Add shrinker to shift to fast/inefficient GP mode Message-ID: <20200513013238.GM2005@dread.disaster.area> References: <20200507004240.GA9156@paulmck-ThinkPad-P72> <20200506175535.d4986a4d497071a410b69765@linux-foundation.org> <20200507170006.GA155220@cmpxchg.org> <20200507170903.GR2869@paulmck-ThinkPad-P72> <20200507183102.GB155220@cmpxchg.org> <20200507190905.GX2869@paulmck-ThinkPad-P72> <6d93affb-6505-1bf3-58a0-c67c34a18a9e@yandex-team.ru> <20200508144638.GF2869@paulmck-ThinkPad-P72> <8671cc58-11e0-b9f0-74b1-264fea58b23c@yandex-team.ru> <20200509160900.GM2869@paulmck-ThinkPad-P72> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200509160900.GM2869@paulmck-ThinkPad-P72> User-Agent: Mutt/1.10.1 (2018-07-13) X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.3 cv=W5xGqiek c=1 sm=1 tr=0 a=ONQRW0k9raierNYdzxQi9Q==:117 a=ONQRW0k9raierNYdzxQi9Q==:17 a=kj9zAlcOel0A:10 a=sTwFKg_x9MkA:10 a=7-415B0cAAAA:8 a=UnOJVGtEmMKoa6n_RpMA:9 a=VP7J8LPk42pBYOac:21 a=FTdNB1hloREqk2B-:21 a=CjuIK1q_8ugA:10 a=biEYGPWJfzWAr4FL6Ov7:22 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, May 09, 2020 at 09:09:00AM -0700, Paul E. McKenney wrote: > On Sat, May 09, 2020 at 11:54:40AM +0300, Konstantin Khlebnikov wrote: > > On 08/05/2020 17.46, Paul E. McKenney wrote: > > > Easy for me to provide "start fast and inefficient mode" and "stop fast > > > and inefficient mode" APIs for MM to call! > > > > > > How about rcu_mempressure_start() and rcu_mempressure_end()? I would > > > expect them not to nest (as in if you need them to nest, please let > > > me know). I would not expect these to be invoked all that often (as in > > > if you do need them to be fast and scalable, please let me know). > > > > RCU would then be in fast/inefficient mode if either MM told it to be > > > or if RCU had detected callback overload on at least one CPU. > > > > > > Seem reasonable? > > > > Not exactly nested calls, but kswapd threads are per numa node. > > So, at some level nodes under pressure must be counted. > > Easy enough, especially given that RCU already "counts" CPUs having > excessive numbers of callbacks. But assuming that the transitions to/from > OOM are rare, I would start by just counting them with a global counter. > If the counter is non-zero, RCU is in fast and inefficient mode. > > > Also forcing rcu calls only for cpus in one numa node might be useful. > > Interesting. RCU currently evaluates a given CPU by comparing the > number of callbacks against a fixed cutoff that can be set at boot using > rcutree.qhimark, which defaults to 10,000. When this cutoff is exceeded, > RCU becomes more aggressive about invoking callbacks on that CPU, for > example, by sacrificing some degree of real-time response. I believe > that this heuristic would also serve the OOM use case well. So one of the things that I'm not sure people have connected here is that memory reclaim done by shrinkers is one of the things that drives huge numbers of call_rcu() callbacks to free memory via rcu. If we are reclaiming dentries and inodes, then we can be pushing thousands to hundreds of thousands of objects into kfree_rcu() and/or direct call_rcu() calls to free these objects in a single reclaim pass. Hence the trigger for RCU going into "excessive callback" mode might, in fact, be kswapd running a pass over the shrinkers. i.e. memory reclaim itself can be responsible for pushing RCU into this "OOM pressure" situation. So perhaps we've missed a trick here by not having the memory reclaim routines trigger RCU callbacks at the end of a priority scan. The shrinkers have queued the objects for freeing, but they haven't actually been freed yet and so things like slab pages haven't actually been returned to the free pool even though the shrinkers have said "freed this many objects"... i.e. perhaps the right solution here is a "rcu_run_callbacks()" function that memory reclaim calls before backing off and/or winding up reclaim priority. > > I wonder if direct-reclaim should at some stage simply wait for RCU QS. > > I.e. call rcu_barrier() or similar somewhere before invoking OOM. > > The rcu_oom_count() function in the patch starting this thread returns the > total number of outstanding callbacks queued on all CPUs. So one approach > would be to invoke this function, and if the return value was truly > huge (taking size of memory and who knows that all else into account), > do the rcu_barrier() to wait for RCU to clear its current backlog. The shrinker scan control structure has a node mask in it to indicate what node (and hence CPUs) it should be reclaiming from. This information comes from the main reclaim scan routine, so it would be trivial to feed straight into the RCU code to have it act on just the CPUs/node that we are reclaiming memory from... > On the NUMA point, it would be dead easy for me to supply a function > that returned the number of callbacks on a given CPU, which would allow > you to similarly evaluate a NUMA node, a cgroup, or whatever. I'd think it runs the other way around - we optimisitically call the RCU layer to do cleanup, and the RCU layer decides if there's enough queued callbacks on the cpus/node to run callbacks immediately. It would even be provided with the scan priority to indicate the level of desperation memory reclaim is under.... > > All GFP_NOFAIL users should allow direct-reclaim, thus this loop > > in page_alloc shouldn't block RCU and doesn't need special care. > > I must defer to you guys on this. The main caution is the duration of > direct reclaim. After all, if it is too long, the kfree_rcu() instance > would have been better of just invoking synchronize_rcu(). Individual callers of kfree_rcu() have no idea of the load on RCU, nor how long direct reclaim is taking. Calling synchronize_rcu() incorrectly has pretty major downsides to it, so nobody should be trying to expedite kfree_rcu() unless there is a good reason to do so (e.g. at unmount to ensure everything allocated by a filesystem has actually been freed). Hence I'd much prefer the decision to expedite callbacks is made by the RCU subsystem based on it's known callback load and some indication of how close memory reclaim is to declaring OOM... Cheers, Dave. -- Dave Chinner david@fromorbit.com