Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 28 Jun 2022 06:46:59 -0700
From:   "Paul E. McKenney" <paulmck@kernel.org>
To:     Marc Zyngier <maz@kernel.org>
Cc:     Neeraj Upadhyay <quic_neeraju@quicinc.com>, frederic@kernel.org,
        josh@joshtriplett.org, rostedt@goodmis.org,
        mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com,
        joel@joelfernandes.org, linux-kernel@vger.kernel.org,
        zhangfei.gao@foxmail.com, boqun.feng@gmail.com, urezki@gmail.com,
        shameerali.kolothum.thodi@huawei.com, pbonzini@redhat.com,
        mtosatti@redhat.com, eric.auger@redhat.com,
        chenxiang66@hisilicon.com
Subject: Re: [PATCH] srcu: Reduce blocking agressiveness of expedited grace
 periods further
Message-ID: <20220628134659.GY1790663@paulmck-ThinkPad-P17-Gen-1>
Reply-To: paulmck@kernel.org
References: <20220627123706.20187-1-quic_neeraju@quicinc.com>
 <875ykl2mb2.wl-maz@kernel.org>
 <dc24db89-05ae-c113-6728-de797e041123@quicinc.com>
 <874k052kxh.wl-maz@kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <874k052kxh.wl-maz@kernel.org>
Precedence: bulk

On Tue, Jun 28, 2022 at 10:31:54AM +0100, Marc Zyngier wrote:
> On Tue, 28 Jun 2022 10:17:24 +0100,
> Neeraj Upadhyay <quic_neeraju@quicinc.com> wrote:
> > 
> > 
> > 
> > On 6/28/2022 2:32 PM, Marc Zyngier wrote:
> > > On Mon, 27 Jun 2022 13:37:06 +0100,
> > > Neeraj Upadhyay <quic_neeraju@quicinc.com> wrote:
> > >> 
> > >> Commit 640a7d37c3f4 ("srcu: Block less aggressively for expedited
> > >> grace periods") highlights a problem where aggressively blocking
> > >> SRCU expedited grace periods, as was introduced in commit
> > >> 282d8998e997 ("srcu: Prevent expedited GPs and blocking readers
> > >> from consuming CPU"), introduces ~2 minutes delay to the overall
> > >> ~3.5 minutes boot time, when starting VMs with "-bios QEMU_EFI.fd"
> > >> cmdline on qemu, which results in very high rate of memslots
> > >> add/remove, which causes > ~6000 synchronize_srcu() calls for
> > >> kvm->srcu SRCU instance.
> > >> 
> > >> Below table captures the experiments done by Zhangfei Gao, Shameer,
> > >> to measure the boottime impact with various values of non-sleeping
> > >> per phase counts, with HZ_250 and preemption enabled:
> > >> 
> > >> +──────────────────────────+────────────────+
> > >> | SRCU_MAX_NODELAY_PHASE   | Boot time (s)  |
> > >> +──────────────────────────+────────────────+
> > >> | 100                      | 30.053         |
> > >> | 150                      | 25.151         |
> > >> | 200                      | 20.704         |
> > >> | 250                      | 15.748         |
> > >> | 500                      | 11.401         |
> > >> | 1000                     | 11.443         |
> > >> | 10000                    | 11.258         |
> > >> | 1000000                  | 11.154         |
> > >> +──────────────────────────+────────────────+
> > >> 
> > >> Analysis on the experiment results showed improved boot time
> > >> with non blocking delays close to one jiffy duration. This
> > >> was also seen when number of per-phase iterations were scaled
> > >> to one jiffy.
> > >> 
> > >> So, this change scales per-grace-period phase number of non-sleeping
> > >> polls, soiuch that, non-sleeping polls are done for one jiffy. In addition
> > >> to this, srcu_get_delay() call in srcu_gp_end(), which is used to calculate
> > >> the delay used for scheduling callbacks, is replaced with the check for
> > >> expedited grace period. This is done, to schedule cbs for completed expedited
> > >> grace periods immediately, which results in improved boot time seen in
> > >> experiments.
> > >> 
> > >> In addition to the changes to default per phase delays, this change
> > >> adds 3 new kernel parameters - srcutree.srcu_max_nodelay,
> > >> srcutree.srcu_max_nodelay_phase, srcutree.srcu_retry_check_delay.
> > >> This allows users to configure the srcu grace period scanning delays,
> > >> depending on their system configuration requirements.
> > >> 
> > >> Signed-off-by: Neeraj Upadhyay <quic_neeraju@quicinc.com>
> > > 
> > > I've given this a go on one of my test platforms (the one I noticed
> > > the issue on the first place), and found that the initial part of the
> > > EFI boot under KVM (pointlessly wiping the emulated flash) went down
> > > to 1m7s from 3m50s (HZ=250).
> > > 
> > > Clearly a massive improvement, but still a far cry from the original
> > > ~40s (yes, this box is utter crap -- which is why I use it).
> > 
> > Do you see any improvement by using "srcutree.srcu_max_nodelay=1000"
> > bootarg, on top of this patch?
> 
> Yup, this brings it back to 43s on a quick test run, which is close
> enough to what I had before.
> 
> How does a random user come up with such a value though?

There was some talk of moving from synchronize_srcu_expedited() to
call_srcu() with the occasional srcu_barrier() to avoid OOM.  If that
proves to be practical, that should get decent performance with little
tuning.  But in the meantime, we need to avoid hangs due to CPU-bound
tasks in one workload while still avoiding massive boot-time slowdowns
in your workload.

Right now, Neeraj's carefully  tuned approach is the one way we know to
square this particular circle.

							Thanx, Paul