Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Uladzislau Rezki <urezki@gmail.com>
Date:   Thu, 3 Nov 2022 13:54:23 +0100
To:     Joel Fernandes <joel@joelfernandes.org>
Cc:     Uladzislau Rezki <urezki@gmail.com>,
        "Paul E. McKenney" <paulmck@kernel.org>,
        linux-kernel@vger.kernel.org, rcu@vger.kernel.org
Subject: Re: [PATCH RFC] rcu/kfree: Do not request RCU when not needed
Message-ID: <Y2O5/3vPoWILRj10@pc638.lan>
References: <20221029132856.3752018-1-joel@joelfernandes.org>
 <Y2JkoVV3jaVS4y0Q@pc636>
 <CAEXW_YRZO086TUJCFzuGpAWRpQ-uKD1S5wDipJ9hR9=XddNFSw@mail.gmail.com>
 <20221102163544.GM5600@paulmck-ThinkPad-P17-Gen-1>
 <Y2KnxKDebPKiqTFZ@pc636>
 <CAEXW_YRquk15oGMCrYXLTKWtMzfPJhEJpjENM_rqt4qjwtAt+g@mail.gmail.com>
 <Y2K3jFv12yKJIAYP@pc638.lan>
 <CAEXW_YQWYfJPpeXoV0ZDGC7Kd585LJ+h2YbKfB3unDDZinxTRQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAEXW_YQWYfJPpeXoV0ZDGC7Kd585LJ+h2YbKfB3unDDZinxTRQ@mail.gmail.com>
Precedence: bulk

> 
> 
> Nice, I am happy you worked on it. A comment bellow:
> 
> Below what i have came up with to switch for polling APIs:
> >
> > From 799ce1653d159ef3d35f34a284f738c2c267c75f Mon Sep 17 00:00:00 2001
> > From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
> > Date: Wed, 2 Nov 2022 19:26:27 +0100
> > Subject: [PATCH 1/1] rcu: kvfree_rcu: Reduce a memory footptint by using
> >  polling APIs
> >
> > Total time taken by all kfree'ers: 6564718459 ns, loops: 10000, batches:
> > 1110, memory footprint: 5057MB
> > Total time taken by all kfree'ers: 8431051895 ns, loops: 10000, batches:
> > 1109, memory footprint: 2749MB
> > Total time taken by all kfree'ers: 9477830789 ns, loops: 10000, batches:
> > 1158, memory footprint: 2934MB
> > Total time taken by all kfree'ers: 9950211144 ns, loops: 10000, batches:
> > 981, memory footprint: 2704MB
> >
> > with a patch:
> 
> 
> >
> > Total time taken by all kfree'ers: 7712110118 ns, loops: 10000, batches:
> > 1660, memory footprint: 91MB
> > Total time taken by all kfree'ers: 7002403664 ns, loops: 10000, batches:
> > 1482, memory footprint: 86MB
> > Total time taken by all kfree'ers: 7842282319 ns, loops: 10000, batches:
> > 1738, memory footprint: 86MB
> > Total time taken by all kfree'ers: 7230161977 ns, loops: 10000, batches:
> > 1542, memory footprint: 72MB
> >
> > Tested with NOCB option, all offloading CPUs:
> 
> 
> > kvm.sh --memory 10G --torture rcuscale --allcpus --duration 1 \
> >   --kconfig CONFIG_NR_CPUS=64 \
> >   --kconfig CONFIG_RCU_NOCB_CPU=y \
> >   --kconfig CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y \
> >   --bootargs "rcuscale.kfree_rcu_test=1 rcuscale.kfree_nthreads=16 \
> >   rcuscale.holdoff=20 rcuscale.kfree_loops=10000
> > torture.disable_onoff_at_boot" --trust-make
> >
> > According to data there is a big gain in memory footprint with a patch.
> > It is because of call_rcu() and call_rcu_flush() take more effort and
> > time to queue a callback and then wait for a gp.
> 
> 
> >
> > With polling API:
> >   a) we do not need to queue any callback;
> >   b) we might not even need wait for a GP completion.
> >
> 
> Wow, I did not expect an extra grace period or 2 to cause so much memory
> unreclaimed.
> 
> Does this kernel have the 5 second timeout change?
> 
Yes. But it does not matter. Why? Because 5 second time out happens only
when a system lightly loaded. It can be 10 or 100 seconds.

Once a flood is detected we start to offload back-logged memory.

>
> If yes, then I am confused why extra grace period matters for memory. My
> goal with the idea was to avoid pointless RCU work when we don’t need it.
> But I can’t complaint.
> 
This patch does the same. It avoids of using any call_rcu() because we
do not need it. I mean extra layer.

Why it matters for the memory. I explained it in previous answer. But it
is because of extra layer. Since it is a flood scenario such delays can
lead to higher memory footprint.

The test easily detects it.

>
> Or is it that this test floods the pointer page so we do not wait full 5
> seconds, and benefit when we reclaim sooner with polled API, than waiting
> for grace periods?
> 
Right. Reclaim sooner with polling API.

--
Uladzislau Rezki