Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
From:   "Zhang, Qiang1" <qiang1.zhang@intel.com>
To:     "paulmck@kernel.org" <paulmck@kernel.org>
CC:     "frederic@kernel.org" <frederic@kernel.org>,
        "rcu@vger.kernel.org" <rcu@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v2] rcu/nocb: Avoid polling when myrdp->nocb_head_rdp list
 is empty
Thread-Topic: [PATCH v2] rcu/nocb: Avoid polling when myrdp->nocb_head_rdp
 list is empty
Thread-Index: AQHYekOLcM/pjDOLHUOfGqJWFKxX8a1EH1EAgACISxCAAAiugIAAAQywgAA0YoCAAB/80A==
Date:   Wed, 8 Jun 2022 06:28:33 +0000
Message-ID: <PH0PR11MB58809FEA1314CF0A7AFAD2BEDAA49@PH0PR11MB5880.namprd11.prod.outlook.com>
References: <20220607075057.909070-1-qiang1.zhang@intel.com>
 <20220607161620.GH1790663@paulmck-ThinkPad-P17-Gen-1>
 <PH0PR11MB5880551F642B1490CEFA65A8DAA49@PH0PR11MB5880.namprd11.prod.outlook.com>
 <20220608005513.GK1790663@paulmck-ThinkPad-P17-Gen-1>
 <PH0PR11MB588014C2451C2E65E583924DDAA49@PH0PR11MB5880.namprd11.prod.outlook.com>
 <20220608040627.GL1790663@paulmck-ThinkPad-P17-Gen-1>
In-Reply-To: <20220608040627.GL1790663@paulmck-ThinkPad-P17-Gen-1>
Accept-Language: en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-version: 11.6.500.17
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?UKvJMYxUa4GNwi8pC+v3s7gVpEmL2LUSJBrSPG8NTf9EiwShyEgL+d52LEPj?=
 =?us-ascii?Q?A+qGXr/IcFk0ElIoqr50aKBwWHJTGP260UhxL8phTsWbYhy8fDEb6NKsxbIx?=
 =?us-ascii?Q?eTq5zJuE/twkxFpbs49nTEMQ5oynWGiUBeoEuoq2Zxlk4fW8RNijVMIP9VMm?=
 =?us-ascii?Q?xxvp9dlggTmlEswkOVA/R22YE3WYnrxNCoj8C+m+t4QPM6v49/oiNvP8IjCh?=
 =?us-ascii?Q?BXJ/4p1/fTlGeJiW3Dq5ygRzOItpCFnlhCYjn8NDQZAEeoWUPmOvz1hPoIZM?=
 =?us-ascii?Q?ePyyOisHw2AWtf+lWuk/IJB16I1rrWsjak5FMgP1Svrx6zlBjb2hQ5SbhsBS?=
 =?us-ascii?Q?9M5LTmZ/zASIdWnFyhwzKHKJvo8hYAJ+CknvW4I5w+tYr6IBQU5Dr8MesJ1i?=
 =?us-ascii?Q?8e2ma0C/1wkRkCT++3JzWTuGE7zdcxKdRwompeGZq3Li8byMwbWGf7nAK3kk?=
 =?us-ascii?Q?WqC1NTkvsJVfsG+RXWIJREA6bHAAiP0YNMaBrH0v5pOBIvT1rX/x0f92hbV9?=
 =?us-ascii?Q?XmnZWM8HnF6tUUK/6gRGkdVWUMF9jJDDthcXVpI/qOcuyniRY9N+3woUJIms?=
 =?us-ascii?Q?wsIYHDtTmvN8rnao0GFUuePzNGNRqv4YG5cohkSc5DnGNIy3gStZa4el3oek?=
 =?us-ascii?Q?pnU9kLRyg/8bhraFv/+Ec8WJOLvpJxoscgJmyCEnI/kjOWUqIj2B57j0Z/15?=
 =?us-ascii?Q?bBHctKGOuXTfp0jaUCAdkbrQ6bGXjV6wld2TEZKcX58+MJKfXi6KF0cIIf5n?=
 =?us-ascii?Q?uikt3JmL+jIc7Ebi4Rxz8HwnLiLiUgPIgdUa/+GtgOfm1ID9QcVxvXf/uE1f?=
 =?us-ascii?Q?62BJ3uEW5Z/pGIBy2bOY6W0X63k6jAIKyhildlBh42NRMygUhzD1VrJiV8/X?=
 =?us-ascii?Q?Lsd6MKWbKhII+nJPweGVvZHfU3MCvwvpY2Ex7f7tDY+YPmnet1pWoCf/2rO8?=
 =?us-ascii?Q?w1uBfqXtWvJKf9s1mJExYCwN0m+se/rvrZR89mo2fQ7FJmCf9znbqwhUztyD?=
 =?us-ascii?Q?kKWfXTTTfYmJZDsXVHUKS+eOioistYr7hP0Bs3Y/S5ozbIz5DWf8+i0ZEb8S?=
 =?us-ascii?Q?xGb+MIoUpYFJrhBqvV0Q5tE8AFH30XHFum7WF9nN6ZhODp328Tb4lTDwRdVA?=
 =?us-ascii?Q?RKIelDJLSxVaoO7+5m7XW5vja28IxIrJZvys2odOviAFQK26HTjuHg5v1LqY?=
 =?us-ascii?Q?N0vOfK0NAZZaMBoZjLdUTlRPsh3avF8pUUSw1laDSEJ3jPfnGJJQ3nBuRgPj?=
 =?us-ascii?Q?IKtxij0zQgiIyfizx3q0ilAqSZXYj0mj3B9/51G0drohGo/Wyt/D07jzds8h?=
 =?us-ascii?Q?QjIMBGsJbutViCUU6ZaCM2LBzoQHnGozzdtIj+eZho0e4yEwCFlhxsfqKcGW?=
 =?us-ascii?Q?5deeUHIex8mTLvijJ0LSmrwmwU5otQbIsaPKeoM1kaxVxUnm3DLVP4v3Kgxb?=
 =?us-ascii?Q?k6C7VEAq8tcWdKVaZtdTzoLYjyigcy5Bs3zW760pOitYIKf/RwESHyt8uw0V?=
 =?us-ascii?Q?N1JJ0kLNYlVXt76triefQq8Q2T8gMv4rB4WSEzVJF+SHq38CSSqvlxa0GBZ9?=
 =?us-ascii?Q?waIUzZ54d9t7vKUmx0VzQ+3FxIrrjsy8cAugh4yfB/wpRBH2V+RB0Ap/wV+2?=
 =?us-ascii?Q?kk/v0gK7NFd4THpX4N/ht8QcodTk2CVww4abV7mmCMWbzahDhwQSt8vEra0T?=
 =?us-ascii?Q?KekKNLowXPrwxwC/WtjUYGvfVIUclx4jRurFqOJA9kPZzXAYqFpmRJw53woS?=
 =?us-ascii?Q?WfNMdprKtw=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5880.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 435690e2-2ff0-4d17-f0b2-08da49181cf8
X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Jun 2022 06:28:33.9997
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: +YaphI6BygzC9YO/o1FVC0am7A5TDWE5IPeJA43DtkAhFeXXUptWILIpoa9f7LQdbcI4qBRW0v1U1vOypzsRcA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB4855
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-3.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Jun 08, 2022 at 01:01:31AM +0000, Zhang, Qiang1 wrote:
> On Wed, Jun 08, 2022 at 12:41:28AM +0000, Zhang, Qiang1 wrote:
> > On Tue, Jun 07, 2022 at 03:50:57PM +0800, Zqiang wrote:
> > > Currently, If the 'rcu_nocb_poll' bootargs is enable, all rcuog=20
> > > kthreads enter polling mode. however, due to only insert CPU's rdp=20
> > > which belong to rcu_nocb_mask to 'nocb_head_rdp' list or all CPU's=20
> > > rdp served by rcuog kthread have been de-offloaded, these cause=20
> > > the 'nocb_head_rdp' list served by rcuog kthread is empty, when=20
> > > the 'nocb_head_rdp' is empty, the rcuog kthread in polling mode=20
> > > not actually do anything. fix it by exiting polling mode when the=20
> > > 'nocb_head_rdp'list is empty, otherwise entering polling mode.
> > >=20
> > > Signed-off-by: Zqiang <qiang1.zhang@intel.com>
> > >
> > >This looks a bit more plausible, but what have you done to test this?
> >=20
> > Yes , I have only tested as follows and I added two trace events.
> >=20
> > +                      trace_rcu_nocb_wake(rcu_state.name, cpu,=20
> > + TPS("EnterNoPoll"));
> >                        =20
> > rcu_wait(READ_ONCE(my_rdp->nocb_toggling_rdp));
> > +                       trace_rcu_nocb_wake(rcu_state.name, cpu,=20
> > + TPS("ExitNoPoll"));
> >=20
> > runqemu kvm slirp nographic qemuparams=3D"-m 2048 -smp 4"=20
> > bootparams=3D"rcu_nocbs=3D2,3 rcutree.dump_tree=3D1 rcu_nocb_poll
> > rcutorture.nocbs_nthreads=3D4 rcutorture.test_boost=3D0" -d

Hi Paul, The test trace log is as follows:

         rcuog/0-19      [003] .....   169.400494: rcu_nocb_wake: rcu_preem=
pt 0 EnterNoPoll
         rcuog/0-19      [003] .....   172.592072: rcu_nocb_wake: rcu_preem=
pt 0 ExitNoPoll
         rcuog/2-36      [002] .....   173.553717: rcu_nocb_wake: rcu_preem=
pt 2 EnterNoPoll
         rcuog/2-36      [002] .....   173.750084: rcu_nocb_wake: rcu_preem=
pt 2 ExitNoPoll

Thanks
Zqiang

>=20
> >To exercise your change, could you please also add an appropriate value =
for the rcutorture.nocbs_nthreads kernel boot parameter?  Without that boot=
 parameter, your kernel will not actually offload or deoffload any CPUs.
>=20
> I have set ' rcutorture.nocbs_nthreads=3D4 ' in bootargs.
>
>I see it now, apologies for my confusion!
>
>							Thanx, Paul
>
> >An alternative approach is to run rcutorture scenario TREE01.
>=20
> I will also use TREE01 for testing
>=20
> Thanks
> Zqiang
>=20
> >
> >							Thanx, Paul
>=20
> > Thanks
> > Zqiang
> >=20
> > >
> > >							Thanx, Paul
> > >
> > > ---
> > >  v1->v2:
> > >  Move rcu_nocb_poll flags check from rdp_offload_toggle() to=20
> > > rcu_nocb_rdp_offload/deoffload(), avoid unnecessary setting of=20
> > > rdp_gp->nocb_gp_sleep flags, because when rcu_nocb_poll is set =20
> > > the rdp_gp->nocb_gp_sleep is not used.
> > >=20
> > >  kernel/rcu/tree_nocb.h | 16 ++++++++++------
> > >  1 file changed, 10 insertions(+), 6 deletions(-)
> > >=20
> > > diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index
> > > fa8e4f82e60c..2a52c9abc681 100644
> > > --- a/kernel/rcu/tree_nocb.h
> > > +++ b/kernel/rcu/tree_nocb.h
> > > @@ -698,10 +698,14 @@ static void nocb_gp_wait(struct rcu_data *my_rd=
p)
> > >  				   TPS("WakeBypassIsDeferred"));
> > >  	}
> > >  	if (rcu_nocb_poll) {
> > > -		/* Polling, so trace if first poll in the series. */
> > > -		if (gotcbs)
> > > -			trace_rcu_nocb_wake(rcu_state.name, cpu, TPS("Poll"));
> > > -		schedule_timeout_idle(1);
> > > +		if (list_empty(&my_rdp->nocb_head_rdp)) {
> > > +			rcu_wait(READ_ONCE(my_rdp->nocb_toggling_rdp));
> > > +		} else {
> > > +			/* Polling, so trace if first poll in the series. */
> > > +			if (gotcbs)
> > > +				trace_rcu_nocb_wake(rcu_state.name, cpu, TPS("Poll"));
> > > +			schedule_timeout_idle(1);
> > > +		}
> > >  	} else if (!needwait_gp) {
> > >  		/* Wait for callbacks to appear. */
> > >  		trace_rcu_nocb_wake(rcu_state.name, cpu, TPS("Sleep")); @@
> > > -1030,7
> > > +1034,7 @@ static long rcu_nocb_rdp_deoffload(void *arg)
> > > =20
> > >  	mutex_lock(&rdp_gp->nocb_gp_kthread_mutex);
> > >  	if (rdp_gp->nocb_gp_kthread) {
> > > -		if (wake_gp)
> > > +		if (wake_gp || rcu_nocb_poll)
> > >  			wake_up_process(rdp_gp->nocb_gp_kthread);
> > > =20
> > >  		/*
> > > @@ -1152,7 +1156,7 @@ static long rcu_nocb_rdp_offload(void *arg)
> > >  	 *      rcu_nocb_unlock()         rcu_nocb_unlock()
> > >  	 */
> > >  	wake_gp =3D rdp_offload_toggle(rdp, true, flags);
> > > -	if (wake_gp)
> > > +	if (wake_gp || rcu_nocb_poll)
> > >  		wake_up_process(rdp_gp->nocb_gp_kthread);
> > >  	swait_event_exclusive(rdp->nocb_state_wq,
> > >  			      rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB)=20
> > > &&
> > > --
> > > 2.25.1
> > >=20