Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Zhang, Qiang1" <qiang1.zhang@intel.com>
To:     Frederic Weisbecker <frederic@kernel.org>
CC:     "paulmck@kernel.org" <paulmck@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Neeraj Upadhyay <quic_neeraju@quicinc.com>,
        Uladzislau Rezki <uladzislau.rezki@sony.com>,
        Boqun Feng <boqun.feng@gmail.com>
Subject: RE: [PATCH] rcu/nocb: Clear rdp offloaded flags when rcuop/rcuog
 kthreads spawn failed
Thread-Topic: [PATCH] rcu/nocb: Clear rdp offloaded flags when rcuop/rcuog
 kthreads spawn failed
Thread-Index: AQHYLIaY7+I2utbBBk2DOZbNtjWjdKyt5C0AgAbbncCAAtphgIAASQVg
Date:   Thu, 10 Mar 2022 02:37:41 +0000
Message-ID: <PH0PR11MB58803A987BB0D7B51D3FD990DA0B9@PH0PR11MB5880.namprd11.prod.outlook.com>
References: <20220228093629.3746473-1-qiang1.zhang@intel.com>
 <20220303164914.GA87151@lothringen>
 <PH0PR11MB5880F450C2DDD04D4C76F814DA099@PH0PR11MB5880.namprd11.prod.outlook.com>
 <20220309210657.GA68899@lothringen>
In-Reply-To: <20220309210657.GA68899@lothringen>
Accept-Language: en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-version: 11.6.401.20
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?FAbphzxkz3RRhesyLPmB+/Vv1kgtEFTkwv6+xEcrpGWX+AH7VXzGkPLw54fU?=
 =?us-ascii?Q?GLIMaTw6GIfP9ecc+91XcBZCE5cZtPQ/N8+rSxqud4TrE7jzjXHSLd2owE+2?=
 =?us-ascii?Q?p4uN08tsMyb8hWcv7+B0u7EWi631vPpHEDEg31puhmisoSCplGHvLrg2K6NX?=
 =?us-ascii?Q?SBd4w4/IcxJbFIJ+6NyfS88U/8IBYY6lFLVscxMTnkudqak3Uyb8R8unmFQn?=
 =?us-ascii?Q?9vu7zCzs3zb9dYkmbqWRMtmDm41uaZh5Gh3snfJT7fQSRKaZ3pmyERbANbKd?=
 =?us-ascii?Q?FdHK+ulPh6hct0smt3tyyR7zbnq7HfkZfrodbDLjyPKlzukzAd1J4U2RLZgA?=
 =?us-ascii?Q?6iLRpTwcgeCEuqnSQqU4/aipf7lNkjOGAmZDeGPXs46kLE2Qaafjn+C5OK4t?=
 =?us-ascii?Q?Q69BGMoBuB+ciXeoAk6byy/XPVX+ZFnwqccbkCKNsMV8932ve9yZ7s626MF/?=
 =?us-ascii?Q?e4Kx3oVIszToRZLApKGxyJQMt9Vc2QakqZIW1owE61a9FcarHQGhMVQDFmK+?=
 =?us-ascii?Q?Zq+mUpD8ZpfEaLvDhvkzG2Zex+a1WT87vM1GOi0cr3G2kFUpm+FSZM7jAV7u?=
 =?us-ascii?Q?VVlRNv8HRiAefMy6TTwgtCvzrj5pdQscCFA+ITvQcvsuKVeG7LOuYcnmGuNo?=
 =?us-ascii?Q?A+QuSFDAswq4dsBljMu+yOIzg+pNHOgDOTqBkQQUKvgUokqtq0oxZJ+OHnFO?=
 =?us-ascii?Q?g3j8Fj9VY9ENsk/fPSUZab/rG6vEM8yogrUGrfri62ca6daJ7ORbUG3JF9EV?=
 =?us-ascii?Q?mVuwERVB6Tcbwn+572+83sWjfRmmLVJjheF2iJhtA2RYVI5MW3ctdEmMJHcK?=
 =?us-ascii?Q?wy7MAD3HDb9mUTBcbd8LaGXpzGWaK8qtj0Ss8p8Xj3Tx7XKXDdrsDRI5F7SH?=
 =?us-ascii?Q?TY1uCH8R/L/M1biulYteMeQcfZP/Vve8ifrc/KLuTHleKzJ+Y05J/COvGq6e?=
 =?us-ascii?Q?i1gBtGdix2L1etF+bHvAToq0vmD4U7UNSVORf7ABt+icrImhdSnli2k5xjzg?=
 =?us-ascii?Q?jgQTyqUnzg8e1wwg8yWkuNMmLqkCiEtBzFqw1mFKE8QyLdC5WCv1U6cEhy/h?=
 =?us-ascii?Q?xudbDrpasgCL2/yHH6W+dU6Svrx7WsBR7bD7WzK49O0ffYcXQfREBosDIYDk?=
 =?us-ascii?Q?hOFACBv6bR4gXyf0VadhcqD1I40A5s6wsAueUkqP6QCyxFwJS0ZIFlbWyDYP?=
 =?us-ascii?Q?mn8JidFHQgKoSWbrXDb7aEw3xoT170w5mgEmay9k4Yu9tx/OkKgjGrGCbNHp?=
 =?us-ascii?Q?LeQUpHBcUc8Iajw0+EziInpIXfUKRZxy2m7K5Q6+GH14tlEjEFXgnL6Q6iYs?=
 =?us-ascii?Q?E8yOQKl0hi9hxrufRB20wl9A69ZmHcPtQIjFw4UI5kTJ7DaRTA+Z+mOjsnLf?=
 =?us-ascii?Q?NMhcE6XRC6mvTggdVwazEXOaekGzVEJGFPqog2LGWsmLRlksuBVwIykAbO7R?=
 =?us-ascii?Q?vI0EeS2mfxcMlVLzhyVGriVnnS3EUwpr?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5880.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4463d2ef-5d44-47ed-2c13-08da023ef316
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Mar 2022 02:37:41.4797
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: HKq4tg232GSmqIChRN3qoIz0STfm5FJhonMH8gGgH6bc+CBeWssLu1gtnKZ8tEq9/CnFpeoASlX7zkbjyF2qEw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB4944
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Mar 08, 2022 at 07:37:24AM +0000, Zhang, Qiang1 wrote:
>=20
> On Mon, Feb 28, 2022 at 05:36:29PM +0800, Zqiang wrote:
> > When CONFIG_RCU_NOCB_CPU is enabled and 'rcu_nocbs' is set, the=20
> > rcuop and rcuog kthreads is created. however the rcuop or rcuog=20
> > kthreads creation may fail, if failed, clear rdp offloaded flags.
> >=20
> > Signed-off-by: Zqiang <qiang1.zhang@intel.com>
> > ---
> >  kernel/rcu/tree_nocb.h | 14 ++++++++++++--
> >  1 file changed, 12 insertions(+), 2 deletions(-)
> >=20
> > diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index
> > 46694e13398a..94b279147954 100644
> > --- a/kernel/rcu/tree_nocb.h
> > +++ b/kernel/rcu/tree_nocb.h
> > @@ -1246,7 +1246,7 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu)
> >  				"rcuog/%d", rdp_gp->cpu);
> >  		if (WARN_ONCE(IS_ERR(t), "%s: Could not start rcuo GP kthread, OOM i=
s now expected behavior\n", __func__)) {
> >  			mutex_unlock(&rdp_gp->nocb_gp_kthread_mutex);
> > -			return;
> > +			goto end;
> >  		}
> >  		WRITE_ONCE(rdp_gp->nocb_gp_kthread, t);
> >  		if (kthread_prio)
> > @@ -1258,12 +1258,22 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu)
> >  	t =3D kthread_run(rcu_nocb_cb_kthread, rdp,
> >  			"rcuo%c/%d", rcu_state.abbr, cpu);
> >  	if (WARN_ONCE(IS_ERR(t), "%s: Could not start rcuo CB kthread, OOM is=
 now expected behavior\n", __func__))
> > -		return;
> > +		goto end;
> > =20
> >  	if (kthread_prio)
> >  		sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
> >  	WRITE_ONCE(rdp->nocb_cb_kthread, t);
> >  	WRITE_ONCE(rdp->nocb_gp_kthread, rdp_gp->nocb_gp_kthread);
> > +	return;
> > +end:
> > +	if (cpumask_test_cpu(cpu, rcu_nocb_mask)) {
> > +		rcu_segcblist_offload(&rdp->cblist, false);
> > +		rcu_segcblist_clear_flags(&rdp->cblist,
> > +				SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP);
> > +		rcu_segcblist_clear_flags(&rdp->cblist, SEGCBLIST_LOCKING);
> > +		rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_RCU_CORE);
> > +	}
> >>
> >>Thanks you, consequences are indeed bad otherwise because the target is=
 considered offloaded but nothing actually handles the callbacks.
> >>
> >>A few issues though:
> >>
> >>* The rdp_gp kthread may be running concurrently. If it's iterating=20
> >>this rdp and
> >>  the SEGCBLIST_LOCKING flag is cleared in the middle,=20
> >>rcu_nocb_unlock() won't
> >>  release (among many other possible issues).
> >>
> >>* we should clear the cpu from rcu_nocb_mask or we won't be able to=20
> >>later
> >>  re-offload it.
> >>
> >>* we should then delete the rdp from the group list:
> >>
> >>     list_del_rcu(&rdp->nocb_entry_rdp);
> >>
> >>So ideally we should call rcu_nocb_rdp_deoffload(). But then bear in mi=
nd:
> >>
> >>1) We must lock rcu_state.barrier_mutex and hotplug read lock. But sinc=
e we
> >>   are calling rcutree_prepare_cpu(), we maybe holding hotplug write lo=
ck
> >>   already.
> >>
> >>   Therefore we first need to invert the locking dependency order betwe=
en
> >>   rcu_state.barrier_mutex and hotplug lock and then just lock the barr=
ier_mutex
> >>   before calling rcu_nocb_rdp_deoffload() from our failure path.
> >>  =20
> >>
> >>2) On rcu_nocb_rdp_deoffload(), handle non-existing nocb_gp and/or nocb=
_cb
> >>   kthreads. Make sure we are holding nocb_gp_kthread_mutex.
>=20
> Sorry for my late reply,  Is the nocb_gp_kthread_mutex really necessary?
> Because the cpu online/offline is serial operation,  It is protected=20
> by  cpus_write_lock()
>
>And you're right! But some people are working on making cpu_up() able to w=
ork in parallel for faster bring-up on boot.

Thank you for  explanation. Are you making the above changes to this patch?

Thanks
Zqiang