Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
From:   "Zhang, Qiang1" <qiang1.zhang@intel.com>
To:     "paulmck@kernel.org" <paulmck@kernel.org>
CC:     "frederic@kernel.org" <frederic@kernel.org>,
        "rcu@vger.kernel.org" <rcu@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH] rcu: Add rnp->cbovldmask check in
 rcutree_migrate_callbacks()
Thread-Topic: [PATCH] rcu: Add rnp->cbovldmask check in
 rcutree_migrate_callbacks()
Thread-Index: AQHYYJkonORSfrYVw02H56cLgBUTL60Qq84AgABKvvA=
Date:   Fri, 6 May 2022 00:40:09 +0000
Message-ID: <PH0PR11MB5880A25078A7D4DBAB1267C2DAC59@PH0PR11MB5880.namprd11.prod.outlook.com>
References: <20220505155236.1559619-1-qiang1.zhang@intel.com>
 <20220505192956.GX1790663@paulmck-ThinkPad-P17-Gen-1>
In-Reply-To: <20220505192956.GX1790663@paulmck-ThinkPad-P17-Gen-1>
Accept-Language: en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-version: 11.6.401.20
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?+6R2oq1tT1JT2GrP2ho0JYJ1eqaTvrOZG1SpwBJ1dY4tHdUUjLNOEXevJsNr?=
 =?us-ascii?Q?ChJoWJIvfoOqCej3+pLiButczOIz0fk+wEOS8kzBqEHRal0BY7/mDH98pq0u?=
 =?us-ascii?Q?wS0tOrOFdeW6pMox6YfVJh/AUMijMViQmZeiJbRR9yttmMJqS0D/apfuYKOU?=
 =?us-ascii?Q?2SMi2YTXDlhJOjmY7ruieqnzURSgucVCwDx6vqLkzHcUOU7zqKpd5MAGyOcB?=
 =?us-ascii?Q?mXEip9kedRKmWrfsTuC8t0J/HfF+/6hixhSRSHFXkibFYAvHn2HD64DUWOFm?=
 =?us-ascii?Q?gvivEXZB6syqIExiZ289fBZQEp1NObpHpLSCbaiflUrndb5ZrG3YPGe2bpyD?=
 =?us-ascii?Q?YoLN+BVBAdSwUEWfOqb7l1HTdG5UFSMQ4HbcmhZIHw77IDh+bNt1bqkXN3yo?=
 =?us-ascii?Q?wqnJBLgE0xXxed5+Y3e0MQa1Sq7Qt+JaV1OTxgf/+5Dif5pL1RhQ61znqRzK?=
 =?us-ascii?Q?Cz7JhsHD3f73g/zEoxGL16RqtrEMB7CSlaQJePNX22btcnN2uQeDzHXEAjBw?=
 =?us-ascii?Q?ToXx1UPd9r0ME4Y1oekkoOw4KiV1X5R2+NZeJZPN/Edv0PuTullIIg/ODNmg?=
 =?us-ascii?Q?s08cZw9tX8iafgl4bzi6dEmdS7QMJVeMZkG2C8emcK90nJXhUj90gMOOQZgg?=
 =?us-ascii?Q?ADCThzbI06ggMmsJG3+fN5CMqFeucFR+ZSKq/ThWt46w7q9gG2Mp1kj5xfE3?=
 =?us-ascii?Q?v4wpGAymEKU7tI5Luoxq1JgkH5KfcybNailn6eg5MM7eaLZilzrHfRCyt/Xg?=
 =?us-ascii?Q?NhmaV6a4HW7Ry3dSLhtfeQwhqwfctdlHbzn6qtT4v//KyLdnsh7Bs56SADRT?=
 =?us-ascii?Q?swKrsBdF1u2zPy7Msrwzm5Ui3uOufnf8oSrQ5tSnx8lpf2/8IfSqH/pYoHcM?=
 =?us-ascii?Q?EPAs9lrsSL8OJfqW/Hq0RiEy5e4APTXJIhaACSNSw4X6JqbCfsbJBo8SzVy5?=
 =?us-ascii?Q?5VACZqwTQeorjhyipDBfk3tMmYc7Vn/gj4shuaTA5mU6bBUMzNoR/dN9pmfj?=
 =?us-ascii?Q?7l9x+NoFtDzjhV82LBXZTWDIhMJYzehxJJ/13UIn1dH8vUOvaP84PTk8zwy+?=
 =?us-ascii?Q?+Xd0+1LVNoXreV2T5tX8nw4g9zPhZpSNlmN4qozyy08EvwpjuKhjTcYuf0Dy?=
 =?us-ascii?Q?21fC5tDYFsRO8RLBDwkVoUQ6A9N/yS/VXuSQQ/+06hzwZkOpEbq9KQxZAefh?=
 =?us-ascii?Q?pNnD1bDXsb530kb0wmp0ItCKMca0K/jhWQdn4Kvybl7wzepDF37cd0MSBSWp?=
 =?us-ascii?Q?lNki6N4qgKxEkhaJ5CaDhyUXD1A0MHpHzBGbVgQcwWbTEya4mBDV6Y1CyGD3?=
 =?us-ascii?Q?iMsNUGNy1quKo90UCzrRRUKKENV3MbA2lV4+pRV0ADmyCXnN7Cxn0KxcfPUE?=
 =?us-ascii?Q?2fGL5wOwq1SBK0IF6h7NJ7TxhxeiXEbSGsNOWJIpCb45mkRnzt36Hr4A0dss?=
 =?us-ascii?Q?Uc87+CkSqhG1eIW9p/vJsQyWeszLT52TohtLUUOFMLhLxC5oYgBcabq000TA?=
 =?us-ascii?Q?hgFf/RaKesDOaWQfNo3bu1YopxMWCC/s5TTF8Jp5cRcHtN/rs1r6u6imUmwk?=
 =?us-ascii?Q?EnmCvQZlijIP+UDhCiWPfL75SieBU7R+L95XE1gY3Mj8opURKP0dPk4CTIy6?=
 =?us-ascii?Q?7WZvl4tUs5gOWp/FOgCfaoJwr/giqf27YHtLu9PqCHH512eu14lzj+oSkN6P?=
 =?us-ascii?Q?zo2MJsPhUNxTKAbVR/TrTYdXhLOaI7DWA5JUEM70BCdNx6cMtLl7bSKZ6cV6?=
 =?us-ascii?Q?JwSyRatRfQ=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5880.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fda34cdf-84f5-4958-0512-08da2ef8f98b
X-MS-Exchange-CrossTenant-originalarrivaltime: 06 May 2022 00:40:09.9282
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: l3OIc7K1TEOaOiaPDaH6RYqMG7HfujkmH5JrzmTEOYO9QTSQ6zgKsAQDeKZKiThHHBq9S+GOSTkfjfvfMbHWRA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1330
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, May 05, 2022 at 11:52:36PM +0800, Zqiang wrote:
> Currently, the rnp's cbovlmask is set in call_rcu(). when CPU going=20
> offline, the outgoing CPU's callbacks is migrated to target CPU, the=20
> number of callbacks on the my_rdp may be overloaded, if overload and=20
> there is no call_rcu() call on target CPU for a long time, the rnp's=20
> cbovldmask is not set in time. in order to fix this situation, add
> check_cb_ovld_locked() in rcutree_migrate_callbacks() to help CPU more=20
> quickly reach quiescent states.
>=20
> Signed-off-by: Zqiang <qiang1.zhang@intel.com>

>Doesn't this get set right at the end of the current grace period?
>Given that there is a callback overload, there should be a grace period in=
 progress.
>
>See this code in rcu_gp_cleanup():
>
>		if (rcu_is_leaf_node(rnp))
>			for_each_leaf_node_cpu_mask(rnp, cpu, rnp->cbovldmask) {
>				rdp =3D per_cpu_ptr(&rcu_data, cpu);
>				check_cb_ovld_locked(rdp, rnp);
>			}
>
>So what am I missing here?  Or are you planning to remove the above code?

We only checked the overloaded rdp at the end of current grace period, for
my_rdp overloaded cause by migration callbacks to it,  if the my_rdp overlo=
aded,
and the my_rdp->mynode 's cbovldmask  is empty,  the my_rdp overloaded may =
be
not checked at end of the current grace period.

I have another question, why don't we call check_cb_ovld_locked() when rdp'=
s n_cbs decreases.
for example call check_cb_ovld_locked() in rcu_do_bacth(), not at the end o=
f grace period.

>
>If so, wouldn't you also need to clear the indication for the CPU that is =
going offline, being careful to handle the case where the two CPUs have dif=
ferent leaf rcu_node structures?

Yes the offline CPU need to clear.

Thanks,
Zqiang

>
>							Thanx, Paul
>
> ---
>  kernel/rcu/tree.c | 1 +
>  1 file changed, 1 insertion(+)
>=20
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index=20
> 9dc4c4e82db6..bcc5876c9753 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -4577,6 +4577,7 @@ void rcutree_migrate_callbacks(int cpu)
>  	needwake =3D needwake || rcu_advance_cbs(my_rnp, my_rdp);
>  	rcu_segcblist_disable(&rdp->cblist);
>  	WARN_ON_ONCE(rcu_segcblist_empty(&my_rdp->cblist) !=3D=20
> !rcu_segcblist_n_cbs(&my_rdp->cblist));
> +	check_cb_ovld_locked(my_rdp, my_rnp);
>  	if (rcu_rdp_is_offloaded(my_rdp)) {
>  		raw_spin_unlock_rcu_node(my_rnp); /* irqs remain disabled. */
>  		__call_rcu_nocb_wake(my_rdp, true, flags);
> --
> 2.25.1
>=20