From:   "Zhang, Qiang1" <qiang1.zhang@intel.com>
To:     Frederic Weisbecker <frederic@kernel.org>,
        Nicholas Piggin <npiggin@gmail.com>
CC:     "paulmck@kernel.org" <paulmck@kernel.org>,
        "quic_neeraju@quicinc.com" <quic_neeraju@quicinc.com>,
        "joel@joelfernandes.org" <joel@joelfernandes.org>,
        "Zhuo, Qiuxu" <qiuxu.zhuo@intel.com>,
        "rcu@vger.kernel.org" <rcu@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v2] rcu: Keeping rcu-related kthreads running on
 housekeeping CPUS
Thread-Topic: [PATCH v2] rcu: Keeping rcu-related kthreads running on
 housekeeping CPUS
Thread-Index: AQHZPHB1PeN6hngoGkqA2pE7E/WY3K7HVWEAgABP0ZA=
Date:   Fri, 10 Feb 2023 05:26:04 +0000
Message-ID: <PH0PR11MB58803E9D69F2FAC51C5E5589DADE9@PH0PR11MB5880.namprd11.prod.outlook.com>
References: <20230209102730.974465-1-qiang1.zhang@intel.com>
 <Y+WQkmiypKUUUfcj@lothringen>
In-Reply-To: <Y+WQkmiypKUUUfcj@lothringen>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?cuHTJuWVLplaGQ+YCVbOn3Fl9x3eBVfdodEn0+r6I6ZJklg3Q022IzDBjtbv?=
 =?us-ascii?Q?/UiayjjW7L92f4as/pBwnEpdlpvuJItlFskZ8fHAs2WER8fEIg4geNxhXx2L?=
 =?us-ascii?Q?mvC3n/vpcaXEbKxXyBT+gExtlVbGPx0bZbQDzrNSOA8C1yLr891sxOBumWWK?=
 =?us-ascii?Q?qJSmhI2ThwLRVSj6E00fTsU9APC3mQlRxI/uzZ2AEGrvoEuqRwibqBu6g0rc?=
 =?us-ascii?Q?NbZMuwuWkanMfMcaxjHePtIJYKInb85WhVTkaKPlhdb6BIWLmT76k9wjTP+i?=
 =?us-ascii?Q?z79kLYOBf81jB8mIXVWhsTR5HmPHZK5TVqoTpSoGg7HpTNb7zBpXHEH7qqiu?=
 =?us-ascii?Q?6NibNTjy6c5VvwwsupXV0mmDbu+O98cD+/YtfsyKFsjEiwUITPx1w0SzrBNU?=
 =?us-ascii?Q?0Fm7BaEU+EicrlScKneXkauE0/5852YLG2N6alKAuqINuKg2kRo31C9CIloj?=
 =?us-ascii?Q?tl9XVr43xC8ANifnEzj5cjV0SserROKYBFnt6eGbUlZhtQs7dHjM9VNTDK+1?=
 =?us-ascii?Q?Z9o59gURwNkMxmGTsw2By6ptGH9tc8UqLMr5dSQFcxXDt0+tfw6HW/kZ2JMU?=
 =?us-ascii?Q?0ziKDqTZBmAtcvhV3gHDkQAsrqZ5ehi9MagjIkPRVnI/nxtQK2oyAgos7X96?=
 =?us-ascii?Q?8zfUi+Lgmy56CRyv+k39KqaQR+jGEEo9V++TnBr5Ft7AvjWyXer5uyRiSZXU?=
 =?us-ascii?Q?OaG40p1T2YBZEoGshhdSLiJQ93iAm1abo9IpOZuCuCy9AVLxMBjLlxtaASvT?=
 =?us-ascii?Q?fQYEHFwf5wP6m05NFIpM3h+SU+GkfsqiM1qAI7EXGIgXqIAVq4ry7OFm6/+1?=
 =?us-ascii?Q?67++cL3+vzcCiSqo0gwMICCHv/l9wlx2ok+7Qz57bOVPKWYJ0H43egJp5Nej?=
 =?us-ascii?Q?Q2utrGXhnR8ca4v1hQujlIUVj6uydX/fBtGKHDG3AZplcXLWHCnIi7LfJwIY?=
 =?us-ascii?Q?c4noaevpdI3b1Kml61/O18BdExt7WFSDFXaPjkhluXXVnPYSinSTKO6zLGC0?=
 =?us-ascii?Q?2H5Hpe0h7faJxivE83bJeiKF5bEXj7smCsodcTlvxeGBHD69jCeBcR+xsgcL?=
 =?us-ascii?Q?FnwUOyisRiK5MXm5kPsKgu2ZOgQtg6smT0sU6KH0lTd/stJB8Wni9bzr4GhF?=
 =?us-ascii?Q?cfAqdvT8QhZSDrYggz6AOPoPlsKyi+tN6gmd/a5zLI7JwQ8e3sk7+5YdIZSp?=
 =?us-ascii?Q?QVFOMBTdPGWtRxOJaVQt7Wm4bIHoNOgft33kwyJbJslHBLVkyp6ckxQqOcdU?=
 =?us-ascii?Q?DHpICFICItn74ttWyDrMF0IVZ8e35jwUk0nGbD1DypHUlBVev3pbzHeOF8N3?=
 =?us-ascii?Q?yXpZtbljnD2GOyr/dIHw73TeL+jmBQI5DzlBfq99TEodltw5MI0hsM38aPIq?=
 =?us-ascii?Q?N/ce7nS4RhB4C/5T7VU+gfuLKWMQMGsFk3r78kpuVCqUkQUgPgvad8+g1Hdd?=
 =?us-ascii?Q?iIehqZPeVc/njR555Qitg8oqABFIgaqoc1xEp04t3j7yCA3Hc8pAgGN26aHl?=
 =?us-ascii?Q?wGndlLVIN86ae2FqfpHuxCr/g9/FiW3yMCcYziBfsllLDHkNLcNBivBs+K+2?=
 =?us-ascii?Q?rhKYbUdpTnsSJUDI2mt9esx1eW9ibsDuvH5C9nUT?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5880.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cccf9ee9-4c25-4ecb-2077-08db0b274ddb
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Feb 2023 05:26:04.0581
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: UcvFBeYWtBq01iMLAEX5m4k4aI7PPjgDlb9UeNVW70GxK44E4fd8GBnb5l8/4/el0tL0HDeLv0m/ZHrb+nPyZQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6514
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> For kernels built with CONFIG_NO_HZ_FULL=3Dy and CONFIG_RCU_NOCB_CPU=3Dy,
> when passing cpulist to "isolcpus=3D", "nohz_full=3D" and "rcu_nocbs=3D"
> bootparams, after system starting, the rcu-related kthreads(include
> rcu_gp, rcuog*, rcuop* kthreads etc) will running on housekeeping
> CPUs, but for cpulist contains CPU0, the result will deferent, these
> rcu-related kthreads will be restricted to running on CPU0.
>=20
> Although invoke kthread_create() to spwan rcu-related kthreads and
> when it's starting, invoke set_cpus_allowed_ptr() to allowed cpumaks
> is housekeeping_cpumask(HK_TYPE_KTHREAD), but due to these rcu-related
> kthreads are created before starting other CPUS, that is to say, at
> this time, only CPU0 is online, when these rcu-related kthreads running
> and set allowed cpumaks is housekeeping cpumask, if find that only CPU0
> is online and CPU0 exists in "isolcpus=3D", "nohz_full=3D" and "rcu_nocbs=
=3D"
> bootparams, invoke set_cpus_allowed_ptr() will return error.
>=20
> set_cpus_allowed_ptr()
>  ->__set_cpus_allowed_ptr()
>    ->__set_cpus_allowed_ptr_locked
>      {
>                 bool kthread =3D p->flags & PF_KTHREAD;
>                 ....
>                 if (kthread || is_migration_disabled(p))
>                         cpu_valid_mask =3D cpu_online_mask;
>                 ....
>                 dest_cpu =3D cpumask_any_and_distribute(cpu_valid_mask, c=
tx->new_mask);
>                 if (dest_cpu >=3D nr_cpu_ids) {
>                         ret =3D -EINVAL;
>                         goto out;
>                 }
>                 ....
>      }
>=20
> At this time, only CPU0 is set in the cpu_online_mask, the ctx->new_mask
> is housekeeping cpumask and not contains CPU0, this will result dest_cpu
> is illegal cpu value, the set_cpus_allowed_ptr() will return -EINVAL and
> failed to set housekeeping cpumask.
>=20
> This commit therefore add additional cpus_allowed_ptr() call in CPU hotpl=
ug
> path. and reset the CPU affinity of rcuboost, rcuog, rcuop kthreads after
> all other CPUs are online.
>=20
> Signed-off-by: Zqiang <qiang1.zhang@intel.com>
>
>Good catch! But based on that and your other fix, I suspect that
>nohz_full=3D0... has never been seriously used.
>
>A few points:
>
>* This is a problem for kthreads in general. And since HK_TYPE_KTHREAD =3D
>  HK_TYPE_RCU and both are going to be merged in the future, I think we sh=
ould
>  stop handling the RCU kthreads housekeeping affinity from RCU but let th=
e
>  kthread code handle that and also fix the issue from the kthread code.
>  RCU boost may be an exception since we try to enforce some node locality
>  within the housekeeping range. =20

Agree.  indeed, these works that set housekeeping CPU affinity should not b=
e handled by RCU,=20
and not only RCU-related kthreads are affected, other kthreads created earl=
ier also have the
same problem.

>
>* If CPU 0 is isolated and it is the boot CPU, we should wait for a second=
ary
>  CPU to boot before activating nohz_full at all. Unfortunately the nohz_f=
ull
>  code is not yet ready for runtime housekeeping cpumask change but work i=
s
>  in progress (I'm saying that for 10 years...)
>
>* I'm tempted to revert 08ae95f4fd3b (nohz_full: Allow the boot CPU to be
>  nohz_full) since it doesn't work and nobody ever complained?

Yes if remove 08ae95f4fd3b, this problem will disappear.

Thanks
Zqiang

>
>Thanks.