Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Zhang, Qiang1" <qiang1.zhang@intel.com>
To:     Frederic Weisbecker <frederic@kernel.org>
CC:     "paulmck@kernel.org" <paulmck@kernel.org>,
        "joel@joelfernandes.org" <joel@joelfernandes.org>,
        "rcu@vger.kernel.org" <rcu@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH] rcu: Avoid invalid wakeup for rcuc kthreads in
 RCU_KTHREAD_OFFCPU status
Thread-Topic: [PATCH] rcu: Avoid invalid wakeup for rcuc kthreads in
 RCU_KTHREAD_OFFCPU status
Thread-Index: AQHY+PQlxGZ/hW5SmUe9kA4qPJUqOq5BgTQAgAARpnCAABPbAIAAiFzw
Date:   Wed, 16 Nov 2022 23:23:35 +0000
Message-ID: <PH0PR11MB5880615D66ADF23A21AA9E04DA079@PH0PR11MB5880.namprd11.prod.outlook.com>
References: <20221115131926.3409974-1-qiang1.zhang@intel.com>
 <20221116124716.GA813995@lothringen>
 <PH0PR11MB5880FD7499114743BCE3E5A3DA079@PH0PR11MB5880.namprd11.prod.outlook.com>
 <20221116150130.GD816333@lothringen>
In-Reply-To: <20221116150130.GD816333@lothringen>
Accept-Language: en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-version: 11.6.500.17
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?jS/CGOBL9jLNCfXcEo6776gv5yjpikybabg79KFQ8l8dEmVRglbIetMjWvkc?=
 =?us-ascii?Q?O9W5Ube4FEtGC7Fn7tEKdp9r/BUnNfdcqoNk+fjq2mU8qPEXlNohliX9RKTS?=
 =?us-ascii?Q?NLGaWOUrwanezcYxZMN43+NDdUhJZcjky64pwLxwfHjiYyJMgEMLDMKSw2SK?=
 =?us-ascii?Q?J6QxdW/x2RJKJopLIAghPSlDF5iCjlE+2y0RxRSytzsarn5clGzAVILyaPiL?=
 =?us-ascii?Q?AE/UhJQ4r4Xzce2sIJo4cVfLmGglYRQQwjBJ+AB/FjNqWt2Rw6lax+eUpFx1?=
 =?us-ascii?Q?mtheo4csw0afjqCv8OO0YpD4zDFbSszQpXk+pxPBABQtDm7TXyXjhdTyHn0C?=
 =?us-ascii?Q?GuX/D03s0vE37piVQzg3eDDn5Vi03BDKBVc4WPFGKbgeTgKrecK1XBqyFjdi?=
 =?us-ascii?Q?oLdgDzEpIqCD2/bH3oLDuQBwJZUso+8BJr7mbhUKhJAa5UQP0W96QV+1KC33?=
 =?us-ascii?Q?0GTOtSytMdoiL1Ty6DaXj5OdK6vcJJ4tDQrYrCTqGDGbjYDRpEOav5NxZObU?=
 =?us-ascii?Q?UfRJqabq8bgjFtpwvPaFBnClbwklTsyJn8AqeZZckRTURTporOpMDDFTpKVe?=
 =?us-ascii?Q?sCHK5q52LM+QHbCTA582sxzqsW9qN6ljsudqNRLIEm+/RH3HcWFksru1xqXG?=
 =?us-ascii?Q?REN5EgYQWf1Hw1BfQE+CHR5kZ3VV6BcrgNffcqnlPuhhfboiDBV+z2GOrBkc?=
 =?us-ascii?Q?+zsL8jvSv/hfvaN3lPhZvDcE0ZrL7RJYtlZObhloS50ehuSZ6OVhsukxOcVZ?=
 =?us-ascii?Q?kK+QJt4lOWbZHDU+KkGuGGw8aeHfG2f5A8bkyMeADoNM8V1kPIjUFsZTClRz?=
 =?us-ascii?Q?CBdvmnn+m1DI4BWdLJsq+DE6mLS7jbjOiPYVZe4iDuTAGttJzqinUuLZySHp?=
 =?us-ascii?Q?Sc3LPY/9HRn4HxiCqpMG4xXfxQx8jjOlLcNwZRrg80Sukvu1KccFwHxs5CpS?=
 =?us-ascii?Q?wQu16k/kg8DqqSEEQuiGvejQleCC83otyhRIwCiLO4Xl1pfZtRVZQLgxNBDX?=
 =?us-ascii?Q?8cN51FZkbjH21SJFPFt9pTo2nYt5ID3xkTPMF82/FT3ROxTDucLzzCdCtzZt?=
 =?us-ascii?Q?X7AWPnr7dylDsFmpFaycjGle6ycLC7bbIZeCzHbedowYu52FfMhuibRqujki?=
 =?us-ascii?Q?/LLgd0I91NvUJnmBLaJlkJZPkKPS7QjS7w3YU+BWTb7nSiSzgF7QXRPequXu?=
 =?us-ascii?Q?43nzf4bk5zrCz2IDTH9xFXMam45ih9rW0LK5TgJn3/blClooBOLGceimQpvl?=
 =?us-ascii?Q?lljafHcEtxfs3DWXnXm4ws7qT/kANb2tcgmX7uysU8gtXeVR6kOrKJA7Q8qr?=
 =?us-ascii?Q?f5Ca8U4UYpo0fuZ1qz1CCrsRGQkQ9GzmZsIzWCMidBz++Dk9Ppmsr19V6pD1?=
 =?us-ascii?Q?SiLEpsR702T2xxcB/vgsoRo0jCSdBfPWQlnB9qvWtP5tekZJsw/LJq4HNQd3?=
 =?us-ascii?Q?Ejd1lfecjcx5b7YByMQPOAcmt4kIqgum+ZlI/X5+Y4kAvRGkqnSj+iTDalv1?=
 =?us-ascii?Q?5z49Qy5OLbCuI50az/GXXX+ag33ZBVQPPGTvvSralTgTrHd6xg+FhRbFVcLQ?=
 =?us-ascii?Q?B1jj8/62+IOfxeKrP6P/Ip1XFNfYUUgHnj9r42dF?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5880.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 241f15e2-5293-44a4-6885-08dac8299590
X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Nov 2022 23:23:35.4361
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: gh46gL2XXYKAQOVvz7x1YmVspBR3HjS9BLT45RociR7HgDLOk2wW5F1DTefGAaCXxeugjdbTurwpsPZD84zBZg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR11MB4714
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Nov 16, 2022 at 02:07:28PM +0000, Zhang, Qiang1 wrote:
> On Tue, Nov 15, 2022 at 09:19:26PM +0800, Zqiang wrote:
> >And more important! On unpark time RCU_KTHREAD_OFFCPU isn't cleared. Onl=
y the
> >rcuc kthread does it, and after your patch it couldn't be awaken to perf=
orm
> >that, unless rcuc is lucky enough to have rcu_data.rcu_cpu_has_work =3D =
1
> >by the time it unparks and that isn't guaranteed. So rcuc may sleep fore=
ver.
>=20
> Thanks for review, yes I should register an unpark function to clear RCU_=
KTHREAD_OFFCPU.
> Is the following modification more appropriate?
>=20
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 3ccad468887e..a2248af0ccda 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -2375,7 +2375,8 @@ static void rcu_wake_cond(struct task_struct *t, in=
t status)
>          * If the thread is yielding, only wake it when this
>          * is invoked from idle
>          */
> -       if (t && (status !=3D RCU_KTHREAD_YIELDING || is_idle_task(curren=
t)))
> +       if (t && (status !=3D RCU_KTHREAD_YIELDING || is_idle_task(curren=
t)) &&
> +                               status !=3D RCU_KTHREAD_OFFCPU)
>                 wake_up_process(t);
>  }
>=20
> @@ -2407,7 +2408,14 @@ static void invoke_rcu_core(void)
>=20
>  static void rcu_cpu_kthread_park(unsigned int cpu)
>  {
> -       per_cpu(rcu_data.rcu_cpu_kthread_status, cpu) =3D RCU_KTHREAD_OFF=
CPU;
> +       WARN_ON_ONCE(cpu !=3D smp_processor_id());
> +       __this_cpu_write(rcu_data.rcu_cpu_kthread_status, RCU_KTHREAD_OFF=
CPU);
> +}
> +
> +static void rcu_cpu_kthread_unpark(unsigned int cpu)
> +{
> +       WARN_ON_ONCE(cpu !=3D smp_processor_id());
> +       __this_cpu_write(rcu_data.rcu_cpu_kthread_status, RCU_KTHREAD_ONC=
PU);
>  }
>=20
>  static int rcu_cpu_kthread_should_run(unsigned int cpu)
> @@ -2460,6 +2468,7 @@ static struct smp_hotplug_thread rcu_cpu_thread_spe=
c =3D {
>         .thread_comm            =3D "rcuc/%u",
>         .setup                  =3D rcu_cpu_kthread_setup,
>         .park                   =3D rcu_cpu_kthread_park,
> +       .unpark                 =3D rcu_cpu_kthread_unpark,

>Well, personally I don't think it's worth the burden because wake_up_proce=
ss()
>already does an early exit if it's not dealing with a TASK_[UN]INTERRUPTIB=
LE task and
>the window is so short and rare that it doesn't look like a good candidate=
 for
>extra optimization;

Agree,  in try_to_wake_up(), acquire rcuc task->pi_lock(there's almost no l=
ock contention here)
and then check tasks state,  for TASK_[UN]INTERRUPTIBLE task,  not dealing =
and early exit.

Thanks
Zqiang

>
>Thanks.