Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   "Zhang, Qiang1" <qiang1.zhang@intel.com>
To:     "paulmck@kernel.org" <paulmck@kernel.org>
CC:     "frederic@kernel.org" <frederic@kernel.org>,
        "rcu@vger.kernel.org" <rcu@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH] rcu: Put the irq work into hard interrupt context for
 execution
Thread-Topic: [PATCH] rcu: Put the irq work into hard interrupt context for
 execution
Thread-Index: AQHYQ/uAOn2CUwsFkEWWhrezlLtpoqzYXhIAgAAlweA=
Date:   Wed, 30 Mar 2022 22:47:05 +0000
Message-ID: <PH0PR11MB58802C1246C6F4FB895BDDA9DA1F9@PH0PR11MB5880.namprd11.prod.outlook.com>
References: <20220330060012.2470054-1-qiang1.zhang@intel.com>
 <20220330201620.GM4285@paulmck-ThinkPad-P17-Gen-1>
In-Reply-To: <20220330201620.GM4285@paulmck-ThinkPad-P17-Gen-1>
Accept-Language: en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-version: 11.6.401.20
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?WmqkVcv8uejcpQtbApFlTEbhwx+RehcHjG2lcs1r+fokejaFCoKTDyK9tf4w?=
 =?us-ascii?Q?wmSum7/Lvn78oQHf3epch6EtfOKPCUatL3c5+qKgesbgOLJ8Us3Ah3YTWTeI?=
 =?us-ascii?Q?aqIN3ZaVtjx+nrULiyaVevIUaXHPKpS30AqW4JMWoMpQUurfbDd03esiLmvb?=
 =?us-ascii?Q?Gk1KM/BIB7EtbHdhLgacQSlofDuYTM77Y+El0IgxguNJGyNd4osyjGwGxD1b?=
 =?us-ascii?Q?lY8DUzDEm4UdZGedJtU2BFBxi7d9KA9Az0V8/Z2IsBDr2zkw9JkwZBK0pR5q?=
 =?us-ascii?Q?+nsyp1SBWC1Ql+tEOq0pN1a2UCh2w8IJ+73IInSWDlpjg+5FEO1WIdgqBa1b?=
 =?us-ascii?Q?T+j51lu+sND5eMTZDlvLNH5Q/WXwAze9PuzlsmWRNUxlwLw/6NqQyQrbNCV/?=
 =?us-ascii?Q?OhGxa2d4FougX8v4iD1AI7gEt9CQaeflfrNbZcTXp001grBOS5bfryo3oFXB?=
 =?us-ascii?Q?EkngKbBmTAw4NE3IGCPwP8ao6T5OAZZKwsnMLUckVpOr+IHKypJJ8qJUVDrP?=
 =?us-ascii?Q?QO8gz0ElgB+vByS6Eb5lVkrgoDkkB1mt3nhoZ/+z+EeT8TbUuMbQp8SSMvsQ?=
 =?us-ascii?Q?JQ1iGug5GGpmWzafnnbjWY7l9OR64rpYngP0uFcrZF4qh6GGOjHEHJ0NZogp?=
 =?us-ascii?Q?Nl7Ewj+4rJBbmuumeT1W/6xVRF8Y5ENhvYuJ7VSGkgWkq9Y+gMk+E8UTearv?=
 =?us-ascii?Q?WFN49O28Ewks3bB1DAvMbMKvW8rZ9dj+L2zDvaDrqGBaurM8bNmT6JT39BRC?=
 =?us-ascii?Q?vu/7N5eEHGuROwTws8IJXJl92o9z6RCP/uvRQ9WDEa6dwPt/bmTdION23EIk?=
 =?us-ascii?Q?Ci6nqDjks2WaYQpTO454TCuLHDYgcru/fbI8n0Em+Dh4DzbtyRorCgqOr0OJ?=
 =?us-ascii?Q?cuUybQh9cK9Lrn9nJA7nHMCKNvnyMBCz+rZSXLOGacv23qrpcrOAkyRRIgAb?=
 =?us-ascii?Q?LgK4PllWYyxgwcxq+ZcI8pR49KFPM2kIWcWGlSbAUtrjeSzHZthidnL7K6me?=
 =?us-ascii?Q?UWYshIw2xb2zityfiNOTiayxMslITo3xwBZlewgw2rhYw1D6TNfuq3CQlO0Y?=
 =?us-ascii?Q?u/UN8zkxQOg/dI+FTutNfJPQSzXEqiMigilIbE3HP8dgAg1SJhdafQCg2pfZ?=
 =?us-ascii?Q?3OWHnp/RswaiCvE8e5NOwVDQFLPx4wtczk813vW/MO1SlOcztpZTyJPa3Pq3?=
 =?us-ascii?Q?29g75M7hvaqbifrUXEzrdCRuDTVPvDQEJv0Xb/KHosxCgRr73OKqEXMx+3wZ?=
 =?us-ascii?Q?B14U1uSzGkHNzv//tAptZWb3AluJbw4woaLBep0XlBBpEou5RnqW/q4E+M+I?=
 =?us-ascii?Q?kagOfBHK6MaQt3IOtfDDFH/SzqLm+fST/qvFVEO1IF7raksBbDj+d9PetPPl?=
 =?us-ascii?Q?OGbGKGc5WXnpIhHVOTAVZg6abqT6W5Dl422Zj4Lc3jhFyt7gSzPavT0O2FKd?=
 =?us-ascii?Q?sSRN+X6Gc5VG2H1ZoyO0Xfl6fAnV+evpPwLenYCkxaiTjRoMjOaivDc2Efrg?=
 =?us-ascii?Q?9Zyim3dL7og9Z9iSz0jcsA/sCb/PcG8oyr8tkElEWQrJXRUoTEnIEmRONRaw?=
 =?us-ascii?Q?sTVyVO/WuvNQ8AwdxeRB8nMtc/o0xFwHuTfh1WLTEivz9RzWhjtL4vudFGvz?=
 =?us-ascii?Q?E0YLjGufpwOoEuUDqo6NtX0XAXhR0QgFTbzS4jmdZ6R2Y6d59Ki3OKKj78ov?=
 =?us-ascii?Q?F6hv1Z7oo6K2MpHgY4qH0Sgbnh8OPOEnhqvjMYsjFk6PaF0NQYkThvJN/wPu?=
 =?us-ascii?Q?kG1ckPJJNQ=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5880.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6f6bf35b-d63d-4e4b-bc94-08da129f36cb
X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Mar 2022 22:47:05.3987
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: zH7VRhRwJV+BqNSbw9Ryg01zRm3rt+GHFzdL0OwFFZwIdxzJw32YYmDa+XLTB7nuY3Z8j/hZMRVP2TKIMz88mw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB3431
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Mar 30, 2022 at 02:00:12PM +0800, Zqiang wrote:
> In PREEMPT_RT kernel, if irq work flags is not set, it will be=20
> executed in per-CPU irq_work kthreads. set IRQ_WORK_HARD_IRQ flags to=20
> irq work, put it in the context of hard interrupt execution,=20
> accelerate scheduler to re-evaluate.
>=20
> Signed-off-by: Zqiang <qiang1.zhang@intel.com>
> ---
>  kernel/rcu/tree.c        | 2 +-
>  kernel/rcu/tree_plugin.h | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>=20
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index=20
> e2ffbeceba69..a69587773a85 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -678,7 +678,7 @@ static void late_wakeup_func(struct irq_work=20
> *work)  }
> =20
>  static DEFINE_PER_CPU(struct irq_work, late_wakeup_work) =3D
> -	IRQ_WORK_INIT(late_wakeup_func);
> +	IRQ_WORK_INIT_HARD(late_wakeup_func);

>This is used only by rcu_irq_work_resched(), which is invoked only by rcu_=
user_enter(), which is never invoked until userspace is enabled, by which t=
ime all of the various kthreads will have been spawned, correct?
>
>Either way, please show me the exact sequence of events that lead to a pro=
blem with the current IRQ_WORK_INIT().
>
>  /*
>   * If either:
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index=20
> 3037c2536e1f..cf7bd28af8ef 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -661,7 +661,7 @@ static void rcu_read_unlock_special(struct task_struc=
t *t)
>  			    expboost && !rdp->defer_qs_iw_pending && cpu_online(rdp->cpu)) {
>  				// Get scheduler to re-evaluate and call hooks.
>  				// If !IRQ_WORK, FQS scan will eventually IPI.
> -				init_irq_work(&rdp->defer_qs_iw, rcu_preempt_deferred_qs_handler);
> +				rdp->defer_qs_iw =3D=20
> +IRQ_WORK_INIT_HARD(rcu_preempt_deferred_qs_handler);
>  				rdp->defer_qs_iw_pending =3D true;
>  				irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);
>  			}
>
>OK, in theory, rcu_read_unlock() could get to this point before all of the=
 various kthreads were spawned.  In practice, the next time that the boot C=
PU went idle, the end of the quiescent state would be noticed.

Through my understanding, use irq_work in order to make the quiescent state=
 be noticed earlier,
Because the irq_work execute in interrupt, this irq_work can be executed in=
 time, but In RT kernel
The irq_work  is put into the kthread for execution, when it is executed, i=
t is affected by the scheduling delay.
Is there anything I missed?

Thanks
Zqiang=09

>
>Or has this been failing in some other manner?  If so, please let me know =
the exact sequence of events.
>
>							Thanx, Paul