From:   "Zhang, Qiang1" <qiang1.zhang@intel.com>
To:     "Zhuo, Qiuxu" <qiuxu.zhuo@intel.com>,
        "paulmck@kernel.org" <paulmck@kernel.org>,
        "frederic@kernel.org" <frederic@kernel.org>,
        "quic_neeraju@quicinc.com" <quic_neeraju@quicinc.com>,
        "joel@joelfernandes.org" <joel@joelfernandes.org>
CC:     "rcu@vger.kernel.org" <rcu@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH] rcu-tasks: Directly invoke rcuwait_wake_up() in
 call_rcu_tasks_generic()
Thread-Topic: [PATCH] rcu-tasks: Directly invoke rcuwait_wake_up() in
 call_rcu_tasks_generic()
Thread-Index: AQHZR0+ogcttGObez02wrCHslcltHq7cNR4AgAAAcHA=
Date:   Thu, 23 Feb 2023 08:43:05 +0000
Message-ID: <PH0PR11MB588084641FFB675A102BA503DAAB9@PH0PR11MB5880.namprd11.prod.outlook.com>
References: <20230223063022.2592212-1-qiang1.zhang@intel.com>
 <IA1PR11MB6171F5F6B525B6C599024C9D89AB9@IA1PR11MB6171.namprd11.prod.outlook.com>
In-Reply-To: <IA1PR11MB6171F5F6B525B6C599024C9D89AB9@IA1PR11MB6171.namprd11.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?mcMYW1l21sTIelSHlF9yk4Zlo3s3poMF3m5cx7PemWuUNvx9/ahhChq4hJ5m?=
 =?us-ascii?Q?tJZHgGZuu5XktClTZmdV579e6CABHO4XdmmKHkFRHGEiHerd/Qvwq7kVqLId?=
 =?us-ascii?Q?t8Ymv8QsjUsJCMnoarim1cNpZdnMnE+6EbbWPWGZ/aeZOYbr0rESag40lEca?=
 =?us-ascii?Q?PVFwTHmGJs2gVywmdl8hOQlOl8l3w9c+9inEFFM+ftk9o8tc892d/99Z+D4S?=
 =?us-ascii?Q?zyNuFAYlyHJ4W+htFGJh5m9LzFiAziHYaMBDdR0Rdj8E1eVj6Pr434quFg2a?=
 =?us-ascii?Q?a7purh5y+jV3GQEKV0ETGjKEKd5nsCPIQeRLD/mC/NiLb6Z45/v+ye+fQptV?=
 =?us-ascii?Q?mIr7xcPiMdYVoAEG0qnWdfnPi+LZQnw2BAOs6o5MfO0wHrijyl5U+h+STM6W?=
 =?us-ascii?Q?rBUXOZo9VBYSof6vYSPAW0sVOLsR8Ds1aDmbjI/0Txa9cTvJ7wENyFO65XB9?=
 =?us-ascii?Q?31BNXOD5fbJjeVdsfiiasCGhj96Bn7hR76NAtO2h7Yf7as4vvMMM8kRZDWrb?=
 =?us-ascii?Q?IfKcaRpcXPk8+nH1qmwOlYGvqEGkzQ0HwyvQ9gFkc0zo5RlupjvrS0GLMAZm?=
 =?us-ascii?Q?hNKg/WAN4OOTZEpz2c4EaHcEVlQTLy1aD2dPvakUdZrgtZqZvbtMo/igI7UN?=
 =?us-ascii?Q?OBo1kqIf28C6Ds/XmXw+fc0Z2cuiplgj1IK+K6ePbC92Dl66MFTUQKL0bELn?=
 =?us-ascii?Q?g8tYDs6Xr9QkiL2eKF/rYJak5PpdjdLFE11ys3RKfE81ZK2kWO5l+F9X/e3/?=
 =?us-ascii?Q?3rBvpcxttzEK/jSUq0rWw8WvGvK3+gy9Z8BORH40sWaB2fIipEDN8D0Vrrre?=
 =?us-ascii?Q?2n6WcE106AucGdskTDvpgEwK3sUqFBdYWrqvR2TmYUHrB8hXAT12uUkV3YN1?=
 =?us-ascii?Q?QeduCKCpER35X4rGATaj+lc48kPo7GnjmLUKbYL77J3bEy/kkBbz7xZuvhLf?=
 =?us-ascii?Q?ufqZ6qAyCoH8P7xJCzK339S/JDw2L0C9WW/11eE0Zgh/VEsvFImZfn7d/FhX?=
 =?us-ascii?Q?UiyMHHEaZXDe/ibtFozz9HwiiwwpeRqXMYMI7I8HjVlRbdtIduvCZFR6QfRZ?=
 =?us-ascii?Q?7LjvXmgoQqKOegawmoJ0ITR0/DADgAEMtSIEASM0BQ5YdWXJHIVTkA5lpU6V?=
 =?us-ascii?Q?Wx21xzE6ywEbp/E3Hv4SxBkXqLiz9b77rLm6H4JIF1AitMFm7HKl+BISDT0h?=
 =?us-ascii?Q?he0+Zi/Y7c3iLLTh758NidDRrLJty182IKzDGh1XbaA2ozE1rsd/CLiC+rqs?=
 =?us-ascii?Q?MBE/sae6caMDiuH1Qi7hkbGluLFGszgH1ilyRgTb/rswaGWGXX6fDB4JMGmO?=
 =?us-ascii?Q?frnnUkDBTKZ5pkymAlYphNa/ptC6r1wZhuSiUrwrBdsTTg3MqmfbVrne/Pzi?=
 =?us-ascii?Q?BHp5N2mauRIas2AGStbg5LjGCORCRLbplsgO/shMC4HLX0o4xA8l+sDaCTO/?=
 =?us-ascii?Q?rZRqifwCvOh2DkeWlW2ISD6s37d798dj+XY8t5Vo4R/Scl45AmxTgzRaFSPa?=
 =?us-ascii?Q?flU4Hc/sJTXWbin/CGoZli4Jz+NpAORjNplt/TXCPp0EbEvqqyBKsBccMlgT?=
 =?us-ascii?Q?eECt3TACzpT8eH03kXbFIQ/r4o0cGgRNi0NV3dsx?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5880.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 12ffedb5-5221-41f7-ce6e-08db1579fb27
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Feb 2023 08:43:05.1504
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: z/jFtrVoDmB33Hak8k/Pde13iHahlzHNaluLzWX/d0R5gmym17b4tcdKixC9nH0r124cW1CF2/DSPTyLeJ10BA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR11MB5687
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Zqiang <qiang1.zhang@intel.com>
> Sent: Thursday, February 23, 2023 2:30 PM
> To: paulmck@kernel.org; frederic@kernel.org; quic_neeraju@quicinc.com;
> joel@joelfernandes.org
> Cc: rcu@vger.kernel.org; linux-kernel@vger.kernel.org
> Subject: [PATCH] rcu-tasks: Directly invoke rcuwait_wake_up() in
> call_rcu_tasks_generic()
>=20
> According to commit '3063b33a347c ("Avoid raw-spinlocked wakeups from
> call_rcu_tasks_generic()")', the grace-period kthread is delayed to wakeu=
p
> using irq_work_queue() is because if the caller of
> call_rcu_tasks_generic() holds a raw spinlock, when the kernel is built w=
ith
> CONFIG_PROVE_RAW_LOCK_NESTING=3Dy, due to a spinlock will be hold in
> wake_up(), so the lockdep splats will happen. but now using
> rcuwait_wake_up() to wakeup grace-period kthread instead of wake_up(), in
> rcuwait_wake_up() no spinlock will be acquired, so this commit remove usi=
ng
>
>There are still spinlock-acquisition and spinlock-release invocations with=
in the call path from rcuwait_wake_up().
>
>rcuwait_wake_up() -> wake_up_process() -> try_to_wake_up(), then:
>
>    raw_spin_lock_irqsave()
>    ...
>    raw_spin_unlock_irqrestore

Yes, but this is raw_spinlock acquisition and release(note: spinlock will c=
onvert to
sleepable lock in Preempt-RT kernel, but raw spinlock is not change).

acquire raw_spinlock -> acquire spinlock  will trigger lockdep warning.

Thanks
Zqiang

>
>
> irq_work_queue(), invoke rcuwait_wake_up() directly in
> call_rcu_tasks_generic().
>=20
> Signed-off-by: Zqiang <qiang1.zhang@intel.com>
> ---
>  kernel/rcu/tasks.h | 16 +---------------
>  1 file changed, 1 insertion(+), 15 deletions(-)
>=20
> diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index
> baf7ec178155..757b8c6da1ad 100644
> --- a/kernel/rcu/tasks.h
> +++ b/kernel/rcu/tasks.h
> @@ -39,7 +39,6 @@ struct rcu_tasks_percpu {
>  	unsigned long rtp_jiffies;
>  	unsigned long rtp_n_lock_retries;
>  	struct work_struct rtp_work;
> -	struct irq_work rtp_irq_work;
>  	struct rcu_head barrier_q_head;
>  	struct list_head rtp_blkd_tasks;
>  	int cpu;
> @@ -112,12 +111,9 @@ struct rcu_tasks {
>  	char *kname;
>  };
>=20
> -static void call_rcu_tasks_iw_wakeup(struct irq_work *iwp);
> -
>  #define DEFINE_RCU_TASKS(rt_name, gp, call, n)
> 			\
>  static DEFINE_PER_CPU(struct rcu_tasks_percpu, rt_name ## __percpu) =3D =
{
> 			\
>  	.lock =3D __RAW_SPIN_LOCK_UNLOCKED(rt_name ##
> __percpu.cbs_pcpu_lock),		\
> -	.rtp_irq_work =3D IRQ_WORK_INIT_HARD(call_rcu_tasks_iw_wakeup),
> 			\
>  };
> 		\
>  static struct rcu_tasks rt_name =3D
> 		\
>  {
> 		\
> @@ -273,16 +269,6 @@ static void cblist_init_generic(struct rcu_tasks *rt=
p)
>  	pr_info("%s: Setting shift to %d and lim to %d.\n", __func__,
> data_race(rtp->percpu_enqueue_shift), data_race(rtp-
> >percpu_enqueue_lim));
>  }
>=20
> -// IRQ-work handler that does deferred wakeup for call_rcu_tasks_generic=
().
> -static void call_rcu_tasks_iw_wakeup(struct irq_work *iwp) -{
> -	struct rcu_tasks *rtp;
> -	struct rcu_tasks_percpu *rtpcp =3D container_of(iwp, struct
> rcu_tasks_percpu, rtp_irq_work);
> -
> -	rtp =3D rtpcp->rtpp;
> -	rcuwait_wake_up(&rtp->cbs_wait);
> -}
> -
>  // Enqueue a callback for the specified flavor of Tasks RCU.
>  static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t =
func,
>  				   struct rcu_tasks *rtp)
> @@ -334,7 +320,7 @@ static void call_rcu_tasks_generic(struct rcu_head
> *rhp, rcu_callback_t func,
>  	rcu_read_unlock();
>  	/* We can't create the thread unless interrupts are enabled. */
>  	if (needwake && READ_ONCE(rtp->kthread_ptr))
> -		irq_work_queue(&rtpcp->rtp_irq_work);
> +		rcuwait_wake_up(&rtp->cbs_wait);
>  }
>=20
>  // RCU callback function for rcu_barrier_tasks_generic().
> --
> 2.25.1