Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Zhang, Qiang1" <qiang1.zhang@intel.com>
To:     "paulmck@kernel.org" <paulmck@kernel.org>
CC:     "frederic@kernel.org" <frederic@kernel.org>,
        "joel@joelfernandes.org" <joel@joelfernandes.org>,
        "rcu@vger.kernel.org" <rcu@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "qiang.zhang1211@gmail.com" <qiang.zhang1211@gmail.com>
Subject: RE: [PATCH] srcu: Fix flush sup work warning in cleanup_srcu_struct()
Thread-Topic: [PATCH] srcu: Fix flush sup work warning in
 cleanup_srcu_struct()
Thread-Index: AQHZW8x3wcHPlwuNM0Wizzk5AGbgwq8FVToAgAClkaCAADGigIAACIWAgACwsYCAAHV/oA==
Date:   Wed, 22 Mar 2023 22:08:54 +0000
Message-ID: <PH0PR11MB58804435C08AEADF23B095D1DA869@PH0PR11MB5880.namprd11.prod.outlook.com>
References: <20230321081346.192000-1-qiang1.zhang@intel.com>
 <e7951c31-4a24-4493-be4b-603fec0730ff@paulmck-laptop>
 <PH0PR11MB588087645FE02C9F2C113CC8DA869@PH0PR11MB5880.namprd11.prod.outlook.com>
 <f13e878f-65c3-40cb-aa3f-eb87ae6c9d6b@paulmck-laptop>
 <PH0PR11MB5880CAE4D4A58BD71AE984CFDA869@PH0PR11MB5880.namprd11.prod.outlook.com>
 <dc8b5a44-02ca-4e5a-bd4a-644af3e7c11e@paulmck-laptop>
In-Reply-To: <dc8b5a44-02ca-4e5a-bd4a-644af3e7c11e@paulmck-laptop>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?kLLDQrwify/empMcrnPsCsyLT3Q0GBVQPGB7Wa9WVUgtmJ9b/BIJPFHDjGu7?=
 =?us-ascii?Q?ifGckZi5BA1/PYTTJCddYytDddOOpdUBmmhRe9rxktB7kg0DOF4tioDzjm/Y?=
 =?us-ascii?Q?4upO+/eUHmhdflwpSprL8T8nw/FSlERlrNic/f1DsK6n0YatYmmFYE8jdQxm?=
 =?us-ascii?Q?VeUWVQkTSVS0JWQgc9YebENGEvHx7FWsSCQkE5esHKVPbUF8jn/cLmHoCApD?=
 =?us-ascii?Q?9skZzEan/kao95ywiEk1HGIMrQolXDFABXqqd7VPNpC0TJWxxjywESxE5QbN?=
 =?us-ascii?Q?R/UwmTDGWdSywcyU52R1c/EuDEti3jOXcGWSx/uV7u74AZm+Bt55N5yfRNGn?=
 =?us-ascii?Q?PECy8SHtH41z3k0g7RK9qEj8SlCwMVfCsJ+6tdsax1RuWtCT9QoQdEJBf0IO?=
 =?us-ascii?Q?VVKgpgwrG7uRn8Lm2K4fTKeF0QXWRFDmfFt78QFJ+TP0fP6oJ7aSaSGHjILS?=
 =?us-ascii?Q?AcfaDyIc/txFHMuj2of2+R95EEB/ZD7kOGGf3WZjSJpllLtYgR5ac/ibyRFj?=
 =?us-ascii?Q?0jmaR80oQbiRZRiBf/xLY7+TXIVCEAe5ypHYJjg92+km0w7G7CXbJbqju19Y?=
 =?us-ascii?Q?ZcxGyCZ50Gqj5DgNveeKHdAtGJSvsz5za77YcQTuWsDPfzL0fHm6k0+P9EFD?=
 =?us-ascii?Q?g42uNC+nkhUyl09Mc8XcendYy1TxHFUmOOz93XSP5mWJ91O1Zone3OvF4Ubl?=
 =?us-ascii?Q?9S1yLFX7mHn0+QkTSsdxEtDEBVOZ5baJIIuQbdCK6tqZS7AQ7CWJ23oFP4KU?=
 =?us-ascii?Q?z82pvU4A/sfMX/mciyQ9DHtXIw8nPHvrKoWJDsZFZdi3ahyyeb/fQ3BNAPmz?=
 =?us-ascii?Q?74K/BxFs8A3nn+9V+pTVBPuflryq+7zzfA5EKWpo+Ec+mJnt0VeyAvomczE+?=
 =?us-ascii?Q?uV11aWXh1td1ikwyqnAH4/oDdB+Nti9zSKvMCNe+ZgAOcmor2tGkzDGufKB/?=
 =?us-ascii?Q?BErdh9tHXHO0cjvoUKJCVEnQCZ1E30/8uo1EmEb+D9Rj+PnYSMa8gVdaH826?=
 =?us-ascii?Q?QfZUb5JFgtqrElYWNdpu+CofrQ3FwwfztL2ut/E1TWQs5snxanIyxwIaiwX1?=
 =?us-ascii?Q?i8hFVkjPYWh9QPwMOWh1FhjTDUr5zhYMjAs1kmk8z5pW5TrT2L1APhpyu/bZ?=
 =?us-ascii?Q?MejNB0F+jYvdZjq3hSM/lSYu2oKn8ur8Y0YlwbLy9TIMjIebPLEsTlw3mcLM?=
 =?us-ascii?Q?y1BkcMU7ZQZx4xkMucM5MaF2kKL6gYCGLUgDO4U2yHyq4Nf8v/7OEyjXpGxJ?=
 =?us-ascii?Q?psaBakOuAjgSXF7T27B/2thzIVFClxseq/PQfMbF3hNo8XNk6EIYtWpyAOP1?=
 =?us-ascii?Q?PzhbjmyOA/bXquXrAxGYucy2AY/4KNQ2ImOXs+TnZQytDT32yDIGQ/WQJ8zS?=
 =?us-ascii?Q?DkRUdinD42ITwzyqIrwsGxlQXgj2DqcJCjXjh/emQEKQ/ES/yzWhjKJkOxFt?=
 =?us-ascii?Q?gJDYd5K+BLSVqXPtwjvYdBRYpDE9t+F13EVjo110B97Nb6X9vAFcLHYF+xLa?=
 =?us-ascii?Q?Pkk0sSfJ3a/+rhJ2JBxP+upe8YFK/ves/P4BOOutX9BqAlP8sfmndr6Cirvt?=
 =?us-ascii?Q?R0wnR7KYnIPN9rjBUXe8QxWL7new44tZvaFadfYF?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5880.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0ff1a17e-5bb3-48e1-e129-08db2b2206a7
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Mar 2023 22:08:54.2667
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: u5l1NSIrbNKAm78FBcuHFo2HTMFxQ4CUkCx8AEofeK0gMkc8U28SMKf0tCwYbXoOnPOR7m3orWl+L8JLn0Q0rg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB7131
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,
        SPF_NONE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> > > insmod rcutorture.ko
> > > rmmod rcutorture.ko
> > >=20
> > > [  209.437327] WARNING: CPU: 0 PID: 508 at kernel/workqueue.c:3167=20
> > > __flush_work+0x50a/0x540 [  209.437346] Modules linked in:=20
> > > rcutorture(-) torture [last unloaded: rcutorture] [  209.437382]=20
> > > CPU: 0 PID: 508 Comm: rmmod Tainted: G  W  6.3.0-rc1-yocto-standard+=
=20
> > > [  209.437406] RIP: 0010:__flush_work+0x50a/0x540 .....
> > > [  209.437758]  flush_delayed_work+0x36/0x90 [  209.437776] =20
> > > cleanup_srcu_struct+0x68/0x2e0 [  209.437817] =20
> > > srcu_module_notify+0x71/0x140 [  209.437854] =20
> > > blocking_notifier_call_chain+0x9d/0xd0
> > > [  209.437880]  __x64_sys_delete_module+0x223/0x2e0
> > > [  209.438046]  do_syscall_64+0x43/0x90 [  209.438062] =20
> > > entry_SYSCALL_64_after_hwframe+0x72/0xdc
> > >=20
> > > For srcu objects defined with DEFINE_SRCU() or DEFINE_STATIC_SRCU(),=
=20
> > > when compiling and loading as modules, the srcu_module_coming() is=20
> > > invoked, allocate memory for srcu structure's->sda and initialize=20
> > > sda structure, due to not fully initialize srcu structure's->sup, so=
=20
> > > at this time the sup structure's->delaywork.func is null, if not=20
> > > invoke init_srcu_struct_fields() before unloading modules, in=20
> > > srcu_module_going() the __flush_work() find
> > > work->func is empty, will raise the warning above.
> > >=20
> > > This commit add init_srcu_struct_fields() to initialize srcu=20
> > > structure's->sup, in srcu_module_coming().
> > >=20
> > > Signed-off-by: Zqiang <qiang1.zhang@intel.com>
> > >
> > >Good catch, and thank you for testing the in-module case!
> > >
> > >One question below...
> > >
> > >							Thanx, Paul
> > >
> > > ---
> > >  kernel/rcu/srcutree.c | 11 ++++++++---
> > >  1 file changed, 8 insertions(+), 3 deletions(-)
> > >=20
> > > diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index=20
> > > 1fb078abbdc9..42d8720e016c 100644
> > > --- a/kernel/rcu/srcutree.c
> > > +++ b/kernel/rcu/srcutree.c
> > > @@ -1921,7 +1921,8 @@ static int srcu_module_coming(struct module *mo=
d)
> > >  		ssp->sda =3D alloc_percpu(struct srcu_data);
> > >  		if (WARN_ON_ONCE(!ssp->sda))
> > >  			return -ENOMEM;
> > > -		init_srcu_struct_data(ssp);
> > > +		if (WARN_ON_ONCE(init_srcu_struct_fields(ssp, true)))
> > > +			return -ENOMEM;
> > >
> > >Wouldn't it be better to simply delete the init_srcu_struct_data()?
> > >
> > >Then the first call to check_init_srcu_struct() would take care of=20
> > >the initialization, just as for the non-module case.  Or am I missing=
=20
> > >something subtle?
> >=20
> > Hi Paul
> >=20
> > Maybe the check_init_srcu_struct() is always not invoked, for example,
> > In rcutorture.c,   here is such a definition DEFINE_STATIC_SRCU(srcu_ct=
l),
> > but we use torture_type=3Drcu to test,  there will not be any interface=
=20
> > calling
> > check_init_srcu_struct() to initialize srcu_ctl and set =20
> > structure's->delaywork.func is process_srcu().
> > when we unload the rcutorture module, invoke cleanup_srcu_struct() to=20
> > flush sup structure's->delaywork.func, due to the func pointer is not=20
> > initialize, it's null, will trigger warning.
> >=20
> > About kernel/workqueue.c:3167
> >=20
> > __flush_work
> >      if (WARN_ON(!work->func))   <---------trigger waning
> > 	return false;
> >=20
> >=20
> > and  in  init_srcu_struct_fields(ssp, true), wil set=20
> > srcu_sup->sda_is_static is true and set srcu_sup-> srcu_gp_seq_needed=20
> > is zero,  after that when we call
> >  check_init_srcu_struct() again, it not be initialized again.
> >
> >
> >Good point!  In the non-module statically allocated case there is never =
a call to cleanup_srcu_struct().
> >
> >So suppose the code in srcu_module_coming() only did the alloc_percpu(),=
 and then the
> >code in srcu_module_going() only did the the matching
> >free_percpu() instead of the current cleanup_srcu_struct()?
>=20
> But in modules, for srcu objects defined with DEFINE_SRCU() or DEFINE_STA=
TIC_SRCU(),
> when a module is unloaded, we usually don't call cleanup_srcu_struct() in=
 the module
> unload function.
> If in srcu_module_going() only do free_percpu(), the srcu_sup->node memor=
y maybe
> can not free and also lost the opportunity to refresh the running work.
>
>
>But in the module case, isn't the srcu_sup->node also statically
>allocated via the "static struct srcu_usage" declaration?

static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags=
)
{
	sp->srcu_sup->node =3D kcalloc(rcu_num_nodes, sizeof(*ssp->srcu_sup->node)=
, gfp_flags);
	...
}

Regardless of whether the srcu object is declared in the module or not, sup=
->node is dynamically allocated.
right?

Thanks
Zqiang

>
>							Thanx, Paul
>
>------------------------------------------------------------------------
>
>#ifdef MODULE
># define __DEFINE_SRCU(name, is_static)								\
>	static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s=
rcu_usage);	\
>	is_static struct srcu_struct name =3D __SRCU_STRUCT_INIT_MODULE(name, nam=
e##_srcu_usage);	\
>	extern struct srcu_struct * const __srcu_struct_##name;					\
>	struct srcu_struct * const __srcu_struct_##name						\
>		__section("___srcu_struct_ptrs") =3D &name
>#else
># define __DEFINE_SRCU(name, is_static)								\
>	static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data);				\
>	static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s=
rcu_usage);	\
>	is_static struct srcu_struct name =3D							\
>		__SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data)
>#endif
>
> Thanks
> Zqiang
>=20
>=20
> >
> >							Thanx, Paul
> >
> > Thanks
> > Zqiang
> >=20
> > >
> > >It should also be possible to eliminate duplicate code between the=20
> > >in-module and non-module statically allocated initialization cases,=20
> > >come to think of it.
> > >
> > >  	}
> > >  	return 0;
> > >  }
> > > @@ -1931,9 +1932,13 @@ static void srcu_module_going(struct module=20
> > > *mod)  {
> > >  	int i;
> > >  	struct srcu_struct **sspp =3D mod->srcu_struct_ptrs;
> > > +	struct srcu_struct *ssp;
> > > =20
> > > -	for (i =3D 0; i < mod->num_srcu_structs; i++)
> > > -		cleanup_srcu_struct(*(sspp++));
> > > +	for (i =3D 0; i < mod->num_srcu_structs; i++) {
> > > +		ssp =3D *(sspp++);
> > > +		cleanup_srcu_struct(ssp);
> > > +		free_percpu(ssp->sda);
> > > +	}
> > >
> > >And good catch on another memory leak with this one, looks proper to=20
> > >me.
> > >
> > >  }
> > > =20
> > >  /* Handle one module, either coming or going. */
> > > --
> > > 2.25.1
> > >=20