Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 16 Feb 2022 11:07:00 -0800
From:   "Paul E. McKenney" <paulmck@kernel.org>
To:     Padmanabha Srinivasaiah <treasure4paddy@gmail.com>
Cc:     Tejun Heo <tj@kernel.org>, jiangshanlai@gmail.com,
        linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org,
        linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH] workqueue: Fix race in schedule and flush work
Message-ID: <20220216190700.GL4285@paulmck-ThinkPad-P17-Gen-1>
Reply-To: paulmck@kernel.org
References: <20220210184319.25009-1-treasure4paddy@gmail.com>
 <Ygqw+EHo//6VGs6q@slm.duckdns.org>
 <20220216184939.GA3868@pswork>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220216184939.GA3868@pswork>
Precedence: bulk

On Wed, Feb 16, 2022 at 07:49:39PM +0100, Padmanabha Srinivasaiah wrote:
> On Mon, Feb 14, 2022 at 09:43:52AM -1000, Tejun Heo wrote:
> > Hello,
> > 
> > > diff --git a/kernel/workqueue.c b/kernel/workqueue.c
> > > index 33f1106b4f99..a3f53f859e9d 100644
> > > --- a/kernel/workqueue.c
> > > +++ b/kernel/workqueue.c
> > > @@ -3326,28 +3326,38 @@ EXPORT_SYMBOL(cancel_delayed_work_sync);
> > >   */
> > >  int schedule_on_each_cpu(work_func_t func)
> > >  {
> > > -	int cpu;
> > >  	struct work_struct __percpu *works;
> > > +	cpumask_var_t sched_cpumask;
> > > +	int cpu, ret = 0;
> > >  
> > > -	works = alloc_percpu(struct work_struct);
> > > -	if (!works)
> > > +	if (!alloc_cpumask_var(&sched_cpumask, GFP_KERNEL))
> > >  		return -ENOMEM;
> > >  
> > > +	works = alloc_percpu(struct work_struct);
> > > +	if (!works) {
> > > +		ret = -ENOMEM;
> > > +		goto free_cpumask;
> > > +	}
> > > +
> > >  	cpus_read_lock();
> > >  
> > > -	for_each_online_cpu(cpu) {
> > > +	cpumask_copy(sched_cpumask, cpu_online_mask);
> > > +	for_each_cpu_and(cpu, sched_cpumask, cpu_online_mask) {
> > 
> > This definitely would need a comment explaining what's going on cuz it looks
> > weird to be copying the cpumask which is supposed to stay stable due to the
> > cpus_read_lock().Given that it can only happen during early boot and the
> > online cpus can only be expanding, maybe just add sth like:
> > 
> >         if (early_during_boot) {
> >                 for_each_possible_cpu(cpu)
> >                         INIT_WORK(per_cpu_ptr(works, cpu), func);
> >         }
> > 
> 
> Thanks tejun for the reply and suggestions.
> 
> Yes, unfortunately cpus_read_lock not keeping cpumask stable at
> secondary boot. Not sure, may be it only gurantee 'cpu' dont go down
> under cpus_read_[lock/unlock].
> 
> As suggested will tryout something like:
> 	if (system_state != RUNNING) {
> 		:
> 	}
> > BTW, who's calling schedule_on_each_cpu() that early during boot. It makes
> > no sense to do this while the cpumasks can't be stabilized.
> >
> It is  implemenation of CONFIG_TASKS_RUDE_RCU.

Another option would be to adjust CONFIG_TASKS_RUDE_RCU based on where
things are in the boot process.  For example:

	// Wait for one rude RCU-tasks grace period.
	static void rcu_tasks_rude_wait_gp(struct rcu_tasks *rtp)
	{
		if (num_online_cpus() <= 1)
			return;  // Fastpath for only one CPU.
		rtp->n_ipis += cpumask_weight(cpu_online_mask);
		schedule_on_each_cpu(rcu_tasks_be_rude);
	}

Easy enough either way!

							Thanx, Paul