Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 31 Mar 2022 10:29:43 -0700
From:   "Paul E. McKenney" <paulmck@kernel.org>
To:     "Zhang, Qiang1" <qiang1.zhang@intel.com>
Cc:     "frederic@kernel.org" <frederic@kernel.org>,
        "rcu@vger.kernel.org" <rcu@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] rcu: Put the irq work into hard interrupt context for
 execution
Message-ID: <20220331172943.GV4285@paulmck-ThinkPad-P17-Gen-1>
Reply-To: paulmck@kernel.org
References: <20220330060012.2470054-1-qiang1.zhang@intel.com>
 <20220330201620.GM4285@paulmck-ThinkPad-P17-Gen-1>
 <PH0PR11MB58802C1246C6F4FB895BDDA9DA1F9@PH0PR11MB5880.namprd11.prod.outlook.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <PH0PR11MB58802C1246C6F4FB895BDDA9DA1F9@PH0PR11MB5880.namprd11.prod.outlook.com>
Precedence: bulk

On Wed, Mar 30, 2022 at 10:47:05PM +0000, Zhang, Qiang1 wrote:
> On Wed, Mar 30, 2022 at 02:00:12PM +0800, Zqiang wrote:
> > In PREEMPT_RT kernel, if irq work flags is not set, it will be 
> > executed in per-CPU irq_work kthreads. set IRQ_WORK_HARD_IRQ flags to 
> > irq work, put it in the context of hard interrupt execution, 
> > accelerate scheduler to re-evaluate.
> > 
> > Signed-off-by: Zqiang <qiang1.zhang@intel.com>
> > ---
> >  kernel/rcu/tree.c        | 2 +-
> >  kernel/rcu/tree_plugin.h | 2 +-
> >  2 files changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 
> > e2ffbeceba69..a69587773a85 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -678,7 +678,7 @@ static void late_wakeup_func(struct irq_work 
> > *work)  }
> >  
> >  static DEFINE_PER_CPU(struct irq_work, late_wakeup_work) =
> > -	IRQ_WORK_INIT(late_wakeup_func);
> > +	IRQ_WORK_INIT_HARD(late_wakeup_func);
> 
> >This is used only by rcu_irq_work_resched(), which is invoked only by rcu_user_enter(), which is never invoked until userspace is enabled, by which time all of the various kthreads will have been spawned, correct?
> >
> >Either way, please show me the exact sequence of events that lead to a problem with the current IRQ_WORK_INIT().
> >
> >  /*
> >   * If either:
> > diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 
> > 3037c2536e1f..cf7bd28af8ef 100644
> > --- a/kernel/rcu/tree_plugin.h
> > +++ b/kernel/rcu/tree_plugin.h
> > @@ -661,7 +661,7 @@ static void rcu_read_unlock_special(struct task_struct *t)
> >  			    expboost && !rdp->defer_qs_iw_pending && cpu_online(rdp->cpu)) {
> >  				// Get scheduler to re-evaluate and call hooks.
> >  				// If !IRQ_WORK, FQS scan will eventually IPI.
> > -				init_irq_work(&rdp->defer_qs_iw, rcu_preempt_deferred_qs_handler);
> > +				rdp->defer_qs_iw = 
> > +IRQ_WORK_INIT_HARD(rcu_preempt_deferred_qs_handler);
> >  				rdp->defer_qs_iw_pending = true;
> >  				irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);
> >  			}
> >
> >OK, in theory, rcu_read_unlock() could get to this point before all of the various kthreads were spawned.  In practice, the next time that the boot CPU went idle, the end of the quiescent state would be noticed.
> 
> Through my understanding, use irq_work in order to make the quiescent state be noticed earlier,
> Because the irq_work execute in interrupt, this irq_work can be executed in time, but In RT kernel
> The irq_work  is put into the kthread for execution, when it is executed, it is affected by the scheduling delay.
> Is there anything I missed?

Yes, in that I am not seeing any actual data showing that this fix really
makes things better.  Please in mind that IRQ_WORK_INIT_HARD does have
performance disadvantages of its own.  So although I agree with your
words saying that IRQ_WORK_INIT_HARD -might- be helpful, those words
are not sufficient.

So, can you show a statistically significant benefit on a real system?
For example, by measuring the time required for a expedited grace period
to complete?  That would argue for this change, though it would need to be
conditional, so that systems that don't care that much about the latency
of expedited RCU grace periods don't need to pay the IRQ_WORK_INIT_HARD
performance penalties.  Or you would need to demonstrate that these
performance penalties don't cause problems.  (But such a demonstration
is not easy given the wide variety of systems that Linux supports.)

Now, I could imagine that the current code could cause problems during
boot on CONFIG_PREEMPT_RT kernels.  But, believe me, I can imagine all
sorts of horrible problems.  But we should fix those that happen not
just in my imagination, but also in the real world.  ;-)

So if you can make such a problem happen in real life, then I would be
happy to take a patch that fixed this on CONFIG_PREEMPT_RT but kept the
current code otherwise.

							Thanx, Paul

> Thanks
> Zqiang	
> 
> >
> >Or has this been failing in some other manner?  If so, please let me know the exact sequence of events.
> >
> >							Thanx, Paul