Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Sat, 23 Jun 2018 10:56:34 -0700
From:   "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To:     Joel Fernandes <joel@joelfernandes.org>
Cc:     Steven Rostedt <rostedt@goodmis.org>,
        Byungchul Park <max.byungchul.park@gmail.com>,
        Byungchul Park <byungchul.park@lge.com>,
        jiangshanlai@gmail.com, josh@joshtriplett.org,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        linux-kernel@vger.kernel.org, kernel-team@lge.com, luto@kernel.org
Subject: Re: [RFC 2/2] rcu: Remove ->dynticks_nmi_nesting from struct
 rcu_dynticks
Reply-To: paulmck@linux.vnet.ibm.com
References: <20180620145814.GQ3593@linux.vnet.ibm.com>
 <CANrsvRMYc5u3L53nnnGC6p-QcjBTfLPjkJGZyubGWAPauc-W8w@mail.gmail.com>
 <20180620164902.GW3593@linux.vnet.ibm.com>
 <20180622055659.GA255098@joelaf.mtv.corp.google.com>
 <20180622132843.GN3593@linux.vnet.ibm.com>
 <20180622181916.GA13628@joelaf.mtv.corp.google.com>
 <20180622205813.GV3593@linux.vnet.ibm.com>
 <20180622170042.4adfbe21@gandalf.local.home>
 <20180622211600.GX3593@linux.vnet.ibm.com>
 <20180623154839.GA12939@joelaf.mtv.corp.google.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20180623154839.GA12939@joelaf.mtv.corp.google.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Message-Id: <20180623175634.GI3593@linux.vnet.ibm.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Sat, Jun 23, 2018 at 08:48:39AM -0700, Joel Fernandes wrote:
> On Fri, Jun 22, 2018 at 02:16:00PM -0700, Paul E. McKenney wrote:
> > On Fri, Jun 22, 2018 at 05:00:42PM -0400, Steven Rostedt wrote:
> > > On Fri, 22 Jun 2018 13:58:13 -0700
> > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > 
> > > > Something like this:
> > > > 
> > > > 	IRQ entered
> > > > 
> > > > And never exited.  Ever.  I actually saw this in 2011.
> > > 
> > > I still believe this was actually a bug. And perhaps you made the RCU
> > > code robust enough to handle this bug ;-)
> > 
> > Welcome to my world!
> > 
> > But I recall it being used in several places, so if it was a bug, it
> > was an intentional bug.  Probably the worst kind.
> > 
> > Sort of like nested NMIs and interrupts within NMI handlers.  ;-)
> > 
> > > > Or something like this:
> > > > 
> > > > 	IRQ exited
> > > > 
> > > > Without a corresponding IRQ enter.
> > > > 
> > > > The current code handles both of these situations, at least assuming
> > > > that the interrupt entry/exit happens during a non-idle period.
> > > > 
> > > > > > So why this function-call structure?  Well, you see, NMI handlers can
> > > > > > take what appear to RCU to be normal interrupts...
> > > > > > 
> > > > > > (And I just added that fun fact to Requirements.html.)  
> > > > > 
> > > > > Yes, I'll definitely go through all the interrupt requirements in the doc and
> > > > > thanks for referring me to it.  
> > > > 
> > > > My concern may well be obsolete.  It would be good if it was!  ;-)
> > > 
> > > I'd love to mandate that irq_enter() must be paired with irq_exit(). I
> > > don't really see any rationale for it to be otherwise. If there is a
> > > case, perhaps it needs to be fixed.
> > 
> > Given that the usermode helpers now look to be common code using
> > workqueues, kthreads, and calls to do_execve(), it might well be that
> > the days of half-interrupts are behind us.
> > 
> > But how to actually validate this?  My offer of adding a WARN_ON_ONCE()
> > and waiting a few years still stands, but perhaps you have a better
> > approach.
> 
> Hi Paul, I am Ok with adding a warning for a couple of releases if you and
> others are Ok with it, how about something like this? Feel free to use the
> diff as a starting point or a different approach if you/others prefer
> something else. Thanks.

A few years rather than a few releases, but yes.  ;-)

The checks would need to go just before the "crowbar" stores.  I will put
something together after Byungchul's patches in this area have had time
to burn in for a few days.

							Thanx, Paul

> ---8<-----------------------
> 
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 8788ddbc0d13..176de74f5027 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -767,6 +767,21 @@ void rcu_idle_enter(void)
>   */
>  void rcu_user_enter(void)
>  {
> +	struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
> +
> +	/*
> +	 * Add warning to ensure: no known instances of entering userspace from
> +	 * IRQ/NMI handlers exist today. Currently special crowbarring of the
> +	 * dynticks_nmi_nesting and maintaining of this separate counter when
> +	 * dynticks_nesting exists, is done in order to handle entering of
> +	 * userspace from an IRQ context and never returning. Lets track it for
> +	 * a couple of kernel releases and then if the warning doesn't occur,
> +	 * we can try to simplify the code and combine/eliminate the counters.
> +	 * See: http://lkml.kernel.org/r/20180622181422.GT3593@linux.vnet.ibm.com
> +	 */
> +	WARN_ON_ONCE(rdtp->dynticks_nmi_nesting &&
> +		     rdtp->dynticks_nmi_nesting != DYNTICK_IRQ_NONIDLE);
> +
>  	lockdep_assert_irqs_disabled();
>  	rcu_eqs_enter(true);
>  }
>