Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Mon, 25 Jun 2018 10:19:20 -0700
From:   "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To:     Joel Fernandes <joel@joelfernandes.org>
Cc:     Byungchul Park <byungchul.park@lge.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Byungchul Park <max.byungchul.park@gmail.com>,
        jiangshanlai@gmail.com, josh@joshtriplett.org,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        linux-kernel@vger.kernel.org, kernel-team@lge.com, luto@kernel.org
Subject: Re: [RFC 2/2] rcu: Remove ->dynticks_nmi_nesting from struct
 rcu_dynticks
Reply-To: paulmck@linux.vnet.ibm.com
References: <20180620145814.GQ3593@linux.vnet.ibm.com>
 <CANrsvRMYc5u3L53nnnGC6p-QcjBTfLPjkJGZyubGWAPauc-W8w@mail.gmail.com>
 <20180620164902.GW3593@linux.vnet.ibm.com>
 <20180622055659.GA255098@joelaf.mtv.corp.google.com>
 <20180622132843.GN3593@linux.vnet.ibm.com>
 <20180622181916.GA13628@joelaf.mtv.corp.google.com>
 <20180622143247.781028b1@gandalf.local.home>
 <20180622200548.GA114655@joelaf.mtv.corp.google.com>
 <20180625082824.GB21377@X58A-UD3R>
 <20180625163951.GA52646@joelaf.mtv.corp.google.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20180625163951.GA52646@joelaf.mtv.corp.google.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Message-Id: <20180625171920.GR3593@linux.vnet.ibm.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, Jun 25, 2018 at 09:39:51AM -0700, Joel Fernandes wrote:
> On Mon, Jun 25, 2018 at 05:28:24PM +0900, Byungchul Park wrote:
> > On Fri, Jun 22, 2018 at 01:05:48PM -0700, Joel Fernandes wrote:
> > > On Fri, Jun 22, 2018 at 02:32:47PM -0400, Steven Rostedt wrote:
> > > > On Fri, 22 Jun 2018 11:19:16 -0700
> > > > Joel Fernandes <joel@joelfernandes.org> wrote:
> > > > 
> > > > > Sure. So in a later thread you mentioned "usermode helpers". I took a closer
> > > > > look at that subsystem, and it seems you can execute usermode helpers from
> > > > > atomic sections with help of UMH_NO_WAIT flag.
> > > > > 
> > > > > Then I checked where this flag is used and it turns out its from the
> > > > > mce_work_trigger function in x86/kernel/cpu/mcheck/dev-mcelog.c which can be
> > > > > called infact from an interrupt context (mce_notify_irq).
> > > > > 
> > > > > Is this the usecase you remember causing this weird transitions to userspace?
> > > > 
> > > > But this case still looks like it uses work queues, it just doesn't
> > > > wait for the result.
> > > > 
> > > > I'll have to look at the code from what it looked like back in 2011, to
> > > > see if there was an actual issue here back then.
> > > 
> > > Good point Steve. So I guess in the current kernel sources, there's no code
> > > that uses UMH in IRQ context AFAICT. I'll go through the google group thread
> > > Paul pointed as well to study the history of the problem a bit more.
> > 
> > Me too. Good discussion we had thanks to you, Joel.
> 
> No problem, thanks for the patch in the first place which triggered this
> discussion.
> 
> For whatever its worth, I made some notes of what I understood from reading
> the code and old posts because I was sure I would otherwise forget
> everything:
> http://www.joelfernandes.org/linuxinternals/2018/06/15/rcu-dynticks.html
> 
> Feel free to comment on that post directly (or here) if you feel something is
> grossly wrong.
> 
> Again thank you and everyone for the discussion! ;-)

Not a bad writeup!  A few comments, as usual...

							Thanx, Paul

------------------------------------------------------------------------

	When I traced rdtp->dynticks_nesting, I could only find its
	value to be either a 0 or a 1. However looking back at old kernel
	sources, it appears that these can be nested becaues of so called
	“half-interrupts”. I believe these are basically interrupts
	that cause a transition to usermode due to usermode upcalls
	(usermode helper subsystem). So a nesting situation could be
	something like: 1. Transition from idle to process context which
	makes dynticks_nesting == 1. Next, an interrupt comes in which
	makes a usermode upcall. This usermode call now makes a system
	call causing entry back into process context, which increments
	the dynticks_nesting counter to 2. Such a crazy situation is
	perhaps possible.

The half-interrupts can instead cause ->dynticks_nmi_nesting to either
fail to return to zero or to go negative, depending on which half of
the interrupt was present.  I don't immediately recall the reason for
allowing nested process-level entry/exit.  Might be another place to
put a WARN_ON_ONCE(), as eliminating this capability would save another
conditional branch.

	Any time the rdtp->dynticks counter’s second-lowest most bit
	is not set, we are in an EQS, and if its set, then we are not
	(second lowest because lowest is reserved for something else as
	of v4.18-rc1). This function is not useful to check if we’re
	in an EQS from a timer tick though, because its possible the
	timer tick interrupt entry caused an EQS exit which updated
	the counter. IOW, the ‘dynticks’ counter is not capable of
	checking if we had already exited the EQS before. To check if
	we were in an EQS or not from the timer tick, we instead must
	use dynticks_nesting counter. More on that later. The above
	function is probably just useful to make sure that interrupt
	entry/exit is properly updating the dynticks counter, and also
	to make sure from non-interrupt context that RCU is in an EQS
	(see rcu_gp_fqs function).

You lost me on this one.  There is rcu_is_cpu_rrupt_from_idle(), but
I am not sure what you are trying to achieve here, so I am not sure
whether this function does what you want.

	When dynticks_nesting is decremented to 0 (the outermost
	process-context nesting level exit causes an eqs-entry), the
	dynticks_nmi_nesting is reset to

I think you want "0." at the end of this sentence.  Or maybe my browser
is messing things up.

							Thanx, Paul