On Tue, Oct 15, 2013 at 02:01:04AM +0000, Liu, Chuansheng wrote:
> We meet one issue that during trigger all CPU backtrace, but during in the NMI handler arch_trigger_all_cpu_backtrace_handler,
> It hit the PAGE fault, then PAGE fault is in loop, at last the thread stack overflow, and system panic.
>
> Anyone can give some help? Thanks.
Looks like we re-enter the fault several times. On x86-32, NMIs can
fault if they dereference vmalloc'ed area. I wonder if the module thing
we lookup in the NMI is stored on some vmalloc'ed area.
Also, may be we enter the fault, trigger the WARN_ON_ONCE(in_nmi()) warning,
which in turn dumps the stack, lookup that module address, refaults, etc...
On Tue, Oct 15, 2013 at 02:18:53PM +0200, Frederic Weisbecker wrote:
> On Tue, Oct 15, 2013 at 02:01:04AM +0000, Liu, Chuansheng wrote:
> > We meet one issue that during trigger all CPU backtrace, but during in the NMI handler arch_trigger_all_cpu_backtrace_handler,
> > It hit the PAGE fault, then PAGE fault is in loop, at last the thread stack overflow, and system panic.
> >
> > Anyone can give some help? Thanks.
>
> Looks like we re-enter the fault several times. On x86-32, NMIs can
> fault if they dereference vmalloc'ed area. I wonder if the module thing
> we lookup in the NMI is stored on some vmalloc'ed area.
IIRC modules are indeed allocated using vmalloc. See module_alloc()
using vmalloc_exec()
On Tue, Oct 15, 2013 at 02:37:17PM +0200, Peter Zijlstra wrote:
> On Tue, Oct 15, 2013 at 02:18:53PM +0200, Frederic Weisbecker wrote:
> > On Tue, Oct 15, 2013 at 02:01:04AM +0000, Liu, Chuansheng wrote:
> > > We meet one issue that during trigger all CPU backtrace, but during in the NMI handler arch_trigger_all_cpu_backtrace_handler,
> > > It hit the PAGE fault, then PAGE fault is in loop, at last the thread stack overflow, and system panic.
> > >
> > > Anyone can give some help? Thanks.
> >
> > Looks like we re-enter the fault several times. On x86-32, NMIs can
> > fault if they dereference vmalloc'ed area. I wonder if the module thing
> > we lookup in the NMI is stored on some vmalloc'ed area.
>
> IIRC modules are indeed allocated using vmalloc. See module_alloc()
> using vmalloc_exec()
This might then be a module that uses call_rcu(), but which does not have
the needed rcu_barrier() in the module-exit function.
Thanx, Paul
On Tue, Oct 15, 2013 at 02:37:17PM +0200, Peter Zijlstra wrote:
> On Tue, Oct 15, 2013 at 02:18:53PM +0200, Frederic Weisbecker wrote:
> > On Tue, Oct 15, 2013 at 02:01:04AM +0000, Liu, Chuansheng wrote:
> > > We meet one issue that during trigger all CPU backtrace, but during in the NMI handler arch_trigger_all_cpu_backtrace_handler,
> > > It hit the PAGE fault, then PAGE fault is in loop, at last the thread stack overflow, and system panic.
> > >
> > > Anyone can give some help? Thanks.
> >
> > Looks like we re-enter the fault several times. On x86-32, NMIs can
> > fault if they dereference vmalloc'ed area. I wonder if the module thing
> > we lookup in the NMI is stored on some vmalloc'ed area.
>
> IIRC modules are indeed allocated using vmalloc. See module_alloc()
> using vmalloc_exec()
Right. At least the module text. Now I'm not sure the module_address_lookup()
dereference that. But there many other objects allocated in module.c that use
alloc_percpu(), which in turn can use vmalloc.
IIRC Steve made the NMIs safely faultable. So may be we can remove the WARN_ON in
do_page_fault(). It may not be a good idea to allow fault in NMIs though. Steve?
On Tue, Oct 15, 2013 at 05:48:37AM -0700, Paul E. McKenney wrote:
> On Tue, Oct 15, 2013 at 02:37:17PM +0200, Peter Zijlstra wrote:
> > On Tue, Oct 15, 2013 at 02:18:53PM +0200, Frederic Weisbecker wrote:
> > > On Tue, Oct 15, 2013 at 02:01:04AM +0000, Liu, Chuansheng wrote:
> > > > We meet one issue that during trigger all CPU backtrace, but during in the NMI handler arch_trigger_all_cpu_backtrace_handler,
> > > > It hit the PAGE fault, then PAGE fault is in loop, at last the thread stack overflow, and system panic.
> > > >
> > > > Anyone can give some help? Thanks.
> > >
> > > Looks like we re-enter the fault several times. On x86-32, NMIs can
> > > fault if they dereference vmalloc'ed area. I wonder if the module thing
> > > we lookup in the NMI is stored on some vmalloc'ed area.
> >
> > IIRC modules are indeed allocated using vmalloc. See module_alloc()
> > using vmalloc_exec()
>
> This might then be a module that uses call_rcu(), but which does not have
> the needed rcu_barrier() in the module-exit function.
I rather believe it's due to the lazy paging of vmalloc area in x86-32. We had issues
like that in the past. For example that's the reason why we do an ad-hoc per-cpu
allocation on callchain buffers in perf rather than using alloc_percpu() which might
use vmalloc.
On Tue, Oct 15, 2013 at 02:59:15PM +0200, Frederic Weisbecker wrote:
> On Tue, Oct 15, 2013 at 05:48:37AM -0700, Paul E. McKenney wrote:
> > On Tue, Oct 15, 2013 at 02:37:17PM +0200, Peter Zijlstra wrote:
> > > On Tue, Oct 15, 2013 at 02:18:53PM +0200, Frederic Weisbecker wrote:
> > > > On Tue, Oct 15, 2013 at 02:01:04AM +0000, Liu, Chuansheng wrote:
> > > > > We meet one issue that during trigger all CPU backtrace, but during in the NMI handler arch_trigger_all_cpu_backtrace_handler,
> > > > > It hit the PAGE fault, then PAGE fault is in loop, at last the thread stack overflow, and system panic.
> > > > >
> > > > > Anyone can give some help? Thanks.
> > > >
> > > > Looks like we re-enter the fault several times. On x86-32, NMIs can
> > > > fault if they dereference vmalloc'ed area. I wonder if the module thing
> > > > we lookup in the NMI is stored on some vmalloc'ed area.
> > >
> > > IIRC modules are indeed allocated using vmalloc. See module_alloc()
> > > using vmalloc_exec()
> >
> > This might then be a module that uses call_rcu(), but which does not have
> > the needed rcu_barrier() in the module-exit function.
>
> I rather believe it's due to the lazy paging of vmalloc area in x86-32. We had issues
> like that in the past. For example that's the reason why we do an ad-hoc per-cpu
> allocation on callchain buffers in perf rather than using alloc_percpu() which might
> use vmalloc.
I must defer to your greater experience with this type of bug.
Thanx, Paul
On Tue, Oct 15, 2013 at 06:06:59AM -0700, Paul E. McKenney wrote:
> On Tue, Oct 15, 2013 at 02:59:15PM +0200, Frederic Weisbecker wrote:
> > On Tue, Oct 15, 2013 at 05:48:37AM -0700, Paul E. McKenney wrote:
> > > On Tue, Oct 15, 2013 at 02:37:17PM +0200, Peter Zijlstra wrote:
> > > > On Tue, Oct 15, 2013 at 02:18:53PM +0200, Frederic Weisbecker wrote:
> > > > > On Tue, Oct 15, 2013 at 02:01:04AM +0000, Liu, Chuansheng wrote:
> > > > > > We meet one issue that during trigger all CPU backtrace, but during in the NMI handler arch_trigger_all_cpu_backtrace_handler,
> > > > > > It hit the PAGE fault, then PAGE fault is in loop, at last the thread stack overflow, and system panic.
> > > > > >
> > > > > > Anyone can give some help? Thanks.
> > > > >
> > > > > Looks like we re-enter the fault several times. On x86-32, NMIs can
> > > > > fault if they dereference vmalloc'ed area. I wonder if the module thing
> > > > > we lookup in the NMI is stored on some vmalloc'ed area.
> > > >
> > > > IIRC modules are indeed allocated using vmalloc. See module_alloc()
> > > > using vmalloc_exec()
> > >
> > > This might then be a module that uses call_rcu(), but which does not have
> > > the needed rcu_barrier() in the module-exit function.
> >
> > I rather believe it's due to the lazy paging of vmalloc area in x86-32. We had issues
> > like that in the past. For example that's the reason why we do an ad-hoc per-cpu
> > allocation on callchain buffers in perf rather than using alloc_percpu() which might
> > use vmalloc.
>
> I must defer to your greater experience with this type of bug.
With some chances I'll be proved wrong. I hope, because that issue is not easily fixed.
On Tue, Oct 15, 2013 at 03:17:59PM +0200, Frederic Weisbecker wrote:
> On Tue, Oct 15, 2013 at 06:06:59AM -0700, Paul E. McKenney wrote:
> > On Tue, Oct 15, 2013 at 02:59:15PM +0200, Frederic Weisbecker wrote:
> > > On Tue, Oct 15, 2013 at 05:48:37AM -0700, Paul E. McKenney wrote:
> > > > On Tue, Oct 15, 2013 at 02:37:17PM +0200, Peter Zijlstra wrote:
> > > > > On Tue, Oct 15, 2013 at 02:18:53PM +0200, Frederic Weisbecker wrote:
> > > > > > On Tue, Oct 15, 2013 at 02:01:04AM +0000, Liu, Chuansheng wrote:
> > > > > > > We meet one issue that during trigger all CPU backtrace, but during in the NMI handler arch_trigger_all_cpu_backtrace_handler,
> > > > > > > It hit the PAGE fault, then PAGE fault is in loop, at last the thread stack overflow, and system panic.
> > > > > > >
> > > > > > > Anyone can give some help? Thanks.
> > > > > >
> > > > > > Looks like we re-enter the fault several times. On x86-32, NMIs can
> > > > > > fault if they dereference vmalloc'ed area. I wonder if the module thing
> > > > > > we lookup in the NMI is stored on some vmalloc'ed area.
> > > > >
> > > > > IIRC modules are indeed allocated using vmalloc. See module_alloc()
> > > > > using vmalloc_exec()
> > > >
> > > > This might then be a module that uses call_rcu(), but which does not have
> > > > the needed rcu_barrier() in the module-exit function.
> > >
> > > I rather believe it's due to the lazy paging of vmalloc area in x86-32. We had issues
> > > like that in the past. For example that's the reason why we do an ad-hoc per-cpu
> > > allocation on callchain buffers in perf rather than using alloc_percpu() which might
> > > use vmalloc.
> >
> > I must defer to your greater experience with this type of bug.
>
> With some chances I'll be proved wrong. I hope, because that issue is not easily fixed.
Fair enough!
Thanx, Paul
On Tue, 15 Oct 2013 14:54:11 +0200
Frederic Weisbecker <[email protected]> wrote:
> On Tue, Oct 15, 2013 at 02:37:17PM +0200, Peter Zijlstra wrote:
> > On Tue, Oct 15, 2013 at 02:18:53PM +0200, Frederic Weisbecker wrote:
> > > On Tue, Oct 15, 2013 at 02:01:04AM +0000, Liu, Chuansheng wrote:
> > > > We meet one issue that during trigger all CPU backtrace, but during in the NMI handler arch_trigger_all_cpu_backtrace_handler,
> > > > It hit the PAGE fault, then PAGE fault is in loop, at last the thread stack overflow, and system panic.
> > > >
> > > > Anyone can give some help? Thanks.
> > >
> > > Looks like we re-enter the fault several times. On x86-32, NMIs can
> > > fault if they dereference vmalloc'ed area. I wonder if the module thing
> > > we lookup in the NMI is stored on some vmalloc'ed area.
> >
> > IIRC modules are indeed allocated using vmalloc. See module_alloc()
> > using vmalloc_exec()
>
> Right. At least the module text. Now I'm not sure the module_address_lookup()
> dereference that. But there many other objects allocated in module.c that use
> alloc_percpu(), which in turn can use vmalloc.
>
> IIRC Steve made the NMIs safely faultable. So may be we can remove the WARN_ON in
> do_page_fault(). It may not be a good idea to allow fault in NMIs though. Steve?
NMIs should be safe to fault after my patches went in. My main concern
was with x86-64 as the NMI vector uses IST which resets the stack, but
x86-32 keeps using the same stack if we are in ring0 (switches if we
are in ring3 just like any other interrupt).
What's the original panic? I don't see the original post?
-- Steve