LinuxLists.cc - perf sched record hangs machine

[permalink] [raw]

Subject: Re: perf sched record hangs machine

On Tue, 2009-09-22 at 23:24 +0200, Frederic Weisbecker wrote:
> On Tue, Sep 22, 2009 at 10:09:26PM +0100, Chris Malley wrote:

> > When I run "perf sched record sleep 10", and generate some activity,
> > after a few seconds my machine (single cpu Dell Latitude D400, 32-bit)
> > locks up.
> > This happens almost every time, not always with the same call trace
> > but always in default_send_IPI_self(). Occasionally I get a completed
> > recording but only if the machine is otherwise quiet.
> >
> > Kernel and perf sched version is current Linus tree (as of 43c1266ce4dc06b).

> > [ 389.272175] BUG: unable to handle kernel paging request at ffffb300
> > [ 389.272294] IP: [<c011b0bd>] default_send_IPI_self+0x1d/0x50

> > [ 389.274708] Call Trace:
> > [ 389.274752] [<c010e3b4>] ? set_perf_event_pending+0x14/0x20
> > [ 389.274801] [<c01b9751>] ? perf_output_unlock+0x121/0x1a0
> > [ 389.274848] [<c01b981a>] ? perf_output_end+0x4a/0x70

> Ah, that calls perf_output_wakeup(), then perf_pending_counter(),
> ....., wake_up_all()
>
> And it looks like this event is called with the rq lock
> held, then we have a deadlock.

Not really, rq->lock is IRQ-safe, that is we should have IRQs disabled
while holding it, so the self-IPI should get delayed until we release
rq->lock.

The code from the OOPS seems to decode to something like:

00000000 <_EIP>:
0: 0f 44 c1 cmove %ecx,%eax
3: 89 02 mov %eax,(%edx)
5: 5b pop %ebx
6: 5d pop %ebp
7: c3 ret
8: 8d b6 00 00 00 00 lea 0x0(%esi),%esi
e: 55 push %ebp
f: 89 00 mov %eax,(%eax)

Which matches the tail of:

.size default_send_IPI_all, .-default_send_IPI_all
.p2align 4,,15
.globl default_send_IPI_self
.type default_send_IPI_self, @function
default_send_IPI_self:
.LFB1592:
.loc 1 136 0
.LVL7:
pushl %ebp #
.LCFI3:
movl %esp, %ebp #,
.LCFI4:
pushl %ebx #
.LCFI5:
.loc 1 136 0
movl %eax, %ebx # vector, vector
.loc 1 137 0
movl apic, %eax # apic, apic
.LVL8:
movl 32(%eax), %ecx # <variable>.dest_logical, D.27494
jmp .L27 #
.p2align 4,,7
.p2align 3
.L28:
.LBB287:
.LBB288:
.LBB289:
.LBB290:
.LBB291:
.LBB292:
.LBB293:
.LBB294:
.loc 3 709 0
#APP
# 709 "/mnt/build/linux-2.6/arch/x86/include/asm/processor.h" 1
rep; nop
# 0 "" 2
#NO_APP
.L27:
.LBE294:
.LBE293:
.LBE292:
.LBE291:
.LBB295:
.LBB296:
.loc 4 112 0
movl __FIXADDR_TOP, %eax # __FIXADDR_TOP, __FIXADDR_TOP
leal -15616(%eax), %edx #, D.27700
movl -15616(%eax), %eax #, temp.615
.LBE296:
.LBE295:
.loc 2 56 0
testb $16, %ah #, temp.615
jne .L28 #,
.LBE290:
.LBE289:
.LBB297:
.LBB298:
.loc 2 36 0
movl %ecx, %eax # D.27494, icr
.LVL9:
.loc 2 40 0
orl $263168, %ecx #, D.27494
.loc 2 36 0
orl $262144, %eax #, icr
.loc 2 40 0
orl %ebx, %eax # vector, icr
cmpl $2, %ebx #, vector
cmove %ecx, %eax # D.27494,, icr
.LBE298:
.LBE297:
.LBB299:
.LBB300:
.loc 4 105 0
#APP
# 105 "/mnt/build/linux-2.6/arch/x86/include/asm/apic.h" 1
661:
movl %eax, (%edx) # v,* D.27700
662:
.section .altinstructions,"a"
.balign 4
.long 661b
.long 663f
.byte (3*32+19)
.byte 662b-661b
.byte 664f-663f
.previous
.section .altinstr_replacement, "ax"
663:
xchgl %eax, (%edx) # v,* D.27700
664:
.previous
# 0 "" 2
.LVL10:
#NO_APP
.LBE300:
.LBE299:
.LBE288:
.LBE287:
.loc 1 138 0
popl %ebx #
.LVL11:
popl %ebp #
ret
.LFE1592:
.size default_send_IPI_self, .-default_send_IPI_self

Which seems to suggest that cmove is in __prepare_ICR, but I'm not sure
how that can cause a page-fault, as that function is rather
pointer-less.

Did it maybe delay evaluating apic->dest_logical that late, it appear to
be the first usage of that argument?

Ingo, any ideas?

2009-09-23 07:33:12

[permalink] [raw]

Subject: Re: perf sched record hangs machine

* Peter Zijlstra <[email protected]> wrote:

> .size default_send_IPI_self, .-default_send_IPI_self
>
> Which seems to suggest that cmove is in __prepare_ICR, but I'm not
> sure how that can cause a page-fault, as that function is rather
> pointer-less.
>
> Did it maybe delay evaluating apic->dest_logical that late, it appear
> to be the first usage of that argument?
>
> Ingo, any ideas?

Yeah, the problem is:

[ 0.042445] Local APIC not detected. Using dummy APIC emulation.

we shouldnt be trying to program the lapic in that case - all ops should
be a NOP, including send-ipi-self. Cyrill?

Ingo

2009-09-23 07:40:44

[permalink] [raw]

Subject: Re: perf sched record hangs machine

* Ingo Molnar <[email protected]> wrote:

>
> * Peter Zijlstra <[email protected]> wrote:
>
> > .size default_send_IPI_self, .-default_send_IPI_self
> >
> > Which seems to suggest that cmove is in __prepare_ICR, but I'm not
> > sure how that can cause a page-fault, as that function is rather
> > pointer-less.
> >
> > Did it maybe delay evaluating apic->dest_logical that late, it appear
> > to be the first usage of that argument?
> >
> > Ingo, any ideas?
>
> Yeah, the problem is:
>
> [ 0.042445] Local APIC not detected. Using dummy APIC emulation.
>
> we shouldnt be trying to program the lapic in that case - all ops
> should be a NOP, including send-ipi-self. Cyrill?

Cyrill, the right fix would be to define a:

struct apic apic_none;

with dummy functions injected. Set struct apic *apic to _that_, instead
of apic_default.

This means that if we manage to detect a local APIC, we will set it to
apic_default (or some other driver) - but the bootup default will be 'no
APIC'.

This is a lot cleaner and a lot less error-prone than the direct
frobbing of the function pointers.

Ingo

2009-09-23 07:44:37

[permalink] [raw]

Subject: Re: perf sched record hangs machine

On 9/23/09, Ingo Molnar <[email protected]> wrote:
>
> * Ingo Molnar <[email protected]> wrote:
>
>>
>> * Peter Zijlstra <[email protected]> wrote:
>>
>> > .size default_send_IPI_self, .-default_send_IPI_self
>> >
>> > Which seems to suggest that cmove is in __prepare_ICR, but I'm not
>> > sure how that can cause a page-fault, as that function is rather
>> > pointer-less.
>> >
>> > Did it maybe delay evaluating apic->dest_logical that late, it appear
>> > to be the first usage of that argument?
>> >
>> > Ingo, any ideas?
>>
>> Yeah, the problem is:
>>
>> [ 0.042445] Local APIC not detected. Using dummy APIC emulation.
>>
>> we shouldnt be trying to program the lapic in that case - all ops
>> should be a NOP, including send-ipi-self. Cyrill?
>
> Cyrill, the right fix would be to define a:
>
> struct apic apic_none;
>
> with dummy functions injected. Set struct apic *apic to _that_, instead
> of apic_default.
>
> This means that if we manage to detect a local APIC, we will set it to
> apic_default (or some other driver) - but the bootup default will be 'no
> APIC'.
>

May it wait till evening (i'll back from office and take a look)?

> This is a lot cleaner and a lot less error-prone than the direct
> frobbing of the function pointers.
>
> Ingo
>

2009-09-23 08:48:04

[permalink] [raw]

Subject: Re: perf sched record hangs machine

On 9/23/09, Cyrill Gorcunov <[email protected]> wrote:
> On 9/23/09, Ingo Molnar <[email protected]> wrote:
>>
>> * Ingo Molnar <[email protected]> wrote:
>>
>>>
>>> * Peter Zijlstra <[email protected]> wrote:
>>>
>>> > .size default_send_IPI_self, .-default_send_IPI_self
>>> >
>>> > Which seems to suggest that cmove is in __prepare_ICR, but I'm not
>>> > sure how that can cause a page-fault, as that function is rather
>>> > pointer-less.
>>> >
>>> > Did it maybe delay evaluating apic->dest_logical that late, it appear
>>> > to be the first usage of that argument?
>>> >
>>> > Ingo, any ideas?
>>>
>>> Yeah, the problem is:
>>>
>>> [ 0.042445] Local APIC not detected. Using dummy APIC emulation.
>>>
>>> we shouldnt be trying to program the lapic in that case - all ops
>>> should be a NOP, including send-ipi-self. Cyrill?
>>
>> Cyrill, the right fix would be to define a:
>>
>> struct apic apic_none;
>>
>> with dummy functions injected. Set struct apic *apic to _that_, instead
>> of apic_default.
>>
>> This means that if we manage to detect a local APIC, we will set it to
>> apic_default (or some other driver) - but the bootup default will be 'no
>> APIC'.
>>
>
> May it wait till evening (i'll back from office and take a look)?
>
>> This is a lot cleaner and a lot less error-prone than the direct
>> frobbing of the function pointers.
>>
>> Ingo
>>
>

Btw, meanwhile Chris may try to pass lapic boot-option in attempt to
reenable apic via msr registers. Also (iirc) i feel we may be hiding
errors if complete noop apic would be used since i belive we need to
check out under which condition a particular operation is called and
when apic is disabled it's mean we're switched to UP mode and
inter-cpu interrupts are under suspicion too. Will take a look during
~6 hours ;)

2009-09-23 08:52:48

[permalink] [raw]

Subject: Re: perf sched record hangs machine

On Wed, 2009-09-23 at 12:48 +0400, Cyrill Gorcunov wrote:
> On 9/23/09, Cyrill Gorcunov <[email protected]> wrote:
> > On 9/23/09, Ingo Molnar <[email protected]> wrote:
> >>
> >> * Ingo Molnar <[email protected]> wrote:
> >>
> >>>
> >>> * Peter Zijlstra <[email protected]> wrote:
> >>>
> >>> > .size default_send_IPI_self, .-default_send_IPI_self
> >>> >
> >>> > Which seems to suggest that cmove is in __prepare_ICR, but I'm not
> >>> > sure how that can cause a page-fault, as that function is rather
> >>> > pointer-less.
> >>> >
> >>> > Did it maybe delay evaluating apic->dest_logical that late, it appear
> >>> > to be the first usage of that argument?
> >>> >
> >>> > Ingo, any ideas?
> >>>
> >>> Yeah, the problem is:
> >>>
> >>> [ 0.042445] Local APIC not detected. Using dummy APIC emulation.
> >>>
> >>> we shouldnt be trying to program the lapic in that case - all ops
> >>> should be a NOP, including send-ipi-self. Cyrill?
> >>
> >> Cyrill, the right fix would be to define a:
> >>
> >> struct apic apic_none;
> >>
> >> with dummy functions injected. Set struct apic *apic to _that_, instead
> >> of apic_default.
> >>
> >> This means that if we manage to detect a local APIC, we will set it to
> >> apic_default (or some other driver) - but the bootup default will be 'no
> >> APIC'.
> >>
> >
> > May it wait till evening (i'll back from office and take a look)?
> >
> >> This is a lot cleaner and a lot less error-prone than the direct
> >> frobbing of the function pointers.
> >>
> >> Ingo
> >>
> >
>
> Btw, meanwhile Chris may try to pass lapic boot-option in attempt to
> reenable apic via msr registers. Also (iirc) i feel we may be hiding
> errors if complete noop apic would be used since i belive we need to
> check out under which condition a particular operation is called and
> when apic is disabled it's mean we're switched to UP mode and
> inter-cpu interrupts are under suspicion too. Will take a look during
> ~6 hours ;)

Thing is, this is a self-ipi, nothing inter-cpu about it.

2009-09-23 09:03:42

[permalink] [raw]

Subject: Re: perf sched record hangs machine

On Wed, 2009-09-23 at 09:32 +0200, Ingo Molnar wrote:
> * Peter Zijlstra <[email protected]> wrote:
>
> > .size default_send_IPI_self, .-default_send_IPI_self
> >
> > Which seems to suggest that cmove is in __prepare_ICR, but I'm not
> > sure how that can cause a page-fault, as that function is rather
> > pointer-less.
> >
> > Did it maybe delay evaluating apic->dest_logical that late, it appear
> > to be the first usage of that argument?
> >
> > Ingo, any ideas?
>
> Yeah, the problem is:
>
> [ 0.042445] Local APIC not detected. Using dummy APIC emulation.

The below seems like the best fix. You disabled all lapic bits, except
that one.

---
arch/x86/kernel/cpu/perf_event.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index a3c7adb..b5801c3 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1790,6 +1790,9 @@ void smp_perf_pending_interrupt(struct pt_regs *regs)
void set_perf_event_pending(void)
{
#ifdef CONFIG_X86_LOCAL_APIC
+ if (!x86_pmu.apic || !x86_pmu_initialized())
+ return;
+
apic->send_IPI_self(LOCAL_PENDING_VECTOR);
#endif
}

2009-09-23 09:15:06

[permalink] [raw]

Subject: Re: perf sched record hangs machine

2009/9/23 Cyrill Gorcunov <[email protected]>:
>
> Btw, meanwhile Chris may try to pass lapic boot-option in attempt to
> reenable apic via msr registers. Also (iirc) i feel we may be hiding
> errors if complete noop apic would be used since i belive we need to
> check out under which condition a particular operation is called and
> when apic is disabled it's mean we're switched to UP mode and
> inter-cpu interrupts are under suspicion too. Will take a look during
> ~6 hours ;)
>

Hi Cyrill

Heh, yes that just occurred to me as well. With the lapic boot option
I can't reproduce the problem, and get a good recording every time.
Don't know why the BIOS had disabled it (can't see any specific
option).

cheers
Chris

2009-09-23 09:20:38

[permalink] [raw]

Subject: Re: perf sched record hangs machine

* Chris Malley <[email protected]> wrote:

> 2009/9/23 Cyrill Gorcunov <[email protected]>:
> >
> > Btw, meanwhile Chris may try to pass lapic boot-option in attempt to
> > reenable apic via msr registers. Also (iirc) i feel we may be hiding
> > errors if complete noop apic would be used since i belive we need to
> > check out under which condition a particular operation is called and
> > when apic is disabled it's mean we're switched to UP mode and
> > inter-cpu interrupts are under suspicion too. Will take a look during
> > ~6 hours ;)
> >
>
> Hi Cyrill
>
> Heh, yes that just occurred to me as well. With the lapic boot option
> I can't reproduce the problem, and get a good recording every time.
> Don't know why the BIOS had disabled it (can't see any specific
> option).

Would still be important to fix the crash - there are boxes where lapics
are disabled permanently and cannot be re-enabled. (plus most people
dont touch their defaults and dont add funky boot options - so crashing
is not an option)

I have such a test-box:

[ 0.000000] Using APIC driver default
[ 0.000000] ACPI: PM-Timer IO Port: 0x8008
[ 0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[ 0.000000] Local APIC disabled by BIOS -- reenabling.
[ 0.000000] Could not enable APIC!
[ 0.000000] APIC: disable apic facility

Btw., perf events can work even without a lapic (albeit without NMI
driven sampling):

[ 0.052051] Performance Events:
[ 0.055138] no APIC, boot with the "lapic" boot parameter to force-enable it.
[ 0.056014] no hardware sampling interrupt available.
[ 0.060014] p6 PMU driver.
[ 0.062955] ... version: 0
[ 0.064014] ... bit width: 32
[ 0.068014] ... generic registers: 2
[ 0.072015] ... value mask: 00000000ffffffff
[ 0.076014] ... max period: 000000007fffffff
[ 0.080014] ... fixed-purpose events: 0
[ 0.084014] ... event mask: 0000000000000003

That's what it did on your box too:

[ 0.013679] Performance Events:
[ 0.013705] no APIC, boot with the "lapic" boot parameter to force-enable it.
[ 0.013783] no hardware sampling interrupt available.
[ 0.013826] p6 PMU driver.
[ 0.013882] ... version: 0
[ 0.013922] ... bit width: 32
[ 0.013962] ... generic registers: 2
[ 0.014002] ... value mask: 00000000ffffffff
[ 0.014045] ... max period: 000000007fffffff
[ 0.014088] ... fixed-purpose events: 0
[ 0.014128] ... event mask: 0000000000000003

Unfortunately i cannot reproduce the crash you've been seeing. (but i'm
quite sure it's due to self-IPI not working fine with dummy lapic.)

Ingo

2009-09-23 09:48:50

[permalink] [raw]

Subject: Re: perf sched record hangs machine

On 9/23/09, Ingo Molnar <[email protected]> wrote:
>
> * Chris Malley <[email protected]> wrote:
>
>> 2009/9/23 Cyrill Gorcunov <[email protected]>:
>> >
>> > Btw, meanwhile Chris may try to pass lapic boot-option in attempt to
>> > reenable apic via msr registers. Also (iirc) i feel we may be hiding
>> > errors if complete noop apic would be used since i belive we need to
>> > check out under which condition a particular operation is called and
>> > when apic is disabled it's mean we're switched to UP mode and
>> > inter-cpu interrupts are under suspicion too. Will take a look during
>> > ~6 hours ;)
>> >
>>
>> Hi Cyrill
>>
>> Heh, yes that just occurred to me as well. With the lapic boot option
>> I can't reproduce the problem, and get a good recording every time.
>> Don't know why the BIOS had disabled it (can't see any specific
>> option).
>
> Would still be important to fix the crash - there are boxes where lapics
> are disabled permanently and cannot be re-enabled. (plus most people
> dont touch their defaults and dont add funky boot options - so crashing
> is not an option)
>

Ingo, Chris, could you try Peter's patch? It seems like what we need.

(Peter, self-ipi shouldn't be separated from others ipi, yes it may
not issue any cycle on fsb, but iirc it uses the same logic as other
ipi use)

> I have such a test-box:
>
> [ 0.000000] Using APIC driver default
> [ 0.000000] ACPI: PM-Timer IO Port: 0x8008
> [ 0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
> [ 0.000000] Local APIC disabled by BIOS -- reenabling.
> [ 0.000000] Could not enable APIC!
> [ 0.000000] APIC: disable apic facility
>
> Btw., perf events can work even without a lapic (albeit without NMI
> driven sampling):
>
> [ 0.052051] Performance Events:
> [ 0.055138] no APIC, boot with the "lapic" boot parameter to force-enable
> it.
> [ 0.056014] no hardware sampling interrupt available.
> [ 0.060014] p6 PMU driver.
> [ 0.062955] ... version: 0
> [ 0.064014] ... bit width: 32
> [ 0.068014] ... generic registers: 2
> [ 0.072015] ... value mask: 00000000ffffffff
> [ 0.076014] ... max period: 000000007fffffff
> [ 0.080014] ... fixed-purpose events: 0
> [ 0.084014] ... event mask: 0000000000000003
>
> That's what it did on your box too:
>
> [ 0.013679] Performance Events:
> [ 0.013705] no APIC, boot with the "lapic" boot parameter to force-enable
> it.
> [ 0.013783] no hardware sampling interrupt available.
> [ 0.013826] p6 PMU driver.
> [ 0.013882] ... version: 0
> [ 0.013922] ... bit width: 32
> [ 0.013962] ... generic registers: 2
> [ 0.014002] ... value mask: 00000000ffffffff
> [ 0.014045] ... max period: 000000007fffffff
> [ 0.014088] ... fixed-purpose events: 0
> [ 0.014128] ... event mask: 0000000000000003
>
> Unfortunately i cannot reproduce the crash you've been seeing. (but i'm
> quite sure it's due to self-IPI not working fine with dummy lapic.)
>
> Ingo
>

2009-09-23 10:13:19

[permalink] [raw]

Subject: Re: perf sched record hangs machine

2009/9/23 Cyrill Gorcunov <[email protected]>:
> On 9/23/09, Ingo Molnar <[email protected]> wrote:
>>
>> Would still be important to fix the crash - there are boxes where lapics
>> are disabled permanently and cannot be re-enabled. (plus most people
>> dont touch their defaults and dont add funky boot options - so crashing
>> is not an option)
>>
>
> Ingo, Chris, could you try Peter's patch? It seems like what we need.
>
> (Peter, self-ipi shouldn't be separated from others ipi, yes it ?may
> not issue any cycle on fsb, but iirc it uses the same logic as other
> ipi use)
>

Applied Peter's patch, doesn't seem to have fixed the problem:

[ 246.408893] BUG: unable to handle kernel paging request at ffffb300
[ 246.408939] IP: [<c011b0bd>] default_send_IPI_self+0x1d/0x50
[ 246.408961] *pde = 0073f067 *pte = 00000000
[ 246.408985] Oops: 0000 [#1] SMP
[ 246.408996] last sysfs file:
/sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
[ 246.409007] Modules linked in: netconsole configfs binfmt_misc
snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm
snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event
snd_seq snd_timer snd_seq_device ipw2200 libipw snd dcdbas cfg80211
intel_agp video soundcore sr_mod lib80211 output joydev pcspkr
snd_page_alloc agpgart usb_storage usbhid ohci1394 tg3 ieee1394
[ 246.409112]
[ 246.409121] Pid: 4188, comm: firefox Not tainted
(2.6.31-cjm-07092-g819307a #4) Latitude D400
[ 246.409126] EIP: 0060:[<c011b0bd>] EFLAGS: 00010046 CPU: 0
[ 246.409131] EIP is at default_send_IPI_self+0x1d/0x50
[ 246.409135] EAX: fffff000 EBX: 000000ec ECX: 00000800 EDX: ffffb300
[ 246.409140] ESI: f16cdc64 EDI: 00000000 EBP: f16cdc00 ESP: f16cdbfc
[ 246.409144] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[ 246.409150] Process firefox (pid: 4188, ti=f16cc000 task=f1465aa0
task.ti=f16cc000)
[ 246.409154] Stack:
[ 246.409158] f16c3e14 f16cdc08 c010e3b4 f16cdc28 c01b9751 f1602024
f1602020 00115838
[ 246.409179] <0> 00000000 f1602000 f16c2c00 f16cdc38 c01b981a
f16cdc64 f16cdc84 f16cdc98
[ 246.409199] <0> c01ba690 f16c2c00 00000001 c030963e ffffffff
ffffffff 00000000 00000001
[ 246.409223] Call Trace:
[ 246.409234] [<c010e3b4>] ? set_perf_event_pending+0x14/0x20
[ 246.409244] [<c01b9751>] ? perf_output_unlock+0x121/0x1a0
[ 246.409249] [<c01b981a>] ? perf_output_end+0x4a/0x70
[ 246.409255] [<c01ba690>] ? __perf_event_overflow+0x240/0x2f0
[ 246.409264] [<c030963e>] ? atomic64_cmpxchg+0x1e/0x30
[ 246.409270] [<c01ba8f4>] ? perf_swevent_ctx_event+0x1b4/0x1c0
[ 246.409276] [<c01ba773>] ? perf_swevent_ctx_event+0x33/0x1c0
[ 246.409281] [<c01ba9a7>] ? do_perf_sw_event+0xa7/0x160
[ 246.409286] [<c01baae2>] ? perf_tp_event+0x82/0xa0
[ 246.409296] [<c012e9c6>] ? ftrace_profile_sched_stat_runtime+0xe6/0x120
[ 246.409301] [<c012e8e0>] ? ftrace_profile_sched_stat_runtime+0x0/0x120
[ 246.409307] [<c013c85a>] ? update_curr+0x18a/0x230
[ 246.409313] [<c013e965>] ? enqueue_entity+0x15/0x460
[ 246.409319] [<c0132447>] ? task_rq_lock+0x47/0x80
[ 246.409324] [<c013f2d1>] ? enqueue_task_fair+0x31/0x70
[ 246.409331] [<c012acad>] ? enqueue_task+0x6d/0x90
[ 246.409336] [<c012ae50>] ? activate_task+0x20/0x30
[ 246.409343] [<c013beeb>] ? try_to_wake_up+0x1fb/0x2f0
[ 246.409351] [<c015ef50>] ? hrtimer_wakeup+0x0/0x20
[ 246.409357] [<c013c00f>] ? wake_up_process+0xf/0x20
[ 246.409365] [<c015ef68>] ? hrtimer_wakeup+0x18/0x20
[ 246.409370] [<c015efdc>] ? __run_hrtimer+0x6c/0xc0
[ 246.409379] [<c04e748a>] ? _spin_lock+0x3a/0x40
[ 246.409384] [<c015f2f5>] ? hrtimer_interrupt+0x185/0x230
[ 246.409391] [<c010564c>] ? timer_interrupt+0x3c/0x50
[ 246.409402] [<c0199bd0>] ? handle_IRQ_event+0x50/0x140
[ 246.409407] [<c04e7335>] ? _spin_unlock_irqrestore+0x55/0x60
[ 246.409413] [<c019bfa4>] ? handle_level_irq+0x64/0xf0
[ 246.409418] [<c019bfae>] ? handle_level_irq+0x6e/0xf0
[ 246.409423] [<c01050da>] ? handle_irq+0x1a/0x30
[ 246.409428] [<c0104896>] ? do_IRQ+0x46/0xc0
[ 246.409437] [<c016f3cc>] ? trace_hardirqs_on_caller+0x12c/0x170
[ 246.409442] [<c010372e>] ? common_interrupt+0x2e/0x34
[ 246.409448] Code: 0f 44 c1 89 02 5b 5d c3 8d b6 00 00 00 00 55 89
e5 53 89 c3 a1 5c de 68 c0 8b 48 20 eb 02 f3 90 a1 c8 10 69 c0 8d 90
00 c3 ff ff <8b> 80 00 c3 ff ff f6 c4 10 75 e8 89 c8 81 c9 00 04 04 00
0d 00
[ 246.409591] EIP: [<c011b0bd>] default_send_IPI_self+0x1d/0x50
SS:ESP 0068:f16cdbfc
[ 246.409601] CR2: 00000000ffffb300
[ 246.409609] ---[ end trace 237505c339f73345 ]---
[ 246.409616] Kernel panic - not syncing: Fatal exception in interrupt
[ 246.409623] Pid: 4188, comm: firefox Tainted: G D
2.6.31-cjm-07092-g819307a #4
[ 246.409627] Call Trace:
[ 246.409633] [<c04e3eb5>] ? printk+0x18/0x1b
[ 246.409638] [<c04e3de0>] panic+0x43/0x100
[ 246.409643] [<c04e8569>] oops_end+0xb9/0xc0
[ 246.409648] [<c0124d66>] no_context+0xb6/0x150
[ 246.409653] [<c0124e63>] __bad_area_nosemaphore+0x63/0x180
[ 246.409659] [<c016fb13>] ? __lock_acquire+0x193/0x1240
[ 246.409664] [<c016fb13>] ? __lock_acquire+0x193/0x1240
[ 246.409670] [<c016fb13>] ? __lock_acquire+0x193/0x1240
[ 246.409675] [<c016fb13>] ? __lock_acquire+0x193/0x1240
[ 246.409680] [<c0124f92>] bad_area_nosemaphore+0x12/0x20
[ 246.409687] [<c04e9b4c>] do_page_fault+0x31c/0x3c0
[ 246.409692] [<c04e9830>] ? do_page_fault+0x0/0x3c0
[ 246.409697] [<c04e79d3>] error_code+0x6b/0x70
[ 246.409703] [<c016007b>] ? down_write_trylock+0x1b/0x50
[ 246.409708] [<c04e9830>] ? do_page_fault+0x0/0x3c0
[ 246.409714] [<c011b0bd>] ? default_send_IPI_self+0x1d/0x50
[ 246.409720] [<c010e3b4>] set_perf_event_pending+0x14/0x20
[ 246.409725] [<c01b9751>] perf_output_unlock+0x121/0x1a0
[ 246.409732] [<c01b981a>] perf_output_end+0x4a/0x70

2009-09-23 10:12:31

[permalink] [raw]

Subject: Re: perf sched record hangs machine

On Wednesday, September 23, 2009, Chris Malley <[email protected]> wrote:
> 2009/9/23 Cyrill Gorcunov <[email protected]>:
>> On 9/23/09, Ingo Molnar <[email protected]> wrote:
>>>
>>> Would still be important to fix the crash - there are boxes where lapics
>>> are disabled permanently and cannot be re-enabled. (plus most people
>>> dont touch their defaults and dont add funky boot options - so crashing
>>> is not an option)
>>>
>>
>> Ingo, Chris, could you try Peter's patch? It seems like what we need.
>>
>> (Peter, self-ipi shouldn't be separated from others ipi, yes it ?may
>> not issue any cycle on fsb, but iirc it uses the same logic as other
>> ipi use)
>>
>
> Applied Peter's patch, doesn't seem to have fixed the problem:
>

thanks Chris! I'll take a look on this today evening (if someone
wouldn't beat me ;)

> [ ?246.408893] BUG: unable to handle kernel paging request at ffffb300
> [ ?246.408939] IP: [<c011b0bd>] default_send_IPI_self+0x1d/0x50
> [ ?246.408961] *pde = 0073f067 *pte = 00000000
> [ ?246.408985] Oops: 0000 [#1] SMP
> [ ?246.408996] last sysfs file:
> /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
> [ ?246.409007] Modules linked in: netconsole configfs binfmt_misc
> snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm
> snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event
> snd_seq snd_timer snd_seq_device ipw2200 libipw snd dcdbas cfg80211
> intel_agp video soundcore sr_mod lib80211 output joydev pcspkr
> snd_page_alloc agpgart usb_storage usbhid ohci1394 tg3 ieee1394
> [ ?246.409112]
> [ ?246.409121] Pid: 4188, comm: firefox Not tainted
> (2.6.31-cjm-07092-g819307a #4) Latitude D400
> [ ?246.409126] EIP: 0060:[<c011b0bd>] EFLAGS: 00010046 CPU: 0
> [ ?246.409131] EIP is at default_send_IPI_self+0x1d/0x50
> [ ?246.409135] EAX: fffff000 EBX: 000000ec ECX: 00000800 EDX: ffffb300
> [ ?246.409140] ESI: f16cdc64 EDI: 00000000 EBP: f16cdc00 ESP: f16cdbfc
> [ ?246.409144] ?DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
> [ ?246.409150] Process firefox (pid: 4188, ti=f16cc000 task=f1465aa0
> task.ti=f16cc000)
> [ ?246.409154] Stack:
> [ ?246.409158] ?f16c3e14 f16cdc08 c010e3b4 f16cdc28 c01b9751 f1602024
> f1602020 00115838
> [ ?246.409179] <0> 00000000 f1602000 f16c2c00 f16cdc38 c01b981a
> f16cdc64 f16cdc84 f16cdc98
> [ ?246.409199] <0> c01ba690 f16c2c00 00000001 c030963e ffffffff
> ffffffff 00000000 00000001
> [ ?246.409223] Call Trace:
> [ ?246.409234] ?[<c010e3b4>] ? set_perf_event_pending+0x14/0x20
> [ ?246.409244] ?[<c01b9751>] ? perf_output_unlock+0x121/0x1a0
> [ ?246.409249] ?[<c01b981a>] ? perf_output_end+0x4a/0x70
> [ ?246.409255] ?[<c01ba690>] ? __perf_event_overflow+0x240/0x2f0
> [ ?246.409264] ?[<c030963e>] ? atomic64_cmpxchg+0x1e/0x30
> [ ?246.409270] ?[<c01ba8f4>] ? perf_swevent_ctx_event+0x1b4/0x1c0
> [ ?246.409276] ?[<c01ba773>] ? perf_swevent_ctx_event+0x33/0x1c0
> [ ?246.409281] ?[<c01ba9a7>] ? do_perf_sw_event+0xa7/0x160
> [ ?246.409286] ?[<c01baae2>] ? perf_tp_event+0x82/0xa0
> [ ?246.409296] ?[<c012e9c6>] ? ftrace_profile_sched_stat_runtime+0xe6/0x120
> [ ?246.409301] ?[<c012e8e0>] ? ftrace_profile_sched_stat_runtime+0x0/0x120
> [ ?246.409307] ?[<c013c85a>] ? update_curr+0x18a/0x230
> [ ?246.409313] ?[<c013e965>] ? enqueue_entity+0x15/0x460
> [ ?246.409319] ?[<c0132447>] ? task_rq_lock+0x47/0x80
> [ ?246.409324] ?[<c013f2d1>] ? enqueue_task_fair+0x31/0x70
> [ ?246.409331] ?[<c012acad>] ? enqueue_task+0x6d/0x90
> [ ?246.409336] ?[<c012ae50>] ? activate_task+0x20/0x30
> [ ?246.409343] ?[<c013beeb>] ? try_to_wake_up+0x1fb/0x2f0
> [ ?246.409351] ?[<c015ef50>] ? hrtimer_wakeup+0x0/0x20
> [ ?246.409357] ?[<c013c00f>] ? wake_up_process+0xf/0x20
> [ ?246.409365] ?[<c015ef68>] ? hrtimer_wakeup+0x18/0x20
> [ ?246.409370] ?[<c015efdc>] ? __run_hrtimer+0x6c/0xc0
> [ ?246.409379] ?[<c04e748a>] ? _spin_lock+0x3a/0x40
> [ ?246.409384] ?[<c015f2f5>] ? hrtimer_interrupt+0x185/0x230
> [ ?246.409391] ?[<c010564c>] ? timer_interrupt+0x3c/0x50
> [ ?246.409402] ?[<c0199bd0>] ? handle_IRQ_event+0x50/0x140
> [ ?246.409407] ?[<c04e7335>] ? _spin_unlock_irqrestore+0x55/0x60
> [ ?246.409413] ?[<c019bfa4>] ? handle_level_irq+0x64/0xf0
> [ ?246.409418] ?[<c019bfae>] ? handle_level_irq+0x6e/0xf0
> [ ?246.409423] ?[<c01050da>] ? handle_irq+0x1a/0x30
> [ ?246.409428] ?[<c0104896>] ? do_IRQ+0x46/0xc0
> [ ?246.409437] ?[<c016f3cc>] ? trace_hardirqs_on_caller+0x12c/0x170
> [ ?246.409442] ?[<c010372e>] ? common_interrupt+0x2e/0x34
> [ ?246.409448] Code: 0f 44 c1 89 02 5b 5d c3 8d b6 00 00 00 00 55 89
> e5 53 89 c3 a1 5c de 68 c0 8b 48 20 eb 02 f3 90 a1 c8 10 69 c0 8d 90
> 00 c3 ff ff <8b> 80 00 c3 ff ff f6 c4 10 75 e8 89 c8 81 c9 00 04 04 00
> 0d 00
> [ ?246.409591] EIP: [<c011b0bd>] default_send_IPI_self+0x1d/0x50
> SS:ESP 0068:f16cdbfc
> [ ?246.409601] CR2: 00000000ffffb300
> [ ?246.409609] ---[ end trace 237505c339f73345 ]---
> [ ?246.409616] Kernel panic - not syncing: Fatal exception in interrupt
> [ ?246.409623] Pid: 4188, comm: firefox Tainted: G ? ? ?D
> 2.6.31-cjm-07092-g819307a #4
> [ ?246.409627] Call Trace:
> [ ?246.409633] ?[<c04e3eb5>] ? printk+0x18/0x1b
> [ ?246.409638] ?[<c04e3de0>] panic+0x43/0x100
> [ ?246.409643] ?[<c04e8569>] oops_end+0xb9/0xc0
> [ ?246.409648] ?[<c0124d66>] no_context+0xb6/0x150
> [ ?246.409653] ?[<c0124e63>] __bad_area_nosemaphore+0x63/0x180
> [ ?246.409659] ?[<c016fb13>] ? __lock_acquire+0x193/0x1240
> [ ?246.409664] ?[<c016fb13>] ? __lock_acquire+0x193/0x1240
> [ ?246.409670] ?[<c016fb13>] ? __lock_acquire+0x193/0x1240
> [ ?246.409675] ?[<c016fb13>] ? __lock_acquire+0x193/0x1240
> [ ?246.409680] ?[<c0124f92>] bad_area_nosemaphore+0x12/0x20
> [ ?246.409687] ?[<c04e9b4c>] do_page_fault+0x31c/0x3c0
> [ ?246.409692] ?[<c04e9830>] ? do_page_fault+0x0/0x3c0
> [ ?246.409697] ?[<c04e79d3>] error_code+0x6b/0x70
> [ ?246.409703] ?[<c016007b>] ? down_write_trylock+0x1b/0x50
> [ ?246.409708] ?[<c04e9830>] ? do_page_fault+0x0/0x3c0
> [ ?246.409714] ?[<c011b0bd>] ? default_send_IPI_self+0x1d/0x50
> [ ?246.409720] ?[<c010e3b4>] set_perf_event_pending+0x14/0x20
> [ ?246.409725] ?[<c01b9751>] perf_output_unlock+0x121/0x1a0
> [ ?246.409732] ?[<c01b981a>] perf_output_end+0x4a/0x70
>

2009-09-23 10:41:12

[permalink] [raw]

Subject: Re: perf sched record hangs machine

* Cyrill Gorcunov <[email protected]> wrote:

> On 9/23/09, Ingo Molnar <[email protected]> wrote:
> >
> > * Chris Malley <[email protected]> wrote:
> >
> >> 2009/9/23 Cyrill Gorcunov <[email protected]>:
> >> >
> >> > Btw, meanwhile Chris may try to pass lapic boot-option in attempt to
> >> > reenable apic via msr registers. Also (iirc) i feel we may be hiding
> >> > errors if complete noop apic would be used since i belive we need to
> >> > check out under which condition a particular operation is called and
> >> > when apic is disabled it's mean we're switched to UP mode and
> >> > inter-cpu interrupts are under suspicion too. Will take a look during
> >> > ~6 hours ;)
> >> >
> >>
> >> Hi Cyrill
> >>
> >> Heh, yes that just occurred to me as well. With the lapic boot option
> >> I can't reproduce the problem, and get a good recording every time.
> >> Don't know why the BIOS had disabled it (can't see any specific
> >> option).
> >
> > Would still be important to fix the crash - there are boxes where lapics
> > are disabled permanently and cannot be re-enabled. (plus most people
> > dont touch their defaults and dont add funky boot options - so crashing
> > is not an option)
> >
>
> Ingo, Chris, could you try Peter's patch? It seems like what we need.

It should fix the problem, but also the underlying uncleanliness should
be fixed as well. We need an apic_none driver template with safe
methods. Then all the scattered 'is the lapic enabled' open-coded checks
of global flags can be removed.

Ingo

2009-09-23 10:42:49

[permalink] [raw]

Subject: Re: perf sched record hangs machine

* Chris Malley <[email protected]> wrote:

> Applied Peter's patch, doesn't seem to have fixed the problem:
>
> [ 246.408893] BUG: unable to handle kernel paging request at ffffb300
> [ 246.408939] IP: [<c011b0bd>] default_send_IPI_self+0x1d/0x50

could you please send the full bootlog, up to the crash?

Thanks,

Ingo

2009-09-23 11:41:47

[permalink] [raw]

Subject: Re: perf sched record hangs machine

2009/9/23 Ingo Molnar <[email protected]>:
>
> * Chris Malley <[email protected]> wrote:
>
>> Applied Peter's patch, doesn't seem to have fixed the problem:
>>
>> [ ?246.408893] BUG: unable to handle kernel paging request at ffffb300
>> [ ?246.408939] IP: [<c011b0bd>] default_send_IPI_self+0x1d/0x50
>
> could you please send the full bootlog, up to the crash?
>
> Thanks,
>
> ? ? ? ?Ingo
>

Sure, but I'm just rebuilding from clean first in case the change
wasn't picked up for some reason, because I don't see how it can get
to default_send_IPI_self() in my configuration...

Chris

2009-09-23 11:43:08

[permalink] [raw]

Subject: Re: perf sched record hangs machine

* Chris Malley <[email protected]> wrote:

> 2009/9/23 Ingo Molnar <[email protected]>:
> >
> > * Chris Malley <[email protected]> wrote:
> >
> >> Applied Peter's patch, doesn't seem to have fixed the problem:
> >>
> >> [ ?246.408893] BUG: unable to handle kernel paging request at ffffb300
> >> [ ?246.408939] IP: [<c011b0bd>] default_send_IPI_self+0x1d/0x50
> >
> > could you please send the full bootlog, up to the crash?
> >
> > Thanks,
> >
> > ? ? ? ?Ingo
> >
>
> Sure, but I'm just rebuilding from clean first in case the change
> wasn't picked up for some reason, because I don't see how it can get
> to default_send_IPI_self() in my configuration...

yeah - i have that thinking too - according to your earlier boot
messages x86_pmu.apic must be 0 and that codepath shouldnt trigger.

Ingo

2009-09-23 11:49:52

[permalink] [raw]

Subject: [tip:perf/urgent] perf_event, x86: Fix 'perf sched record' crashing the machine

Commit-ID: 7d42896628202a551ad1107697cd215dc5fca099
Gitweb: http://git.kernel.org/tip/7d42896628202a551ad1107697cd215dc5fca099
Author: Peter Zijlstra <[email protected]>
AuthorDate: Wed, 23 Sep 2009 11:03:37 +0200
Committer: Ingo Molnar <[email protected]>
CommitDate: Wed, 23 Sep 2009 11:25:56 +0200

perf_event, x86: Fix 'perf sched record' crashing the machine

Chris Malley reported that 'perf sched record' sometimes
crashes his box with:

[ 389.272175] BUG: unable to handle kernel paging request at ffffb300
[ 389.272294] IP: [<c011b0bd>] default_send_IPI_self+0x1d/0x50
[ 389.272366] *pde = 0073f067 *pte = 00000000
[ 389.274708] Call Trace:
[ 389.274752] [<c010e3b4>] ? set_perf_event_pending+0x14/0x20
[ 389.274801] [<c01b9751>] ? perf_output_unlock+0x121/0x1a0
[ 389.274848] [<c01b981a>] ? perf_output_end+0x4a/0x70
[ 389.274893] [<c01ba690>] ? __perf_event_overflow+0x240/0x2f0
[ 389.274942] [<c030963e>] ? atomic64_cmpxchg+0x1e/0x30
[ 389.274988] [<c01ba8f4>] ? perf_swevent_ctx_event+0x1b4/0x1c0
[ 389.275035] [<c01ba773>] ? perf_swevent_ctx_event+0x33/0x1c0
[ 389.275081] [<c01ba9a7>] ? do_perf_sw_event+0xa7/0x160
[ 389.275127] [<c01baae2>] ? perf_tp_event+0x82/0xa0
[ 389.275174] [<c012e9c6>] ? ftrace_profile_sched_stat_runtime+0xe6/0x120
[ 389.275224] [<c012e8e0>] ? ftrace_profile_sched_stat_runtime+0x0/0x120
[ 389.275273] [<c013c85a>] ? update_curr+0x18a/0x230
[ 389.275318] [<c013cdc5>] ? put_prev_task_fair+0x155/0x160
[ 389.275366] [<c01618b5>] ? sched_clock_cpu+0xd5/0x110
[ 389.275413] [<c04e7525>] ? _spin_lock_irq+0x45/0x50
[ 389.275458] [<c04e424e>] ? schedule+0x20e/0xb10

The problem is that the box has no lapic enabled:

[ 0.042445] Local APIC not detected. Using dummy APIC emulation.

The below seems like the best fix. We disabled all lapic bits, except
the self-IPI-resend logic.

Reported-by: Chris Malley <[email protected]>
Signed-off-by: Peter Zijlstra <[email protected]>
Cc: Cyrill Gorcunov <[email protected]>
Cc: Frederic Weisbecker <[email protected]>
LKML-Reference: <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>

---
arch/x86/kernel/cpu/perf_event.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index a3c7adb..b5801c3 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1790,6 +1790,9 @@ void smp_perf_pending_interrupt(struct pt_regs *regs)
void set_perf_event_pending(void)
{
#ifdef CONFIG_X86_LOCAL_APIC
+ if (!x86_pmu.apic || !x86_pmu_initialized())
+ return;
+
apic->send_IPI_self(LOCAL_PENDING_VECTOR);
#endif
}

2009-09-23 12:15:05

[permalink] [raw]

Subject: Re: [tip:perf/urgent] perf_event, x86: Fix 'perf sched record' crashing the machine

2009/9/23 tip-bot for Peter Zijlstra <[email protected]>:
> Commit-ID: ?7d42896628202a551ad1107697cd215dc5fca099
> Gitweb: ? ? http://git.kernel.org/tip/7d42896628202a551ad1107697cd215dc5fca099
> Author: ? ? Peter Zijlstra <[email protected]>
> AuthorDate: Wed, 23 Sep 2009 11:03:37 +0200
> Committer: ?Ingo Molnar <[email protected]>
> CommitDate: Wed, 23 Sep 2009 11:25:56 +0200
>
> perf_event, x86: Fix 'perf sched record' crashing the machine
>
> Chris Malley reported that 'perf sched record' sometimes
> crashes his box with:
>
> [ ?389.272175] BUG: unable to handle kernel paging request at ffffb300
> [ ?389.272294] IP: [<c011b0bd>] default_send_IPI_self+0x1d/0x50
> [ ?389.272366] *pde = 0073f067 *pte = 00000000
> [ ?389.274708] Call Trace:
> [ ?389.274752] ?[<c010e3b4>] ? ?set_perf_event_pending+0x14/0x20
> [ ?389.274801] ?[<c01b9751>] ? ?perf_output_unlock+0x121/0x1a0
> [ ?389.274848] ?[<c01b981a>] ? perf_output_end+0x4a/0x70
> [ ?389.274893] ?[<c01ba690>] ? ?__perf_event_overflow+0x240/0x2f0
> [ ?389.274942] ?[<c030963e>] ? atomic64_cmpxchg+0x1e/0x30
> [ ?389.274988] ?[<c01ba8f4>] ? ?perf_swevent_ctx_event+0x1b4/0x1c0
> [ ?389.275035] ?[<c01ba773>] ? ?perf_swevent_ctx_event+0x33/0x1c0
> [ ?389.275081] ?[<c01ba9a7>] ? do_perf_sw_event+0xa7/0x160
> [ ?389.275127] ?[<c01baae2>] ? perf_tp_event+0x82/0xa0
> [ ?389.275174] ?[<c012e9c6>] ? ?ftrace_profile_sched_stat_runtime+0xe6/0x120
> [ ?389.275224] ?[<c012e8e0>] ? ?ftrace_profile_sched_stat_runtime+0x0/0x120
> [ ?389.275273] ?[<c013c85a>] ? update_curr+0x18a/0x230
> [ ?389.275318] ?[<c013cdc5>] ? ?put_prev_task_fair+0x155/0x160
> [ ?389.275366] ?[<c01618b5>] ? sched_clock_cpu+0xd5/0x110
> [ ?389.275413] ?[<c04e7525>] ? _spin_lock_irq+0x45/0x50
> [ ?389.275458] ?[<c04e424e>] ? schedule+0x20e/0xb10
>
> The problem is that the box has no lapic enabled:
>
> ?[ ? ?0.042445] Local APIC not detected. Using dummy APIC emulation.
>
> The below seems like the best fix. We disabled all lapic bits, except
> the self-IPI-resend logic.
>
> Reported-by: Chris Malley <[email protected]>
> Signed-off-by: Peter Zijlstra <[email protected]>
> Cc: Cyrill Gorcunov <[email protected]>
> Cc: Frederic Weisbecker <[email protected]>
> LKML-Reference: <[email protected]>
> Signed-off-by: Ingo Molnar <[email protected]>
>
>
> ---
> ?arch/x86/kernel/cpu/perf_event.c | ? ?3 +++
> ?1 files changed, 3 insertions(+), 0 deletions(-)
>
> diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
> index a3c7adb..b5801c3 100644
> --- a/arch/x86/kernel/cpu/perf_event.c
> +++ b/arch/x86/kernel/cpu/perf_event.c
> @@ -1790,6 +1790,9 @@ void smp_perf_pending_interrupt(struct pt_regs *regs)
> ?void set_perf_event_pending(void)
> ?{
> ?#ifdef CONFIG_X86_LOCAL_APIC
> + ? ? ? if (!x86_pmu.apic || !x86_pmu_initialized())
> + ? ? ? ? ? ? ? return;
> +
> ? ? ? ?apic->send_IPI_self(LOCAL_PENDING_VECTOR);
> ?#endif
> ?}
>

Confirmed, this fixes the problem, so FWIW:

Tested-by: Chris Malley <[email protected]>

2009-09-23 12:20:45

[permalink] [raw]

Subject: Re: perf sched record hangs machine

2009/9/23 Ingo Molnar <[email protected]>:
>
> * Chris Malley <[email protected]> wrote:
>
>> 2009/9/23 Ingo Molnar <[email protected]>:
>> >
>> > * Chris Malley <[email protected]> wrote:
>> >
>> >> Applied Peter's patch, doesn't seem to have fixed the problem:
>> >>
>> >> [ ?246.408893] BUG: unable to handle kernel paging request at ffffb300
>> >> [ ?246.408939] IP: [<c011b0bd>] default_send_IPI_self+0x1d/0x50
>> >
>> > could you please send the full bootlog, up to the crash?
>> >
>> > Thanks,
>> >
>> > ? ? ? ?Ingo
>> >
>>
>> Sure, but I'm just rebuilding from clean first in case the change
>> wasn't picked up for some reason, because I don't see how it can get
>> to default_send_IPI_self() in my configuration...
>
> yeah - i have that thinking too - according to your earlier boot
> messages x86_pmu.apic must be 0 and that codepath shouldnt trigger.
>
> ? ? ? ?Ingo
>

Yep, my bad. The build process had helpfully appended a -dirty tag to
the kernel version which I hadn't noticed.

I can't make it hang any more, so the fix looks good to me, thanks everyone.

Chris

2009-09-23 12:27:34