Hi,
Writing to /proc/sys/kernel/watchdog_thresh causes the following BUG in
at least v3.13-rc2-625-g06151db, v3.15 and v3.16-rc1. Kernel config is
attached.
It was originally found on bare metal, since then reproduced in QEMU in
init, and when directly executing it.
Regards,
Peter
[ 0.676701] BUG: sleeping function called from invalid context at /tmp/linux/mm/slub.c:965
[ 0.679396] in_atomic(): 1, irqs_disabled(): 0, pid: 1, name: init
[ 0.681204] 3 locks held by init/1:
[ 0.682371] #0: (sb_writers#3){.+.+.+}, at: [<ffffffff8117b663>] vfs_write+0x143/0x180
[ 0.685887] #1: (watchdog_proc_mutex){+.+.+.}, at: [<ffffffff810e02d3>] proc_dowatchdog+0x33/0x110
[ 0.689631] #2: (cpu_hotplug.lock){.+.+.+}, at: [<ffffffff810589c2>] get_online_cpus+0x32/0x80
[ 0.693117] Preemption disabled at:[<ffffffff810e0384>] proc_dowatchdog+0xe4/0x110
[ 0.695753]
[ 0.696588] CPU: 0 PID: 1 Comm: init Not tainted 3.16.0-rc1-testing #34
[ 0.698404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[ 0.704622] ffff88003f250000 ffff88003f1ffd10 ffffffff81624c59 0000000000000000
[ 0.707749] ffff88003f1ffd30 ffffffff8108df1d 0000000000000010 ffffffff81c56360
[ 0.711010] ffff88003f1ffd78 ffffffff8116a5de ffffffff811159a5 ffffffff810a5f4a
[ 0.714053] Call Trace:
[ 0.715015] [<ffffffff81624c59>] dump_stack+0x4e/0x7a
[ 0.716619] [<ffffffff8108df1d>] __might_sleep+0x11d/0x190
[ 0.718232] [<ffffffff8116a5de>] kmem_cache_alloc_trace+0x4e/0x1e0
[ 0.720214] [<ffffffff811159a5>] ? perf_event_alloc+0x55/0x440
[ 0.721910] [<ffffffff810a5f4a>] ? mark_held_locks+0x6a/0x90
[ 0.723558] [<ffffffff811159a5>] perf_event_alloc+0x55/0x440
[ 0.725304] [<ffffffff810e0050>] ? restart_watchdog_hrtimer+0x50/0x50
[ 0.727279] [<ffffffff81116d16>] perf_event_create_kernel_counter+0x26/0xe0
[ 0.729269] [<ffffffff810dfe05>] watchdog_nmi_enable+0x75/0x140
[ 0.730965] [<ffffffff810dffb3>] update_timers_all_cpus+0x53/0xa0
[ 0.732953] [<ffffffff810e0384>] proc_dowatchdog+0xe4/0x110
[ 0.738408] [<ffffffff811d8c93>] proc_sys_call_handler+0xb3/0xc0
[ 0.740266] [<ffffffff811d8cb4>] proc_sys_write+0x14/0x20
[ 0.742086] [<ffffffff8117b5cd>] vfs_write+0xad/0x180
[ 0.743669] [<ffffffff810a606d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
[ 0.745593] [<ffffffff8117b799>] SyS_write+0x49/0xb0
[ 0.747101] [<ffffffff8162fe92>] system_call_fastpath+0x16/0x1b
[ 0.749069] NMI watchdog: disabled (cpu0): hardware events not enabled
On Mon, Jun 16, 2014 at 04:12:44PM +0200, Peter Wu wrote:
> Hi,
>
> Writing to /proc/sys/kernel/watchdog_thresh causes the following BUG in
> at least v3.13-rc2-625-g06151db, v3.15 and v3.16-rc1. Kernel config is
> attached.
>
> It was originally found on bare metal, since then reproduced in QEMU in
> init, and when directly executing it.
>
> Regards,
> Peter
Hi Peter,
I assume the following patch will work?
Michal, do you remember why we needed preempt here? I wouldn't think it
mattered as we are not doing anything per-cpu specific.
Cheers,
Don
diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index 516203e..30e4822 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -527,10 +527,8 @@ static void update_timers_all_cpus(void)
int cpu;
get_online_cpus();
- preempt_disable();
for_each_online_cpu(cpu)
update_timers(cpu);
- preempt_enable();
put_online_cpus();
}
On Monday 16 June 2014 16:52:45 Don Zickus wrote:
> On Mon, Jun 16, 2014 at 04:12:44PM +0200, Peter Wu wrote:
> > Hi,
> >
> > Writing to /proc/sys/kernel/watchdog_thresh causes the following BUG in
> > at least v3.13-rc2-625-g06151db, v3.15 and v3.16-rc1. Kernel config is
> > attached.
> >
> > It was originally found on bare metal, since then reproduced in QEMU in
> > init, and when directly executing it.
> >
> > Regards,
> > Peter
>
> Hi Peter,
>
> I assume the following patch will work?
Yes, the patch works (no more BUG). No idea whether it is
(in)correct though.
Regards,
Peter
> Michal, do you remember why we needed preempt here? I wouldn't think it
> mattered as we are not doing anything per-cpu specific.
>
> Cheers,
> Don
>
> diff --git a/kernel/watchdog.c b/kernel/watchdog.c
> index 516203e..30e4822 100644
> --- a/kernel/watchdog.c
> +++ b/kernel/watchdog.c
> @@ -527,10 +527,8 @@ static void update_timers_all_cpus(void)
> int cpu;
>
> get_online_cpus();
> - preempt_disable();
> for_each_online_cpu(cpu)
> update_timers(cpu);
> - preempt_enable();
> put_online_cpus();
> }
>
On Mon 16-06-14 16:52:45, Don Zickus wrote:
[...]
> Michal, do you remember why we needed preempt here? I wouldn't think it
> mattered as we are not doing anything per-cpu specific.
Yes, you are right. Preempt can be enabled because we are always
accessing watchdog_ev on a specific cpu so reschedule doesn't matter.
> Cheers,
> Don
>
> diff --git a/kernel/watchdog.c b/kernel/watchdog.c
> index 516203e..30e4822 100644
> --- a/kernel/watchdog.c
> +++ b/kernel/watchdog.c
> @@ -527,10 +527,8 @@ static void update_timers_all_cpus(void)
> int cpu;
>
> get_online_cpus();
> - preempt_disable();
> for_each_online_cpu(cpu)
> update_timers(cpu);
> - preempt_enable();
> put_online_cpus();
> }
>
--
Michal Hocko
SUSE Labs