Hi,
Linux-next booted with the below warnings on powerpc
Test: Reboot
Machine Type : Power 8 bare-metal
Kernel version : 4.13.0-rc4-next-20170808
gcc : 4.8.5
config: Tul-NV-config file attached
Issue is rare to hit (found once for 3 retries)
A WARN_ON_ONCE is being triggered from function __blk_mq_run_hw_queue in
file block/blk-mq.c at line 1111
which is :
static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
{
int srcu_idx;
/*
* We should be running this queue from one of the CPUs that
* are mapped to it.
*/
WARN_ON(!cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask)
&&
cpu_online(hctx->next_cpu));
/*
* We can't run the queue inline with ints disabled. Ensure that
* we catch bad users of this early.
*/
>>> WARN_ON_ONCE(in_interrupt());
if (!(hctx->flags & BLK_MQ_F_BLOCKING)) {
rcu_read_lock();
boot warnings:
--------------
kvm: exiting hardware virtualization
------------[ cut here ]------------
WARNING: CPU: 15 PID: 0 at block/blk-mq.c:1111 __blk_mq_run_hw_queue
+0x1d8/0x1f0
Modules linked in: iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4
iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp tun bridge
stp llc kvm_hv kvm iptable_filter vmx_crypto ipmi_powernv leds_powernv
led_class powernv_rng ipmi_devintf ipmi_msghandler rng_core
powernv_op_panel binfmt_misc nfsd ip_tables x_tables autofs4
CPU: 15 PID: 0 Comm: swapper/15 Not tainted 4.13.0-rc4-next-20170808 #4
task: c0000007f8439000 task.stack: c0000007f84e0000
NIP: c00000000088f4d8 LR: c00000000088f7b0 CTR: c000000000dafcc0
REGS: c000000ffff376d0 TRAP: 0700 Not tainted
(4.13.0-rc4-next-20170808)
MSR: 9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>
CR: 42004022 XER: 00000000
CFAR: c00000000088f3c4 SOFTE: 1
GPR00: c00000000088f7b0 c000000ffff37950 c000000001d8ff00
c0000007eb707c00
GPR04: 0000000000000000 0000000000000000 c0000000022fff00
c00000000224ff00
GPR08: c00000000224ff00 0000000000000001 0000000000000100
9000000000001003
GPR12: 0000000000004400 c00000000fd45280 c0000007f84e3f90
0000000000200042
GPR16: 0000000100009ad1 c000000ffff34000 0000000000000000
c0000000015c4e80
GPR20: c000000001dc3b00 c0000000015c4e80 000000000000000a
c000000ffff34000
GPR24: 0000000000000000 c0000007eb3e1818 c000000ffff37a70
0000000000000001
GPR28: c0000007eb3e0000 0000000000000000 c0000007eb707c00
c0000007eb707c00
NIP [c00000000088f4d8] __blk_mq_run_hw_queue+0x1d8/0x1f0
LR [c00000000088f7b0] __blk_mq_delay_run_hw_queue+0x1f0/0x210
Call Trace:
[c000000ffff37990] [c00000000088f7b0] __blk_mq_delay_run_hw_queue
+0x1f0/0x210
[c000000ffff379d0] [c00000000088fcb8] blk_mq_start_hw_queue+0x58/0x80
[c000000ffff379f0] [c00000000088fd40] blk_mq_start_hw_queues+0x60/0xb0
[c000000ffff37a30] [c000000000ae2b54] scsi_kick_queue+0x34/0xa0
[c000000ffff37a50] [c000000000ae2f70] scsi_run_queue+0x3b0/0x660
[c000000ffff37ac0] [c000000000ae7ed4] scsi_run_host_queues+0x64/0xc0
[c000000ffff37b00] [c000000000ae7f64] scsi_unblock_requests+0x34/0x60
[c000000ffff37b20] [c000000000b14998] ipr_ioa_bringdown_done+0xf8/0x3a0
[c000000ffff37bc0] [c000000000b12528] ipr_reset_ioa_job+0xd8/0x170
[c000000ffff37c00] [c000000000b18790] ipr_reset_timer_done+0x110/0x160
[c000000ffff37c50] [c00000000024db50] call_timer_fn+0xa0/0x3a0
[c000000ffff37ce0] [c00000000024e058] expire_timers+0x1b8/0x350
[c000000ffff37d50] [c00000000024e2f0] run_timer_softirq+0x100/0x3e0
[c000000ffff37df0] [c000000000162edc] __do_softirq+0x20c/0x620
[c000000ffff37ee0] [c000000000163a80] irq_exit+0x230/0x290
[c000000ffff37f10] [c00000000001d770] __do_irq+0x170/0x410
[c000000ffff37f90] [c00000000003ea20] call_do_irq+0x14/0x24
[c0000007f84e3a70] [c00000000001dae0] do_IRQ+0xd0/0x190
[c0000007f84e3ac0] [c000000000008c58] hardware_interrupt_common
+0x158/0x160
--- interrupt: 501 at .L1^B42+0x0/0x4
LR = arch_local_irq_restore+0x124/0x160
[c0000007f84e3db0] [c00000000001c9c8] arch_local_irq_restore+0xa8/0x160
(unreliable)
[c0000007f84e3dd0] [c000000000db5038] cpuidle_enter_state+0x238/0x6e0
[c0000007f84e3e30] [c000000000db5588] cpuidle_enter+0x38/0x60
[c0000007f84e3e50] [c0000000001f22e4] call_cpuidle+0x74/0xe0
[c0000007f84e3e70] [c0000000001f2a78] do_idle+0x4b8/0x5a0
[c0000007f84e3ee0] [c0000000001f2f64] cpu_startup_entry+0x74/0x90
[c0000007f84e3f20] [c000000000068c14] start_secondary+0x4a4/0x550
[c0000007f84e3f90] [c00000000000b16c] start_secondary_prolog+0x10/0x14
Instruction dump:
e9280e58 eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 39290001 f9280e58 4e800020
3ce2004c e9270e38 39290001 f9270e38 <0fe00000> 3d02004c e9280e40
39290001
---[ end trace 5632db71d3bf5b30 ]---
WARN_ON_ONCE(in_interrupt()) was first introduced in the commit :
commit b7a71e66d4d274d627cabc17c5e41330bcf47c2d
Author: Jens Axboe <[email protected]>
Date: Tue Aug 1 09:28:24 2017 -0600
blk-mq: add warning to __blk_mq_run_hw_queue() for ints disabled
We recently had a bug in the IPR SCSI driver, where it would end up
making the SCSI mid layer run the mq hardware queue with interrupts
disabled. This isn't legal, since the software queue locking relies
on never being grabbed from interrupt context. Additionally, drivers
that set BLK_MQ_F_BLOCKING may schedule from this context.
Add a WARN_ON_ONCE() to catch bad users up front.
Signed-off-by: Jens Axboe <[email protected]>
--
Regard's
Abdul Haleem
IBM Linux Technology Centre
On Wed, 2017-08-16 at 23:37 +0530, Abdul Haleem wrote:
> Linux-next booted with the below warnings on powerpc
>
> [ ... ]
>
> boot warnings:
> --------------
> kvm: exiting hardware virtualization
> ------------[ cut here ]------------
> WARNING: CPU: 15 PID: 0 at block/blk-mq.c:1111 __blk_mq_run_hw_queue
> +0x1d8/0x1f0
> Modules linked in: iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4
> Call Trace:
> [c000000ffff37990] [c00000000088f7b0] __blk_mq_delay_run_hw_queue
> +0x1f0/0x210
> [c000000ffff379d0] [c00000000088fcb8] blk_mq_start_hw_queue+0x58/0x80
> [c000000ffff379f0] [c00000000088fd40] blk_mq_start_hw_queues+0x60/0xb0
> [c000000ffff37a30] [c000000000ae2b54] scsi_kick_queue+0x34/0xa0
> [c000000ffff37a50] [c000000000ae2f70] scsi_run_queue+0x3b0/0x660
> [c000000ffff37ac0] [c000000000ae7ed4] scsi_run_host_queues+0x64/0xc0
> [c000000ffff37b00] [c000000000ae7f64] scsi_unblock_requests+0x34/0x60
> [c000000ffff37b20] [c000000000b14998] ipr_ioa_bringdown_done+0xf8/0x3a0
> [c000000ffff37bc0] [c000000000b12528] ipr_reset_ioa_job+0xd8/0x170
> [c000000ffff37c00] [c000000000b18790] ipr_reset_timer_done+0x110/0x160
> [c000000ffff37c50] [c00000000024db50] call_timer_fn+0xa0/0x3a0
> [c000000ffff37ce0] [c00000000024e058] expire_timers+0x1b8/0x350
> [c000000ffff37d50] [c00000000024e2f0] run_timer_softirq+0x100/0x3e0
> [c000000ffff37df0] [c000000000162edc] __do_softirq+0x20c/0x620
> [c000000ffff37ee0] [c000000000163a80] irq_exit+0x230/0x290
> [c000000ffff37f10] [c00000000001d770] __do_irq+0x170/0x410
> [c000000ffff37f90] [c00000000003ea20] call_do_irq+0x14/0x24
> [c0000007f84e3a70] [c00000000001dae0] do_IRQ+0xd0/0x190
> [c0000007f84e3ac0] [c000000000008c58] hardware_interrupt_common
> +0x158/0x160
Hello Brian,
In the MAINTAINERS file I found the following:
IBM Power Linux RAID adapter
M: Brian King <[email protected]>
S: Supported
F: drivers/scsi/ipr.*
Is that information up-to-date? Do you agree that the above message indicates
a bug in the ipr driver?
Thanks,
Bart.
On 08/16/2017 01:15 PM, Bart Van Assche wrote:
> On Wed, 2017-08-16 at 23:37 +0530, Abdul Haleem wrote:
>> Linux-next booted with the below warnings on powerpc
>>
>> [ ... ]
>>
>> boot warnings:
>> --------------
>> kvm: exiting hardware virtualization
>> ------------[ cut here ]------------
>> WARNING: CPU: 15 PID: 0 at block/blk-mq.c:1111 __blk_mq_run_hw_queue
>> +0x1d8/0x1f0
>> Modules linked in: iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4
>> Call Trace:
>> [c000000ffff37990] [c00000000088f7b0] __blk_mq_delay_run_hw_queue
>> +0x1f0/0x210
>> [c000000ffff379d0] [c00000000088fcb8] blk_mq_start_hw_queue+0x58/0x80
>> [c000000ffff379f0] [c00000000088fd40] blk_mq_start_hw_queues+0x60/0xb0
>> [c000000ffff37a30] [c000000000ae2b54] scsi_kick_queue+0x34/0xa0
>> [c000000ffff37a50] [c000000000ae2f70] scsi_run_queue+0x3b0/0x660
>> [c000000ffff37ac0] [c000000000ae7ed4] scsi_run_host_queues+0x64/0xc0
>> [c000000ffff37b00] [c000000000ae7f64] scsi_unblock_requests+0x34/0x60
>> [c000000ffff37b20] [c000000000b14998] ipr_ioa_bringdown_done+0xf8/0x3a0
>> [c000000ffff37bc0] [c000000000b12528] ipr_reset_ioa_job+0xd8/0x170
>> [c000000ffff37c00] [c000000000b18790] ipr_reset_timer_done+0x110/0x160
>> [c000000ffff37c50] [c00000000024db50] call_timer_fn+0xa0/0x3a0
>> [c000000ffff37ce0] [c00000000024e058] expire_timers+0x1b8/0x350
>> [c000000ffff37d50] [c00000000024e2f0] run_timer_softirq+0x100/0x3e0
>> [c000000ffff37df0] [c000000000162edc] __do_softirq+0x20c/0x620
>> [c000000ffff37ee0] [c000000000163a80] irq_exit+0x230/0x290
>> [c000000ffff37f10] [c00000000001d770] __do_irq+0x170/0x410
>> [c000000ffff37f90] [c00000000003ea20] call_do_irq+0x14/0x24
>> [c0000007f84e3a70] [c00000000001dae0] do_IRQ+0xd0/0x190
>> [c0000007f84e3ac0] [c000000000008c58] hardware_interrupt_common
>> +0x158/0x160
>
> Hello Brian,
>
> In the MAINTAINERS file I found the following:
>
> IBM Power Linux RAID adapter
> M: Brian King <[email protected]>
> S: Supported
> F: drivers/scsi/ipr.*
>
> Is that information up-to-date? Do you agree that the above message indicates
> a bug in the ipr driver?
Yes. Can you try with this patch that is in 4.13/scsi-fixes:
https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=4.13/scsi-fixes&id=b0e17a9b0df29590c45dfb296f541270a5941f41
Thanks,
Brian
--
Brian King
Power Linux I/O
IBM Linux Technology Center
On Wed, 2017-08-16 at 15:10 -0500, Brian King wrote:
> On 08/16/2017 01:15 PM, Bart Van Assche wrote:
> > On Wed, 2017-08-16 at 23:37 +0530, Abdul Haleem wrote:
> > > Linux-next booted with the below warnings on powerpc
> > >
> > > [ ... ]
> > >
> > > boot warnings:
> > > --------------
> > > kvm: exiting hardware virtualization
> > > ------------[ cut here ]------------
> > > WARNING: CPU: 15 PID: 0 at block/blk-mq.c:1111 __blk_mq_run_hw_queue
> > > +0x1d8/0x1f0
> > > Modules linked in: iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4
> > > Call Trace:
> > > [c000000ffff37990] [c00000000088f7b0] __blk_mq_delay_run_hw_queue
> > > +0x1f0/0x210
> > > [c000000ffff379d0] [c00000000088fcb8] blk_mq_start_hw_queue+0x58/0x80
> > > [c000000ffff379f0] [c00000000088fd40] blk_mq_start_hw_queues+0x60/0xb0
> > > [c000000ffff37a30] [c000000000ae2b54] scsi_kick_queue+0x34/0xa0
> > > [c000000ffff37a50] [c000000000ae2f70] scsi_run_queue+0x3b0/0x660
> > > [c000000ffff37ac0] [c000000000ae7ed4] scsi_run_host_queues+0x64/0xc0
> > > [c000000ffff37b00] [c000000000ae7f64] scsi_unblock_requests+0x34/0x60
> > > [c000000ffff37b20] [c000000000b14998] ipr_ioa_bringdown_done+0xf8/0x3a0
> > > [c000000ffff37bc0] [c000000000b12528] ipr_reset_ioa_job+0xd8/0x170
> > > [c000000ffff37c00] [c000000000b18790] ipr_reset_timer_done+0x110/0x160
> > > [c000000ffff37c50] [c00000000024db50] call_timer_fn+0xa0/0x3a0
> > > [c000000ffff37ce0] [c00000000024e058] expire_timers+0x1b8/0x350
> > > [c000000ffff37d50] [c00000000024e2f0] run_timer_softirq+0x100/0x3e0
> > > [c000000ffff37df0] [c000000000162edc] __do_softirq+0x20c/0x620
> > > [c000000ffff37ee0] [c000000000163a80] irq_exit+0x230/0x290
> > > [c000000ffff37f10] [c00000000001d770] __do_irq+0x170/0x410
> > > [c000000ffff37f90] [c00000000003ea20] call_do_irq+0x14/0x24
> > > [c0000007f84e3a70] [c00000000001dae0] do_IRQ+0xd0/0x190
> > > [c0000007f84e3ac0] [c000000000008c58] hardware_interrupt_common
> > > +0x158/0x160
> >
> > Hello Brian,
> >
> > In the MAINTAINERS file I found the following:
> >
> > IBM Power Linux RAID adapter
> > M: Brian King <[email protected]>
> > S: Supported
> > F: drivers/scsi/ipr.*
> >
> > Is that information up-to-date? Do you agree that the above message indicates
> > a bug in the ipr driver?
>
> Yes. Can you try with this patch that is in 4.13/scsi-fixes:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=4.13/scsi-fixes&id=b0e17a9b0df29590c45dfb296f541270a5941f41
Hello Brian,
Sorry but I don't have access to a setup on which I can test the ipr driver ...
Bart.
On 08/17/2017 10:32 AM, Bart Van Assche wrote:
> On Wed, 2017-08-16 at 15:10 -0500, Brian King wrote:
>> On 08/16/2017 01:15 PM, Bart Van Assche wrote:
>>> On Wed, 2017-08-16 at 23:37 +0530, Abdul Haleem wrote:
>>>> Linux-next booted with the below warnings on powerpc
>>>>
>>>> [ ... ]
>>>>
>>>> boot warnings:
>>>> --------------
>>>> kvm: exiting hardware virtualization
>>>> ------------[ cut here ]------------
>>>> WARNING: CPU: 15 PID: 0 at block/blk-mq.c:1111 __blk_mq_run_hw_queue
>>>> +0x1d8/0x1f0
>>>> Modules linked in: iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4
>>>> Call Trace:
>>>> [c000000ffff37990] [c00000000088f7b0] __blk_mq_delay_run_hw_queue
>>>> +0x1f0/0x210
>>>> [c000000ffff379d0] [c00000000088fcb8] blk_mq_start_hw_queue+0x58/0x80
>>>> [c000000ffff379f0] [c00000000088fd40] blk_mq_start_hw_queues+0x60/0xb0
>>>> [c000000ffff37a30] [c000000000ae2b54] scsi_kick_queue+0x34/0xa0
>>>> [c000000ffff37a50] [c000000000ae2f70] scsi_run_queue+0x3b0/0x660
>>>> [c000000ffff37ac0] [c000000000ae7ed4] scsi_run_host_queues+0x64/0xc0
>>>> [c000000ffff37b00] [c000000000ae7f64] scsi_unblock_requests+0x34/0x60
>>>> [c000000ffff37b20] [c000000000b14998] ipr_ioa_bringdown_done+0xf8/0x3a0
>>>> [c000000ffff37bc0] [c000000000b12528] ipr_reset_ioa_job+0xd8/0x170
>>>> [c000000ffff37c00] [c000000000b18790] ipr_reset_timer_done+0x110/0x160
>>>> [c000000ffff37c50] [c00000000024db50] call_timer_fn+0xa0/0x3a0
>>>> [c000000ffff37ce0] [c00000000024e058] expire_timers+0x1b8/0x350
>>>> [c000000ffff37d50] [c00000000024e2f0] run_timer_softirq+0x100/0x3e0
>>>> [c000000ffff37df0] [c000000000162edc] __do_softirq+0x20c/0x620
>>>> [c000000ffff37ee0] [c000000000163a80] irq_exit+0x230/0x290
>>>> [c000000ffff37f10] [c00000000001d770] __do_irq+0x170/0x410
>>>> [c000000ffff37f90] [c00000000003ea20] call_do_irq+0x14/0x24
>>>> [c0000007f84e3a70] [c00000000001dae0] do_IRQ+0xd0/0x190
>>>> [c0000007f84e3ac0] [c000000000008c58] hardware_interrupt_common
>>>> +0x158/0x160
>>>
>>> Hello Brian,
>>>
>>> In the MAINTAINERS file I found the following:
>>>
>>> IBM Power Linux RAID adapter
>>> M: Brian King <[email protected]>
>>> S: Supported
>>> F: drivers/scsi/ipr.*
>>>
>>> Is that information up-to-date? Do you agree that the above message indicates
>>> a bug in the ipr driver?
>>
>> Yes. Can you try with this patch that is in 4.13/scsi-fixes:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=4.13/scsi-fixes&id=b0e17a9b0df29590c45dfb296f541270a5941f41
>
> Hello Brian,
>
> Sorry but I don't have access to a setup on which I can test the ipr driver ...
Understood. That request was intended for Abdul, who reported the issue.
I should have just cc'ed you...
Thanks,
Brian
--
Brian King
Power Linux I/O
IBM Linux Technology Center
On Thu, 2017-08-17 at 14:18 -0500, Brian King wrote:
> On 08/17/2017 10:32 AM, Bart Van Assche wrote:
> > On Wed, 2017-08-16 at 15:10 -0500, Brian King wrote:
> >> On 08/16/2017 01:15 PM, Bart Van Assche wrote:
> >>> On Wed, 2017-08-16 at 23:37 +0530, Abdul Haleem wrote:
> >>>> Linux-next booted with the below warnings on powerpc
> >>>>
> >>>> [ ... ]
> >>>>
> >>>> boot warnings:
> >>>> --------------
> >>>> kvm: exiting hardware virtualization
> >>>> ------------[ cut here ]------------
> >>>> WARNING: CPU: 15 PID: 0 at block/blk-mq.c:1111 __blk_mq_run_hw_queue
> >>>> +0x1d8/0x1f0
> >>>> Modules linked in: iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4
> >>>> Call Trace:
> >>>> [c000000ffff37990] [c00000000088f7b0] __blk_mq_delay_run_hw_queue
> >>>> +0x1f0/0x210
> >>>> [c000000ffff379d0] [c00000000088fcb8] blk_mq_start_hw_queue+0x58/0x80
> >>>> [c000000ffff379f0] [c00000000088fd40] blk_mq_start_hw_queues+0x60/0xb0
> >>>> [c000000ffff37a30] [c000000000ae2b54] scsi_kick_queue+0x34/0xa0
> >>>> [c000000ffff37a50] [c000000000ae2f70] scsi_run_queue+0x3b0/0x660
> >>>> [c000000ffff37ac0] [c000000000ae7ed4] scsi_run_host_queues+0x64/0xc0
> >>>> [c000000ffff37b00] [c000000000ae7f64] scsi_unblock_requests+0x34/0x60
> >>>> [c000000ffff37b20] [c000000000b14998] ipr_ioa_bringdown_done+0xf8/0x3a0
> >>>> [c000000ffff37bc0] [c000000000b12528] ipr_reset_ioa_job+0xd8/0x170
> >>>> [c000000ffff37c00] [c000000000b18790] ipr_reset_timer_done+0x110/0x160
> >>>> [c000000ffff37c50] [c00000000024db50] call_timer_fn+0xa0/0x3a0
> >>>> [c000000ffff37ce0] [c00000000024e058] expire_timers+0x1b8/0x350
> >>>> [c000000ffff37d50] [c00000000024e2f0] run_timer_softirq+0x100/0x3e0
> >>>> [c000000ffff37df0] [c000000000162edc] __do_softirq+0x20c/0x620
> >>>> [c000000ffff37ee0] [c000000000163a80] irq_exit+0x230/0x290
> >>>> [c000000ffff37f10] [c00000000001d770] __do_irq+0x170/0x410
> >>>> [c000000ffff37f90] [c00000000003ea20] call_do_irq+0x14/0x24
> >>>> [c0000007f84e3a70] [c00000000001dae0] do_IRQ+0xd0/0x190
> >>>> [c0000007f84e3ac0] [c000000000008c58] hardware_interrupt_common
> >>>> +0x158/0x160
> >>>
> >>> Hello Brian,
> >>>
> >>> In the MAINTAINERS file I found the following:
> >>>
> >>> IBM Power Linux RAID adapter
> >>> M: Brian King <[email protected]>
> >>> S: Supported
> >>> F: drivers/scsi/ipr.*
> >>>
> >>> Is that information up-to-date? Do you agree that the above message indicates
> >>> a bug in the ipr driver?
> >>
> >> Yes. Can you try with this patch that is in 4.13/scsi-fixes:
> >>
> >> https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=4.13/scsi-fixes&id=b0e17a9b0df29590c45dfb296f541270a5941f41
> >
Hi Brian,
The patch fixes the warning, Thanks for the fix.
Tested-by : Abdul Haleem <[email protected]>
--
Regard's
Abdul Haleem
IBM Linux Technology Centre