On Wed, Aug 17, 2022, at 12:30 PM, Jan Kara wrote:
> BTW, are you sure the first problematic kernel is 5.12?
100%
It consistently reproduces with any 5.12 series kernel, including from c03c21ba6f4e which is before rc1. It's frustrating that git bisect produces kernels that won't boot, I was more than half way through! :D And could have been done by now...
We've been running on 5.11 series kernels for a year because of this problem.
> BTW that may be an
> interesting thing to try: Can you boot with
> "megaraid_sas.host_tagset_enable = 0" kernel option and see whether the
> issue reproduces?
Yep.
--
Chris Murphy
On Wed, Aug 17, 2022, at 12:47 PM, Chris Murphy wrote:
Can you boot with
>> "megaraid_sas.host_tagset_enable = 0" kernel option and see whether the
>> issue reproduces?
This has been running an hour without symptoms. It's strongly suggestive, but needs to run overnight to be sure. Anecdotally, the max write IO is less than what I'm used to seeing.
[ 0.583121] Kernel command line: BOOT_IMAGE=(md/0)/vmlinuz-5.12.5-300.fc34.x86_64 root=UUID=04f1fb7f-5cc4-4dfb-a7cf-b6b6925bf895 ro rootflags=subvol=root rd.md.uuid=e7782150:092e161a:68395862:31375bca biosdevname=1 net.ifnames=0 log_buf_len=8M plymouth.enable=0 megaraid_sas.host_tagset_enable=0
...
[ 6.745964] megasas: 07.714.04.00-rc1
[ 6.758472] megaraid_sas 0000:02:00.0: BAR:0x1 BAR's base_addr(phys):0x0000000092000000 mapped virt_addr:0x00000000c54554ff
[ 6.758477] megaraid_sas 0000:02:00.0: FW now in Ready state
[ 6.770658] megaraid_sas 0000:02:00.0: 63 bit DMA mask and 32 bit consistent mask
[ 6.795060] megaraid_sas 0000:02:00.0: firmware supports msix : (96)
[ 6.807537] megaraid_sas 0000:02:00.0: requested/available msix 49/49
[ 6.819259] megaraid_sas 0000:02:00.0: current msix/online cpus : (49/48)
[ 6.830800] megaraid_sas 0000:02:00.0: RDPQ mode : (disabled)
[ 6.842031] megaraid_sas 0000:02:00.0: Current firmware supports maximum commands: 928 LDIO threshold: 0
[ 6.871246] megaraid_sas 0000:02:00.0: Performance mode :Latency (latency index = 1)
[ 6.882265] megaraid_sas 0000:02:00.0: FW supports sync cache : No
[ 6.893034] megaraid_sas 0000:02:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[ 6.988550] megaraid_sas 0000:02:00.0: FW provided supportMaxExtLDs: 1 max_lds: 64
[ 6.988554] megaraid_sas 0000:02:00.0: controller type : MR(2048MB)
[ 6.988555] megaraid_sas 0000:02:00.0: Online Controller Reset(OCR) : Enabled
[ 6.988556] megaraid_sas 0000:02:00.0: Secure JBOD support : No
[ 6.988557] megaraid_sas 0000:02:00.0: NVMe passthru support : No
[ 6.988558] megaraid_sas 0000:02:00.0: FW provided TM TaskAbort/Reset timeout : 0 secs/0 secs
[ 6.988559] megaraid_sas 0000:02:00.0: JBOD sequence map support : No
[ 6.988560] megaraid_sas 0000:02:00.0: PCI Lane Margining support : No
[ 7.025160] megaraid_sas 0000:02:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[ 7.025162] megaraid_sas 0000:02:00.0: INIT adapter done
[ 7.025164] megaraid_sas 0000:02:00.0: JBOD sequence map is disabled megasas_setup_jbod_map 5707
[ 7.029878] megaraid_sas 0000:02:00.0: pci id : (0x1000)/(0x005d)/(0x1028)/(0x1f47)
[ 7.029881] megaraid_sas 0000:02:00.0: unevenspan support : yes
[ 7.029882] megaraid_sas 0000:02:00.0: firmware crash dump : no
[ 7.029883] megaraid_sas 0000:02:00.0: JBOD sequence map : disabled
[ 7.029915] megaraid_sas 0000:02:00.0: Max firmware commands: 927 shared with nr_hw_queues = 1
[ 7.029918] scsi host11: Avago SAS based MegaRAID driver
--
Chris Murphy
On Wed 17-08-22 13:57:00, Chris Murphy wrote:
> On Wed, Aug 17, 2022, at 12:47 PM, Chris Murphy wrote:
> Can you boot with
> >> "megaraid_sas.host_tagset_enable = 0" kernel option and see whether the
> >> issue reproduces?
>
> This has been running an hour without symptoms. It's strongly suggestive,
> but needs to run overnight to be sure. Anecdotally, the max write IO is
> less than what I'm used to seeing.
OK, if this indeed passes then b6e68ee82585 ("blk-mq: Improve performance
of non-mq IO schedulers with multiple HW queues") might be what's causing
issues (although I don't know how yet...).
Honza
>
> [ 0.583121] Kernel command line: BOOT_IMAGE=(md/0)/vmlinuz-5.12.5-300.fc34.x86_64 root=UUID=04f1fb7f-5cc4-4dfb-a7cf-b6b6925bf895 ro rootflags=subvol=root rd.md.uuid=e7782150:092e161a:68395862:31375bca biosdevname=1 net.ifnames=0 log_buf_len=8M plymouth.enable=0 megaraid_sas.host_tagset_enable=0
> ...
> [ 6.745964] megasas: 07.714.04.00-rc1
> [ 6.758472] megaraid_sas 0000:02:00.0: BAR:0x1 BAR's base_addr(phys):0x0000000092000000 mapped virt_addr:0x00000000c54554ff
> [ 6.758477] megaraid_sas 0000:02:00.0: FW now in Ready state
> [ 6.770658] megaraid_sas 0000:02:00.0: 63 bit DMA mask and 32 bit consistent mask
> [ 6.795060] megaraid_sas 0000:02:00.0: firmware supports msix : (96)
> [ 6.807537] megaraid_sas 0000:02:00.0: requested/available msix 49/49
> [ 6.819259] megaraid_sas 0000:02:00.0: current msix/online cpus : (49/48)
> [ 6.830800] megaraid_sas 0000:02:00.0: RDPQ mode : (disabled)
> [ 6.842031] megaraid_sas 0000:02:00.0: Current firmware supports maximum commands: 928 LDIO threshold: 0
> [ 6.871246] megaraid_sas 0000:02:00.0: Performance mode :Latency (latency index = 1)
> [ 6.882265] megaraid_sas 0000:02:00.0: FW supports sync cache : No
> [ 6.893034] megaraid_sas 0000:02:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
> [ 6.988550] megaraid_sas 0000:02:00.0: FW provided supportMaxExtLDs: 1 max_lds: 64
> [ 6.988554] megaraid_sas 0000:02:00.0: controller type : MR(2048MB)
> [ 6.988555] megaraid_sas 0000:02:00.0: Online Controller Reset(OCR) : Enabled
> [ 6.988556] megaraid_sas 0000:02:00.0: Secure JBOD support : No
> [ 6.988557] megaraid_sas 0000:02:00.0: NVMe passthru support : No
> [ 6.988558] megaraid_sas 0000:02:00.0: FW provided TM TaskAbort/Reset timeout : 0 secs/0 secs
> [ 6.988559] megaraid_sas 0000:02:00.0: JBOD sequence map support : No
> [ 6.988560] megaraid_sas 0000:02:00.0: PCI Lane Margining support : No
> [ 7.025160] megaraid_sas 0000:02:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
> [ 7.025162] megaraid_sas 0000:02:00.0: INIT adapter done
> [ 7.025164] megaraid_sas 0000:02:00.0: JBOD sequence map is disabled megasas_setup_jbod_map 5707
> [ 7.029878] megaraid_sas 0000:02:00.0: pci id : (0x1000)/(0x005d)/(0x1028)/(0x1f47)
> [ 7.029881] megaraid_sas 0000:02:00.0: unevenspan support : yes
> [ 7.029882] megaraid_sas 0000:02:00.0: firmware crash dump : no
> [ 7.029883] megaraid_sas 0000:02:00.0: JBOD sequence map : disabled
> [ 7.029915] megaraid_sas 0000:02:00.0: Max firmware commands: 927 shared with nr_hw_queues = 1
> [ 7.029918] scsi host11: Avago SAS based MegaRAID driver
>
>
>
>
> --
> Chris Murphy
--
Jan Kara <[email protected]>
SUSE Labs, CR
On Wed, Aug 17, 2022, at 2:15 PM, Jan Kara wrote:
> OK, if this indeed passes then b6e68ee82585 ("blk-mq: Improve performance
> of non-mq IO schedulers with multiple HW queues") might be what's causing
> issues (although I don't know how yet...).
I can revert it from 5.12.0 and try. Let me know which next test is preferred :)
--
Chris Murphy
On 2022-08-17 20:15, Jan Kara wrote:
> On Wed 17-08-22 13:57:00, Chris Murphy wrote:
>> On Wed, Aug 17, 2022, at 12:47 PM, Chris Murphy wrote:
>> Can you boot with
>>>> "megaraid_sas.host_tagset_enable = 0" kernel option and see whether the
>>>> issue reproduces?
>>
>> This has been running an hour without symptoms. It's strongly suggestive,
>> but needs to run overnight to be sure. Anecdotally, the max write IO is
>> less than what I'm used to seeing.
>
> OK, if this indeed passes then b6e68ee82585 ("blk-mq: Improve performance
> of non-mq IO schedulers with multiple HW queues") might be what's causing
> issues (although I don't know how yet...).
>
> Honza
Certainly explains why BFQ turned up as a suspect, considering it's still
single-queue (fair MQ scheduling is .. complicated).
-h
On Wed 17-08-22 14:18:01, Chris Murphy wrote:
>
>
> On Wed, Aug 17, 2022, at 2:15 PM, Jan Kara wrote:
>
> > OK, if this indeed passes then b6e68ee82585 ("blk-mq: Improve performance
> > of non-mq IO schedulers with multiple HW queues") might be what's causing
> > issues (although I don't know how yet...).
>
> I can revert it from 5.12.0 and try. Let me know which next test is preferred :)
Let's try to revert this first so that we have it narrowed down what
started causing the issues.
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR