Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 17 Aug 2022 20:15:54 +0200
From:   Jan Kara <jack@suse.cz>
To:     Chris Murphy <lists@colorremedies.com>
Cc:     Jan Kara <jack@suse.cz>,
        Holger =?utf-8?Q?Hoffst=C3=A4tte?= 
        <holger@applied-asynchrony.com>,
        Nikolay Borisov <nborisov@suse.com>,
        Jens Axboe <axboe@kernel.dk>,
        Paolo Valente <paolo.valente@linaro.org>,
        Linux-RAID <linux-raid@vger.kernel.org>,
        linux-block <linux-block@vger.kernel.org>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Josef Bacik <josef@toxicpanda.com>
Subject: Re: stalling IO regression since linux 5.12, through 5.18
Message-ID: <20220817181554.znqljc6mmci45ukd@quack3>
References: <61e5ccda-a527-4fea-9850-91095ffa91c4@www.fastmail.com>
 <4995baed-c561-421d-ba3e-3a75d6a738a3@www.fastmail.com>
 <dcd8beea-d2d9-e692-6e5d-c96b2d29dfd1@suse.com>
 <2b8a38fa-f15f-45e8-8caa-61c5f8cd52de@www.fastmail.com>
 <7c830487-95a6-b008-920b-8bc4a318f10a@applied-asynchrony.com>
 <20220817114933.66c4g4xjsi4df2tg@quack3>
 <85a141ae-56a7-4dcd-b75a-04be4b276b3a@www.fastmail.com>
 <20220817163059.kigrvdfmxfswmhls@quack3>
 <f6f899a5-97e2-460f-ad73-73d4b5e38eb6@www.fastmail.com>
 <51cd43f9-ab6b-4dd6-814f-e0c1ace3143c@www.fastmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <51cd43f9-ab6b-4dd6-814f-e0c1ace3143c@www.fastmail.com>
Precedence: bulk

On Wed 17-08-22 13:57:00, Chris Murphy wrote:
> On Wed, Aug 17, 2022, at 12:47 PM, Chris Murphy wrote:
> Can you boot with
> >> "megaraid_sas.host_tagset_enable = 0" kernel option and see whether the
> >> issue reproduces?
> 
> This has been running an hour without symptoms. It's strongly suggestive,
> but needs to run overnight to be sure. Anecdotally, the max write IO is
> less than what I'm used to seeing.

OK, if this indeed passes then b6e68ee82585 ("blk-mq: Improve performance
of non-mq IO schedulers with multiple HW queues") might be what's causing
issues (although I don't know how yet...).

								Honza

> 
> [    0.583121] Kernel command line: BOOT_IMAGE=(md/0)/vmlinuz-5.12.5-300.fc34.x86_64 root=UUID=04f1fb7f-5cc4-4dfb-a7cf-b6b6925bf895 ro rootflags=subvol=root rd.md.uuid=e7782150:092e161a:68395862:31375bca biosdevname=1 net.ifnames=0 log_buf_len=8M plymouth.enable=0 megaraid_sas.host_tagset_enable=0
> ...
> [    6.745964] megasas: 07.714.04.00-rc1
> [    6.758472] megaraid_sas 0000:02:00.0: BAR:0x1  BAR's base_addr(phys):0x0000000092000000  mapped virt_addr:0x00000000c54554ff
> [    6.758477] megaraid_sas 0000:02:00.0: FW now in Ready state
> [    6.770658] megaraid_sas 0000:02:00.0: 63 bit DMA mask and 32 bit consistent mask
> [    6.795060] megaraid_sas 0000:02:00.0: firmware supports msix	: (96)
> [    6.807537] megaraid_sas 0000:02:00.0: requested/available msix 49/49
> [    6.819259] megaraid_sas 0000:02:00.0: current msix/online cpus	: (49/48)
> [    6.830800] megaraid_sas 0000:02:00.0: RDPQ mode	: (disabled)
> [    6.842031] megaraid_sas 0000:02:00.0: Current firmware supports maximum commands: 928	 LDIO threshold: 0
> [    6.871246] megaraid_sas 0000:02:00.0: Performance mode :Latency (latency index = 1)
> [    6.882265] megaraid_sas 0000:02:00.0: FW supports sync cache	: No
> [    6.893034] megaraid_sas 0000:02:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
> [    6.988550] megaraid_sas 0000:02:00.0: FW provided supportMaxExtLDs: 1	max_lds: 64
> [    6.988554] megaraid_sas 0000:02:00.0: controller type	: MR(2048MB)
> [    6.988555] megaraid_sas 0000:02:00.0: Online Controller Reset(OCR)	: Enabled
> [    6.988556] megaraid_sas 0000:02:00.0: Secure JBOD support	: No
> [    6.988557] megaraid_sas 0000:02:00.0: NVMe passthru support	: No
> [    6.988558] megaraid_sas 0000:02:00.0: FW provided TM TaskAbort/Reset timeout	: 0 secs/0 secs
> [    6.988559] megaraid_sas 0000:02:00.0: JBOD sequence map support	: No
> [    6.988560] megaraid_sas 0000:02:00.0: PCI Lane Margining support	: No
> [    7.025160] megaraid_sas 0000:02:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
> [    7.025162] megaraid_sas 0000:02:00.0: INIT adapter done
> [    7.025164] megaraid_sas 0000:02:00.0: JBOD sequence map is disabled megasas_setup_jbod_map 5707
> [    7.029878] megaraid_sas 0000:02:00.0: pci id		: (0x1000)/(0x005d)/(0x1028)/(0x1f47)
> [    7.029881] megaraid_sas 0000:02:00.0: unevenspan support	: yes
> [    7.029882] megaraid_sas 0000:02:00.0: firmware crash dump	: no
> [    7.029883] megaraid_sas 0000:02:00.0: JBOD sequence map	: disabled
> [    7.029915] megaraid_sas 0000:02:00.0: Max firmware commands: 927 shared with nr_hw_queues = 1
> [    7.029918] scsi host11: Avago SAS based MegaRAID driver
> 
> 
> 
> 
> -- 
> Chris Murphy
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR