Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20221108025602.E48005EC04F9@us226.sjc.aristanetworks.com> <Y2qJE2Bpbvnf5Ejb@kbusch-mbp>
In-Reply-To: <Y2qJE2Bpbvnf5Ejb@kbusch-mbp>
From:   Francesco Ruggeri <fruggeri@arista.com>
Date:   Tue, 8 Nov 2022 09:30:23 -0800
Message-ID: <CA+HUmGijWOvmt_+UZyc8bArdOYMecDUbwpMiSBBjnfJsLx6ofg@mail.gmail.com>
Subject: Re: nvme: hung task in blk_mq_freeze_queue_wait
To:     Keith Busch <kbusch@kernel.org>
Cc:     linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
        sagi@grimberg.me, hch@lst.de, axboe@fb.com
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Tue, Nov 8, 2022 at 8:51 AM Keith Busch <kbusch@kernel.org> wrote:
> That is an old kernel, I think it'd be worth trying something newer to
> confirm if this observation isn't already fixed. Specifically, it looks
> like you have multiple namespaces timing out IO near simultaneously, and
> causing a mismatched handling. That kind of thing was fixed after the
> follow (plus some prior dependencies):
>
>   commit d6135c3a1ec0cddda7b8b8e1b5b4abeeafd98289
>   Author: Keith Busch <kbusch@kernel.org>
>   Date:   Tue May 14 14:46:09 2019 -0600
>
>       nvme-pci: Sync queues on reset


On Tue, Nov 8, 2022 at 3:29 AM Hillf Danton <hdanton@sina.com> wrote:
>
> Given blk_mq_freeze_queue_wait() in the call trace, another explanation
> may be percpu refcount [1], which can be tested with the diff attached.
>
> Only for thoughts.
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/block/blk-mq.c?h=next-20221104#n228
>
> Hillf

Thanks Keith, Hillf, I will look into that.

Francesco