Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 8 Nov 2022 09:51:31 -0700
From:   Keith Busch <kbusch@kernel.org>
To:     Francesco Ruggeri <fruggeri@arista.com>
Cc:     linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
        sagi@grimberg.me, hch@lst.de, axboe@fb.com
Subject: Re: nvme: hung task in blk_mq_freeze_queue_wait
Message-ID: <Y2qJE2Bpbvnf5Ejb@kbusch-mbp>
References: <20221108025602.E48005EC04F9@us226.sjc.aristanetworks.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20221108025602.E48005EC04F9@us226.sjc.aristanetworks.com>
Precedence: bulk

On Mon, Nov 07, 2022 at 06:56:02PM -0800, Francesco Ruggeri wrote:
> [ 4390.119745] INFO: task kworker/u80:2:8279 blocked for more than 300 seconds.
> [ 4390.471456]       Tainted: P           O      4.19.142.Ar-29007847.buytenhb7335070 #1

That is an old kernel, I think it'd be worth trying something newer to
confirm if this observation isn't already fixed. Specifically, it looks
like you have multiple namespaces timing out IO near simultaneously, and
causing a mismatched handling. That kind of thing was fixed after the
follow (plus some prior dependencies): 

  commit d6135c3a1ec0cddda7b8b8e1b5b4abeeafd98289
  Author: Keith Busch <kbusch@kernel.org>
  Date:   Tue May 14 14:46:09 2019 -0600
  
      nvme-pci: Sync queues on reset