Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Lei Lei2 Yin <yinlei2@lenovo.com>
To:     "kbusch@kernel.org" <kbusch@kernel.org>,
        "axboe@fb.com" <axboe@fb.com>, "hch@lst.de" <hch@lst.de>,
        Sagi Grimberg <sagi@grimberg.me>
CC:     "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "cybeyond@foxmail.com" <cybeyond@foxmail.com>
Subject: [PATCH v2] nvme: fix double blk_mq_complete_request for timeout
 request with low probability
Thread-Topic: [PATCH v2] nvme: fix double blk_mq_complete_request for timeout
 request with low probability
Thread-Index: Adlp7D0a6eHDJ2XCQ12f5YXOFDMnBA==
Date:   Sat, 8 Apr 2023 07:44:41 +0000
Message-ID: <PS1PR03MB49397BF066FA9BC5BC5EE72488979@PS1PR03MB4939.apcprd03.prod.outlook.com>
Accept-Language: zh-CN, en-US
Content-Language: zh-CN
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?6D9a/o8v2jaFyYMGBcG0wQbi43aX1cMCMTFkhxV2ogUoIMdh9Qwyg8szAey4?=
 =?us-ascii?Q?ZzXYIHCFXp9XjWLTCYafEUkC4byGBwjiA0/+Y+wJQZhMLJBxCDC1PT4zmWnP?=
 =?us-ascii?Q?dHWzvH3yiYMU4trgCZESsBeKqMbODibwD1kqpo/tCPu6kiEd8YFOj+idX+lf?=
 =?us-ascii?Q?dEhbWf+8lk9tlWDTudmSzN7A9fQUKCJE/G6KAgxanXkrpzEPFb1KaU9cc3Dg?=
 =?us-ascii?Q?Tg9rT4w1QZ4R5/eIhRGZ1V+RkNvqcetRwoeAMwtyyPxN9O0AhSYOrRd6al9r?=
 =?us-ascii?Q?563++dpjaXW1DyDD0PIuIdR4K5915xMrmbsdi6xhPzmsgP31mvOSQ5Fyz4Ps?=
 =?us-ascii?Q?hXtdgFZNPVmA1iIaEO8SpfvOSZ48XPNCZjOGSsf6G6LHZvz08Apw3kDxmB2y?=
 =?us-ascii?Q?j/h+foitG4m/uP1xvBT+smQ3X/d+A/D7rTBru+8RfRQKplHuo10OkWKoYxob?=
 =?us-ascii?Q?oGTPGGgnkztkI9bRh1U6OprFnCBg06i+BY7tbbpNdT4cl0h7r1OTZtCGokNZ?=
 =?us-ascii?Q?48WRuXJTSd9+wbLfVY8Aww/bAl27eI6OtdF8HPyy53RELgJH1ZfS9NgGIDIj?=
 =?us-ascii?Q?/E+gISpalgkz2V28mtLhvJO/Y35TcXHfgO/MjT8SiQmutAQf+yJtndYZX8mY?=
 =?us-ascii?Q?PJwIkTtrgC8NcXqUdQigAdNCWzFGp4teQljYOlYBwv7FWL58ngqBOm/SNtgO?=
 =?us-ascii?Q?qZE5bCtWpIMsgx4s79yqYzbvtoyIRpClvEBxW8PuuFmzLfioXf8DwsXMMzHS?=
 =?us-ascii?Q?Uboy2m6O3VYDkjJ2rGwIuH9cQ5GGnPnpDpoA22GEX2fz6T9QY3QNHcI+60oJ?=
 =?us-ascii?Q?5mqpvgrYRh4+kh7WkO+/m9p8b0R1wWVUa7WI1yF17Pp7e4uC+G13WFyzO1Uz?=
 =?us-ascii?Q?4RxtXZ+qVu42m7jLC6CBhrmdfBOl09Co/SSmOSKt43JVjq1QLm94+3ujTH3t?=
 =?us-ascii?Q?Ocdrw8Vkrs9g479OofCmqUTkuHyC/7OLjxXOfF0KeI/1WMVNtxz742yv4O5+?=
 =?us-ascii?Q?zG200gSNJa/xKH6CwZc/lzJLEJXCsYozMtVIqwghnpbiSfYSIWdXYlQUPYvV?=
 =?us-ascii?Q?SwYfArZQ3HmgJRHyDyBwA0BYd3VZd9AZUAbac5aWCrJE9FtP8BUVWOhnkuVN?=
 =?us-ascii?Q?ifLFnzyAxBLtGy5Ilhdbf8hEhKFeAaJ0t/I99OCbKAncXoVH3zVw5ARb6M/D?=
 =?us-ascii?Q?R9PsGTm0VL4cdfTYXGszO2f/4qY3/AE8VzEHoxsvdvR3MKjiAaT0Nduccv9M?=
 =?us-ascii?Q?2J107nFXFkUrEd3DcsC757UoOzvKsqUw7ACn3LVf59G7TLBNStRvNG3pcOQA?=
 =?us-ascii?Q?t8ie1PEfpfBd7TxQXX8JewrqzkQ9ZYL6Mt0brXFnfvJ5g8z715TVnFWKv84A?=
 =?us-ascii?Q?ejS1+BI18t6tqWYu7NkuM1BlL+o1MSLLBDhhl3XaBas0ry1QBQNZIkoF6O0b?=
 =?us-ascii?Q?5Z2YnyRhJFZ7WmrBPDcYAeEFO6TNZdFCPnlfg7CfORKjGZanLXsjF1mpQVbt?=
 =?us-ascii?Q?AeLw0yHkhiA2bX02f2maD+xilQugc19+DKWeUYTMaPksn1UcKzwMHX/4KIz8?=
 =?us-ascii?Q?JD+O1CNRVsB/XzEBsrg=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: lenovo.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PS1PR03MB4939.apcprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8e1cd836-8595-43bd-6887-08db38051d35
X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Apr 2023 07:44:41.8535
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 5c7d0b28-bdf8-410c-aa93-4df372b16203
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 3ba1tb6XLRVBuRmkBxd0IBkwTqt743LO0OKgLvCzQtGKKTVyaNUjgc6ZHO39T3Hzb/61sK/5UsYWynF6RWVtJQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEZPR03MB6644
X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

From 5148d52554d6bcf6134786d40f1c6f9f22e18978 Mon Sep 17 00:00:00 2001
From: Lei Yin <yinlei2@lenovo.com>
Date: Thu, 6 Apr 2023 23:39:11 +0800
Subject: [PATCH v2] nvme: fix double blk_mq_complete_request for timeout
 request with low probability

When nvme_cancel_tagset traverses all tagsets and executes
nvme_cancel_request, this request may be executing blk_mq_free_request
that is called by nvme_rdma_complete_timed_out/nvme_tcp_complete_timed_out.
When blk_mq_free_request executes to WRITE_ONCE(rq->state, MQ_RQ_IDLE) and
__blk_mq_free_request(rq), it will cause double blk_mq_complete_request for
this request, and it will cause a null pointer error in the second
execution of this function because rq->mq_hctx has set to NULL in first
execution.

With multipath, by injecting a large number of requests timed out, I have
reproduced the issue that caused kernel crashes in three versions of the
kernel(include 5.10.167, 6.2.10 and upstream version which compiled by
myself). The error stack is as follows:

[ 2777.253091]  <TASK>
[ 2777.253102]  nvme_failover_req+0x10a/0x120 [nvme_core]
[ 2777.255302]  blk_complete_reqs+0x3e/0x60
[ 2777.255726]  __do_softirq+0xb6/0x2ad
[ 2777.256139]  ? __pfx_smpboot_thread_fn+0x10/0x10
[ 2777.256556]  run_ksoftirqd+0x28/0x40
[ 2777.256978]  smpboot_thread_fn+0xdb/0x1d0
[ 2777.257399]  kthread+0xd7/0x100
[ 2777.257827]  ? __pfx_kthread+0x10/0x10
[ 2777.258253]  ret_from_fork+0x29/0x50
[ 2777.258695]  </TASK>

Signed-off-by: Lei Yin <yinlei2@lenovo.com>
---
 drivers/nvme/host/core.c    | 4 ++--
 drivers/nvme/host/fabrics.h | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 53ef028596c6..c1cc384f4f3e 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -450,8 +450,8 @@ bool nvme_cancel_request(struct request *req, void *dat=
a)
 	dev_dbg_ratelimited(((struct nvme_ctrl *) data)->device,
 				"Cancelling I/O %d", req->tag);
=20
-	/* don't abort one completed request */
-	if (blk_mq_request_completed(req))
+	/* don't abort one completed or idle request */
+	if (blk_mq_rq_state(req) !=3D MQ_RQ_IN_FLIGHT)
 		return true;
=20
 	nvme_req(req)->status =3D NVME_SC_HOST_ABORTED_CMD;
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index dcac3df8a5f7..a25a0118b722 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -197,7 +197,7 @@ static inline char *nvmf_ctrl_subsysnqn(struct nvme_ctr=
l *ctrl)
=20
 static inline void nvmf_complete_timed_out_request(struct request *rq)
 {
-	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq)) {
+	if (blk_mq_rq_state(rq) =3D=3D MQ_RQ_IN_FLIGHT) {
 		nvme_req(rq)->status =3D NVME_SC_HOST_ABORTED_CMD;
 		blk_mq_complete_request(rq);
 	}
--=20
2.39.1