Received: by 2002:ab2:1149:0:b0:1f3:1f8c:d0c6 with SMTP id z9csp2726lqz; Fri, 29 Mar 2024 06:56:47 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUX43ha2byoiPrG0FAgJyPijH3HPrRBNicUJN3sHokqCT1cYdMTSZj/pry9u9mhfxPscarNGsfy5olq/mrpzGvyR5sqYffOk8c6pXrexg== X-Google-Smtp-Source: AGHT+IEV+sKk3UiubFQeFTE+PdyMmy9GZ4sAMMNWHS3JDJATG6lvBZxPVL/hoJnnV4uGgzL6TNPd X-Received: by 2002:a05:6358:5409:b0:183:4d1d:dcaa with SMTP id u9-20020a056358540900b001834d1ddcaamr2487822rwe.3.1711720606816; Fri, 29 Mar 2024 06:56:46 -0700 (PDT) Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id q200-20020a632ad1000000b005e84e725502si3671165pgq.637.2024.03.29.06.56.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Mar 2024 06:56:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-124796-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@kernel.org header.s=k20201202 header.b=FUa92wbB; arc=fail (body hash mismatch); spf=pass (google.com: domain of linux-kernel+bounces-124796-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-124796-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 699AE289F91 for ; Fri, 29 Mar 2024 13:53:57 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 86F9614883F; Fri, 29 Mar 2024 12:42:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="FUa92wbB" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A412E18B5FC; Fri, 29 Mar 2024 12:42:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711716152; cv=none; b=g0TjZg8WB4YveO2r7CoApDhyjEBcKqcEKILyvdTRpfFXUTHcO5of8uHfmdhL5ALi24W9ouL1eQqto310ba/OqIOayUyWERi4yk8Ap5OOXpmGxBvazjBK0nqOZmHl+K5jqtHLtCv2vkdvEJr02M4nlTuMZZsfnltSxBF2w8xUEhg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711716152; c=relaxed/simple; bh=Xc3uF32qcm2T61fHfLRI1sXsN9HQIKCA58nheKM7b4Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MQZ1siQ9qQPUohYALUmi++rS/zbJMbpBFaEOsFt4uvtp6LHO6MCBcnz+fvwMgqDiGa7b7HDuPQakSTp8QWlBLA5Ikr/U1iY7Axsku1mv7jOkT2+7AVePCKAuiiB9+TV1mmtXPc5gdkpwwGcMVjqqQ5vBGRXRHC6J6L7WYCZxq0E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=FUa92wbB; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 704FCC433C7; Fri, 29 Mar 2024 12:42:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711716152; bh=Xc3uF32qcm2T61fHfLRI1sXsN9HQIKCA58nheKM7b4Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FUa92wbBuRind570l6CgAZMpBI7B6DYO75iyDDdBAxcoQvuBUP8n1Px/J3V+qa4Pp 4qWj/5w3NIxUI5VSZfOvwOht7Ky2E7f+0kS1MgOb+ayXKldC7e8BDX99DhLOcyW/+B Zv5GNI6D6MawMCOjynD/IOqkgMZ7T+urTRKGjkseY3m4UUt3ftHX1X3jFIEYsdbh+U n8g29zNTpTR5gSgxUT/0sLt74hrvoRu6de224ZcG2tO0lWDPUkwRKhuz9ieuxnfNPa 024HRCafd2Xt3Ef5x14Hcm0Bjcx3TGqus2n4mcbkubk5cWPUnvdUHirW3a85fjW4yk 8nejxQ3iQAKuw== From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Manjunath Patil , Leon Romanovsky , Sasha Levin , markzhang@nvidia.com, linux-rdma@vger.kernel.org Subject: [PATCH AUTOSEL 6.8 85/98] RDMA/cm: add timeout to cm_destroy_id wait Date: Fri, 29 Mar 2024 08:37:56 -0400 Message-ID: <20240329123919.3087149-85-sashal@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240329123919.3087149-1-sashal@kernel.org> References: <20240329123919.3087149-1-sashal@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-stable: review X-Patchwork-Hint: Ignore X-stable-base: Linux 6.8.2 Content-Transfer-Encoding: 8bit From: Manjunath Patil [ Upstream commit 96d9cbe2f2ff7abde021bac75eafaceabe9a51fa ] Add timeout to cm_destroy_id, so that userspace can trigger any data collection that would help in analyzing the cause of delay in destroying the cm_id. New noinline function helps dtrace/ebpf programs to hook on to it. Existing functionality isn't changed except triggering a probe-able new function at every timeout interval. We have seen cases where CM messages stuck with MAD layer (either due to software bug or faulty HCA), leading to cm_id getting stuck in the following call stack. This patch helps in resolving such issues faster. kernel: ... INFO: task XXXX:56778 blocked for more than 120 seconds. .. Call Trace: __schedule+0x2bc/0x895 schedule+0x36/0x7c schedule_timeout+0x1f6/0x31f ? __slab_free+0x19c/0x2ba wait_for_completion+0x12b/0x18a ? wake_up_q+0x80/0x73 cm_destroy_id+0x345/0x610 [ib_cm] ib_destroy_cm_id+0x10/0x20 [ib_cm] rdma_destroy_id+0xa8/0x300 [rdma_cm] ucma_destroy_id+0x13e/0x190 [rdma_ucm] ucma_write+0xe0/0x160 [rdma_ucm] __vfs_write+0x3a/0x16d vfs_write+0xb2/0x1a1 ? syscall_trace_enter+0x1ce/0x2b8 SyS_write+0x5c/0xd3 do_syscall_64+0x79/0x1b9 entry_SYSCALL_64_after_hwframe+0x16d/0x0 Signed-off-by: Manjunath Patil Link: https://lore.kernel.org/r/20240309063323.458102-1-manjunath.b.patil@oracle.com Signed-off-by: Leon Romanovsky Signed-off-by: Sasha Levin --- drivers/infiniband/core/cm.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c index ff58058aeadca..bf0df6ee4f785 100644 --- a/drivers/infiniband/core/cm.c +++ b/drivers/infiniband/core/cm.c @@ -34,6 +34,7 @@ MODULE_AUTHOR("Sean Hefty"); MODULE_DESCRIPTION("InfiniBand CM"); MODULE_LICENSE("Dual BSD/GPL"); +#define CM_DESTROY_ID_WAIT_TIMEOUT 10000 /* msecs */ static const char * const ibcm_rej_reason_strs[] = { [IB_CM_REJ_NO_QP] = "no QP", [IB_CM_REJ_NO_EEC] = "no EEC", @@ -1025,10 +1026,20 @@ static void cm_reset_to_idle(struct cm_id_private *cm_id_priv) } } +static noinline void cm_destroy_id_wait_timeout(struct ib_cm_id *cm_id) +{ + struct cm_id_private *cm_id_priv; + + cm_id_priv = container_of(cm_id, struct cm_id_private, id); + pr_err("%s: cm_id=%p timed out. state=%d refcnt=%d\n", __func__, + cm_id, cm_id->state, refcount_read(&cm_id_priv->refcount)); +} + static void cm_destroy_id(struct ib_cm_id *cm_id, int err) { struct cm_id_private *cm_id_priv; struct cm_work *work; + int ret; cm_id_priv = container_of(cm_id, struct cm_id_private, id); spin_lock_irq(&cm_id_priv->lock); @@ -1135,7 +1146,14 @@ static void cm_destroy_id(struct ib_cm_id *cm_id, int err) xa_erase(&cm.local_id_table, cm_local_id(cm_id->local_id)); cm_deref_id(cm_id_priv); - wait_for_completion(&cm_id_priv->comp); + do { + ret = wait_for_completion_timeout(&cm_id_priv->comp, + msecs_to_jiffies( + CM_DESTROY_ID_WAIT_TIMEOUT)); + if (!ret) /* timeout happened */ + cm_destroy_id_wait_timeout(cm_id); + } while (!ret); + while ((work = cm_dequeue_work(cm_id_priv)) != NULL) cm_free_work(work); -- 2.43.0