Received-SPF: pass (google.com: domain of linux-nfs+bounces-3178-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Mon, 6 May 2024 16:06:06 -0400
From: Chuck Lever <chuck.lever@oracle.com>
To: Anna Schumaker <anna@kernel.org>
Cc: linux-nfs@vger.kernel.org, Sagi Grimberg <sagi.grimberg@vastdata.com>,
        Dan Aloni <dan.aloni@vastdata.com>
Subject: Re: [PATCH v3] rpcrdma: fix handling for RDMA_CM_EVENT_DEVICE_REMOVAL
Message-ID: <Zjk4LuOnti/ouaVw@tissot.1015granger.net>
References: <20240506093759.2934591-1-dan.aloni@vastdata.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240506093759.2934591-1-dan.aloni@vastdata.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?dcaEYCc+tv0EFC6mq79FgrP8KSZK5D8moZC9/HQ67uIUqah9G7jv0d9zn8p5?=
 =?us-ascii?Q?yMUxEKDIgB/1nMREHWJx6nqY4ZnfXUoHHCpLEEoa51Z2No5xdpeaNBmxM8B0?=
 =?us-ascii?Q?Qxsa6mkDTKx8IAY7B6tNzrzQv4mXt9D7klhOjmbSpYEPKvk9TTGW7BQnLT89?=
 =?us-ascii?Q?rc35OOmvxjp3nTEu7PwAcN7BM43CUNE+0S39RBME1aga89f6UnJTIutTafdK?=
 =?us-ascii?Q?UQAzUPq3O8ybjWv4IOTLsoMzpKIuPD/5+DrQLiLcYxSE3+c/LJ9yj2mrnrDM?=
 =?us-ascii?Q?l25yiGaBu33mExQc8Z+nQ//SGsQlV0H4w+UOj1h/jc+vgkJ5Qv0srsnS3vb/?=
 =?us-ascii?Q?qMhPWEhYkhP4P143dbbetZdip8exUU3TFq/+Mdx06XJBEaYXP0kG4Is+LEDr?=
 =?us-ascii?Q?RKDdHM6rg13QA6tL8eRXYC8DxOYWG205Jxk40VZrXfXf4IeN7B8TyOvUCMtd?=
 =?us-ascii?Q?rpHAIuU8NoUVQAvcyDpBsL0y+u87CxcP8PNj7wNfe5vnolyV+LLkzHpP4RNk?=
 =?us-ascii?Q?urrGmOP4/pvVLlrFkYp8eWGPVzQKhLPsEjuus5QYtk8YcemPtWA7rJBA8Jfs?=
 =?us-ascii?Q?EfQykW1DbIJifCfmSmd9u1Gz48zSzhXBFV6NMZBmpkytxpM5bRguPZBhZH+2?=
 =?us-ascii?Q?LKcX1Bh9+i3pF9DfOE6BW7XZ4XTu72PZ1tQ9AcvBQgAsQArJT332ogHOCRIC?=
 =?us-ascii?Q?i0IZHwI4uZwfkf1wf/D+K5ks1R1lxWCEVNFFEHYibOp322KzykpItrw5Yo8s?=
 =?us-ascii?Q?wFhITSZRXbQPdgTUmxmDfY4hp++eZkFcxbT5M5rT6iaTpwGjGGRhogpHAtQg?=
 =?us-ascii?Q?cG6Nyrs8CIcjxo1d0BQ0zcnMr/t4DMjRt18RlpcnGYdc+vXmV/hp4UI89zxs?=
 =?us-ascii?Q?WxNITmsJ+lQ/D7JBS3qT/20HtZ8pQ2XuuDZsykc4lqBeOH5Rns/sJimC6DNO?=
 =?us-ascii?Q?um1DMMaOFi9Tq07gyDISqrO1W0sPjPAAKk8XJrOc5TJfepbiCAqYGAlK46vo?=
 =?us-ascii?Q?FOsqoSrnL+0VKGDXO5Rxe2PGQ/IsbYDcmdhqPS5LbX0QErjYJZOOhM/4rIZh?=
 =?us-ascii?Q?4yjyxMKt/SFSnWHMu5o7L/RPKbb2OSMwBK1kBVT8VEKcGSm3C2QpARYCKUw1?=
 =?us-ascii?Q?BTHOEYnSNqOZpfJgF9bOD3Z0ORizEO7i4CIReBcmA3qOpcTfjXmKw14gAMBG?=
 =?us-ascii?Q?IhmuDVoeGzMnHfq3uwUf+U2bQeJeUb+NgQMppilrBqmDSPVZbRKS3mEA+6TP?=
 =?us-ascii?Q?WniKynPcSR/jQKpvZOx5WVsOSVm9CHWmpS1/neCgnzHruY9dC8jq/STJeaOn?=
 =?us-ascii?Q?Wq3uST14mgeSrDkqg0+trdrRZeKnzQOk6ryOlPzJoEMD6O+bLwORQGHrcYg9?=
 =?us-ascii?Q?IY3BxWGp9g36GBn2k1ixdEnPyOSz/CUvE6y1k2nuPlT3xsTZNBDbFy62tvbe?=
 =?us-ascii?Q?ArNfRtFb4LEa2Q0Z44Di2/bxbrRGJTkWiwqvNfJMGzNR7fC2zJFYFBvhziQ6?=
 =?us-ascii?Q?2JJ+csxylrEWNEJfCkA2hY4/pTasDfTCnBl5GHJmVEr57wBuDmMFGmNnTea6?=
 =?us-ascii?Q?oqCaHbNfOfd30ZnCoRRUr3pnFBOvuQnXEif2xEZa0/hNq0/jedbJpVdHTtqq?=
 =?us-ascii?Q?ew=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	p+ymbvJrxakhpRbCGDGlaju4g28uTOewVisiClYeHsEUvjdinlHByy7+ED7GuaaSAo+JJBRtrWH3Bs0/34vMM8znuvLmp9gZsro6uK/K0SfBOWXEXv4ziTMYl+6LW/EKpI7AXzv+fEkjk9fEc4WMW2RciR40q4U5qpPZxvvfnic2bSD9d3/qtiE8FaQone201nkbjqGaj4e9v0GfMd3A4CEhjtNnvcMNIFXmMT4YrwHj0+3YSn+CUMleFjpaEE1r8QvE0MOa2lJjpf3OdRyp0bOJczuqL9Tw3gZj7F1XRk0LiXtCH4+DXOntGLxMvZ/63J8ywQx3+niIfYN75A/umHShoIhRg0Dx7nwm14xwbSJybAQazLabGO4pTIZmSeWGnSYrXHfTfBgVfnx7z6/wljOXKQu2YlVqsvs94NcgoG/xvC4pPssRzOnut42Vg8tWSa2UkOQZrkaYBSTDkvtDG1WfcWbz00GJZjNbKhjajwWUL1OXDjZ0WhLY5HSKLZ+PEwSxlMHtbbE9wW3vxnFluIE7jsy0n51jzI0eBroARJT8JfUuMb72lAA5Y/eh5AjI/RzqOuRITZllh60122ZqKcU6gAXir1D2XUJqftO3M7Y=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c8221326-5638-44e0-a492-08dc6e07f87e
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 May 2024 20:06:09.5488
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: n/Qa7HnN9eoQWHKTcNnHLpNcoKK1sPPSz2lupk1sWYSAbBWB9V0y+gzma/la9y0JpjMaR7dBx64yXham0Fvueg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4145
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26
 definitions=2024-05-06_14,2024-05-06_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 spamscore=0 mlxscore=0
 mlxlogscore=999 suspectscore=0 malwarescore=0 adultscore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2404010000
 definitions=main-2405060145
X-Proofpoint-GUID: T0RzL3vf42RmX7NGL4BWE5iFyNLD9M3o
X-Proofpoint-ORIG-GUID: T0RzL3vf42RmX7NGL4BWE5iFyNLD9M3o

On Mon, May 06, 2024 at 12:37:59PM +0300, Dan Aloni wrote:
> Under the scenario of IB device bonding, when bringing down one of the
> ports, or all ports, we saw xprtrdma entering a non-recoverable state
> where it is not even possible to complete the disconnect and shut it
> down the mount, requiring a reboot. Following debug, we saw that
> transport connect never ended after receiving the
> RDMA_CM_EVENT_DEVICE_REMOVAL callback.
> 
> The DEVICE_REMOVAL callback is irrespective of whether the CM_ID is
> connected, and ESTABLISHED may not have happened. So need to work with
> each of these states accordingly.
> 
> Fixes: 2acc5cae2923 ('xprtrdma: Prevent dereferencing r_xprt->rx_ep after it is freed')
> Cc: Sagi Grimberg <sagi.grimberg@vastdata.com>
> Signed-off-by: Dan Aloni <dan.aloni@vastdata.com>
> ---
>  net/sunrpc/xprtrdma/verbs.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index 4f8d7efa469f..432557a553e7 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -244,7 +244,11 @@ rpcrdma_cm_event_handler(struct rdma_cm_id *id, struct rdma_cm_event *event)
>  	case RDMA_CM_EVENT_DEVICE_REMOVAL:
>  		pr_info("rpcrdma: removing device %s for %pISpc\n",
>  			ep->re_id->device->name, sap);
> -		fallthrough;
> +		switch (xchg(&ep->re_connect_status, -ENODEV)) {
> +		case 0: goto wake_connect_worker;
> +		case 1: goto disconnected;
> +		}
> +		return 0;
>  	case RDMA_CM_EVENT_ADDR_CHANGE:
>  		ep->re_connect_status = -ENODEV;
>  		goto disconnected;
> -- 
> 2.39.3
> 

Hi Anna,

Please apply this patch with:

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Chuck Lever <chuck.lever@oracle.com>


-- 
Chuck Lever