Received-SPF: pass (google.com: domain of linux-nfs+bounces-3179-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Mon, 6 May 2024 16:08:57 -0400
From: Chuck Lever <chuck.lever@oracle.com>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: Dan Aloni <dan.aloni@vastdata.com>, linux-nfs@vger.kernel.org,
        Sagi Grimberg <sagi.grimberg@vastdata.com>
Subject: Re: [PATCH v3] rpcrdma: fix handling for RDMA_CM_EVENT_DEVICE_REMOVAL
Message-ID: <Zjk42a54FfH6jWMw@tissot.1015granger.net>
References: <20240506093759.2934591-1-dan.aloni@vastdata.com>
 <be4563e5-caa6-4085-98a9-a86e24c99186@grimberg.me>
 <Zjj9iOOJ0px+Lvin@tissot.1015granger.net>
 <bb1f1604-8b3f-4fc4-ac31-1f365c1fe257@grimberg.me>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <bb1f1604-8b3f-4fc4-ac31-1f365c1fe257@grimberg.me>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?BslpJpyEyyF3r1oJALyFRl4vNxbvH2UFjJXdJZ/Ov6PQKVnSls2OrO5pUywT?=
 =?us-ascii?Q?FxyFy3Lum6r1hIVtG1jrLst39LvCY29qAyi2VR74KRORIOho7TDDkLcStliN?=
 =?us-ascii?Q?0n4uZfYKLV5+AP6q1nOLIdTRd6FroC7dXwO+xC6O6e0+hUL9RpMxQA1ievjd?=
 =?us-ascii?Q?umdLidh2DI9R1T3osiL2DD+EljmlMUfFEjH8s+taj1oWJV+r5fZ7ZseyMA9j?=
 =?us-ascii?Q?RBZW4kajuIsxbMGpEgnxD+sjNSIeafx+LNlItvbL4HK6mdZSAnlg/xBggJXB?=
 =?us-ascii?Q?y4z5X0tA5w3hkUZtvHTGXDRN7xZEDY9DM9kjtx/JbXjRv+8xSNbzvC72hL1B?=
 =?us-ascii?Q?H7D2GazFVNgpbe6YTBM9hRjUXorsVW1SgbiF08btdQubIVQJh/kJIpwVOnyF?=
 =?us-ascii?Q?ow2G+0F7PYAvCgETeSF7BRUi6Xg/4y1vgN9rLuGb1t3FEAkrvM5tpVvF0cu/?=
 =?us-ascii?Q?NsMhaQlH0TI/vbIu0eQUsahnsLBITuipeUCqqIlRWX5TXIHRJlh4DQix97UK?=
 =?us-ascii?Q?M7WUPTEMHDRNvXFwdJ6ODlRNELTtOUmlbaUO8H1n0OOPfu6v2fePlYXrTTl1?=
 =?us-ascii?Q?itS/ImKmVFSqoKe9Jkn+9VrTp+Cn4jCyo05LzeYAHUVEFofq7F9pFXDaJv8S?=
 =?us-ascii?Q?lXPudLfKtK/8+M2iQy59gxyOd2aYMn1kw7WkozpfFqyheXrRkGjm14HhTGn5?=
 =?us-ascii?Q?C4O6qJJfmW2zjfqGZ9pGgadPakvlL6NRQXhHC2rktl50uWRfmPYNdSLnsO6z?=
 =?us-ascii?Q?NyOHcKib3Le24TgKm9vHG5cknrAbHlrjh0AZLOpG8iiibwAKJNtbvA+zIxL6?=
 =?us-ascii?Q?ob2IwPJu4SouHLOsOvlvOhyQN2yIjsS3uwuyayuCQEto4Ma7Nft/llr0ZwS4?=
 =?us-ascii?Q?eTqi0Y5ez12NY4C2CO2cx38z5dR0OD5N2Ryk+RYj5C+HNap/YKBwOENJaJHx?=
 =?us-ascii?Q?VPkLLX8vvWUjsurYV+LyFJfSYGidkIOLecLgwvkiG4WOADYvhVvnDeWz06nT?=
 =?us-ascii?Q?+2JSrFCzRMZqQE544Ua7TrFHdsfypKHYak8+RWQZHv7v0cYag8rhRkS7iGgn?=
 =?us-ascii?Q?pjr4whKTqKkgh5c8dDPyTr6/zDcZvkqmkD/QOwAebgszQSU7vPtQub8T4Bww?=
 =?us-ascii?Q?VIAMT5jHVmaHh/uHGQBVK7O2vp27zxi/goVpGwTEtyRhuxxzI88YNnsLsJ0i?=
 =?us-ascii?Q?7SFQzGUOfohctEoeXR9lzgIer/54ixr0vUc44gnLmIK4b6hnkkKueU6tp8QY?=
 =?us-ascii?Q?Gsmf4eZavo9MLaLn2GPN0owt4cyQzsp0q0SXTY521ggDRZR2DsSrNW7x0b3c?=
 =?us-ascii?Q?TmYu7FxKyihbvCI5/93JtH/PnTvWJfcyRTG5dmWQyYgvSO2PTIitqci64qpi?=
 =?us-ascii?Q?QYnLir1f/LGE7tQIAzc6os6fC9WOJfjQcao/ahLwEW1j6J4ndvowXbMdcQXw?=
 =?us-ascii?Q?XHC0tcdcUqMFbq8YpDDFP6DdPRKXCJ+lCfuDuVjhi7sYUtT0emBi++4ZvzQT?=
 =?us-ascii?Q?rDk8MRSugqKNsDfMLcO0ATRwcw6ovdPkoLDACNvNU5ZdHWTxQQrXj3tJQAvD?=
 =?us-ascii?Q?e/cUZnojfShGfw8U/bpKpow4BgCUFYwOctu3vkOU+kLbbnmIqRfpBtYdgGDw?=
 =?us-ascii?Q?+g=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	DZAoL/g7OQug5q2mCDzb417GuIUApvR6UsFHtbZTFJP256uMoIwNEXAR8iZGLQUhL4kTX9EKCUcNavxY9mfS4bSzoMuXxyK0DlH03JPKpVjhpeO+5brNbxUeS/qZ/os96c97schjncWO7kXNq8Kgn17NRobfSjYcUuIyIJzkJUUCcvcnTEr0XCQmFaRNGIlT0I8HJb23wIDwnAKieHn3gttFZyMVAMFFosJ5VvBwqYsJDX/17dAT/cAyAMaXhV3yy3UHWLdP0eiZiMFUnsuQzFGX9ftS8pH9NvfSTGIcvUXNkGsatEfZ0UGOXWrIcVeMWNqwwWLU1e/7JW6FtNnaKl5rkq7j+g4+cYq2KErgEM25NdFnydlvAA1/miWZvceAgMLDKA7uqVwAOz9eGtAeSsS9eoKn83o9L3mbAvRCLRHQKIOD1t5ozjuKpHKwvMFQx5QPlUrTrLlT/OvhMyOdNPdsFVvoF/UR9R1yHIrKsaqHEuOE9vzOrf8BkwxDNS3NBgfBzxb1qm2XIUXndxRwQf5O2vXdmkridc3K0sNTDR9oowq6uLeJL8s0CRk4eMVO0Xir74pfpvKt9AmuhTkyAMjE0gmge8wNe6/UhYlFjsA=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a517b9f1-44f3-4d91-f4ba-08dc6e085e18
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 May 2024 20:08:59.8112
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Q80PNVuRbY7GeCVtxNeibSxMSCsMIJgcQVQbShoy0dyoVzTGA4NVuA8ypOh+q90iezWF+Dcs1RjzIH/HMC2QGA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4578
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26
 definitions=2024-05-06_14,2024-05-06_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999
 bulkscore=0 suspectscore=0 spamscore=0 malwarescore=0 mlxscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2404010000 definitions=main-2405060146
X-Proofpoint-GUID: IbTpns43mjuE6ljRGWLL7UroTCNzdTP_
X-Proofpoint-ORIG-GUID: IbTpns43mjuE6ljRGWLL7UroTCNzdTP_

On Mon, May 06, 2024 at 10:10:12PM +0300, Sagi Grimberg wrote:
> 
> On 06/05/2024 18:55, Chuck Lever wrote:
> > On Mon, May 06, 2024 at 06:09:51PM +0300, Sagi Grimberg wrote:
> > > Question though, in DEVICE_REMOVAL the device is going away as soon as the
> > > cm handler callback returns. Shouldn't nfs release all the device resources
> > > (related to this
> > > cm_id)? afaict it was changed in:
> > > e28ce90083f0 ("xprtrdma: kmalloc rpcrdma_ep separate from rpcrdma_xprt")
> > In the case where a DEVICE_REMOVAL event fires and a connection
> > hasn't yet been established, my guess is the ep reference count will
> > go to zero when rpcrdma_ep_put() is called.
> 
> Yes, I was actually referring to the case where the connection was
> established.
> It looks like rpcrdma_force_disconnect -> xprt_force_disconnect schedules
> async
> work to tear things down no?

Yes because we can't rip out the hardware resources while the
RPC client is still using the transport. Converting to use an
ib_client might help by first triggering a disconnect, done
with proper serialization.


> > > FWIW in nvme we avoided the problem altogether by registering an ib_client
> > > that is
> > > called on .remove() and its a separate context that doesn't have all the
> > > intricacies with
> > > rdma_cm...
> > I looked at ib_client, years ago, and thought it would be a lot of
> > added complexity. With a code sample (NVMe host) maybe I can put
> > something together.
> 
> The plus is that there is no need to handle the DEVICE_REMOVAL cm event,
> which is always nice...

I'll have a look, thanks for the suggestion.


-- 
Chuck Lever