Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever <chuck.lever@oracle.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH v1] svcrdma: Hold private mutex while invoking
 rdma_accept()
Thread-Topic: [PATCH v1] svcrdma: Hold private mutex while invoking
 rdma_accept()
Thread-Index: AQHXAMPXmeY62ULkGUCUAaBhXd4JEapUmXKAgAAB5IA=
Date:   Fri, 12 Feb 2021 14:50:42 +0000
Message-ID: <1103A656-BAB3-40C4-A935-3D432073AD83@oracle.com>
References: <161308170145.1097.4777972218876421176.stgit@klimt.1015granger.net>
 <20210212144355.GA1696322@nvidia.com>
In-Reply-To: <20210212144355.GA1696322@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <8C526F7BCE329E4FB6A7732F0A696CD5@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 95afaed9-9267-451d-d77f-08d8cf659280
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Feb 2021 14:50:42.8075
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: viglYLlJFI6FF4qRXT4luA0sBCrdGS7q/7TuMmeSCw97RsGMKaZPnXC4Ij6wtxkOKYwpWPYOXlUDisQMYOtAfw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3047
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9892 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0
 mlxlogscore=837 malwarescore=0 bulkscore=0 phishscore=0 spamscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2102120116
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9892 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0
 spamscore=0 lowpriorityscore=0 phishscore=0 adultscore=0 impostorscore=0
 suspectscore=0 mlxscore=0 clxscore=1015 mlxlogscore=945 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000
 definitions=main-2102120116
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Hi Jason-

Thanks for your review.


> On Feb 12, 2021, at 9:43 AM, Jason Gunthorpe <jgg@nvidia.com> wrote:
>=20
> On Thu, Feb 11, 2021 at 05:15:30PM -0500, Chuck Lever wrote:
>> RDMA core mutex locking was restructured by d114c6feedfe ("RDMA/cma:
>> Add missing locking to rdma_accept()") [Aug 2020]. When lock
>> debugging is enabled, the RPC/RDMA server trips over the new lockdep
>> assertion in rdma_accept() because it doesn't call rdma_accept()
>> from its CM event handler.
>>=20
>> As a temporary fix, have svc_rdma_accept() take the mutex
>> explicitly. In the meantime, let's consider how to restructure the
>> RPC/RDMA transport to invoke rdma_accept() from the proper context.
>>=20
>> Calls to svc_rdma_accept() are serialized with calls to
>> svc_rdma_free() by the generic RPC server layer.
>>=20
>> Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>=20
> Fixes line

Wasn't clear to me which commit should be listed. d114c6feedfe ?


>> ---
>> net/sunrpc/xprtrdma/svc_rdma_transport.c |    2 ++
>> 1 file changed, 2 insertions(+)
>=20
> It could even be right like this..
>=20
> This should be inside the lock though:
>=20
>        newxprt->sc_cm_id->event_handler =3D svc_rdma_cma_handler;

OK.


> But this really funny looking, before it gets to accept the handler is
> still the listen handler so any incoming events will just be
> discarded.

Yeah, not clear to me why two CM event handlers are necessary.
If they are truly needed, a comment would be helpful.


> However the rdma_accept() should fail if the state machine has been
> moved from the accepting state, and I think the only meaningful event
> that can be delivered here is disconnect. So the rdma_accept() failure
> does trigger destroy_id, which is the right thing on disconnect anyhow.

The mutex needs to be released before the ID is destroyed, right?


--
Chuck Lever