Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     Leon Romanovsky <leon@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH v1] RDMA/core: Fix check_flush_dependency splat on addr_wq
Thread-Topic: [PATCH v1] RDMA/core: Fix check_flush_dependency splat on
 addr_wq
Thread-Index: AQHYtjxDHzOcFbDad0CgCtqOY9xpUa28It+AgABhkYCAAUS4AIAAULmAgAMZTACAAAltAIAAAa6AgABhRICABIF0gIAACD6A
Date:   Mon, 29 Aug 2022 17:14:56 +0000
Message-ID: <90CD6895-348C-4B75-BAC5-2F7A0414494B@oracle.com>
References: <166118222093.3250511.11454048195824271658.stgit@morisot.1015granger.net>
 <YwSLOxyEtV4l2Frc@unreal> <584E7212-BC09-48E1-A27E-725E54FA075E@oracle.com>
 <YwXtePKW+sn/89M6@unreal> <591D1B3D-B04A-4625-8DD0-CA0C2E986345@oracle.com>
 <YwjKpoVbd1WygWwF@nvidia.com>
 <08F23441-1532-4F40-9C2A-5DBD61B11483@oracle.com>
 <YwjT9yz8reC1HDR/@nvidia.com>
 <FF62F78D-95EE-4BA1-9FC6-4C6B1F355520@oracle.com>
 <YwztJVdYq6f5M9yZ@nvidia.com>
In-Reply-To: <YwztJVdYq6f5M9yZ@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?UxNpnlbfzfGIAic4i6D4rflFEPJEhTQaJ0pfNw66zQrNTJSAIEjrNmkOqfL3?=
 =?us-ascii?Q?v5i7wzAKnwnjs3YJZ6K603FMqzjaRcv4SoqFhaeWtwVdCOkZ3y9hdOd6OLm7?=
 =?us-ascii?Q?N0kihGzNKE1ErGsskf0e5BtY+YvGXk7NTBjPVZWjvib8Fl2SMPtCVcaIdEWS?=
 =?us-ascii?Q?hYt2zwvOUb07hrvofkHwPAB1LQ3seFeig75ZfIS9H91PmKFt4d3EmRcqjPOb?=
 =?us-ascii?Q?Se0D/8qvsnOBppN9FtJyYp+b15AHGKN9UNHTz/egSLnXk7rv85fsE+TLM9xP?=
 =?us-ascii?Q?3CEkfM2W8uq++412OTcOqv9a51EZzckU0+NWnxQGM2QJSZUm2NyAbhPNG6hy?=
 =?us-ascii?Q?oRm6Py5WGMFcSSaXrM44NAdId92dq6X+JHFErt9or63YV0uSKdnFhm7PdB8Q?=
 =?us-ascii?Q?0/YCBtC+3l4HVfWCHMbDDDkwsttwFpAmYYVy4j6J8gewDp+kYd3XUji3vOgn?=
 =?us-ascii?Q?I1zdQEa9dXGFkcwWUSN6oOJBFJ80mlupt4OVFllV/St82usgKUXJC+hpcv1z?=
 =?us-ascii?Q?oor8hnKPbkSfZV/YKPbFeesALjkjmMxBqkv5GEIs9aTT655IUS8RL5Ll/xYi?=
 =?us-ascii?Q?WXdtY8Q/c7OabPSIWzP1wKkCAns2ijzEx7ro6CGs2RJm9K4NAN3bt/Dk5/eK?=
 =?us-ascii?Q?7fk53GzkNc8CsqTPlZn/G6Ps+D+mmTUmXayQr0Tz3hTWgZhY9Q3l87UmXoy6?=
 =?us-ascii?Q?++CACsVunPEJo9uE4JC14Jg/WmcY1YJQ645jHwcsNa/wie4lgTlJRnxQCiDH?=
 =?us-ascii?Q?0N/vtCYlp4VXvXKql8YYdlEfMSlDYS8WUMDPJVrR6DnA9b+0XRDalKAlBVoF?=
 =?us-ascii?Q?wAlwGACsSaBUH4uT96sfhgisxy4LVy3zcWNQGsHTo078ctw4zY1AyqEt0p3z?=
 =?us-ascii?Q?xix6Um0PiCDywCmA4xJpuQzB6XuCx9HqN4rpHn77Q6ql+CyychNHJkJiN+CY?=
 =?us-ascii?Q?k761Qu9SinXpcPhIcP+qCGzMgiT/isHtsFU+mh/92k4LzPnYB5h1dSUOlI3t?=
 =?us-ascii?Q?SppjH8oThAgbqhgfV963ArCziayb1a3L+qGQdpBeMH6PExiGonCg5TgDuxaj?=
 =?us-ascii?Q?7ZGQl4st2rszP44H4/AXxeaUzcLMiXAlIE+ECcejGFkxF7T3EHldYWvmXBW1?=
 =?us-ascii?Q?w3X4k5gNqBon4YjF77NxOD24anoqdgsZvOav6C+l5FZ4sQ2+0QAfvm4MVHsY?=
 =?us-ascii?Q?7VnJPrQx/XnLPYVSk9MJlKpPPymmzsdlS44DwFdQykFwAmxjSV+8AiIf/iKC?=
 =?us-ascii?Q?Z6/PhzS841nSyzQofF67toOktvquahgxxDt2SWZUSQqPjppLedzr+lJP63dZ?=
 =?us-ascii?Q?aWo6ht3YByfOS8exTBcxZ6VZC+6ZuodrkYIVus0NoVWUSLK/2dAPhGSOVykR?=
 =?us-ascii?Q?5ge210IyyQwcR7NBlSz5kEWw3bvMpcWC8ziduqMfygWj0HsEAUHyVgcNDXj7?=
 =?us-ascii?Q?MeSWEiBnJBpBx9UPyoox6cg5p88p2ARwMdxgm0FLjO/VKfr4DCATy1LJ/hQL?=
 =?us-ascii?Q?7Oy0JHkmioFYORw4K7Eb4z260jmQ2cOJh632TpQMUuD6rV2/JZOHT4SUF7aY?=
 =?us-ascii?Q?PCDCC4QQ/QNCMk3fJ7KTDbJr+HswjYIlQoEjMuYxgTfvOAe5xjfwkh07K/tn?=
 =?us-ascii?Q?Kw=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <8100F58ED790D747AB50CDF1139A7E19@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f899207e-1d10-41ff-9577-08da89e1fee5
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Aug 2022 17:14:56.3139
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: yvtVUYUMhJM+3CISqcyZjcLwqGjW3jLI+BPxcgQojLkTdHmC/1tfc0/pEJVl3DsXYgo188I/RM8WLyk1N1F++w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2787
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-08-29_09,2022-08-25_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 mlxscore=0
 spamscore=0 mlxlogscore=999 phishscore=0 suspectscore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2207270000
 definitions=main-2208290080
X-Proofpoint-ORIG-GUID: Ep7FKVT3MOUUhuVSaVm-UtvbPHIC8AKQ
X-Proofpoint-GUID: Ep7FKVT3MOUUhuVSaVm-UtvbPHIC8AKQ
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Aug 29, 2022, at 12:45 PM, Jason Gunthorpe <jgg@nvidia.com> wrote:
>=20
> On Fri, Aug 26, 2022 at 07:57:04PM +0000, Chuck Lever III wrote:
>> The connect APIs would be a place to start. In the meantime, though...
>>=20
>> Two or three years ago I spent some effort to ensure that closing
>> an RDMA connection leaves a client-side RPC/RDMA transport with no
>> RDMA resources associated with it. It releases the CQs, QP, and all
>> the MRs. That makes initial connect and reconnect both behave exactly
>> the same, and guarantees that a reconnect does not get stuck with
>> an old CQ that is no longer working or a QP that is in TIMEWAIT.
>>=20
>> However that does mean that substantial resource allocation is
>> done on every reconnect.
>=20
> And if the resource allocations fail then what happens? The storage
> ULP retries forever and is effectively deadlocked?

The reconnection attempt fails, and any resources allocated during
that attempt are released. The ULP waits a bit then tries again
until it works or is interrupted.

A deadlock might occur if one of those allocations triggers
additional reclaim activity.


> How much allocation can you safely do under GFP_NOIO?

My naive take is that doing those allocations under NOIO would
help avoid recursion during memory exhaustion.


--
Chuck Lever