Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 26 Aug 2022 10:29:10 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Chuck Lever III <chuck.lever@oracle.com>
Cc:     Leon Romanovsky <leon@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH v1] RDMA/core: Fix check_flush_dependency splat on addr_wq
Message-ID: <YwjKpoVbd1WygWwF@nvidia.com>
References: <166118222093.3250511.11454048195824271658.stgit@morisot.1015granger.net>
 <YwSLOxyEtV4l2Frc@unreal>
 <584E7212-BC09-48E1-A27E-725E54FA075E@oracle.com>
 <YwXtePKW+sn/89M6@unreal>
 <591D1B3D-B04A-4625-8DD0-CA0C2E986345@oracle.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <591D1B3D-B04A-4625-8DD0-CA0C2E986345@oracle.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?HOG6Bwsaq8WJ3kiqhd3nLo2P4WXbIZwS8Du2ghwdpQfNIuwVzHKIB1GxZ94T?=
 =?us-ascii?Q?5C73dFp0uFb5/FJPCFspjYA0T3KODvdAUyHghW8w+cR9evkYIJqI3R8bZAte?=
 =?us-ascii?Q?S3/m4ZWL/vbCwYhZ8ogHs/yCjZ4mRkYsBj+F0RIN0fcoG6p8lJrm/g54rCCj?=
 =?us-ascii?Q?Q1KIDgG8VvT/HSR0zqoFnwagA1JIByOUvFT+SN9u23OjRH89U4HVRX8dwxGc?=
 =?us-ascii?Q?Bn1OUCX4uBT8fRPuUqs0evub1AEHiS0ZjHlEeQVdrc9/ADQG2DNSPvXVAeuh?=
 =?us-ascii?Q?D+0avQAHTHQJVB7sRJQsOSnKWNMKAJAxQwhVvV5jtuNWMWP1UAQI2dKS+GSo?=
 =?us-ascii?Q?7DwYf1799b/GshAxPQg94nsMD1k8jgXrjMZFQLFc1gKF4Mhx+1nePhTve4A8?=
 =?us-ascii?Q?iHLvJ4ZAn1RkWLOVCFUjuknqDEk4WYwk6PO8aoFGu4H6zJX7Baz+tUspyrYh?=
 =?us-ascii?Q?dRTvU5Lah9uPdRIN9Om97E86ujyzclqydqP1KwkLAMRW57BLIHBwIMArVljY?=
 =?us-ascii?Q?EYYoEfv/DLPrOMzsVuSq8uRlUajOdQ+UHAO59md6Z5848jT8JaAnAEI9aPFC?=
 =?us-ascii?Q?aedHNq042NOemD4gZuQ800uw5WI+KqL0K83FzzKZRK4pYGAbmpOhQqYTkLw3?=
 =?us-ascii?Q?+BYmc9GE90j932crKsiEPUPnuE7levwEeQ+nH3u1Ugb3QZxG335WvbTIFvAq?=
 =?us-ascii?Q?csGU954qVJyl/bIAJmQ3tk/n8bAABPa72B8WVPHhpGaUgaS6x8ZXVUuBKvzM?=
 =?us-ascii?Q?har4Md683dYkrKO92KcN4OJ2TsagLlBxOUyL6huGaoEmrwgzHncV5PXSXeB3?=
 =?us-ascii?Q?FyvM+B6Cxj49wn3hwhZMU1Qxi1+TELzc2IE05qknj90xlXjKzLs9sXxrX5Rq?=
 =?us-ascii?Q?UBL7xNal7N1sORSFI3YnR4TGqg9Rz6d3vtG0FcLheA3c8D7d2aFpQKwTJlg0?=
 =?us-ascii?Q?QUHnpgAn1QKM9O4eorv7m/y10yZw7TL+Q+mTNq78hDoSaornaAkxC/2NvviF?=
 =?us-ascii?Q?1/0x3FgzkkZpQ0jgAkY5cqLQLkkq8e9qli4dcYl/gPI/4VmNK//SvX1PMf/P?=
 =?us-ascii?Q?97qrQwKeB8HSJj/laKbvIJG0NLsjISFb4YGn7bgz3qc+8/DU5lW6Pn9FX5wZ?=
 =?us-ascii?Q?9c4CSnQZ/DAD1M62+RlNzz/KBgUNz8Azqt5YH0L6skHGjkWe0UTZ9n2dcd+/?=
 =?us-ascii?Q?nsLHK3hEQzQ3PsjWQyFOIQy8vADhf3SQojz/95I6S5Di1B16GaWQ5OI2YBrn?=
 =?us-ascii?Q?aLHKHXhv9Sa9iD1lNjNkrQubRwbqBia9BRWQRyJZ6Yp64A3jl4+lG+ofpkuV?=
 =?us-ascii?Q?f1F6aCfPfksT2JF/NC2hk3ZUVGxAJrlvKu2EpBEqR2k3lNAncDi7s1z7EFwU?=
 =?us-ascii?Q?Zkn15FacmkPmLMOuU2nyLlYT2Qm97Q/g9PvP3D0G1fVw1ySZ0G8IPFdt/aQc?=
 =?us-ascii?Q?LV5Ogoxqo5jEpIw+hGpiGsTPDcl5zni4JzF9NSskC4A3EJK4z7QhDZ0Nib6a?=
 =?us-ascii?Q?WNmzBGK+4GyjkxP/THTAojoMAlYLujIGwD0QA3x6a9xBeIAcysbR50D2XTyI?=
 =?us-ascii?Q?fQ7vGaJ78IQBL/Yk00OV1sKEBto0/TzgL/4B3oGh?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c05111a1-e9fe-4b69-a898-08da8766f610
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2022 13:29:11.2297
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EqgIgo/1LJDXZqUEi9XqnTFFTJBi060AeyYr1dZK06AuWULDPIlWX4RgMRDpgT2w
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR12MB2540
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

On Wed, Aug 24, 2022 at 02:09:52PM +0000, Chuck Lever III wrote:
> 
> 
> > On Aug 24, 2022, at 5:20 AM, Leon Romanovsky <leon@kernel.org> wrote:
> > 
> > On Tue, Aug 23, 2022 at 01:58:44PM +0000, Chuck Lever III wrote:
> >> 
> >> 
> >>> On Aug 23, 2022, at 4:09 AM, Leon Romanovsky <leon@kernel.org> wrote:
> >>> 
> >>> On Mon, Aug 22, 2022 at 11:30:20AM -0400, Chuck Lever wrote:
> > 
> > <...>
> > 
> >>>> The xprtiod work queue is WQ_MEM_RECLAIM, so any work queue that
> >>>> one of its work items tries to cancel has to be WQ_MEM_RECLAIM to
> >>>> prevent a priority inversion.
> >>> 
> >>> But why do you have WQ_MEM_RECLAIM in xprtiod?
> >> 
> >> Because RPC is under a filesystem (NFS). Therefore it has to handle
> >> writeback demanded by direct reclaim. All of the storage ULPs have
> >> this constraint, in fact.
> > 
> > I don't know, this ib_addr workqueue is used when connection is created.
> 
> Reconnection is exactly when we need to ensure that creating
> a new connection won't trigger more memory allocation, because
> that will immediately deadlock.
> 
> Again, all network storage ULPs have this constraint.

IMHO this whole concept is broken.

The RDMA stack does not operate globally under RECLAIM, nor should it.

If you attempt to do a reconnection/etc from within a RECLAIM context
it will deadlock on one of the many allocations that are made to
support opening the connection.

The general idea of reclaim is that the entire task context working
under the reclaim is marked with an override of the gfp flags to make
all allocations under that call chain reclaim safe.

But rdmacm does allocations outside this, eg in the WQs processing the
CM packets. So this doesn't work and we will deadlock.

Fixing it is a big deal and needs more that poking WQ_MEM_RECLAIM here
and there..

For instance, this patch is just incorrect, you can't use
WQ_MEM_RECLAIM on a WQ that is doing allocations and expect anything
useful to happen:

addr_resolve()
 addr_resolve_neigh()
  fetch_ha()
   ib_nl_fetch_ha()
     ib_nl_ip_send_msg()
       	skb = nlmsg_new(len, GFP_KERNEL);

So regardless of the MEM_RECLAIM the *actual work* being canceled can
be stuck on an non-reclaim-safe allocation.

Jason