Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Leon Romanovsky <leon@kernel.org>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH v1] RDMA/core: Fix check_flush_dependency splat on addr_wq
Thread-Topic: [PATCH v1] RDMA/core: Fix check_flush_dependency splat on
 addr_wq
Thread-Index: AQHYtjxDHzOcFbDad0CgCtqOY9xpUa28It+AgABhkYA=
Date:   Tue, 23 Aug 2022 13:58:44 +0000
Message-ID: <584E7212-BC09-48E1-A27E-725E54FA075E@oracle.com>
References: <166118222093.3250511.11454048195824271658.stgit@morisot.1015granger.net>
 <YwSLOxyEtV4l2Frc@unreal>
In-Reply-To: <YwSLOxyEtV4l2Frc@unreal>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?rrxVOfhHy/XV+IS36ZXOwRzHWXgoAeDrm/XGYProgSKCYhcuWpiU//wsFfOn?=
 =?us-ascii?Q?LrSUYLufiX9IrIDcTbFFaMMxNbknqylLLk4hCrk/bQex75PHtX8tvqdyD6fW?=
 =?us-ascii?Q?VCc5/pRTFiPSrmUwwudjQKsx9I1DaJoKHswjWh72wKHrkz2JvGfWhFPXMxz/?=
 =?us-ascii?Q?qjuq5V7IAoHSZJ4xGrVMAZ4RHuMto5eX+GmM81wtdqQicxbg8aJZ2rStUKNP?=
 =?us-ascii?Q?1gDoojIOre3aeLC/jJPuhvasanqXNv6/eCjtCLoU5R1PmcDT8mItjlo0N9zx?=
 =?us-ascii?Q?vWFhvL0dSGnfF6j+NFFQGnB2/rvfuI03jetlQIViWQpk5ZzWd3mlLdjnIshU?=
 =?us-ascii?Q?y9NDypgSzdoCRt0clo004KI5AvGVJ+1kmehrqY5sYk/Jfche4sKnT4dFoheY?=
 =?us-ascii?Q?Or0Cm6lpZGj8YC4KIeCj1B3UN1S5u/6laTKpPg6kKRlFUt3Tn4slWTg93VCs?=
 =?us-ascii?Q?y4C9XKPy1J8l8JFWiSpreim5kxkEyurBxFKqbywdDUu5BhfsJwDojtICd+gC?=
 =?us-ascii?Q?QU/8JTYEKp4RLJF0QPNax9bIQ9xH3jx/eZdmlhPk6dIl4l9ZgYuTxtMEXJA4?=
 =?us-ascii?Q?V75Qt8r7M7KheEQE9cWYCBoksCE/mAFGCm2NuAfIrulchUKvHHGxY2oKzW6t?=
 =?us-ascii?Q?LerzdxNA+rXqvyjUwu1xCzyBuuzB9tCxewHs2cVCTozgo3dRHRgt842P/Z+S?=
 =?us-ascii?Q?o4QKsDQvUdPdMy8xWTyCpuv8LeGq/b3/ULkgMLb5iV3GE7MmqwoohvIixwTX?=
 =?us-ascii?Q?r4cE1qhLgCmKKhIU9RdCr9PXgOpNYHLi6p9dBQmvsg70prVNz3oVDTY4jCeo?=
 =?us-ascii?Q?vvDsL36mF7ItMdpKGZWvXwtSCgpKC8rFu2nNRLHv1HOzhHU2zFGMjOD6+xoO?=
 =?us-ascii?Q?vMx0wC3uxDNQNuQRbSI4OthWijZGLvlrY1EdGzTLsMsJtziKn2KJWLIBBLr3?=
 =?us-ascii?Q?YgkjSP/6Ri56ixfHUr19fPaYoM4DQ09xAeeHISWo/u/uMC7pFOk64G0Vg01k?=
 =?us-ascii?Q?ON9ZK81yg4SGh/CUGtPXZHeYMWKhzniAeBWMYjSRQ7+p/wS7+MStk9WZuPi7?=
 =?us-ascii?Q?kYbzNxQkBPsKw90cXxR78vYKwTrgaWLYLHfbfHfmVRGX8bCGaN5kHYyUmN3c?=
 =?us-ascii?Q?v25nPBjeCOqK/suhNNUG8KIhl3kQxfMKHrINteCwNNO2ciO7w7W+aE0gsMFD?=
 =?us-ascii?Q?xFx+5hLnnGzitdN5vSwfwXiQradF4YhrbHGIl0ZlXyHLZmXvu2wFbybaeveC?=
 =?us-ascii?Q?mr+ZUSKZ5veJfa0MX/3owfrY1mZL7aWK+LrdJEgf1Z8z3DWLt/rvaX2dP9/F?=
 =?us-ascii?Q?4B0mhiZooh6tVx6iOg6U9Pp7VULbS/RBuACWbSOWQVs9RiIb233VuAfqbaLw?=
 =?us-ascii?Q?txtZvf4aavS+shZQu5lUcZTGSOEPZU9MfB5DyB8ywaynCkONgd1/o9JPudV8?=
 =?us-ascii?Q?0Nmsba/0WKDPtwd+58QrYATIfxD6XNcqGDLE5c91f7ytwPXAS78n0TBh+ypM?=
 =?us-ascii?Q?A/2fFVN+9PMxzbQ+8ZXle/c4Y9QbvxUDZk+r5GzFjujwg7gmllCPgzTUioBd?=
 =?us-ascii?Q?b3cQRCCj8WzSRp8hixOQtnJmW/TZeerbkPnw+L8pEQCePTHPoJXaOmavru7V?=
 =?us-ascii?Q?2g=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <FC4E7E6A28D1434A8020EC47198F830F@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c0a58893-6082-4a4a-afc2-08da850f97a0
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Aug 2022 13:58:44.0771
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: QpBKWa5aoepLaq6ej+kL49TFwjlzI1XAHSzOqEVKF5dI988fsy1VTjRNPB59MoN3kQTCc+ABGk0+q8HYh6rXUA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5245
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-08-23_05,2022-08-22_02,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 bulkscore=0
 mlxlogscore=999 spamscore=0 phishscore=0 suspectscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2207270000
 definitions=main-2208230056
X-Proofpoint-ORIG-GUID: fzX87KF7HldojEySnL1tS79BWsQwXTVH
X-Proofpoint-GUID: fzX87KF7HldojEySnL1tS79BWsQwXTVH
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Aug 23, 2022, at 4:09 AM, Leon Romanovsky <leon@kernel.org> wrote:
>=20
> On Mon, Aug 22, 2022 at 11:30:20AM -0400, Chuck Lever wrote:
>> While setting up a new lab, I accidentally misconfigured the
>> Ethernet port for a system that tried an NFS mount using RoCE.
>> This made the NFS server unreachable. The following WARNING
>> popped on the NFS client while waiting for the mount attempt to
>> time out:
>>=20
>> Aug 20 17:12:05 bazille kernel: workqueue: WQ_MEM_RECLAIM xprtiod:xprt_r=
dma_connect_worker [rpcrdma] is flushing !WQ_MEM_RECLAI>
>> Aug 20 17:12:05 bazille kernel: WARNING: CPU: 0 PID: 100 at kernel/workq=
ueue.c:2628 check_flush_dependency+0xbf/0xca
>> Aug 20 17:12:05 bazille kernel: Modules linked in: rpcsec_gss_krb5 nfsv4=
 dns_resolver nfs 8021q garp stp mrp llc rfkill rpcrdma>
>> Aug 20 17:12:05 bazille kernel: CPU: 0 PID: 100 Comm: kworker/u8:8 Not t=
ainted 6.0.0-rc1-00002-g6229f8c054e5 #13
>> Aug 20 17:12:05 bazille kernel: Hardware name: Supermicro X10SRA-F/X10SR=
A-F, BIOS 2.0b 06/12/2017
>> Aug 20 17:12:05 bazille kernel: Workqueue: xprtiod xprt_rdma_connect_wor=
ker [rpcrdma]
>> Aug 20 17:12:05 bazille kernel: RIP: 0010:check_flush_dependency+0xbf/0x=
ca
>> Aug 20 17:12:05 bazille kernel: Code: 75 2a 48 8b 55 18 48 8d 8b b0 00 0=
0 00 4d 89 e0 48 81 c6 b0 00 00 00 48 c7 c7 65 33 2e be>
>> Aug 20 17:12:05 bazille kernel: RSP: 0018:ffffb562806cfcf8 EFLAGS: 00010=
092
>> Aug 20 17:12:05 bazille kernel: RAX: 0000000000000082 RBX: ffff97894f8c3=
c00 RCX: 0000000000000027
>> Aug 20 17:12:05 bazille kernel: RDX: 0000000000000002 RSI: ffffffffbe344=
7d1 RDI: 00000000ffffffff
>> Aug 20 17:12:05 bazille kernel: RBP: ffff978941315840 R08: 0000000000000=
000 R09: 0000000000000000
>> Aug 20 17:12:05 bazille kernel: R10: 00000000000008b0 R11: 0000000000000=
001 R12: ffffffffc0ce3731
>> Aug 20 17:12:05 bazille kernel: R13: ffff978950c00500 R14: ffff97894341f=
0c0 R15: ffff978951112eb0
>> Aug 20 17:12:05 bazille kernel: FS:  0000000000000000(0000) GS:ffff97987=
fc00000(0000) knlGS:0000000000000000
>> Aug 20 17:12:05 bazille kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000=
080050033
>> Aug 20 17:12:05 bazille kernel: CR2: 00007f807535eae8 CR3: 000000010b8e4=
002 CR4: 00000000003706f0
>> Aug 20 17:12:05 bazille kernel: DR0: 0000000000000000 DR1: 0000000000000=
000 DR2: 0000000000000000
>> Aug 20 17:12:05 bazille kernel: DR3: 0000000000000000 DR6: 00000000fffe0=
ff0 DR7: 0000000000000400
>> Aug 20 17:12:05 bazille kernel: Call Trace:
>> Aug 20 17:12:05 bazille kernel:  <TASK>
>> Aug 20 17:12:05 bazille kernel:  __flush_work.isra.0+0xaf/0x188
>> Aug 20 17:12:05 bazille kernel:  ? _raw_spin_lock_irqsave+0x2c/0x37
>> Aug 20 17:12:05 bazille kernel:  ? lock_timer_base+0x38/0x5f
>> Aug 20 17:12:05 bazille kernel:  __cancel_work_timer+0xea/0x13d
>> Aug 20 17:12:05 bazille kernel:  ? preempt_latency_start+0x2b/0x46
>> Aug 20 17:12:05 bazille kernel:  rdma_addr_cancel+0x70/0x81 [ib_core]
>> Aug 20 17:12:05 bazille kernel:  _destroy_id+0x1a/0x246 [rdma_cm]
>> Aug 20 17:12:05 bazille kernel:  rpcrdma_xprt_connect+0x115/0x5ae [rpcrd=
ma]
>> Aug 20 17:12:05 bazille kernel:  ? _raw_spin_unlock+0x14/0x29
>> Aug 20 17:12:05 bazille kernel:  ? raw_spin_rq_unlock_irq+0x5/0x10
>> Aug 20 17:12:05 bazille kernel:  ? finish_task_switch.isra.0+0x171/0x249
>> Aug 20 17:12:05 bazille kernel:  xprt_rdma_connect_worker+0x3b/0xc7 [rpc=
rdma]
>> Aug 20 17:12:05 bazille kernel:  process_one_work+0x1d8/0x2d4
>> Aug 20 17:12:05 bazille kernel:  worker_thread+0x18b/0x24f
>> Aug 20 17:12:05 bazille kernel:  ? rescuer_thread+0x280/0x280
>> Aug 20 17:12:05 bazille kernel:  kthread+0xf4/0xfc
>> Aug 20 17:12:05 bazille kernel:  ? kthread_complete_and_exit+0x1b/0x1b
>> Aug 20 17:12:05 bazille kernel:  ret_from_fork+0x22/0x30
>> Aug 20 17:12:05 bazille kernel:  </TASK>
>>=20
>> The xprtiod work queue is WQ_MEM_RECLAIM, so any work queue that
>> one of its work items tries to cancel has to be WQ_MEM_RECLAIM to
>> prevent a priority inversion.
>=20
> But why do you have WQ_MEM_RECLAIM in xprtiod?

Because RPC is under a filesystem (NFS). Therefore it has to handle
writeback demanded by direct reclaim. All of the storage ULPs have
this constraint, in fact.


>  1270         wq =3D alloc_workqueue("xprtiod", WQ_UNBOUND | WQ_MEM_RECLA=
IM, 0);
>=20
> IMHO, It will be nicer if we remove WQ_MEM_RECLAIM instead of adding it.
>=20
> Thanks
>=20
>>=20
>> Suggested-by: Trond Myklebust <trondmy@hammerspace.com>
>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>> ---
>> drivers/infiniband/core/addr.c |    2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>=20
>> diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/ad=
dr.c
>> index f253295795f0..5c36d01ebf0b 100644
>> --- a/drivers/infiniband/core/addr.c
>> +++ b/drivers/infiniband/core/addr.c
>> @@ -872,7 +872,7 @@ static struct notifier_block nb =3D {
>>=20
>> int addr_init(void)
>> {
>> -	addr_wq =3D alloc_ordered_workqueue("ib_addr", 0);
>> +	addr_wq =3D alloc_ordered_workqueue("ib_addr", WQ_MEM_RECLAIM);
>> 	if (!addr_wq)
>> 		return -ENOMEM;

--
Chuck Lever