Received-SPF: pass (google.com: domain of linux-kernel+bounces-128843-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: Kamaljit Singh <Kamaljit.Singh1@wdc.com>
To: Chaitanya Kulkarni <chaitanyak@nvidia.com>, Sagi Grimberg
	<sagi@grimberg.me>
CC: "kbusch@kernel.org" <kbusch@kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "linux-nvme@lists.infradead.org"
	<linux-nvme@lists.infradead.org>
Subject: Re: WQ_UNBOUND workqueue warnings from multiple drivers
Thread-Topic: WQ_UNBOUND workqueue warnings from multiple drivers
Thread-Index: AQHaeXhd/UIUk2VZGEG0Qh+3CF4eQLFAWi8AgAIfawCAEyxD9w==
Date: Tue, 2 Apr 2024 23:50:28 +0000
Message-ID:
 <BYAPR04MB415196EC84756BA1876463DABC3E2@BYAPR04MB4151.namprd04.prod.outlook.com>
References:
 <BYAPR04MB41513F04DB2ECDC3601CC36ABC2D2@BYAPR04MB4151.namprd04.prod.outlook.com>
 <c4057654-97bd-4721-9bed-9dd5ef8b3f8d@grimberg.me>
 <6d3af8dd-30c3-48d4-9083-7f00ea21ff8c@nvidia.com>
In-Reply-To: <6d3af8dd-30c3-48d4-9083-7f00ea21ff8c@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
msip_labels:
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: BYAPR04MB4151:EE_|PH0PR04MB8475:EE_
wdcipoutbound: EOP-TRUE
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info:
 dLC6zcrrknFRql0t1bbUUvBVl2Uk64ehon6eEFbuOXrLbknjgR/DGCPBEdk6IN5VDxj0WxqOqzRhhdENh4SjQZDawpHj8ISzkuGizZE4o1G66ESImv4y3WCn4G3dq/9LNGdNnDh8VDNQcFK+0VIohaT6KqZR52vEvhYgAf4BHfwz0P6SNHXik5f7/kKKsTdOI7vMGdD/JHDR4nfBoaoG/4pCGvYXcRHTy/niqR+uPQ7Tvo0FkUnuYeoTFWopqXbJtnNEv2+tJNC5FPnnyaAYt5noG2W4QSgGwI2kFoFAlvweL2Te4kS+APA81QgWcUazUe5FiagzkClH/K7+vwroLfm7j3VOxKuH/sPYnH5YiR0WXHHuV47mf1G2OX3uW5msfiZQvWw3NIxHHrZgpKnMdcr9bv3l5/EwKhb0VpLu4Qfg9fI6WgjkQc5E9evAX3QAlhtObHj/XKzmxTnnbM/mO3O1v0QCz9X00OKHcoRH+qiepokLUs9B1SkvQ9+2ziCws8EM7k+X5Ho+k6Mk2HsObUaaudwdhqTF0kam3MmmkvqcoJH5Krw5VsblGu9ZW2N4Swjpt4bwCp108pKZLveDRXMJwyr6qGPCzXqHTWI1GjQ=
x-forefront-antispam-report:
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR04MB4151.namprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(1800799015)(366007);DIR:OUT;SFP:1102;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?iso-8859-1?Q?ABVjOs2jIEvDvTxwaYraGlV8d1GeUZbuV7+E3CMdQvXoCLZSyzhfD42GJI?=
 =?iso-8859-1?Q?vH4VfznBqW9MV/I75/ciT+Wts+OB1FvgThM3eD8eUozALgsd2PhPVTxxEP?=
 =?iso-8859-1?Q?mjZMJOINwqOjIkznacLJ5PcRvG3EfqpepUFiEDw4qgr/CmFlu5DD6igfzY?=
 =?iso-8859-1?Q?A40SLgRb4Gg3gxrwVSeTK2Y0oJPQfovv9azPES1W49e8VORCUyQfK9FGbh?=
 =?iso-8859-1?Q?slJCL54PIHkmZNy3nMwF32rU/IPbPhgrKEi/IVnWE3dI4rZaHTEaua5UT+?=
 =?iso-8859-1?Q?/2R2g6frm9vbbqEKHAYgiffDztmhRnrtxRJkVQuZSSPdRcPsTjBCIDSYg2?=
 =?iso-8859-1?Q?YnNEU5tU4y1YaoBPBk87/Q66bl7MPPXcIbhqGyd5bgAGhYhs7QncAV5Svm?=
 =?iso-8859-1?Q?W39ZMUXwhvTvnwgSAYzSK/0OB5JP7nZuUeirQjMhgJPHXjheNTWfnpcPr7?=
 =?iso-8859-1?Q?QyGVpJ7gm8B0zOo+vTcLaokknJrRUgiWnGPwJQddesBlhGlHW+zCR9qP/z?=
 =?iso-8859-1?Q?lkUrxTb2u9xjvfylkoLeGabXVeNOpdi5RiPbNBEnUWKLqx+CxiBVrXGvnE?=
 =?iso-8859-1?Q?lbBskoCzIWz2mPRUnLnqO+icg9QEQ3DB+UYq3Z+x0VGsbDg8tAg53uetYh?=
 =?iso-8859-1?Q?GLraYUdUgydxcPIgb2YoNLAwAabg+z2f7pX3vPoE0weC1APRYqIUu00nXB?=
 =?iso-8859-1?Q?IlPuSaBxYGZv1tsXYMj1dfpD/UH+fyPUyWSwUm+pscgLM9E9ODhdlLe9e+?=
 =?iso-8859-1?Q?0BbAxUaDKlhHy/viBt5hi6vvk6lr6ZTP/TI+koJiqYM3dcdVVLyAC6tNis?=
 =?iso-8859-1?Q?hFkd8tSS16MSoalRcmF2zKgvKA3wyAXByy+DPog82Vpc/inZ6G/QgyhnsS?=
 =?iso-8859-1?Q?RU8GjrqMtWVuhpMQucgK39hCZn1UF8E9r6h8V5K5TSncL+NhUVBd9PGcYE?=
 =?iso-8859-1?Q?3dNix+XHbmRnX9SemUAD93fLIAHhApe1ZPAj4a7+HamFMVZ2HqHBAFrxWX?=
 =?iso-8859-1?Q?hxaJED8gBCyDOahohf2bWUtGPXbvUHMip/lXz6G/wptcq9evmyIoBeoqYJ?=
 =?iso-8859-1?Q?XTInxKATH31humn/D5FymNHDfCzRHxXhBVNPQbG4uONhAyGHg3x3TLHzUZ?=
 =?iso-8859-1?Q?BX31NIZBhlyHA4TJbokdtrspdMvdq9dToD76SYhhIZBmJ+y7C0ZVFbrGU/?=
 =?iso-8859-1?Q?1ax3uzb0L7OeGX9/5u9or/PhQwxhDfHDwxOxzLuIHRkqHclkOxdazMqobo?=
 =?iso-8859-1?Q?d5/lhlIeuajdmp+7iSWHhOvxTuQsm6uoc5qTHtc3u8tbAAREBl8Zz8QhRf?=
 =?iso-8859-1?Q?IFaefQKNxk/nNy3daEsWBRTixz6kIT/ZJpL6+eOiEkg7rqPo2f3u+2dtRC?=
 =?iso-8859-1?Q?K95gUtV0v9HhVCyG2QFrdpodUZdb2qcuOFCmTo0HidPKuFv0KNLLanF0+H?=
 =?iso-8859-1?Q?rS+mXK+5VVJ3kgPgkk5OAK0JowLtdg2wLan5OJf78cidkC84WLBSYRXRnP?=
 =?iso-8859-1?Q?+eKqMaGpNnQRQtyO4RAHrPHU8E6jRoGkM+68juBw9mAI9T13Z7DNpWrR6K?=
 =?iso-8859-1?Q?XSCYiWeze9GcXFqGsqpikdm9xyUq0wSB813YA3pr1PGMSqQ24nQKWmacIg?=
 =?iso-8859-1?Q?nKam0cth+p4uORSMXlljZzlfrOsAhbFdim?=
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0:
	IyEscXllXCsNJ5AZNsUtC4euFADutUZcqWEB0boc2dOJhnydc47amcbIx8ox2Y1jlPEaRmdGOLpMpglY1CdASDTYx3AiOsxfGXBsVPHrLG7frzToGzvUPcIeCwUXlzBAHcSpTBg7VDGp4mUndhYvwypd2R7hPU5WPssnq8bxJHOtjVcNO8Alk2B1ullscp2QDTZytNpTF7drZ/PEHWZwBPySEkm7e/+jEM2/AnZTXzRGzr5p8cxGKvp9PdbeeCq6elcm27D1PjJasPZOk3eKVcthrykmozvjyPr95oo7oipWbfuburCXoUKrz7BbjhSeJ1cwrR7AVjttOffwbu2ZEM+jF2aU4WGag9jvCdqsrcbAiqLqFa8uR+os1fupXTdz+44UBh06x2BYMIbUKI8wyCQP3mXyYp8sJ6wtUMe05aXmD4+4qMIGfORtNYWNpuIS/O5rvqhcQRxFsFmPc1u+Sdpboflx0zZc4SX2gL4Q734UtwuX6bJtDjIu5w/8jtwje4muC4qUOS5QDHFxzYeRtJfz9a27G0ppTgfyQqvGXQYjze1tagHTYrOPs435YfFXjex1PFLCQCpV/HOS2xq9oPZu4ZHppZCHW9OkV2gdOBucV5WdG7rHfBhbh62j1fxo
X-OriginatorOrg: wdc.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BYAPR04MB4151.namprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: aeba80fe-6b85-4889-3e65-08dc536face7
X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Apr 2024 23:50:28.6608
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: tHOjqFZyMpb6g0nVKqjunp4bO6wL4Z8BozbG8CEu3ZLMCQhAHEYYuy7r2g3QfbnbKBxzh2a9wvojgRWqDF88NQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR04MB8475

Sagi, Chaitanya,=0A=
=A0=0A=
Sorry for the delay, found your replies in the junk folder :(=0A=
=A0=0A=
>=A0Was the test you were running read-heavy?=0A=
No, most of the failing fio=A0tests were doing heavy writes. All were with =
8 Controllers and 32 NS each. io-specs are below.=0A=
=0A=
[1] bs=3D16k, iodepth=3D16, rwmixread=3D0, numjobs=3D16=0A=
Failed in ~1 min=0A=
=0A=
Some others were:=0A=
[2] bs=3D8k, iodepth=3D16, rwmixread=3D5, numjobs=3D16 =0A=
[3] bs=3D8k, iodepth=3D16, rwmixread=3D50, numjobs=3D16 =0A=
=A0=0A=
Thanks,=0A=
Kamaljit=0A=
=A0=0A=
From: Chaitanya Kulkarni <chaitanyak@nvidia.com>=0A=
Date: Thursday, March 21, 2024 at 10:36=0A=
To: Sagi Grimberg <sagi@grimberg.me>, Kamaljit Singh <Kamaljit.Singh1@wdc.c=
om>=0A=
Cc: kbusch@kernel.org <kbusch@kernel.org>, linux-kernel@vger.kernel.org <li=
nux-kernel@vger.kernel.org>, linux-nvme@lists.infradead.org <linux-nvme@lis=
ts.infradead.org>=0A=
Subject: Re: WQ_UNBOUND workqueue warnings from multiple drivers=0A=
CAUTION: This email originated from outside of Western Digital. Do not clic=
k on links or open attachments unless you recognize the sender and know tha=
t the content is safe.=0A=
=0A=
=0A=
On 3/20/24 02:11, Sagi Grimberg wrote:=0A=
>=0A=
>=0A=
> On 19/03/2024 0:33, Kamaljit Singh wrote:=0A=
>> Hello,=0A=
>>=0A=
>> After switching from Kernel v6.6.2 to v6.6.21 we're now seeing these=0A=
>> workqueue=0A=
>> warnings. I found a discussion thread about the the Intel drm driver=0A=
>> here=0A=
>> https://lore.kernel.org/lkml/ZO-BkaGuVCgdr3wc@slm.duckdns.org/T/=0A=
>>=0A=
>> and this related bug report=0A=
>> https://gitlab.freedesktop.org/drm/intel/-/issues/9245=0A=
>> but that that drm fix isn't merged into v6.6.21. It appears that we=0A=
>> may need the same=0A=
>> WQ_UNBOUND change to the nvme host tcp driver among others.=0A=
>>=A0=A0 [Fri Mar 15 22:30:06 2024] workqueue: nvme_tcp_io_work [nvme_tcp]=
=0A=
>> hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND=0A=
>> [Fri Mar 15 23:44:58 2024] workqueue: drain_vmap_area_work hogged CPU=0A=
>> for >10000us 4 times, consider switching to WQ_UNBOUND=0A=
>> [Sat Mar 16 09:55:27 2024] workqueue: drain_vmap_area_work hogged CPU=0A=
>> for >10000us 8 times, consider switching to WQ_UNBOUND=0A=
>> [Sat Mar 16 17:51:18 2024] workqueue: nvme_tcp_io_work [nvme_tcp]=0A=
>> hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND=0A=
>> [Sat Mar 16 23:04:14 2024] workqueue: nvme_tcp_io_work [nvme_tcp]=0A=
>> hogged CPU for >10000us 16 times, consider switching to WQ_UNBOUND=0A=
>> [Sun Mar 17 21:35:46 2024] perf: interrupt took too long (2707 >=0A=
>> 2500), lowering kernel.perf_event_max_sample_rate to 73750=0A=
>> [Sun Mar 17 21:49:34 2024] workqueue: drain_vmap_area_work hogged CPU=0A=
>> for >10000us 16 times, consider switching to WQ_UNBOUND=0A=
>> ...=0A=
>> workqueue: drm_fb_helper_damage_work [drm_kms_helper] hogged CPU for=0A=
>> >10000us 32 times, consider switching to WQ_UNBOUND=0A=
>=0A=
> Hey Kamaljit,=0A=
>=0A=
> Its interesting that this happens because nvme_tcp_io_work is bound to=0A=
> 1 jiffie.=0A=
> Although in theory we do not stop receiving from a socket once we=0A=
> started, so=0A=
> I guess this can happen in some extreme cases. Was the test you were=0A=
> running=0A=
> read-heavy?=0A=
>=0A=
> I was thinking that we may want to optionally move the recv path to=0A=
> softirq instead to=0A=
> get some latency improvements, although I don't know if that would=0A=
> improve the situation=0A=
> if we end up spending a lot of time in soft-irq...=0A=
>=0A=
>>=A0=A0=A0=A0 Thanks,=0A=
>> Kamaljit Singh=0A=
>=0A=
>=0A=
=0A=
we need a regular test for this in blktests as it doesn't look like we=0A=
caught this in=0A=
regular testing ...=0A=
=0A=
Kamaljit, can you please provide details of the tests you are running so=0A=
we can=0A=
reproduce ?=0A=
=0A=
-ck=0A=
=0A=