Received: by 2002:a05:6358:4e97:b0:b3:742d:4702 with SMTP id ce23csp4736816rwb; Wed, 17 Aug 2022 05:24:43 -0700 (PDT) X-Google-Smtp-Source: AA6agR7tJqrSsvmd5rpbQ1mEF7Aqcch/ybmDRs5CDW97YsbSJrXci9CTMZlYTQlo+LIB8tLIY+F3 X-Received: by 2002:a17:90b:4b52:b0:1f7:2e06:575f with SMTP id mi18-20020a17090b4b5200b001f72e06575fmr3526303pjb.120.1660739082947; Wed, 17 Aug 2022 05:24:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1660739082; cv=none; d=google.com; s=arc-20160816; b=XO2jA+wUZjWjeYOqmXTBu4nmNj3dORhpX6S8ujaxUkoLWC5EAyQa8zDUw9uU5CJD2T zv9oiDoIdWMgIiDCCtTqlInXgquKJjjhrbrX56QJlTEzcgVZUgvwwG1YBOe6cPSTHR4b /MADGDKBXgrqwgnDaF02Dhpqv9SLhh9EzYg2sGJIJoc7ingJrWgHHiIbV08W9SZUdnaG Si3fF1mmSHn+ASdsEq8FxNOqfW67ECyornPiT1cSBvxrbmMvvoL4XC0KK+ssXHylthFu NJckX8EOA1A3hkbFhRVvIG10bgSjU6qFPFPOeV1fS+WGnT23FAeoilFD8D/CeGHUmOYp warA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:subject:cc:to:from :date:references:in-reply-to:message-id:mime-version:user-agent :feedback-id:dkim-signature:dkim-signature; bh=nMczVAGpca7H++XwV1RmIcObkzLoHhiXbmBHdwyghyU=; b=uR6wWgb4rGlJZLL00CgUgeAdvlDdtLnlNjUUIPnoLyO8PjDHMOYvKEcfsk0FDmQsAN sC+7hPkKPCuz8v7eyZWCAQFq++CFQ+DJ+7WsUwhx8Gb01+/TwaYNRKiHDgBmUfiDrSzT wiw++CWb2nExEP4d0Ec8l5utQg4DJaqwkI9hgIArxcM9UkQABIds0/beh2gIHdiDig4y okHXUQxtSEVSPL2lmKCgr2oTkHnueXDSmhV6e7Zkr98OMiEgaaH5cieymLMhASPj6AIc KhOPdGTUOfsS80NU+Wnc9+n/NHOyr5ehpENdCGGfWfcYFTu2VDj7VKCEZ5RUdk340HZo EEmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@colorremedies.com header.s=fm1 header.b=WLdp1RQQ; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=RiAymQvL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n16-20020a170902e55000b00172889e9ea9si1220370plf.583.2022.08.17.05.24.31; Wed, 17 Aug 2022 05:24:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@colorremedies.com header.s=fm1 header.b=WLdp1RQQ; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=RiAymQvL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239054AbiHQL5d (ORCPT + 99 others); Wed, 17 Aug 2022 07:57:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51248 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238700AbiHQL5b (ORCPT ); Wed, 17 Aug 2022 07:57:31 -0400 Received: from wnew4-smtp.messagingengine.com (wnew4-smtp.messagingengine.com [64.147.123.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D247782D3B; Wed, 17 Aug 2022 04:57:30 -0700 (PDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.west.internal (Postfix) with ESMTP id 830E72B059F6; Wed, 17 Aug 2022 07:57:29 -0400 (EDT) Received: from imap50 ([10.202.2.100]) by compute3.internal (MEProxy); Wed, 17 Aug 2022 07:57:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= colorremedies.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1660737449; x=1660741049; bh=nMczVAGpca 7H++XwV1RmIcObkzLoHhiXbmBHdwyghyU=; b=WLdp1RQQHgQVxczuf7pW15tSAX 2fY2owzazhdQ4Vag4SdC9OvQD8X0ecFf+Jh6e8dyXMnXQuMPp/BdL+jyeMnuOE4H OPnuYzkxkoycBiFkTuCPpA1UXkQ9QdD/Dox/EZ+pXrpaTck4gM7ixADCvMqyuSMd Z0DxqNifJ4mCdf+mow2fZTea6WzwqGYe3tYwdI7gu8iPV7JJYqmIVBPsO0keMjWQ /0Y7t79djaGZTlH9vwhfsT8EAcZhY4/yQ1MUG8wwd2ENASQ0Rge0IG/taXYjwcaP FB69dJMdB3vd398ro1EN64h4y1T4rPL2Q/l+OONmI1Z/JnP8AXYl/OSWIBfA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1660737449; x= 1660741049; bh=nMczVAGpca7H++XwV1RmIcObkzLoHhiXbmBHdwyghyU=; b=R iAymQvL3bRE/7i5N7wzVWFFiIJBHAFb9Iu1SgWK8qjROI9K64jqeK8ZKnhYQZyYc E9FctukwQssAcs4io5G+Jl26u3y3uOV/DrRmHiP1+rdSQI0sxSiofx0G12aApgls 1iWyAfjYhXJkKLtg8UIIwYDshHkyGJLPnedA+BfpDtKhBtDTNcHjWf3ece6tSOla +V3kY2eVhXAceF72W+Xkpwaw7Y79WUh2ec7tTlXK22PQ6lJpNwipV1GifC/D20yD 5N07Zu0GJ0jZU9Gm2Md5ntVrhFtHR2nRVeNZeYrMEcoHpGyqADP4Vwav26oCD2/h eNREEVANYr+vdC9Fj1fLg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrvdehiedggeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtgfesthhqredtreerjeenucfhrhhomhepfdev hhhrihhsucfouhhrphhhhidfuceolhhishhtshestgholhhorhhrvghmvgguihgvshdrtg homheqnecuggftrfgrthhtvghrnhepveeghfeiveehvdeukedvtedtfefhuddtleekffek keefheeivdehgfffleeileehnecuffhomhgrihhnpehkvghrnhgvlhdrohhrghenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehlihhsthhssegt ohhlohhrrhgvmhgvughivghsrdgtohhm X-ME-Proxy: Feedback-ID: i06494636:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 761291700082; Wed, 17 Aug 2022 07:57:28 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-841-g7899e99a45-fm-20220811.002-g7899e99a Mime-Version: 1.0 Message-Id: <8b361f8e-cc4f-466c-90f0-031a43436af2@www.fastmail.com> In-Reply-To: <7c830487-95a6-b008-920b-8bc4a318f10a@applied-asynchrony.com> References: <2220d403-e443-4e60-b7c3-d149e402c13e@www.fastmail.com> <61e5ccda-a527-4fea-9850-91095ffa91c4@www.fastmail.com> <4995baed-c561-421d-ba3e-3a75d6a738a3@www.fastmail.com> <2b8a38fa-f15f-45e8-8caa-61c5f8cd52de@www.fastmail.com> <7c830487-95a6-b008-920b-8bc4a318f10a@applied-asynchrony.com> Date: Wed, 17 Aug 2022 07:57:07 -0400 From: "Chris Murphy" To: =?UTF-8?Q?Holger_Hoffst=C3=A4tte?= , "Nikolay Borisov" , "Jens Axboe" , "Jan Kara" , "Paolo Valente" Cc: Linux-RAID , linux-block , linux-kernel , "Josef Bacik" Subject: Re: stalling IO regression since linux 5.12, through 5.18 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 17, 2022, at 5:52 AM, Holger Hoffst=C3=A4tte wrote: > On 2022-08-16 17:34, Chris Murphy wrote: >>=20 >> On Tue, Aug 16, 2022, at 11:25 AM, Nikolay Borisov wrote: >>> How about changing the scheduler either mq-deadline or noop, just >>> to see if this is also reproducible with a different scheduler. I >>> guess noop would imply the blk cgroup controller is going to be >>> disabled >>=20 >> I already reported on that: always happens with bfq within an hour or >> less. Doesn't happen with mq-deadline for ~25+ hours. Does happen >> with bfq with the above patches removed. Does happen with >> cgroup.disabled=3Dio set. >>=20 >> Sounds to me like it's something bfq depends on and is somehow >> becoming perturbed in a way that mq-deadline does not, and has >> changed between 5.11 and 5.12. I have no idea what's under bfq that >> matches this description. > > Chris, just a shot in the dark but can you try the patch from > > https://lore.kernel.org/linux-block/20220803121504.212071-1-yukuai1@hu= aweicloud.com/ > > on top of something more recent than 5.12? Ideally 5.19 where it appli= es > cleanly. The problem doesn't reliably reproduce on 5.19. A patch for 5.12..5.18 w= ould be much more testable. --=20 Chris Murphy