Received: by 2002:a05:6358:4e97:b0:b3:742d:4702 with SMTP id ce23csp3255rwb; Wed, 17 Aug 2022 21:19:30 -0700 (PDT) X-Google-Smtp-Source: AA6agR50cudypGGHehJZ6FadNFqex+G9ZfGaRg687T6mG/QlNdLtKp8ytKo/ABxPbixO+nw1Hcos X-Received: by 2002:a05:6a00:8cf:b0:522:93a6:46e9 with SMTP id s15-20020a056a0008cf00b0052293a646e9mr1312992pfu.62.1660796369971; Wed, 17 Aug 2022 21:19:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1660796369; cv=none; d=google.com; s=arc-20160816; b=IlwJA0IHGduLzX9pIMyP/JvTidDzocO7Z0gAfCXCNNY+/KtH+YB81sMSDGON3hyUk/ Lu64n35XHrg1MOzDl/P3VvQa4TG8uUmKfCpiCuQpqq4ezeQC2DF29s467ZhigAbkiuqT 7yyX3okNsC7nqHIxP1sy8WLz36XHaQc0xCO44av03GBfAtwGn0eY/AdjJtAmJB00nsqe anr9gqw3kS0T3IToMapDich4qLHHFEeM2EIQAif1BBlMTxg/xIOXnKSCYANPDH1Agjee ZOSlbdBq90rSaWljrY3sG4nEEzeKhz3nOkYBHZ8lvaFM+chtFqmbSKPwbHtx5iPApQDb oWhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:subject:cc:to:from:date:references:in-reply-to :message-id:mime-version:user-agent:feedback-id:dkim-signature :dkim-signature; bh=p5RQmR6ERx2tDB2QJRfxZE9rzRDcq3welkce5QeUbYs=; b=EssufrbCFNsP2VvY28K7eQtYVkia/b6IWwy1QAovO2qWIB4yCTH8PQLk54JhCrCsRO OqGpjbAEHIMB/qj0tuFksV8oEuh0pU7mIHxrou/2qPlvq8cxDHYmLLMQmIusEj9C6uXG 6p3zkFZ9vUQB9X1qEkto2D0uqo5YclRRHe5OFbRml3LskklQjzLEoQpYKDE1+IeqY3Bp +9p0n03oWURuBrcbfhTrwgzctTC6yUjoprrS4GvhqkfH8Ng/kjvPKLAkQGHpH2Za98zu UOJ6bAjXKmTP3od+ipc66NWXIFrKgc0P7hQcAGJW5JwATeK6O8hwayLM3ut4lr8nUENd B40g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@colorremedies.com header.s=fm1 header.b=OoKnylvE; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=yLEoXdwl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id li14-20020a17090b48ce00b001f19523ebadsi587906pjb.78.2022.08.17.21.19.18; Wed, 17 Aug 2022 21:19:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@colorremedies.com header.s=fm1 header.b=OoKnylvE; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=yLEoXdwl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242865AbiHREMs (ORCPT + 99 others); Thu, 18 Aug 2022 00:12:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231547AbiHREMq (ORCPT ); Thu, 18 Aug 2022 00:12:46 -0400 Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com [66.111.4.230]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51D738E0F3; Wed, 17 Aug 2022 21:12:45 -0700 (PDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id A39A1580EA6; Thu, 18 Aug 2022 00:12:43 -0400 (EDT) Received: from imap50 ([10.202.2.100]) by compute3.internal (MEProxy); Thu, 18 Aug 2022 00:12:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= colorremedies.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; t=1660795963; x= 1660799563; bh=p5RQmR6ERx2tDB2QJRfxZE9rzRDcq3welkce5QeUbYs=; b=O oKnylvE/AVYyogIvJltr/srgIDHK8Sanss7o+l3SKKyBopX/mtCW5wbCyZy/V2rq +nb3s71owFWYOhZcNRBlnjE+Nuu1GAJeTobApxWPn8z0E9Uh9u7KraqyC3RK9Q5X bn5cYfofLRhsymLUDfpY20Ja8IK1tJdYVD+q/IO8mxDB0ajDIq6ukB+NXvvupsq4 jv+xz/gWuGS3G4E7GUjGgdtWUzgxJxE8Crhm4OcbGCXyYEfGNvN5vtXvMhfZtvpW ng1ZAksd3ska/qruZqOl4f0Ey0pW0Nl4MxslmsCesKMj9PzLzZg+JSQ4cyliPfvv 5dM2lnG2WN2fhfAeB5Sxw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1660795963; x=1660799563; bh=p5RQmR6ERx2tDB2QJRfxZE9rzRDc q3welkce5QeUbYs=; b=yLEoXdwl1f+R2Klw8SFe3faAxLUwIZaDdvGtjh1oY3sa AXMTbC0erEKcBMfzElMoBVTxXjW7f2HWL+fltGSdAZHjwXPlU6C2AVM+1jByfueu 5GfnRLCNK2l3Bd6GGpkMZGjc4KXGEIxhhOmeW7+R4U/65b7ZMARKOuxIkUdyflAv 8jbX6oeSGZ1aoFwAG+AcffCo19xGA7NVcnMlXY8IWC34uhrVX+I1kwrj+ul1nNjo SZ48mQqu17oyX8O6hLvsrUoWxqyicSuxdQe+WzMh8gNeMOhA5m4lcKlQMGujD5Os rsEIFuPoc/W1vUxK/QGq3VHZ2+DNSwEu4q/uHbJRTQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrvdehjedgkeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne goufhushhpvggtthffohhmrghinhculdegledmnecujfgurhepofgfggfkjghffffhvfev ufgtsehttdertderredtnecuhfhrohhmpedfvehhrhhishcuofhurhhphhihfdcuoehlih hsthhssegtohhlohhrrhgvmhgvughivghsrdgtohhmqeenucggtffrrghtthgvrhhnpeef heeliedugeeuleetffeuheegkeetgfdtveevudffgfejvdegveeljefhvdefhfenucffoh hmrghinhepghhoohhglhgvrdgtohhmnecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomheplhhishhtshestgholhhorhhrvghmvgguihgvshdrtghomh X-ME-Proxy: Feedback-ID: i06494636:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 0C4551700082; Thu, 18 Aug 2022 00:12:42 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-841-g7899e99a45-fm-20220811.002-g7899e99a Mime-Version: 1.0 Message-Id: <9f2f608a-cd5f-4736-9e6d-07ccc2eca12c@www.fastmail.com> In-Reply-To: References: <2b8a38fa-f15f-45e8-8caa-61c5f8cd52de@www.fastmail.com> <35f0d608-7448-4276-8922-19a23d8f9049@www.fastmail.com> <568465de-5c3b-4d94-a74b-5b83ce2f942f@www.fastmail.com> Date: Thu, 18 Aug 2022 00:12:21 -0400 From: "Chris Murphy" To: "Ming Lei" Cc: "Nikolay Borisov" , "Jens Axboe" , "Jan Kara" , "Paolo Valente" , "Btrfs BTRFS" , Linux-RAID , linux-block , linux-kernel , "Josef Bacik" Subject: Re: stalling IO regression since linux 5.12, through 5.18 Content-Type: text/plain X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 17, 2022, at 11:41 PM, Ming Lei wrote: > OK, can you post the blk-mq debugfs log after you trigger it on v5.17? https://drive.google.com/file/d/1n8f66pVLCwQTJ0PMd71EiUZoeTWQk3dB/view?usp=sharing This time it happened pretty quickly. This log is soon after triple digit load and no IO, but not as fully developed as before. The system has become entirely unresponsive to new commands, so I have to issue sysrq+b - if I let it go too long even that won't work. -- Chris Murphy