Received: by 2002:a05:6358:4e97:b0:b3:742d:4702 with SMTP id ce23csp4971633rwb; Wed, 17 Aug 2022 08:54:17 -0700 (PDT) X-Google-Smtp-Source: AA6agR5Q03WttivDz+Qlf0nZPhDV9hUa6PyEzYti8W3u4ahvacBDCCdRG423WIr5KmIafp7Gu44o X-Received: by 2002:a17:907:6d8b:b0:730:c160:1f9c with SMTP id sb11-20020a1709076d8b00b00730c1601f9cmr17370773ejc.342.1660751657597; Wed, 17 Aug 2022 08:54:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1660751657; cv=none; d=google.com; s=arc-20160816; b=z3Rni2cuReExNJxUTfPXkRzoche4gOJHfoZ5RWGM2DrZtQbYcx+qQ2cuEozFTxVe/9 fg/HpEadEOia8iXm8uKTuyL488p54ami6GYrZohjh8FJRR4SX33OxcABLnHfqZetmWx5 /8Wf0YWl/xBS3gSmt3DtOdw+MBwij7xl42V4g6MXaXYkM+ujdg3swCJDz/FTsxnhq72A cGlhTNCMeHkpBlTpOG5hh4BqpL7BZXSSUtXI/EbVyz6qqJRNTh43HjOvB7KWCjYsNZmT utkP6oXdEuUQWdmoJLxXgMXGoazjy+Zn2Jg4haNO5uokjy0FaHLPvvP9V15hkaSBH7wP MepA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=SqR01pq9tq6qm7V2NPRinZ2qHQbpsqAKxq332tGf5Q4=; b=ovLD8Gk0EspCyotSWbXofxtRBT1o2Kw2gP0eiwqiWuNa7HpairWaQpeNGoUEBy+vD0 xOcaAX9DyyzJRhVDT8qySMsIuuNdz1Jgz00HCF2ZSoffd/rqL0DqJV7c0xqg8gxhL66J QQ/G379dr/T07vBqUByh+GcvFHLKxxu31JjqhNaDanHrdmiboakjA5IzpYjsYtsRnPR4 9Ste6AkpFhUK72PtKqM2iS9MJJnJB4TknVqw8gBQ7YOQuzWJ5oWZMFEGDdtmPQoizX96 UAQ36QGwmNtBz0UUuyKgHqimJm5DtfzkldecOB4yfHVUw2mwW479dA/fzRfQ1AJiF+G1 Dcmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=D3f4L18C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s14-20020a508d0e000000b0043cea48785csi13060195eds.192.2022.08.17.08.53.51; Wed, 17 Aug 2022 08:54:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=D3f4L18C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240113AbiHQOxu (ORCPT + 99 others); Wed, 17 Aug 2022 10:53:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41750 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240124AbiHQOxs (ORCPT ); Wed, 17 Aug 2022 10:53:48 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02A9F5B798 for ; Wed, 17 Aug 2022 07:53:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660748027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=SqR01pq9tq6qm7V2NPRinZ2qHQbpsqAKxq332tGf5Q4=; b=D3f4L18CvLe41VBaHYbpzvv4vdUuTI6QGQsi8gZS3wIistfdYbnyyQK2APv490Bq8yH5yX 0MaTzzhRBDeAP+ybCINiIwDx3q/MGvCJxbKWg9cHIThrPJRZs8FH1G/+FWyrlp1HrdgYoc /DI/fw1IseLunx4vprsXEnZHN697Y1k= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-588-DvkFF6R-PC-vl_GTPJvKdA-1; Wed, 17 Aug 2022 10:53:41 -0400 X-MC-Unique: DvkFF6R-PC-vl_GTPJvKdA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2A316185A7B2; Wed, 17 Aug 2022 14:53:41 +0000 (UTC) Received: from T590 (ovpn-8-20.pek2.redhat.com [10.72.8.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3C9D9C15BB3; Wed, 17 Aug 2022 14:53:34 +0000 (UTC) Date: Wed, 17 Aug 2022 22:53:29 +0800 From: Ming Lei To: Chris Murphy Cc: Nikolay Borisov , Jens Axboe , Jan Kara , Paolo Valente , Btrfs BTRFS , Linux-RAID , linux-block , linux-kernel , Josef Bacik Subject: Re: stalling IO regression since linux 5.12, through 5.18 Message-ID: References: <61e5ccda-a527-4fea-9850-91095ffa91c4@www.fastmail.com> <4995baed-c561-421d-ba3e-3a75d6a738a3@www.fastmail.com> <2b8a38fa-f15f-45e8-8caa-61c5f8cd52de@www.fastmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Scanned-By: MIMEDefang 2.85 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 17, 2022 at 10:34:38AM -0400, Chris Murphy wrote: > > > On Wed, Aug 17, 2022, at 8:06 AM, Ming Lei wrote: > > > blk-mq debugfs log is usually helpful for io stall issue, care to post > > the blk-mq debugfs log: > > > > (cd /sys/kernel/debug/block/$disk && find . -type f -exec grep -aH . {} \;) > > This is only sda > https://drive.google.com/file/d/1aAld-kXb3RUiv_ShAvD_AGAFDRS03Lr0/view?usp=sharing From the log, there isn't any in-flight IO request. So please confirm that it is collected after the IO stall is triggered. If yes, the issue may not be related with BFQ, and should be related with blk-cgroup code. Thanks, Ming