Received: by 2002:a05:6358:4e97:b0:b3:742d:4702 with SMTP id ce23csp4753731rwb; Wed, 17 Aug 2022 05:41:53 -0700 (PDT) X-Google-Smtp-Source: AA6agR4CspNF+dHYhzt6XYVmX4hUX02YjmXJctXa9gJdYnYFQC7fbNW6FC3u0lBSfvYrY8wdTmaO X-Received: by 2002:a17:907:a40d:b0:733:17c1:a246 with SMTP id sg13-20020a170907a40d00b0073317c1a246mr16638335ejc.132.1660740112685; Wed, 17 Aug 2022 05:41:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1660740112; cv=none; d=google.com; s=arc-20160816; b=sZjK4pJ5+aiCh344Rf1CvlRJK00P0tGWNoW1PTSYPqZTIwXUnIFooPWSSlWR3qvXYv TUg8wbNMHGJsGPbZ/qlMEeenI9TqXOol+Q9anePDQPmxI+90q24+5FDo8ElvNEUkmrme C3MfV06IcXm8xdN6TJ4apRvt8NmH6e9fRvMBPRIYhnOzhvEplpvem1rhHGUrfZGYUq39 h0nXEtZyKjzK9ftGbdxHRsy5e851ZY173Jv6LgKNLVXUx+ZQmVg0388ncM7GfN5aFwvV KgAd3CYqAzEcO3EwYNZQqc5A7U0ZzRct9NnghuUs9hsoGcTRwbh135aQd30lUUdEbgK3 tnfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:in-reply-to :references:mime-version:dkim-signature; bh=uvED31tz28w3zR0MRsRpAF+4dRa0mrzWJzSKyavcZMM=; b=Cj2P8d7EsNiTaDfIse3MIEsUySZ54S8rj70mShLLgrDOZR/ndRVl0tNsbyDQ2TIqEk 9Kdsd4wJbnveejLtDbzmCFKwV+rPkqdT4hUN7A5hJXzdqamx2elnBYGc+n4HlumVTtA6 vfSUN3z4oq2t9B7ldjiLeLw+jGm9se8ACtBUEvSnM0bwQ4h292S2Dr7AUtcAVOL2SLkV ABq+hXJ3T9mhG34YbRUYB83xYlAMb5jUn5hNm5SH6SftNE3pgAxKByA8XmCQN83zb0+X qejZ3fv1g9OLwBCYJ3FTPkqeKW1vKZ/1nsOI0VPBTuX5xqTFlvN8Sj/adFib64B8ENyt bWFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=TpOUYd6N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gt30-20020a1709072d9e00b007269af37e73si81388ejc.685.2022.08.17.05.41.27; Wed, 17 Aug 2022 05:41:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=TpOUYd6N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236617AbiHQMH5 (ORCPT + 99 others); Wed, 17 Aug 2022 08:07:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239369AbiHQMHl (ORCPT ); Wed, 17 Aug 2022 08:07:41 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6255586B61 for ; Wed, 17 Aug 2022 05:07:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660738024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=uvED31tz28w3zR0MRsRpAF+4dRa0mrzWJzSKyavcZMM=; b=TpOUYd6Nk3jdrpmp41LJCoXXzz5JI0lPMuHGvbxQ6sTqByk9OJe4L9NOMMtf19Q0+199uS Ciig3F9XPyTqcSPzLmtZ7JWm2e9FZLzxXtb6M2x6Wh0/lBFL+9kJTe3rCoN6MHU/C35N6U RVj+J0HOI2K5sEg29SjG2/0UVRStpr0= Received: from mail-yw1-f199.google.com (mail-yw1-f199.google.com [209.85.128.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-512-d0F_4Vc6NMushh-C6ow1ig-1; Wed, 17 Aug 2022 08:07:03 -0400 X-MC-Unique: d0F_4Vc6NMushh-C6ow1ig-1 Received: by mail-yw1-f199.google.com with SMTP id 00721157ae682-31f3959ba41so150462837b3.2 for ; Wed, 17 Aug 2022 05:07:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc; bh=uvED31tz28w3zR0MRsRpAF+4dRa0mrzWJzSKyavcZMM=; b=Aw/Rde93vJpkFdOYl9yuNevOjeoKThcIpDT/L1x04Si1B1cVbe7ERZ2N+fPnUM7wDN ehICGwL/U+HD6XDFoMad7kJ+Ahi6+BRc1UddJxa3seXslMJhgT/GJiLq77WAwjJ+0QdK pWAtOQsmQZD49xv3qUCvK1jYwOq2RnqupZnI4Py4++j45oNUwZGE7+Rs4mz/Pi+rX10y BvJnbExErngxo2lmTYXMeAuECQ+p8x6NJIeJ8md1tGhlkS8oXutfcCDspdgg+xc1o4wA Umz0TE0zdyW6OJHnPyoCtcni6I6bCz3qzGPASQucWRfmVqzyQK4El7CoIuCDO/l6wdST 7pZA== X-Gm-Message-State: ACgBeo23+0u9SDOWqSUijrVKU0+Xx8WTK+co9kDPCLaL6CwwAReLi15Q jyrvoVPxAbWnfjsFEup8Q3wDF6olQam/GMQ+hVC5YGM47DTzT3HD5zOuWUdBFU5k3dhFmqXmvdL FTSRR5wpdEwtbbZfaFnCxXLL/ohjNPbsy4AnxrEbY X-Received: by 2002:a25:e209:0:b0:67c:234a:f08c with SMTP id h9-20020a25e209000000b0067c234af08cmr19261953ybe.19.1660738022542; Wed, 17 Aug 2022 05:07:02 -0700 (PDT) X-Received: by 2002:a25:e209:0:b0:67c:234a:f08c with SMTP id h9-20020a25e209000000b0067c234af08cmr19261926ybe.19.1660738022293; Wed, 17 Aug 2022 05:07:02 -0700 (PDT) MIME-Version: 1.0 References: <2220d403-e443-4e60-b7c3-d149e402c13e@www.fastmail.com> <61e5ccda-a527-4fea-9850-91095ffa91c4@www.fastmail.com> <4995baed-c561-421d-ba3e-3a75d6a738a3@www.fastmail.com> <2b8a38fa-f15f-45e8-8caa-61c5f8cd52de@www.fastmail.com> In-Reply-To: <2b8a38fa-f15f-45e8-8caa-61c5f8cd52de@www.fastmail.com> From: Ming Lei Date: Wed, 17 Aug 2022 20:06:51 +0800 Message-ID: Subject: Re: stalling IO regression since linux 5.12, through 5.18 To: Chris Murphy Cc: Nikolay Borisov , Jens Axboe , Jan Kara , Paolo Valente , Btrfs BTRFS , Linux-RAID , linux-block , linux-kernel , Josef Bacik , Ming Lei Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello Chris, On Tue, Aug 16, 2022 at 11:35 PM Chris Murphy wrote: > > > ... > > I already reported on that: always happens with bfq within an hour or less. Doesn't happen with mq-deadline for ~25+ hours. Does happen with bfq with the above patches removed. Does happen with cgroup.disabled=io set. > > Sounds to me like it's something bfq depends on and is somehow becoming perturbed in a way that mq-deadline does not, and has changed between 5.11 and 5.12. I have no idea what's under bfq that matches this description. > blk-mq debugfs log is usually helpful for io stall issue, care to post the blk-mq debugfs log: (cd /sys/kernel/debug/block/$disk && find . -type f -exec grep -aH . {} \;) Thanks, Ming