Received: by 2002:a05:6358:701b:b0:131:369:b2a3 with SMTP id 27csp3253392rwo; Mon, 24 Jul 2023 08:23:22 -0700 (PDT) X-Google-Smtp-Source: APBJJlGFWSk7vYtVohPx63Re9cHNZZg+PkXsSWAvwKImGv8tbZLnilEhAy9imb9BjFa2RV2brobx X-Received: by 2002:a05:6e02:1646:b0:348:d360:f180 with SMTP id v6-20020a056e02164600b00348d360f180mr357619ilu.0.1690212202189; Mon, 24 Jul 2023 08:23:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690212202; cv=none; d=google.com; s=arc-20160816; b=s2hckVexq4FAn+FvOZFp53Cc4rU0HnHlCH6Zs4UCSEwnu7XBULCYiuQEw9eSFuP/UK Gx8hNVNqzuwfrIXudccEnkORz6iEZI6PmFhqQV6Ki6aRFOdz/TZHBoUVJ8C9tFh7T+62 LBpIyJVgzOxyeGXeceni8KSgTeofNqF9Yst7IK4kC6JtU4C/8NWGsdwseyoINlx7XzFx eHGO9gn5KJHDZ2FFulr7l7ZwU8SSKhrZvtLgqGpZzGOgAYXO4nYXASTS9n4egHL5neN3 tyBsjf6GsI2PapePnKvvnLQ5qFQWoOagh+hAFZONteyns+KQ9TrcLylU0kBhafofGyXo w5ww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:cc:to:subject :message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=FDCS4BThuikrxfZ3pVdXGKzBm4hBT5BkVcG2+9VDxdM=; fh=bWzlJVHhpsw5jQVu//HYQhgUpvgEM1vo0Vz7i1xUlI8=; b=QTEMlXvj+ZO/ZK+loyfwgROOdguLtzy76D2RZDQAje9223x5XCjgjyGR7saOhrq+/U 7Mwlqp83JUYAEh220pH8wIhaeyn9zvNdnWZrA6l6H9f3ZXtaKsN7OlX99XwMJKU5Z8G7 meGpwQ/SYlnsRnJs3+tmi6/3Dwr8lscus+9FHhKNelf9aLEQ44xhowxuaLJkgJisSNxt KXdEGd0ZgREpe+4X0o4ukBkMWbk1aSVVNk71FypLig0qjT7Mo3nae8TMoTmPOW8T+dV8 ujr3WcZpqaT+AVdw7xCOAajLU6vIRR/JgjVX0fLQp+uNsfNOSjv06RcOD0rylOfKVex3 nc0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=HeqltN6Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h19-20020a17090aa89300b002680b23aff7si3626182pjq.161.2023.07.24.08.23.10; Mon, 24 Jul 2023 08:23:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=HeqltN6Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231218AbjGXOjq (ORCPT + 99 others); Mon, 24 Jul 2023 10:39:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230035AbjGXOjp (ORCPT ); Mon, 24 Jul 2023 10:39:45 -0400 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA11E19C for ; Mon, 24 Jul 2023 07:39:42 -0700 (PDT) Received: by mail-ej1-x630.google.com with SMTP id a640c23a62f3a-99b9421aaebso155109966b.2 for ; Mon, 24 Jul 2023 07:39:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1690209580; x=1690814380; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=FDCS4BThuikrxfZ3pVdXGKzBm4hBT5BkVcG2+9VDxdM=; b=HeqltN6YysoNpPmF8jOq4BMenkVF/UoLLGTs2f73eDVL6RZGg995n4rlajiD4OwBuD x0vh4COeuwAnTWyjIuPulyOqktMXVNRXkQvWwZ2l7X/x2xyzf/5z6rmKcfg7LEQUKape g2rrI9p67ze5EjPEeChlbuzjserRkK4xiP6HA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690209580; x=1690814380; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FDCS4BThuikrxfZ3pVdXGKzBm4hBT5BkVcG2+9VDxdM=; b=NRlRTJtP66Kzjt4y2zxQgslx1xFt//MTS/kmJWjoZDB0XOM/058KEmbltaHaxiSyYm CyH9zs84XWrrzQuW+mOA6wxGB97NS+3aYPD7HQuZlep233JnJt1QDEDxJ0r6SSYTrKvh p25/1X53qleIphJlZOgOIhgghkBFRLEAADIcKxcUa49Ib2GFugO8LtQeUs5pdKMemuKO 0ZTCgDWBeJJ3uFflL6g8am9jzIA1tS++yLgd0yx0AlaHLuNuINgRfIxIYUiRMG3K/Syd wqb2WgYcF1fLgWDS0W2sPIdtV6jkcU4emOg7GTZcZAyPfa9/X+KDgDoqGj+Difi4IywD 3mKQ== X-Gm-Message-State: ABy/qLYzwfuXr+qAYyYbi4yg0kdcaFvkGaNYfkFV4kDhcEAqEuiZXyvX o2IWlZ9NW44PtlnZMBaUEmaznMH/zbpoBGuFwMaSBg== X-Received: by 2002:a17:907:771a:b0:993:d617:bdc5 with SMTP id kw26-20020a170907771a00b00993d617bdc5mr9486318ejc.37.1690209579912; Mon, 24 Jul 2023 07:39:39 -0700 (PDT) Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com. [209.85.208.52]) by smtp.gmail.com with ESMTPSA id k17-20020a170906681100b00997d7aa59fasm6911354ejr.14.2023.07.24.07.39.39 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Jul 2023 07:39:39 -0700 (PDT) Received: by mail-ed1-f52.google.com with SMTP id 4fb4d7f45d1cf-51e24210395so15318a12.0 for ; Mon, 24 Jul 2023 07:39:39 -0700 (PDT) X-Received: by 2002:a50:ab56:0:b0:51e:5773:891d with SMTP id t22-20020a50ab56000000b0051e5773891dmr168221edc.4.1690209578716; Mon, 24 Jul 2023 07:39:38 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Doug Anderson Date: Mon, 24 Jul 2023 07:39:26 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: VM Boot Hangs with Commit "Revert "scsi: core: run queue if SCSI device queue isn't ready and queue is idle"" on linux-5.4.y To: Sherry Yang Cc: Harshit Mogalapalli , George Kennedy , "linux-scsi@vger.kernel.org" , "linux-kernel@vger.kernel.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, On Sun, Jul 23, 2023 at 10:28=E2=80=AFPM Sherry Yang wrote: > > Hi Douglas, > > We observed linux-stable v5.4 VM boot hangs, but probably only 1 in thous= ands of boots (less than 10,000 boots). We started 16 VMs on a Bare Metal = with loop reboots, I chose 10,000 boots as a threshold, and bisected it. Af= ter a painful bisection, I found the culprit commit 578c8f09c04b (=E2=80=9C= Revert scsi: core: run queue if SCSI device queue isnt ready and queue is i= dle=E2=80=9D). This commit actually was merged to v5.8 the 1st time. It's a= series of patch set (https://www.spinics.net/lists/linux-block/msg51866.ht= ml). Actually, in the 4-patch series, 2 of them have already been backporte= d to linux-stable v5.4, but not at the same time: > > 1) ab3cee3762e5 (=E2=80=9Cblk-mq: In blk_mq_dispatch_rq_list() no budget = is a reason to kick=E2=80=9D) in tag v5.4.86 > 2) 578c8f09c04b (=E2=80=9CRevert scsi: core: run queue if SCSI device que= ue isnt ready and queue is idle=E2=80=9D) in tag v5.4.235, it=E2=80=99s bac= kported as stable dependency for another commit > > Signed-off-by: Douglas Anderson > Reviewed-by: Ming Lei > Acked-by: Martin K. Petersen > Signed-off-by: Jens Axboe > Stable-dep-of: c31e76bcc379 ("blk-mq: remove stale comment for bl= k_mq_sched_mark_restart_hctx=E2=80=9D) > Signed-off-by: Sasha Levin > > And I tried backporting the other 2 patches to v5.4, the issue is still r= eproducible. > > I tested multiple kernels, the issue is not reproducible within 10,000 bo= ots in the following kernels: > 1) Linux v5.9 > 2) Linux v5.4.249 + revert of 578c8f09c04b (=E2=80=9CRevert scsi: core: r= un queue if SCSI device queue isnt ready and queue is idle=E2=80=9D) > > Not exactly sure how this commit is affecting linux-stable v5.4, but I su= spect some prerequisite commits are missing which lead to boot hangs on lin= ux-stable v5.4 but not on higher releases. Could you take a look at this is= sue and share your insight? Ugh, I spent many days pouring over the code and digging through debug traces in order to write those patches. I don't think I'd be able to give any concrete advice without spending many days and being able to reproduce multiple times with traces since pretty much any knowledge I learned during the course of developing those patches has decayed over the last several years. :( I don't happen to know any dependencies offhand... That being said, it seems like: 1. Backporting the revert (the 4th patch in the series) without all the other patches in the series feels wrong. In the text of the revert I explicitly refer to the other patches in the series as prerequisites. I guess you said you tried backporting the other two patches and they didn't help, though? That's no good. :( 2. I don't think the revert is actually important to backport to stable. While the first 3 patches were important to fix the problems I was seeing, the revert was just a cleanup. If the revert is causing problems in 5.4.x then I'd suggest removing it from 5.4.x Does that make sense? So ideally you'd submit 3 patches to the stable kerne= l: a) Revert the revert b) Pick ("blk-mq: Add blk_mq_delay_run_hw_queues() API call") c) Pick ("blk-mq: Rerun dispatching in the case of budget contention") FWIW, we seem to have all 4 patches in the ChromeOS 5.4 kernel tree. They all landed together plus 1 prerequisite. * https://crrev.com/c/2155423 - FROMGIT: Revert "scsi: core: run queue if SCSI device queue isn't ready and queue is idle" * https://crrev.com/c/2133069 - FROMGIT: blk-mq: Rerun dispatching in the case of budget contention * https://crrev.com/c/2155422 - FROMGIT: blk-mq: Add blk_mq_delay_run_hw_queues() API call * https://crrev.com/c/2125232 - FROMGIT: blk-mq: In blk_mq_dispatch_rq_list() "no budget" is a reason to kick * https://crrev.com/c/2155421 - UPSTREAM: blk-mq: Put driver tag in blk_mq_dispatch_rq_list() when no budget When I saw the prerequisite in there I was hopeful that it was the one you needed, but it looks like that's already in 5.4 stable so (I presume) you've already been testing with it... -Doug