Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <d39e9149-fcb6-1f7c-4c19-234e74f286f8@kernel.dk>
Date:   Tue, 20 Sep 2022 08:05:06 -0600
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:102.0) Gecko/20100101
 Thunderbird/102.2.2
Subject: Re: regression caused by block: freeze the queue earlier in
 del_gendisk
To:     Thorsten Leemhuis <regressions@leemhuis.info>,
        Dusty Mabe <dusty@dustymabe.com>,
        Ming Lei <ming.lei@redhat.com>, Christoph Hellwig <hch@lst.de>
Cc:     linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-raid@vger.kernel.org,
        "regressions@lists.linux.dev" <regressions@lists.linux.dev>
References: <017845ae-fbae-70f6-5f9e-29aff2742b8c@dustymabe.com>
 <YxBZ4BBjxvAkvI2A@T590> <20220907073324.GB23826@lst.de>
 <Yxr4SD4d0rZ9TZik@T590> <20220912071618.GA4971@lst.de>
 <Yx/jLTknQm9VeHi4@T590> <95cbd47d-46ed-850e-7d4f-851b35d03069@dustymabe.com>
 <f2c28043-59e6-0aee-b8bf-df38525ee899@leemhuis.info>
Content-Language: en-US
From:   Jens Axboe <axboe@kernel.dk>
In-Reply-To: <f2c28043-59e6-0aee-b8bf-df38525ee899@leemhuis.info>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 9/20/22 3:11 AM, Thorsten Leemhuis wrote:
> Hi, this is your Linux kernel regression tracker.
> 
> On 13.09.22 04:36, Dusty Mabe wrote:
>> On 9/12/22 21:55, Ming Lei wrote:
>>> On Mon, Sep 12, 2022 at 09:16:18AM +0200, Christoph Hellwig wrote:
>>>> On Fri, Sep 09, 2022 at 04:24:40PM +0800, Ming Lei wrote:
>>>>> On Wed, Sep 07, 2022 at 09:33:24AM +0200, Christoph Hellwig wrote:
>>>>>> On Thu, Sep 01, 2022 at 03:06:08PM +0800, Ming Lei wrote:
>>>>>>> It is a bit hard to associate the above commit with reported issue.
>>>>>>
>>>>>> So the messages clearly are about something trying to open a device
>>>>>> that went away at the block layer, but somehow does not get removed
>>>>>> in time by udev (which seems to be a userspace bug in CoreOS).  But
>>>>>> even with that we really should not hang.
>>>>>
>>>>> Xiao Ni provides one script[1] which can reproduce the issue more or less.
>>>>
>>>> I've run the reproduced 10000 times on current mainline, and while
>>>> it prints one of the autoloading messages per run, I've not actually
>>>> seen any kind of hang.
>>>
>>> I can't reproduce the hang too.
>>
>> I obviously can reproduce the issue with the test in our Fedora CoreOS
>> test suite. It's part of a framework (i.e. it's not simple some script
>> you can run) but it is very reproducible so one can add some instrumentation
>> to the kernel and feed it through a build/test cycle to see different
>> results or logs.
>>
>> I'm willing to share this with other people (maybe a screen share or
>> some written down instructions) if anyone would be interested.
> 
> This thread looked stalled, or was there any progress in the past week?
> If not: Fedora apparently removed the patch in their kernels a while
> ago, as quite a few users where hitting it. What is preventing us from
> doing the same in mainline and 5.19.y until the issue can be resolved?
> The description of a09b314005f3 ("block: freeze the queue earlier in
> del_gendisk") doesn't sound like the change does something crucial that
> can't wait a bit. I might be totally wrong with that, but I think it's
> my duty to ask that question at this point.

Christoph and I discussed this one last week, and he has a plan to try
a flag approach. Christoph, did you get a chance to bang that out? Would
be nice to get this one wrapped up.

-- 
Jens Axboe