Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Feedback-ID: i13394474:Fastmail
Message-ID: <017845ae-fbae-70f6-5f9e-29aff2742b8c@dustymabe.com>
Date:   Fri, 26 Aug 2022 12:15:22 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.8.0
Content-Language: en-US
To:     Jens Axboe <axboe@kernel.dk>, linux-block@vger.kernel.org,
        linux-kernel@vger.kernel.org
Cc:     hch@lst.de
From:   Dusty Mabe <dusty@dustymabe.com>
Subject: regression caused by block: freeze the queue earlier in del_gendisk
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hey All,

I think I've found a regression introduced by:

a09b314 o block: freeze the queue earlier in del_gendisk

In Fedora CoreOS we have tests that set up RAID1 on the /boot/ and /root/ partitions
and then subsequently removes one of the disks to simulate a failure. Sometime recently
this test started timing out occasionally. Looking a bit closer it appears instances are
getting stuck during reboot with a bunch of looping messages:

```
[   17.978854] block device autoloading is deprecated and will be removed.
[   17.982555] block device autoloading is deprecated and will be removed.
[   17.985537] block device autoloading is deprecated and will be removed.
[   17.987546] block device autoloading is deprecated and will be removed.
[   17.989540] block device autoloading is deprecated and will be removed.
[   17.991547] block device autoloading is deprecated and will be removed.
[   17.993555] block device autoloading is deprecated and will be removed.
[   17.995539] block device autoloading is deprecated and will be removed.
[   17.997577] block device autoloading is deprecated and will be removed.
[   17.999544] block device autoloading is deprecated and will be removed.
[   22.979465] blkdev_get_no_open: 1666 callbacks suppressed
...
...
...
[  618.221270] blkdev_get_no_open: 1664 callbacks suppressed
[  618.221273] block device autoloading is deprecated and will be removed.
[  618.224274] block device autoloading is deprecated and will be removed.
[  618.227267] block device autoloading is deprecated and will be removed.
[  618.229274] block device autoloading is deprecated and will be removed.
[  618.231277] block device autoloading is deprecated and will be removed.
[  618.233277] block device autoloading is deprecated and will be removed.
[  618.235282] block device autoloading is deprecated and will be removed.
[  618.237370] block device autoloading is deprecated and will be removed.
[  618.239356] block device autoloading is deprecated and will be removed.
[  618.241290] block device autoloading is deprecated and will be removed.
```

Using the Fedora kernels I narrowed it down to being introduced between 
`kernel-5.19.0-0.rc3.27.fc37` (good) and `kernel-5.19.0-0.rc4.33.fc37` (bad).

I then did a bisect and found:

```
$ git bisect bad
a09b314005f3a0956ebf56e01b3b80339df577cc is the first bad commit
commit a09b314005f3a0956ebf56e01b3b80339df577cc
Author: Christoph Hellwig <hch@lst.de>
Date:   Tue Jun 14 09:48:27 2022 +0200

    block: freeze the queue earlier in del_gendisk
    
    Freeze the queue earlier in del_gendisk so that the state does not
    change while we remove debugfs and sysfs files.
    
    Ming mentioned that being able to observer request in debugfs might
    be useful while the queue is being frozen in del_gendisk, which is
    made possible by this change.
    
    Signed-off-by: Christoph Hellwig <hch@lst.de>
    Link: https://lore.kernel.org/r/20220614074827.458955-5-hch@lst.de
    Signed-off-by: Jens Axboe <axboe@kernel.dk>

 block/genhd.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
```

Reverting this commit on top of latest git master (4c612826b) gave me successful results.

Any ideas on what could be amiss here? Luckily the patch is tiny so hopefully it might
be obvious.

More details (including logs) in the following locations:

- https://bugzilla.redhat.com/show_bug.cgi?id=2121791
- https://github.com/coreos/fedora-coreos-tracker/issues/1282


Thanks!
Dusty