Hi Linus. I'm down to four regressions that I and regzbot are aware of
and worth mentioning. All of them are worked on by developers. See below
for details.
What currently worries me more are two regressions from the 6.2 cycle
still not fixed.
Wake-on-lan (WOL) apparently is broken for a huge number of users since
6.2 was released[1]. This is known for 8 weeks now and about 4 weeks ago
it was bisected to commit 5c62d5aab87 ("ACPICA: Events: Support fixed
PCIe wake event") we immediately could have reverted. The developer that
looked into this was even willing to do the revert late March, but then
got discouraged by a maintainer [2]. But well, a fix was apparently[3]
finally posted for review last week (to the acpica-devel list); with a
bit of luck your might get it next week. Still a bit sad that 6.2 is
broken for so long now, as Greg wants to see it fixed in mainline first.
[1] https://bugzilla.kernel.org/show_bug.cgi?id=217069
[2] https://bugzilla.kernel.org/show_bug.cgi?id=217069#c50
[3]
https://lore.kernel.org/all/[email protected]/
I have see similar situations a few times already. I sometimes wonder if
I should post reverts for review in cases like this (say a few days or a
week after bisection, is no fix is in sight) to enforce the issue and at
least trigger a discussion about the revert -- and depending on its
outcome ask you a few days later to consider picking it up. But I'm not
sure if that's really a wise move.
Another issue from the 6.2 days still not fixed are a huge number of
DISCARD request on NVME devices with Btrfs caused by 63a7cb13071
("btrfs: auto enable discard=async when possible") [1, 2]. This is known
for 7 weeks and likely results in a performance regression for some
users and might also reduce the life time of some devices. One user even
reported his laptop now uses ~10W more due to the constant drive
activity [3]. After some back and forth the Btrfs developers agreed on
working towards a fix recently [4]; but I doubt we'll see it this week.
I have no idea if a revert would be possible here. But I thought you
might want to be aware of this.
[1] https://lore.kernel.org/linux-btrfs/Y%2F%2Bn1wS%2F4XAH7X1p@nz/
[2]
https://lore.kernel.org/linux-btrfs/CAHmG9huwQcQXvy3HS0OP9bKFxwUa3aQj9MXZCr74emn0U+efqQ@mail.gmail.com/
[3]
https://lore.kernel.org/linux-btrfs/[email protected]/
[4] https://lore.kernel.org/linux-btrfs/20230404193909.GC344341@zen/
HTH, Ciao, Thorsten
P.S.: Sorry for bringing up a regression last week that was just a
kernel warning; for some reason I was under the wrong impression that
video apps were crashing; I guess my mind confused this with another
regression.
---
Hi, this is regzbot, the Linux kernel regression tracking bot.
Currently I'm aware of 4 regressions in linux-mainline. Find the
current status below and the latest on the web:
https://linux-regtracking.leemhuis.info/regzbot/mainline/
Bye bye, hope to see you soon for the next report.
Regzbot (on behalf of Thorsten Leemhuis)
======================================================
current cycle (v6.2.. aka v6.3-rc), culprit identified
======================================================
net: mlx5: InfiniBand devices were no longer present
----------------------------------------------------
https://linux-regtracking.leemhuis.info/regzbot/regression/CAHC9VhQ7A4+msL38WpbOMYjAqLp0EtOjeLh4Dc6SQtD6OUvCQg@mail.gmail.com/
https://lore.kernel.org/netdev/CAHC9VhQ7A4%[email protected]/
By Paul Moore; 18 days ago; 18 activities, latest 1 days ago.
Introduced in fe998a3c77b9 (v6.3-rc1)
Recent activities from: Jakub Kicinski (6), Saeed Mahameed (5), Paul
Moore (3), Leon Romanovsky (1)
Noteworthy links:
* [PATCH net] Revert "net/mlx5: Enable management PF initialization"
https://lore.kernel.org/netdev/[email protected]/
2 days ago, by Jakub Kicinski; thread monitored.
sched: PostgreSQL performance regression introduced by mm_cid
-------------------------------------------------------------
https://linux-regtracking.leemhuis.info/regzbot/regression/20230327080502.GA570847@ziqianlu-desk2/
https://lore.kernel.org/lkml/20230327080502.GA570847@ziqianlu-desk2/
By Aaron Lu; 20 days ago; 75 activities, latest 2 days ago.
Introduced in af7f588d8f73 (v6.3-rc1)
Recent activities from: Mathieu Desnoyers (15), Aaron Lu (13), Peter
Zijlstra (11), [email protected] (1)
5 patch postings are associated with this regression, the latest is this:
* Re: [RFC PATCH v4] sched: Fix performance regression introduced by mm_cid
https://lore.kernel.org/lkml/20230413131353.GA214119@ziqianlu-desk2/
3 days ago, by Aaron Lu
Noteworthy links:
* [RFC PATCH] sched: Introduce per-mm/cpu concurrency id state
https://lore.kernel.org/lkml/[email protected]/
16 days ago, by Mathieu Desnoyers; thread monitored.
* [RFC PATCH] sched: Fix performance regression introduced by mm_cid
https://lore.kernel.org/lkml/[email protected]/
12 days ago, by Mathieu Desnoyers; thread monitored.
* [RFC PATCH] sched: Fix performance regression introduced by mm_cid (v2)
https://lore.kernel.org/lkml/[email protected]/
11 days ago, by Mathieu Desnoyers; thread monitored.
* [RFC PATCH v3] sched: Fix performance regression introduced by mm_cid
https://lore.kernel.org/lkml/[email protected]/
11 days ago, by Mathieu Desnoyers; thread monitored.
* [RFC PATCH v4] sched: Fix performance regression introduced by mm_cid
https://lore.kernel.org/lkml/[email protected]/
6 days ago, by Mathieu Desnoyers; thread monitored.
* [RFC PATCH] sched: Rate limit migrations
https://lore.kernel.org/lkml/[email protected]/
4 days ago, by Mathieu Desnoyers; thread monitored.
* [RFC PATCH v5] sched: Fix performance regression introduced by mm_cid
https://lore.kernel.org/lkml/[email protected]/
3 days ago, by Mathieu Desnoyers; thread monitored.
* [RFC PATCH v6] sched: Fix performance regression introduced by mm_cid
https://lore.kernel.org/lkml/[email protected]/
2 days ago, by Mathieu Desnoyers; thread monitored.
firmware/sysfb: wrong mode and display garbled on 16-year old i686 laptop
-------------------------------------------------------------------------
https://linux-regtracking.leemhuis.info/regzbot/regression/[email protected]/
https://lore.kernel.org/dri-devel/[email protected]/
By [email protected]; 10 days ago; 24 activities, latest 2 days ago.
Introduced in f35cd3fa7729 (v6.3-rc1)
Recent activities from: Javier Martinez Canillas (12), Pierre
Asselin (9), [email protected] (1)
4 patch postings are associated with this regression, the latest is this:
* Re: [PATCH] firmware/sysfb: Fix wrong stride when bits-per-pixel is calculated
https://lore.kernel.org/lkml/[email protected]/
3 days ago, by Javier Martinez Canillas
Noteworthy links:
* [PATCH] firmware/sysfb: Fix wrong stride when bits-per-pixel is calculated
https://lore.kernel.org/lkml/[email protected]/
4 days ago, by Javier Martinez Canillas; thread monitored.
pci: / net: igb: hangs during boot on PowerEdge R620
----------------------------------------------------
https://linux-regtracking.leemhuis.info/regzbot/regression/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
By Donald Hunter; 16 days ago; 9 activities, latest 4 days ago.
Introduced in 6fffbc7ae137 (v6.3-rc1)
Recent activities from: Donald Hunter (2), Andy Shevchenko (1), Rob
Herring (1), Bjorn Helgaas (1)
=============
End of report
=============
All regressions marked '[ *NEW* ]' were added since the previous report,
which can be found here:
https://lore.kernel.org/r/[email protected]
Thanks for your attention, have a nice day!
Regzbot, your hard working Linux kernel regression tracking robot
P.S.: Wanna know more about regzbot or how to use it to track regressions
for your subsystem? Then check out the getting started guide or the
reference documentation:
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md
The short version: if you see a regression report you want to see
tracked, just send a reply to the report where you Cc
[email protected] with a line like this:
#regzbot introduced: v5.13..v5.14-rc1
If you want to fix a tracked regression, just do what is expected
anyway: add a 'Link:' tag with the url to the report, e.g.:
Link: https://lore.kernel.org/all/[email protected]/
On Sun, Apr 16, 2023 at 10:59 AM Regzbot (on behalf of Thorsten
Leemhuis) <[email protected]> wrote:
>
> Wake-on-lan (WOL) apparently is broken for a huge number of users since
> 6.2 was released[1]. This is known for 8 weeks now and about 4 weeks ago
> it was bisected to commit 5c62d5aab87 ("ACPICA: Events: Support fixed
> PCIe wake event") we immediately could have reverted. The developer that
> looked into this was even willing to do the revert late March, but then
> got discouraged by a maintainer [2]. But well, a fix was apparently[3]
> finally posted for review last week (to the acpica-devel list); with a
> bit of luck your might get it next week. Still a bit sad that 6.2 is
> broken for so long now, as Greg wants to see it fixed in mainline first.
>
> [1] https://bugzilla.kernel.org/show_bug.cgi?id=217069
> [2] https://bugzilla.kernel.org/show_bug.cgi?id=217069#c50
> [3] https://lore.kernel.org/all/[email protected]/
I find that bugzilla discussion very confusing, it's not clear what
the status of the patch actually is.
And the sane lkml thread just says "the patch is under review" without
actually saying *where* said patch is, or where the review is.
It sounds like it got perhaps into some internal ACPCICA queue? None
of those links are very clear on any of this.
Rafael?
> Another issue from the 6.2 days still not fixed are a huge number of
> DISCARD request on NVME devices with Btrfs caused by 63a7cb13071
> ("btrfs: auto enable discard=async when possible") [1, 2].
Yeah, automatic discard is a broken idea, and that should just be reverted.
The number of devices that have *horrible* problems with discard is
too high for any kind of "do discard by default" logic.
David, please don't do that. Just because discard happens to work well
on some disk, *you* care about, does not in any way mean that "do
discard by default" is sane.
Discard needs to be opt-in. Not out-opt.
Linus
On 16.04.23 22:48, Linus Torvalds wrote:
> On Sun, Apr 16, 2023 at 10:59 AM Regzbot (on behalf of Thorsten
> Leemhuis) <[email protected]> wrote:
>>
>> Wake-on-lan (WOL) apparently is broken for a huge number of users since
>> 6.2 was released[1]. This is known for 8 weeks now and about 4 weeks ago
>> it was bisected to commit 5c62d5aab87 ("ACPICA: Events: Support fixed
>> PCIe wake event") we immediately could have reverted. The developer that
>> looked into this was even willing to do the revert late March, but then
>> got discouraged by a maintainer [2]. But well, a fix was apparently[3]
>> finally posted for review last week (to the acpica-devel list); with a
>> bit of luck your might get it next week. Still a bit sad that 6.2 is
>> broken for so long now, as Greg wants to see it fixed in mainline first.
>>
>> [1] https://bugzilla.kernel.org/show_bug.cgi?id=217069
>> [2] https://bugzilla.kernel.org/show_bug.cgi?id=217069#c50
>> [3] https://lore.kernel.org/all/[email protected]/
>
> I find that bugzilla discussion very confusing, it's not clear what
> the status of the patch actually is.
>
> And the sane lkml thread just says "the patch is under review" without
> actually saying *where* said patch is, or where the review is.
>
> It sounds like it got perhaps into some internal ACPCICA queue? None
> of those links are very clear on any of this.
>
> Rafael?
FWIW, yeah, had the same problems and already had asked for a link to
the patch submission. A few hours ago Jianmin Lv replied and pointed me
here: https://github.com/acpica/acpica/pull/866
/me meanwhile wonders if acpica-devel was abandoned or if its list
archive (https://lists.linuxfoundation.org/pipermail/acpica-devel/ ) is
just broken.
>> Another issue from the 6.2 days still not fixed are a huge number of
>> DISCARD request on NVME devices with Btrfs caused by 63a7cb13071
>> ("btrfs: auto enable discard=async when possible") [1, 2].
>
> Yeah, automatic discard is a broken idea, and that should just be reverted.
>
> The number of devices that have *horrible* problems with discard is
> too high for any kind of "do discard by default" logic.
>
> David, please don't do that. Just because discard happens to work well
> on some disk, *you* care about, does not in any way mean that "do
> discard by default" is sane.
>
> Discard needs to be opt-in. Not out-opt.
Not my area of expertise, but is this maybe something where we can
automatically enable async discard by default if some easily detectable
conditions are met? E.g. if the device supports PCIe 12.0, NVMe 9.1, or
was produced in 2030 and later (note, I used bogus numbers here on purpose).
Ciao, Thorsten
On Sun, Apr 16, 2023 at 10:49 PM Linus Torvalds
<[email protected]> wrote:
>
> On Sun, Apr 16, 2023 at 10:59 AM Regzbot (on behalf of Thorsten
> Leemhuis) <[email protected]> wrote:
> >
> > Wake-on-lan (WOL) apparently is broken for a huge number of users since
> > 6.2 was released[1]. This is known for 8 weeks now and about 4 weeks ago
> > it was bisected to commit 5c62d5aab87 ("ACPICA: Events: Support fixed
> > PCIe wake event") we immediately could have reverted. The developer that
> > looked into this was even willing to do the revert late March, but then
> > got discouraged by a maintainer [2]. But well, a fix was apparently[3]
> > finally posted for review last week (to the acpica-devel list); with a
> > bit of luck your might get it next week. Still a bit sad that 6.2 is
> > broken for so long now, as Greg wants to see it fixed in mainline first.
> >
> > [1] https://bugzilla.kernel.org/show_bug.cgi?id=217069
> > [2] https://bugzilla.kernel.org/show_bug.cgi?id=217069#c50
> > [3] https://lore.kernel.org/all/[email protected]/
>
> I find that bugzilla discussion very confusing, it's not clear what
> the status of the patch actually is.
>
> And the sane lkml thread just says "the patch is under review" without
> actually saying *where* said patch is, or where the review is.
>
> It sounds like it got perhaps into some internal ACPCICA queue? None
> of those links are very clear on any of this.
>
> Rafael?
There is a pull request for ACPICA that corresponds to this (IIUC),
https://github.com/acpica/acpica/pull/866
On Sun, Apr 16, 2023 at 01:48:52PM -0700, Linus Torvalds wrote:
> On Sun, Apr 16, 2023 at 10:59 AM Regzbot (on behalf of Thorsten
> Leemhuis) <[email protected]> wrote:
> > Another issue from the 6.2 days still not fixed are a huge number of
> > DISCARD request on NVME devices with Btrfs caused by 63a7cb13071
> > ("btrfs: auto enable discard=async when possible") [1, 2].
>
> Yeah, automatic discard is a broken idea, and that should just be reverted.
>
> The number of devices that have *horrible* problems with discard is
> too high for any kind of "do discard by default" logic.
>
> David, please don't do that. Just because discard happens to work well
> on some disk, *you* care about, does not in any way mean that "do
> discard by default" is sane.
>
> Discard needs to be opt-in. Not out-opt.
What we've enabled is the async mode that waits and batches the ranges
to trim so it's in big chunks and it does not hammer the device with
short discard requests among other IO. The problems reported is due to
the throttling logic that was too slow to keep up after large freed
data. We have fixes for that.
There's also in-memory cache of already trimmed ranges since last mount
so even running discard repeatedly (either fstrim or as mount option)
will not do extra IO. We try hard not to provoke the firmware bugs.
The request to make it default came from Fedora guys, we're not pushing
defaults just because it works on somebody's newest hardware. Feature
introduced in 5.6, enabled in 6.2 provides time to let people test it
and report bugs. Of course we can't cover all different types of
hardware so changing defaults for everyone comes with a risk. I've
scanned IRC logs for feedback related to the discard=async, the
community there tends to be from the more technical users that do try
new things and report back. This is usually a good sample.
- https://pagure.io/fedora-btrfs/project/issue/6
- https://lore.kernel.org/linux-btrfs/CAEg-Je_b1YtdsCR0zS5XZ_SbvJgN70ezwvRwLiCZgDGLbeMB=w@mail.gmail.com/t/#u
Opt-in yes ideally with information available at least to end users
about which devices are safe and which to avoid. This is sadly
nonexistent, we get only broad claims that "don't do that on old
device", vaguely mentioning some vendors and not on record for legal
reasons. There are posts on some forums about bad vendor/model or if
it's really bad then it hits the news sites as well.
- https://www.truenas.com/community/threads/ssd-install-failures-possible-trim-issue.56162/
> The number of devices that have horrible problems with discard is too
> high for any kind of “do discard by default” logic.
As you state it, I would be concerned about enabling discard on any
device and not even the widely configured weekly fstrim jobs. If the
number of the affected devices is too high, please show me where is the
evidence for that. I've been searching web for that now and back then,
the mailing list archives, the in-tree quirk tables. The MMC has a
specific quirk MMC_QUIRK_TRIM_BROKEN, some SATA devices used to require
ncq=1 set which in turn made it to the quirk tables, NVMe has an
impressive list of quirks but not related to discard.
I'm genuinely interested, this can only improve the shared knowledge. I
hope we won't end up reverting it just "because Linus said so".
On Tue, Apr 18, 2023 at 11:20 AM David Sterba <[email protected]> wrote:
>
> There's also in-memory cache of already trimmed ranges since last mount
> so even running discard repeatedly (either fstrim or as mount option)
> will not do extra IO. We try hard not to provoke the firmware bugs.
So we've had devices that claim to support discard, and then simply don't.
I have dim memories of people reporting IO simply stopping working
after a discard when it confused the GC logic too much.
And yes, those dim memories are from many years ago when SSD's were
new and fancy, and we had all kinds of crazy stuff going on, including
serious big SSD manufacturers that came to the kernel summit and said
that we need to do IO in 64kB aligned hunks, because doing GC was too
hard.
Those people have now thankfully gone off and done shingled drives
instead and we can mostly ignore them (although I do note that btrfs
seems to be gulping down the shingled drive koolaid too), but I'm
afraid that some of that incompetence still exists in the form of old
drives.
And some of it isn't even that old. See commit 07d2872bf4c8 ("mmc:
core: Add SD card quirk for broken discard") which is from late last
year. I'm not sure what the failure case there was (apart from the
"mk2fs failed", which I _assume_ was mkfs or mke2fs).
The real problem cases tend to be things like random USB memory sticks
etc. I think the Sandisk MMC case is not that different. A lot of odd
small embedded flash controllers that have never been tested except
under Windows or in cameras or whatever.
So discard tends to have two classes of problems
(a) performance problems due to being non-queued, or simply because
the flash controller is small and latency can be absolutely *huge*
when it processes trims
(b) the "it doesn't work at all" problem
and it's really that "it doesn't work" case I worry about.
We have quite a few trim-related quirks. Do this:
git grep HORKAGE.*TRIM
to see just the libata cases. Yes, some of those are "the queued
version doesn't work". Others are just "it's not zero after trim".
Whatever. But some of them are literally "do not use trim at all".
See commit cda57b1b05cf ("libata: force disable trim for SuperSSpeed
S238"), and tell me that the sentence
"This device loses blocks, often the partition table area, on trim"
doesn't worry you? Ok, so that's from 2015, so "old drives only".
Or how about c8ea23d5fa59 ("ata: libata-core: Disable TRIM on M88V29")
from last year:
"While it also advertises TRIM support, I/O errors are reported
when the discard mount option fstrim is used. TRIM also fails
when disabling NCQ and not just as an NCQ command"
Again, that's libata - odd crazy hardware. But it's exactly the odd
crazy hardware that worries me. When the failure mode isn't "it's
slow", but "it ATE MY WHOLE DISK", that's a scary scary problem.
Hmm?
I dunno. Maybe you have reason to believe that all of these cases have
been fixed, or that some of these were caused by kernel bugs because
we did things wrong, and those have been fixed.
But the failure modes just makes me worry. From your email, it *seems*
like you think that the failures were primarily performance-related.
Linus
Hi Linus and David,
I'm the guy that sort of kickstarted this whole thing a year ago.
From my perspective in Fedora-land, we've been running automatic
weekly fstrim on every Fedora system for three years now[1] and
have not received any complaints about SSDs pushing daises from
that.
When we started discussing btrfs discard=async within Fedora
two years ago[2], I started soliciting feedback and information
from the Btrfs developers I was regularly working with at the time.
Last year, I had a face-to-face with Chris Mason and we discussed
the idea in depth and decided to go for this, based on both Fedora's
data with consumer disks and Facebook's data with their datacenters.
The only real surprise we had was the so-called "discard storm",
which Boris Burkov made adjustments to resolve a couple weeks ago[3].
With all that context in mind, I'm not sure we really should be panicking
about having async discard enabled, since it's the same operation
that the fstrim timer was doing before, just queued by btrfs itself instead.
So personally, I would prefer *not* to revert the new default.
Thanks in advance and best regards,
Neal
[1]: https://fedoraproject.org/wiki/Changes/EnableFSTrimTimer
[2]: https://pagure.io/fedora-btrfs/project/issue/6
[3]: https://lore.kernel.org/linux-btrfs/[email protected]/T/#t
--
真実はいつも一つ！/ Always, there's only one truth!
On Tue, Apr 18, 2023 at 2:33 PM Neal Gompa <[email protected]> wrote:
>
> From my perspective in Fedora-land, we've been running automatic
> weekly fstrim on every Fedora system for three years now[1] and
> have not received any complaints about SSDs pushing daises from
> that.
Ahh, good. So we do have a lot more test coverage than I expected.
Then I withdraw my concerns. Hopefully it all works now...
Linus
On 18.04.23 23:32, Neal Gompa wrote:
>
> I'm the guy that sort of kickstarted this whole thing a year ago.
>>From my perspective in Fedora-land, we've been running automatic
> weekly fstrim on every Fedora system for three years now[1] and
> have not received any complaints about SSDs pushing daises from
> that.
>
> When we started discussing btrfs discard=async within Fedora
> two years ago[2], I started soliciting feedback and information
> from the Btrfs developers I was regularly working with at the time.
>
> Last year, I had a face-to-face with Chris Mason and we discussed
> the idea in depth and decided to go for this, based on both Fedora's
> data with consumer disks and Facebook's data with their datacenters.
>
> The only real surprise we had was the so-called "discard storm",
> which Boris Burkov made adjustments to resolve a couple weeks ago[3].
> [...]
> [3]: https://lore.kernel.org/linux-btrfs/[email protected]/T/#t
Wait, what? Argh. Sorry, if I had seen that patch, I wouldn't have
brought this up in my report at all. I missed it, as I wasn't CCed; and
regzbot missed it, because the patch uses a odd format for the lore link
(but not totally uncommon, will change regzbot to ensure that doesn't
happen again).
Ciao, Thorsten
P.S.: /me meanwhile yet again wonders if we should tell people to add a
"CC: <[email protected]>" on patches fixing regressions. Then
in this case I would have become aware of the patch. And it makes it
obvious for anybody handling patches that the patch is fixing a
regression. But whatever, might not be worth it.
On Tue, Apr 18, 2023 at 12:11:51PM -0700, Linus Torvalds wrote:
> On Tue, Apr 18, 2023 at 11:20 AM David Sterba <[email protected]> wrote:
> >
> > There's also in-memory cache of already trimmed ranges since last mount
> > so even running discard repeatedly (either fstrim or as mount option)
> > will not do extra IO. We try hard not to provoke the firmware bugs.
[...]
> Again, that's libata - odd crazy hardware. But it's exactly the odd
> crazy hardware that worries me. When the failure mode isn't "it's
> slow", but "it ATE MY WHOLE DISK", that's a scary scary problem.
>
> Hmm?
>
> I dunno. Maybe you have reason to believe that all of these cases have
> been fixed, or that some of these were caused by kernel bugs because
> we did things wrong, and those have been fixed.
>
> But the failure modes just makes me worry. From your email, it *seems*
> like you think that the failures were primarily performance-related.
No, the main concern is if discard works without destroying data,
performance is more like an optimization. I too worry about buggy
hardware, we have a page just about that
https://btrfs.readthedocs.io/en/latest/Hardware.html .
I've taken notes from your reply and will enhance the page, or page
about discard in particular. The info about device quirks/horkage could
be linked too or I thought about generating a static page from the
per-bus tables so it's on one page.
I'll send pull request with fixes for the regression.
On Wed, Apr 19, 2023 at 07:03:31AM +0200, Linux regression tracking (Thorsten Leemhuis) wrote:
> On 18.04.23 23:32, Neal Gompa wrote:
> >
> > I'm the guy that sort of kickstarted this whole thing a year ago.
> >>From my perspective in Fedora-land, we've been running automatic
> > weekly fstrim on every Fedora system for three years now[1] and
> > have not received any complaints about SSDs pushing daises from
> > that.
> >
> > When we started discussing btrfs discard=async within Fedora
> > two years ago[2], I started soliciting feedback and information
> > from the Btrfs developers I was regularly working with at the time.
> >
> > Last year, I had a face-to-face with Chris Mason and we discussed
> > the idea in depth and decided to go for this, based on both Fedora's
> > data with consumer disks and Facebook's data with their datacenters.
> >
> > The only real surprise we had was the so-called "discard storm",
> > which Boris Burkov made adjustments to resolve a couple weeks ago[3].
> > [...]
> > [3]: https://lore.kernel.org/linux-btrfs/[email protected]/T/#t
>
> Wait, what? Argh. Sorry, if I had seen that patch, I wouldn't have
> brought this up in my report at all. I missed it, as I wasn't CCed; and
> regzbot missed it, because the patch uses a odd format for the lore link
> (but not totally uncommon, will change regzbot to ensure that doesn't
> happen again).
I'd need pay more attention when the regression tracking process is
involved in case there are more patch versions floating around. People
usually don't "CC enough" so that you have the regzbot in place helps
to track the state.
> Ciao, Thorsten
>
> P.S.: /me meanwhile yet again wonders if we should tell people to add a
> "CC: <[email protected]>" on patches fixing regressions. Then
> in this case I would have become aware of the patch. And it makes it
> obvious for anybody handling patches that the patch is fixing a
> regression. But whatever, might not be worth it.
I'm not sure if it would fit how regzbot workflow works, but syzbot
provides links with the reports and then changes the state when the
patch is committed containing the links. I don't see anything similar in
the process/handling-regression document. If the "Link: <report>" is
sufficient then it should work already but there's no guarantee that the
submitted patches would contain that. I add links to the committed
versions but then you'd need to scan at least linux-next. In any case
with the regzbot it's fixable.
On Thu, Apr 20, 2023 at 12:02 PM David Sterba <[email protected]> wrote:
>
> No, the main concern is if discard works without destroying data,
I actually ended up googling it, and I'm a lot less worried now,
because at least according to my limited searching it seems like
Windows enables trim by default at least since Win10.
Of course, I also found some conflicting statements claiming that USB
drives aren't considered SSDs, so who knows. But it does seem like
probably new hardware ends up getting tested for trim, and the bad
situation is likely really limited to only the bad old days.
So together with Fedora having apparently enabled timed auto-trimming
for some time, I guess I will just have to believe that the world has
turned into a less dark and scary place than I thought it was.
Knock wood.
Linus
On 20.04.23 21:21, David Sterba wrote:
> On Wed, Apr 19, 2023 at 07:03:31AM +0200, Linux regression tracking (Thorsten Leemhuis) wrote:
>> On 18.04.23 23:32, Neal Gompa wrote:
>>>
>>> I'm the guy that sort of kickstarted this whole thing a year ago.
>>> >From my perspective in Fedora-land, we've been running automatic
>>> weekly fstrim on every Fedora system for three years now[1] and
>>> have not received any complaints about SSDs pushing daises from
>>> that.
>>>
>>> When we started discussing btrfs discard=async within Fedora
>>> two years ago[2], I started soliciting feedback and information
>>> from the Btrfs developers I was regularly working with at the time.
>>>
>>> Last year, I had a face-to-face with Chris Mason and we discussed
>>> the idea in depth and decided to go for this, based on both Fedora's
>>> data with consumer disks and Facebook's data with their datacenters.
>>>
>>> The only real surprise we had was the so-called "discard storm",
>>> which Boris Burkov made adjustments to resolve a couple weeks ago[3].
>>> [...]
>>> [3]: https://lore.kernel.org/linux-btrfs/[email protected]/T/#t
>>
>> Wait, what? Argh. Sorry, if I had seen that patch, I wouldn't have
>> brought this up in my report at all. I missed it, as I wasn't CCed; and
>> regzbot missed it, because the patch uses a odd format for the lore link
>> (but not totally uncommon, will change regzbot to ensure that doesn't
>> happen again).
[for the record: I noticed the odd format was not the real problem; see
below]
> I'd need pay more attention when the regression tracking process is
> involved in case there are more patch versions floating around. People
> usually don't "CC enough" so that you have the regzbot in place helps
> to track the state.
First off: Sorry again for the trouble I caused.
Yes, a CC would have prevent it, as then I (aka the "human fallback")
would have seen the patch and noticed that regzbot missed something.
But normally it should have worked fine without the CC. In this case it
just didn't due to a bug in regzbot: the msg-id of the tracked report
contains slashes and it seems something in the url en/decoding within
regzbot went wrong somewhere. IOW: the fix in next (2e55571fddf ("btrfs:
set default discard iops_limit to 1000")) is fine the way it is from
regzbot point of view and I have to fix regzbot.
>> P.S.: /me meanwhile yet again wonders if we should tell people to add a
>> "CC: <[email protected]>" on patches fixing regressions. Then
>> in this case I would have become aware of the patch. And it makes it
>> obvious for anybody handling patches that the patch is fixing a
>> regression. But whatever, might not be worth it.
>
> I'm not sure if it would fit how regzbot workflow works, but syzbot
> provides links with the reports and then changes the state when the
> patch is committed containing the links.
You can't see it on the lists and only it regzbot web-ui, but regzbot
works similarly: it marks a regression as "fix incoming" when it notices
the fix hit linux-next.
> I don't see anything similar in
> the process/handling-regression document. If the "Link: <report>" is
> sufficient
It is, but maybe handling-regressions should point that out more
clearly. It currently is more implicit: "These tags are also crucial for
tools and scripts used by other kernel developers or Linux
distributions; one of these tools is regzbot, which heavily relies on
the “Link:” tags to associate reports for regression with changes
resolving them."
> then it should work already but there's no guarantee that the
> submitted patches would contain that. I add links to the committed
> versions
Thx for that, as the one Boris initially used[1] in the submission[2] is
not easily to resolve.
[1]
https://lore.kernel.org/linux-btrfs/[email protected]/T/#m6ebdeb475809ed7714b21b8143103fb7e5a966da
[2]
https://lore.kernel.org/linux-btrfs/[email protected]/T/#me406ebc5dc35e7fdbb59aece2158ac1a86b0c11b
> but then you'd need to scan at least linux-next. In any case
> with the regzbot it's fixable.
Ciao, Thorsten
On 17.04.23 13:38, Rafael J. Wysocki wrote:
> On Sun, Apr 16, 2023 at 10:49 PM Linus Torvalds
> <[email protected]> wrote:
>>
>> On Sun, Apr 16, 2023 at 10:59 AM Regzbot (on behalf of Thorsten
>> Leemhuis) <[email protected]> wrote:
>>>
>>> Wake-on-lan (WOL) apparently is broken for a huge number of users since
>>> 6.2 was released[1]. This is known for 8 weeks now and about 4 weeks ago
>>> it was bisected to commit 5c62d5aab87 ("ACPICA: Events: Support fixed
>>> PCIe wake event") we immediately could have reverted. The developer that
>>> looked into this was even willing to do the revert late March, but then
>>> got discouraged by a maintainer [2]. But well, a fix was apparently[3]
>>> finally posted for review last week (to the acpica-devel list); with a
>>> bit of luck your might get it next week. Still a bit sad that 6.2 is
>>> broken for so long now, as Greg wants to see it fixed in mainline first.
>>>
>>> [1] https://bugzilla.kernel.org/show_bug.cgi?id=217069
>>> [2] https://bugzilla.kernel.org/show_bug.cgi?id=217069#c50
>>> [3] https://lore.kernel.org/all/[email protected]/
>>
>> I find that bugzilla discussion very confusing, it's not clear what
>> the status of the patch actually is.
>>
>> And the sane lkml thread just says "the patch is under review" without
>> actually saying *where* said patch is, or where the review is.
>>
>> It sounds like it got perhaps into some internal ACPCICA queue? None
>> of those links are very clear on any of this.
>>
>> Rafael?
>
> There is a pull request for ACPICA that corresponds to this (IIUC),
> https://github.com/acpica/acpica/pull/866
Rafael, what happened to this? From the github acpica discussion is
looks a lot like there was no real progress. And
drivers/acpi/acpica/evevent.c in next looks unchanged; and from
searching for ACPI_EVENT_PCIE_WAKE on lore it also looks like neither
the fix nor a revert was posted. But I might be missing something.
/me still hopes we can get this fixed somehow before Linus releases 6.3
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
On Fri, Apr 21, 2023 at 3:49 PM Linux regression tracking (Thorsten
Leemhuis) <[email protected]> wrote:
>
> On 17.04.23 13:38, Rafael J. Wysocki wrote:
> > On Sun, Apr 16, 2023 at 10:49 PM Linus Torvalds
> > <[email protected]> wrote:
> >>
> >> On Sun, Apr 16, 2023 at 10:59 AM Regzbot (on behalf of Thorsten
> >> Leemhuis) <[email protected]> wrote:
> >>>
> >>> Wake-on-lan (WOL) apparently is broken for a huge number of users since
> >>> 6.2 was released[1]. This is known for 8 weeks now and about 4 weeks ago
> >>> it was bisected to commit 5c62d5aab87 ("ACPICA: Events: Support fixed
> >>> PCIe wake event") we immediately could have reverted. The developer that
> >>> looked into this was even willing to do the revert late March, but then
> >>> got discouraged by a maintainer [2]. But well, a fix was apparently[3]
> >>> finally posted for review last week (to the acpica-devel list); with a
> >>> bit of luck your might get it next week. Still a bit sad that 6.2 is
> >>> broken for so long now, as Greg wants to see it fixed in mainline first.
> >>>
> >>> [1] https://bugzilla.kernel.org/show_bug.cgi?id=217069
> >>> [2] https://bugzilla.kernel.org/show_bug.cgi?id=217069#c50
> >>> [3] https://lore.kernel.org/all/[email protected]/
> >>
> >> I find that bugzilla discussion very confusing, it's not clear what
> >> the status of the patch actually is.
> >>
> >> And the sane lkml thread just says "the patch is under review" without
> >> actually saying *where* said patch is, or where the review is.
> >>
> >> It sounds like it got perhaps into some internal ACPCICA queue? None
> >> of those links are very clear on any of this.
> >>
> >> Rafael?
> >
> > There is a pull request for ACPICA that corresponds to this (IIUC),
> > https://github.com/acpica/acpica/pull/866
>
> Rafael, what happened to this?
It will get fixed, most likely by reverting the offending commit and
most likely during the 6.4 merge window.
Note that ACPICA is involved, so the analogous revert needs to be
submitted there and I'm traveling right now.
Thanks!
On Fri, Apr 21, 2023 at 12:22 PM Rafael J. Wysocki <[email protected]> wrote:
>
> It will get fixed, most likely by reverting the offending commit and
> most likely during the 6.4 merge window.
No.
It's now reverted in my tree.
We're not doing *another* release with this known-broken garbage. It's
been pending for much too long already.
Known-broken commits either
(a) get a timely fix that doesn't have other questions
or
(b) get reverted
Not this kind of "this is broken, has been known to be broken for a
long time, people have bisected it, and we're just sitting here
wondering what to do".
> Note that ACPICA is involved, so the analogous revert needs to be
> submitted there and I'm traveling right now.
No, we're not waiting for "it's broken in the ACPICA tree" and using
that as an excuse to have a broken kernel.
If the ACPICA tree can't get their act together in two months, that's
their problem. It does not mean that users should need to suffer known
issues.
Linus
On Fri, Apr 21, 2023 at 10:45 PM Linus Torvalds
<[email protected]> wrote:
>
> On Fri, Apr 21, 2023 at 12:22 PM Rafael J. Wysocki <[email protected]> wrote:
> >
> > It will get fixed, most likely by reverting the offending commit and
> > most likely during the 6.4 merge window.
>
> No.
>
> It's now reverted in my tree.
Thanks for taking care of this and sorry for the trouble.
I was traveling Fri - Sun and I wouldn't have been able to push the
revert myself before today.
> We're not doing *another* release with this known-broken garbage. It's
> been pending for much too long already.
>
> Known-broken commits either
>
> (a) get a timely fix that doesn't have other questions
>
> or
>
> (b) get reverted
>
> Not this kind of "this is broken, has been known to be broken for a
> long time, people have bisected it, and we're just sitting here
> wondering what to do".
>
> > Note that ACPICA is involved, so the analogous revert needs to be
> > submitted there and I'm traveling right now.
>
> No, we're not waiting for "it's broken in the ACPICA tree" and using
> that as an excuse to have a broken kernel.
>
> If the ACPICA tree can't get their act together in two months, that's
> their problem. It does not mean that users should need to suffer known
> issues.
OK, in the future I'll deal with problematic commits coming from
ACPICA more timely without waiting for upstream.