Received-SPF: pass (google.com: domain of linux-kernel+bounces-158302-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <9fb128b1-d625-477f-8a16-aade00b13bc6@leemhuis.info>
Date: Thu, 25 Apr 2024 11:33:08 +0200
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: regression fixes sitting in subsystem git trees for a week or
 longer
To: Benjamin Tissoires <bentiss@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
 Jiri Kosina <jikos@kernel.org>, Douglas Anderson <dianders@chromium.org>,
 Hans de Goede <hdegoede@redhat.com>, linux-input@vger.kernel.org,
 linux-kernel@vger.kernel.org, Kenny Levinsen <kl@kl.wtf>,
 Benjamin Tissoires <benjamin.tissoires@redhat.com>,
 Linux regressions mailing list <regressions@lists.linux.dev>
References: <20240331182440.14477-1-kl@kl.wtf>
 <14d1b38e-0f11-4852-9c52-92b4bedb0a77@leemhuis.info>
 <CAO-hwJJtK2XRHK=HGaNUFb3mQhY5XbNGeCQwuAB0nmG2bjHX-Q@mail.gmail.com>
 <a810561a-14f3-412e-9903-acaba7a36160@leemhuis.info>
 <CAHk-=wjy_ph9URuFt-pq+2AJ__p7gFDx=yzVSCsx16xAYvNw9g@mail.gmail.com>
 <87698732-5439-42bd-b2b2-864bb4f3b3ec@leemhuis.info>
 <qcd5klmhyx23rowpbm4egshm6hemhh4stq7r6soblnuul55524@yyktdlowepw7>
From: "Linux regression tracking (Thorsten Leemhuis)"
 <regressions@leemhuis.info>
Content-Language: en-US, de-DE
Reply-To: Linux regressions mailing list <regressions@lists.linux.dev>
In-Reply-To: <qcd5klmhyx23rowpbm4egshm6hemhh4stq7r6soblnuul55524@yyktdlowepw7>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

On 25.04.24 10:44, Benjamin Tissoires wrote:
> On Apr 25 2024, Thorsten Leemhuis wrote:
>> On 24.04.24 20:53, Linus Torvalds wrote:
>>> On Wed, 24 Apr 2024 at 09:56, Thorsten Leemhuis
>>> <regressions@leemhuis.info> wrote:
>> [...]
>> And the arch linux wiki even documents a workaround:
>> https://wiki.archlinux.org/title/Lenovo_ThinkPad_Z16_Gen_2#Initialization_failure
>>
>> Those are just the reports and discussions I found. And you know how
>> it is: many people that struggle will never report a problem.
> 
> short FYI, (I've Cc-ed you on the PR), but I just sent the PR for HID,
> which includes this fix.

Great, many thx. Saw it right after sending my mail... :-/

>> Is cherry picking from -next as easy for you? Maintainers sometimes
>> improve small details when merging a fix, so it might be better to
>> take fixes from there instead of pulling them from lore.
> 
> Maybe one suggestion that might help to reduce these kind of situations
> in the future: can you configure your bot to notify the maintainers
> after a couple of days that the patch has been merged that it would be
> nice if they could send the PR to Linus?

Yes, that is an idea in the long run, but I'm not sure if it's wise now
or later. People easily get annoyed by these mails (which I totally
understand!) and then will start hating the bot or regression tracking
in general. That's why I'm really careful here.

There are also the subsystems that regularly flush their fixes shortly
before a new -rc, so they likely never want to see such reminders. And
sending them right after a new -rc is better than nothing, but not ideal
either.

IOW: it's complicated. :-/

> In this case I bet Jiri forgot to send it because he was overloaded and
> so was I.

Understood and no worries. But this became a good opportunity to raise
the general problem, as that is something that bugs me. Sorry. Hope you
don't mind to much that I used that chance.

> So a friendly reminder could make things go faster.

I'll already did this occasionally manually, but that of course does not
scale. Sometimes I wonder if it would be more efficient for nearly all
of us if subsystems just flushed their -fixes branch shortly before each
new -rc, as Linus apparently is not bothered by PRs that contain just a
change or two. But that of course creates work for each of the subsystem
maintainers, unless they creates scripts to handle that work nearly for
free (it seems to me the x86 folks have something like that).

Of course that would mean...

> And maybe, before sending the reminder, if you could also check that the
> target branch hasn't been touched in 2 days that would prevent annoyances
> when we just added a commit and want to let it stay in for-next for 24h
> before sending the full branch.

..that nothing big or slightly dangerous should be merged to -fixes
branches on Fridays.

>> P.S: Wondering if I should team up with the kernel package maintainers
>> of Arch Linux, Fedora, and openSUSE and start a git tree based on the
>> latest stable tree with additional fixes and reverts for regressions
>> not yet fixed upstream...[1] But that feels kinda wrong: it IMHO
>> would be better to resolve those problems quickly in the proper
>> upstream trees.
> 
> I would also say that this is wrong. Unless all regressions go through
> your tree and you then send PR to Linus, [...]

Ohh, sorry, I was not clear here, as that would be totally wrong --
fixes definitely should go through the subsystems trees, as they have
the knowledge and the infra to check them (hmm, maybe a dedicated tree
might make sense for the smaller subsystems, but let's ignore that).

What I meant was just a tree those distros could merge into their
kernels to quickly resolve issues that upstream is slow to fix. But that
obviously has downsides, too. And is yet more work.

> However, do you have some kind of dashboard that you could share with
> the package maintainers? This way they could easily compare the not-yet
> applied fixes with their bugs and decide to backport them themselves.

I have for the kernel overall, but nothing subsystem specific. But that
is pretty high on my todo list, as...

> In other words: let others do the hard work, you are doing a lot already

..I'm very well aware of this. :-/

Ciao, Thorsten