Hi Linus, hi Greg,
I got the impression that early stable releases with a huge number of
patches (like 6.3.2 with ~690 changes) seems to cause a few regressions.
As you know, those releases usually contain many backports of changes
merged during the merge window for the following mainline release (e.g.
6.4). That made me wonder:
How many patches do we have in linux-next right now that better should
be merged this cycle (e.g. ahead of the 6.4 release) instead of merging
them in the merge window for 6.5 and backporting them shortly afterwards?
To check I briefly set down and quickly hacked together a python
script[1] that looks at linux-next for patches with tags like 'Cc:
stable...' and 'Fixes: ', as all respectively some (or many?) of those
will be backported. I made the script ignore a few things, like commits
from the past eight days and commits that fix changes committed to
mainline more that a year ago.
I ran this a few minutes ago and it spilled out about 260 changes (about
80 of them with a stable tag). I put the results into a table:
https://docs.google.com/spreadsheets/d/1OnMrde1e7LBMPhOPJL0Sn9rd3W32mTGls_qGMoZS8z8/edit?usp=sharing
And now I'm not really sure what to do with this data. :-/ :-D
Some of those commits the script found to my untrained eyes look like
something that might have been worth having in 6.4 -- especially those
with a stable-tag. But not all of them. And I'm not suggesting to merge
them. It was just a exercise to see if this might be useful.
What do you think about this? Is this helpful somehow? Or can it be made
more helpful with a few changes? Especially if someone would regularly
run this?
Ciao, Thorsten
[1]
https://gitlab.com/knurd42/kernel-scripts/-/blob/master/stats/next_stable_candidates.py
(as any code likely will contain bugs)
On 18.06.23 10:49, Thorsten Leemhuis wrote:
> Hi Linus, hi Greg,
>
> I got the impression that early stable releases with a huge number of
> patches (like 6.3.2 with ~690 changes) seems to cause a few regressions.
> As you know, those releases usually contain many backports of changes
> merged during the merge window for the following mainline release (e.g.
> 6.4). That made me wonder:
>
> How many patches do we have in linux-next right now that better should
> be merged this cycle (e.g. ahead of the 6.4 release) instead of merging
> them in the merge window for 6.5 and backporting them shortly afterwards?
>
> To check I briefly set down and quickly hacked together a python
> script[1] that looks at linux-next for patches with tags like 'Cc:
> stable...' and 'Fixes: ', as all respectively some (or many?) of those
> will be backported. I made the script ignore a few things, like commits
> from the past eight days and commits that fix changes committed to
> mainline more that a year ago.
>
> I ran this a few minutes ago and it spilled out about 260 changes (about
> 80 of them with a stable tag). I put the results into a table:
> https://docs.google.com/spreadsheets/d/1OnMrde1e7LBMPhOPJL0Sn9rd3W32mTGls_qGMoZS8z8/edit?usp=sharing
TWIMC, I just updated the data slightly, as I updated the script to also
look for commits that revert changes from the current mainline cycle.
Did that while I was preparing this weeks regression report and noticed
a series of reverts[1] in next where my brain said "hmmm, Andrew merged
those more than a week ago to mm-hotfixes-unstable and -rc7 is due
today; I don't see a pr from him and wonder if these revert are
something that better should be in rc7 to help preventing a rc8?"
[1] https://lore.kernel.org/all/[email protected]/
noticed it via
https://lore.kernel.org/lkml/[email protected]/
BTW, quick reminder for the data in the spreadsheet, to ensure nobody
gets this wrong:
> [...] And I'm not suggesting to merge them [to mainline].
> It was just a exercise to see if this might be useful.
IOW: the spreadsheet definitely mentioned commits in next that
definitely are not suited for this merge window.
Ciao, Thorsten
On Sun, Jun 18, 2023 at 03:40:15PM +0200, Thorsten Leemhuis wrote:
> On 18.06.23 10:49, Thorsten Leemhuis wrote:
> > Hi Linus, hi Greg,
> >
> > I got the impression that early stable releases with a huge number of
> > patches (like 6.3.2 with ~690 changes) seems to cause a few regressions.
> > As you know, those releases usually contain many backports of changes
> > merged during the merge window for the following mainline release (e.g.
> > 6.4). That made me wonder:
> >
> > How many patches do we have in linux-next right now that better should
> > be merged this cycle (e.g. ahead of the 6.4 release) instead of merging
> > them in the merge window for 6.5 and backporting them shortly afterwards?
> >
> > To check I briefly set down and quickly hacked together a python
> > script[1] that looks at linux-next for patches with tags like 'Cc:
> > stable...' and 'Fixes: ', as all respectively some (or many?) of those
> > will be backported. I made the script ignore a few things, like commits
> > from the past eight days and commits that fix changes committed to
> > mainline more that a year ago.
> >
> > I ran this a few minutes ago and it spilled out about 260 changes (about
> > 80 of them with a stable tag). I put the results into a table:
> > https://docs.google.com/spreadsheets/d/1OnMrde1e7LBMPhOPJL0Sn9rd3W32mTGls_qGMoZS8z8/edit?usp=sharing
>
> TWIMC, I just updated the data slightly, as I updated the script to also
> look for commits that revert changes from the current mainline cycle.
>
> Did that while I was preparing this weeks regression report and noticed
> a series of reverts[1] in next where my brain said "hmmm, Andrew merged
> those more than a week ago to mm-hotfixes-unstable and -rc7 is due
> today; I don't see a pr from him and wonder if these revert are
> something that better should be in rc7 to help preventing a rc8?"
>
> [1] https://lore.kernel.org/all/[email protected]/
> noticed it via
> https://lore.kernel.org/lkml/[email protected]/
Having "only" 82 commits that cc: stable is a _HUGE_ decrease, so that's
great to see. That's a very low % overall from the number of
non-cc-stable commits in the tree, so this feels good to me.
Most of those are tiny stuff, dts fixes, fixes for minor
platforms/drivers, lockdep fixes, and other tiny things that maintainers
probably just don't think worthy of getting into -final now, as long as
they make it into the tree "eventually". I know it's that way for some
of the commits in my trees that are tagged that way.
The "fixes-only" commits are a bit more interesting, we still have huge
swaths of subsystems that refuse to actually tag commits for stable, but
luckily developers know to at least put a "Fixes:" tag on their fixes,
which help us out in classifying where they should go.
The "reverts" look not so good to me, but it's hard to know if they are
followed up with a "real fix" afterward, which is common at times.
Anyway, thanks for doing this, looks pretty sane to me.
greg k-h
On 19.06.23 10:49, Greg KH wrote:
> On Sun, Jun 18, 2023 at 03:40:15PM +0200, Thorsten Leemhuis wrote:
>> On 18.06.23 10:49, Thorsten Leemhuis wrote:
>>>
>>> I got the impression that early stable releases with a huge number of
>>> patches (like 6.3.2 with ~690 changes) seems to cause a few regressions.
>>> As you know, those releases usually contain many backports of changes
>>> merged during the merge window for the following mainline release (e.g.
>>> 6.4). That made me wonder:
>>>
>>> How many patches do we have in linux-next right now that better should
>>> be merged this cycle (e.g. ahead of the 6.4 release) instead of merging
>>> them in the merge window for 6.5 and backporting them shortly afterwards?
>>>
>>> To check I briefly set down and quickly hacked together a python
>>> script[1] that looks at linux-next for patches with tags like 'Cc:
>>> stable...' and 'Fixes: ', as all respectively some (or many?) of those
>>> will be backported. I made the script ignore a few things, like commits
>>> from the past eight days and commits that fix changes committed to
>>> mainline more that a year ago.
>>>
>>> I ran this a few minutes ago and it spilled out about 260 changes (about
>>> 80 of them with a stable tag). I put the results into a table:
>>> https://docs.google.com/spreadsheets/d/1OnMrde1e7LBMPhOPJL0Sn9rd3W32mTGls_qGMoZS8z8/edit?usp=sharing
>>
>> TWIMC, I just updated the data slightly, as I updated the script to also
>> look for commits that revert changes from the current mainline cycle.
>>
>> Did that while I was preparing this weeks regression report and noticed
>> a series of reverts[1] in next where my brain said "hmmm, Andrew merged
>> those more than a week ago to mm-hotfixes-unstable and -rc7 is due
>> today; I don't see a pr from him and wonder if these revert are
>> something that better should be in rc7 to help preventing a rc8?"
>>
>> [1] https://lore.kernel.org/all/[email protected]/
>> noticed it via
>> https://lore.kernel.org/lkml/[email protected]/
>
> Having "only" 82 commits that cc: stable is a _HUGE_ decrease, so that's
> great to see. That's a very low % overall from the number of
> non-cc-stable commits in the tree, so this feels good to me.
Yeah, from my point of view it also looks like things improved.
> Most of those are tiny stuff, dts fixes, fixes for minor
> platforms/drivers, lockdep fixes, and other tiny things that maintainers
> probably just don't think worthy of getting into -final now, as long as
> they make it into the tree "eventually". I know it's that way for some
> of the commits in my trees that are tagged that way.
Yup. But some seem to fix regressions. a0f19091d4f5, 6a87679626b5 and
88d341716b83 for example looked suspicious to my untrained eyes during a
brief look at the first 20 changes CCed stable. But developers and
maintainers of course are in a way better position to decide how to
handle those patches.
BTW, some of those 20 changes and a off-list conversation about this
thread reminded me of something related:
Last year on the maintainers summit we discussed this "delayed stable
backport" thingy that afaik works something like this:
Cc: <[email protected]> # after 4 weeks
Or is it more like this?
Cc: <[email protected]> # 6.2 [after 4 weeks]
I think the conclusion was to add this to the documentation, but that
afaics never happened. If you tell me the format you or your scripts
expect, I'd be willing to create a patch.
> The "fixes-only" commits are a bit more interesting, we still have huge
> swaths of subsystems that refuse to actually tag commits for stable, but
> luckily developers know to at least put a "Fixes:" tag on their fixes,
> which help us out in classifying where they should go.
Various aspects contribute to this, but due to my regression tracking
efforts two of them jumped to my mind here:
* A clear statement from Linus wrt to the stable tags in changes that
fix regressions would be good. E.g. something along the lines of "always
add a CC: <stable@... tag when fixing a regression caused by change
mainlined during the past year to ensure the fix reaches the users of
stable trees quickly".
My recent changes to handling-regressions.rst[1] touched on that bit
already and told developers to do so. But it did so without any clear
precedent from Linus (as mentioned in the patch description). Hence it
afaics would help a lot if Linus said something like that to make it
"official".
[1] See eed892da9cd ("docs: handling-regressions: rework section about
fixing procedures") in next. To quote: ```If a regression made it into a
proper mainline release during the past twelve months, ensure to tag the
fix with "Cc: [email protected]", as a "Fixes:" tag alone does not
guarantee a backport. Please add the same tag, in case you know the
culprit was backported to stable or longterm kernels.```
* a (big?) part of the problem afaics is that many developers and
maintainers seem to think that a "Fixes:" tag is enough to ensure a
backport. You efforts educating them[2] at least from here look a bit
like a endless game of whac-a-mole, as you sent such mails for quite a
while already and it seems nothing much has changed. Sometimes I wonder
if we should spam everyone in MAINTAINERS (and some of the regular
developers as well?) with a short PSA trying to kill that myth. But I
don't really like that idea myself. Maybe it would help if Linus
mentions it two or three times in release announcements?
[2] for the unaware and the record, here are two recent ones:
https://lore.kernel.org/all/2023060703-colony-shakily-3514@gregkh/
https://lore.kernel.org/all/2023061137-algorithm-almanac-1337@gregkh/
> The "reverts" look not so good to me, but it's hard to know if they are
> followed up with a "real fix" afterward, which is common at times.
Yup.
> Anyway, thanks for doing this, looks pretty sane to me.
Thx. Maybe with a bit of fine tuning this becomes more useful and
something I can run regularly.
We'll see.
Ciao, Thorsten
On Mon, Jun 19, 2023 at 02:19:49PM +0200, Thorsten Leemhuis wrote:
> Last year on the maintainers summit we discussed this "delayed stable
> backport" thingy that afaik works something like this:
>
> Cc: <[email protected]> # after 4 weeks
>
> Or is it more like this?
>
> Cc: <[email protected]> # 6.2 [after 4 weeks]
>
> I think the conclusion was to add this to the documentation, but that
> afaics never happened. If you tell me the format you or your scripts
> expect, I'd be willing to create a patch.
Either of them will work, as my "script" is me reviewing each patch :)
It's rare that I see this, I have been doing it for a few USB patches
recently, but I don't mark them as I know to hold off on stable
integration a bit, but I can't read the minds of other maintainers.
> > The "fixes-only" commits are a bit more interesting, we still have huge
> > swaths of subsystems that refuse to actually tag commits for stable, but
> > luckily developers know to at least put a "Fixes:" tag on their fixes,
> > which help us out in classifying where they should go.
>
> Various aspects contribute to this, but due to my regression tracking
> efforts two of them jumped to my mind here:
>
> * A clear statement from Linus wrt to the stable tags in changes that
> fix regressions would be good. E.g. something along the lines of "always
> add a CC: <stable@... tag when fixing a regression caused by change
> mainlined during the past year to ensure the fix reaches the users of
> stable trees quickly".
Sounds good to me for you to say that! It should happen, but remember
many maintainers still don't want to, or feel they need to, tag anything
for stable. And that's fine, I can't tell people what to do and the
stable tree stuff was ALWAYS designed to never require maintainers to do
more work than they wanted to.
Hence Sasha's great AUTOSEL work in digging out patches from those
subsystems that do NOT mark anything for stable.
So while we can ask, we can never require.
> * a (big?) part of the problem afaics is that many developers and
> maintainers seem to think that a "Fixes:" tag is enough to ensure a
> backport. You efforts educating them[2] at least from here look a bit
> like a endless game of whac-a-mole, as you sent such mails for quite a
> while already and it seems nothing much has changed. Sometimes I wonder
> if we should spam everyone in MAINTAINERS (and some of the regular
> developers as well?) with a short PSA trying to kill that myth. But I
> don't really like that idea myself. Maybe it would help if Linus
> mentions it two or three times in release announcements?
>
> [2] for the unaware and the record, here are two recent ones:
> https://lore.kernel.org/all/2023060703-colony-shakily-3514@gregkh/
> https://lore.kernel.org/all/2023061137-algorithm-almanac-1337@gregkh/
Given that the cc: stable predates the Fixes: tag by years, it's funny
that people don't realize this.
BUT it's the fixes tag that we have been using for those subsystems that
do NOT tag stuff for stable, so I guess when people saw patches flow in,
they just "assumed" that this was the normal process.
So yes, I'll keep reminding people, and Sasha does a great job in
sweeping up the Fixes: only patches, it is never guaranteed that this
will get into a stable release.
Except for a yearly "here is how stable works" email like you say to all
MAINTAINERS, I don't know how it would be any more explicit than what we
have documented today. Maybe I should do that yearly type of an email,
consider it a christmas card update or something :)
thanks,
greg k-h
On 19.06.23 15:58, Greg KH wrote:
> On Mon, Jun 19, 2023 at 02:19:49PM +0200, Thorsten Leemhuis wrote:
>> Last year on the maintainers summit we discussed this "delayed stable
>> backport" thingy that afaik works something like this:
>>
>> Cc: <[email protected]> # after 4 weeks
>>
>> Or is it more like this?
>>
>> Cc: <[email protected]> # 6.2 [after 4 weeks]
>>
>> I think the conclusion was to add this to the documentation, but that
>> afaics never happened. If you tell me the format you or your scripts
>> expect, I'd be willing to create a patch.
>
> Either of them will work, as my "script" is me reviewing each patch :)
>
> It's rare that I see this, [...]
I guess that partially because many people are not aware of this option. :-D
>>> The "fixes-only" commits are a bit more interesting, we still have huge
>>> swaths of subsystems that refuse to actually tag commits for stable, but
>>> luckily developers know to at least put a "Fixes:" tag on their fixes,
>>> which help us out in classifying where they should go.
>>
>> Various aspects contribute to this, but due to my regression tracking
>> efforts two of them jumped to my mind here:
>>
>> * A clear statement from Linus wrt to the stable tags in changes that
>> fix regressions would be good. E.g. something along the lines of "always
>> add a CC: <stable@... tag when fixing a regression caused by change
>> mainlined during the past year to ensure the fix reaches the users of
>> stable trees quickly".
>
> Sounds good to me for you to say that! It should happen, but remember
> many maintainers still don't want to, or feel they need to, tag anything
> for stable. And that's fine, I can't tell people what to do and the
> stable tree stuff was ALWAYS designed to never require maintainers to do
> more work than they wanted to.
Yeah, I know and that's totally fine with me. The problem is that the
tag is overloaded with two meanings ("I want to backport this" and "I'm
willing to help with backporting in case of trouble"). Something along
the lines of
Cc: <[email protected]> # notmyjob
could help for this case, so people can indicate "this really should be
backported, but I won't help with that".
> So while we can ask, we can never require.
Well, we IMHO could require to tag them to ensure they are not missed
and quickly picked up.
>> * a (big?) part of the problem afaics is that many developers and
>> maintainers seem to think that a "Fixes:" tag is enough to ensure a
>> backport. You efforts educating them[2] at least from here look a bit
>> like a endless game of whac-a-mole, as you sent such mails for quite a
>> while already and it seems nothing much has changed. Sometimes I wonder
>> if we should spam everyone in MAINTAINERS (and some of the regular
>> developers as well?) with a short PSA trying to kill that myth. But I
>> don't really like that idea myself. Maybe it would help if Linus
>> mentions it two or three times in release announcements?
>>
>> [2] for the unaware and the record, here are two recent ones:
>> https://lore.kernel.org/all/2023060703-colony-shakily-3514@gregkh/
>> https://lore.kernel.org/all/2023061137-algorithm-almanac-1337@gregkh/
>
> Given that the cc: stable predates the Fixes: tag by years, it's funny
> that people don't realize this.
>
> BUT it's the fixes tag that we have been using for those subsystems that
> do NOT tag stuff for stable, so I guess when people saw patches flow in,
> they just "assumed" that this was the normal process.
Guess so.
> So yes, I'll keep reminding people, and Sasha does a great job in
> sweeping up the Fixes: only patches, it is never guaranteed that this
> will get into a stable release.
>
> Except for a yearly "here is how stable works" email like you say to all
> MAINTAINERS, I don't know how it would be any more explicit than what we
> have documented today. Maybe I should do that yearly type of an email,
> consider it a christmas card update or something :)
How about "10 myths Linux developers believe about kernels development
process; number 6 will astound you".
/me runs and hides
Ciao, Thorsten