Date:   Mon, 27 Feb 2023 19:52:39 -0500
From:   Sasha Levin <sashal@kernel.org>
To:     Matthew Wilcox <willy@infradead.org>
Cc:     Eric Biggers <ebiggers@kernel.org>, linux-kernel@vger.kernel.org,
        stable@vger.kernel.org, viro@zeniv.linux.org.uk,
        linux-fsdevel@vger.kernel.org
Subject: Re: AUTOSEL process
Message-ID: <Y/1QV9mQ31wbqFnp@sashalap>
References: <Y/rbGxq8oAEsW28j@sol.localdomain>
 <Y/rufenGRpoJVXZr@sol.localdomain>
 <Y/ux9JLHQKDOzWHJ@sol.localdomain>
 <Y/y70zJj4kjOVfXa@sashalap>
 <Y/zswi91axMN8OsA@sol.localdomain>
 <Y/zxKOBTLXFjSVyI@sol.localdomain>
 <Y/0U8tpNkgePu00M@sashalap>
 <Y/0i5pGYjrVw59Kk@gmail.com>
 <Y/0wMiOwoeLcFefc@sashalap>
 <Y/01z4EJNfioId1d@casper.infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <Y/01z4EJNfioId1d@casper.infradead.org>
Precedence: bulk

On Mon, Feb 27, 2023 at 10:59:27PM +0000, Matthew Wilcox wrote:
>On Mon, Feb 27, 2023 at 05:35:30PM -0500, Sasha Levin wrote:
>> On Mon, Feb 27, 2023 at 09:38:46PM +0000, Eric Biggers wrote:
>> > Just because you can't be 100% certain whether a commit is a fix doesn't mean
>> > you should be rushing to backport random commits that have no indications they
>> > are fixing anything.
>>
>> The difference in opinion here is that I don't think it's rushing: the
>> stable kernel rules say a commit must be in a released kernel, while the
>> AUTOSEL timelines make it so a commit must have been in two released
>> kernels.
>
>Patches in -rc1 have been in _no_ released kernels.  I'd feel a lot
>better about AUTOSEL if it didn't pick up changes until, say, -rc4,
>unless they were cc'd to stable.

This happened before my time, but -rc are considered releases.

The counter point to your argument/ask is that if you run the numbers on
regressions between -rc releases, it's the later one that tend to
introduce (way) more issues.

I've actually written about it a few years back to ksummit discuss
(here: https://lwn.net/Articles/753329/) because the numbers I saw
indicate that later -rc releases are 3x likely to introduce a
regression.

Linus pushed back on it saying that it is "by design" because those
commits are way more complex than ones that land during the early -rc
cycles.

So yes, I don't mind modifying the release workflow to decrease the
regressions we introduce, but I think that there's a difference between
what folks see as "helpful" and the outcome it would have.

>> > Nothing has changed, but that doesn't mean that your process is actually
>> > working.  7 days might be appropriate for something that looks like a security
>> > fix, but not for a random commit with no indications it is fixing anything.
>>
>> How do we know if this is working or not though? How do you quantify the
>> amount of useful commits?
>
>Sasha, 7 days is too short.  People have to be allowed to take holiday.

That's true, and I don't have strong objections to making it longer. How
often did it happen though? We don't end up getting too many replies
past the 7 day window.

I'll bump it to 14 days for a few months and see if it changes anything.

>> I'd love to improve the process, but for that we need to figure out
>> criteria for what we consider good or bad, collect data, and make
>> decisions based on that data.
>>
>> What I'm getting from this thread is a few anecdotal examples and
>> statements that the process isn't working at all.
>>
>> I took Jon's stablefixes script which he used for his previous articles
>> around stable kernel regressions (here:
>> https://lwn.net/Articles/812231/) and tried running it on the 5.15
>> stable tree (just a random pick). I've proceeded with ignoring the
>> non-user-visible regressions as Jon defined in his article (basically
>> issues that were introduced and fixed in the same releases) and ended up
>> with 604 commits that caused a user visible regression.
>>
>> Out of those 604 commits:
>>
>>  - 170 had an explicit stable tag.
>>  - 434 did not have a stable tag.
>
>I think a lot of people don't realise they have to _both_ put a Fixes
>tag _and_ add a Cc: stable.  How many of those 604 commits had a Fixes
>tag?

What do you mean? Just a cc: stable tag is enough to land it in stable,
you don't have to do both. The numbers above reflect that.

Running the numbers, there are 9422 commits with a Fixes tag in the 5.15
tree, out of which 360 had a regression, so 360 / 9422 = 3.82%.

-- 
Thanks,
Sasha