Date:   Sat, 11 Mar 2023 15:59:30 -0500
From:   Sasha Levin <sashal@kernel.org>
To:     Willy Tarreau <w@1wt.eu>
Cc:     Eric Biggers <ebiggers@kernel.org>, Theodore Ts'o <tytso@mit.edu>,
        Matthew Wilcox <willy@infradead.org>,
        Pavel Machek <pavel@ucw.cz>, linux-kernel@vger.kernel.org,
        stable@vger.kernel.org, viro@zeniv.linux.org.uk,
        linux-fsdevel@vger.kernel.org
Subject: Re: AUTOSEL process
Message-ID: <ZAzrsgAMX+LC9E5A@sashalap>
References: <ZATC3djtr9/uPX+P@duo.ucw.cz>
 <ZAewdAql4PBUYOG5@gmail.com>
 <ZAwe95meyCiv6qc4@casper.infradead.org>
 <ZAyK0KM6JmVOvQWy@sashalap>
 <20230311161644.GH860405@mit.edu>
 <ZAy+3f1/xfl6dWpI@sol.localdomain>
 <ZAzJltJaydwjCN6E@1wt.eu>
 <ZAzVbzthi8IfptFZ@sol.localdomain>
 <ZAzafagDchRQRxWi@sol.localdomain>
 <ZAzianzvIOUrH5pr@1wt.eu>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <ZAzianzvIOUrH5pr@1wt.eu>
Precedence: bulk

On Sat, Mar 11, 2023 at 09:19:54PM +0100, Willy Tarreau wrote:
>On Sat, Mar 11, 2023 at 11:46:05AM -0800, Eric Biggers wrote:
>> (And please note, the key word here is *confidence*.  We all agree that it's
>> never possible to be absolutely 100% sure whether a commit is appropriate for
>> stable or not.  That's a red herring.
>
>In fact even developers themselves sometimes don't know, and even when they
>know, sometimes they know after committing it. Many times we've found that
>a bug was accidently resolved by a small change. Just for this it's important
>to support a post-merge analysis.
>> And I would assume, or at least hope, that the neural network thing being used
>> for AUTOSEL outputs a confidence rating and not just a yes/no answer.  If it
>> actually just outputs yes/no, well how is anyone supposed to know that and fix
>> that, given that it does not seem to be an open source project?)
>
>Honestly I don't know. I ran a few experiments with natural language
>processors such as GPT-3 on commit messages which contained human-readable
>instructions, and asking "what am I expected to do with these patches", and
>seeing the bot respond "you should backport them to this version, change
>this and that in that version, and preliminary take that patch". It
>summarized extremely well the instructions delivered by the developer,
>which is awesome, but was not able to provide any form of confidence
>level. I don't know what Sasha uses but wouldn't be surprised it shares
>some such mechanisms and that it might not always be easy to get such a
>confidence level. But I could be wrong.

It's actually pretty stupid: it uses the existence of ~10k of the most
common words in commit messages + metrics from cqmetrics
(github.com/dspinellis/cqmetrics) as input.

Although I get a score, which is already set pretty high, confidence is
really non-existant here: at the end it depends mostly on the writing
style of said commit author more than anything.

-- 
Thanks,
Sasha