Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Sasha Levin <Alexander.Levin@microsoft.com>
To:     Greg KH <gregkh@linuxfoundation.org>,
        "julia.lawall@lip6.fr" <julia.lawall@lip6.fr>
CC:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: bug-introducing patches (or: -rc cycles suck)
Thread-Topic: bug-introducing patches (or: -rc cycles suck)
Thread-Index: AQHT4KzY5hZ/zYBxbECTHPKrktWH/g==
Date:   Mon, 30 Apr 2018 17:58:30 +0000
Message-ID: <20180430175829.GB1544@sasha-vm>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: microsoft.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-ID: <DCC89C330ED1A945955D5303C6764F20@namprd21.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 4d7eb0b3-7194-42a4-0bdd-08d5aec3fbdf
X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Apr 2018 17:58:30.9176
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR2101MB0984
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Working on AUTOSEL, it became even more obvious to me how difficult it is f=
or a patch to get a proper review. Maintainers found it difficult to keep u=
p with the upstream work for their subsystem, and reviewing additional -sta=
ble patches put even more load on them which some suggested would be more t=
han what they can handle.

While AUTOSEL tries to understand if a patch fixes a bug, this was a bit la=
te: the bug was already introduced, folks already have to deal with it, and=
 the kernel is broken. I was wondering if I can do a similar process to AUT=
OSEL, but teach the AI about bug-introducing patches.

When someone fixes a bug, he would describe the patch differently than he w=
ould if he was writing a new feature. This lets AUTOSEL build on different =
commit message constructs, among various inputs, to recognize bug fixes. Ho=
wever, people are unaware that they introduce a bug, so the commit message =
for bug introducing patches is essentially the same as for commits that don=
't introduce a bug. This meant that I had to try and source data out of dif=
ferent sources.

Few of the parameters I ended up using are:
 - -next data (days spent in -next, changes in the patch between -next tree=
s, ...)
 - Mailing list data (was this patch ever sent to a ML? How long before it =
was merged? How many replies did it get? ...)
 - Author/commiter/maintainer chain data. Just like sports, some folks are =
more likely to produce better results than others. This goes beyond just "s=
kill", but also looks at things such as whether the author patches a subsys=
tem he's "familiar with" (=3D=3D subsystem where most of his patches usuall=
y go), or is he modifying a subsystem he never sent a patch for.
 - Patch complexity metrics - various code metrics to indicate how "complex=
" a patch is. Think 100 lines of whitespace fixes vs 100 lines that signifi=
cantly changes a subsystem.
 - Kernel process correctness - I tried using "violations" of the kernel pr=
ocess (patch formatting, correctness of the mailing to lkml, etc) as an ind=
icator of how familiar the author is with the kernel, with the presumption =
that folks who are newer to kernel development are more likely to introduce=
 bugs

Running an initial iteration on a set of commits made two things very obvio=
us to me:

1. -rc releases suck. seriously suck. The quality of commits that went in -=
rc cycles was much worse that merge window commit:
 - All commits had the same chance of introducing a bug whether they came i=
n a merge window or an -rc cycle. This means that -rc commits mostly end up=
 replacing obvious bugs with less obvious ones.
 - While the average merge window commit changes, on average, 3x more lines=
 than an -rc commit, the chances of a bug introduced per patch is the same,=
 which means that bugs-per-line metric of code is much higher with -rc patc=
hes.
 - A merge window commit spent 50% more days, on average, in -next than a -=
rc commit.
 - The number of -rc commits that never saw any mailing list or has never b=
een replied to on a mailing list was **way** higher than merge window commi=
ts.
 - For some reason, the odds of a -rc commit to be targetted for -stable is=
 over 20%, while for merge window commits it's about 3%. I can't quite expl=
ain why that happens, but this would suggest that -rc commits end up hurtin=
g -stable pretty badly.

2. Maintainers need to stop writing patches, commiting them, and pushing th=
em in without reviews.
In -rc cycles there is quite a large number of commits that were either wri=
tten by maintainers, commited, and merged upstream the same day. These patc=
hes are very likely to introduce a new bug.


I don't really have a proposal beyond "tighten up -rc cycles", but I think =
it's a discussion worth having. We have enough data to show what parts of k=
ernel development work, and what parts are just hurting us.=