Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <Yp59WCODvEDbpxOY@sol.localdomain> <CANp29Y7yEocOnLMhE_hc37L8wAzpvON9hwpjvuBLoMdQzhw+xA@mail.gmail.com>
 <Yp/wGYVsKR8M9eXI@sol.localdomain>
In-Reply-To: <Yp/wGYVsKR8M9eXI@sol.localdomain>
From:   Aleksandr Nogikh <nogikh@google.com>
Date:   Mon, 13 Jun 2022 17:47:22 +0200
Message-ID: <CANp29Y7Y-5KY8D8=tFQfYF_aHwNOh8pz3sumLuKM930EHW7msg@mail.gmail.com>
Subject: Re: Auto-invalidating old syzbot reports?
To:     Eric Biggers <ebiggers@kernel.org>
Cc:     syzkaller <syzkaller@googlegroups.com>,
        Dmitry Vyukov <dvyukov@google.com>,
        LKML <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Wed, Jun 8, 2022 at 2:41 AM Eric Biggers <ebiggers@kernel.org> wrote:
>
> On Tue, Jun 07, 2022 at 04:37:42PM +0200, 'Aleksandr Nogikh' via syzkaller wrote:
> > Hi Eric,
> >
> > Thanks for contacting us!
> > These are very interesting points.
> >
> > Syzbot indeed only closes old bugs without a reproducer, because if we
> > have a repro, then we can periodically do a fix bisection. And yes,
> > this mechanism unfortunately does not always work perfectly.
> >
> > I think we could split the problem you described into two parts.
> > 1) Some bugs that are "open" on the dashboard are actually no longer
> > relevant and should be closed.
> >
> > If you know some old opened bugs with repro, which are actually
> > already fixed, could you please share them? It would be helpful to
> > figure out the exact reason why they are still open.
> > There are some cases when we can close bugs with a repro without
> > losing too much -- e.g. for bugs from -next there was a discussion in
> > https://github.com/google/syzkaller/issues/1957.
> > Also, if the fix bisection fails, but the repro no longer triggers the
> > crash on the HEAD, then we could probably "cancel" the repro and let
> > the bug be auto-closed (actually, auto-invalidated) later?
> >
>
> Just to give the first definitive example I could find,
> https://syzkaller.appspot.com/bug?id=06c43cd0a71aec08de8ea55ca5cda816c45ab4e0
> ("KMSAN: uninit-value in _mix_pool_bytes") is a 3-year old bug that is still
> open even though it was fixed by commit f45a4248ea4c ("net: usb: rtl8150: set
> random MAC address when set_ethernet_addr() fails").

Thanks for providing the example!
Yes, KMSAN bugs are unfortunately a bit special at the moment - we
cannot do a proper fix bisection because the KMSAN repo is regularly
rebased. Correct Reported-by tag and manual fix reporting (#syz fix)
are right now the only ways to get KMSAN bugs closed.

For this particular bug the problem was that the fixing commit
referenced (Reported-by:
syzbot+abbc768b560c84d92fd3@syzkaller.appspotmail.com) another bug
(https://syzkaller.appspot.com/bug?extid=abbc768b560c84d92fd3).

>
> From working on syzbot reports in the past, I can say that the "already fixed"
> case for old reports is very common.  It is hard and time-consuming to actually
> identify them as such though, since it requires root-causing the bug.  If it was
> quick and easy to do so, there wouldn't be hundreds of such open reports...
>
> > 2) Some bugs were reported to the mailing lists, but became forgotten.
> >
> > We could periodically take maintainers as per the latest commit and
> > send a reminder email to them. What do you think, would people go mad
> > if we did that for each bug e.g. every 6 months? :) Only if the bug
> > still happens on syzbot, of course.
>
> That is greatly needed, but to get there we first need to get past the
> assumption that every syzbot report will get properly handled by humans and thus
> should never be automatically closed.  That assumption has been tried for the
> last 5 years, and unfortunately it isn't working.  (If responding to syzbot
> reports was being properly funded, it would be possible, but it's not.)  It
> looks like you agree, as per your suggestion that only crashes that still happen
> in syzbot should be reminded about.  I think syzbot actually needs to go further
> and close the old bug reports, not merely suppress reminders about them...

I've filed an issue about making syzbot periodically re-test the
reproducers. If the repro doesn't trigger a bug anymore, syzkaller
will pretend there's no repro and close the bug once crashes are not
happening anymore. This should hopefully resolve the problem with old
irrelevant bugs.
https://github.com/google/syzkaller/issues/3193

>
> In any case, reminders *must* include the latest crash details in a way that
> clearly shows that the bug is still relevant.

That's a good point, thanks!

>
> > At some point we were also considering sending aggregated reminders
> > (e.g. sth like "we still have X open bugs for the subsystem you
> > maintain/have actively contributed to, here they are:"), but to do
> > that, we first need to learn how to more or less reliably classify the
> > bugs into the subsystems.
>
> Well syzbot already identifies subsystems via the stack trace; it's just not as
> good as a human expert, and probably never will be since the correct subsystem
> can be very non-obvious.  For a short time, I was actually manually classifying
> the subsystems for syzbot reports and sending out reminders --- see
> https://lore.kernel.org/linux-kernel/?q=%22open+syzbot+bugs%22 --- but I gave up
> due to lack of support from my job for doing that work, combined with receiving
> somewhat less engagement than I had hoped.

Hmm, yes, maybe we can indeed give even the existing subsystem
detection mechanism a try.
And it looks like it's better to only send grouped reminders -- given
how much negative reaction one can see under your link(s), I wonder
what would there be if we send per-bug reminders :)

Hopefully I'll be able to get to the implementation in some foreseeable future.

>
> Perhaps the best solution would be to crowdsource by providing a self-service
> "#syz subsystem $FOO" command.
>
> - Eric