Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
MIME-Version: 1.0
References: <Yp59WCODvEDbpxOY@sol.localdomain>
In-Reply-To: <Yp59WCODvEDbpxOY@sol.localdomain>
From:   Aleksandr Nogikh <nogikh@google.com>
Date:   Tue, 7 Jun 2022 16:37:42 +0200
Message-ID: <CANp29Y7yEocOnLMhE_hc37L8wAzpvON9hwpjvuBLoMdQzhw+xA@mail.gmail.com>
Subject: Re: Auto-invalidating old syzbot reports?
To:     Eric Biggers <ebiggers@kernel.org>
Cc:     syzkaller <syzkaller@googlegroups.com>,
        Dmitry Vyukov <dvyukov@google.com>,
        LKML <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Hi Eric,

Thanks for contacting us!
These are very interesting points.

Syzbot indeed only closes old bugs without a reproducer, because if we
have a repro, then we can periodically do a fix bisection. And yes,
this mechanism unfortunately does not always work perfectly.

I think we could split the problem you described into two parts.
1) Some bugs that are "open" on the dashboard are actually no longer
relevant and should be closed.

If you know some old opened bugs with repro, which are actually
already fixed, could you please share them? It would be helpful to
figure out the exact reason why they are still open.
There are some cases when we can close bugs with a repro without
losing too much -- e.g. for bugs from -next there was a discussion in
https://github.com/google/syzkaller/issues/1957.
Also, if the fix bisection fails, but the repro no longer triggers the
crash on the HEAD, then we could probably "cancel" the repro and let
the bug be auto-closed (actually, auto-invalidated) later?

2) Some bugs were reported to the mailing lists, but became forgotten.

We could periodically take maintainers as per the latest commit and
send a reminder email to them. What do you think, would people go mad
if we did that for each bug e.g. every 6 months? :) Only if the bug
still happens on syzbot, of course.

At some point we were also considering sending aggregated reminders
(e.g. sth like "we still have X open bugs for the subsystem you
maintain/have actively contributed to, here they are:"), but to do
that, we first need to learn how to more or less reliably classify the
bugs into the subsystems.

--
Best Regards,
Aleksandr

On Tue, Jun 7, 2022 at 12:19 AM Eric Biggers <ebiggers@kernel.org> wrote:
>
> Currently the upstream Linux kernel has 888 open syzbot reports
> (https://syzkaller.appspot.com/upstream).  However, nearly two-thirds of them
> (577) were reported more than 1 year ago.  Old reports are often for bugs that
> were already fixed.  They can also be reports that got overlooked, forgotten
> about, not sent to the right place, etc.  Kernel maintainers also change over
> time, so the current maintainer(s) might never have received the original report
> even if syzbot sent the original report to the correct maintainer(s).
>
> Having these old reports open is preventing syzbot from re-reporting any bugs
> with the same crash signature (where a crash signature is something like
> "KASAN: null-ptr-deref Read in percpu_ref_exit") if it is still being seen.
>
> syzbot does auto-invalidate some old bugs, but only ones without a reproducer.
>
> Given that humans aren't keeping up with these reports, has it been considered
> to auto-invalidate all old syzbot reports -- not just ones without a reproducer?
>
> - Eric
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/Yp59WCODvEDbpxOY%40sol.localdomain.