Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Wed, 24 Jul 2019 09:30:14 -0700
From:   Eric Biggers <ebiggers@kernel.org>
To:     Eric Dumazet <eric.dumazet@gmail.com>
Cc:     Dmitry Vyukov <dvyukov@google.com>, netdev@vger.kernel.org,
        "David S. Miller" <davem@davemloft.net>,
        Florian Westphal <fw@strlen.de>,
        Ilya Maximets <i.maximets@samsung.com>,
        Eric Dumazet <edumazet@google.com>,
        David Ahern <dsahern@gmail.com>, linux-kernel@vger.kernel.org,
        syzkaller-bugs@googlegroups.com
Subject: Re: Reminder: 99 open syzbot bugs in net subsystem
Message-ID: <20190724163014.GC673@sol.localdomain>
Mail-Followup-To: Eric Dumazet <eric.dumazet@gmail.com>,
        Dmitry Vyukov <dvyukov@google.com>, netdev@vger.kernel.org,
        "David S. Miller" <davem@davemloft.net>,
        Florian Westphal <fw@strlen.de>,
        Ilya Maximets <i.maximets@samsung.com>,
        Eric Dumazet <edumazet@google.com>, David Ahern <dsahern@gmail.com>,
        linux-kernel@vger.kernel.org, syzkaller-bugs@googlegroups.com
References: <20190724013813.GB643@sol.localdomain>
 <63f12327-dd4b-5210-4de2-705af6bc4ba4@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <63f12327-dd4b-5210-4de2-705af6bc4ba4@gmail.com>
User-Agent: Mutt/1.12.1 (2019-06-15)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, Jul 24, 2019 at 08:39:05AM +0200, Eric Dumazet wrote:
> 
> 
> On 7/24/19 3:38 AM, Eric Biggers wrote:
> > [This email was generated by a script.  Let me know if you have any suggestions
> > to make it better, or if you want it re-generated with the latest status.]
> > 
> > Of the currently open syzbot reports against the upstream kernel, I've manually
> > marked 99 of them as possibly being bugs in the net subsystem.  This category
> > only includes the networking bugs that I couldn't assign to a more specific
> > component (bpf, xfrm, bluetooth, tls, tipc, sctp, wireless, etc.).  I've listed
> > these reports below, sorted by an algorithm that tries to list first the reports
> > most likely to be still valid, important, and actionable.
> > 
> > Of these 99 bugs, 17 were seen in mainline in the last week.
> > 
> > Of these 99 bugs, 4 were bisected to commits from the following people:
> > 
> > 	Florian Westphal <fw@strlen.de>
> > 	Ilya Maximets <i.maximets@samsung.com>
> > 	Eric Dumazet <edumazet@google.com>
> > 	David Ahern <dsahern@gmail.com>
> > 
> > If you believe a bug is no longer valid, please close the syzbot report by
> > sending a '#syz fix', '#syz dup', or '#syz invalid' command in reply to the
> > original thread, as explained at https://goo.gl/tpsmEJ#status
> > 
> > If you believe I misattributed a bug to the net subsystem, please let me know,
> > and if possible forward the report to the correct people or mailing list.
> >
> 
> Some of the bugs have been fixed already, before syzbot found them.
> 
> Why force human to be gentle to bots and actually replying to them ?
> 
> I usually simply wait that syzbot is finding the bug does not repro anymore,
> but now if you send these emails, we will have even more pressure on us.
> 

First, based on experience, I'd guess about 30-45 of these are still valid.  17
were seen in mainline in the last week, but some others are valid too.  The ones
most likely to still be valid are at the beginning of the list.  So let's try
not use the presence of outdated bugs as an excuse not to fix current bugs.

Second, all these bug reports are still open, regardless of whether reminders
are sent or not.  I think you're really suggesting that possibly outdated bug
reports should be automatically invalidated by syzbot.

syzbot already does that for bugs with no reproducer.  However, that still
leaves a lot of outdated bugs with reproducers.

Since the kernel community is basically in continuous bug bankruptcy and lots of
syzbot reports are being ignored anyway, I'm in favor of making the invalidation
criteria more aggressive, so we can best focus people's efforts.  I understand
that Dmitry has been against this though, since a significant fraction of bugs
that syzbot stopped hitting for some reason actually turn out to be still valid.

But we probably have no choice.  So I suggest we agree on new criteria for
invalidating bugs.  I'd suggest assigning a timeout to each bug, based on
attributes like "seen in mainline?", "reproducer type", "bisected?", "does it
look like a 'bad' crash (e.g. use-after-free)"; similar to the algorithm I'm
using to sort the bugs when sorting these reminders.  I.e., bugs most likely to
still be valid, important, and actionable get longest timeouts.

Then if no crash or activity was seen in the timeout, the bug is closed.

Any thoughts from anyone?

- Eric