Date: Thu, 21 Dec 2017 18:05:05 +0100
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: Dmitry Vyukov <dvyukov@google.com>
Cc: LKML <linux-kernel@vger.kernel.org>,
        syzkaller <syzkaller@googlegroups.com>,
        Eric Dumazet <edumazet@google.com>, Eric Biggers <ebiggers3@gmail.com>,
        Kostya Serebryany <kcc@google.com>,
        Alexander Potapenko <glider@google.com>,
        andreyknvl <andreyknvl@google.com>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>,
        David Miller <davem@davemloft.net>,
        Willem de Bruijn <willemb@google.com>,
        Guenter Roeck <groeck@google.com>,
        Stephan Mueller <smueller@chronox.de>
Subject: Re: [RFC] syzbot process
Message-ID: <20171221170505.GB10438@kroah.com>
References: <CACT4Y+bkOpUyjEGUT96T4tPGFeoQ2iP8ZGoE_FMiAg3h5RKxDA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CACT4Y+bkOpUyjEGUT96T4tPGFeoQ2iP8ZGoE_FMiAg3h5RKxDA@mail.gmail.com>
User-Agent: Mutt/1.9.2 (2017-12-15)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 4309
Lines: 91

On Thu, Dec 21, 2017 at 01:52:40PM +0100, Dmitry Vyukov wrote:
> Hello,
> 
> You might have seen bug reports coming from syzbot on LKML recently.
> syzbot is an automated system that continuously fuzzes main Linux
> kernel branches using syzkaller fuzzer and reports all found bugs:
> https://github.com/google/syzkaller/blob/master/docs/syzbot.md
> So far it has reported ~280 bugs in slightly less than 2 months:
> https://groups.google.com/forum/#!forum/syzkaller-bugs
> 
> One of the ideas behind syzbot was maximum automation because we
> simply could not keep track, and not even report all bugs that
> syzkaller finds (the number has crossed 1000). Bugs were massively
> lost, not reported, context was lost (e.g. kernel commit, config,
> etc), we could not report similarly looking crashes, we could not test
> proposed fixes, etc. syzbot aims at solving all of these problems.
> 
> However, the cost is that it needs to understand statuses of bugs:
> most importantly, what commit fixes what bug. It also has support for
> marking a bug as "invalid", e.g. happened once but most likely was
> caused by a previous silent memory corruption. And support for marking
> bugs as duplicates of other bugs, i.e. the same root cause and will be
> fixed when the target bug is fixed. These simple rules are outlined in
> the footer of each report and also explained in more detail at the
> referenced link:
> 
> ----------------------------------
> This bug is generated by a dumb bot. It may contain errors.
> See https://goo.gl/tpsmEJ for details.
> Direct all questions to syzkaller@googlegroups.com.
> Please credit me with: Reported-by: syzbot <syzkaller@googlegroups.com>
> syzbot will keep track of this bug report.
> Once a fix for this bug is merged into any tree, reply to this email with:
> #syz fix: exact-commit-title
> If you want to test a patch for this bug, please reply with:
> #syz test: git://repo/address.git branch
> and provide the patch inline or as an attachment.
> To mark this as a duplicate of another syzbot report, please reply with:
> #syz dup: exact-subject-of-another-report
> If it's a one-off invalid bug report, please reply with:
> #syz invalid
> Note: if the crash happens again, it will cause creation of a new bug  report.
> Note: all commands must start from beginning of the line in the email body.
> ----------------------------------
> 
> Status tracking allows syzbot to (1) keep track of still unfixed bugs
> (more than half actually gets lost in LKML archives if nobody keeps
> track of them), (2) be able to ever report similarly looking crashes
> as new bugs in future, (3) be able to test fixes.
> 
> The problem is that these rules are mostly not followed.
> 
> I understand that following these rules is a moderate pain and I am
> reaching to you to discuss potential solutions for this problem. I've
> tried hard to come up with a scheme that would eliminate sending these
> replies altogether, but failed.
> We can simply to agree to follow the current convention, which is not
> too hard in the end and sounds like a fair deal -- we give you bugs,
> you give us fixing commits.
> Or, we could somehow employ bugzilla.kernel.org for systematic bug
> tracking. However, I've got an impression that it's mostly unused and
> not used systematically even when used (e.g. bugs fixed 5 years ago
> still rest there as open).
> One idea that was proposed is do it the other way around. Namely,
> syzbot gives you bug id, and you need to add a tag along the lines of
> "syzbot-fix: HASH" to the commit. I don't know if kernel community
> finds this easier to use or not. And dups, invalid bugs and missed
> tags still needs to be handled in some other way (e.g. the current
> way).
> Any other proposals, thoughts, ideas?

No, don't use bugzilla :)

But, use what bugzilla does do well, provide a unique identifier that
can be referenced in git commit messages to point people at the problem
report.

Why not generate a unique one per "problem" you find?  And have a way to
look them up somehow?

Then you can put:
	syzkaller-id: XXXXXX
in the commit log and you can track it easier?

That's what all other "tools" do that try to track kernel bug reports.
We do that for coverity as one such example, and lots of different bug
trackers as well.

thanks,

greg k-h