Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753307AbdL1KYE (ORCPT ); Thu, 28 Dec 2017 05:24:04 -0500 Received: from mail-pl0-f49.google.com ([209.85.160.49]:36040 "EHLO mail-pl0-f49.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752015AbdL1KYB (ORCPT ); Thu, 28 Dec 2017 05:24:01 -0500 X-Google-Smtp-Source: ACJfBou/wku2B2jTifyGTkDydPE5gx03ecQsclAla0sjvpSaZV48khhYzoQQqhoxoccf1ERaZYzCjYXuMyFvTxyGJgw= MIME-Version: 1.0 In-Reply-To: <20171221170505.GB10438@kroah.com> References: <20171221170505.GB10438@kroah.com> From: Dmitry Vyukov Date: Thu, 28 Dec 2017 11:23:39 +0100 Message-ID: Subject: Re: [RFC] syzbot process To: Greg Kroah-Hartman Cc: LKML , syzkaller , Eric Dumazet , Eric Biggers , Kostya Serebryany , Alexander Potapenko , andreyknvl , Linus Torvalds , Andrew Morton , Tetsuo Handa , David Miller , Willem de Bruijn , Guenter Roeck , Stephan Mueller Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4887 Lines: 100 On Thu, Dec 21, 2017 at 6:05 PM, Greg Kroah-Hartman wrote: > On Thu, Dec 21, 2017 at 01:52:40PM +0100, Dmitry Vyukov wrote: >> Hello, >> >> You might have seen bug reports coming from syzbot on LKML recently. >> syzbot is an automated system that continuously fuzzes main Linux >> kernel branches using syzkaller fuzzer and reports all found bugs: >> https://github.com/google/syzkaller/blob/master/docs/syzbot.md >> So far it has reported ~280 bugs in slightly less than 2 months: >> https://groups.google.com/forum/#!forum/syzkaller-bugs >> >> One of the ideas behind syzbot was maximum automation because we >> simply could not keep track, and not even report all bugs that >> syzkaller finds (the number has crossed 1000). Bugs were massively >> lost, not reported, context was lost (e.g. kernel commit, config, >> etc), we could not report similarly looking crashes, we could not test >> proposed fixes, etc. syzbot aims at solving all of these problems. >> >> However, the cost is that it needs to understand statuses of bugs: >> most importantly, what commit fixes what bug. It also has support for >> marking a bug as "invalid", e.g. happened once but most likely was >> caused by a previous silent memory corruption. And support for marking >> bugs as duplicates of other bugs, i.e. the same root cause and will be >> fixed when the target bug is fixed. These simple rules are outlined in >> the footer of each report and also explained in more detail at the >> referenced link: >> >> ---------------------------------- >> This bug is generated by a dumb bot. It may contain errors. >> See https://goo.gl/tpsmEJ for details. >> Direct all questions to syzkaller@googlegroups.com. >> Please credit me with: Reported-by: syzbot >> syzbot will keep track of this bug report. >> Once a fix for this bug is merged into any tree, reply to this email with: >> #syz fix: exact-commit-title >> If you want to test a patch for this bug, please reply with: >> #syz test: git://repo/address.git branch >> and provide the patch inline or as an attachment. >> To mark this as a duplicate of another syzbot report, please reply with: >> #syz dup: exact-subject-of-another-report >> If it's a one-off invalid bug report, please reply with: >> #syz invalid >> Note: if the crash happens again, it will cause creation of a new bug report. >> Note: all commands must start from beginning of the line in the email body. >> ---------------------------------- >> >> Status tracking allows syzbot to (1) keep track of still unfixed bugs >> (more than half actually gets lost in LKML archives if nobody keeps >> track of them), (2) be able to ever report similarly looking crashes >> as new bugs in future, (3) be able to test fixes. >> >> The problem is that these rules are mostly not followed. >> >> I understand that following these rules is a moderate pain and I am >> reaching to you to discuss potential solutions for this problem. I've >> tried hard to come up with a scheme that would eliminate sending these >> replies altogether, but failed. >> We can simply to agree to follow the current convention, which is not >> too hard in the end and sounds like a fair deal -- we give you bugs, >> you give us fixing commits. >> Or, we could somehow employ bugzilla.kernel.org for systematic bug >> tracking. However, I've got an impression that it's mostly unused and >> not used systematically even when used (e.g. bugs fixed 5 years ago >> still rest there as open). >> One idea that was proposed is do it the other way around. Namely, >> syzbot gives you bug id, and you need to add a tag along the lines of >> "syzbot-fix: HASH" to the commit. I don't know if kernel community >> finds this easier to use or not. And dups, invalid bugs and missed >> tags still needs to be handled in some other way (e.g. the current >> way). >> Any other proposals, thoughts, ideas? > > No, don't use bugzilla :) That was my impression as well :) > But, use what bugzilla does do well, provide a unique identifier that > can be referenced in git commit messages to point people at the problem > report. > > Why not generate a unique one per "problem" you find? And have a way to > look them up somehow? > > Then you can put: > syzkaller-id: XXXXXX > in the commit log and you can track it easier? > > That's what all other "tools" do that try to track kernel bug reports. > We do that for coverity as one such example, and lots of different bug > trackers as well. syzbot is capable of generating unique id's itself, and later we plan to create an UI to look up bugs, etc (a-la bugzilla) because syzbot does have all that info. So integrating with bugzilla won't directly help to solve _this_ problem. We could upload them to bugzilla as well, but taking into account amount of additional work, this is not top priority right now. Thanks