Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753783AbdLUMxF (ORCPT ); Thu, 21 Dec 2017 07:53:05 -0500 Received: from mail-pl0-f65.google.com ([209.85.160.65]:41141 "EHLO mail-pl0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750768AbdLUMxC (ORCPT ); Thu, 21 Dec 2017 07:53:02 -0500 X-Google-Smtp-Source: ACJfBov5KSmR2vTi5PlMd43leaJxYAJHabfYXt90cHfqNbjzCyDjFAwrBdAGzJ+BmgskLBAdFPmKPHdTSzK3nxRTKCY= MIME-Version: 1.0 From: Dmitry Vyukov Date: Thu, 21 Dec 2017 13:52:40 +0100 Message-ID: Subject: [RFC] syzbot process To: LKML , syzkaller , Eric Dumazet , Eric Biggers , Kostya Serebryany , Alexander Potapenko , andreyknvl , Linus Torvalds , Greg Kroah-Hartman , Andrew Morton , Tetsuo Handa , David Miller , Willem de Bruijn , Guenter Roeck , Stephan Mueller Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3577 Lines: 71 Hello, You might have seen bug reports coming from syzbot on LKML recently. syzbot is an automated system that continuously fuzzes main Linux kernel branches using syzkaller fuzzer and reports all found bugs: https://github.com/google/syzkaller/blob/master/docs/syzbot.md So far it has reported ~280 bugs in slightly less than 2 months: https://groups.google.com/forum/#!forum/syzkaller-bugs One of the ideas behind syzbot was maximum automation because we simply could not keep track, and not even report all bugs that syzkaller finds (the number has crossed 1000). Bugs were massively lost, not reported, context was lost (e.g. kernel commit, config, etc), we could not report similarly looking crashes, we could not test proposed fixes, etc. syzbot aims at solving all of these problems. However, the cost is that it needs to understand statuses of bugs: most importantly, what commit fixes what bug. It also has support for marking a bug as "invalid", e.g. happened once but most likely was caused by a previous silent memory corruption. And support for marking bugs as duplicates of other bugs, i.e. the same root cause and will be fixed when the target bug is fixed. These simple rules are outlined in the footer of each report and also explained in more detail at the referenced link: ---------------------------------- This bug is generated by a dumb bot. It may contain errors. See https://goo.gl/tpsmEJ for details. Direct all questions to syzkaller@googlegroups.com. Please credit me with: Reported-by: syzbot syzbot will keep track of this bug report. Once a fix for this bug is merged into any tree, reply to this email with: #syz fix: exact-commit-title If you want to test a patch for this bug, please reply with: #syz test: git://repo/address.git branch and provide the patch inline or as an attachment. To mark this as a duplicate of another syzbot report, please reply with: #syz dup: exact-subject-of-another-report If it's a one-off invalid bug report, please reply with: #syz invalid Note: if the crash happens again, it will cause creation of a new bug report. Note: all commands must start from beginning of the line in the email body. ---------------------------------- Status tracking allows syzbot to (1) keep track of still unfixed bugs (more than half actually gets lost in LKML archives if nobody keeps track of them), (2) be able to ever report similarly looking crashes as new bugs in future, (3) be able to test fixes. The problem is that these rules are mostly not followed. I understand that following these rules is a moderate pain and I am reaching to you to discuss potential solutions for this problem. I've tried hard to come up with a scheme that would eliminate sending these replies altogether, but failed. We can simply to agree to follow the current convention, which is not too hard in the end and sounds like a fair deal -- we give you bugs, you give us fixing commits. Or, we could somehow employ bugzilla.kernel.org for systematic bug tracking. However, I've got an impression that it's mostly unused and not used systematically even when used (e.g. bugs fixed 5 years ago still rest there as open). One idea that was proposed is do it the other way around. Namely, syzbot gives you bug id, and you need to add a tag along the lines of "syzbot-fix: HASH" to the commit. I don't know if kernel community finds this easier to use or not. And dups, invalid bugs and missed tags still needs to be handled in some other way (e.g. the current way). Any other proposals, thoughts, ideas? Thank you