Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
In-Reply-To: <20180119014803.n75l5vrxlpifm3sc@wfg-t540p.sh.intel.com>
References: <CACT4Y+bvA6hrVT5u-Cy_zJLNTM5SYf9kt+sXYmfq13rPGfiqFQ@mail.gmail.com>
 <CABXOdTf+CoB6EkhG1yPtB4NNtohbex=ebHLX7YW5TyQQGn9byw@mail.gmail.com>
 <CACT4Y+aGMoobn069+Lq1BT2YGqi9qYY9vHFtiXT2DLsJ5ZUh9Q@mail.gmail.com> <20180119014803.n75l5vrxlpifm3sc@wfg-t540p.sh.intel.com>
From:   Dmitry Vyukov <dvyukov@google.com>
Date:   Mon, 22 Jan 2018 14:34:30 +0100
Message-ID: <CACT4Y+ZvGAX0V0B1W8cZtWqEdXdD0AE-Pym_988tL3neu1Xu5w@mail.gmail.com>
Subject: Re: what trees/branches to test on syzbot
To:     Fengguang Wu <fengguang.wu@intel.com>
Cc:     Guenter Roeck <groeck@google.com>,
        LKML <linux-kernel@vger.kernel.org>,
        "Theodore Ts'o" <tytso@mit.edu>,
        "Eric W. Biederman" <ebiederm@xmission.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        syzkaller <syzkaller@googlegroups.com>,
        Stephen Rothwell <sfr@canb.auug.org.au>,
        David Miller <davem@davemloft.net>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, Jan 19, 2018 at 2:48 AM, Fengguang Wu <fengguang.wu@intel.com> wrote:
> Hi Dmitry,
>
>
> On Tue, Jan 16, 2018 at 10:58:51AM +0100, Dmitry Vyukov wrote:
>>
>> On Tue, Jan 16, 2018 at 10:45 AM, Guenter Roeck <groeck@google.com> wrote:
>>>
>>> On Mon, Jan 15, 2018 at 11:51 PM, Dmitry Vyukov <dvyukov@google.com>
>>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> Several people proposed that linux-next should not be tested on
>>>> syzbot. While some people suggested that it needs to test as many
>>>> trees as possible. I've initially included linux-next as it is a
>>>> staging area before upstream tree, with the intention that patches are
>>>> _tested_ there, is they are not tested there, bugs enter upstream
>>>> tree. And then it takes much longer to get fix into other trees.
>>>>
>>>> So the question is: what trees/branches should be tested? Preferably
>>>> in priority order as syzbot can't test all of them.
>>>>
>>>
>>> I always thought that -next existed specifically to give people a
>>> chance to test the code in it. Maybe the question is where to report
>>> the test results ?
>>
>>
>> FTR, from Guenter on another thread:
>>
>>> Interesting. Assuming that refers to linux-next, not linux-net, that
>>> may explain why linux-next tends to deteriorate. I wonder if I should
>>> drop it from my testing as well. I'll be happy to follow whatever the
>>> result of this exchange is and do the same.
>>
>>
>> If we agree on some list of important branches, and what branches
>> specifically should not be tested with automatic reporting, I think it
>> will benefit everybody.
>> +Fengguang, can you please share your list and rationale behind it?
>
>
> 0-day aims to aggressively test as much tree and branches as possible,
> including various developer trees, maintainer, linux-next, mainline and
> stable trees. Here are the complete list of 800+ trees we monitored:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git/tree/repo/linux
>
> The rationale is obvious. IMHO what really matters here is about
> capability rather than rationale: that policy heavily relies on the
> fundamental capability of auto bisecting. Once regressions are
> bisected, we know the owners of problem to auto send report to, ie.
> the first bad commit's author and committer.
>
> For the bugs that cannot be bisected, they tend to be old ones and
> we report more often on mainline tree than linux-next.

Thanks for the info, Fengguang.

Bisecting is something we need to syzbot in future. However about 50%
of syzbot bugs are due to races and are somewhat difficult to bisect
reliably.