Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH v2] Add kernel config option for fuzz testing.
To:     Dmitry Vyukov <dvyukov@google.com>
Cc:     Jiri Slaby <jslaby@suse.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Matthew Garrett <mjg59@google.com>,
        Andi Kleen <ak@linux.intel.com>,
        "Theodore Y . Ts'o" <tytso@mit.edu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Petr Mladek <pmladek@suse.com>,
        Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
        Arnd Bergmann <arnd@arndb.de>,
        LKML <linux-kernel@vger.kernel.org>
References: <20200307135822.3894-1-penguin-kernel@I-love.SAKURA.ne.jp>
 <6f2e27de-c820-7de3-447d-cd9f7c650add@suse.com>
 <20200308065258.GE3983392@kroah.com>
 <3e9f47f7-a6c1-7cec-a84f-e621ae5426be@suse.com>
 <CACT4Y+a6KExbggs4mg8pvoD554PcDqQNW4sM15X-tc=YONCzYw@mail.gmail.com>
 <7728c978-d359-227f-0f3e-f975c45ca218@i-love.sakura.ne.jp>
 <CACT4Y+YEYYNv-=AgNthQvLkeK-8uEeRs4XEuRqphfZBhc2hc5g@mail.gmail.com>
From:   Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Message-ID: <55c906c8-ca70-9d1b-a90f-49660773856b@i-love.sakura.ne.jp>
Date:   Thu, 26 Mar 2020 20:10:01 +0900
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:68.0) Gecko/20100101
 Thunderbird/68.6.0
MIME-Version: 1.0
In-Reply-To: <CACT4Y+YEYYNv-=AgNthQvLkeK-8uEeRs4XEuRqphfZBhc2hc5g@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 2020/03/24 19:37, Dmitry Vyukov wrote:
>> So, we have three questions.
>>
>> Q1: Can we agree with adding build-time branching (i.e. kernel config options) ?
>>
>>     I fear bugs (e.g. unexpectedly overwrting flag variables) in run-time
>>     branching mechanisms. Build-time branching mechanisms cannot have such bugs.
> 
> My vote is for build config. It's simplest to configure (every testing
> system should already have control over config) and most reliable
> (e.g. fuzzer figures out a way to disable a runtime option).

Right. For example, console loglevel has been unexpectedly changed via syscall
arguments. For another example, /dev/mem has been unexpectedly accessed using
mount operation ( https://github.com/google/syzkaller/issues/1436 ). Providing
an interface (e.g. debugfs files) for branching after boot has a risk of
unexpectedly overwriting flag variables.

Since it seems that there is no objection to Q1, I think that we can go with
build-time branching.

>> Q3: If we can agree with kernel config options but we can't start with single
>>     (or fewer) kernel config option, can we agree with adding another kernel
>>     config option which selects all options (e.g.
>>     ) ?
> 
> I think this option is the best. It both allows fine-grained control
> (and documentation for each switch), and coarse-grained control for
> testing systems.
> We could also add "depends on DEBUG_KERNEL" just to make sure.
> 

OK. But I think that KERNEL_BUILT_FOR_FUZZ_TESTING_SELECT_ALL might fail to
work, for it is possible that a new option was added but that option was not
added to "select" list of KERNEL_BUILT_FOR_FUZZ_TESTING_SELECT_ALL. Also,
there might be cases where a new option was added but that option should not
be selected for some fuzzers (e.g. syzkaller wants to hardcode console loglevel
to foo while fuzzer1 and fuzzer2 want to hardcode console loglevel to bar).
That is, something like

      config KERNEL_BUILT_FOR_FUZZ_TESTING
             bool "Build kernel for fuzz testing"

      config KERNEL_BUILT_FOR_SYZKALLER
             bool "Build kernel for syzkaller testing"
             depends on KERNEL_BUILT_FOR_FUZZ_TESTING
             select KERNEL_DISABLE_FOO1
             select KERNEL_DISABLE_BAR1
             select KERNEL_DISABLE_BUZ1
             select KERNEL_ENABLE_BAR2

      config KERNEL_BUILT_FOR_FUZZER1
             bool "Build kernel for fuzzer1 testing"
             depends on KERNEL_BUILT_FOR_FUZZ_TESTING
             select KERNEL_DISABLE_FOO1
             select KERNEL_DISABLE_BAR1

      config KERNEL_BUILT_FOR_FUZZER2
             bool "Build kernel for fuzzer2 testing"
             depends on KERNEL_BUILT_FOR_FUZZ_TESTING
             select KERNEL_DISABLE_FOO1
             select KERNEL_DISABLE_BUZ1

      config KERNEL_DISABLE_FOO1
             bool "Disable foo1"
             depends on KERNEL_BUILT_FOR_FUZZ_TESTING

      config KERNEL_DISABLE_BAR1
             bool "Disable bar1"
             depends on KERNEL_BUILT_FOR_FUZZ_TESTING

      config KERNEL_DISABLE_BUZ1
             bool "Disable buz1"
             depends on KERNEL_BUILT_FOR_FUZZ_TESTING

      config KERNEL_CHANGE_FOO2
             bool "Change foo2"
             depends on KERNEL_BUILT_FOR_FUZZ_TESTING

      config KERNEL_ENABLE_BAR2
             bool "Enable bar2"
             depends on KERNEL_BUILT_FOR_FUZZ_TESTING

in order to allow each testing system to select what it wants with
"only two options" ("CONFIG_KERNEL_BUILT_FOR_FUZZ_TESTING=y" and one of
"CONFIG_KERNEL_BUILT_FOR_SYZKALLER=y" or "CONFIG_KERNEL_BUILT_FOR_FUZZER1=y"
or "CONFIG_KERNEL_BUILT_FOR_FUZZER2=y").

CONFIG_KERNEL_BUILT_FOR_FUZZ_TESTING option remains there in order to
avoid needlessly prompting choices to users who do not intend to build
for fuzz testing.