Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Message-ID: <1537804564.23582.113.camel@oracle.com>
Subject: Re: [Announce] LPC 2018: Testing and Fuzzing Microconference
From:   Knut Omang <knut.omang@oracle.com>
To:     Dmitry Vyukov <dvyukov@google.com>,
        Matthew Wilcox <willy@infradead.org>
Cc:     Dhaval Giani <dhaval.giani@gmail.com>,
        Sasha Levin <alexander.levin@microsoft.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        alice.ferrazzi@gmail.com, Kevin Hilman <khilman@baylibre.com>,
        Tim Bird <tbird20d@gmail.com>,
        Laura Abbott <labbott@redhat.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        gustavo.padovan@collabora.co.uk,
        "Carpenter,Dan" <dan.carpenter@oracle.com>
Date:   Mon, 24 Sep 2018 17:56:04 +0200
In-Reply-To: <CACT4Y+ZN0OcTkY0ahvF0CGZv1bbLHGH2aEbCFuQzpC498cNojw@mail.gmail.com>
References: <CAPhKKr_XFzGcBBuQ51w7HjZd_Kq3nTOB6uAFG2vP5JH+Vx_eng@mail.gmail.com>
         <20180922125256.GB14042@bombadil.infradead.org>
         <CACT4Y+ZN0OcTkY0ahvF0CGZv1bbLHGH2aEbCFuQzpC498cNojw@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, 2018-09-24 at 15:42 +0200, Dmitry Vyukov wrote:
> On Sat, Sep 22, 2018 at 2:52 PM, Matthew Wilcox <willy@infradead.org> wrote:
> > On Wed, Sep 19, 2018 at 10:13:15AM -0700, Dhaval Giani wrote:
> >> Sasha and I are pleased to announce the Testing and Fuzzing track at
> >> LPC [ 1 ]. We are planning to continue the discussions from last
> >> year's microconference [2]. Many discussions from the Automated
> >> Testing Summit [3] will also continue, and a final agenda will come up
> >> only soon after that.
> >>
> >> Suggested Topics
> >>
> >> - Syzbot/syzkaller
> >> - ATS
> >> - Distro/stable testing
> >> - kernelci
> >> - kernelci auto bisection
> >> - Unit testing framework
> >>
> >> We look forward to other interesting topics for this microconference
> >> as a reply to this email.
> >
> > I would like to talk about the IDA test suite that was recently merged.
> > See lib/test_ida.c.  It can be built as a module (CONFIG_TEST_IDA=m),
> > built-in (=y) or built in userspace (as part of the radix tree test
> > suite for historical reasons) along with the current kernel code.
> >
> > Being able to build the test suite in userspace allows for much more
> > rapid development.  Building it in kernel space offers testing across
> > a wide range of configurations that I don't have access to and can't
> > necessarily simulate well in userspace.
> >
> > My userspace implementation of kmalloc() simulates failures (in a rather
> > heavy-handed way; every non-GFP_KERNEL allocation fails).  That's rather
> > harder to simulate in kernel space.  I'd like there to be a way for a
> > kernel space test suite to ask for kmalloc failures so that the failure
> > paths can be tested.
> 
> Hi Matthew,
> 
> kmalloc fault injection is already implemented with
> CONFIG_FAULT_INJECTION. For original fault injection, you more-or-less
> ask to fail X% of allocations at random. It's great for testing
> servers for stability, but not so well suited for testing. Recently
> I've added so-called "systematic" fault injection
> (/proc/thread-self/fail-nth) which is perfect for unit testing. It
> allows to ask to fail N-th allocation request in the current task. So
> a unit test for a syscall can do:
> 
> for (i = 0;; i++) {
>   write(/proc/thread-self/fail-nth, i);
>   syscall_under_test();
>   if (read(/proc/thread-self/fail-nth) != 0)
>     break;
> }
> 
> which allows to examine each failure site one-by-one systematically
> (without making random processes on the machine fail too).
> 
> 
> > I think the idea of building parts of the core kernel libraries in
> > userspace and testing them there has greater generality than just the
> > IDA, IDR, XArray & radix tree and might profitably be adopted by other
> > parts of lib/.  The userspace simulation of other parts of the kernel
> > may well need to be extended.
> 
> This would be great. Besides providing faster turn-around time, this
> allow us to finally have something like:
> 
> make test [subsystem]
> 
> which will run all tests for the subsystem locally (in parallel,
> giving final OK/FAIL). This is not possible to in-kernel tests,
> because they require a machine and an image, and it's not possible to
> support everything that people use.

With KTF (https://github.com/oracle/ktf) we are using kprobes to inject 
modifying behaviour (by modifying a function's return value or input).

To cater for the needs for failover testing from user space, our approach 
is what we call "hybrid tests", where parts of the test execute in user land and 
parts of the test execute in the kernel. 

One limitation with config based test functionality is that it requires
rebuilding the kernel to enable the functionality. I think having the tests
available for running even on pre-existing kernels can be of great value.

I agree with Matthew in that there's good time savings to be made to be
able to "lift" code out of the kernel and execute it completely in user space.
The challenge is to be able to compile the kernel code in user land unmodified. 
The pieces under lib/ is probably the easiest ones since they have few 
dependencies on other kernel components.

If interesting, I could say a few words in this context about some work I did 
to allow me to run the Infiniband driver I was working on - and also developed 
the precursor of KTF for 
(https://github.com/oracle/linux-uek/tree/uek4/master/drivers/infiniband/hw/sif)
- entirely in user space under valgrind.
I used it a.o. to test the fairly complex page table management driver code for the on-
board MMU. I have wanted to make it available in a similar way as KTF, just haven't had
the time to get back to it yet.

Knut