Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [ANN] Kselftest integration into Kernel CI
To:     "Bird, Tim" <Tim.Bird@sony.com>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@vger.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Cc:     Kevin Hilman <khilman@baylibre.com>,
        Michael Ellerman <mpe@ellerman.id.au>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Shuah Khan <skhan@linuxfoundation.org>
References: <3b3dc707-7ae7-955b-69fe-b9abe9ae26c5@linuxfoundation.org>
 <MWHPR13MB08954921D3B29D9AA824A2A0FDE80@MWHPR13MB0895.namprd13.prod.outlook.com>
From:   Shuah Khan <skhan@linuxfoundation.org>
Message-ID: <f361d23d-f721-ab2c-6797-3b04f0616567@linuxfoundation.org>
Date:   Fri, 28 Feb 2020 11:43:33 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.4.1
MIME-Version: 1.0
In-Reply-To: <MWHPR13MB08954921D3B29D9AA824A2A0FDE80@MWHPR13MB0895.namprd13.prod.outlook.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 2/28/20 10:50 AM, Bird, Tim wrote:
> 
> 
>> -----Original Message-----
>> From:  Shuah Khan
>>
>> Integrating Kselftest into Kernel CI rings depends on Kselftest build
>> and install framework to support Kernel CI use-cases. I am kicking off
>> an effort to support Kselftest runs in Kernel CI rings. Running these
>> tests in Kernel CI rings will help quality of kernel releases, both
>> stable and mainline.
>>
>> What is required for full support?
>>
>> 1. Cross-compilation & relocatable build support
>> 2. Generates objects in objdir/kselftest without cluttering main objdir
>> 3. Leave source directory clean
>> 4. Installs correctly in objdir/kselftest/kselftest_install and adds
>>      itself to run_kselftest.sh script generated during install.
>>
>> Note that install step is necessary for all files to be installed for
>> run time support.
>>
>> I looked into the current status and identified problems. The work is
>> minimal to add full support. Out of 80+ tests, 7 fail to cross-build
>> and 1 fails to install correctly.
>>
>> List is below:
>>
>> Tests fails to build: bpf, capabilities, kvm, memfd, mqueue, timens, vm
>> Tests fail to install: android (partial failure)
>> Leaves source directory dirty: bpf, seccomp
>>
>> I have patches ready for the following issues:
>>
>> Kselftest objects (test dirs) clutter top level object directory.
>> seccomp_bpf generates objects in the source directory.
>>
>> I created a topic branch to collect all the patches:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/?h=kernelci
>>
>> I am going to start working on build problems. If anybody is
>> interested in helping me with this effort, don't hesitate to
>> contact me. I first priority is fixing build and install and
>> then look into tests that leave the source directory dirty.
> 
> I'm interested in this.  I'd like the same cleanups in order to run
> kselftest in Fuego, and I can try it with additional toolchains
> and boards.  Unfortunately, in terms of running tests, almost all
> the boards in my lab are running old kernels.  So the tests results
> aren't useful for upstream work.  But I can still test
> compilation and install issues, for the kselftest tests themselves.
> 

Testing compilation and install issues is very valuable. This is one
area that hasn't been test coverage compared to running tests. So it
great if you can help with build/install on linux-next to catch
problems in new tests. I am finding that older tests have been stable
and as new tests come in, we tend to miss catching these types of
problems.

Especially cross-builds and installs on arm64 and others.

>>
>> Detailed report can be found here:
>>
>> https://drive.google.com/file/d/11nnWOKIzzOrE4EiucZBn423lzSU_eNNv/view?usp=sharing
> 
> Is there anything you'd like me to look at specifically?  Do you want me to start
> at the bottom of the list and work up?  I could look at 'vm' or 'timens'.
> 

Yes you can start with vm and timens.

thanks,
-- Shuah