Received-SPF: pass (google.com: domain of linux-kernel+bounces-155449-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <02c3926c-84fe-4526-a746-6964f5290c14@collabora.com>
Date: Tue, 23 Apr 2024 20:17:18 +0500
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Cc: Muhammad Usama Anjum <usama.anjum@collabora.com>,
 automated-testing@lists.yoctoproject.org,
 "kernel@collabora.com" <kernel@collabora.com>, kernelci@lists.linux.dev,
 open list <linux-kernel@vger.kernel.org>
Content-Language: en-US
To: Shuah Khan <shuah@kernel.org>, gustavo.padovan@collabora.com,
 "open list : KERNEL SELFTEST FRAMEWORK" <linux-kselftest@vger.kernel.org>,
 Laura Nao <laura.nao@collabora.com>,
 Shreeya Patel <shreeya.patel@collabora.com>,
 =?UTF-8?B?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= <nfraprado@collabora.com>,
 Helen Mae Koike Fornazier <helen.koike@collabora.com>
From: Muhammad Usama Anjum <usama.anjum@collabora.com>
Subject: Improving the testing quality of the kernel
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Hi,

I've been working with colleagues at Collabora to improve the testing
quality overall for kselftests since several months. We had identified
following key points to improve:
* Make non-TAP conformant tests, TAP conformant to catch the bugs/failures
in the CI and non-CI environment. Without clear success or failure message,
it is difficult and at times impossible to notice the bugs and which part
of the test or sub-test has failed.
* Add config fragment if it is already isn't present.
* Improve ancient tests to be robust. KernelCI helps a lot in finding
flakiness or non-robust pieces of code by building it in range of
configurations and running it on range of hardware.

As new KernelCI is in works, we are identifying which kselftest suites
could be most suitable to be enabled in the start to keep the noise to a
minimum. The criteria to enable a suite on KernelCI is:
* The test suite is TAP compliant
* The test suite passes in preliminary testing on x86_64 and ARM64 platforms
* The test suite builds fine with Clang in-addition to gcc

To facilitate transparency and track our progress, I am diligently
maintaining a spreadsheet [1] with detailed information about each suite's
status and requirements.

Furthermore, I have commenced exploration into KUnit testing, and initial
results are promising. I plan to delve deeper into KUnit testing and will
provide further updates in the coming days.

Your feedback, insights and collaboration on any aspect of these
initiatives would be highly valuable. We greatly appreciate your
collaboration and support as we continue to enhance the testing quality of
kselftests.

[1]
https://docs.google.com/spreadsheets/d/1XRCgxlY1b74aIOIXQ7qDR-62KTJmuV1ow9st4fpeVhA/edit?usp=sharing

-- 
BR,
Muhammad Usama Anjum