Received-SPF: pass (google.com: domain of linux-kernel+bounces-211074-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <7588c2a8-27c3-4b1c-bebd-9c3b099c8b80@collabora.com>
Date: Wed, 12 Jun 2024 12:27:10 +0500
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Cc: Muhammad Usama Anjum <usama.anjum@collabora.com>, kernel@collabora.com,
 linux-kselftest@vger.kernel.org, linux-doc@vger.kernel.org,
 linux-kernel@vger.kernel.org
Subject: Re: [PATCH] selftests: Add information about TAP conformance in tests
To: Shuah Khan <skhan@linuxfoundation.org>, Shuah Khan <shuah@kernel.org>,
 Jonathan Corbet <corbet@lwn.net>
References: <20240610083245.1938001-1-usama.anjum@collabora.com>
 <1f46ef8d-2435-4c73-88b0-f954d685d1a9@linuxfoundation.org>
Content-Language: en-US
From: Muhammad Usama Anjum <usama.anjum@collabora.com>
In-Reply-To: <1f46ef8d-2435-4c73-88b0-f954d685d1a9@linuxfoundation.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

On 6/12/24 1:27 AM, Shuah Khan wrote:
> On 6/10/24 02:32, Muhammad Usama Anjum wrote:
>> Although "TAP" word is being used already in documentation, but it hasn't
>> been defined in informative way for developers that how to write TAP
>> conformant tests and what are the benefits. Write a short brief about it.
>>
>> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
>> ---
>>   Documentation/dev-tools/kselftest.rst | 8 ++++++++
>>   1 file changed, 8 insertions(+)
>>
>> diff --git a/Documentation/dev-tools/kselftest.rst
>> b/Documentation/dev-tools/kselftest.rst
>> index dcf634e411bd9..b579f491f3e97 100644
>> --- a/Documentation/dev-tools/kselftest.rst
>> +++ b/Documentation/dev-tools/kselftest.rst
>> @@ -228,6 +228,14 @@ In general, the rules for selftests are
>>    * Don't cause the top-level "make run_tests" to fail if your feature is
>>      unconfigured.
>>   + * The output of tests must conform to the TAP standard to ensure high
>> +   testing quality and to capture failures/errors with specific details.
>> +   The kselftest.h and kselftest_harness.h headers provide wrappers for
>> +   outputting test results such as pass, fail, or skip etc. These wrappers
>> +   should be used instead of reinventing the wheel or using raw printf and
> 
> When there are multiple tests supported by a test suite, it doesn't
> make sense to convert every single print statement. That is the reason
> why we have the run_tests wrapper that prints the summary in TAP format.
> 
> I would rephrase this to say "should be used for pass, fail, exit, and
> skip messages".
Makes sense. I'll update.

> 
> This rule is applicable to only the tests that are in the selftests Makefile
> and included in the default run.
> 
> There is no need to convert every single print statement in tests.
> 
>> +   exit statements. CI systems can easily parse TAP output messages to
>> +   detect test failures.
> 
> I would rather see people spending energy writing new tests and fixing
> warnings. TAP conversions need to be focused if they add value.
Some tests don't print pass/fail and just return status by exit value.
Adding pass/fail statements there really add value by exposing what went
wrong. So it can be reproduced in debug environment to fix it.

> 
> thanks,
> -- Shuah
> 

-- 
BR,
Muhammad Usama Anjum