Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH v2 1/1] Documentation: dev-tools: clarify KTAP
 specification wording
From:   Frank Rowand <frowand.list@gmail.com>
To:     Jonathan Corbet <corbet@lwn.net>, David Gow <davidgow@google.com>,
        Shuah Khan <skhan@linuxfoundation.org>,
        Kees Cook <keescook@chromium.org>, Rae Moar <rmoar@google.com>,
        Tim.Bird@sony.com, Brendan Higgins <brendanhiggins@google.com>
Cc:     rmr167@gmail.com, guillaume.tucker@collabora.com,
        dlatypov@google.com, kernelci@groups.io,
        kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org,
        linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org
References: <20220207162813.3091899-1-frowand.list@gmail.com>
 <e795ad1a-ddb1-f0af-4a74-7c2c6a0a949a@gmail.com>
Message-ID: <0de1426c-5b5d-9160-cd5c-00185fa516f5@gmail.com>
Date:   Mon, 7 Feb 2022 10:55:39 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.14.0
MIME-Version: 1.0
In-Reply-To: <e795ad1a-ddb1-f0af-4a74-7c2c6a0a949a@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 2/7/22 10:38 AM, Frank Rowand wrote:
> On 2/7/22 10:28 AM, frowand.list@gmail.com wrote:
>> From: Frank Rowand <frank.rowand@sony.com>
>>
>> Add the spec version to the title line.
>>
>> Explain likely source of "Unknown lines".
>>
>> "Unknown lines" in nested tests are optionally indented.
>>
>> Add "Unknown lines" items to differences between TAP & KTAP list
>>
>> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
>> ---
> 
> Please do not apply this version (v2).  I was working off the KTAP specification
                                                                ^^^^ TAP
> from a different github repo than I should have been referencing.

OK to proceed with this version of the patch.  I was referencing the proposed
TAP version 14, which is in a different github repo than the TAP version 13
specification.  Everything in this patch is consistent with TAP version 13,
as well as the proposed TAP version 14.

-Frank

> 
> v3 upcoming.
> 
> -Frank
> 
>>
>> Changes since version 1
>>   - Explain likely source of "Unknown lines"
>>   - "Unknown line" in nested tests are optionally indented
>>   - Add "Unknown lines" items to differences between TAP & KTAP list
>>
>>  Documentation/dev-tools/ktap.rst | 24 ++++++++++++++++++------
>>  1 file changed, 18 insertions(+), 6 deletions(-)
>>
>> diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
>> index 878530cb9c27..9c40c94d3f12 100644
>> --- a/Documentation/dev-tools/ktap.rst
>> +++ b/Documentation/dev-tools/ktap.rst
>> @@ -1,8 +1,8 @@
>>  .. SPDX-License-Identifier: GPL-2.0
>>  
>> -========================================
>> -The Kernel Test Anything Protocol (KTAP)
>> -========================================
>> +===================================================
>> +The Kernel Test Anything Protocol (KTAP), version 1
>> +===================================================
>>  
>>  TAP, or the Test Anything Protocol is a format for specifying test results used
>>  by a number of projects. It's website and specification are found at this `link
>> @@ -174,6 +174,13 @@ There may be lines within KTAP output that do not follow the format of one of
>>  the four formats for lines described above. This is allowed, however, they will
>>  not influence the status of the tests.
>>  
>> +This is an important difference from TAP.  Kernel tests may print messages
>> +to the system console or a log file.  Both of these destinations may contain
>> +messages either from unrelated kernel or userspace activity, or kernel
>> +messages from non-test code that is invoked by the test.  The kernel code
>> +invoked by the test likely is not aware that a test is in progress and
>> +thus can not print the message as a diagnostic message.
>> +
>>  Nested tests
>>  ------------
>>  
>> @@ -186,10 +193,13 @@ starting with another KTAP version line and test plan, and end with the overall
>>  result. If one of the subtests fail, for example, the parent test should also
>>  fail.
>>  
>> -Additionally, all result lines in a subtest should be indented. One level of
>> +Additionally, all lines in a subtest should be indented. One level of
>>  indentation is two spaces: "  ". The indentation should begin at the version
>>  line and should end before the parent test's result line.
>>  
>> +"Unknown lines" are not considered to be lines in a subtest and thus are
>> +allowed to be either indented or not indented.
>> +
>>  An example of a test with two nested subtests:
>>  
>>  .. code-block::
>> @@ -225,9 +235,11 @@ Major differences between TAP and KTAP
>>  --------------------------------------
>>  
>>  Note the major differences between the TAP and KTAP specification:
>> -- yaml and json are not recommended in diagnostic messages
>> -- TODO directive not recognized
>> +- yaml and json are not recommended in KTAP diagnostic messages
>> +- TODO directive not recognized in KTAP
>>  - KTAP allows for an arbitrary number of tests to be nested
>> +- TAP includes "Unknown lines" in the category of "Anything else"
>> +- TAP says "Unknown lines" are "incorrect; KTAP allows "Unknown lines"
>>  
>>  The TAP14 specification does permit nested tests, but instead of using another
>>  nested version line, uses a line of the form
>>
>