Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <197889b6-5773-094c-8699-26843c6519fd@gmail.com>
Date:   Sun, 26 Mar 2023 21:41:50 -0500
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.7.1
Subject: Re: [KTAP V2 PATCH] ktap_v2: add recognized test name line
Content-Language: en-US
To:     Rae Moar <rmoar@google.com>, davidgow@google.com,
        skhan@linuxfoundation.org, keescook@chromium.org,
        Tim.Bird@sony.com, brendanhiggins@google.com
Cc:     corbet@lwn.net, guillaume.tucker@collabora.com,
        dlatypov@google.com, kernelci@lists.linux.dev,
        kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org,
        linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org
References: <20230316225915.494688-1-rmoar@google.com>
From:   Frank Rowand <frowand.list@gmail.com>
In-Reply-To: <20230316225915.494688-1-rmoar@google.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 3/16/23 17:59, Rae Moar wrote:
> Add recognition of the test name line ("# Subtest: <name>") to the KTAP v2
> spec.
> 
> The purpose of this line is to declare the name of a test before its
> results. This functionality is especially useful when trying to parse test
> results incrementally and when interpretting results after a crash.
> 
> This line is already compliant with KTAP v1 as it is interpretted as a
> diagnostic line by parsers. Additionally, the line is currently used by
> KUnit tests and was derived from the TAP 14 spec:
> https://testanything.org/tap-version-14-specification.html.

It is convenient that "# Subtest: <name>" is compatible with v1, but I think
that there is a negative that overrides the convenience.

The "# Subtest: <name>" syntax means that we need to restrict the format of
diagnostic lines, such that "#Subtest:" is an illegal diagnostic, at least
for the line immediately following the Version line.

I think it would be cleaner to modify the Version line syntax to be:

  KTAP version 2 [# <subtest_name>]

I notice that the KTAP Specification version 1 fails to specify the
Version line syntax.  So the Specification would be updated from:

  All KTAP-formatted results begin with a "version line" which specifies which
  version of the (K)TAP standard the result is compliant with.

  For example:
  - "KTAP version 1"
  - "TAP version 13"
  - "TAP version 14"

to:

  The Version line is required and must have the format:

  .. code-block:: none

        KTAP version 2 [# <subtest_name>]

  All KTAP-formatted results begin with a "version line" which specifies which
  version of the (K)TAP standard the result is compliant with.

  For example:
  - "KTAP version 2"
  - "TAP version 13"
  - "TAP version 14"

> 
> Recognition of this line would create an accepted way for different test
> frameworks to declare the name of a test before its results.
> 
> The proposed location for this line is between the version line and the
> test plan line. This location ensures that the line would not be
> accidentally parsed as a subtest's diagnostic lines. Note this proposed
> location would be a slight differentiation from KTAP v1.
> 
> Example of test name line:
> 
>  KTAP version 2
>  # Subtest: main_test
>  1..1
>    KTAP version 2
>    # Subtest: sub_test
>    1..2
>    ok 1 test_1
>    ok 2 test_2
>  ok 1 sub_test
> 
> Here is a link to a version of the KUnit parser that is able to parse the
> test name line for KTAP version 2. Note this includes a test name line for
> the main level of KTAP.
> 
> Link: https://kunit-review.googlesource.com/c/linux/+/5709
> 
> Signed-off-by: Rae Moar <rmoar@google.com>
> ---
> 
> This is a RFC. I would like to know what people think and use this as a
> platform for discussion on KTAP v2.
> 
> Note: this patch is based on Frank's ktap_spec_version_2 branch.
> 
>  Documentation/dev-tools/ktap.rst | 19 ++++++++++++++-----
>  1 file changed, 14 insertions(+), 5 deletions(-)
> 
> diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
> index ff77f4aaa6ef..9c7ed66d9f77 100644
> --- a/Documentation/dev-tools/ktap.rst
> +++ b/Documentation/dev-tools/ktap.rst
> @@ -28,8 +28,7 @@ KTAP output is built from four different types of lines:
>  In general, valid KTAP output should also form valid TAP output, but some
>  information, in particular nested test results, may be lost. Also note that
>  there is a stagnant draft specification for TAP14, KTAP diverges from this in
> -a couple of places (notably the "Subtest" header), which are described where
> -relevant later in this document.
> +a couple of places, which are described where relevant later in this document.
>  
>  Version lines
>  -------------
> @@ -44,8 +43,8 @@ For example:
>  - "TAP version 14"
>  
>  Note that, in KTAP, subtests also begin with a version line, which denotes the

> -start of the nested test results. This differs from TAP14, which uses a
> -separate "Subtest" line.

^^^^ This is an error in the KTAP Specification version 1.  TAP14 allows the case
of "Bare Subtests", which would be the equivalent of the KTAP v1 method.

> +start of the nested test results. This differs from TAP14, which uses only a
> +"Subtest" line.
>  
>  While, going forward, "KTAP version 2" should be used by compliant tests, it
>  is expected that most parsers and other tooling will accept the other versions
> @@ -166,6 +165,12 @@ even if they do not start with a "#": this is to capture any other useful
>  kernel output which may help debug the test. It is nevertheless recommended
>  that tests always prefix any diagnostic output they have with a "#" character.
>  
> +One recognized diagnostic line is the "# Subtest: <name>" line. This line
> +is used to declare the name of a test before subtest results are printed. This
> +is helpful for parsing and for providing context during crashes. As a rule,
> +this line is placed after the version line and before the plan line. Note
> +this line can be used for the main test, as well as subtests.
> +
>  Unknown lines
>  -------------
>  
> @@ -206,6 +211,7 @@ An example of a test with two nested subtests:
>  	KTAP version 2
>  	1..1
>  	  KTAP version 2
> +	  # Subtest: example
>  	  1..2
>  	  ok 1 test_1
>  	  not ok 2 test_2
> @@ -219,6 +225,7 @@ An example format with multiple levels of nested testing:
>  	KTAP version 2
>  	1..2
>  	  KTAP version 2
> +	  # Subtest: example_test_1
>  	  1..2
>  	    KTAP version 2
>  	    1..2
> @@ -245,7 +252,7 @@ allows an arbitrary number of tests to be nested     no         yes
>  
>  The TAP14 specification does permit nested tests, but instead of using another
>  nested version line, uses a line of the form
> -"Subtest: <name>" where <name> is the name of the parent test.
> +"Subtest: <name>" where <name> is the name of the parent test as discussed above.
>  
>  Example KTAP output
>  --------------------
> @@ -254,6 +261,7 @@ Example KTAP output
>  	KTAP version 2
>  	1..1
>  	  KTAP version 2
> +	  # Subtest: main_test
>  	  1..3
>  	    KTAP version 2
>  	    1..1
> @@ -266,6 +274,7 @@ Example KTAP output
>  	    ok 2 test_2
>  	  ok 2 example_test_2
>  	    KTAP version 2
> +		# Subtest: example_test_3
>  	    1..3
>  	    ok 1 test_1
>  	    # test_2: FAIL
> 
> base-commit: 906f02e42adfbd5ae70d328ee71656ecb602aaf5