Message-ID: <155efcdb-2be6-16c4-42bc-37930639060a@gmail.com>
Date:   Wed, 15 Mar 2023 16:45:29 -0500
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.7.1
Subject: Re: [KTAP V2 PATCH] ktap_v2: add skip test result
Content-Language: en-US
To:     Mark Brown <broonie@kernel.org>, kernelci@groups.io,
        rmoar@google.com
Cc:     "Bird, Tim" <Tim.Bird@sony.com>,
        "davidgow@google.com" <davidgow@google.com>,
        "skhan@linuxfoundation.org" <skhan@linuxfoundation.org>,
        "keescook@chromium.org" <keescook@chromium.org>,
        "brendanhiggins@google.com" <brendanhiggins@google.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "guillaume.tucker@collabora.com" <guillaume.tucker@collabora.com>,
        "dlatypov@google.com" <dlatypov@google.com>,
        "kunit-dev@googlegroups.com" <kunit-dev@googlegroups.com>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <20230310222002.3633162-1-rmoar@google.com>
 <BYAPR13MB2503C590A2AE6FEF6BCAC529FDBB9@BYAPR13MB2503.namprd13.prod.outlook.com>
 <CA+GJov5O6hGdjYMXjRd34MEZuyBuukyJCOsS=HeO30h43eLQbQ@mail.gmail.com>
 <4568b302-2a5a-4499-b2f7-12f89c031495@sirena.org.uk>
From:   Frank Rowand <frowand.list@gmail.com>
In-Reply-To: <4568b302-2a5a-4499-b2f7-12f89c031495@sirena.org.uk>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 3/15/23 07:53, Mark Brown wrote:
> On Tue, Mar 14, 2023 at 06:03:59PM -0400, Rae Moar via groups.io wrote:
> 
>> One thing to note on the created churn: I have noticed a proportion of
>> kselftests currently implement skipped tests in a way that does not
>> use the SKIP directive. They use a comment of the format "# [SKIP]"
>> prior to a test result line with no SKIP directive. Thus, in order to
>> reach KTAP compliance the way skip tests are handled would need to be
>> changed in these cases anyways.
> 
> This is the documented way of reporting a skip in KTAP:
> 
>    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/dev-tools/ktap.rst#n97
> 
> TBH I'm finding it really hard to summon much enthusiasm for changing
> this except as part of some other incompatible update - the current
> format isn't ideal but deploying a change would be a bunch of hassle for
> the existing test automation systems.

Yes, there is no need to do a single specification change that results
in incompatibility.  But given the previous discussions there seem to
be plenty of other desired changes that will result in incompatibility.

My desire is to take our time to capture as much of the desired changes
as possible in version 2 of the specification.  And an expectation that
there will not need to be a version 3 of the specification for many years.
I will support not rushing version 2 of the specification so that this
goal can be realized.