Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com>
 <202006141120.96FF8C5@keescook> <CY4PR13MB11757D57CD441C5CAEC3F257FD9C0@CY4PR13MB1175.namprd13.prod.outlook.com>
 <7161fadb-45ba-c4c0-8bbb-cb47d2dd0265@redhat.com> <dc853d83-649e-b652-819f-4766ea294d78@gmail.com>
In-Reply-To: <dc853d83-649e-b652-819f-4766ea294d78@gmail.com>
From:   David Gow <davidgow@google.com>
Date:   Sat, 20 Jun 2020 14:44:56 +0800
Message-ID: <CABVgOS=qSMY9xP7db4-hkSt71YKyPpJuQv=fqcfzV-kCC1k9qQ@mail.gmail.com>
Subject: Re: RFC - kernel selftest result documentation (KTAP)
To:     Frank Rowand <frowand.list@gmail.com>
Cc:     Paolo Bonzini <pbonzini@redhat.com>,
        "Bird, Tim" <Tim.Bird@sony.com>, Kees Cook <keescook@chromium.org>,
        "shuah@kernel.org" <shuah@kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        Brendan Higgins <brendanhiggins@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Sat, Jun 20, 2020 at 1:58 AM Frank Rowand <frowand.list@gmail.com> wrote:
>
> On 2020-06-16 07:08, Paolo Bonzini wrote:
> > On 15/06/20 21:07, Bird, Tim wrote:

> >>>> Finally,
> >>>>   - Should a SKIP result be 'ok' (TAP13 spec) or 'not ok' (current kselftest practice)?
> >>>> See https://testanything.org/tap-version-13-specification.html
> >>>
> >>> Oh! I totally missed this. Uhm. I think "not ok" makes sense to me "it
> >>> did not run successfully". ... but ... Uhhh ... how do XFAIL and SKIP
> >>> relate? Neither SKIP nor XFAIL count toward failure, though, so both
> >>> should be "ok"? I guess we should change it to "ok".
> >
> > See above for XFAIL.
> >
> > I initially raised the issue with "SKIP" because I have a lot of tests
> > that depend on hardware availability---for example, a test that does not
> > run on some processor kinds (e.g. on AMD, or old Intel)---and for those
> > SKIP should be considered a success.
>
> No, SKIP should not be considered a success.  It should also not be considered
> a failure.  Please do not blur the lines between success, failure, and
> skipped.

I agree that skipped tests should be their own thing, separate from
success and failure, but the way they tend to behave tends to be
closer to a success than a failure.

I guess the important note here is that a suite of tests, some of
which are SKIPped, can be listed as having passed, so long as none of
them failed. So, the rule for "bubbling up" test results is that any
failures cause the parent to fail, the parent is marked as skipped if
_all_ subtests are skipped, and otherwise is marked as having
succeeded. (Reversing the last part: having a suite be marked as
skipped if _any_ of the subtests are skipped also makes sense, and has
its advantages, but anecdotally seems less common in other systems.)

The other really brave thing one could do to break from the TAP
specification would be to add a "skipped" value alongside "ok" and
"not ok", and get rid of the whole "SKIP" directive/comment stuff.
Possibly not worth the departure from the spec, but it would sidestep
part of the problem.


Cheers,
-- David