Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 16 Nov 2022 16:02:23 +0000
From:   Mark Brown <broonie@kernel.org>
To:     Naresh Kamboju <naresh.kamboju@linaro.org>
Cc:     Kees Cook <keescook@chromium.org>,
        open list <linux-kernel@vger.kernel.org>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@vger.kernel.org>, lkft-triage@lists.linaro.org,
        Shuah Khan <shuah@kernel.org>,
        Anders Roxell <anders.roxell@linaro.org>,
        Remi Duraffort <remi.duraffort@linaro.org>
Subject: Re: kselftest: Bad test result: from results parsing in LAVA
Message-ID: <Y3UJj6v8xv2rvMp5@sirena.org.uk>
References: <CA+G9fYu0p4cFdkrkQyV4OC-iPpY18asV0VTP=O_sysLNaJJAQw@mail.gmail.com>
 <Y3TlVAgo9XSWKm1h@sirena.org.uk>
 <CA+G9fYuckvzw_t4iSQ5Kj_FvRb-bfWaT66DazEejrKXePaDGuA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
        protocol="application/pgp-signature"; boundary="QuIAI3jeL4BDw//t"
Content-Disposition: inline
In-Reply-To: <CA+G9fYuckvzw_t4iSQ5Kj_FvRb-bfWaT66DazEejrKXePaDGuA@mail.gmail.com>
Precedence: bulk


--QuIAI3jeL4BDw//t
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Nov 16, 2022 at 07:36:10PM +0530, Naresh Kamboju wrote:
> On Wed, 16 Nov 2022 at 18:57, Mark Brown <broonie@kernel.org> wrote:
> > On Wed, Nov 16, 2022 at 05:46:33PM +0530, Naresh Kamboju wrote:

> > > Test results parser showing =E2=80=9CBad test results: =E2=80=9C,

> > When reporting an issue can you please try to provide some
> > analysis which goes beyond the level of "I saw an error message"
> > - for example here it's hard to tell if you think you're seeing
> > an issue somewhere in your test automation system or if you're
> > trying to report something in the tests.

> Let me add more information about this,

> Kees Cook, has done the work of a kselftest results parser in perl
> which is in the test-definitions repository. which was working well
> for two years now. please refer to the below commit log and link to
> the kselftest test-definitions [3].

> The new test cases output is not coping up with the old results parser
> and both KernelCI [1] and LKFT [2] using LAVA have noticed.
> Kselftest results parser problem [4].

I'm still not clear if you believe there is an issue in the tests
or in your test infrastructure here.  I can't identify any
problem with the test output, everything appears to be within the
KTAP spec:

   https://www.kernel.org/doc/html/latest/dev-tools/ktap.html#test-case-res=
ult-lines

My best guess is that either this script which the infrastructure
is using or something else that uses the results of the script is
broken when the test description includes spaces.  The KTAP spec
says that each test case result is a line in the form:

   <result> <number> [<description>][ # [<directive>] [<diagnostic data>]]

where

   The description is a description of the test, generally the
   name of the test, and can be any string of words (can=E2=80=99t include =
#).
   The description is optional, but recommended.

ie, anything after the space following the test number up to a #
or the end of the line is a valid test description (it's not the
intent but it does appear that the # needs to have a space before
it).  You could argue that non-alphabetic characters are out of
spec for the description since any separators between words
aren't themselves words but realistically that'd invalidate a
fairly large subset of the selftests which probably isn't
constructive.

--QuIAI3jeL4BDw//t
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAABCgAdFiEEreZoqmdXGLWf4p/qJNaLcl1Uh9AFAmN1CY8ACgkQJNaLcl1U
h9D+Hwf/WcLRCEFjVXwxjSQevEenU/4QXRyju/vgjjnV5cmJ973j+KnguN17ofSy
bwcuSBrmx1AhtsCSg5qT+DSpBCQ89rBEA2qG07nn4qknOP6ISZCJ18rOWNjlHkIL
Fa9IbAsPRlvDIdVd0R8IZpT3k+A0CpLPr6Mv1JCLGeZM2wYfOu8TQXTsvgYgCH6I
nslVd9O4QrQdKgweQ/HZkdJES+mzFhgMkg4d+ZEzRfn0c+/WEz2vREojyLnfwjkx
7sUeelLSGDQJvAMX0oJE+r7wRkKr4+t/OiAFXkfNucm/Yam8HLBg5yxwgZ5EVCx5
TtcsIv1230xO/vKLsLJ1C2gi3UymUA==
=4vqB
-----END PGP SIGNATURE-----

--QuIAI3jeL4BDw//t--