Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CA+G9fYu0p4cFdkrkQyV4OC-iPpY18asV0VTP=O_sysLNaJJAQw@mail.gmail.com>
 <Y3TlVAgo9XSWKm1h@sirena.org.uk>
In-Reply-To: <Y3TlVAgo9XSWKm1h@sirena.org.uk>
From:   Naresh Kamboju <naresh.kamboju@linaro.org>
Date:   Wed, 16 Nov 2022 19:36:10 +0530
Message-ID: <CA+G9fYuckvzw_t4iSQ5Kj_FvRb-bfWaT66DazEejrKXePaDGuA@mail.gmail.com>
Subject: Re: kselftest: Bad test result: from results parsing in LAVA
To:     Mark Brown <broonie@kernel.org>, Kees Cook <keescook@chromium.org>
Cc:     open list <linux-kernel@vger.kernel.org>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@vger.kernel.org>, lkft-triage@lists.linaro.org,
        Shuah Khan <shuah@kernel.org>,
        Anders Roxell <anders.roxell@linaro.org>,
        Remi Duraffort <remi.duraffort@linaro.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Wed, 16 Nov 2022 at 18:57, Mark Brown <broonie@kernel.org> wrote:
>
> On Wed, Nov 16, 2022 at 05:46:33PM +0530, Naresh Kamboju wrote:
> > kselftest running on LAVA infrastures provides test results parser
> > from test-definitions repository.
> > which is getting  "Bad test result:".
> >
> > I have noticed this on kernelci [1] and LKFT LAVA instances [2].
> > We need to investigate and change parse_output [3] inside test-definiti=
ons.
> >
> > Report-by: Linux Kernel Functional Testing <lkft@linaro.org>
> >
> > Test results parser showing =E2=80=9CBad test results: =E2=80=9C,
>
> When reporting an issue can you please try to provide some
> analysis which goes beyond the level of "I saw an error message"
> - for example here it's hard to tell if you think you're seeing
> an issue somewhere in your test automation system or if you're
> trying to report something in the tests.


Let me add more information about this,

Kees Cook, has done the work of a kselftest results parser in perl
which is in the test-definitions repository. which was working well
for two years now. please refer to the below commit log and link to
the kselftest test-definitions [3].

The new test cases output is not coping up with the old results parser
and both KernelCI [1] and LKFT [2] using LAVA have noticed.
Kselftest results parser problem [4].

---
Repo: test-definitions
commit id:  8bd338bbcfa5a03efcf1d12e25b5d341d5a29cbc
---
kselftest: Parse kTAP for test results

This stops mangling test output for humans and performs a best-effort
kTAP parsing for the kselftest output (including recursive kTAP output).

Signed-off-by: Kees Cook <keescook@chromium.org>


[1] https://storage.kernelci.org/next/master/next-20221116/arm64/defconfig+=
arm64-chromebook/gcc-10/lab-collabora/kselftest-arm64-mt8173-elm-hana.html#=
L3105
[2] https://linaro.atlassian.net/browse/LKQ-934
[3] https://github.com/Linaro/test-definitions/blob/master/automated/linux/=
kselftest/kselftest.sh#L124
[4] https://lkft.validation.linaro.org/scheduler/job/5729151#L2511

--
Linaro LKFT
https://lkft.linaro.org