Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20210611035725.1248874-1-davidgow@google.com> <CANpmjNOvLz=71PXSi+LGvKZ+9b_rfY1+wp8HfeyZa8u2QZURcg@mail.gmail.com>
 <8a7e6806-97f5-6f13-5e97-328cf0d3a95d@linuxfoundation.org>
In-Reply-To: <8a7e6806-97f5-6f13-5e97-328cf0d3a95d@linuxfoundation.org>
From:   David Gow <davidgow@google.com>
Date:   Sat, 12 Jun 2021 07:16:49 +0800
Message-ID: <CABVgOSk5r3-m_dfF=tGUDo_hO=HWK=VbO42MHpbdM6n4oXjb+w@mail.gmail.com>
Subject: Re: [PATCH] kunit: Fix result propagation for parameterised tests
To:     Shuah Khan <skhan@linuxfoundation.org>
Cc:     Marco Elver <elver@google.com>,
        Brendan Higgins <brendanhiggins@google.com>,
        Arpitha Raghunandan <98.arpi@gmail.com>,
        KUnit Development <kunit-dev@googlegroups.com>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Sat, Jun 12, 2021 at 1:44 AM Shuah Khan <skhan@linuxfoundation.org> wrote:
>
> On 6/11/21 2:29 AM, Marco Elver wrote:
> > On Fri, 11 Jun 2021 at 05:57, David Gow <davidgow@google.com> wrote:
> >>
> >> When one parameter of a parameterised test failed, its failure would be
> >> propagated to the overall test, but not to the suite result (unless it
> >> was the last parameter).
> >>
> >> This is because test_case->success was being reset to the test->success
> >> result after each parameter was used, so a failing test's result would
> >> be overwritten by a non-failing result. The overall test result was
> >> handled in a third variable, test_result, but this was disacarded after
> >> the status line was printed.
> >>
> >> Instead, just propagate the result after each parameter run.
> >>
> >> Signed-off-by: David Gow <davidgow@google.com>
> >> Fixes: fadb08e7c750 ("kunit: Support for Parameterized Testing")
> >
> > Reviewed-by: Marco Elver <elver@google.com>
> >
> > Would Cc: stable be appropriate?
> >
> > Thanks,
> > -- Marco
> >
> >> ---
> >>
> >> This is fixing quite a serious bug where some test suites would appear
> >> to succeed even if some of their component tests failed. It'd be nice to
> >> get this into kunit-fixes ASAP.
> >>
>
> Will apply this with cc stable.
>

Thanks!

> >> (This will require a rework of some of the skip tests work, for which
> >> I'll send out a new version soon.)
> >>
>
> Thanks for the heads up. I will wait for new version.
>

Thanks: I've sent out v4 which fixes this:
https://lore.kernel.org/linux-kselftest/20210611070802.1318911-1-davidgow@google.com/

It's rebased on top of this patch, so depends on it, and also depends
on the first two patches in the "Do not typecheck binary assertions"
series:
https://lore.kernel.org/linux-kselftest/20210513193204.816681-1-davidgow@google.com/

> thanks,
> -- Shuah