Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20200817043028.76502-1-98.arpi@gmail.com> <f408efbd-10f7-f1dd-9baa-0f1233cafffc@rasmusvillemoes.dk>
 <20200821113710.GA26290@alley> <4e26f696-3b50-d781-00fd-7fc6fdc2c3eb@rasmusvillemoes.dk>
 <CAFd5g45VABAd-Z3A39ORJ-VJM0oz=YQDjE=4C_jjw1LCzh67iw@mail.gmail.com>
In-Reply-To: <CAFd5g45VABAd-Z3A39ORJ-VJM0oz=YQDjE=4C_jjw1LCzh67iw@mail.gmail.com>
From:   Brendan Higgins <brendanhiggins@google.com>
Date:   Mon, 12 Oct 2020 13:48:52 -0700
Message-ID: <CAFd5g47wrVks-+rfPp=3qm+RYm9f+qvStw8TVwH_Eh8wh_Fzng@mail.gmail.com>
Subject: Re: [PATCH] lib: Convert test_printf.c to KUnit
To:     Rasmus Villemoes <linux@rasmusvillemoes.dk>
Cc:     Petr Mladek <pmladek@suse.com>,
        Arpitha Raghunandan <98.arpi@gmail.com>,
        Shuah Khan <skhan@linuxfoundation.org>,
        Andy Shevchenko <andriy.shevchenko@linux.intel.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
        KUnit Development <kunit-dev@googlegroups.com>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@vger.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        linux-kernel-mentees@lists.linuxfoundation.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Mon, Oct 12, 2020 at 1:13 PM Brendan Higgins
<brendanhiggins@google.com> wrote:
>
> On Fri, Aug 21, 2020 at 5:19 AM Rasmus Villemoes
> <linux@rasmusvillemoes.dk> wrote:
>
> Sorry about the late reply. I saw activity on this before and thought
> it was under control. I only saw the unresolved state now looking
> through patchwork.
>
> > On 21/08/2020 13.37, Petr Mladek wrote:
> > > On Mon 2020-08-17 09:06:32, Rasmus Villemoes wrote:
> > >> On 17/08/2020 06.30, Arpitha Raghunandan wrote:
> > >>> Converts test lib/test_printf.c to KUnit.
> > >>> More information about KUnit can be found at
> > >>> https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html.
> > >>> KUnit provides a common framework for unit tests in the kernel.
> > >>
> > >> So I can continue to build a kernel with some appropriate CONFIG set=
 to
> > >> y, boot it under virt-me, run dmesg and see if I broke printf? That'=
s
> > >> what I do now, and I don't want to have to start using some enterpri=
sy
> > >> framework.
> > >
> > > I had the same concern. I have tried it.
> >
> > Thanks for doing that and reporting the results.
> >
> > >     #> modprobe printf_kunit
> > >
> > > produced the following in dmesg:
> > >
> > > [   60.931175] printf_kunit: module verification failed: signature an=
d/or required key missing - tainting kernel
> > > [   60.942209] TAP version 14
> > > [   60.945197]     # Subtest: printf-kunit-test
> > > [   60.945200]     1..1
> > > [   60.951092]     ok 1 - selftest
> > > [   60.953414] ok 1 - printf-kunit-test
> > >
> > > I could live with the above. Then I tried to break a test by the foll=
owing change:
> > >
> > >
> > > diff --git a/lib/printf_kunit.c b/lib/printf_kunit.c
> > > index 68ac5f9b8d28..1689dadd70a3 100644
> > > --- a/lib/printf_kunit.c
> > > +++ b/lib/printf_kunit.c
> > > @@ -395,7 +395,7 @@ ip4(struct kunit *kunittest)
> > >         sa.sin_port =3D cpu_to_be16(12345);
> > >         sa.sin_addr.s_addr =3D cpu_to_be32(0x7f000001);
> > >
> > > -       test(kunittest, "127.000.000.001|127.0.0.1", "%pi4|%pI4", &sa=
.sin_addr, &sa.sin_addr);
> > > +       test(kunittest, "127-000.000.001|127.0.0.1", "%pi4|%pI4", &sa=
.sin_addr, &sa.sin_addr);
> > >         test(kunittest, "127.000.000.001|127.0.0.1", "%piS|%pIS", &sa=
, &sa);
> > >         sa.sin_addr.s_addr =3D cpu_to_be32(0x01020304);
> > >         test(kunittest, "001.002.003.004:12345|1.2.3.4:12345", "%piSp=
|%pISp", &sa, &sa);
> > >
> > >
> > > It produced::
> > >
> > > [   56.786858] TAP version 14
> > > [   56.787493]     # Subtest: printf-kunit-test
> > > [   56.787494]     1..1
> > > [   56.788612]     # selftest: EXPECTATION FAILED at lib/printf_kunit=
.c:76
> > >                    Expected memcmp(test_buffer, expect, written) =3D=
=3D 0, but
> > >                        memcmp(test_buffer, expect, written) =3D=3D 1
> > >                        0 =3D=3D 0
> > >                vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.0=
00.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
> > > [   56.795433]     # selftest: EXPECTATION FAILED at lib/printf_kunit=
.c:76
> > >                    Expected memcmp(test_buffer, expect, written) =3D=
=3D 0, but
> > >                        memcmp(test_buffer, expect, written) =3D=3D 1
> > >                        0 =3D=3D 0
> > >                vsnprintf(buf, 20, "%pi4|%pI4", ...) wrote '127.000.00=
0.001|127', expected '127-000.000.001|127'
> > > [   56.800909]     # selftest: EXPECTATION FAILED at lib/printf_kunit=
.c:108
> > >                    Expected memcmp(p, expect, elen+1) =3D=3D 0, but
> > >                        memcmp(p, expect, elen+1) =3D=3D 1
> > >                        0 =3D=3D 0
> > >                kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.00=
0.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
> > > [   56.806497]     not ok 1 - selftest
> > > [   56.806497] not ok 1 - printf-kunit-test
> > >
> > > while the original code would have written the following error messag=
es:
> > >
> > > [   95.859225] test_printf: loaded.
> > > [   95.860031] test_printf: vsnprintf(buf, 256, "%pi4|%pI4", ...) wro=
te '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
> > > [   95.862630] test_printf: vsnprintf(buf, 6, "%pi4|%pI4", ...) wrote=
 '127.0', expected '127-0'
> > > [   95.864118] test_printf: kvasprintf(..., "%pi4|%pI4", ...) returne=
d '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
> > > [   95.866589] test_printf: failed 3 out of 388 tests
> > >
> > >
> > > Even the error output is acceptable for me.
> >
> > Urgh. Yeah, perhaps, but the original is much more readable; it really
> > doesn't matter that a memcmp() fails to compare equal to 0, that's
> > merely how we figured out that the output was wrong.
>
> We can go back to the original error reporting format, just as long as
> you don't mind the "ok" lines interspersed throughout (this is part of
> an attempt to standardize around the KTAP reporting format[1].
>
> > I am just curious why
> > > the 2nd failure is different:
> > >
> > >    + original code: vsnprintf(buf, 6, "%pi4|%pI4", ...) wrote '127.0'=
, expected '127-0'
> > >    + kunit code: vsnprintf(buf, 20, "%pi4|%pI4", ...) wrote '127.000.=
000.001|127', expected '127-000.000.001|127'
> >
> > That's by design. If you read the code, there's a comment that says we
> > do every test case four times: With a buffer that is large enough to do
> > the whole output, with a 0 size buffer (that's essential to allowing
> > kasprintf to know how much to allocate),  with kvasprintf - but also
> > with a buffer size that's guaranteed to ensure the output gets truncate=
d
> > somewhere. And that size is chosen randomly - I guess one could test
> > every single buffer size between 0 and elen+1, but that's overkill.
> >
> > Now I should probably have made the tests deterministic in the sense of
> > getting a random seed for a PRNG, printing that seed and allowing a
> > module parameter to set the seed in order to repeat the exact same
> > tests. But so far I haven't really seen any bugs caught by test_printf
> > which would have been easier to fix with that.
> >
> > The reason I added that "chop it off somewhere randomly" was, IIRC, due
> > to some %p extensions that behaved rather weirdly depending on whether
> > there was enough room left or not, but I fixed those bugs before
> > creating test_printf (and they were in turn the reason for creating
> > test_printf). See for example 41416f2330, where %pE at the beginning of
> > the format string would work ok, but if anything preceded it and the
> > buffer was too small we'd crash.
> >
> > >
> > > I am also a bit scared by the following note at
> > > https://www.kernel.org/doc/html/latest/dev-tools/kunit/start.html#run=
ning-tests-without-the-kunit-wrapper
> > >
> > >    "KUnit is not designed for use in a production system, and it=E2=
=80=99s
> > >    possible that tests may reduce the stability or security of the
> > >    system."
> > >
> > > What does it mean thay it might reduce stability or security?
> > > Is it because the tests might cause problems?
> > > Or because the kunit framework modifies functionality of the running
> > > system all the time?
>
> Oh yeah, that's just because we are afraid that tests might cause
> problems. KUnit by itself does nothing to affect the stability or
> security of the system.

And I forgot the link to KTAP[1]. I am really batting a thousand here.

[1] https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC81=
63FD830@CY4PR13MB1175.namprd13.prod.outlook.com/