Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com>
 <20200616204817.GA212825@google.com> <CY4PR13MB1175A71AE3937C0786721ABFFD9D0@CY4PR13MB1175.namprd13.prod.outlook.com>
 <202006161703.B2E51605@keescook> <CY4PR13MB1175DCC4066FC0839A6B2E84FD9A0@CY4PR13MB1175.namprd13.prod.outlook.com>
 <202006162032.9BF6F8F4E@keescook> <CABVgOSn48gw3pMSCccmF8FK=3UPF+Spe47AiDQcMiXkk3k4yqQ@mail.gmail.com>
In-Reply-To: <CABVgOSn48gw3pMSCccmF8FK=3UPF+Spe47AiDQcMiXkk3k4yqQ@mail.gmail.com>
From:   Brendan Higgins <brendanhiggins@google.com>
Date:   Fri, 19 Jun 2020 12:44:53 -0700
Message-ID: <CAFd5g4610GBAcyY9DDoQjoCVC0AG-=5FYJAZnS_5+L-aU7e4XA@mail.gmail.com>
Subject: Re: RFC - kernel selftest result documentation (KTAP)
To:     David Gow <davidgow@google.com>
Cc:     Kees Cook <keescook@chromium.org>, "Bird, Tim" <Tim.Bird@sony.com>,
        "shuah@kernel.org" <shuah@kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Jun 16, 2020 at 9:06 PM David Gow <davidgow@google.com> wrote:
>
> On Wed, Jun 17, 2020 at 11:36 AM Kees Cook <keescook@chromium.org> wrote:
> >
> > On Wed, Jun 17, 2020 at 02:30:45AM +0000, Bird, Tim wrote:
> > > Agreed.  You only need machine-parsable data if you expect the CI
> > > system to do something more with the data than just present it.
> > > What that would be, that would be common for all tests (or at least
> > > many test), is unclear.  Maybe there are patterns in the diagnostic
> > > data that could lead to higher-level analysis, or even automated
> > > fixes, that don't become apparent if the data is unstructured.  But
> > > it's hard to know until you have lots of data.  I think just getting
> > > the other things consistent is a good priority right now.
> >
> > Yeah. I think the main place for this is performance analysis, but I
> > think that's a separate system entirely. TAP is really strictly yes/no,
> > where as performance analysis a whole other thing. The only other thing
> > I can think of is some kind of feature analysis, but that would be built
> > out of the standard yes/no output. i.e. if I create a test that checks
> > for specific security mitigation features (*cough*LKDTM*cough*), having
> > a dashboard that shows features down one axis and architectures and/or
> > kernel versions on other axes, then I get a pretty picture. But it's
> > still being built out of the yes/no info.
> >
> > *shrug*
> >
> > I think diagnostic should be expressly non-machine-oriented.
>
> So from the KUnit side, we sort-of have three kinds of diagnostic lines:
> - Lines printed directly from tests (typically using kunit_info() or
> similar functions): as I understand it, these are basically the
> equivalent of what kselftest typically uses diagnostics for --
> test-specific, human-readable messages. I don't think we need/want to
> parse these much.
> - Kernel messages during test execution. If we get the results from
> scraping the kernel log (which is still the default for KUnit, though
> there is also a debugfs info), other kernel logs can be interleaved
> with the results. Sometimes these are irrelevant things happening on
> another thread, sometimes they're something directly related to the
> test which we'd like to capture (KASAN errors, for instance). I don't
> think we want these to be machine oriented, but we may want to be able
> to filter them out.
> - Expectation failures: as Brendan mentioned, KUnit will print some
> diagnostic messages for individual assertion/expectation failures,
> including the expected and actual values. We'd ideally like to be able
> to identify and parse these, but keeping them human-readable is
> definitely also a goal.

Seems like a fair characterization to me.

> Now, to be honest, I doubt that the distinction here would be of much
> use to kselftest, but it could be nice to not go out of our way to
> make parsing some diagnostic lines possible. That being said,
> personally I'm all for avoiding the yaml for diagnostic messages stuff
> and sticking to something simple and line-based, possibly
> standardising a the format of a few common diagnostic measurements
> (e.g., assertions/expected values/etc) in a way that's both
> human-readable and parsable if possible.

Agreed.

> I agree that there's a lot of analysis that is possible with just the
> yes/no data. There's probably some fancy correlation one could do even
> with unstructured diagnostic logs, so I don't think overstructuring
> things is a necessity by any means. Where we have different tests
> doing similar sorts of things, though, consistency in message
> formatting could help even if things are not explicitly parsed.
> Ensuring that helper functions that log and the like are spitting
> things out in the same format is probably a good starting step down
> that path.

Super agreed. I am sure that there are people who can do some very
clever things with what is being described here, but I would prefer
not to overcomplicate the spec right out of the gate. It feels like we
only have two TAP versions to reconcile here, and we are mostly in
agreement on how to do that.