On 03/26/2015 01:33 PM, Tyler Baker wrote:
> On 26 March 2015 at 09:29, John Stultz <[email protected]> wrote:
>> On Thu, Mar 26, 2015 at 4:31 AM, Prarit Bhargava <[email protected]> wrote:
>>> On 03/25/2015 07:44 PM, John Stultz wrote:
>>>> + printf("%-22s %s missing CAP_WAKE_ALARM? : [UNSUPPORTED]\n",
>>>> + clockstring(clock_id),
>>>> + flags ? "ABSTIME":"RELTIME");
>>>
>>> Something to think about: Do you want to write these tests to be more human
>>> readable or machine readable? In theory with awk I guess it doesn't matter too
>>> much, however, it is something that we should think about moving forward.
>>
>> So this came up at ELC in a few discussions. Right now there isn't any
>> established output format, but there's some nice and simple
>> infrastructure for counting pass/fails.
>>
>> However, in talking to Tyler, I know he has started looking at how to
>> integrate the selftests into our automated infrastructure and was
>> interested in how we improve the output parsing for reports. So there
>> is interest in improving this, and I'm open to whatever changes might
>> be needed (adding extra arguments to the test to put them into "easy
>> parse" mode or whatever).
>
> Thanks for looping me in John. My interest in kselftest stems from my
> involvement with kernelci.org, a communityservice focused on upstream
> kernel validation across multiple architectures. In it's current form,
> it is merely build and boot testing boards. However, we are at a point
> where we'd like to start running some tests. The automation framework
> (LAVA) used to execute these tests essentially uses a regular
> expression to parse the test's standard output. This is advantageous
> as a test can be written in any language, as long as it produces sane
> uniform output.
>
> Ideally, we would like to perform the kernel builds as we do today
> along with building all the kseltests present in the tree, and
> inserting them into a 'testing' ramdisk for deployment. Once we
> successfully boot the platform, we execute all the kselftests, parse
> standard out, and report the results. The benefit from this
> implementation is that a developer writing a test does have to do
> anything 'special' to get his/her test to run once it has been applied
> to a upstream tree. I'll explain below some concerns I have about
> accomplishing this.
>
> Currently, we have had to write wrappers[1][2] for some kselftests to
> be able parse the output. If we can choose/agree on a standard output
> format all of this complexity goes away, and then we can dynamically
> run kselftests. Integration of new tests will not be needed, as they
> all produce output in standard way. I've taken a look at the wiki page
> for standardizing output[3] and TAP looks like the good format IMO.
>
> Also, for arch != x86 there are some barriers to overcome to get all
> the kselftests cross compiling, which would be nice to have as well.
>
> I realize this may be a good amount of work, so I'd like to help out.
> Perhaps working John to convert his timer tests to use TAP output
> would be a good starting point?
John, I could probably do that for you. I'm always willing to give it a shot.
>
>>
>> thanks
>> -john
>
> [1] https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/common/scripts/kselftest-runner.sh
> [2] https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/common/scripts/kselftest-mqueue.sh
> [3] https://kselftest.wiki.kernel.org/standardize_the_test_output
I'll go off and look at this and wait for the current patchset(s) to make it
into Linus' tree before posting or suggesting patches.
P.
>
> Cheers,
>
> Tyler
>
On 04/02/2015 04:18 AM, Prarit Bhargava wrote:
>
>
> On 03/26/2015 01:33 PM, Tyler Baker wrote:
>> On 26 March 2015 at 09:29, John Stultz <[email protected]> wrote:
>>> On Thu, Mar 26, 2015 at 4:31 AM, Prarit Bhargava <[email protected]> wrote:
>>>> On 03/25/2015 07:44 PM, John Stultz wrote:
>>>>> + printf("%-22s %s missing CAP_WAKE_ALARM? : [UNSUPPORTED]\n",
>>>>> + clockstring(clock_id),
>>>>> + flags ? "ABSTIME":"RELTIME");
>>>>
>>>> Something to think about: Do you want to write these tests to be more human
>>>> readable or machine readable? In theory with awk I guess it doesn't matter too
>>>> much, however, it is something that we should think about moving forward.
>>>
>>> So this came up at ELC in a few discussions. Right now there isn't any
>>> established output format, but there's some nice and simple
>>> infrastructure for counting pass/fails.
>>>
>>> However, in talking to Tyler, I know he has started looking at how to
>>> integrate the selftests into our automated infrastructure and was
>>> interested in how we improve the output parsing for reports. So there
>>> is interest in improving this, and I'm open to whatever changes might
>>> be needed (adding extra arguments to the test to put them into "easy
>>> parse" mode or whatever).
>>
>> Thanks for looping me in John. My interest in kselftest stems from my
>> involvement with kernelci.org, a communityservice focused on upstream
>> kernel validation across multiple architectures. In it's current form,
>> it is merely build and boot testing boards. However, we are at a point
>> where we'd like to start running some tests. The automation framework
>> (LAVA) used to execute these tests essentially uses a regular
>> expression to parse the test's standard output. This is advantageous
>> as a test can be written in any language, as long as it produces sane
>> uniform output.
>>
>> Ideally, we would like to perform the kernel builds as we do today
>> along with building all the kseltests present in the tree, and
>> inserting them into a 'testing' ramdisk for deployment. Once we
>> successfully boot the platform, we execute all the kselftests, parse
>> standard out, and report the results. The benefit from this
>> implementation is that a developer writing a test does have to do
>> anything 'special' to get his/her test to run once it has been applied
>> to a upstream tree. I'll explain below some concerns I have about
>> accomplishing this.
>>
>> Currently, we have had to write wrappers[1][2] for some kselftests to
>> be able parse the output. If we can choose/agree on a standard output
>> format all of this complexity goes away, and then we can dynamically
>> run kselftests. Integration of new tests will not be needed, as they
>> all produce output in standard way. I've taken a look at the wiki page
>> for standardizing output[3] and TAP looks like the good format IMO.
>>
>> Also, for arch != x86 there are some barriers to overcome to get all
>> the kselftests cross compiling, which would be nice to have as well.
>>
>> I realize this may be a good amount of work, so I'd like to help out.
>> Perhaps working John to convert his timer tests to use TAP output
>> would be a good starting point?
>
> John, I could probably do that for you. I'm always willing to give it a shot.
>
Improving reporting and output is a good idea, as long as the
reporting framework doesn't add external dependencies and makes
it difficult to build and run tests from the kernel git tree
environment. Being able to run tests on a development system
is the primary objective for these tests.
As long as the TAP doesn't require additional libraries and
tools to be installed on the development systems, I will be
happy with the improvements to reporting and output.
thanks,
-- Shuah
--
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
[email protected] | (970) 217-8978
On 2 April 2015 at 06:43, Shuah Khan <[email protected]> wrote:
> On 04/02/2015 04:18 AM, Prarit Bhargava wrote:
>>
>>
>> On 03/26/2015 01:33 PM, Tyler Baker wrote:
>>> On 26 March 2015 at 09:29, John Stultz <[email protected]> wrote:
>>>> On Thu, Mar 26, 2015 at 4:31 AM, Prarit Bhargava <[email protected]> wrote:
>>>>> On 03/25/2015 07:44 PM, John Stultz wrote:
>>>>>> + printf("%-22s %s missing CAP_WAKE_ALARM? : [UNSUPPORTED]\n",
>>>>>> + clockstring(clock_id),
>>>>>> + flags ? "ABSTIME":"RELTIME");
>>>>>
>>>>> Something to think about: Do you want to write these tests to be more human
>>>>> readable or machine readable? In theory with awk I guess it doesn't matter too
>>>>> much, however, it is something that we should think about moving forward.
>>>>
>>>> So this came up at ELC in a few discussions. Right now there isn't any
>>>> established output format, but there's some nice and simple
>>>> infrastructure for counting pass/fails.
>>>>
>>>> However, in talking to Tyler, I know he has started looking at how to
>>>> integrate the selftests into our automated infrastructure and was
>>>> interested in how we improve the output parsing for reports. So there
>>>> is interest in improving this, and I'm open to whatever changes might
>>>> be needed (adding extra arguments to the test to put them into "easy
>>>> parse" mode or whatever).
>>>
>>> Thanks for looping me in John. My interest in kselftest stems from my
>>> involvement with kernelci.org, a communityservice focused on upstream
>>> kernel validation across multiple architectures. In it's current form,
>>> it is merely build and boot testing boards. However, we are at a point
>>> where we'd like to start running some tests. The automation framework
>>> (LAVA) used to execute these tests essentially uses a regular
>>> expression to parse the test's standard output. This is advantageous
>>> as a test can be written in any language, as long as it produces sane
>>> uniform output.
>>>
>>> Ideally, we would like to perform the kernel builds as we do today
>>> along with building all the kseltests present in the tree, and
>>> inserting them into a 'testing' ramdisk for deployment. Once we
>>> successfully boot the platform, we execute all the kselftests, parse
>>> standard out, and report the results. The benefit from this
>>> implementation is that a developer writing a test does have to do
>>> anything 'special' to get his/her test to run once it has been applied
>>> to a upstream tree. I'll explain below some concerns I have about
>>> accomplishing this.
>>>
>>> Currently, we have had to write wrappers[1][2] for some kselftests to
>>> be able parse the output. If we can choose/agree on a standard output
>>> format all of this complexity goes away, and then we can dynamically
>>> run kselftests. Integration of new tests will not be needed, as they
>>> all produce output in standard way. I've taken a look at the wiki page
>>> for standardizing output[3] and TAP looks like the good format IMO.
>>>
>>> Also, for arch != x86 there are some barriers to overcome to get all
>>> the kselftests cross compiling, which would be nice to have as well.
>>>
>>> I realize this may be a good amount of work, so I'd like to help out.
>>> Perhaps working John to convert his timer tests to use TAP output
>>> would be a good starting point?
>>
>> John, I could probably do that for you. I'm always willing to give it a shot.
>>
>
> Improving reporting and output is a good idea, as long as the
> reporting framework doesn't add external dependencies and makes
> it difficult to build and run tests from the kernel git tree
> environment. Being able to run tests on a development system
> is the primary objective for these tests.
>
> As long as the TAP doesn't require additional libraries and
> tools to be installed on the development systems, I will be
> happy with the improvements to reporting and output.
This is a very valid concern IMO. There is a C library for TAP
providers[0] but I agree we want to avoid external libraries as
dependencies. Perhaps, it would be simple enough to write a selftest
harness that provides a basic interface for displaying test results,
messages, and warning/errors. This harness would ensure all the
logging was done in a TAP compliant manner. Food for thought :)
>
> thanks,
> -- Shuah
>
> --
> Shuah Khan
> Sr. Linux Kernel Developer
> Open Source Innovation Group
> Samsung Research America (Silicon Valley)
> [email protected] | (970) 217-8978
Cheers,
Tyler
[0] http://www.eyrie.org/~eagle/software/c-tap-harness/
On 04/02/2015 11:17 AM, Tyler Baker wrote:
> On 2 April 2015 at 06:43, Shuah Khan <[email protected]> wrote:
>> On 04/02/2015 04:18 AM, Prarit Bhargava wrote:
>>>
>>>
>>> On 03/26/2015 01:33 PM, Tyler Baker wrote:
>>>> On 26 March 2015 at 09:29, John Stultz <[email protected]> wrote:
>>>>> On Thu, Mar 26, 2015 at 4:31 AM, Prarit Bhargava <[email protected]> wrote:
>>>>>> On 03/25/2015 07:44 PM, John Stultz wrote:
>>>>>>> + printf("%-22s %s missing CAP_WAKE_ALARM? : [UNSUPPORTED]\n",
>>>>>>> + clockstring(clock_id),
>>>>>>> + flags ? "ABSTIME":"RELTIME");
>>>>>>
>>>>>> Something to think about: Do you want to write these tests to be more human
>>>>>> readable or machine readable? In theory with awk I guess it doesn't matter too
>>>>>> much, however, it is something that we should think about moving forward.
>>>>>
>>>>> So this came up at ELC in a few discussions. Right now there isn't any
>>>>> established output format, but there's some nice and simple
>>>>> infrastructure for counting pass/fails.
>>>>>
>>>>> However, in talking to Tyler, I know he has started looking at how to
>>>>> integrate the selftests into our automated infrastructure and was
>>>>> interested in how we improve the output parsing for reports. So there
>>>>> is interest in improving this, and I'm open to whatever changes might
>>>>> be needed (adding extra arguments to the test to put them into "easy
>>>>> parse" mode or whatever).
>>>>
>>>> Thanks for looping me in John. My interest in kselftest stems from my
>>>> involvement with kernelci.org, a communityservice focused on upstream
>>>> kernel validation across multiple architectures. In it's current form,
>>>> it is merely build and boot testing boards. However, we are at a point
>>>> where we'd like to start running some tests. The automation framework
>>>> (LAVA) used to execute these tests essentially uses a regular
>>>> expression to parse the test's standard output. This is advantageous
>>>> as a test can be written in any language, as long as it produces sane
>>>> uniform output.
>>>>
>>>> Ideally, we would like to perform the kernel builds as we do today
>>>> along with building all the kseltests present in the tree, and
>>>> inserting them into a 'testing' ramdisk for deployment. Once we
>>>> successfully boot the platform, we execute all the kselftests, parse
>>>> standard out, and report the results. The benefit from this
>>>> implementation is that a developer writing a test does have to do
>>>> anything 'special' to get his/her test to run once it has been applied
>>>> to a upstream tree. I'll explain below some concerns I have about
>>>> accomplishing this.
>>>>
>>>> Currently, we have had to write wrappers[1][2] for some kselftests to
>>>> be able parse the output. If we can choose/agree on a standard output
>>>> format all of this complexity goes away, and then we can dynamically
>>>> run kselftests. Integration of new tests will not be needed, as they
>>>> all produce output in standard way. I've taken a look at the wiki page
>>>> for standardizing output[3] and TAP looks like the good format IMO.
>>>>
>>>> Also, for arch != x86 there are some barriers to overcome to get all
>>>> the kselftests cross compiling, which would be nice to have as well.
>>>>
>>>> I realize this may be a good amount of work, so I'd like to help out.
>>>> Perhaps working John to convert his timer tests to use TAP output
>>>> would be a good starting point?
>>>
>>> John, I could probably do that for you. I'm always willing to give it a shot.
>>>
>>
>> Improving reporting and output is a good idea, as long as the
>> reporting framework doesn't add external dependencies and makes
>> it difficult to build and run tests from the kernel git tree
>> environment. Being able to run tests on a development system
>> is the primary objective for these tests.
>>
>> As long as the TAP doesn't require additional libraries and
>> tools to be installed on the development systems, I will be
>> happy with the improvements to reporting and output.
>
> This is a very valid concern IMO. There is a C library for TAP
> providers[0] but I agree we want to avoid external libraries as
> dependencies. Perhaps, it would be simple enough to write a selftest
> harness that provides a basic interface for displaying test results,
> messages, and warning/errors. This harness would ensure all the
> logging was done in a TAP compliant manner. Food for thought :)
>
The harness has to cover tests in written in C as well as shell
scripts. I added a simple framework in kselftest.h as a step
towards harness. However, it needs more work. I would welcome
any patches that would provide a harness that meets the needs
of external selftest use-cases such as yours.
Is writing a simple TAP compliant harness something you can take
on?? Looking for volunteers :)
thanks,
-- Shuah
--
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
[email protected] | (970) 217-8978
On Thu, Apr 2, 2015 at 3:18 AM, Prarit Bhargava <[email protected]> wrote:
> On 03/26/2015 01:33 PM, Tyler Baker wrote:
>> I realize this may be a good amount of work, so I'd like to help out.
>> Perhaps working John to convert his timer tests to use TAP output
>> would be a good starting point?
>
> John, I could probably do that for you. I'm always willing to give it a shot.
I took a quick look into it, since I'm definitely interested in
improving output formatting, but man, TAP is a fairly ugly output
format if you ask me.
It only has binary "ok" or "not ok" (why not "fail", or something else
that's exclusively grep-able, I don't know). So I'm not sure if cases
where functionality is unsupported should be a pass or fail.
Most problematically: It seems to want enumeration in the test output
(so test 2 needs to print: "ok 2 Test complete") which means either
there needs to be a wrapper that does the TAP output knowing which
test of N its currently running, or the test number needs to be
submitted as an runtime argument to the test, and the test then has to
add it to its output line.
Anyway, if we do want to go with that format, I suspect it should be
something we add to the kselftest pass/fail hooks, rather then to the
individual tests. Then its just a matter of prefixing normal test
output with #'s so they can be ignored by the parser.
thanks
-john
On 2 April 2015 at 10:48, Shuah Khan <[email protected]> wrote:
> On 04/02/2015 11:17 AM, Tyler Baker wrote:
>> On 2 April 2015 at 06:43, Shuah Khan <[email protected]> wrote:
>>> On 04/02/2015 04:18 AM, Prarit Bhargava wrote:
>>>>
>>>>
>>>> On 03/26/2015 01:33 PM, Tyler Baker wrote:
>>>>> On 26 March 2015 at 09:29, John Stultz <[email protected]> wrote:
>>>>>> On Thu, Mar 26, 2015 at 4:31 AM, Prarit Bhargava <[email protected]> wrote:
>>>>>>> On 03/25/2015 07:44 PM, John Stultz wrote:
>>>>>>>> + printf("%-22s %s missing CAP_WAKE_ALARM? : [UNSUPPORTED]\n",
>>>>>>>> + clockstring(clock_id),
>>>>>>>> + flags ? "ABSTIME":"RELTIME");
>>>>>>>
>>>>>>> Something to think about: Do you want to write these tests to be more human
>>>>>>> readable or machine readable? In theory with awk I guess it doesn't matter too
>>>>>>> much, however, it is something that we should think about moving forward.
>>>>>>
>>>>>> So this came up at ELC in a few discussions. Right now there isn't any
>>>>>> established output format, but there's some nice and simple
>>>>>> infrastructure for counting pass/fails.
>>>>>>
>>>>>> However, in talking to Tyler, I know he has started looking at how to
>>>>>> integrate the selftests into our automated infrastructure and was
>>>>>> interested in how we improve the output parsing for reports. So there
>>>>>> is interest in improving this, and I'm open to whatever changes might
>>>>>> be needed (adding extra arguments to the test to put them into "easy
>>>>>> parse" mode or whatever).
>>>>>
>>>>> Thanks for looping me in John. My interest in kselftest stems from my
>>>>> involvement with kernelci.org, a communityservice focused on upstream
>>>>> kernel validation across multiple architectures. In it's current form,
>>>>> it is merely build and boot testing boards. However, we are at a point
>>>>> where we'd like to start running some tests. The automation framework
>>>>> (LAVA) used to execute these tests essentially uses a regular
>>>>> expression to parse the test's standard output. This is advantageous
>>>>> as a test can be written in any language, as long as it produces sane
>>>>> uniform output.
>>>>>
>>>>> Ideally, we would like to perform the kernel builds as we do today
>>>>> along with building all the kseltests present in the tree, and
>>>>> inserting them into a 'testing' ramdisk for deployment. Once we
>>>>> successfully boot the platform, we execute all the kselftests, parse
>>>>> standard out, and report the results. The benefit from this
>>>>> implementation is that a developer writing a test does have to do
>>>>> anything 'special' to get his/her test to run once it has been applied
>>>>> to a upstream tree. I'll explain below some concerns I have about
>>>>> accomplishing this.
>>>>>
>>>>> Currently, we have had to write wrappers[1][2] for some kselftests to
>>>>> be able parse the output. If we can choose/agree on a standard output
>>>>> format all of this complexity goes away, and then we can dynamically
>>>>> run kselftests. Integration of new tests will not be needed, as they
>>>>> all produce output in standard way. I've taken a look at the wiki page
>>>>> for standardizing output[3] and TAP looks like the good format IMO.
>>>>>
>>>>> Also, for arch != x86 there are some barriers to overcome to get all
>>>>> the kselftests cross compiling, which would be nice to have as well.
>>>>>
>>>>> I realize this may be a good amount of work, so I'd like to help out.
>>>>> Perhaps working John to convert his timer tests to use TAP output
>>>>> would be a good starting point?
>>>>
>>>> John, I could probably do that for you. I'm always willing to give it a shot.
>>>>
>>>
>>> Improving reporting and output is a good idea, as long as the
>>> reporting framework doesn't add external dependencies and makes
>>> it difficult to build and run tests from the kernel git tree
>>> environment. Being able to run tests on a development system
>>> is the primary objective for these tests.
>>>
>>> As long as the TAP doesn't require additional libraries and
>>> tools to be installed on the development systems, I will be
>>> happy with the improvements to reporting and output.
>>
>> This is a very valid concern IMO. There is a C library for TAP
>> providers[0] but I agree we want to avoid external libraries as
>> dependencies. Perhaps, it would be simple enough to write a selftest
>> harness that provides a basic interface for displaying test results,
>> messages, and warning/errors. This harness would ensure all the
>> logging was done in a TAP compliant manner. Food for thought :)
>>
>
> The harness has to cover tests in written in C as well as shell
> scripts. I added a simple framework in kselftest.h as a step
> towards harness. However, it needs more work. I would welcome
> any patches that would provide a harness that meets the needs
> of external selftest use-cases such as yours.
Ok, I'll have a closer look at kselftest.h.
>
> Is writing a simple TAP compliant harness something you can take
> on?? Looking for volunteers :)
I'll volunteer to pick up these work items :) I should be able to
start on this work at some point next week.
John brought up some good points about the TAP format, and I'm
personally not yet convinced it is the best format. That being said,
I'll work to keep the changes to the harness generic so that we can
easily switch test output formats if we decide there is something
better than TAP in the future.
>
> thanks,
> -- Shuah
>
> --
> Shuah Khan
> Sr. Linux Kernel Developer
> Open Source Innovation Group
> Samsung Research America (Silicon Valley)
> [email protected] | (970) 217-8978
Thanks,
Tyler
On 04/02/2015 02:02 PM, John Stultz wrote:
> On Thu, Apr 2, 2015 at 3:18 AM, Prarit Bhargava <[email protected]> wrote:
>> On 03/26/2015 01:33 PM, Tyler Baker wrote:
>>> I realize this may be a good amount of work, so I'd like to help out.
>>> Perhaps working John to convert his timer tests to use TAP output
>>> would be a good starting point?
>>
>> John, I could probably do that for you. I'm always willing to give it a shot.
>
> I took a quick look into it, since I'm definitely interested in
> improving output formatting, but man, TAP is a fairly ugly output
> format if you ask me.
>
> It only has binary "ok" or "not ok" (why not "fail", or something else
> that's exclusively grep-able, I don't know). So I'm not sure if cases
> where functionality is unsupported should be a pass or fail.
>
> Most problematically: It seems to want enumeration in the test output
> (so test 2 needs to print: "ok 2 Test complete") which means either
> there needs to be a wrapper that does the TAP output knowing which
> test of N its currently running, or the test number needs to be
> submitted as an runtime argument to the test, and the test then has to
> add it to its output line.
>
> Anyway, if we do want to go with that format, I suspect it should be
> something we add to the kselftest pass/fail hooks, rather then to the
> individual tests. Then its just a matter of prefixing normal test
> output with #'s so they can be ignored by the parser.
I've been looking into this and while TAP is useful, I find myself with a few
questions:
1. I want the output to be both human and machine readable. I find that "not
okay" vs "okay" is confusing at best for humans to read. I suppose that is a
minor issue, but it is a real issue when laymen are asked to execute a test.
2. I think we can get around the TAP numbering, however, maybe a better
approach is to limit tests to doing one, and only one, test? ie) something like
rtctest.c would have be be broken up into individual tests.
3. TAP appears to have a BSD-style license? Not knowing enough about cross
licensing issues like this, is that going to be a problem?
4. I think John is moving in the right direction by suggesting that we move to
putting this into kselftest.h as hooks, rather than modifying each test.
Another idea is to include a script that allows the running of multiple tests
from the top level tests directory.
Let me muck around with the timer tests and see what I can come up with. I'm
trying to keep in mind that we're going to be asking people to potentially
modify their tests, as well as what we're going to be telling people to do when
they add new tests, while I make modifications.
P.
>
> thanks
> -john
>
On Thu, 2015-04-02 at 11:02 -0700, John Stultz wrote:
> On Thu, Apr 2, 2015 at 3:18 AM, Prarit Bhargava <[email protected]> wrote:
> > On 03/26/2015 01:33 PM, Tyler Baker wrote:
> >> I realize this may be a good amount of work, so I'd like to help out.
> >> Perhaps working John to convert his timer tests to use TAP output
> >> would be a good starting point?
> >
> > John, I could probably do that for you. I'm always willing to give it a shot.
>
> I took a quick look into it, since I'm definitely interested in
> improving output formatting, but man, TAP is a fairly ugly output
> format if you ask me.
>
> It only has binary "ok" or "not ok" (why not "fail", or something else
> that's exclusively grep-able, I don't know). So I'm not sure if cases
> where functionality is unsupported should be a pass or fail.
>
> Most problematically: It seems to want enumeration in the test output
> (so test 2 needs to print: "ok 2 Test complete") which means either
> there needs to be a wrapper that does the TAP output knowing which
> test of N its currently running, or the test number needs to be
> submitted as an runtime argument to the test, and the test then has to
> add it to its output line.
Yeah TAP is horrible, for the reasons you describe.
I *think* in practice most tools will handle just "ok" / "not ok" without the
tests being numbered, but I don't know that for sure.
For the powerpc tests I'm using "subunit v1" [1], which is basically just:
- "^success: <test name>"
- "^failure: <test name>"
See: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/powerpc/subunit.h
So I can do eg:
$ cd tools/testing/selftests/powerpc
$ make run_tests 2>&1 | subunit-1to2 | subunit-stats --no-passthrough
Total tests: 35
Passed tests: 31
Failed tests: 0
Skipped tests: 4
Seen tags: git_version:v4.0-rc7-0-gf22e6e8
But unfortunately TAP has a lot more traction with tools.
It wouldn't be too hard to convert the subunit stream into TAP I think, but for
some reason no one seems to have written that. Maybe we should, I just haven't
had time to do it.
cheers
[1]: https://github.com/testing-cabal/subunit/blob/master/README#L343