is there any sort of standard for testing and integration into
mainline ? in the Blackfin world, we've been developing little
external kernel modules and adding them to our own testsuite, but
often times these things are not Blackfin specific. case in point,
we're integrating a string testsuite to make sure all of the fun str*
and mem* functions are sane and operate as they expected, but rather
than having just Blackfin benefit here, i'd like to see this pushed
upstream ...
i'm fully aware of LTP (as i work on it), but i feel that serves a
great purpose for exercising the API/ABI exposed to userspace either
directly through the kernel or indirectly through the system libc ...
it isnt a very good tool for testing kernel internals, especially as
the kernel changes and evolves.
is there a framework already in place i'm not aware of ? should there
be ? should this all live in LTP ? i wouldnt mind an option under
kernel hacking "Enable testsuites" ... as far as i can tell, the
testing process is really based extensively on feedback from people,
nothing really automated.
-mike
On Fri 31 Aug 2007 17:22, Mike Frysinger pondered:
> is there any sort of standard for testing and integration into
> mainline ? in the Blackfin world, we've been developing little
> external kernel modules and adding them to our own testsuite, but
> often times these things are not Blackfin specific. case in point,
> we're integrating a string testsuite to make sure all of the fun str*
> and mem* functions are sane and operate as they expected, but rather
> than having just Blackfin benefit here, i'd like to see this pushed
> upstream ...
I know there have been some discussions at past OLS about some testing that
Martin was working on.
http://www.linuxsymposium.org/2006/view_abstract.php?content_key=13
But I think this was more functional tests, less unit tests that what you are
talking/asking about.
http://test.kernel.org/functional/index.html
Also - If I remember - most of the existing tests were for a self hosted
environment - and might not be well suited to embedded (like Blackfin) which
requires cross compile, and pretty thin runtime environment (uClibc +
busybox's msh as shell).
Martin?
-Robin
"Mike Frysinger" <[email protected]> writes:
> is there any sort of standard for testing and integration into
> mainline ?
Everybody does their own.
> in the Blackfin world, we've been developing little
> external kernel modules and adding them to our own testsuite, but
> often times these things are not Blackfin specific. case in point,
> we're integrating a string testsuite to make sure all of the fun str*
> and mem* functions are sane and operate as they expected, but rather
> than having just Blackfin benefit here, i'd like to see this pushed
> upstream ...
I would like to see this too. I wrote a couple of unit tests
during x86-64 development too and they would be handy
to have in a central place.
The SUSE kernel also has a crasher module that exercises the
allocators and is pretty useful to stress code in general.
Disadvantage is that test code tends to be hackish and ugly
and very specialized and will probably not pass standard review procedures.
Also the rt people seem to have pushed a couple of tests in already,
but I always hated it that they're in the standard directories.
RCU also has, in fact they added a "eat all my CPU in tests"
CONFIG option. Just making that dependent on a CONFIG_UNIT_TESTS
would be a good change in itself. And then there are the slab
failure inducers of course.
BTW string functions are best tested in user space. That's
a relatively bad example.
-Andi
On 02 Sep 2007 00:08:57 +0200, Andi Kleen <[email protected]> wrote:
> BTW string functions are best tested in user space. That's
> a relatively bad example.
in theory, maybe ... in reality, i really dont think so
the string implementations are spread out over the kernel ... there's
implementations in lib/, include/asm-*/, and arch/*/lib/ ... so any
test code that would use these sources is going to be an ugly hack to
make sure it grabs all the right pieces from all the right places.
then there's the issue of API ... compiling code that is designed to
be in the kernel and nowhere else is going to require a mighty bit of
crappy glue logic in order for it to compile properly in userspace
but let's cheat and just use the kernel build system to produce .o's
for us ... then there's the issue of ABI. there is no guarantee the
ABI the kernel uses internally is the same as the ABI userspace uses
(and in fact, on some architectures it is simply not possible). so
you're left with a solution that may work for some, but not for
others.
or we put one common piece of C test code right into the kernel and be
done. no worrying about the tested API/ABI/environment being
different from the actual runtime the kernel itself has.
-mike
On 9/1/07, Robin Getz <[email protected]> wrote:
> On Fri 31 Aug 2007 17:22, Mike Frysinger pondered:
> > is there any sort of standard for testing and integration into
> > mainline ? in the Blackfin world, we've been developing little
> > external kernel modules and adding them to our own testsuite, but
> > often times these things are not Blackfin specific. case in point,
> > we're integrating a string testsuite to make sure all of the fun str*
> > and mem* functions are sane and operate as they expected, but rather
> > than having just Blackfin benefit here, i'd like to see this pushed
> > upstream ...
>
> I know there have been some discussions at past OLS about some testing that
> Martin was working on.
>
> http://www.linuxsymposium.org/2006/view_abstract.php?content_key=13
>
> But I think this was more functional tests, less unit tests that what you are
> talking/asking about.
>
> http://test.kernel.org/functional/index.html
>
> Also - If I remember - most of the existing tests were for a self hosted
> environment - and might not be well suited to embedded (like Blackfin) which
> requires cross compile, and pretty thin runtime environment (uClibc +
> busybox's msh as shell).
i flipped through the autotest code a bit but i dont think this can do
what i'm looking for. it appears to be a mix of LTP provides and what
we have for our Blackfin test infrastructure -- iow, all userspace
stuff and no real way of exercising kernel internals.
-mike
Mike Frysinger wrote:
> is there any sort of standard for testing and integration into
> mainline ? in the Blackfin world, we've been developing little
> external kernel modules and adding them to our own testsuite, but
> often times these things are not Blackfin specific. case in point,
> we're integrating a string testsuite to make sure all of the fun str*
> and mem* functions are sane and operate as they expected, but rather
> than having just Blackfin benefit here, i'd like to see this pushed
> upstream ...
>
> i'm fully aware of LTP (as i work on it), but i feel that serves a
> great purpose for exercising the API/ABI exposed to userspace either
> directly through the kernel or indirectly through the system libc ...
> it isnt a very good tool for testing kernel internals, especially as
> the kernel changes and evolves.
>
> is there a framework already in place i'm not aware of ? should there
> be ? should this all live in LTP ? i wouldnt mind an option under
> kernel hacking "Enable testsuites" ... as far as i can tell, the
> testing process is really based extensively on feedback from people,
> nothing really automated.
>
If you want to test that stuff and run it on the current code in the
kernel, how about a kernel module? You could "modprobe sanitytest" or
something and report to syslog at module load time. And maybe have a
parameter which does something drastic if something core is so hosed
that filesystem damage is likely. Or just optional init code run by a
kernel option, perhaps sanity testing after boot is not a great idea.
--
Bill Davidsen <[email protected]>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot
On 9/1/07, Bill Davidsen <[email protected]> wrote:
> If you want to test that stuff and run it on the current code in the
> kernel, how about a kernel module? You could "modprobe sanitytest" or
> something and report to syslog at module load time. And maybe have a
> parameter which does something drastic if something core is so hosed
> that filesystem damage is likely. Or just optional init code run by a
> kernel option, perhaps sanity testing after boot is not a great idea.
i'm fully capable of doing my own testing right now; that isnt the
point of this thread ... the point is for everyone to benefit, not
just the Blackfin architecture
-mike
On Sat, Sep 01, 2007 at 06:50:30PM -0400, Mike Frysinger wrote:
> On 02 Sep 2007 00:08:57 +0200, Andi Kleen <[email protected]> wrote:
> > BTW string functions are best tested in user space. That's
> > a relatively bad example.
>
> in theory, maybe ... in reality, i really dont think so
>
> the string implementations are spread out over the kernel ... there's
> implementations in lib/, include/asm-*/, and arch/*/lib/ ... so any
> test code that would use these sources is going to be an ugly hack to
> make sure it grabs all the right pieces from all the right places.
string functions tend to be self contained.
The other issue to test some of them properly you need unmapped pages
etc. That gets much easier to do in user space. There are some other
issues.
-Andi
On 9/2/07, Andi Kleen <[email protected]> wrote:
> On Sat, Sep 01, 2007 at 06:50:30PM -0400, Mike Frysinger wrote:
> > On 02 Sep 2007 00:08:57 +0200, Andi Kleen <[email protected]> wrote:
> > > BTW string functions are best tested in user space. That's
> > > a relatively bad example.
> >
> > in theory, maybe ... in reality, i really dont think so
> >
> > the string implementations are spread out over the kernel ... there's
> > implementations in lib/, include/asm-*/, and arch/*/lib/ ... so any
> > test code that would use these sources is going to be an ugly hack to
> > make sure it grabs all the right pieces from all the right places.
>
> string functions tend to be self contained.
"tend to be" does not lend itself to being cleanly tested ... and i
imagine i'd get some pretty heavy resistance if i proposed
re-organzing code just so that i could compile it outside of the
kernel
there is still the ABI issue ... code written in kernel space in pure
asm cannot always be compiled in userspace and work properly/the same
> The other issue to test some of them properly you need unmapped pages
> etc. That gets much easier to do in user space. There are some other
> issues.
you mean testing boundary overflows ? can be handled with canaries
rather than segfaults i imagine ...
-mike
On Sun, Sep 02, 2007 at 11:15:23AM -0400, Mike Frysinger wrote:
> there is still the ABI issue ... code written in kernel space in pure
> asm cannot always be compiled in userspace and work properly/the same
Is that a blackfin weirdness?
>
> > The other issue to test some of them properly you need unmapped pages
> > etc. That gets much easier to do in user space. There are some other
> > issues.
>
> you mean testing boundary overflows ? can be handled with canaries
> rather than segfaults i imagine ...
Not for reads, no.
-Andi
On 9/2/07, Andi Kleen <[email protected]> wrote:
> The other issue to test some of them properly you need unmapped pages
> etc. That gets much easier to do in user space. There are some other
> issues.
vmalloc, vmap, etc. always put a guard page after the allocation. So
if you test string operations on vmalloc()'ed memory, you have
unmapped pages in both ends.
I like the idea of a test suite for kernel internals, but I also think
it would be best to run it in kernel mode since things like atomics,
etc. may work differently in user space on some architectures.
Haavard
Mike Frysinger wrote:
> On 9/1/07, Bill Davidsen <[email protected]> wrote:
>
>> If you want to test that stuff and run it on the current code in the
>> kernel, how about a kernel module? You could "modprobe sanitytest" or
>> something and report to syslog at module load time. And maybe have a
>> parameter which does something drastic if something core is so hosed
>> that filesystem damage is likely. Or just optional init code run by a
>> kernel option, perhaps sanity testing after boot is not a great idea.
>>
>
> i'm fully capable of doing my own testing right now; that isnt the
> point of this thread ... the point is for everyone to benefit, not
> just the Blackfin architecture
>
That's exactly the point I was making, if your tests and any others
people feel are useful were in a module and/or could be run as init code
only when wanted, everyone could benefit from the tests. Anything
related to build sanity could be in one place, to guard against
surprises like new versions of gcc, typos in patches, etc.
--
bill davidsen <[email protected]>
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979
On Sat 1 Sep 2007 18:08, Andi Kleen pondered:
> "Mike Frysinger" <[email protected]> writes:
>
> > is there any sort of standard for testing and integration into
> > mainline ?
>
> Everybody does their own.
That kind of stinks - and seems to be a potential duplication of effort all
over the place.
> > in the Blackfin world, we've been developing little
> > external kernel modules and adding them to our own testsuite, but
> > often times these things are not Blackfin specific. case in point,
> > we're integrating a string testsuite to make sure all of the fun str*
> > and mem* functions are sane and operate as they expected, but rather
> > than having just Blackfin benefit here, i'd like to see this pushed
> > upstream ...
>
> I would like to see this too. I wrote a couple of unit tests
> during x86-64 development too and they would be handy
> to have in a central place.
>
> The SUSE kernel also has a crasher module that exercises the
> allocators and is pretty useful to stress code in general.
>
> Disadvantage is that test code tends to be hackish and ugly
> and very specialized and will probably not pass standard review
> procedures.
Then this makes perfect sense to actually have it in the proper places, and
have it go through the standard review process. I have run into problems
recently, where ugly hackish tests seem never really catch actual faults.
(Since they were written by the same person who wrote the driver - Design a
fault in, code the fault, test the fault is there).
With many eyes - and all that stuff - hopefully we would catch things.
> Also the rt people seem to have pushed a couple of tests in already,
> but I always hated it that they're in the standard directories.
> RCU also has, in fact they added a "eat all my CPU in tests"
> CONFIG option. Just making that dependent on a CONFIG_UNIT_TESTS
> would be a good change in itself. And then there are the slab
> failure inducers of course.
Putting them close to the standard directories might be OK - keeping the tests
close to the thing that is being tested may help with maintainers keeping an
eye on things that might be missed otherwise.
Maybe these types of tests should go into a separate directory - power on self
tests (post) directories in the respective places? ./kernel/post
and ./drivers/xxxx/post?
So Mike's proposed string tests would go into ./lib/post/strings.c
When things pass - you get a single indication (all tests for unit X passed)
or a pass for each test?
-Robin
On 9/2/07, Andi Kleen <[email protected]> wrote:
> On Sun, Sep 02, 2007 at 11:15:23AM -0400, Mike Frysinger wrote:
> > there is still the ABI issue ... code written in kernel space in pure
> > asm cannot always be compiled in userspace and work properly/the same
>
> Is that a blackfin weirdness?
yes, Blackfin is weird in this respect due to being a no-mmu and
wanting ELF, but i was being sufficiently general to bolster my claims
and to leave the door open for any other port out there i'm not
familiar with and any port that comes along
> > > The other issue to test some of them properly you need unmapped pages
> > > etc. That gets much easier to do in user space. There are some other
> > > issues.
> >
> > you mean testing boundary overflows ? can be handled with canaries
> > rather than segfaults i imagine ...
>
> Not for reads, no.
true, but generally the string functions arent searching for crashes
in the boundary conditions ... they just happen to sometimes find them
:)
-mike
On 9/4/07, Mike Frysinger <[email protected]> wrote:
> On 9/2/07, Andi Kleen <[email protected]> wrote:
> > On Sun, Sep 02, 2007 at 11:15:23AM -0400, Mike Frysinger wrote:
> > > there is still the ABI issue ... code written in kernel space in pure
> > > asm cannot always be compiled in userspace and work properly/the same
> >
> > Is that a blackfin weirdness?
>
> yes, Blackfin is weird in this respect due to being a no-mmu and
> wanting ELF, but i was being sufficiently general to bolster my claims
> and to leave the door open for any other port out there i'm not
> familiar with and any port that comes along
actually, there was another case i was thinking of ... it isnt
uncommon for architectures to use compiler flags which affect the ABI
and thus make userland not a 1-to-1 test example for the kernel ...
consider the regparam stuff we use on i386
-mike