Hi,
The Linux Test Project test suite has been released for the month of
FEBRUARY 2011. Please see ltp/INSTALL file carefully, as there has
been multiple changes for building/installing the test suite after the
recent changes in Makefile infrastructure.
The latest version of the test-suite contains 3000+ tests for the Linux
OS and can be found at:
http://ltp.sourceforge.net/,
Latest happenings in LTP can also be found at:
http://ltp.sourceforge.net/wiki/,
http://ltp.sourceforge.net/wikiArchives.php, and,
IRC: irc.freenode.org #ltp.
Our web site also contains other information such as:
- A Linux test tools matrix
- Technical papers
- How To's on Linux testing
- Code coverage analysis tool.
We would encourage the community to post results to
[email protected],
patches, new tests, bugs or comments/questions [email protected],
http://sourceforge.net/tracker/?func=add&group_id=3382&atid=103382 (for
New Bug(s)),
http://sourceforge.net/tracker/?func=add&group_id=3382&atid=303382 (for
New Patch(s)),
http://sourceforge.net/tracker/?func=add&group_id=3382&atid=353382 (for
New Feature Request(s))
**Note: Some pending patches will be checked in for next monthś release.
Happy testing,
Regards,
Shubham
On Wed, Mar 2, 2011 at 12:40 PM, Shubham Goyal
<[email protected]> wrote:
> Hi,
>
> The Linux Test Project test suite has been released for the month of
> FEBRUARY 2011. Please see ltp/INSTALL file carefully, as there has
> been multiple changes for building/installing the test suite after the
> recent changes in Makefile infrastructure.
Wouldn't make sense to integrate this test suite in the kernel source tree?
Regards,
Paolo
On Wed, 2011-03-02 at 13:06 +0100, Paolo Ciarrocchi wrote:
> On Wed, Mar 2, 2011 at 12:40 PM, Shubham Goyal
> <[email protected]> wrote:
> > Hi,
> >
> > The Linux Test Project test suite has been released for the month of
> > FEBRUARY 2011. Please see ltp/INSTALL file carefully, as there has
> > been multiple changes for building/installing the test suite after the
> > recent changes in Makefile infrastructure.
>
> Wouldn't make sense to integrate this test suite in the kernel source tree?
There was discussion like this some few years back. The idea was to get
some core tests from LTP to the kernel source tree. But then the idea
was dropped probably to avoid maintenance overhead ;-)
Regards--
Subrata
>
> Regards,
> Paolo
On Wed, Mar 2, 2011 at 4:23 AM, Subrata Modak
<[email protected]> wrote:
> On Wed, 2011-03-02 at 13:06 +0100, Paolo Ciarrocchi wrote:
>> On Wed, Mar 2, 2011 at 12:40 PM, Shubham Goyal
>> <[email protected]> wrote:
>> > Hi,
>> >
>> > The Linux Test Project test suite has been released for the month of
>> > FEBRUARY 2011. Please see ltp/INSTALL file carefully, as there has
>> > been multiple changes for building/installing the test suite after the
>> > recent changes in Makefile infrastructure.
>>
>> Wouldn't make sense to integrate this test suite in the kernel source tree?
>
> There was discussion like this some few years back. The idea was to get
> some core tests from LTP to the kernel source tree. But then the idea
> was dropped probably to avoid maintenance overhead ;-)
Putting LTP in the kernel.org sources really doesn't make sense for
the following reasons:
1. LTP isn't really tied to a single kernel release.
2. LTP isn't the only test project out there for Linux.
3. LTP has more stuff than it needs to have for testing out the kernel
(well, it did more in the past before I started cleaning it up in the
past couple of months).
4. Maintaining it will become a political bloodbath for both parties
as Linux is loosely managed by Linus et all, and LTP has been largely
developed by SGI and maintained by IBM and a few other parties like
Fujitsu, Nokia, Redhat, etc.
5. Integrating LTP into Kbuild, etc would probably be non-trivial due
to the size of LTP (but it might be easier after the Makefile
restructuring I did a year and a half ago).
That being said, if Linux devs took the initiative and submitted
testcases that either illustrated past regressions in the kernel,
feature tested enhancements, and submitted documentation that actually
described their changes to the Linux kernel and the manpages project,
I would take this over having LTP in the linux sources because right
now things largely work because the Linux sources haven't really been
rototilled since 2.4 -> 2.6, but they are somewhat bitrotted and when
Linux rototills its sources again, we'll have to go through
rototilling our stuff as well.
Right now multiple QA engineers are sort of playing whack-a-mole
trying to figure out requirements and submit tests to LTP, and since
they don't have 100% context into the actual problem, some information
is lost in translation when the tests are submitted. This isn't
desirable.
Thanks,
-Garrett
On Wednesday, March 02, 2011 14:45:38 Garrett Cooper wrote:
> On Wed, Mar 2, 2011 at 4:23 AM, Subrata Modak
>
> <[email protected]> wrote:
> > On Wed, 2011-03-02 at 13:06 +0100, Paolo Ciarrocchi wrote:
> >> On Wed, Mar 2, 2011 at 12:40 PM, Shubham Goyal
> >>
> >> <[email protected]> wrote:
> >> > Hi,
> >> >
> >> > The Linux Test Project test suite has been released for the month of
> >> > FEBRUARY 2011. Please see ltp/INSTALL file carefully, as there has
> >> > been multiple changes for building/installing the test suite after the
> >> > recent changes in Makefile infrastructure.
> >>
> >> Wouldn't make sense to integrate this test suite in the kernel source
> >> tree?
> >
> > There was discussion like this some few years back. The idea was to get
> > some core tests from LTP to the kernel source tree. But then the idea
> > was dropped probably to avoid maintenance overhead ;-)
>
> Putting LTP in the kernel.org sources really doesn't make sense for
> the following reasons:
>
> 1. LTP isn't really tied to a single kernel release.
> 2. LTP isn't the only test project out there for Linux.
> 3. LTP has more stuff than it needs to have for testing out the kernel
> (well, it did more in the past before I started cleaning it up in the
> past couple of months).
> 4. Maintaining it will become a political bloodbath for both parties
> as Linux is loosely managed by Linus et all, and LTP has been largely
> developed by SGI and maintained by IBM and a few other parties like
> Fujitsu, Nokia, Redhat, etc.
> 5. Integrating LTP into Kbuild, etc would probably be non-trivial due
> to the size of LTP (but it might be easier after the Makefile
> restructuring I did a year and a half ago).
these are all very good reasons. additional points:
- ltp is pretty fsckin huge
- ltp often times tests both sides of the userspace API/ABI -- between the
kernel and the C library, and the C library and end applications
-mike
> There was discussion like this some few years back. The idea was to get
> some core tests from LTP to the kernel source tree. But then the idea
> was dropped probably to avoid maintenance overhead ;-)
Where are the places in kernel source tree for those?
Those days, there just too many tests and testing projects for kernel like
LTP, autotest, xfstests and so on. Why not have somewhere to collabrate and
then to extract the best?
LTP has so many goals and focus which isn't going to be only to test kernel
any more and it is increasing difficult to support so many distros, kernel
versions, and so on.
There are some CORE tests like memory management tests, ksm, oom etc have
benefit from the developers' bless and review. It also need to be updated
to keep the tests relevant to the current git tree, since the features/specs
are changing consistent inside the kernel.
This could be also useful to improve the kernel quality by providing test
code inside the kernel tree that to be used during the code review process
that for example, a ksm patchset needs to pass that particular sanity tests
in order to catch the regression. It provide benefit that when the changelog
said that it passed the ksm tests inside the kernel, we knew exactly what it is
without needing to sync up with another project like LTP.
In term of maintenance, it needs to be selectively which tests need to be
inside the kernel. There should ideally have a dedicated maintainer from the
testing point of view to review them. The criteria can be something like,
1) purely purpose of the tests are to test kernel written in C with the kernel
coding style. Userspace and integration tests should be better to put into
LTP and other projects.
2) tests need to pass sub-system maintainers' review that for example, ksm
tests need MM sub-system maintainers' review-by and sign-off-by and alike.
3) they need to be working with and sync up to the latest git version.
4) they have to be proved to be the best tests we can have to test those
particular kernel code. There are many tests in LTP, but there are also
many duplicated tests as well. Those need to be solved when considering to
be moved inside the kernel source.
5) they should really be functional testing. Non-functional tests like stress
or performance tests are usually more complex to setup hence defeat the
purpose of quick regression checking.
Once we have those tests in-place, the next step to improve the kernel quality
is to have more patches had Tested-by tags before been accepted in the kernel
git tree. Those testers can simply use the non-ambiguious references for tests
provided by those in-kernel tests.
In addition, those could be a follow-up items with the kernel regression reports
that after fixed/analyzed a regression in kernel, the next natural thing to do
is to fix/add missing tests to close this gap in the future by providing efficient
tests to our users if all possible.
CAI Qian
Hi!
I'm not completely against this. At least LTP and possibly other
testsuites would gain by attention from developers (it seems that the QA
people themselves are the only ones interested). But some things are not
just as simple as you see them (at least from my point of view).
> Those days, there just too many tests and testing projects for kernel like
> LTP, autotest, xfstests and so on. Why not have somewhere to collabrate and
> then to extract the best?
That is IMHO just too much work. You would need somebody to extract it
and then somebody to keep things in sync.
> LTP has so many goals and focus which isn't going to be only to test kernel
> any more and it is increasing difficult to support so many distros, kernel
> versions, and so on.
Frankly, LTP has maybe quite a lot of goals, but has a very little
manpower, so just now it's mostly broken and rotten code. I would
rather focus on cleaning up and fixing up the LTP, dropping ambiguous
tests and so. We already did some progress in that area (I tend to say
"we" but besides random contributions we have like three or five people).
> There are some CORE tests like memory management tests, ksm, oom etc have
> benefit from the developers' bless and review. It also need to be updated
> to keep the tests relevant to the current git tree, since the features/specs
> are changing consistent inside the kernel.
That's true indeed.
> This could be also useful to improve the kernel quality by providing test
> code inside the kernel tree that to be used during the code review process
> that for example, a ksm patchset needs to pass that particular sanity tests
> in order to catch the regression. It provide benefit that when the changelog
> said that it passed the ksm tests inside the kernel, we knew exactly what it is
> without needing to sync up with another project like LTP.
Well, I don't see what would be gained by merging parts of the LTP into
kernel tree. As I said before, this would probably lead to splitting of
the forces (and not that we have a lot to split anyway). LTP already has
directory called testcases/kernel/, LTP is in the git repository and we
have a mailing list. All that is needed is people start noticing that
we are here.
> In term of maintenance, it needs to be selectively which tests need to be
> inside the kernel. There should ideally have a dedicated maintainer from the
> testing point of view to review them. The criteria can be something like,
>
> 1) purely purpose of the tests are to test kernel written in C with the kernel
> coding style. Userspace and integration tests should be better to put into
> LTP and other projects.
I don't think that it's easy to say if some tests are testing
kernel/userspace. Sometimes the line isn't that clear.
> 2) tests need to pass sub-system maintainers' review that for example, ksm
> tests need MM sub-system maintainers' review-by and sign-off-by and alike.
Well, requiring maintainers to sign-off your tests is kind of dull. That
would probably block the tests from being accepted just because
maintainers don't care too much/have different things to do.
> 3) they need to be working with and sync up to the latest git version.
> 4) they have to be proved to be the best tests we can have to test those
> particular kernel code. There are many tests in LTP, but there are also
> many duplicated tests as well. Those need to be solved when considering to
> be moved inside the kernel source.
You can't easily prove that something is best ;).
> 5) they should really be functional testing. Non-functional tests like stress
> or performance tests are usually more complex to setup hence defeat the
> purpose of quick regression checking.
>
> Once we have those tests in-place, the next step to improve the kernel quality
> is to have more patches had Tested-by tags before been accepted in the kernel
> git tree. Those testers can simply use the non-ambiguious references for tests
> provided by those in-kernel tests.
Once again, LTP does exist so reference to LTP is not ambiguous. Yes,
it's, for historical reasons, hosted on sourceforge rather than
kernel.org. But there it is.
> In addition, those could be a follow-up items with the kernel regression reports
> that after fixed/analyzed a regression in kernel, the next natural thing to do
> is to fix/add missing tests to close this gap in the future by providing efficient
> tests to our users if all possible.
That's right thing to do, but once more, LTP is here for you ;). So my
conclusion is that the major point here is make LTP more visible among
linux hackers. On that note, I'm planing to prepare some presentation to
let the people know what is the current state and what we are doing to
get it better.
--
Cyril Hrubis
[email protected]
> Well, I don't see what would be gained by merging parts of the LTP into
> kernel tree. As I said before, this would probably lead to splitting of
> the forces (and not that we have a lot to split anyway). LTP already has
> directory called testcases/kernel/, LTP is in the git repository and we
> have a mailing list. All that is needed is people start noticing that
> we are here.
Then, the approach to merge parts of LTP to kernel is to say "Here we are,
please accept our best". On the other hand, I have noticed that there are
many developers tend to have test code in their kernel submit changelog
which isn't it better to make life easier for them to add those testing
code in a proper place in kernel which in-turn to benefit in a long run.
> I don't think that it's easy to say if some tests are testing
> kernel/userspace. Sometimes the line isn't that clear.
There are C code as in kernel coding style. Scripting code like Bash, Perl
better to re-written in C that in a long run when there are something like
thousands of tests to run that performance/scalabitlies/maintenence is going
to matter just like to write an OS.
> Well, requiring maintainers to sign-off your tests is kind of dull. That
> would probably block the tests from being accepted just because
> maintainers don't care too much/have different things to do.
The idea is to raise a bar to get the best out of it. If maintainers don't
care too much about the testing right now that is fine. There are many people
they do care. A particular subsystem maintainer and its tests maintainer
aren't necessary to be the same person because subsystem maintainer isn't
necessary to be the best one to find/acknowledge defeats for code he maintained.
> You can't easily prove that something is best ;).
The best will at lest be reviewed by eyes from the kernel community and
experts, and will be the one to be accepted by the community.
> Once again, LTP does exist so reference to LTP is not ambiguous. Yes,
> it's, for historical reasons, hosted on sourceforge rather than
> kernel.org. But there it is.
By accepted into the kernel, it certainly make it easier to reference
without dealing with two projects and trees.
CAI Qian
On Fri, Mar 4, 2011 at 12:21 AM, CAI Qian <[email protected]> wrote:
>
>> Well, I don't see what would be gained by merging parts of the LTP into
>> kernel tree. As I said before, this would probably lead to splitting of
>> the forces (and not that we have a lot to split anyway). LTP already has
>> directory called testcases/kernel/, LTP is in the git repository and we
>> have a mailing list. All that is needed is people start noticing that
>> we are here.
> Then, the approach to merge parts of LTP to kernel is to say "Here we are,
> please accept our best". On the other hand, I have noticed that there are
> many developers tend to have test code in their kernel submit changelog
> which isn't it better to make life easier for them to add those testing
> code in a proper place in kernel which in-turn to benefit in a long run.
>
>> I don't think that it's easy to say if some tests are testing
>> kernel/userspace. Sometimes the line isn't that clear.
> There are C code as in kernel coding style. Scripting code like Bash, Perl
> better to re-written in C that in a long run when there are something like
> thousands of tests to run that performance/scalabitlies/maintenence is going
> to matter just like to write an OS.
>
>> Well, requiring maintainers to sign-off your tests is kind of dull. That
>> would probably block the tests from being accepted just because
>> maintainers don't care too much/have different things to do.
> The idea is to raise a bar to get the best out of it. If maintainers don't
> care too much about the testing right now that is fine. There are many people
> they do care. A particular subsystem maintainer and its tests maintainer
> aren't necessary to be the same person because subsystem maintainer isn't
> necessary to be the best one to find/acknowledge defeats for code he maintained.
>
>> You can't easily prove that something is best ;).
> The best will at lest be reviewed by eyes from the kernel community and
> experts, and will be the one to be accepted by the community.
>
>> Once again, LTP does exist so reference to LTP is not ambiguous. Yes,
>> it's, for historical reasons, hosted on sourceforge rather than
>> kernel.org. But there it is.
> By accepted into the kernel, it certainly make it easier to reference
> without dealing with two projects and trees.
I'm sorry for even starting this bikeshed discussion. Let's just bury
the hatchet and get back to work on more fruitful things.
Thanks,
-Garrett
On Mar 2, 2011, at 8:52 PM, CAI Qian wrote:
> Those days, there just too many tests and testing projects for kernel like
> LTP, autotest, xfstests and so on. Why not have somewhere to collabrate and
> then to extract the best?
Part of the problem is that every single testing project has different goals and
priorities. For example xfstests is maintained by the XFS folks, as well as people
from some of the other file system development efforts (the ext4 one in particular,
thanks to people like Eric Sandeen), to test file systems.
At least at one point, I had heard a complaint that LTP was more focused on
increasing test coverage as measured by a code coverage tool in the kernel
than it was about about covering edge conditions, or races. There's nothing
wrong with that, per se, and I don't know if it was true then or now, but it's a very
different focus from one which is focused increasing the data reliability of file
systems, quickly and efficiently.
And then there's the LSB test suites, which is really code at testing correctness
from a standards perspective, which is a different focus yet again from the LTP
and xfstests approach.
Bottom line, I'm a big fan of having different test suites, with different philosophies.
Each philosophy has its strengths and blind spots, and so a problem that might
be missed by one test suite might get caught by another.
The only real problem is an operational one. There are some programs which
are used by both LTP and xfstests, and changes that is made in one, don't
necessarily get propagated to the other unless someone manually does it.
But I think we can solve that without trying to merge all of these tests into a
single Grand Unified Test Suite.
-- Ted
On Fri, Mar 04, 2011 at 01:58:45PM -0500, Theodore Tso wrote:
>
> On Mar 2, 2011, at 8:52 PM, CAI Qian wrote:
>
> > Those days, there just too many tests and testing projects for kernel like
> > LTP, autotest, xfstests and so on. Why not have somewhere to collabrate and
> > then to extract the best?
>
> Part of the problem is that every single testing project has different goals and
> priorities. For example xfstests is maintained by the XFS folks, as well as people
> from some of the other file system development efforts (the ext4 one in particular,
> thanks to people like Eric Sandeen), to test file systems.
Perhaps we need to get developer's tests into the kernel. We now have a
tools/testing directory lets use it.
>
> At least at one point, I had heard a complaint that LTP was more focused on
> increasing test coverage as measured by a code coverage tool in the kernel
> than it was about about covering edge conditions, or races. There's nothing
> wrong with that, per se, and I don't know if it was true then or now, but it's a very
> different focus from one which is focused increasing the data reliability of file
> systems, quickly and efficiently.
>
> And then there's the LSB test suites, which is really code at testing correctness
> from a standards perspective, which is a different focus yet again from the LTP
> and xfstests approach.
>
> Bottom line, I'm a big fan of having different test suites, with different philosophies.
> Each philosophy has its strengths and blind spots, and so a problem that might
> be missed by one test suite might get caught by another.
>
> The only real problem is an operational one. There are some programs which
> are used by both LTP and xfstests, and changes that is made in one, don't
> necessarily get propagated to the other unless someone manually does it.
> But I think we can solve that without trying to merge all of these tests into a
> single Grand Unified Test Suite.
>
How about having the developers write tests and place them in the
tools/testing directory and then the folks at LTP and xfstests et al.
can update their code with the code from the kernel.
Currently only ktest.pl exists in this directory. I use it constantly
and post bugs that it finds. It focuses on just the building and booting
of a kernel. It can run several randconfigs, do bisects and such, but it
just has a single command to do any tests. This command is just a shell
command the user can put in. I leave what tests to run to the user. Thus
LTP could be the test that gets kicked off.
For example: I have this script I run on my x86 box:
TEST_START ITERATE 10
TEST_TYPE = test
BUILD_TYPE = randconfig
MIN_CONFIG = /home/rostedt/work/autotest/configs/mitest/config-mitest-net
CHECKOUT = origin/master
TEST = ssh root@mitest /work/c/hackbench_32 50
TEST_START ITERATE 10
TEST_TYPE = boot
BUILD_TYPE = randconfig
MIN_CONFIG = /home/rostedt/work/autotest/configs/mitest/config-mitest-min
MAKE_CMD = make ARCH=i386
DEFAULTS
REBOOT_ON_ERROR = 0
POWEROFF_ON_ERROR = 1
POWEROFF_ON_SUCCESS = 1
REBOOT_ON_SUCCESS = 0
DIE_ON_FAILURE = 0
STORE_FAILURES = /home/rostedt/work/autotest/nobackup/failures
POWEROFF_AFTER_HALT = 60
CLEAR_LOG = 1
MIN_CONFIG = /home/rostedt/work/autotest/configs/mitest/config-mitest-min
SSH_USER = root
BUILD_DIR = /home/rostedt/work/autotest/nobackup/linux-test.git
OUTPUT_DIR = /home/rostedt/work/autotest/nobackup/mitest
BUILD_TARGET = arch/x86/boot/bzImage
TARGET_IMAGE = /boot/vmlinuz-test
POWER_CYCLE = /home/rostedt/work/autotest/cycle-mxtest
CONSOLE = nc -d fedora 3001
LOCALVERSION = -test
GRUB_MENU = Test Kernel
MAKE_CMD = distmake-32 ARCH=i386
POWER_OFF = /home/rostedt/work/autotest/poweroff-mxtest
BUILD_OPTIONS = -j20
LOG_FILE = /home/rostedt/work/autotest/nobackup/mitest/mitest.log
TEST = ssh root@mitest cat /debug/tracing/trace
ADD_CONFIG = /home/rostedt/work/autotest/configs/config-broken /home/rostedt/work/autotest/config-general
Each command is documented in the samples.conf that is also in that
directory.
The above test runs two sets of tests. The first runs randconfig 10
times with a minimum config that will allow my box to have a network
connection, and after it boots, it runs hackbench.
The second test runs ten randconfig builds 10 times and just makes sure
the box can boot.
I'm working on getting to run commands via the console so I do not need
the network active to run tests.
Once could change the TEST = ... to run LTP tests, or anything else.
Having tests to run that I can add to my automated testing would be
helpful. I could build randconfigs and run these tests making sure that
they work with different configurations.
Maybe it would also be helpful to have the CONFIGs that are needed by
the tests. It would not make sense to test iptables if iptables is not
configured ;)
-- Steve
> Perhaps we need to get developer's tests into the kernel. We now have a
> tools/testing directory lets use it.
There are a few test code span over other directories too.
Documentation/vm/hugepage-mmap.c
Documentation/vm/hugepage-shm.c
Documentation/vm/map_hugetlb.c
...
CAI Qian
On Mon, 2011-03-07 at 21:33 -0500, CAI Qian wrote:
> > Perhaps we need to get developer's tests into the kernel. We now have a
> > tools/testing directory lets use it.
> There are a few test code span over other directories too.
> Documentation/vm/hugepage-mmap.c
> Documentation/vm/hugepage-shm.c
> Documentation/vm/map_hugetlb.c
These all look like "Examples", which means they should probably be
moved to samples? As "samples" can be compiled on boot. But perhaps
because they are user space programs its fine to keep them here?
-- Steve
> ...
> > > Perhaps we need to get developer's tests into the kernel. We now
> > > have a
> > > tools/testing directory lets use it.
> > There are a few test code span over other directories too.
> > Documentation/vm/hugepage-mmap.c
> > Documentation/vm/hugepage-shm.c
> > Documentation/vm/map_hugetlb.c
>
> These all look like "Examples", which means they should probably be
> moved to samples? As "samples" can be compiled on boot. But perhaps
> because they are user space programs its fine to keep them here?
Yes, but also test cases to check if basic hugetlb code is working.
Well, the problem is that they are all over the place, and some of
good test cases are within commit logs as well which makes things
harder to maintain, use...
CAI Qian