Our team would like to assist the community in quickly identifying
patches that provide
performance improvements or regressions in the 2.5 kernel tree. The way
to do this
will be to run a nightly regression test suite against the current bk
tree, and then compare
the results against the previous night's results, showing the
differences. Additionally,
also comparing against the 2.5 point release.
We have dedicated a machine and thrown together some scripts that will grab
and build the latest kernel files, execute the regression suite,
collecting (hopefully)
enough system state information to allow meaningful analysis of any peculiar
results encountered.
Here are links to the current regression results obtained:
2.5.70 vs 2.5.70-bk1:
http://www.ibm.com/developerworks/oss/linuxperf/regression/2.5.70-bk1/2.5.70-vs-2.5.70-bk1/
2.5.70 vs 2.5.70-bk2:
http://www.ibm.com/developerworks/oss/linuxperf/regression/2.5.70-bk2/2.5.70-vs-2.5.70-bk2/
2.5.70-bk1 vs 2.5.70-bk2
http://www.ibm.com/developerworks/oss/linuxperf/regression/2.5.70-bk2/2.5.70-bk1-vs-2.5.70-bk2/
The regression suite executes in about 7.5 hours currently. We would
like to keep
the execution time below 12 hours, so when a problem is encountered, we
will have
time to recover without falling behind on the daily snapshots. We have
attempted
to strike a balance between test execution time and test coverage. Work
is still
ongoing in the area to provide the best balance and maintain repeatability.
Currently the regression suite operates on the 2.5 kernel bk tree. We do
plan on
adding another machine that will perform similiar regression comparisons
for the
-mm and -mjb patches.
Please bear in mind this is work in progress and there might be a few
rough edges.
However, with your input, we feel it can provide a useful function.
Please do not
hesitate to provide feedback or suggestions on improvements including
content
and presentation.
Mark Peloquin
IBM Linux performance team
Mark Peloquin <[email protected]> writes:
> We have dedicated a machine and thrown together some scripts that will grab
> and build the latest kernel files, execute the regression suite,
> collecting (hopefully)
> enough system state information to allow meaningful analysis of any peculiar
> results encountered.
How about doing a LTP run too with some difference file for new FAILs/BROKs ?
That's not strictly a benchmark, but would help catching regressions
quickly.
I notice your benchmark mix is very IO heavy, it would be nice to test other
aspects of the system too. Perhaps lmbench and reaim compute workload?
It would be nice if we had a new linux-testresults list where such
updates could be posted regularly. I don't think it belong on l-k
because it would be too noisy. Perhaps such a list could be added to
vger. David, what do you think?
-Andi
From: Andi Kleen <[email protected]>
Date: 29 May 2003 23:11:17 +0200
David, what do you think?
Would it have a single poster?
On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
> From: Andi Kleen <[email protected]>
> Date: 29 May 2003 23:11:17 +0200
>
> David, what do you think?
>
> Would it have a single poster?
OSDL, Mark's IBM team and possible LTP ?
I assume there will be more once the list exists; automated regression
tests seem to be currently in fashion.
-Andi
On Thu, 29 May 2003 23:29:29 +0200 Andi Kleen <[email protected]> wrote:
| On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
| > From: Andi Kleen <[email protected]>
| > Date: 29 May 2003 23:11:17 +0200
| >
| > David, what do you think?
| >
| > Would it have a single poster?
|
| OSDL, Mark's IBM team and possible LTP ?
|
| I assume there will be more once the list exists; automated regression
| tests seem to be currently in fashion.
If DaveM doesn't want to do it, I think that we can do it.
(I say without checking.... :)
--
~Randy
From: "Randy.Dunlap" <[email protected]>
Date: Thu, 29 May 2003 14:38:20 -0700
On Thu, 29 May 2003 23:29:29 +0200 Andi Kleen <[email protected]> wrote:
| On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
| > Would it have a single poster?
|
| OSDL, Mark's IBM team and possible LTP ?
|
| I assume there will be more once the list exists; automated regression
| tests seem to be currently in fashion.
If DaveM doesn't want to do it, I think that we can do it.
(I say without checking.... :)
Please do :-)
The issue is that I'm easier about adding a new list if I can
restrict the poster list.
On Thu, 2003-05-29 at 14:29, Andi Kleen wrote:
> On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
> > From: Andi Kleen <[email protected]>
> > Date: 29 May 2003 23:11:17 +0200
> >
> > David, what do you think?
> >
> > Would it have a single poster?
>
> OSDL, Mark's IBM team and possible LTP ?
>
> I assume there will be more once the list exists; automated regression
> tests seem to be currently in fashion.
>
> -Andi
> -
OSDL has a linux stabilization web page where several tests are run
automatically when a new kernel is built. It currently runs Linus'
kernel as well as the -mm series. We do run LTP, I/O tests, memory
tests, reaim, and database tests as part of an automated regression
run. Some of you are familiar with the web page, but for those who are
not, it is located here: http://www.osdl.org/projects/linstab/
In addition, there are links to other sites, most notably IBM's LTC
test results.
We have just completed a physical move to a new office and we believe
we have all of our systems working again, so test results for the
latest kernels are a bit behind. We hope to have completed runs for
all tests by the weekend. Note, we are experiencing some test failures
but we suspect it is due to the move and not the kernels at the moment.
--
Craig Thomas
[email protected]
On Thu, May 29, 2003 at 11:11:17PM +0200, Andi Kleen wrote:
> It would be nice if we had a new linux-testresults list where such
> updates could be posted regularly. I don't think it belong on l-k
> because it would be too noisy. Perhaps such a list could be added to
> vger. David, what do you think?
The OSDL has a serious amount of automated testing we could point the
results of to a separate list if it is created.
Right now we avoid pointing that sort of thing to l-k because it would
drive people nuts. On average we complete 40+ tests a day.
With all the testing efforts going on, a central list to post and
analyze results would be good. People interested in helping out could
easily work with testers to look for trends and help with root cause
analysis.
When results are found to contain significant data, we can always notify l-k.
-Nathan Dabney
On Thu, 2003-05-29 at 14:48, David S. Miller wrote:
> From: "Randy.Dunlap" <[email protected]>
> Date: Thu, 29 May 2003 14:38:20 -0700
>
> On Thu, 29 May 2003 23:29:29 +0200 Andi Kleen <[email protected]> wrote:
>
> | On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
> | > Would it have a single poster?
> |
> | OSDL, Mark's IBM team and possible LTP ?
> |
> | I assume there will be more once the list exists; automated regression
> | tests seem to be currently in fashion.
>
> If DaveM doesn't want to do it, I think that we can do it.
> (I say without checking.... :)
>
> Please do :-)
>
> The issue is that I'm easier about adding a new list if I can
> restrict the poster list.
>
OSDL has a mail list that is used to discuss the stability of the linux
kernel. This would be a perfect list to use for posting test results.
The list name is [email protected]. It is a public list administered by
OSDL. To subscribe: http://www.osdl.org/mailman/listinfo/linstab
--
Craig Thomas
[email protected]
> OSDL has a mail list that is used to discuss the stability of the linux
> kernel. This would be a perfect list to use for posting test results.
> The list name is [email protected]. It is a public list administered by
> OSDL. To subscribe: http://www.osdl.org/mailman/listinfo/linstab
That's fairly obscure (Nobody knew of it before). Perhaps a well publicized
list on vger would be better.
-Andi
> OSDL has a linux stabilization web page where several tests are run
[...] Would you be willing to change your scripts to report
any new results to this new list?
-Andi
> > OSDL has a linux stabilization web page where several tests are run
>
> [...] Would you be willing to change your scripts to report
> any new results to this new list?
>
I would be very interested in this if it leads to more people *looking* at the
tests results.
Automating this stuff is the easy part - getting intelligence out of the
results is harder.
The more eyeballs we can get to look, the easier this gets.
If a new list, or better use of an old list will help, i'll change whatever is
necessary.
If people like the Web, but don't like our paper layout, i'll change that too.
cliffw
As
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
On Fri, May 30, 2003 at 12:03:54AM +0200, Andi Kleen wrote:
> > OSDL has a linux stabilization web page where several tests are run
>
> [...] Would you be willing to change your scripts to report
> any new results to this new list?
The linux stabilization web page uses results from the STP runs I
mentioned (40+ per day). The STP emails results summaries after test
runs so we could easily redirect the results to this new list.
-Nathan Dabney
> > OSDL has a mail list that is used to discuss the stability of the linux
> > kernel. This would be a perfect list to use for posting test results.
> > The list name is [email protected]. It is a public list administered by
> > OSDL. To subscribe: http://www.osdl.org/mailman/listinfo/linstab
>
> That's fairly obscure (Nobody knew of it before). Perhaps a well publicized
> list on vger would be better.
Perhaps - though we can publicize any new list.
We're content to leave the decision to DaveM and the list team -
if they don't want the extra work, we're always glad to help.
cliffw
OSDL
>
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Andi Kleen wrote:
>Mark Peloquin <[email protected]> writes:
>
>
>
>>We have dedicated a machine and thrown together some scripts that will grab
>>and build the latest kernel files, execute the regression suite,
>>collecting (hopefully)
>>enough system state information to allow meaningful analysis of any peculiar
>>results encountered.
>>
>>
>
>How about doing a LTP run too with some difference file for new FAILs/BROKs ?
>That's not strictly a benchmark, but would help catching regressions
>quickly.
>
I'm under the impression that LTP and other test efforts seemed to focus
more on functional evaluation, which is fine. We are trying to focus
purely on the performance differences seen from day to day.
>
>I notice your benchmark mix is very IO heavy, it would be nice to test other
>aspects of the system too. Perhaps lmbench and reaim compute workload?
>
Your correct. We're just getting started with this effort and we used
this mix to get things going. Once ppl are happy with the presentation
of data, we planned to add more tests to provide a more balanced mix.
But since you asked, we have added lmbench to our -bk3 regression run. :)
Mark
David S. Miller wrote:
> From: Andi Kleen <[email protected]>
> Date: 29 May 2003 23:11:17 +0200
>
> David, what do you think?
>
>Would it have a single poster?
>
Our intention was to have one "main" poster and another person for backup.
Mark
Nathan wrote:
>On Thu, May 29, 2003 at 11:11:17PM +0200, Andi Kleen wrote:
>
>
>>It would be nice if we had a new linux-testresults list where such
>>updates could be posted regularly. I don't think it belong on l-k
>>because it would be too noisy. Perhaps such a list could be added to
>>vger. David, what do you think?
>>
>>
>
>The OSDL has a serious amount of automated testing we could point the
>results of to a separate list if it is created.
>
>Right now we avoid pointing that sort of thing to l-k because it would
>drive people nuts. On average we complete 40+ tests a day.
>
>With all the testing efforts going on, a central list to post and
>analyze results would be good. People interested in helping out could
>easily work with testers to look for trends and help with root cause
>analysis.
>
>When results are found to contain significant data, we can always notify l-k.
>
Easy of viewing should be considered. We have tried to show a high level
summary that allows the users to quickly, looking in one place,
determine if any significant data is found. When the users seems
something of interest, they only need follow the links to see the
details. Its shouldn't be necessary for users to sift through one email
for each test. If finding signficant data was easier, and I think it can
be made easier, users would look at it themselves and there wouldn't be
the need to have to notify l-k.
I'm not trying to be competetive here. I just think results and
comparisons can be made that covers a large amount of tests in a single
page or note. One note per day does not IMHO seem like too much. That
note can always be the "tip of the iceberg" pointing to many other
things. Thus those not interested can simply skip that note.
Mark
Andi Kleen wrote:
>>OSDL has a linux stabilization web page where several tests are run
>>
>>
>
>[...] Would you be willing to change your scripts to report
>any new results to this new list?
>
>-Andi
>
We have not automated the posting process ... yet, and will be happy to
post whereever is acceptable.
Mark
On May 29, 2003 17:48 -0500, Mark Peloquin wrote:
> Your correct. We're just getting started with this effort and we used
> this mix to get things going. Once ppl are happy with the presentation
> of data, we planned to add more tests to provide a more balanced mix.
> But since you asked, we have added lmbench to our -bk3 regression run. :)
Mark, it would be nice to get a graph of the combined results for each
test. Something like:
tiobench sequential write rate
| +++++++++++++++++ + = -mm-ext3
M | ++++++++++++++++++++++++++***************** * = linus-ext3
B | +++++++***************** ###### # = -ac-ext3
/ | . = -mm-XFS
s | = = -ac-XFS
| ********* etc
|
+----------------------------------------------------
date
This allows at-a-glance trends for each group of tests and (as in the
example above you could see easily when a performance bug was added
and fixed in -ac before in the linus kernel, for example). Probably
having all of the comparable results on the same page, or even in the
same graph results is a win.
Bonus points if you can click on a spot in the graph and get the results
page for that date/test ;-).
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
> On May 29, 2003 17:48 -0500, Mark Peloquin wrote:
> > Your correct. We're just getting started with this effort and we used
> > this mix to get things going. Once ppl are happy with the presentation
> > of data, we planned to add more tests to provide a more balanced mix.
> > But since you asked, we have added lmbench to our -bk3 regression run. :)
>
> Mark, it would be nice to get a graph of the combined results for each
> test. Something like:
>
> tiobench sequential write rate
> | +++++++++++++++++ + = -mm-ext3
> M | ++++++++++++++++++++++++++***************** * = linus-ext3
> B | +++++++***************** ###### # = -ac-ext3
> / | . = -mm-XFS
> s | = = -ac-XFS
> | ********* etc
> |
> +----------------------------------------------------
> date
>
> This allows at-a-glance trends for each group of tests and (as in the
> example above you could see easily when a performance bug was added
> and fixed in -ac before in the linus kernel, for example). Probably
> having all of the comparable results on the same page, or even in the
> same graph results is a win.
>
> Bonus points if you can click on a spot in the graph and get the results
> page for that date/test ;-).
This idea the STP team really, really likes. We'll start working on getting
this type of report
into our framework.
Thanks much, Andreas - great explaination.
cliffw
>
> Cheers, Andreas
> --
> Andreas Dilger
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
From: Andi Kleen <[email protected]>
Date: Fri, 30 May 2003 00:05:40 +0200
> To subscribe: http://www.osdl.org/mailman/listinfo/linstab
That's fairly obscure (Nobody knew of it before). Perhaps a well publicized
list on vger would be better.
I don't see why the OSDL list cannot be widely publicized and
the vger variant could. Don't be rediculious Andi :-)
Sorry I didn't see this sooner, I'm unsubscribed for the moment until my
email provider can get exim/procmail talking nicely.
LTP has had a mailing list for a long time that is explicitly for the
purpose of posting results. It's currently underutilized so I'd love to
see more results getting posted there again. Please consider using that
one for posting results of all types (LTP and non-ltp)
[email protected]
Thanks,
Paul Larson