Date: Fri, 12 Oct 2012 15:51:14 +0100
From: Mel Gorman <mgorman@suse.de>
To: Linux-MM <linux-mm@kvack.org>
Cc: LKML <linux-kernel@vger.kernel.org>
Subject: MMTests 0.06
Message-ID: <20121012145114.GZ29125@suse.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-15
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 3516
Lines: 69

MMTests 0.06 is a configurable test suite that runs a number of common
workloads of interest to MM developers. There are multiple additions
all but in many respects the most useful will be automatic package
installation. The package names are based on openSUSE but it's easy to
create mappings in bin/install-depends where the package names differ. The
very basics of monitoring NUMA efficiency is there as well and the autonuma
benchmark has a test. The stats it reports for NUMA need significant
improvement but for the most part that should be straight forward.

Changelog since v0.05
o Automatically install packages (need name mappings for other distros)
o Add benchmark for autonumabench
o Add support for benchmarking NAS with MPI
o Add pgbench for autonumabench (may need a bit more work)
o Upgrade postgres version to 9.2.1
o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains
o Alter mailserver config to finish in a reasonable time
o Add monitor for perf sched
o Add moinitor that gathers ftrace information with trace-cmd
o Add preliminary monitors for NUMA stats (very basic)
o Specify ftrace events to monitor from config file
o Remove the bulk of whats left of VMRegress
o Convert shellpacks to a template format to auto-generate boilerplate code
o Collect lock_stat information if enabled
o Run multiple iterations of aim9
o Add basic regression tests for Cross Memory Attach
o Copy with preempt being enabled in highalloc stres tests
o Have largedd cope with a missing large file to work with
o Add a monitor-only mode to just capture logs
o Report receive-side throughput in netperf for results

At LSF/MM at some point a request was made that a series of tests
be identified that were of interest to MM developers and that could be
used for testing the Linux memory management subsystem. There is renewed
interest in some sort of general testing framework during discussions for
Kernel Summit 2012 so here is what I use.

http://www.csn.ul.ie/~mel/projects/mmtests/
http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz

There are a number of stock configurations stored in configs/.  For example
config-global-dhp__pagealloc-performance runs a number of tests that
may be able to identify performance regressions or gains in the page
allocator. Similarly there network and scheduler configs. There are also
more complex options. config-global-dhp__parallelio-memcachetest will run
memcachetest in the foreground while doing IO of different sizes in the
background to measure how much unrelated IO affects the throughput of an
in-memory database.

This release is also a little rough and the extraction scripts could
have been tidier but they were mostly written in an airport and for the
most part they work as advertised. I'll fix bugs as according as they are
brought to my attention.

The stats reporting still needs work because while some tests know how
to make a better estimate of mean by filtering outliers it is not being
handled consistently and the methodology needs work. I know filtering
statistics like this is a major flaw in the methodology but the decision
was made in this case in the interest of the benchmarks with unstable
results completing in a reasonable time.

-- 
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/