Hi all,
Lately I've been having some fun playing with bcache, dmcache, and enhanceio.
I've an fio config[1] that thinks it simulates a certain kind of workload
<cough> by randomly reading and writing 8k blocks to 8x 1GB files. Since the
storage summit is in a week, I thought I might get the party started.
I had two machines to play with. The first ("sub3") has a three-way RAID1 of
7200rpm disks, and a 1.2GB partition on an otherwise empty SSD. The second
("blackbox") is a laptop with a single 5400rpm disk and a 1.2GB partition on an
SSD. These configurations are a little silly, but I constrained the SSD size
to try to observe cache demotions. None of the spinny disks have enough
streaming bandwidth to beat the SSD.
I set up each cache, formatted the cache device with "mkfs.ext4 -E
lazy_itable_init=1 -F <device>", mounted it, and ran fio for five minutes.
dmcache was set up with the default ("mq") policy in writeback mode with the
sequential threshold set to 16M, and all other settings set to defaults. I
changed dmcache's sequential threshold because I observed that with its default
of 256K, setting readahead to 256K sometimes caused dmcache to confuse
readahead for streaming IO, and ignore the IO. bcache and enhanceio were set
up with LRU replacement and writeback mode. The kernel was 3.9-rc6, with
bcache pulled in from bcache-testing on Kent's git tree, and enhanceio pulled
in from STEC's github tree. There seems to be a reworked version of bcache in
-next, but it appears to depend on the aio rework, which I didn't feel like
backporting. :)
The results live at [2]. In hindsight I wish I'd let it run longer so that in
all cases we would end up performing more IO than there was cache space, but
there's still time for that. bcache pretty consistently delivered 2-3x the
bandwidth of the raw device, and the latencies went down by about the same
amount. In general, I managed about a 70% hit rate with bcache. EnhanceIO
provided a smaller boost, between 1.5-2x the bandwidth. Read latencies went
down a bit, and write latencies seemed to decrease substantially, and enhanceio
was pretty consistent about the boost as well. I don't know how to get hit
rate data out of enhanceio.
However, the weirdest results came from dmcache. I reran the mkcache + format
+ fio loop quite a few times. Most of the time it would produce a really nice
3-20x speedup, like what I posted at [2]. Other times, however, it was barely
faster than the spinny disk. It was always the case that the cache hit full
utilization (i.e. all cache blocks in use), however in the fast case I would
see about a 90% hit rate and in the slow case, about 1%. Sadly, I don't have a
dumper tool that would let me examine which cache blocks remained in the cache,
however I /did/ observe that if I ran mkfs with lazy_itable_init=0, the cache
would be pre-stuffed with ~600MB of inode tables, causing a very low hit rate
and poor performance. I don't think those inode tables are all that hot,
seeing as there are only 20 inodes in the whole filesystem. Oddly, the
latencies are higher on dmcache even despite the higher throughput. My
speculation here is that I've not yet figured out what circumstances are
favorable to convincing dmcache to demote a cache block in favor of promoting
another. The demotion count was almost zero.
Maybe I just need to run a longer workload?
<shrug> It's midnight here, I'm going to sleep on it for now and see what
people have to say in the morning.
Hopefully I'll see some of you at the upcoming storage/fs/etc summit?
--D
[1] http://djwong.org/docs/ssd-cache-bakeoff/database-funtime.fio
[2] http://djwong.org/docs/ssd-cache-bakeoff/
Hi Darrick,
On Thu, Apr 11, 2013 at 12:22:39AM -0700, Darrick J. Wong wrote:
> Hi all,
>
> Lately I've been having some fun playing with bcache, dmcache, and enhanceio.
I pushed some tweaks to the mq policy today to my thin-dev tree. They
show some improvements to these fio based tests.
In addition I've written a blog post trying to explain what's going on in dm-cache:
http://device-mapper.org/blog/2013/04/13/benchmarking-dm-cache-with-fio/
- Joe
On Sat, Apr 13 2013 at 12:09pm -0400,
Joe Thornber <[email protected]> wrote:
> Hi Darrick,
>
> On Thu, Apr 11, 2013 at 12:22:39AM -0700, Darrick J. Wong wrote:
> > Hi all,
> >
> > Lately I've been having some fun playing with bcache, dmcache, and enhanceio.
>
> I pushed some tweaks to the mq policy today to my thin-dev tree. They
> show some improvements to these fio based tests.
>
> In addition I've written a blog post trying to explain what's going on in dm-cache:
> http://device-mapper.org/blog/2013/04/13/benchmarking-dm-cache-with-fio/
Darrick,
Joe has a few other dm-cache-target.c changes in his thin-dev branch
that are required in order to realize the gains from his mq changes. I
haven't yet isolated which changes are important but if I just use the
3.9-rc6's dm-cache-tagret.c with thin-dev's mq changes I cannot
reproduce the improved performance Joe mentions in his blog post.
Also, even before these changes I wasn't able to reproduce your dm-cache
results (either the spike in performance or the inconsistencies you
saw across runs).
BTW, I have added 'test_fio_database_funtime' to both the cache and
bcache testsuites in my thinp-test-suite repo (master branch):
git://github.com/snitm/thinp-test-suite.git
You'd run it with somwthing like:
./run_tests --profile mix_fio --suite cache -n /test_fio_database_funtime/
or
./run_tests --profile mix_fio --suite bcache -n /test_fio_database_funtime/
I've been testing against the v3.9-rc6 kernel with Jens' for-next bcache
code merged in, see 'thin-dev-bcache' branch of my linux repo:
git://github.com/snitm/linux.git
On Sat, Apr 13 2013 at 3:33pm -0400,
Mike Snitzer <[email protected]> wrote:
> On Sat, Apr 13 2013 at 12:09pm -0400,
> Joe Thornber <[email protected]> wrote:
>
> > Hi Darrick,
> >
> > On Thu, Apr 11, 2013 at 12:22:39AM -0700, Darrick J. Wong wrote:
> > > Hi all,
> > >
> > > Lately I've been having some fun playing with bcache, dmcache, and enhanceio.
> >
> > I pushed some tweaks to the mq policy today to my thin-dev tree. They
> > show some improvements to these fio based tests.
> >
> > In addition I've written a blog post trying to explain what's going on in dm-cache:
> > http://device-mapper.org/blog/2013/04/13/benchmarking-dm-cache-with-fio/
>
> Darrick,
>
> Joe has a few other dm-cache-target.c changes in his thin-dev branch
> that are required in order to realize the gains from his mq changes. I
> haven't yet isolated which changes are important but if I just use the
> 3.9-rc6's dm-cache-tagret.c with thin-dev's mq changes I cannot
> reproduce the improved performance Joe mentions in his blog post.
I've now isolated the dm-cache-target.c commit in Joe's thin-dev that is
important to have in conjunction with the mq changes (from commit 7362f43a):
commit f134e6382a534dd3622c3850e4824ae5929885d9
Author: Joe Thornber <[email protected]>
Date: Wed Mar 27 14:46:57 2013 +0000
[dm-cache] Change the default migration_threshold to something more suitable.
diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
index 8a4bcf3..f215307 100644
--- a/drivers/md/dm-cache-target.c
+++ b/drivers/md/dm-cache-target.c
@@ -1948,7 +1948,7 @@ static sector_t calculate_discard_block_size(sector_t cache_block_size,
return discard_block_size;
}
-#define DEFAULT_MIGRATION_THRESHOLD (2048 * 100)
+#define DEFAULT_MIGRATION_THRESHOLD 2048
static int cache_create(struct cache_args *ca, struct cache **result)
{
On Mon, Apr 15, 2013 at 09:55:39AM -0400, Mike Snitzer wrote:
> On Sat, Apr 13 2013 at 3:33pm -0400,
> Mike Snitzer <[email protected]> wrote:
>
> > On Sat, Apr 13 2013 at 12:09pm -0400,
> > Joe Thornber <[email protected]> wrote:
> >
> > > Hi Darrick,
> > >
> > > On Thu, Apr 11, 2013 at 12:22:39AM -0700, Darrick J. Wong wrote:
> > > > Hi all,
> > > >
> > > > Lately I've been having some fun playing with bcache, dmcache, and enhanceio.
> > >
> > > I pushed some tweaks to the mq policy today to my thin-dev tree. They
> > > show some improvements to these fio based tests.
> > >
> > > In addition I've written a blog post trying to explain what's going on in dm-cache:
> > > http://device-mapper.org/blog/2013/04/13/benchmarking-dm-cache-with-fio/
> >
> > Darrick,
> >
> > Joe has a few other dm-cache-target.c changes in his thin-dev branch
> > that are required in order to realize the gains from his mq changes. I
> > haven't yet isolated which changes are important but if I just use the
> > 3.9-rc6's dm-cache-tagret.c with thin-dev's mq changes I cannot
> > reproduce the improved performance Joe mentions in his blog post.
>
> I've now isolated the dm-cache-target.c commit in Joe's thin-dev that is
> important to have in conjunction with the mq changes (from commit 7362f43a):
>
> commit f134e6382a534dd3622c3850e4824ae5929885d9
> Author: Joe Thornber <[email protected]>
> Date: Wed Mar 27 14:46:57 2013 +0000
>
> [dm-cache] Change the default migration_threshold to something more suitable.
>
> diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
> index 8a4bcf3..f215307 100644
> --- a/drivers/md/dm-cache-target.c
> +++ b/drivers/md/dm-cache-target.c
> @@ -1948,7 +1948,7 @@ static sector_t calculate_discard_block_size(sector_t cache_block_size,
> return discard_block_size;
> }
>
> -#define DEFAULT_MIGRATION_THRESHOLD (2048 * 100)
> +#define DEFAULT_MIGRATION_THRESHOLD 2048
>
> static int cache_create(struct cache_args *ca, struct cache **result)
> {
You'll certainly need the new mq writeback stuff too. I don't know
why you're intent on cherry picking.
- Joe
You misunderstood me. I was just making sure Darrick knew about this other migration change because without it the specific commit you pointed to for mq won't get the performance improvement. So I was just trying to make sure if Darrick or others cherry picked they didn't miss out.????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay? ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?