From: Jacek Luczak Subject: Re: getdents - ext4 vs btrfs performance Date: Wed, 29 Feb 2012 14:55:11 +0100 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE To: linux-ext4@vger.kernel.org, linux-fsdevel , LKML , linux-btrfs@vger.kernel.org, chris.mason@oracle.com Return-path: In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org List-Id: linux-ext4.vger.kernel.org Hi Chris, the last one was borked :) Please check this one. -jacek 2012/2/29 Jacek Luczak : > Hi All, > > /*Sorry for sending incomplete email, hit wrong button :) I guess I > can't use Gmail */ > > Long story short: We've found that operations on a directory structur= e > holding many dirs takes ages on ext4. > > The Question: Why there's that huge difference in ext4 and btrfs? See > below test results for real values. > > Background: I had to backup a Jenkins directory holding workspace for > few projects which were co from svn (implies lot of extra .svn dirs). > The copy takes lot of time (at least more than I've expected) and > process was mostly in D (disk sleep). I've dig more and done some > extra test to see if this is not a regression on block/fs site. To > isolate the issue I've also performed same tests on btrfs. > > Test environment configuration: > 1) HW: HP ProLiant BL460 G6, 48 GB of memory, 2x 6 core Intel X5670 H= T > enabled, Smart Array P410i, RAID 1 on top of 2x 10K RPM SAS HDDs. > 2) Kernels: All tests were done on following kernels: > =A0- 2.6.39.4-3 -- the build ID (3) is used here for internal tacking= of > config changes mostly. In -3 we've introduced ,,fix readahead pipelin= e > break caused by block plug'' patch. Otherwise it's pure 2.6.39.4. > =A0- 3.2.7 -- latest kernel at the time of testing (3.2.8 has been > release recently). > 3) A subject of tests, directory holding: > =A0- 54GB of data (measured on ext4) > =A0- 1978149 files > =A0- 844008 directories > 4) Mount options: > =A0- ext4 -- errors=3Dremount-ro,noatime, > data=3Dwriteback > =A0- btrfs -- noatime,nodatacow and for later investigation on > copression effect: noatime,nodatacow,compress=3Dlzo > > In all tests I've been measuring time of execution. Following tests > were performed: > - find . -type d > - find . -type f > - cp -a > - rm -rf > > Ext4 results: > | Type =A0 =A0 | 2.6.39.4-3 =A0 | 3.2.7 > | Dir cnt =A0| 17m 40sec =A0| 11m 20sec > | File cnt | =A017m 36sec | 11m 22sec > | Copy =A0 =A0| 1h 28m =A0 =A0 =A0 =A0| 1h 27m > | Remove| 3m 43sec =A0 =A0| 3m 38sec > > Btrfs results (without lzo comression): > | Type =A0 =A0 | 2.6.39.4-3 =A0 | 3.2.7 > | Dir cnt =A0| 2m 22sec =A0| 2m 21sec > | File cnt | =A02m 26sec | 2m 23sec > | Copy =A0 =A0| 36m 22sec | 39m 35sec > | Remove| 7m 51sec =A0 | 10m 43sec > > From above one can see that copy takes close to 1h less on btrfs. I'v= e > done strace counting times of calls, results are as follows (from > 3.2.7): > 1) Ext4 (only to elements): > % time =A0 =A0 seconds =A0usecs/call =A0 =A0 calls =A0 =A0errors sysc= all > ------ ----------- ----------- --------- --------- ---------------- > =A057.01 =A0 13.257850 =A0 =A0 =A0 =A0 =A0 1 =A015082163 =A0 =A0 =A0 = =A0 =A0 read > =A023.40 =A0 =A05.440353 =A0 =A0 =A0 =A0 =A0 3 =A0 1687702 =A0 =A0 =A0= =A0 =A0 getdents > =A06.15 =A0 =A01.430559 =A0 =A0 =A0 =A0 =A0 0 =A0 3672418 =A0 =A0 =A0= =A0 =A0 lstat > =A03.80 =A0 =A00.883767 =A0 =A0 =A0 =A0 =A0 0 =A013106961 =A0 =A0 =A0= =A0 =A0 write > =A02.32 =A0 =A00.539959 =A0 =A0 =A0 =A0 =A0 0 =A0 4794099 =A0 =A0 =A0= =A0 =A0 open > =A01.69 =A0 =A00.393589 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0843695 =A0 =A0 =A0= =A0 =A0 mkdir > =A01.28 =A0 =A00.296700 =A0 =A0 =A0 =A0 =A0 0 =A0 5637802 =A0 =A0 =A0= =A0 =A0 setxattr > =A00.80 =A0 =A00.186539 =A0 =A0 =A0 =A0 =A0 0 =A0 7325195 =A0 =A0 =A0= =A0 =A0 stat > > 2) Btrfs: > % time =A0 =A0 seconds =A0usecs/call =A0 =A0 calls =A0 =A0errors sysc= all > ------ ----------- ----------- --------- --------- ---------------- > 53.38 =A0 =A09.486210 =A0 =A0 =A0 =A0 =A0 1 =A015179751 =A0 =A0 =A0 =A0= =A0 read > 11.38 =A0 =A02.021662 =A0 =A0 =A0 =A0 =A0 1 =A0 1688328 =A0 =A0 =A0 =A0= =A0 getdents > =A010.64 =A0 =A01.890234 =A0 =A0 =A0 =A0 =A0 0 =A0 4800317 =A0 =A0 =A0= =A0 =A0 open > =A06.83 =A0 =A01.213723 =A0 =A0 =A0 =A0 =A0 0 =A013201590 =A0 =A0 =A0= =A0 =A0 write > =A04.85 =A0 =A00.862731 =A0 =A0 =A0 =A0 =A0 0 =A0 5644314 =A0 =A0 =A0= =A0 =A0 setxattr > =A03.50 =A0 =A00.621194 =A0 =A0 =A0 =A0 =A0 1 =A0 =A0844008 =A0 =A0 =A0= =A0 =A0 mkdir > =A02.75 =A0 =A00.489059 =A0 =A0 =A0 =A0 =A0 0 =A0 3675992 =A0 =A0 =A0= =A0 1 lstat > =A01.71 =A0 =A00.303544 =A0 =A0 =A0 =A0 =A0 0 =A0 5644314 =A0 =A0 =A0= =A0 =A0 llistxattr > =A01.50 =A0 =A00.265943 =A0 =A0 =A0 =A0 =A0 0 =A0 1978149 =A0 =A0 =A0= =A0 =A0 utimes > =A01.02 =A0 =A00.180585 =A0 =A0 =A0 =A0 =A0 0 =A0 5644314 =A0 =A08440= 08 getxattr > > On btrfs getdents takes much less time which prove the bottleneck in > copy time on ext4 is this syscall. In 2.6.39.4 it shows even less tim= e > for getdents: > % time =A0 =A0 seconds =A0usecs/call =A0 =A0 calls =A0 =A0errors sysc= all > ------ ----------- ----------- --------- --------- ---------------- > =A050.77 =A0 10.978816 =A0 =A0 =A0 =A0 =A0 1 =A015033132 =A0 =A0 =A0 = =A0 =A0 read > =A014.46 =A0 =A03.125996 =A0 =A0 =A0 =A0 =A0 1 =A0 4733589 =A0 =A0 =A0= =A0 =A0 open > =A07.15 =A0 =A01.546311 =A0 =A0 =A0 =A0 =A0 0 =A0 5566988 =A0 =A0 =A0= =A0 =A0 setxattr > =A05.89 =A0 =A01.273845 =A0 =A0 =A0 =A0 =A0 0 =A0 3626505 =A0 =A0 =A0= =A0 =A0 lstat > =A05.81 =A0 =A01.255858 =A0 =A0 =A0 =A0 =A0 1 =A0 1667050 =A0 =A0 =A0= =A0 =A0 getdents > =A05.66 =A0 =A01.224403 =A0 =A0 =A0 =A0 =A0 0 =A013083022 =A0 =A0 =A0= =A0 =A0 write > =A03.40 =A0 =A00.735114 =A0 =A0 =A0 =A0 =A0 1 =A0 =A0833371 =A0 =A0 =A0= =A0 =A0 mkdir > =A01.96 =A0 =A00.424881 =A0 =A0 =A0 =A0 =A0 0 =A0 5566988 =A0 =A0 =A0= =A0 =A0 llistxattr > > > Why so huge difference in the getdents timings? > > -Jacek