For your information;
I've been running a bunch of benchmarks on a 250GB XFS filesystem.
After the benchmarks had run for a few hours and almost filled up the
fs, I removed all the files and did a "df -h" with interresting
results :
/dev/mapper/Data1-test
250G -64Z 251G 101% /mnt/test
"df -k" reported this :
/dev/mapper/Data1-test
262144000 -73786976294838202960 262147504 101% /mnt/test
I then did an umount and remount of the filesystem and then things
look more sane :
"df -h" :
/dev/mapper/Data1-test
250G 126M 250G 1% /mnt/test
"df -k" :
/dev/mapper/Data1-test
262144000 128280 262015720 1% /mnt/test
The filesystem is mounted like this :
/dev/mapper/Data1-test on /mnt/test type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log1/test_log,usrquota)
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
Also sprach "Jesper Juhl" <[email protected]> (Wed, 6 Sep 2006
10:54:34 +0200):
> For your information;
>
> I've been running a bunch of benchmarks on a 250GB XFS filesystem.
> After the benchmarks had run for a few hours and almost filled up the
> fs, I removed all the files and did a "df -h" with interresting
> results :
>
> /dev/mapper/Data1-test
> 250G -64Z 251G 101% /mnt/test
>
[...]
> I then did an umount and remount of the filesystem and then things
> look more sane :
>
> "df -h" :
> /dev/mapper/Data1-test
> 250G 126M 250G 1% /mnt/test
[...]
> The filesystem is mounted like this :
>
> /dev/mapper/Data1-test on /mnt/test type xfs
> (rw,noatime,ihashsize=64433,logdev=/dev/Log1/test_log,usrquota)
I once (2.6.12?) had to copy a quite large directory to an XFS
partition. It "should" had fit onto it (by what df said), but I ran into
"disk full". I think the reason was related to a large xfsbufd_centisecs
or xfssyncd_centisecs and indeed I could watch free space to grow and
shrink in regulaer intervals (watch df -k). I may well be wrong here (as
I'm sure no XFS-expert), but it looked like old data gets some kind of
"comressed" or "ordered" by the XFS-driver while newly written data took
more place. A "slow" copy did it, as well as a later try to an reiserfs
or ext.
sl ritch
On Wed, Sep 06, 2006 at 10:54:34AM +0200, Jesper Juhl wrote:
> For your information;
>
> I've been running a bunch of benchmarks on a 250GB XFS filesystem.
> After the benchmarks had run for a few hours and almost filled up the
> fs, I removed all the files and did a "df -h" with interresting
> results :
>
> /dev/mapper/Data1-test
> 250G -64Z 251G 101% /mnt/test
>
> "df -k" reported this :
>
> /dev/mapper/Data1-test
> 262144000 -73786976294838202960 262147504 101% /mnt/test
....
> The filesystem is mounted like this :
>
> /dev/mapper/Data1-test on /mnt/test type xfs
> (rw,noatime,ihashsize=64433,logdev=/dev/Log1/test_log,usrquota)
So the in-core accounting has underflowed by a small amount but the
on disk accounting is correct.
We've had a few reports of this that I know of over the past couple of years,
but we've never managed to find a reproducable test case for it.
Can you describe what benchmark you were runnin, wht kernel you were
using and whether any of the tests hit an ENOSPC condition?
Also, in future can you cc [email protected] on XFS bug reports?
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
On 07/09/06, David Chinner <[email protected]> wrote:
> On Wed, Sep 06, 2006 at 10:54:34AM +0200, Jesper Juhl wrote:
> > For your information;
> >
> > I've been running a bunch of benchmarks on a 250GB XFS filesystem.
> > After the benchmarks had run for a few hours and almost filled up the
> > fs, I removed all the files and did a "df -h" with interresting
> > results :
> >
> > /dev/mapper/Data1-test
> > 250G -64Z 251G 101% /mnt/test
> >
> > "df -k" reported this :
> >
> > /dev/mapper/Data1-test
> > 262144000 -73786976294838202960 262147504 101% /mnt/test
> ....
> > The filesystem is mounted like this :
> >
> > /dev/mapper/Data1-test on /mnt/test type xfs
> > (rw,noatime,ihashsize=64433,logdev=/dev/Log1/test_log,usrquota)
>
> So the in-core accounting has underflowed by a small amount but the
> on disk accounting is correct.
>
> We've had a few reports of this that I know of over the past couple of years,
> but we've never managed to find a reproducable test case for it.
>
> Can you describe what benchmark you were runnin, wht kernel you were
> using
The kernel is 2.6.18-rc6 SMP
> and whether any of the tests hit an ENOSPC condition?
>
That I don't know.
The script I was running is this one :
###-----[ cut here ]-----
#!/bin/bash
DIR=/mnt/test
DIR1=random
DIR2=bigfiles
DIR3=smallfiles
BONNIEDIR=bonnie
RACERDIR=racer
WRIGGLERDIR=wriggler
IOZONEDIR=iozone
cd $DIR
rm -rf -- *
mkdir $DIR1
cd $DIR1
while true; do
cd $DIR/$DIR1
for i in `seq 1 100`; do
touch -- "`head --bytes 15 /dev/urandom`";
done >/dev/null 2>&1
find . | while read i; do mv -f -- "$i" "`head --bytes 16
/dev/urandom`"; done >/dev/null 2>&1
rm -f -- *
sleep 1
done >/dev/null 2>&1 &
while true; do
cd $DIR/$DIR1
for i in `seq 1 100`; do
touch -- "`head --bytes 25 /dev/urandom`";
done >/dev/null 2>&1
find . | while read i; do mv -f -- "$i" "`head --bytes 30
/dev/urandom`"; done >/dev/null 2>&1
rm -f -- *
sleep 2
done >/dev/null 2>&1 &
cd $DIR
mkdir $DIR/$BONNIEDIR
mkdir $DIR/$RACERDIR
mkdir $DIR/$WRIGGLERDIR
mkdir $DIR/$IOZONEDIR
mkdir $DIR/$DIR2
mkdir $DIR/$DIR3
while true; do
(cd /usr/local/racer ; sh ./racer-lustre.sh >/dev/null 2>&1); done
>/dev/null 2>&1 &
while true; do
(cd $DIR/$IOZONEDIR ; /usr/local/iozone/iozone -a >/dev/null 2>&1);
done >/dev/null 2>&1 &
while true; do
(chown nobody $DIR/$BONNIEDIR ; cd $DIR/$BONNIEDIR ;
/usr/local/bonnie++/sbin/bonnie++ -u nobody >/dev/null 2>&1);
done >/dev/null 2>&1 &
while true; do
(cd $DIR/$WRIGGLERDIR ; /usr/local/diskWriggler/diskWriggler
-n 500 -2K -o $DIR/$WRIGGLERDIR >/dev/null 2>&1);
done >/dev/null 2>&1 &
while true; do
(cd $DIR/$DIR2 ; dd if=/dev/zero of=$DIR/$DIR2/biggie bs=10M
count=15000 >/dev/null 2>&1; sync ; rm -f $DIR/$DIR2/biggie );
done >/dev/null 2>&1 &
while true; do
(cd $DIR/$DIR3 ; export SIZE=$(($RANDOM*200000/32767)) ; dd
if=/dev/urandom of=$DIR/$DIR3/$SIZE bs=2048 count=$SIZE >/dev/null
2>&1; rm -f -- $DIR/$D
IR3/* );
done >/dev/null 2>&1 &
while true; do
for i in `seq 60 300`; do sleep $i; sync; done ;
done >/dev/null 2>&1 &
while true; do
for i in `seq 7 21`; do sleep $i; find $DIR; done ;
done >/dev/null 2>&1 &
while true; do date ; echo "tests running"; sleep 300; done
###-----[ cut here ]-----
/usr/local/racer/ holds this test script :
ftp://ftp.lustre.org/pub/benchmarks/racer-lustre.tar.gz
The script is modified to use /mnt/test/racer for its tests.
/usr/local/iozone/ holds iozone version 3.263
/usr/local/bonnie++/ holds bonnie++ version 1.03
/usr/local/diskWriggler/ holds diskWriggler version 1.0.1
> Also, in future can you cc [email protected] on XFS bug reports?
>
I keep forgetting, sorry. I'll try harder to remember :)
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
On Thu, Sep 07, 2006 at 04:17:53PM +0200, Jesper Juhl wrote:
> On 07/09/06, David Chinner <[email protected]> wrote:
> >On Wed, Sep 06, 2006 at 10:54:34AM +0200, Jesper Juhl wrote:
> >> For your information;
> >>
> >> I've been running a bunch of benchmarks on a 250GB XFS filesystem.
> >> After the benchmarks had run for a few hours and almost filled up the
> >> fs, I removed all the files and did a "df -h" with interresting
> >> results :
.....
> >So the in-core accounting has underflowed by a small amount but the
> >on disk accounting is correct.
> >
> >We've had a few reports of this that I know of over the past
> >couple of years, but we've never managed to find a reproducable
> >test case for it.
> >Can you describe what benchmark you were runnin, wht kernel you were
> >using
>
> The kernel is 2.6.18-rc6 SMP
Ok, so it's a current problem....
> >and whether any of the tests hit an ENOSPC condition?
> >
> That I don't know.
>
> The script I was running is this one :
<snip>
That doesn't really narrow down the scope at all. All that script
tells me is that problem is <waves hands> somewhere inside XFS.... :/
Can you try to isolate which of the loads is causing the problem?
That being said, this looks like a good stress load - I'll pass it
onto our QA folks...
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group