Hi
My work (as in what brings in money to buy things from) file-system is
now "slow as hell". Last time this happened you guys told me that this
was because I had millions of hardlinks and way too many files.
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0 815M 4.1M 811M 1% /recover2
I have only 4.1M files. Only 1% inodes "in use".
The filesystem was filling up lately, but i've managed to create some
free space.
Filesystem Size Used Avail Use% Mounted on
/dev/md0 6.3T 5.3T 720G 89% /recover2
The question is: How do I get the filesystem to perform
normally again?
All mkdir calls are taking around 62 milliseconds:
14:57:02 mkdir("...ME_25c8e/", 0777) = 0 <0.062312>
14:57:02 mkdir("...ME_112ff/", 0777) = 0 <0.062658>
14:57:02 mkdir("...ME_183cf/", 0777) = 0 <0.062344>
while on my workstation where i'm not supposed to be working
with client-data I get:
14:58:18 mkdir("...ME_2a6c6/", 0777) = 0 <0.000033>
14:58:18 mkdir("...ME_2b3d9/", 0777) = 0 <0.000031>
14:58:18 mkdir("...ME_2c63f/", 0777) = 0 <0.000034>
So, things on my big datastorage drive are about 2000 times slower
than on my workstation.
Roger.
--
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
On 5/13/11 7:59 AM, Rogier Wolff wrote:
>
> Hi
>
> My work (as in what brings in money to buy things from) file-system is
> now "slow as hell". Last time this happened you guys told me that this
> was because I had millions of hardlinks and way too many files.
>
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/md0 815M 4.1M 811M 1% /recover2
>
> I have only 4.1M files. Only 1% inodes "in use".
>
>
> The filesystem was filling up lately, but i've managed to create some
> free space.
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/md0 6.3T 5.3T 720G 89% /recover2
>
> The question is: How do I get the filesystem to perform
> normally again?
>
>
> All mkdir calls are taking around 62 milliseconds:
>
> 14:57:02 mkdir("...ME_25c8e/", 0777) = 0 <0.062312>
> 14:57:02 mkdir("...ME_112ff/", 0777) = 0 <0.062658>
> 14:57:02 mkdir("...ME_183cf/", 0777) = 0 <0.062344>
I don't know what the workload is, and "slow" isn't very well characterized here, but from this can I assume that mkdir is the bottleneck?
Did you ever try that patch I sent you for this issue in March?
Using oprofile to see where the fs is spending its time might also be useful.
-Eric
> while on my workstation where i'm not supposed to be working
> with client-data I get:
>
> 14:58:18 mkdir("...ME_2a6c6/", 0777) = 0 <0.000033>
> 14:58:18 mkdir("...ME_2b3d9/", 0777) = 0 <0.000031>
> 14:58:18 mkdir("...ME_2c63f/", 0777) = 0 <0.000034>
>
> So, things on my big datastorage drive are about 2000 times slower
> than on my workstation.
>
> Roger.
>
On May 13, 2011, at 06:59, Rogier Wolff wrote:
> My work (as in what brings in money to buy things from) file-system is
> now "slow as hell". Last time this happened you guys told me that this
> was because I had millions of hardlinks and way too many files.
>
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/md0 815M 4.1M 811M 1% /recover2
>
> I have only 4.1M files. Only 1% inodes "in use".
>
>
> The filesystem was filling up lately, but i've managed to create some
> free space.
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/md0 6.3T 5.3T 720G 89% /recover2
I suspect that the free inodes in your filesystem are very fragmented, due to the filesystem filling up and then deleting a lot of files that create "holes" of free inodes within the inode table blocks. That means that possibly finding free inodes is a bunch of work, and then possibly only one or two inodes are allocated from each block.
> The question is: How do I get the filesystem to perform
> normally again?
It is probably worthwhile to do a blktrace to see what the IO is looking like, and as Eric suggested the oprofile information may be interesting if a lot of CPU is being used.
> All mkdir calls are taking around 62 milliseconds:
>
> 14:57:02 mkdir("...ME_25c8e/", 0777) = 0 <0.062312>
> 14:57:02 mkdir("...ME_112ff/", 0777) = 0 <0.062658>
> 14:57:02 mkdir("...ME_183cf/", 0777) = 0 <0.062344>
>
> while on my workstation where i'm not supposed to be working
> with client-data I get:
>
> 14:58:18 mkdir("...ME_2a6c6/", 0777) = 0 <0.000033>
> 14:58:18 mkdir("...ME_2b3d9/", 0777) = 0 <0.000031>
> 14:58:18 mkdir("...ME_2c63f/", 0777) = 0 <0.000034>
>
> So, things on my big datastorage drive are about 2000 times slower
> than on my workstation.
>
> Roger.
>
> --
> ** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
> ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 **
> *-- BitWizard writes Linux device drivers for any device you may have! --*
> Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
> Does it sit on the couch all day? Is it unemployed? Please be specific!
> Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Cheers, Andreas
On Fri, May 13, 2011 at 11:21:52PM -0600, Andreas Dilger wrote:
> On May 13, 2011, at 06:59, Rogier Wolff wrote:
> > My work (as in what brings in money to buy things from) file-system is
> > now "slow as hell". Last time this happened you guys told me that this
> > was because I had millions of hardlinks and way too many files.
> >
> > Filesystem Inodes IUsed IFree IUse% Mounted on
> > /dev/md0 815M 4.1M 811M 1% /recover2
> >
> > I have only 4.1M files. Only 1% inodes "in use".
> >
> >
> > The filesystem was filling up lately, but i've managed to create some
> > free space.
> >
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/md0 6.3T 5.3T 720G 89% /recover2
> I suspect that the free inodes in your filesystem are very
> fragmented, due to the filesystem filling up and then deleting a lot
> of files that create "holes" of free inodes within the inode table
> blocks. That means that possibly finding free inodes is a bunch of
> work, and then possibly only one or two inodes are allocated from
> each block.
How can the free inodes be fragmented?!? 99% of the available inodes
ARE free.
> > The question is: How do I get the filesystem to perform
> > normally again?
> It is probably worthwhile to do a blktrace to see what the IO is
> looking like, and as Eric suggested the oprofile information may be
> interesting if a lot of CPU is being used.
A lot of CPU time is being used, yes. The 62 milliseconds are almost
completely CPU time.
I was running a big "backup" (image of a 500Gb disk) onto the
partition. Now that this has finished I'll try booting into a recent
kernel. Next I'll try finding and applying the patch....
Roger.
--
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ