2013-04-10 11:06:10

by Tomasz Chmielewski

Subject: something's using space, can't figure out what - how to debug?

Here is one of my ext4 filesystems, which uses 2x more space as it
should:

# df -h /srv
Filesystem Size Used Avail Use% Mounted on
/dev/xvdf 194G 134G 51G 73% /srv

# du -sh /srv/
60G /srv/

# lsof -n|grep /srv
#


So df says 134 GB is used, while du says 60 GB is used.
No removed but used files, according to lsof.

How do I best debug what happened to ~70 GB?

This is EC2 kernel, nothing interesting in dmesg:

# uname -a
Linux ec2.agenteimovel.com.br 3.0.0-12-virtual #20-Ubuntu SMP Fri Oct 7
18:19:02 UTC 2011 x86_64 GNU/Linux


--
Tomasz Chmielewski
http://www.ptraveler.com

2013-04-11 00:26:48

by Theodore Ts'o

Subject: Re: something's using space, can't figure out what - how to debug?

On Wed, Apr 10, 2013 at 07:56:06PM +0900, Tomasz Chmielewski wrote:
> # lsof -n|grep /srv
> #
>
>
> So df says 134 GB is used, while du says 60 GB is used.
> No removed but used files, according to lsof.
>
> How do I best debug what happened to ~70 GB?

Well, nothing is using /srv, so why don't you unmount it and run
"e2fsck -f /dev/xvdf", and see e2fsck reports in terms of the number
of blocks in use, i.e:

/dev/lambda/backup: 581320/4915200 files (0.1% non-contiguous), 6837767/19660800 blocks

- Ted

2013-04-11 02:31:35

by Tomasz Chmielewski

Subject: Re: something's using space, can't figure out what - how to debug?

On Wed, 10 Apr 2013 20:26:43 -0400
Theodore Ts'o <[email protected]> wrote:

> On Wed, Apr 10, 2013 at 07:56:06PM +0900, Tomasz Chmielewski wrote:
> > # lsof -n|grep /srv
> > #
> >
> >
> > So df says 134 GB is used, while du says 60 GB is used.
> > No removed but used files, according to lsof.
> >
> > How do I best debug what happened to ~70 GB?
>
> Well, nothing is using /srv, so why don't you unmount it and run
> "e2fsck -f /dev/xvdf", and see e2fsck reports in terms of the number
> of blocks in use, i.e:
>
> /dev/lambda/backup: 581320/4915200 files (0.1% non-contiguous),
> 6837767/19660800 blocks

If it was only so easy!

# lsof -n|grep /srv

# fuser /srv

# ls /sys/block/xvdf/holders/

# umount /srv
umount: /srv: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))

Yet after reboot, back to normal:

root@ec2:~# df -h /srv
Filesystem Size Used Avail Use% Mounted on
/dev/xvdf 194G 62G 123G 34% /srv


--
Tomasz Chmielewski
http://www.ptraveler.com