Hi.
Periodically there is an overflow on /tmp on my boxes, and I can't find
the reaason why. May be it isn't a bug in tmpfs and you can help me find
the reason, and if it is an error, I hope you will help me to eliminate
it.
I know that df, du can show different values if some files were deleted,
but the processes which opened these files are still acting.
Well now I see this picture on one of my boxes. Below I post the results
of how I was trying to find what was /tmp full of.
# df -h /tmp
Filesystem Size Used Avail Use% Mounted on
tmpfs 4.1G 3.3G 762M 82% /tmp
# du -hs /tmp
518M /tmp
# lsof /tmp | awk '{if ($7 != 0) print$0}'
(in attach)
I still don't see any files taking 3G. Now I know only one solution
mount/remount /tmp.
--
BRGDS. Alexey Vlasov.
Alexey Vlasov <[email protected]> writes:
>
> I know that df, du can show different values if some files were deleted,
> but the processes which opened these files are still acting.
You can check for that by doing ls -l /proc/*/fd | grep deleted
It won't tell you the file size though.
-Andi
--
[email protected] -- Speaking for myself only.
On Mon, Jul 13, 2009 at 11:09:29AM +0200, Andi Kleen wrote:
> Alexey Vlasov <[email protected]> writes:
> >
> > I know that df, du can show different values if some files were deleted,
> > but the processes which opened these files are still acting.
>
> You can check for that by doing ls -l /proc/*/fd | grep deleted
>
> It won't tell you the file size though.
There comes a big list, like this:
lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
[cut]
lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
But you can see the file date, these are new files and by next search
you won't see them any more.
The most interesting is that the /tmp size constantly grows.
--
BRGDS. Alexey Vlasov.
> But you can see the file date, these are new files and by next search
I appended a patch that reports the size.
-Andi
---
VFS: Report file size for deleted files in /proc
It's useful to know how big deleted files are, in case a file system
is filled up unexpectedly and the admin wants to track down which
process is the culprit. This patch reports the file size
in addition to (deleted) for /proc/*/fd links.
This modifies a public interface, but it's reasonable to assume
that noone relies on the particular format here.
Signed-off-by: Andi Kleen <[email protected]>
---
fs/dcache.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
Index: linux-2.6.31-rc1-ak/fs/dcache.c
===================================================================
--- linux-2.6.31-rc1-ak.orig/fs/dcache.c
+++ linux-2.6.31-rc1-ak/fs/dcache.c
@@ -1910,9 +1910,19 @@ char *__d_path(const struct path *path,
spin_lock(&vfsmount_lock);
prepend(&end, &buflen, "\0", 1);
- if (d_unlinked(dentry) &&
- (prepend(&end, &buflen, " (deleted)", 10) != 0))
+ if (d_unlinked(dentry)) {
+ char buf[40];
+ int n;
+ if (dentry->d_inode) { /* check needed? */
+ n = snprintf(buf, sizeof buf, " (deleted; size %lluK)",
+ dentry->d_inode->i_size >> 10);
+ } else {
+ strcpy(buf, " (deleted)");
+ n = 10;
+ }
+ if (prepend(&end, &buflen, buf, n) != 0)
goto Elong;
+ }
if (buflen < 1)
goto Elong;
--
[email protected] -- Speaking for myself only.
Andi Kleen <[email protected]> writes:
>> But you can see the file date, these are new files and by next search
>
> I appended a patch that reports the size.
You can get the size already by using stat instead of lstat, ie. use ls -lL.
Andreas.
--
Andreas Schwab, [email protected]
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84 5EC7 45C6 250E 6F00 984E
"And now for something completely different."
On Mon, Jul 13, 2009 at 03:13:09PM +0200, Andreas Schwab wrote:
> Andi Kleen <[email protected]> writes:
>
> >> But you can see the file date, these are new files and by next search
> >
> > I appended a patch that reports the size.
>
> You can get the size already by using stat instead of lstat, ie. use ls -lL.
True, on the other hand you need two passes because that doesn't give
you file names. So the patch would be still more convenient.
-Andi
--
[email protected] -- Speaking for myself only.
2009/7/13 Alexey Vlasov <[email protected]>:
> On Mon, Jul 13, 2009 at 11:09:29AM +0200, Andi Kleen wrote:
>> Alexey Vlasov <[email protected]> writes:
>> >
>> > I know that df, du can show different values if some files were deleted,
>> > but the processes which opened these files are still acting.
>>
>> You can check for that by doing ls -l /proc/*/fd | grep deleted
>>
>> It won't tell you the file size though.
>
> There comes a big list, like this:
>
> lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
> lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
> lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
> [cut]
> lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
> lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
> lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
>
> But you can see the file date, these are new files and by next search
> you won't see them any more.
>
> The most interesting is that the /tmp size constantly grows.
Try:
lsof -n | grep deleted
This will show the process holding them more easily, and the file size
(in the seventh column). It won't show the date they were opened
though.
On Mon, Jul 13, 2009 at 09:54:22AM -0400, Bryan Donlan wrote:
> >> Alexey Vlasov <[email protected]> writes:
> >> >
> >> > I know that df, du can show different values if some files were deleted,
> >> > but the processes which opened these files are still acting.
> >>
> >> You can check for that by doing ls -l /proc/*/fd | grep deleted
> >>
> >> It won't tell you the file size though.
> >
> > There comes a big list, like this:
> >
> > lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
> >
> > The most interesting is that the /tmp size constantly grows.
>
> lsof -n | grep deleted
# lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'
mysqld 22905 mysql 6u REG
0,13 20 663938322 /tmp/ib1oZTSZ (deleted)
mysqld 22905 mysql 8u REG
0,13 84773 663938424 /tmp/ibRcarbD (deleted)
# df -h /tmp
Filesystem Size Used Avail Use% Mounted on
tmpfs 4.1G 3.4G 630M 85% /tmp
It will soon be up to 100% full and I'll have to remount.
--
BRGDS. Alexey Vlasov.
2009/7/13 Alexey Vlasov <[email protected]>:
> On Mon, Jul 13, 2009 at 09:54:22AM -0400, Bryan Donlan wrote:
>> >> Alexey Vlasov <[email protected]> writes:
>> >> >
>> >> > I know that df, du can show different values if some files were deleted,
>> >> > but the processes which opened these files are still acting.
>> >>
>> >> You can check for that by doing ls -l /proc/*/fd | grep deleted
>> >>
>> >> It won't tell you the file size though.
>> >
>> > There comes a big list, like this:
>> >
>> > lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
>> >
>> > The most interesting is that the /tmp size constantly grows.
>>
>> lsof -n | grep deleted
>
> # lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'
>
> mysqld ? ?22905 ? ? ? ? ? ? ? ? ? ? ? ? ?mysql ? ?6u ? REG
> 0,13 ? ?20 663938322 /tmp/ib1oZTSZ (deleted)
> mysqld ? ?22905 ? ? ? ? ? ? ? ? ? ? ? ? ?mysql ? ?8u ? REG
> 0,13 84773 663938424 /tmp/ibRcarbD (deleted)
>
> # df -h /tmp
> Filesystem ? ? ? ? ? ?Size ?Used Avail Use% Mounted on
> tmpfs ? ? ? ? ? ? ? ? 4.1G ?3.4G ?630M ?85% /tmp
>
> It will soon be up to 100% full and I'll have to remount.
Are those the only two results? That should only be adding up to ~84kb
of space...
On Mon, Jul 13, 2009 at 03:58:36PM -0400, Bryan Donlan wrote:
> 2009/7/13 Alexey Vlasov <[email protected]>:
> > # lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'
> >
> > mysqld ? ?22905 ? ? ? ? ? ? ? ? ? ? ? ? ?mysql ? ?6u ? REG
> > 0,13 ? ?20 663938322 /tmp/ib1oZTSZ (deleted)
> > mysqld ? ?22905 ? ? ? ? ? ? ? ? ? ? ? ? ?mysql ? ?8u ? REG
> > 0,13 84773 663938424 /tmp/ibRcarbD (deleted)
> >
> > # df -h /tmp
> > Filesystem ? ? ? ? ? ?Size ?Used Avail Use% Mounted on
> > tmpfs ? ? ? ? ? ? ? ? 4.1G ?3.4G ?630M ?85% /tmp
>
> Are those the only two results? That should only be adding up to ~84kb
> of space...
Yes, only two lines where the 7-th column isn't equal to 0
--
BRGDS. Alexey Vlasov.
Hi Andi Kleen.
Patch can hardly hellp, because there is another problem. Proc shows
neither the size of deleted files, nor what he's full with /tmp.
On Mon, Jul 13, 2009 at 01:59:17PM +0200, Andi Kleen wrote:
> > But you can see the file date, these are new files and by next search
>
> I appended a patch that reports the size.
>
> -Andi
>
> ---
>
> VFS: Report file size for deleted files in /proc
>
> It's useful to know how big deleted files are, in case a file system
> is filled up unexpectedly and the admin wants to track down which
> process is the culprit. This patch reports the file size
> in addition to (deleted) for /proc/*/fd links.
>
> This modifies a public interface, but it's reasonable to assume
> that noone relies on the particular format here.
>
> Signed-off-by: Andi Kleen <[email protected]>
>
> ---
> fs/dcache.c | 14 ++++++++++++--
> 1 file changed, 12 insertions(+), 2 deletions(-)
>
> Index: linux-2.6.31-rc1-ak/fs/dcache.c
> ===================================================================
> --- linux-2.6.31-rc1-ak.orig/fs/dcache.c
> +++ linux-2.6.31-rc1-ak/fs/dcache.c
> @@ -1910,9 +1910,19 @@ char *__d_path(const struct path *path,
>
> spin_lock(&vfsmount_lock);
> prepend(&end, &buflen, "\0", 1);
> - if (d_unlinked(dentry) &&
> - (prepend(&end, &buflen, " (deleted)", 10) != 0))
> + if (d_unlinked(dentry)) {
> + char buf[40];
> + int n;
> + if (dentry->d_inode) { /* check needed? */
> + n = snprintf(buf, sizeof buf, " (deleted; size %lluK)",
> + dentry->d_inode->i_size >> 10);
> + } else {
> + strcpy(buf, " (deleted)");
> + n = 10;
> + }
> + if (prepend(&end, &buflen, buf, n) != 0)
> goto Elong;
> + }
>
> if (buflen < 1)
> goto Elong;
>
> --
> [email protected] -- Speaking for myself only.
--
BRGDS. Alexey Vlasov.