I have a situation where rpc.mountd is growing continuously and I am not
sure if it's a memory leak or expected behavior that can be controlled
with some configuration option.
Top is used to watch rpc.mountd over time and I can see Virtual size
continually growing. In this environment there is no swap space so RSS
is growing also. When RSS gets close to Virtual size, Virtual size
jumps up by 128K until RSS again "catches up" to VIRTUAL which then
jumps up by 128K again:
VIRT RES
9951 root 16 0 1812 1052 668 S 0.0 0.0 0:20.07 rpc.mountd
--nfs-version 2 --nfs-version 3
9951 root 16 0 1812 1052 668 S 0.0 0.0 0:20.19 rpc.mountd
--nfs-version 2 --nfs-version 3
9951 root 15 0 1812 1052 668 S 0.0 0.0 0:20.44 rpc.mountd
--nfs-version 2 --nfs-version 3
9951 root 15 0 1940 1056 668 S 0.0 0.0 0:20.57 rpc.mountd
--nfs-version 2 --nfs-version 3
9951 root 16 0 1940 1056 668 S 0.0 0.0 0:20.69 rpc.mountd
--nfs-version 2 --nfs-version 3
9951 root 16 0 1940 1056 668 S 0.0 0.0 0:20.70 rpc.mountd
--nfs-version 2 --nfs-version 3
RES keeps incrementing every so often and eventually:
VIRT RES
9951 root 16 0 1940 1180 668 S 0.0 0.0 0:38.85 rpc.mountd
--nfs-version 2 --nfs-version 3
9951 root 16 0 1940 1180 668 S 0.0 0.0 0:38.99 rpc.mountd
--nfs-version 2 --nfs-version 3
9951 root 16 0 2068 1184 668 S 0.0 0.0 0:39.14 rpc.mountd
--nfs-version 2 --nfs-version 3
9951 root 16 0 2068 1184 668 S 0.0 0.0 0:39.33 rpc.mountd
--nfs-version 2 --nfs-version 3
This pattern continues.
I added some xlog() statements in rpc.mountd to try and show me if there
were some malloc's without free's. I didn't see anything, but here are
the routines that seemed to get called frequently:
Apr 26 18:55:13 unit0 mountd[24268]: auth_unix_ip client alloced
0x80695e0
Apr 26 18:55:13 unit0 mountd[24268]: auth_unix_ip client freed 0x80695e0
Apr 26 18:55:13 unit0 mountd[24268]: nfsd_export alocated dom 0x80695e0
Apr 26 18:55:13 unit0 mountd[24268]: nfsd_export alocated path 0x80695f0
Apr 26 18:55:13 unit0 mountd[24268]: nfsd_export : free dom 0x80695e0
Apr 26 18:55:13 unit0 mountd[24268]: nfsd_export : free path 0x80695f0
Apr 26 18:55:13 unit0 mountd[24268]: nfsd_fh : dom allocated 0x8069600
Apr 26 18:55:13 unit0 mountd[24268]: nfsd_fh : freed dom 0x8069600
Apr 26 19:10:33 unit0 mountd[24268]: auth_unix_ip client alloced
0x8069618
Apr 26 19:10:33 unit0 mountd[24268]: auth_unix_ip client freed 0x8069618
Apr 26 19:10:33 unit0 mountd[24268]: nfsd_export alocated dom 0x8069618
Apr 26 19:10:33 unit0 mountd[24268]: nfsd_export alocated path 0x8069628
Apr 26 19:10:33 unit0 mountd[24268]: nfsd_export : free dom 0x8069618
Apr 26 19:10:33 unit0 mountd[24268]: nfsd_export : free path 0x8069628
Apr 26 19:10:33 unit0 mountd[24268]: nfsd_fh : dom allocated 0x8069638
Apr 26 19:10:33 unit0 mountd[24268]: nfsd_fh : freed dom 0x8069638
It seems to be these three routines that get called over and over, at
least so far that I have added xlog()'s to. These seem to be making some
interactions with the kernel nfsd via /proc, but I am not real sure how
that would impact rpc.mountd's virtual size.
I am using the kernel NFSD, the kernel version is 2.6.10, the nfs-utils
version is nfs-utils-1.0.7
I don't know if this growth is expected or if it will eventually stop,
over a period of days it has not stopped growing, and if there may be
some configuration option that can control it.
Has anyone seen this behavior before ?
Is it expected and controllable or is there a memory leak here ?
Thanks
Chris Kottaridis
Senior Engineer
Wind River Systems
719-522-9786
I have this problem too for a long time and already reported it, but
propably forgotten. My rpc.mountd is actually 350MB and continue
growing, with nfs-utils 1.1.0-rc2:
# rpc.mountd --version
kmountd 1.1.0-rc2
# top
2259 root 15 0 340m 331m 744 S 0.0 16.5 0:20.12 rpc.mountd
I already provided a copy of the /proc/<pid>/smaps which show the main
problem :
00514000-15033000 rw-p 00514000 00:00 0
[heap]
Size: 339068 kB
Rss: 338204 kB
I think there is a memory leak here.
Gabriel Barazer
On 04/30/2007 19:41:07 +0200, "Kottaridis, Chris"
<[email protected]> wrote:
> I have a situation where rpc.mountd is growing continuously and I am not
> sure if it's a memory leak or expected behavior that can be controlled
> with some configuration option.
>
> Top is used to watch rpc.mountd over time and I can see Virtual size
> continually growing. In this environment there is no swap space so RSS
> is growing also. When RSS gets close to Virtual size, Virtual size
> jumps up by 128K until RSS again "catches up" to VIRTUAL which then
> jumps up by 128K again:
>
>
> VIRT RES
> 9951 root 16 0 1812 1052 668 S 0.0 0.0 0:20.07 rpc.mountd
> --nfs-version 2 --nfs-version 3
> 9951 root 16 0 1812 1052 668 S 0.0 0.0 0:20.19 rpc.mountd
> --nfs-version 2 --nfs-version 3
> 9951 root 15 0 1812 1052 668 S 0.0 0.0 0:20.44 rpc.mountd
> --nfs-version 2 --nfs-version 3
> 9951 root 15 0 1940 1056 668 S 0.0 0.0 0:20.57 rpc.mountd
> --nfs-version 2 --nfs-version 3
> 9951 root 16 0 1940 1056 668 S 0.0 0.0 0:20.69 rpc.mountd
> --nfs-version 2 --nfs-version 3
> 9951 root 16 0 1940 1056 668 S 0.0 0.0 0:20.70 rpc.mountd
> --nfs-version 2 --nfs-version 3
>
> RES keeps incrementing every so often and eventually:
>
> VIRT RES
> 9951 root 16 0 1940 1180 668 S 0.0 0.0 0:38.85 rpc.mountd
> --nfs-version 2 --nfs-version 3
> 9951 root 16 0 1940 1180 668 S 0.0 0.0 0:38.99 rpc.mountd
> --nfs-version 2 --nfs-version 3
> 9951 root 16 0 2068 1184 668 S 0.0 0.0 0:39.14 rpc.mountd
> --nfs-version 2 --nfs-version 3
> 9951 root 16 0 2068 1184 668 S 0.0 0.0 0:39.33 rpc.mountd
> --nfs-version 2 --nfs-version 3
>
> This pattern continues.
>
> I added some xlog() statements in rpc.mountd to try and show me if there
> were some malloc's without free's. I didn't see anything, but here are
> the routines that seemed to get called frequently:
>
> Apr 26 18:55:13 unit0 mountd[24268]: auth_unix_ip client alloced 0x80695e0
> Apr 26 18:55:13 unit0 mountd[24268]: auth_unix_ip client freed 0x80695e0
> Apr 26 18:55:13 unit0 mountd[24268]: nfsd_export alocated dom 0x80695e0
> Apr 26 18:55:13 unit0 mountd[24268]: nfsd_export alocated path 0x80695f0
> Apr 26 18:55:13 unit0 mountd[24268]: nfsd_export : free dom 0x80695e0
> Apr 26 18:55:13 unit0 mountd[24268]: nfsd_export : free path 0x80695f0
> Apr 26 18:55:13 unit0 mountd[24268]: nfsd_fh : dom allocated 0x8069600
> Apr 26 18:55:13 unit0 mountd[24268]: nfsd_fh : freed dom 0x8069600
> Apr 26 19:10:33 unit0 mountd[24268]: auth_unix_ip client alloced 0x8069618
> Apr 26 19:10:33 unit0 mountd[24268]: auth_unix_ip client freed 0x8069618
> Apr 26 19:10:33 unit0 mountd[24268]: nfsd_export alocated dom 0x8069618
> Apr 26 19:10:33 unit0 mountd[24268]: nfsd_export alocated path 0x8069628
> Apr 26 19:10:33 unit0 mountd[24268]: nfsd_export : free dom 0x8069618
> Apr 26 19:10:33 unit0 mountd[24268]: nfsd_export : free path 0x8069628
> Apr 26 19:10:33 unit0 mountd[24268]: nfsd_fh : dom allocated 0x8069638
> Apr 26 19:10:33 unit0 mountd[24268]: nfsd_fh : freed dom 0x8069638
>
> It seems to be these three routines that get called over and over, at
> least so far that I have added xlog()'s to. These seem to be making some
> interactions with the kernel nfsd via /proc, but I am not real sure how
> that would impact rpc.mountd's virtual size.
>
> I am using the kernel NFSD, the kernel version is 2.6.10, the nfs-utils
> version is nfs-utils-1.0.7
>
> I don't know if this growth is expected or if it will eventually stop,
> over a period of days it has not stopped growing, and if there may be
> some configuration option that can control it.
>
> Has anyone seen this behavior before ?
>
> Is it expected and controllable or is there a memory leak here ?
>
> Thanks
>
> Chris Kottaridis
> Senior Engineer
> Wind River Systems
> 719-522-9786
>
>
>
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
There's a comment in the support/rpc/svc.c file for the svc_getreq()
routine that makes the following statement:
/* ********************* SERVER INPUT STUFF ********************** */
/*
* Get server side input from some transport.
*
* Statement of authentication parameters management:
* This function owns and manages all authentication parameters,
specifically
* the "raw" parameters (msg.rm_call.cb_cred and msg.rm_call.cb_verf)
and
* the "cooked" credentials (rqst->rq_clntcred).
* However, this function does not know the structure of the cokked
* credentials, so it make the following assumptions:
* a) the structure is contiguous (no pointers), and
* b) the cred structure size does not exceed RQCRED_SIZE bytes.
* In all events, all three parameters are freed upon exit from this
routine.
* The storage is trivially management on the call stack in user land,
but
* is mallocated in kernel land.
*/
I don't really know what's going on here, so I am not too sure about the
reference in the last two lines about memory management. This wouldn't
be some way in which the kernel is allocating space to the rpc.mountd
process, so if there's a leak it's in the kernel and not in the
rpc.mountd userland code would it ?
Thanks
Chris Kottaridis
Senior Engineer
Wind River Systems
719-522-9786
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Monday April 30, [email protected] wrote:
> I have a situation where rpc.mountd is growing continuously and I am not
> sure if it's a memory leak or expected behavior that can be controlled
> with some configuration option.
>
...
>
> I am using the kernel NFSD, the kernel version is 2.6.10, the nfs-utils
> version is nfs-utils-1.0.7
Can you try this patch - unless you are using rpcsec/gss, it is the
only known leak likely to affect that version.
If that doesn't help you might need to try valgrind or similar.
NeilBrown
commit 08964495c2a65f6228426e4565a50dae5b75834d
Author: Steinar H. Gunderson <[email protected]>
Date: Fri Mar 16 11:26:35 2007 +1100
Fix memory leak in mountd.
Signed-off-by: Neil Brown <[email protected]>
diff --git a/support/export/client.c b/support/export/client.c
index 33dfdb0..686c744 100644
--- a/support/export/client.c
+++ b/support/export/client.c
@@ -329,6 +329,7 @@ add_name(char *old, char *add)
strcat(new, ",");
strcat(new, cp);
}
+ free(old);
return new;
}
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Monday April 30, [email protected] wrote:
> I have this problem too for a long time and already reported it, but
> propably forgotten. My rpc.mountd is actually 350MB and continue
> growing, with nfs-utils 1.1.0-rc2:
>
> # rpc.mountd --version
> kmountd 1.1.0-rc2
Is this compiled with libblkid (the default)?
There is a memory leak in libblkid that was only fixed quite recently
so unless your libblkid is very new, you should use
./configure --without-uuid
NeilBrown
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
I'll give it a try, but time on the machine can be a problem to get so
it may take a while to actually try it.
I'll let you know how it goes
Thanks
Chris Kottaridis
Senior Engineer
Wind River Systems
719-522-9786
-----Original Message-----
From: Neil Brown [mailto:[email protected]]
Sent: Monday, April 30, 2007 3:20 PM
To: Kottaridis, Chris
Cc: [email protected]
Subject: Re: [NFS] Rpc.mountd growing 6 MB/day
On Monday April 30, [email protected] wrote:
> I have a situation where rpc.mountd is growing continuously and I am
> not sure if it's a memory leak or expected behavior that can be
> controlled with some configuration option.
>
...
>
> I am using the kernel NFSD, the kernel version is 2.6.10, the
> nfs-utils version is nfs-utils-1.0.7
Can you try this patch - unless you are using rpcsec/gss, it is the only
known leak likely to affect that version.
If that doesn't help you might need to try valgrind or similar.
NeilBrown
commit 08964495c2a65f6228426e4565a50dae5b75834d
Author: Steinar H. Gunderson <[email protected]>
Date: Fri Mar 16 11:26:35 2007 +1100
Fix memory leak in mountd.
Signed-off-by: Neil Brown <[email protected]>
diff --git a/support/export/client.c b/support/export/client.c index
33dfdb0..686c744 100644
--- a/support/export/client.c
+++ b/support/export/client.c
@@ -329,6 +329,7 @@ add_name(char *old, char *add)
strcat(new, ",");
strcat(new, cp);
}
+ free(old);
return new;
}
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On 05/01/2007 5:47:19 +0200, Neil Brown <[email protected]> wrote:
> On Tuesday May 1, [email protected] wrote:
>> I will try with mountd compiled without libblkid, although my version is
>> the most up to date release : 1.39 from e2fsprogs.
>
> Bug is in 1.39, will be fix in 1.40. More info at
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=413661
>
I think the bug was indeed related to the libblkid memleak : 100's of
mounts later, mountd size is kept low ~ 1MB.
Is this dependency to libblkid really needed in the nfs-utils ?
According to the mountd sources (the only place where there is BLKID
related code), the get_uuid function in onyl used in the nfsd_fh
function, used to find a mount point from a fsid export entry.
Since the current version of e2fsprogs is still 1.39, this bug is likely
to happen to much people, hence the best thing to do would be disabling
the uuid support by default, wouldn't be ?
Gabriel
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wednesday May 2, [email protected] wrote:
> On 05/01/2007 5:47:19 +0200, Neil Brown <[email protected]> wrote:
>
> > On Tuesday May 1, [email protected] wrote:
> >> I will try with mountd compiled without libblkid, although my version is
> >> the most up to date release : 1.39 from e2fsprogs.
> >
> > Bug is in 1.39, will be fix in 1.40. More info at
> > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=413661
> >
>
> I think the bug was indeed related to the libblkid memleak : 100's of
> mounts later, mountd size is kept low ~ 1MB.
>
> Is this dependency to libblkid really needed in the nfs-utils ?
> According to the mountd sources (the only place where there is BLKID
> related code), the get_uuid function in onyl used in the nfsd_fh
> function, used to find a mount point from a fsid export entry.
>
> Since the current version of e2fsprogs is still 1.39, this bug is likely
> to happen to much people, hence the best thing to do would be disabling
> the uuid support by default, wouldn't be ?
Yes... I guess it should default to not providing uuid support unless
e2fsprogs 1.40 or later is installed. People with a prepatch should
still be able to do that.
I'll see if I can work out the auto-conf magic to make that happen
(though as I cannot install 1.40 or later, it might be hard to
test completely....)
NeilBrown
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
I will try with mountd compiled without libblkid, although my version is
the most up to date release : 1.39 from e2fsprogs.
Do I have to apply the previously posted patch as well ?
Gabriel
On 04/30/2007 23:24:07 +0200, Neil Brown <[email protected]> wrote:
> On Monday April 30, [email protected] wrote:
>> I have this problem too for a long time and already reported it, but
>> propably forgotten. My rpc.mountd is actually 350MB and continue
>> growing, with nfs-utils 1.1.0-rc2:
>>
>> # rpc.mountd --version
>> kmountd 1.1.0-rc2
>
> Is this compiled with libblkid (the default)?
> There is a memory leak in libblkid that was only fixed quite recently
> so unless your libblkid is very new, you should use
> ./configure --without-uuid
>
> NeilBrown
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
BTW, there is a bug in the configure script provided in the
nfs-utils-1.1.0-rc2 package : --without-uuid does't work, but
--disable-uuid does for excluding libblkid linking.
Gabriel
On 04/30/2007 23:24:07 +0200, Neil Brown <[email protected]> wrote:
> On Monday April 30, [email protected] wrote:
>> I have this problem too for a long time and already reported it, but
>> propably forgotten. My rpc.mountd is actually 350MB and continue
>> growing, with nfs-utils 1.1.0-rc2:
>>
>> # rpc.mountd --version
>> kmountd 1.1.0-rc2
>
> Is this compiled with libblkid (the default)?
> There is a memory leak in libblkid that was only fixed quite recently
> so unless your libblkid is very new, you should use
> ./configure --without-uuid
>
> NeilBrown
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tuesday May 1, [email protected] wrote:
> I will try with mountd compiled without libblkid, although my version is
> the most up to date release : 1.39 from e2fsprogs.
Bug is in 1.39, will be fix in 1.40. More info at
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=413661
>
> Do I have to apply the previously posted patch as well ?
>
No, that fix in already in 1.1.0-rc2
NeilBrown
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tuesday May 1, [email protected] wrote:
> BTW, there is a bug in the configure script provided in the
> nfs-utils-1.1.0-rc2 package : --without-uuid does't work, but
> --disable-uuid does for excluding libblkid linking.
Yes, --disable-uuid is the correct flag. I will fix the help message.
Thanks.
NeilBrown
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs