Hi,
I'm running a server that has 1500+ mounted local filesystems.
All of these local filesystems are exported through NFS by this server.
When an NFS-client tries to access one of these exported filesystems,
for example doing a simple 'ls' in a mounted NFS, rpc.mountd on the
server takes 100% CPU and starts to spin over reading /etc/mtab which is
linked to /proc/mounts and totals to about 200KB in size. The client
stalls all the while rpc.mountd is busy reading /etc/mtab for every
filesystem mounted, and then some more.
I made a strace of rpc.mountd during such a run to gather some stats:
# cut -d'(' -f1 straceout.25103 | sort | uniq -c | sort -n
3 fadvise64
6 ioctl
11 lseek
33 readlink
43 select
53 write
809 statfs
814 lstat
1282 stat
17043 mmap
17043 munmap
17088 fstat
17895 open
21728 openat
37295 close
40352 getdents
458616 newfstatat
3763233 read
# grep -c "open.*mtab" straceout.25103
17028
This entire process of reading /etc/mtab 17028 times takes a lot of time
during which the client stalls, but in the end 'it just works fine'.
It just takes ages when you try to tab-complete on a client.
What would be needed to debug and optimise this?
Could someone point me to the code that is involved in doing this?
I'm not a good C-coder but perhaps i can debug more...
Any insights appreciated!
With regards,
-Sander.
--
| Happiness isn't enough for me. I demand EUPHORIA!
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2
On 03/25/2015 05:37 AM, Sander Smeenk wrote:
> I'm running a server that has 1500+ mounted local filesystems.
> All of these local filesystems are exported through NFS by this server.
>
> When an NFS-client tries to access one of these exported filesystems,
> for example doing a simple 'ls' in a mounted NFS, rpc.mountd on the
> server takes 100% CPU and starts to spin over reading /etc/mtab which is
> linked to /proc/mounts and totals to about 200KB in size. The client
> stalls all the while rpc.mountd is busy reading /etc/mtab for every
> filesystem mounted, and then some more.
>
> This entire process of reading /etc/mtab 17028 times takes a lot of time
> during which the client stalls, but in the end 'it just works fine'.
> It just takes ages when you try to tab-complete on a client.
Taking a quick look... it appears the only way rpc.mountd
will read /etc/mtab is if the 'crossmnt' export flag is
set. Is this the case?
>
> What would be needed to debug and optimise this?
Maybe reorder you exports so the 'hot' file systems are on the
top of the list...
> Could someone point me to the code that is involved in doing this?
utils/mountd/cache.c:nfsd_fh():path = next_mnt(&mnt, exp->m_export.e_path);
steved.
Quoting Steve Dickson ([email protected]):
> > This entire process of reading /etc/mtab 17028 times takes a lot of
> > time during which the client stalls, but in the end 'it just works
> > fine'. It just takes ages when you try to tab-complete on a client.
> Taking a quick look... it appears the only way rpc.mountd will read
> /etc/mtab is if the 'crossmnt' export flag is set. Is this the case?
Yes.
IIRC this was necessary for this situation:
Server has filesytems mounted on /mnt/pt1 and /mnt/pt1/foo.
Both /mnt/pt1 and /mnt/pt1/foo are exported with 'crossmnt'.
Clients that mount just /mnt/pt1 can also reach /mnt/pt1/foo.
> > Could someone point me to the code that is involved in doing this?
> utils/mountd/cache.c:nfsd_fh():path = next_mnt(&mnt, exp->m_export.e_path);
Thanks. I'll dig around some more and see what i can find.
-Sndr.
--
| What are the little plastic endings on your shoelaces called?
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2