Hello NFS hackers,
I am working on a netbooted linux setup with / living on tmpfs and a
shared /usr NFS mounted (readonly) from a Netapp server.
While working on this I noticed that certain workloads ran much slower
on the netbooted system than they would on a standalone one. One
example is running ./configure for any package using GNU autoconf, say
emacs-21.1. After some digging around I found that during the
./configure run a large number of NFS requests are generated against
the server holding /usr (even though the emacs source tree lived
somewhere else), about 120000 requests total. Most of these (over
100000) are "access" requests. My guess is that these are generated by
cache validations when executables or libraries are accessed from
/usr/bin and /usr/lib and perhaps during PATH searches as well.
I switched the /usr mount to NFS v2 and this reduced the number of
requests during the ./configure run to about 2000 calls total. That's
120000 for v3 vs. 2000 for v2!
Here are the mount options from /proc/mounts in each case:
ro,v3,rsize=8192,wsize=8192,hard,udp,nocto,nolock
ro,v2,rsize=8192,wsize=8192,hard,udp,nocto,nolock
I have initially tried the v3 mount without the explicit
rsize/wsize=8192, same results. nocto appears to reduce the number of
requests a bit when using v2 but it doesn't seem to make a difference
with v3. The nolock is needed because portmap is not yet running when
/usr is mounted.
The netbooted client is running a vanilla 2.4.20 kernel, the server is
a Netapp. There is plenty of RAM on the client, over 50MB is used for
caching during the run.
Why does the NFS v3 mount generate so much traffic? Mounting via v2
is an acceptable workaround for now but I would rather use v3 if it
wasn't for this issue.
Any clues would be greatly appreciated,
--
Istvan
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>>>>> " " == Istvan Marko <[email protected]> writes:
> total. Most of these (over
> 100000) are "access" requests. My guess is that these are
> generated by
> cache validations when executables or libraries are accessed
> from /usr/bin and /usr/lib and perhaps during PATH searches as
> well.
Solution: Never, ever, ever, run ./configure as 'root'.
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Trond Myklebust <[email protected]> writes:
>>>>>> " " == Istvan Marko <[email protected]> writes:
>
>
> > total. Most of these (over
> > 100000) are "access" requests. My guess is that these are
> > generated by
> > cache validations when executables or libraries are accessed
> > from /usr/bin and /usr/lib and perhaps during PATH searches as
> > well.
>
> Solution: Never, ever, ever, run ./configure as 'root'.
Thank you Trond, you are right on the mark! But what exactly is going
on here? Is this related to root_squash (or the Netapp equivalent)? Is
it just ./configure that does something weird or are there other
common situations where this "access storm" might get triggered with
an NFS mounted /usr?
Thanks again for your help!
--
Istvan
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tuesday 10 December 2002 11:58, Istvan Marko wrote:
> Thank you Trond, you are right on the mark! But what exactly is going
> on here? Is this related to root_squash (or the Netapp equivalent)? Is
> it just ./configure that does something weird or are there other
> common situations where this "access storm" might get triggered with
> an NFS mounted /usr?
In 2.4.x, the NFSv3 ACCESS call is not fully implemented: it lacks the ability
to cache results. For this reason it is only used to check for root squashing
and for those specific cases where the kernel tries to override the standard
UNIX mode permission checks.
In 2.5.x, on the other hand, access checking *is* cached.
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Trond Myklebust wrote:
> In 2.4.x, the NFSv3 ACCESS call is not fully implemented: it lacks the
ability
> to cache results. For this reason it is only used to check for root
squashing
> and for those specific cases where the kernel tries to override the
standard
> UNIX mode permission checks.
>
Last week when I reported slowness on 2.4.20+NFS_ALL's TCP client you said
that it was "probably a consequence of the ACCESS patch". It seems like
the ACCESS patch should make the client run faster. Can you clarify a bit
under which conditions the client would run faster and which it would run
slower if the ACCESS patch is installed? In particular is it a benefit to
run the ACCESS patch if you are mounting without root_squash and
performing a lot of actions as the root (or perhaps any other) user? I'm
sure it depends on workload but perhaps there are some general guidelines.
Thanks!
andrew
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs