On Mon, Jul 30, 2007 at 02:55:14PM +0200, Stefan Walter wrote:
> There are however two issues for which we could not find an easy
> solution:
>
> 1. For every client rpc.mountd and the kernel seem to exchange
> and use lists with _all_ netgroups used in exports that are
> relevant for granting permission to some share for a particular
> client. We could imagine two optimizations here:
>
> * Resolve netgroups and only put the (member) netgroups that
> contained the host name that would be used to authorize
> a mount in the list.
>
> * Use the list of mounted paths per client and only put the
> netgroup(s) used to export paths that are actually mounted
> on a client.
>
> This also caused us severe performance problems because
> rpc.mountd queries all these netgroups. We were initially using
> a LDAP and mouting a directory took up to ten seconds
> during which rpc.mountd was busily querying the LDAP server.
> We got this down to two seconds using file based netgroups.
>
> 2. Using a fixed size for NFSCLNT_IDMAX does not scale. Mounting
> shares on a client for which the 'if' clause of the quick fix
> becomes true will not be possible. We thought about enlarging
> NFSCLNT_IDMAX and using a custom kernel but dropped the idea.
>
> Our ultimate goal is to get Red Hat fix the code in nfs-utils 1.0.6
> that is used in RHEL4. A first step would be to get a suitable fix in
> the current nfs-utils.
That's an interesting problem. Thanks for the report!
I don't believe that long comma-delimited string actually has any
meaning to the kernel--as far as the kernel is concerned, it's just an
opaque object that will be passed back to mountd later (along with a
path name) to get export options.
So I suppose that string could be replaced by a hash, or maybe even just
by the ip address of the particular host--the disadvantage to the latter
being that it would require the kernel to keep a separate export for
each client address.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tue, 31 Jul 2007 10:48:24 -0400
"J. Bruce Fields" <[email protected]> wrote:
> On Mon, Jul 30, 2007 at 02:55:14PM +0200, Stefan Walter wrote:
> > There are however two issues for which we could not find an easy
> > solution:
> >
> > 1. For every client rpc.mountd and the kernel seem to exchange
> > and use lists with _all_ netgroups used in exports that are
> > relevant for granting permission to some share for a particular
> > client. We could imagine two optimizations here:
> >
> > * Resolve netgroups and only put the (member) netgroups that
> > contained the host name that would be used to authorize
> > a mount in the list.
> >
> > * Use the list of mounted paths per client and only put the
> > netgroup(s) used to export paths that are actually mounted
> > on a client.
> >
> > This also caused us severe performance problems because
> > rpc.mountd queries all these netgroups. We were initially using
> > a LDAP and mouting a directory took up to ten seconds
> > during which rpc.mountd was busily querying the LDAP server.
> > We got this down to two seconds using file based netgroups.
> >
> > 2. Using a fixed size for NFSCLNT_IDMAX does not scale. Mounting
> > shares on a client for which the 'if' clause of the quick fix
> > becomes true will not be possible. We thought about enlarging
> > NFSCLNT_IDMAX and using a custom kernel but dropped the idea.
> >
> > Our ultimate goal is to get Red Hat fix the code in nfs-utils 1.0.6
> > that is used in RHEL4. A first step would be to get a suitable fix in
> > the current nfs-utils.
>
> That's an interesting problem. Thanks for the report!
>
> I don't believe that long comma-delimited string actually has any
> meaning to the kernel--as far as the kernel is concerned, it's just an
> opaque object that will be passed back to mountd later (along with a
> path name) to get export options.
>
> So I suppose that string could be replaced by a hash, or maybe even just
> by the ip address of the particular host--the disadvantage to the latter
> being that it would require the kernel to keep a separate export for
> each client address.
>
> --b.
>
This issue with rpc.mountd crashing sounds familiar. It may be the same
as RHBZ 242465 (which is unfortunately marked private so I can't really
refer you to it). If it is the same, then there is a fix slated for
inclusion in RHEL 4.6.
If you have a non-critical place to do so, I'd suggest testing the
RHEL4 nfs-utils packages available on my people page:
http://people.redhat.com/jlayton
--
Jeff Layton <[email protected]>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Jeff Layton wrote:
> This issue with rpc.mountd crashing sounds familiar. It may be the same
> as RHBZ 242465 (which is unfortunately marked private so I can't really
> refer you to it). If it is the same, then there is a fix slated for
> inclusion in RHEL 4.6.
>
> If you have a non-critical place to do so, I'd suggest testing the
> RHEL4 nfs-utils packages available on my people page:
>
> http://people.redhat.com/jlayton
>
>
I just downloaded and tried nfs-utils-1.0.6-81.EL4.i386.rpm but the included
rpc.mountd crashes the same way. I checked the source rpm on your
website and the relevant sections of the code (including the use of strcpy)
are the same as in the current RHEL4 nfs-utils source rpm.
BTW, I already opened a case with the Red Hat support (case 1736884).
- Stefan
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tue, 31 Jul 2007 10:48:24 -0400
"J. Bruce Fields" <[email protected]> wrote:
> On Mon, Jul 30, 2007 at 02:55:14PM +0200, Stefan Walter wrote:
> > There are however two issues for which we could not find an easy
> > solution:
> >
> > 1. For every client rpc.mountd and the kernel seem to exchange
> > and use lists with _all_ netgroups used in exports that are
> > relevant for granting permission to some share for a particular
> > client. We could imagine two optimizations here:
> >
> > * Resolve netgroups and only put the (member) netgroups that
> > contained the host name that would be used to authorize
> > a mount in the list.
> >
> > * Use the list of mounted paths per client and only put the
> > netgroup(s) used to export paths that are actually mounted
> > on a client.
> >
> > This also caused us severe performance problems because
> > rpc.mountd queries all these netgroups. We were initially using
> > a LDAP and mouting a directory took up to ten seconds
> > during which rpc.mountd was busily querying the LDAP server.
> > We got this down to two seconds using file based netgroups.
> >
> > 2. Using a fixed size for NFSCLNT_IDMAX does not scale. Mounting
> > shares on a client for which the 'if' clause of the quick fix
> > becomes true will not be possible. We thought about enlarging
> > NFSCLNT_IDMAX and using a custom kernel but dropped the idea.
> >
> > Our ultimate goal is to get Red Hat fix the code in nfs-utils 1.0.6
> > that is used in RHEL4. A first step would be to get a suitable fix in
> > the current nfs-utils.
>
> That's an interesting problem. Thanks for the report!
>
> I don't believe that long comma-delimited string actually has any
> meaning to the kernel--as far as the kernel is concerned, it's just an
> opaque object that will be passed back to mountd later (along with a
> path name) to get export options.
>
> So I suppose that string could be replaced by a hash, or maybe even just
> by the ip address of the particular host--the disadvantage to the latter
> being that it would require the kernel to keep a separate export for
> each client address.
>
> --b.
>
I started having a look at this today. The original patches that I
proposed to clean up the rmtab a few months ago also eliminated this
comma-delimited string. Neil had valid objections to it at the time,
but if we switched to using the IP address as a cache key like Bruce
describes then doing that becomes more reasonable.
The only downside I see is the one Bruce points out -- the size of
the kernel export cache would increase. I don't have a feel for whether
this is a show stopper, however.
Neil, do you have thoughts on what you'd consider a reasonable approach
to fixing this?
--
Jeff Layton <[email protected]>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, Aug 02, 2007 at 11:32:55AM -0400, Jeff Layton wrote:
> On Tue, 31 Jul 2007 10:48:24 -0400
> "J. Bruce Fields" <[email protected]> wrote:
> > That's an interesting problem. Thanks for the report!
> >
> > I don't believe that long comma-delimited string actually has any
> > meaning to the kernel--as far as the kernel is concerned, it's just an
> > opaque object that will be passed back to mountd later (along with a
> > path name) to get export options.
> >
> > So I suppose that string could be replaced by a hash, or maybe even just
> > by the ip address of the particular host--the disadvantage to the latter
> > being that it would require the kernel to keep a separate export for
> > each client address.
>
> I started having a look at this today. The original patches that I
> proposed to clean up the rmtab a few months ago also eliminated this
> comma-delimited string. Neil had valid objections to it at the time,
> but if we switched to using the IP address as a cache key like Bruce
> describes then doing that becomes more reasonable.
>
> The only downside I see is the one Bruce points out -- the size of
> the kernel export cache would increase. I don't have a feel for whether
> this is a show stopper, however.
Yeah, there might be some risk of solving that problem at the expense of
people with tons of clients all matching *.example.com. The actual
export objects are actually pretty small--68 bytes, last I checked?
You'd also need two upcalls to mountd per client (for ip address and
export, as opposed to just ip address). I don't know what other costs
there might be.
What about the idea of hashing the comma-delimited list and passing
that? You'd want hash large enough to be collision-free--might as well
use some cryptographic hash, I guess, though it may be mild overkill.
Is there any reason why that wouldn't work?
Could you remind me what problems you were trying to fix with your rmtab
cleanup?
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, 2 Aug 2007 12:05:36 -0400
"J. Bruce Fields" <[email protected]> wrote:
> On Thu, Aug 02, 2007 at 11:32:55AM -0400, Jeff Layton wrote:
> > On Tue, 31 Jul 2007 10:48:24 -0400
> > "J. Bruce Fields" <[email protected]> wrote:
> > > That's an interesting problem. Thanks for the report!
> > >
> > > I don't believe that long comma-delimited string actually has any
> > > meaning to the kernel--as far as the kernel is concerned, it's just an
> > > opaque object that will be passed back to mountd later (along with a
> > > path name) to get export options.
> > >
> > > So I suppose that string could be replaced by a hash, or maybe even just
> > > by the ip address of the particular host--the disadvantage to the latter
> > > being that it would require the kernel to keep a separate export for
> > > each client address.
> >
> > I started having a look at this today. The original patches that I
> > proposed to clean up the rmtab a few months ago also eliminated this
> > comma-delimited string. Neil had valid objections to it at the time,
> > but if we switched to using the IP address as a cache key like Bruce
> > describes then doing that becomes more reasonable.
> >
> > The only downside I see is the one Bruce points out -- the size of
> > the kernel export cache would increase. I don't have a feel for whether
> > this is a show stopper, however.
>
> Yeah, there might be some risk of solving that problem at the expense of
> people with tons of clients all matching *.example.com. The actual
> export objects are actually pretty small--68 bytes, last I checked?
> You'd also need two upcalls to mountd per client (for ip address and
> export, as opposed to just ip address). I don't know what other costs
> there might be.
>
Yeah, that could make using the ipaddr too costly...
> What about the idea of hashing the comma-delimited list and passing
> that? You'd want hash large enough to be collision-free--might as well
> use some cryptographic hash, I guess, though it may be mild overkill.
> Is there any reason why that wouldn't work?
>
Hashing would certainly work. The only issue though is that this
doesn't take care of the second problem, which is that NFSCLNT_IDMAX is
too small for some environments. So this change would need to be
coupled with an increase in this hardcoded value, change m_hostname to
be dynamically allocated, or change the comma-delimited string to some
other scheme entirely (maybe a linked list of some sort?).
As far as the hashing scheme, I guess we could use the glibc crypt()
routine that does md5 hashing. I'd like to avoid adding dependencies
on new libs if possible.
> Could you remind me what problems you were trying to fix with your rmtab
> cleanup?
>
Those patches were really intended to get showmount -a to look correct.
My first pass at it changed the m_hostname field to hold an actual
hostname, got rid of the comma-separated strings and just had mountd do
repeated calls to client_check to check the host's validity.
This was problematic though with a multi-homed client though since the
export table key then became a hostname rather than the comma-delimited
string and more than 1 ip addr could resolve to it. If we use the ipaddr
as a key, then this problem goes away, but then we have the other issues
you've detailed above.
--
Jeff Layton <[email protected]>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs