Hi,
After upgrading NFS client kernel to latest linux-next, NFS mount
failed:
# mount -t nfs pxe:/cc /cc
mount.nfs: access denied by server while mounting pxe:/cc
# uname -a
Linux hp 2.6.31-rc6-next-20090818 #61 SMP Thu Aug 20 14:46:10 CST 2009 x86_64 GNU/Linux
However server log says OK:
Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated mount request from 192.168.11.6:973 for /cc (/cc)
Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated unmount request from 192.168.11.6:974 for /cc (/cc)
However-2: nfsroot can be mounted at boot time. Server kernel has always been 2.6.30.
Any ideas?
Thanks,
Fengguang
On Sat, Aug 22, 2009 at 01:50:52AM +0800, Chuck Lever wrote:
>
> On Aug 20, 2009, at 10:36 PM, Trond Myklebust wrote:
>
> > On Fri, 2009-08-21 at 09:27 +0800, Wu Fengguang wrote:
> >> On Thu, Aug 20, 2009 at 09:02:29PM +0800, Trond Myklebust wrote:
> >>> On Thu, 2009-08-20 at 15:13 +0800, Wu Fengguang wrote:
> >>>> Hi,
> >>>>
> >>>> After upgrading NFS client kernel to latest linux-next, NFS mount
> >>>> failed:
> >>>>
> >>>> # mount -t nfs pxe:/cc /cc
> >>>> mount.nfs: access denied by server while mounting pxe:/cc
> >>>>
> >>>> # uname -a
> >>>> Linux hp 2.6.31-rc6-next-20090818 #61 SMP Thu Aug 20
> >>>> 14:46:10 CST 2009 x86_64 GNU/Linux
> >>>>
> >>>> However server log says OK:
> >>>>
> >>>> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated mount
> >>>> request from 192.168.11.6:973 for /cc (/cc)
> >>>> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated unmount
> >>>> request from 192.168.11.6:974 for /cc (/cc)
> >>>>
> >>>> However-2: nfsroot can be mounted at boot time. Server kernel has
> >>>> always been 2.6.30.
> >>>>
> >>>> Any ideas?
> >>>>
> >>>> Thanks,
> >>>> Fengguang
> >>>
> >>> Can you try again after enabling mount debugging on the NFS client?
> >>>
> >>> echo 512 > /proc/sys/sunrpc/nfs_debug
> >>
> >> I used 1024 and found the mount failed here in nfs_walk_authlist():
> >>
> >> dfprintk(MOUNT, "NFS: server does not support requested auth
> >> flavor\ n");
> >> nfs_umount(request);
> >
> > Thanks Fengguang!
> >
> > Chuck, this looks like one of yours. Could it be that you are hitting
> > the same Linux knfsd bug that Tom Haynes saw with a Solaris client?
> > AFAICR, the problem was that existing nfs servers do not set a default
> > auth flavour, and so you just have to try with auth_sys and see if it
> > succeeds...
>
> With 1024 set, the mount client's XDR routines should have listed the
> server's flavors, if any, in the system log. Should be something like
> "NFS: received 0 auth flavors" if the server didn't return any. Can
> you confirm that?
Yes. The full logs are:
Aug 21 09:54:38 hp kernel: [ 74.671552] NFS: nfs mount opts='nolock,addr=192.168.11.2'
Aug 21 09:54:38 hp kernel: [ 74.677150] NFS: parsing nfs mount option 'nolock'
Aug 21 09:54:38 hp kernel: [ 74.682204] NFS: parsing nfs mount option 'addr=192.168.11.2'
Aug 21 09:54:38 hp kernel: [ 74.688234] NFS: MNTPATH: '/cc'
Aug 21 09:54:38 hp kernel: [ 74.691468] NFS: sending MNT request for pxe:/cc
Aug 21 09:54:38 hp kernel: [ 74.704458] NFS: received 0 auth flavors
Aug 21 09:54:38 hp kernel: [ 74.708587] NFS: MNT request succeeded
Aug 21 09:54:38 hp kernel: [ 74.712435] NFS: server does not support requested auth flavor
Aug 21 09:54:38 hp kernel: [ 74.718358] NFS: sending UMNT request for pxe:/cc
> I'll try to post a fix later today.
Thanks!
Fengguang
On Thu, Aug 20, 2009 at 03:13:15PM +0800, Wu Fengguang wrote:
> Hi,
>
> After upgrading NFS client kernel to latest linux-next, NFS mount
> failed:
>
> # mount -t nfs pxe:/cc /cc
> mount.nfs: access denied by server while mounting pxe:/cc
>
> # uname -a
> Linux hp 2.6.31-rc6-next-20090818 #61 SMP Thu Aug 20 14:46:10 CST 2009 x86_64 GNU/Linux
>
> However server log says OK:
>
> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated mount request from 192.168.11.6:973 for /cc (/cc)
> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated unmount request from 192.168.11.6:974 for /cc (/cc)
>
> However-2: nfsroot can be mounted at boot time. Server kernel has always been 2.6.30.
Another bug:
[ 29.096228] general protection fault: 0000 [#1] SMP
[ 29.099602] last sysfs file: /sys/devices/virtual/net/sit0/flags
[ 29.099602] CPU 1
[ 29.099602] Modules linked in: drm iwlagn iwlcore snd_hda_codec_intelhdmi snd_hda_codec_idt snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq snd_timer snd_seq_device snd soundcore snd_page_alloc
[ 29.099602] Pid: 3104, comm: mount.nfs Not tainted 2.6.31-rc6-next-20090818 #61
[ 29.099602] RIP: 0010:[<ffffffff812d5aa6>] [<ffffffff812d5aa6>] __list_add+0x26/0xa0
[ 29.099602] RSP: 0018:ffff88001a827d18 EFLAGS: 00010246
[ 29.099602] RAX: ff880018fb288048 RBX: ffff880018fb2880 RCX: ffffffff81e0a6a0
[ 29.099602] RDX: ffff880018fb2880 RSI: ff880018fb288048 RDI: ffff8800189df548
[ 29.099602] RBP: ffff88001a827d38 R08: ffff880018fb29b8 R09: 0000000000000000
[ 29.099602] R10: 0000000000000000 R11: 0000000000000000 R12: ff880018fb288048
[ 29.099602] R13: ffff8800189df548 R14: ffff880018fb28f0 R15: ffff880018fb28f0
[ 29.099602] FS: 0000000000000000(0000) GS:ffff880003a00000(0000) knlGS:0000000000000000
[ 29.099602] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 29.099602] CR2: 00007fd2a085ab40 CR3: 0000000001001000 CR4: 00000000000006e0
[ 29.099602] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 29.099602] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 29.099602] Process mount.nfs (pid: 3104, threadinfo ffff88001a826000, task ffff88001b568f00)
[ 29.099602] Stack:
[ 29.099602] ffffffff81204ce8 ffff880018fb29a0 ffff880018fb28f0 ffff8800189df500
[ 29.099602] <0> ffff88001a827d68 ffffffff81204d0a 0000000300000000 ffff88001888e200
[ 29.099602] <0> ffff880018fb28f0 ffff880018fa1000 ffff88001a827d88 ffffffff81202a51
[ 29.099602] Call Trace:
[ 29.099602] [<ffffffff81204ce8>] ? nfs_release+0x48/0x90
[ 29.099602] [<ffffffff81204d0a>] nfs_release+0x6a/0x90
[ 29.099602] [<ffffffff81202a51>] nfs_file_release+0x61/0x90
[ 29.099602] [<ffffffff8112ca1e>] __fput+0x11e/0x280
[ 29.099602] [<ffffffff8112cba5>] fput+0x25/0x30
[ 29.099602] [<ffffffff81183df8>] removed_exe_file_vma+0x38/0x50
[ 29.099602] [<ffffffff81104d10>] remove_vma+0x90/0xa0
[ 29.099602] [<ffffffff81104e18>] exit_mmap+0xf8/0x180
[ 29.099602] [<ffffffff8104d887>] mmput+0x67/0xd0
[ 29.099602] [<ffffffff81052ead>] exit_mm+0x11d/0x160
[ 29.099602] [<ffffffff81054c08>] do_exit+0x138/0x840
[ 29.099602] [<ffffffff8100c06a>] ? sysret_check+0x2e/0x69
[ 29.099602] [<ffffffff8105535c>] do_group_exit+0x4c/0xc0
[ 29.099602] [<ffffffff810553e7>] sys_exit_group+0x17/0x20
[ 29.099602] [<ffffffff8100c032>] system_call_fastpath+0x16/0x1b
[ 29.099602] Code: 00 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 4c 89 6d f8 49 89 f4 48 8b 42 08 49 89 fd 48 89 d3 48 39 f0 75 27 <49> 8b 04 24 48 39 d8 75 43 4c 89 6b 08 49 89 5d 00 4d 89 65 08
[ 29.359600] RIP [<ffffffff812d5aa6>] __list_add+0x26/0xa0
[ 29.359600] RSP <ffff88001a827d18>
[ 29.369139] ---[ end trace 8c118d638a30b42e ]---
[ 29.373847] Fixing recursive fault but reboot is needed!
Thanks,
Fengguang
On Thu, 2009-08-20 at 15:13 +0800, Wu Fengguang wrote:
> Hi,
>
> After upgrading NFS client kernel to latest linux-next, NFS mount
> failed:
>
> # mount -t nfs pxe:/cc /cc
> mount.nfs: access denied by server while mounting pxe:/cc
>
> # uname -a
> Linux hp 2.6.31-rc6-next-20090818 #61 SMP Thu Aug 20 14:46:10 CST 2009 x86_64 GNU/Linux
>
> However server log says OK:
>
> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated mount request from 192.168.11.6:973 for /cc (/cc)
> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated unmount request from 192.168.11.6:974 for /cc (/cc)
>
> However-2: nfsroot can be mounted at boot time. Server kernel has always been 2.6.30.
>
> Any ideas?
>
> Thanks,
> Fengguang
Can you try again after enabling mount debugging on the NFS client?
echo 512 > /proc/sys/sunrpc/nfs_debug
Cheers
Trond
On Thu, Aug 20, 2009 at 09:02:29PM +0800, Trond Myklebust wrote:
> On Thu, 2009-08-20 at 15:13 +0800, Wu Fengguang wrote:
> > Hi,
> >
> > After upgrading NFS client kernel to latest linux-next, NFS mount
> > failed:
> >
> > # mount -t nfs pxe:/cc /cc
> > mount.nfs: access denied by server while mounting pxe:/cc
> >
> > # uname -a
> > Linux hp 2.6.31-rc6-next-20090818 #61 SMP Thu Aug 20 14:46:10 CST 2009 x86_64 GNU/Linux
> >
> > However server log says OK:
> >
> > Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated mount request from 192.168.11.6:973 for /cc (/cc)
> > Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated unmount request from 192.168.11.6:974 for /cc (/cc)
> >
> > However-2: nfsroot can be mounted at boot time. Server kernel has always been 2.6.30.
> >
> > Any ideas?
> >
> > Thanks,
> > Fengguang
>
> Can you try again after enabling mount debugging on the NFS client?
>
> echo 512 > /proc/sys/sunrpc/nfs_debug
I used 1024 and found the mount failed here in nfs_walk_authlist():
dfprintk(MOUNT, "NFS: server does not support requested auth flavor\ n");
nfs_umount(request);
Thanks,
Fengguang
On Fri, 2009-08-21 at 09:27 +0800, Wu Fengguang wrote:
> On Thu, Aug 20, 2009 at 09:02:29PM +0800, Trond Myklebust wrote:
> > On Thu, 2009-08-20 at 15:13 +0800, Wu Fengguang wrote:
> > > Hi,
> > >
> > > After upgrading NFS client kernel to latest linux-next, NFS mount
> > > failed:
> > >
> > > # mount -t nfs pxe:/cc /cc
> > > mount.nfs: access denied by server while mounting pxe:/cc
> > >
> > > # uname -a
> > > Linux hp 2.6.31-rc6-next-20090818 #61 SMP Thu Aug 20 14:46:10 CST 2009 x86_64 GNU/Linux
> > >
> > > However server log says OK:
> > >
> > > Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated mount request from 192.168.11.6:973 for /cc (/cc)
> > > Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated unmount request from 192.168.11.6:974 for /cc (/cc)
> > >
> > > However-2: nfsroot can be mounted at boot time. Server kernel has always been 2.6.30.
> > >
> > > Any ideas?
> > >
> > > Thanks,
> > > Fengguang
> >
> > Can you try again after enabling mount debugging on the NFS client?
> >
> > echo 512 > /proc/sys/sunrpc/nfs_debug
>
> I used 1024 and found the mount failed here in nfs_walk_authlist():
>
> dfprintk(MOUNT, "NFS: server does not support requested auth flavor\ n");
> nfs_umount(request);
Thanks Fengguang!
Chuck, this looks like one of yours. Could it be that you are hitting
the same Linux knfsd bug that Tom Haynes saw with a Solaris client?
AFAICR, the problem was that existing nfs servers do not set a default
auth flavour, and so you just have to try with auth_sys and see if it
succeeds...
Cheers
Trond
On Aug 20, 2009, at 10:36 PM, Trond Myklebust wrote:
> On Fri, 2009-08-21 at 09:27 +0800, Wu Fengguang wrote:
>> On Thu, Aug 20, 2009 at 09:02:29PM +0800, Trond Myklebust wrote:
>>> On Thu, 2009-08-20 at 15:13 +0800, Wu Fengguang wrote:
>>>> Hi,
>>>>
>>>> After upgrading NFS client kernel to latest linux-next, NFS mount
>>>> failed:
>>>>
>>>> # mount -t nfs pxe:/cc /cc
>>>> mount.nfs: access denied by server while mounting pxe:/cc
>>>>
>>>> # uname -a
>>>> Linux hp 2.6.31-rc6-next-20090818 #61 SMP Thu Aug 20
>>>> 14:46:10 CST 2009 x86_64 GNU/Linux
>>>>
>>>> However server log says OK:
>>>>
>>>> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated mount
>>>> request from 192.168.11.6:973 for /cc (/cc)
>>>> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated unmount
>>>> request from 192.168.11.6:974 for /cc (/cc)
>>>>
>>>> However-2: nfsroot can be mounted at boot time. Server kernel has
>>>> always been 2.6.30.
>>>>
>>>> Any ideas?
>>>>
>>>> Thanks,
>>>> Fengguang
>>>
>>> Can you try again after enabling mount debugging on the NFS client?
>>>
>>> echo 512 > /proc/sys/sunrpc/nfs_debug
>>
>> I used 1024 and found the mount failed here in nfs_walk_authlist():
>>
>> dfprintk(MOUNT, "NFS: server does not support requested auth
>> flavor\ n");
>> nfs_umount(request);
>
> Thanks Fengguang!
>
> Chuck, this looks like one of yours. Could it be that you are hitting
> the same Linux knfsd bug that Tom Haynes saw with a Solaris client?
> AFAICR, the problem was that existing nfs servers do not set a default
> auth flavour, and so you just have to try with auth_sys and see if it
> succeeds...
With 1024 set, the mount client's XDR routines should have listed the
server's flavors, if any, in the system log. Should be something like
"NFS: received 0 auth flavors" if the server didn't return any. Can
you confirm that?
I'll try to post a fix later today.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Aug 21, 2009, at 3:07 PM, Thomas Haynes wrote:
> Sent from my iPhone
>
> On Aug 21, 2009, at 1:24 PM, "J. Bruce Fields" =20
> <[email protected]> wrote:
>
>> On Fri, Aug 21, 2009 at 02:16:08PM -0400, Chuck Lever wrote:
>>> I want to understand the server bug a little more. I glanced over =
=20
>>> RFC
>>> 2623 and didn't see anything specific.
>>>
>>> Is it the case that only Linux NFSD does this, or do other servers =
=20
>>> do
>>> it? In other words, is this a typical server response, and if so, =
=20
>>> is
>>> there a specific semantic attached to it?
>>>
>>> If no list is provided, should the client assume that only =20
>>> AUTH_NONE and
>>> AUTH_SYS are supported, or instead, perhaps that the client can =20
>>> try to
>>> use any flavor? In other words, if no list is provided, let the =20
>>> mount
>>> proceed no matter what was specified by sec=3D ?
>>
>> I think the safest behavior on the client would be something like:
>>
>> - If an explicit sec=3D is provided on the client, try that
>> flavor. Otherwise:
>> - If the server returned a nonempty list, pick something
>> off that list. Otherwise:
>> - Try auth_sys.
>>
>
> Which is pretty much what we do. The exception being that the =20
> default flavor is a setting - and probably always AUTH_SYS...
I'm still not sure what is the right thing to do here. Looking at =20
1813, it says:
DESCRIPTION
Procedure MNT maps a pathname on the server to a file
handle. The pathname is an ASCII string that describes a
directory on the server. If the call is successful
(MNT3_OK), the server returns an NFS version 3 protocol
-> file handle and a vector of RPC authentication flavors
-> that are supported with the client=92s use of the file
-> handle (or any file handles derived from it). The
authentication flavors are defined in Section 7.2 and
section 9 of [RFC1057].
IMPLEMENTATION
If mountres3.fhs_status is MNT3_OK, then
mountres3.mountinfo contains the file handle for the
-> directory and a list of acceptable authentication
-> flavors. This file handle may only be used in the NFS
version 3 protocol. This procedure also results in the
server adding a new entry to its mount list recording that
this client has mounted the directory. AUTH_UNIX
authentication or better is required.
This suggests pretty clearly that the client should treat the returned =
=20
auth flavor list as the list of flavors supported for this mount =20
point, not as a preference list to be used only if the client is =20
trying to guess what flavor to use. Is my reading incorrect? =20
Help! :-)
>
>
>
>> --b.
>>
>>>
>>> Thanks for any clarification.
>>>
>>> Begin forwarded message:
>>>
>>>> From: Trond Myklebust <[email protected]>
>>>> Date: August 20, 2009 10:36:11 PM GMT-04:00
>>>> To: Wu Fengguang <[email protected]>, "Mr. Charles Edward =20
>>>> Lever"
>>>> <[email protected]>
>>>> Cc: "[email protected]" <[email protected]>, LKML
>>>> <[email protected]>
>>>> Subject: Re: mount.nfs: access denied by server
>>>>
>>>> On Fri, 2009-08-21 at 09:27 +0800, Wu Fengguang wrote:
>>>>> On Thu, Aug 20, 2009 at 09:02:29PM +0800, Trond Myklebust wrote:
>>>>>> On Thu, 2009-08-20 at 15:13 +0800, Wu Fengguang wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> After upgrading NFS client kernel to latest linux-next, NFS =20
>>>>>>> mount
>>>>>>> failed:
>>>>>>>
>>>>>>> # mount -t nfs pxe:/cc /cc
>>>>>>> mount.nfs: access denied by server while mounting pxe:/cc
>>>>>>>
>>>>>>> # uname -a
>>>>>>> Linux hp 2.6.31-rc6-next-20090818 #61 SMP Thu Aug 20
>>>>>>> 14:46:10 CST 2009 x86_64 GNU/Linux
>>>>>>>
>>>>>>> However server log says OK:
>>>>>>>
>>>>>>> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated mount
>>>>>>> request from 192.168.11.6:973 for /cc (/cc)
>>>>>>> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated unmount
>>>>>>> request from 192.168.11.6:974 for /cc (/cc)
>>>>>>>
>>>>>>> However-2: nfsroot can be mounted at boot time. Server kernel =20
>>>>>>> has
>>>>>>> always been 2.6.30.
>>>>>>>
>>>>>>> Any ideas?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Fengguang
>>>>>>
>>>>>> Can you try again after enabling mount debugging on the NFS =20
>>>>>> client?
>>>>>>
>>>>>> echo 512 > /proc/sys/sunrpc/nfs_debug
>>>>>
>>>>> I used 1024 and found the mount failed here in =20
>>>>> nfs_walk_authlist():
>>>>>
>>>>> dfprintk(MOUNT, "NFS: server does not support requested auth
>>>>> flavor\ n");
>>>>> nfs_umount(request);
>>>>
>>>> Thanks Fengguang!
>>>>
>>>> Chuck, this looks like one of yours. Could it be that you are =20
>>>> hitting
>>>> the same Linux knfsd bug that Tom Haynes saw with a Solaris client=
?
>>>> AFAICR, the problem was that existing nfs servers do not set a =20
>>>> default
>>>> auth flavour, and so you just have to try with auth_sys and see =20
>>>> if it
>>>> succeeds...
>>>>
>>>> Cheers
>>>> Trond
>>>>
>>>
>>> --
>>> Chuck Lever
>>> chuck[dot]lever[at]oracle[dot]com
>>>
>>>
>>>
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Chuck Lever wrote:
> On Aug 21, 2009, at 3:07 PM, Thomas Haynes wrote:
>> Sent from my iPhone
>>
>> On Aug 21, 2009, at 1:24 PM, "J. Bruce Fields" <[email protected]>
>> wrote:
>>
>>> On Fri, Aug 21, 2009 at 02:16:08PM -0400, Chuck Lever wrote:
>>>> I want to understand the server bug a little more. I glanced over RFC
>>>> 2623 and didn't see anything specific.
>>>>
>>>> Is it the case that only Linux NFSD does this, or do other servers do
>>>> it? In other words, is this a typical server response, and if so, is
>>>> there a specific semantic attached to it?
>>>>
>>>> If no list is provided, should the client assume that only
>>>> AUTH_NONE and
>>>> AUTH_SYS are supported, or instead, perhaps that the client can try to
>>>> use any flavor? In other words, if no list is provided, let the mount
>>>> proceed no matter what was specified by sec= ?
>>>
>>> I think the safest behavior on the client would be something like:
>>>
>>> - If an explicit sec= is provided on the client, try that
>>> flavor. Otherwise:
>>> - If the server returned a nonempty list, pick something
>>> off that list. Otherwise:
>>> - Try auth_sys.
>>>
>>
>> Which is pretty much what we do. The exception being that the default
>> flavor is a setting - and probably always AUTH_SYS...
>
> I'm still not sure what is the right thing to do here. Looking at
> 1813, it says:
Sorry, I read this wrong, I thought the question was how should the
client handle
security flavors from the mount command.
With respect to the server returning a list, our implementation does
look at the list
of returned flavors and only tries to use one in the list. That was
actually the fix that
Bruce just applied - the linux server was returning an empty list and
the OpenSolaris
server would fail the mount.
Tom Haynes wrote:
> J. Bruce Fields wrote:
>>
>>> Right now, we use AUTH_SYS if the mount command didn't specify a
>>> flavor, but the flavor we're going to use is always checked against
>>> the server's returned list. You seem to be suggesting we should
>>> ignore the server's list entirely if sec= was specified...?
>>>
>>
>> Yes. I can't see what practical problems that would cause.
>>
Here, I am going to disagree. The client should compare the
flavor specified with the list of authentication flavors
returned by the server and if the specific flavor isn't on
the list, then the mount attempt should be failed.
>> Also, while I hope this is the last bug in the mountd's flavor list
>> return, it isn't the first--only recently did we even start using real
>> information from the export instead of just faking something up. So I
>> think it's safest to preserve the historical sec= behavior and give
>> users a way to override the negotiation, to cut down on bug reports of
>> mount failures on upgrade.
>>
>> --b.
>>
>
>
> While this Linux behavior occasionally bites the OpenSolaris behavior, it
> really means we just have to guard against the client ignoring what we
> sent.
> I.e., if I say we only support krb5 and you try with AUTH_SYS, I should
> reject it.
>
Which should happen correctly, right?
ps
> Historical inertia is hard to overcome, but perhaps you could gradually
> ease your way into using the real data. :->
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Chuck Lever wrote:
> On Aug 21, 2009, at 3:07 PM, Thomas Haynes wrote:
>> Sent from my iPhone
>>
>> On Aug 21, 2009, at 1:24 PM, "J. Bruce Fields" <[email protected]=
>
>> wrote:
>>
>>> On Fri, Aug 21, 2009 at 02:16:08PM -0400, Chuck Lever wrote:
>>>> I want to understand the server bug a little more. I glanced over=
RFC
>>>> 2623 and didn't see anything specific.
>>>>
>>>> Is it the case that only Linux NFSD does this, or do other servers=
do
>>>> it? In other words, is this a typical server response, and if so,=
is
>>>> there a specific semantic attached to it?
>>>>
>>>> If no list is provided, should the client assume that only AUTH_NO=
NE
>>>> and
>>>> AUTH_SYS are supported, or instead, perhaps that the client can tr=
y to
>>>> use any flavor? In other words, if no list is provided, let the m=
ount
>>>> proceed no matter what was specified by sec=3D ?
>>>
>>> I think the safest behavior on the client would be something like:
>>>
>>> - If an explicit sec=3D is provided on the client, try that
>>> flavor. Otherwise:
>>> - If the server returned a nonempty list, pick something
>>> off that list. Otherwise:
>>> - Try auth_sys.
>>>
>>
>> Which is pretty much what we do. The exception being that the defaul=
t
>> flavor is a setting - and probably always AUTH_SYS...
>=20
> I'm still not sure what is the right thing to do here. Looking at 18=
13,
> it says:
>=20
> DESCRIPTION
> Procedure MNT maps a pathname on the server to a file
> handle. The pathname is an ASCII string that describes a
> directory on the server. If the call is successful
> (MNT3_OK), the server returns an NFS version 3 protocol
> -> file handle and a vector of RPC authentication flavors
> -> that are supported with the client=92s use of the file
> -> handle (or any file handles derived from it). The
> authentication flavors are defined in Section 7.2 and
> section 9 of [RFC1057].
>=20
> IMPLEMENTATION
> If mountres3.fhs_status is MNT3_OK, then
> mountres3.mountinfo contains the file handle for the
> -> directory and a list of acceptable authentication
> -> flavors. This file handle may only be used in the NFS
> version 3 protocol. This procedure also results in the
> server adding a new entry to its mount list recording that
> this client has mounted the directory. AUTH_UNIX
> authentication or better is required.
>=20
> This suggests pretty clearly that the client should treat the returne=
d
> auth flavor list as the list of flavors supported for this mount poin=
t,
> not as a preference list to be used only if the client is trying to
> guess what flavor to use. Is my reading incorrect? Help! :-)
>=20
Your interpretation is the correct one, Chuck. The list
returned from the server is supposed to be the definitive
list of authentication flavors that the server will accept
from the client.
A server which returns no flavors is by definition, broken.
ps
>=20
>>
>>
>>
>>> --b.
>>>
>>>>
>>>> Thanks for any clarification.
>>>>
>>>> Begin forwarded message:
>>>>
>>>>> From: Trond Myklebust <[email protected]>
>>>>> Date: August 20, 2009 10:36:11 PM GMT-04:00
>>>>> To: Wu Fengguang <[email protected]>, "Mr. Charles Edward Le=
ver"
>>>>> <[email protected]>
>>>>> Cc: "[email protected]" <[email protected]>, LKML
>>>>> <[email protected]>
>>>>> Subject: Re: mount.nfs: access denied by server
>>>>>
>>>>> On Fri, 2009-08-21 at 09:27 +0800, Wu Fengguang wrote:
>>>>>> On Thu, Aug 20, 2009 at 09:02:29PM +0800, Trond Myklebust wrote:
>>>>>>> On Thu, 2009-08-20 at 15:13 +0800, Wu Fengguang wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> After upgrading NFS client kernel to latest linux-next, NFS mo=
unt
>>>>>>>> failed:
>>>>>>>>
>>>>>>>> # mount -t nfs pxe:/cc /cc
>>>>>>>> mount.nfs: access denied by server while mounting pxe:/cc
>>>>>>>>
>>>>>>>> # uname -a
>>>>>>>> Linux hp 2.6.31-rc6-next-20090818 #61 SMP Thu Aug 20
>>>>>>>> 14:46:10 CST 2009 x86_64 GNU/Linux
>>>>>>>>
>>>>>>>> However server log says OK:
>>>>>>>>
>>>>>>>> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated mount
>>>>>>>> request from 192.168.11.6:973 for /cc (/cc)
>>>>>>>> Aug 20 15:02:09 wu-t61 mountd[4599]: authenticated unmoun=
t
>>>>>>>> request from 192.168.11.6:974 for /cc (/cc)
>>>>>>>>
>>>>>>>> However-2: nfsroot can be mounted at boot time. Server kernel =
has
>>>>>>>> always been 2.6.30.
>>>>>>>>
>>>>>>>> Any ideas?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Fengguang
>>>>>>>
>>>>>>> Can you try again after enabling mount debugging on the NFS cli=
ent?
>>>>>>>
>>>>>>> echo 512 > /proc/sys/sunrpc/nfs_debug
>>>>>>
>>>>>> I used 1024 and found the mount failed here in nfs_walk_authlist=
():
>>>>>>
>>>>>> dfprintk(MOUNT, "NFS: server does not support requested aut=
h
>>>>>> flavor\ n");
>>>>>> nfs_umount(request);
>>>>>
>>>>> Thanks Fengguang!
>>>>>
>>>>> Chuck, this looks like one of yours. Could it be that you are hit=
ting
>>>>> the same Linux knfsd bug that Tom Haynes saw with a Solaris clien=
t?
>>>>> AFAICR, the problem was that existing nfs servers do not set a de=
fault
>>>>> auth flavour, and so you just have to try with auth_sys and see i=
f it
>>>>> succeeds...
>>>>>
>>>>> Cheers
>>>>> Trond
>>>>>
>>>>
>>>> --=20
>>>> Chuck Lever
>>>> chuck[dot]lever[at]oracle[dot]com
>>>>
>>>>
>>>>
>=20
> --=20
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
>=20
>=20
>=20
> --=20
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" =
in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Aug 21, 2009 at 04:39:53PM -0400, Peter Staubach wrote:
> Tom Haynes wrote:
> > J. Bruce Fields wrote:
> >>
> >>> Right now, we use AUTH_SYS if the mount command didn't specify a
> >>> flavor, but the flavor we're going to use is always checked against
> >>> the server's returned list. You seem to be suggesting we should
> >>> ignore the server's list entirely if sec= was specified...?
> >>>
> >>
> >> Yes. I can't see what practical problems that would cause.
> >>
>
> Here, I am going to disagree. The client should compare the
> flavor specified with the list of authentication flavors
> returned by the server and if the specific flavor isn't on
> the list, then the mount attempt should be failed.
Why? What can go wrong if you just try the flavor?
Ugh: to answer my own question: ok, if the server allows auth_sys for
the NFS operations required on mount (as suggested by the security
considerations rfc), then that would mean mount would always succeed,
and nobody would see the error until the first operation on the
filesystem. It would be better to return the error on mount.
So, I think I have to give in and agree with you. People will just have
to upgrade their mountd's....
--b.
On Fri, 2009-08-21 at 16:59 -0400, J. Bruce Fields wrote:
> On Fri, Aug 21, 2009 at 04:39:53PM -0400, Peter Staubach wrote:
> > Tom Haynes wrote:
> > > J. Bruce Fields wrote:
> > >>
> > >>> Right now, we use AUTH_SYS if the mount command didn't specify a
> > >>> flavor, but the flavor we're going to use is always checked against
> > >>> the server's returned list. You seem to be suggesting we should
> > >>> ignore the server's list entirely if sec= was specified...?
> > >>>
> > >>
> > >> Yes. I can't see what practical problems that would cause.
> > >>
> >
> > Here, I am going to disagree. The client should compare the
> > flavor specified with the list of authentication flavors
> > returned by the server and if the specific flavor isn't on
> > the list, then the mount attempt should be failed.
>
> Why? What can go wrong if you just try the flavor?
>
> Ugh: to answer my own question: ok, if the server allows auth_sys for
> the NFS operations required on mount (as suggested by the security
> considerations rfc), then that would mean mount would always succeed,
> and nobody would see the error until the first operation on the
> filesystem. It would be better to return the error on mount.
>
> So, I think I have to give in and agree with you. People will just have
> to upgrade their mountd's....
People have lived with the current situation for years without any
trouble. It was only the addition of the security negotiation that
exposed the mountd bug.
My position is that if mountd returns a clearly invalid (i.e. empty)
list, then that's not a reason to interrupt the mount. We should just
fall back to what we did prior to including security negotiation.
The alternative is simply not to merge security negotiation until people
have had ample time to upgrade mountd (i.e. 2-3 years from now).
Trond
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
http://www.netapp.com
On Fri, Aug 21, 2009 at 05:08:12PM -0400, Trond Myklebust wrote:
> On Fri, 2009-08-21 at 16:59 -0400, J. Bruce Fields wrote:
> > On Fri, Aug 21, 2009 at 04:39:53PM -0400, Peter Staubach wrote:
> > > Tom Haynes wrote:
> > > > J. Bruce Fields wrote:
> > > >>
> > > >>> Right now, we use AUTH_SYS if the mount command didn't specify a
> > > >>> flavor, but the flavor we're going to use is always checked against
> > > >>> the server's returned list. You seem to be suggesting we should
> > > >>> ignore the server's list entirely if sec= was specified...?
> > > >>>
> > > >>
> > > >> Yes. I can't see what practical problems that would cause.
> > > >>
> > >
> > > Here, I am going to disagree. The client should compare the
> > > flavor specified with the list of authentication flavors
> > > returned by the server and if the specific flavor isn't on
> > > the list, then the mount attempt should be failed.
> >
> > Why? What can go wrong if you just try the flavor?
> >
> > Ugh: to answer my own question: ok, if the server allows auth_sys for
> > the NFS operations required on mount (as suggested by the security
> > considerations rfc), then that would mean mount would always succeed,
> > and nobody would see the error until the first operation on the
> > filesystem. It would be better to return the error on mount.
> >
> > So, I think I have to give in and agree with you. People will just have
> > to upgrade their mountd's....
>
> People have lived with the current situation for years without any
> trouble. It was only the addition of the security negotiation that
> exposed the mountd bug.
>
> My position is that if mountd returns a clearly invalid (i.e. empty)
> list, then that's not a reason to interrupt the mount. We should just
> fall back to what we did prior to including security negotiation.
Yeah, so instead of taking my original pseudocode (which just allowed
sec= to always override even a non-empty list from the server), more
reasonable might be just to treat the empty-list case specially.
For example, maybe:
- If the returned list is nonempty:
- Do whatever the latest code does: probably return an
error if there's a clash with the sec= mount option,
otherwise attempt the first on the list.
- Otherwise, if the returned list is empty:
- Use any sec= option if provide, or auth_sys if not.
--b.
Trond Myklebust wrote:
> What is the definition of "recently introduced" here? What is the exact
> commit that introduced the bug in nfs-utils?
>
> The problem is that we're changing the client, and if it turns out that
> every damned Linux server that was installed out there in the past 2
> years displays this bug, then Linus will call 'regression' on us and
> revert the code.
>
> Trond
>
>
The Linux client and server have well defined behaviors in that they always
will default to AUTH_SYS.
It may not be standard compliant, but everyone and their dog knows it.
Newer servers will not hand out empty lists, and why cause yourself grief
by not handling older servers?