If I export a directory that has an inode number > 2^32, the
client-side mount succeeds, but any subsequent action gives a stale
error.
$ sudo mount -t nfs svin:/z/svin/aowitest/foo /mnt
$ ls /mnt
ls: cannot access /mnt: Stale file handle
The underlying filesystem is XFS. The exported directory is not the
root of that filesystem (hence the large inode number).
Server-side debug captures this:
nfsd_dispatch: vers 4 proc 1
nfsv4 compound op #1/3: 22 (OP_PUTFH)
svc: server ffff897ecc64e000, pool 0, transport ffff897eca7fd000, inuse=3
nfsd: fh_verify(28: 00070001 400000c0 00000001 0f52f60a 63438e31 ab541493)
nfsv4 compound op ffff897ecb6e9080 opcnt 3 #1: 22: status 70
nfsv4 compound returned 70
I have worked around this by rmdir/mkdir'ing until I got an inode < 2^32.
This is only an issue with the inode of the / of the NFS exported path.
Have I overlooked a flag somewhere? Is this just impossible to fix
without breaking the world?
--
Anders Ossowicki
> On Nov 14, 2017, at 11:49 AM, Anders Ossowicki <[email protected]> wrote:
>
> If I export a directory that has an inode number > 2^32, the
> client-side mount succeeds, but any subsequent action gives a stale
> error.
>
> $ sudo mount -t nfs svin:/z/svin/aowitest/foo /mnt
> $ ls /mnt
> ls: cannot access /mnt: Stale file handle
>
> The underlying filesystem is XFS. The exported directory is not the
> root of that filesystem (hence the large inode number).
>
> Server-side debug captures this:
>
> nfsd_dispatch: vers 4 proc 1
> nfsv4 compound op #1/3: 22 (OP_PUTFH)
> svc: server ffff897ecc64e000, pool 0, transport ffff897eca7fd000, inuse=3
> nfsd: fh_verify(28: 00070001 400000c0 00000001 0f52f60a 63438e31 ab541493)
> nfsv4 compound op ffff897ecb6e9080 opcnt 3 #1: 22: status 70
> nfsv4 compound returned 70
>
> I have worked around this by rmdir/mkdir'ing until I got an inode < 2^32.
>
> This is only an issue with the inode of the / of the NFS exported path.
>
> Have I overlooked a flag somewhere? Is this just impossible to fix
> without breaking the world?
A fresh XFS file system can be mounted with the "inode32"
mount option (on the server) to prevent the creation of
inodes with numbers that cannot be represented in 32 bits.
But since you already have such inodes, NFS clients will
have to convert the numbers on the fly. You can use a boot
parameter on your clients:
nfs.enable_ino64=0
According to comments in fs/nfs/inode.c .
--
Chuck Lever
On 14 November 2017 at 18:12, Chuck Lever <[email protected]> wrote:
> A fresh XFS file system can be mounted with the "inode32"
> mount option (on the server) to prevent the creation of
> inodes with numbers that cannot be represented in 32 bits.
>
> But since you already have such inodes, NFS clients will
> have to convert the numbers on the fly. You can use a boot
> parameter on your clients:
>
> nfs.enable_ino64=0
>
> According to comments in fs/nfs/inode.c .
Thanks for the suggestion.
I looked at nfs.enable_ino64 while tracking down this issue, but it
looked like a client-side option to me, and the error happens in the
nfsd code. I'll give a shot tomorrow.
There's no problem accessing files or directories with inodes > 2^32 in
our setup, we've been doing that for years. It's only when the root
directory of the export has a high inode that we trigger this.
--
Anders Ossowicki
On 14 November 2017 at 19:11, Anders Ossowicki <[email protected]> wrote:
> On 14 November 2017 at 18:12, Chuck Lever <[email protected]> wrote:
>> But since you already have such inodes, NFS clients will
>> have to convert the numbers on the fly. You can use a boot
>> parameter on your clients:
>>
>> nfs.enable_ino64=0
>>
>> According to comments in fs/nfs/inode.c .
>
> Thanks for the suggestion.
>
> I looked at nfs.enable_ino64 while tracking down this issue, but it
> looked like a client-side option to me, and the error happens in the
> nfsd code. I'll give a shot tomorrow.
No dice, unfortunately:
aowi@otto ~ $ dmesg|grep 'Kernel command line'
[ 0.000000] Kernel command line:
BOOT_IMAGE=/boot/vmlinuz-3.19.0-77-generic
root=UUID=601cb526-8e11-4565-a5ed-0be619db7a5e ro quiet splash
nfs.enable_ino64=0 vt.handoff=7
aowi@otto ~ $ sudo mount -t nfs svin:/z/svin/aowitest/foo /mnt
aowi@otto ~ $ ls /mnt
ls: cannot open directory /mnt: Stale file handle
--
Anders Ossowicki
> On Nov 15, 2017, at 3:14 AM, Anders Ossowicki <[email protected]> wrote:
>
> On 14 November 2017 at 19:11, Anders Ossowicki <[email protected]> wrote:
>> On 14 November 2017 at 18:12, Chuck Lever <[email protected]> wrote:
>>> But since you already have such inodes, NFS clients will
>>> have to convert the numbers on the fly. You can use a boot
>>> parameter on your clients:
>>>
>>> nfs.enable_ino64=0
>>>
>>> According to comments in fs/nfs/inode.c .
>>
>> Thanks for the suggestion.
>>
>> I looked at nfs.enable_ino64 while tracking down this issue, but it
>> looked like a client-side option to me, and the error happens in the
>> nfsd code. I'll give a shot tomorrow.
>
> No dice, unfortunately:
>
> aowi@otto ~ $ dmesg|grep 'Kernel command line'
> [ 0.000000] Kernel command line:
> BOOT_IMAGE=/boot/vmlinuz-3.19.0-77-generic
> root=UUID=601cb526-8e11-4565-a5ed-0be619db7a5e ro quiet splash
> nfs.enable_ino64=0 vt.handoff=7
> aowi@otto ~ $ sudo mount -t nfs svin:/z/svin/aowitest/foo /mnt
> aowi@otto ~ $ ls /mnt
> ls: cannot open directory /mnt: Stale file handle
Does this happen with a recent kernel on the client?
And, it would help if you can capture and post a raw tcpdump
of this interaction so we can see all the details of the
conversation. Start the tcpdump before "sudo mount", and
save the raw bytes with "-w /tmp/sniffer.pcap". gzip and
post it here.
What happens if you mount svin:/ and walk down the server's
namespace into /z/svin/aowitest/foo ?
--
Chuck Lever
On Wed, Nov 15, 2017 at 10:27:49AM -0500, Chuck Lever wrote:
>
> > On Nov 15, 2017, at 3:14 AM, Anders Ossowicki <[email protected]> wrote:
> >
> > On 14 November 2017 at 19:11, Anders Ossowicki <[email protected]> wrote:
> >> On 14 November 2017 at 18:12, Chuck Lever <[email protected]> wrote:
> >>> But since you already have such inodes, NFS clients will
> >>> have to convert the numbers on the fly. You can use a boot
> >>> parameter on your clients:
> >>>
> >>> nfs.enable_ino64=0
> >>>
> >>> According to comments in fs/nfs/inode.c .
> >>
> >> Thanks for the suggestion.
> >>
> >> I looked at nfs.enable_ino64 while tracking down this issue, but it
> >> looked like a client-side option to me, and the error happens in the
> >> nfsd code. I'll give a shot tomorrow.
> >
> > No dice, unfortunately:
> >
> > aowi@otto ~ $ dmesg|grep 'Kernel command line'
> > [ 0.000000] Kernel command line:
> > BOOT_IMAGE=/boot/vmlinuz-3.19.0-77-generic
> > root=UUID=601cb526-8e11-4565-a5ed-0be619db7a5e ro quiet splash
> > nfs.enable_ino64=0 vt.handoff=7
> > aowi@otto ~ $ sudo mount -t nfs svin:/z/svin/aowitest/foo /mnt
> > aowi@otto ~ $ ls /mnt
> > ls: cannot open directory /mnt: Stale file handle
>
> Does this happen with a recent kernel on the client?
>
> And, it would help if you can capture and post a raw tcpdump
> of this interaction so we can see all the details of the
> conversation. Start the tcpdump before "sudo mount", and
> save the raw bytes with "-w /tmp/sniffer.pcap". gzip and
> post it here.
Sounds like the server's returning STALE on PUTFH of a filehandle it
presumably just returned.... So that does seem like a pretty clear
bug on the server side, either in knfsd or xfs or exportfs. Still, it
might be good to see the traffic, yes.
--b.
>
> What happens if you mount svin:/ and walk down the server's
> namespace into /z/svin/aowitest/foo ?
>
>
> --
> Chuck Lever
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html