Hi list,
we're experiencing a serious NFSv4 client caching issue when a
client reads a file which was deleted and recreated by another client.
Steps to reproduce the problem:
root@client1:~# mount -t nfs -o rw,vers=4,sec=sys,hard,intr 129.70.150.53:/vol/testvol5 /mnt
root@client2:~# mount -t nfs -o rw,vers=4,sec=sys,hard,intr 129.70.150.53:/vol/testvol5 /mnt
user@client1:~$ echo foo > /mnt/bar
user@client2:~$ cat /mnt/bar
foo
user@client1:~$ rm /mnt/bar ; echo quux > /mnt/bar
user@client2:~$ cat /mnt/bar
cat: /mnt/bar: Input/output error
Even after waiting some hours the i/o error is still there.
'ls' on target directory fixes the problem immediately:
user@client2:~$ ls /mnt
bar
user@client2:~$ cat /mnt/bar
quux
Dropping the inode cache also works:
root@client2:~# sync
root@client2:~# echo 2 > /proc/sys/vm/drop_caches
We tried different mount options (lookupcache=none, noac, ...),
but nothing helped.
NFS server: EMC ISILON NAS cluster
Clients: Ubuntu 14.04 LTS
Kernels tried: Ubuntu linux-image-3.16.0-40-generic, vanilla 3.18.1,
vanilla 4.4.0
Also affected:
- Debian Jessie (kernel 3.16)
- Ubuntu 16.04 beta/alpha (Ubuntu kernel 4.4.0-12-generic)
No affected:
- CentOS 6 (kernel 2.6.32)
- Debian Wheezy (kernel 3.2)
This problem does only occur when using NFS protocol version 4,
not with vers=3.
I dumped the network traffic between NFS server and client2
(I attached the dump).
Does anybody know what's happening and how to fix this issue?
Cheers,
Sascha
On Tue, Mar 15, 2016 at 10:21:04AM +0100, Sascha Frey wrote:
> Hi list,
>
> we're experiencing a serious NFSv4 client caching issue when a
> client reads a file which was deleted and recreated by another client.
>
> Steps to reproduce the problem:
>
> root@client1:~# mount -t nfs -o rw,vers=4,sec=sys,hard,intr 129.70.150.53:/vol/testvol5 /mnt
> root@client2:~# mount -t nfs -o rw,vers=4,sec=sys,hard,intr 129.70.150.53:/vol/testvol5 /mnt
>
> user@client1:~$ echo foo > /mnt/bar