Found regression introduced by this patch
311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"
# cd /bin
-sh: cd: /bin: Not a directory
root@192:/# pwd
/
root@192:/# ls /bin
ls: /bin
root@192:/# ls -l /bin/sh
lrwxrwxrwx 1 root root 9 Jun 16 13:59 /bin/sh -> /bin/bash
the mount options of the client, an embedded system, mounting nfsroot:
nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
vers: 3.16.0-rc1
the server exports:
/exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux
reverting above patch, dirlisting with 'ls /' works again
/Andreas
2014-07-16 1:54 GMT+02:00 Trond Myklebust <[email protected]>:
> On Sun, Jul 13, 2014 at 11:44 PM, Andreas Fenkart <[email protected]> wrote:
>> 2014-07-10 19:51 GMT+02:00 J. Bruce Fields <[email protected]>:
>>> On Mon, Jun 23, 2014 at 12:07:22PM +0200, Andreas Fenkart wrote:
>>>> Found regression introduced by this patch
>>>>
>>>> 311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"
>>>>
>>>> # cd /bin
>>>> -sh: cd: /bin: Not a directory
>>>> root@192:/# pwd
>>>> /
>>>> root@192:/# ls /bin
>>>> ls: /bin
>>>> root@192:/# ls -l /bin/sh
>>>> lrwxrwxrwx 1 root root 9 Jun 16 13:59 /bin/sh -> /bin/bash
>>>>
>>>> the mount options of the client, an embedded system, mounting nfsroot:
>>>> nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
>>>> vers: 3.16.0-rc1
>>>>
>>>> the server exports:
>>>> /exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
>>>> vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux
>>>>
>>>> reverting above patch, dir listing with 'ls /' works again
>>>
>>> Is this still happening?
>>
>> yes it is, I just reverted the aforementioned commit.
>
> Can you reapply, and then do an "ls /" while recording the traffic to
> the NFS server using wireshark, and then send us the resulting dump?
http://www.pcapr.net/view/afenkart/2014/6/3/1/nfs-client.dump.html
you have to create a login... simple. Or did you mean attach to email?
> If the client is on an embedded system which can't run wireshark, then
> it is fine to record it on the server, but please use something like
>
> 'tshark -w /var/tmp/dump.out host <client-ip>'
captured on embedded system:
tcpdump -w /tmp/nfs-client.dump
>
> Cheers
> Trond
thanks, sincerely
Andreas
On Mon, Jun 23, 2014 at 12:07:22PM +0200, Andreas Fenkart wrote:
> Found regression introduced by this patch
>
> 311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"
>
> # cd /bin
> -sh: cd: /bin: Not a directory
> root@192:/# pwd
> /
> root@192:/# ls /bin
> ls: /bin
> root@192:/# ls -l /bin/sh
> lrwxrwxrwx 1 root root 9 Jun 16 13:59 /bin/sh -> /bin/bash
>
> the mount options of the client, an embedded system, mounting nfsroot:
> nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
> vers: 3.16.0-rc1
>
> the server exports:
> /exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
> vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux
>
> reverting above patch, dirlisting with 'ls /' works again
Is this still happening?
--b.
On Sun, Jul 13, 2014 at 11:44 PM, Andreas Fenkart <[email protected]> wrote:
> 2014-07-10 19:51 GMT+02:00 J. Bruce Fields <[email protected]>:
>> On Mon, Jun 23, 2014 at 12:07:22PM +0200, Andreas Fenkart wrote:
>>> Found regression introduced by this patch
>>>
>>> 311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"
>>>
>>> # cd /bin
>>> -sh: cd: /bin: Not a directory
>>> root@192:/# pwd
>>> /
>>> root@192:/# ls /bin
>>> ls: /bin
>>> root@192:/# ls -l /bin/sh
>>> lrwxrwxrwx 1 root root 9 Jun 16 13:59 /bin/sh -> /bin/bash
>>>
>>> the mount options of the client, an embedded system, mounting nfsroot:
>>> nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
>>> vers: 3.16.0-rc1
>>>
>>> the server exports:
>>> /exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
>>> vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux
>>>
>>> reverting above patch, dir listing with 'ls /' works again
>>
>> Is this still happening?
>
> yes it is, I just reverted the aforementioned commit.
Can you reapply, and then do an "ls /" while recording the traffic to
the NFS server using wireshark, and then send us the resulting dump?
If the client is on an embedded system which can't run wireshark, then
it is fine to record it on the server, but please use something like
'tshark -w /var/tmp/dump.out host <client-ip>'
Cheers
Trond
--
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
On Wed, Jul 16, 2014 at 11:45 AM, Andreas Fenkart <[email protected]> wrote:
> 2014-07-16 16:47 GMT+02:00 Trond Myklebust <[email protected]>:
>> On Wed, Jul 16, 2014 at 1:21 AM, Andreas Fenkart <[email protected]> wrote:
>
>> That unfortunately only shows a few READ requests, probably because
>> the readdirs etc are cached. Could you try again with a directory that
>> is not in cache and/or try "echo 3 >/proc/sys/vm/drop_caches" before
>> retrying the 'ls /'?
>
> # tcpdump -s 0 -w /tmp/nfs-client_v4.dump
> # echo 3 >/proc/sys/vm/drop_caches
> # ls -l /
>
> file is logged to tmpfs, then copied
> http://www.pcapr.net/view/afenkart/2014/6/3/8/nfs-client_v4.dump.html
>
>> Note: I am a little surprised that there are no GETATTRs in that dump.
>> Even if the readdir is cached, the client is supposed to revalidate
>> the directory.
>
> should be there now
Still not seeing the readdir, but the getattrs are there and all looks OK.
Question: the server is knfsd from a Linux 3.14.4 kernel, is that
correct? So it does have commit 83b19121522 (nfsd: revert v2 half of
"nfsd: don't return high mode bits")?
--
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
On Wed, Jul 16, 2014 at 1:21 AM, Andreas Fenkart <[email protected]> wrote:
> 2014-07-16 1:54 GMT+02:00 Trond Myklebust <[email protected]>:
>> On Sun, Jul 13, 2014 at 11:44 PM, Andreas Fenkart <[email protected]> wrote:
>>> 2014-07-10 19:51 GMT+02:00 J. Bruce Fields <[email protected]>:
>>>> On Mon, Jun 23, 2014 at 12:07:22PM +0200, Andreas Fenkart wrote:
>>>>> Found regression introduced by this patch
>>>>>
>>>>> 311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"
>>>>>
>>>>> # cd /bin
>>>>> -sh: cd: /bin: Not a directory
>>>>> root@192:/# pwd
>>>>> /
>>>>> root@192:/# ls /bin
>>>>> ls: /bin
>>>>> root@192:/# ls -l /bin/sh
>>>>> lrwxrwxrwx 1 root root 9 Jun 16 13:59 /bin/sh -> /bin/bash
>>>>>
>>>>> the mount options of the client, an embedded system, mounting nfsroot:
>>>>> nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
>>>>> vers: 3.16.0-rc1
>>>>>
>>>>> the server exports:
>>>>> /exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
>>>>> vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux
>>>>>
>>>>> reverting above patch, dir listing with 'ls /' works again
>>>>
>>>> Is this still happening?
>>>
>>> yes it is, I just reverted the aforementioned commit.
>>
>> Can you reapply, and then do an "ls /" while recording the traffic to
>> the NFS server using wireshark, and then send us the resulting dump?
>
> http://www.pcapr.net/view/afenkart/2014/6/3/1/nfs-client.dump.html
> you have to create a login... simple. Or did you mean attach to email?
>
>> If the client is on an embedded system which can't run wireshark, then
>> it is fine to record it on the server, but please use something like
>>
>> 'tshark -w /var/tmp/dump.out host <client-ip>'
>
> captured on embedded system:
> tcpdump -w /tmp/nfs-client.dump
>
>>
>> Cheers
>> Trond
>
> thanks, sincerely
> Andreas
That unfortunately only shows a few READ requests, probably because
the readdirs etc are cached. Could you try again with a directory that
is not in cache and/or try "echo 3 >/proc/sys/vm/drop_caches" before
retrying the 'ls /'?
Note: I am a little surprised that there are no GETATTRs in that dump.
Even if the readdir is cached, the client is supposed to revalidate
the directory.
--
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
2014-07-16 16:47 GMT+02:00 Trond Myklebust <[email protected]>:
> On Wed, Jul 16, 2014 at 1:21 AM, Andreas Fenkart <[email protected]> wrote:
> That unfortunately only shows a few READ requests, probably because
> the readdirs etc are cached. Could you try again with a directory that
> is not in cache and/or try "echo 3 >/proc/sys/vm/drop_caches" before
> retrying the 'ls /'?
# tcpdump -s 0 -w /tmp/nfs-client_v4.dump
# echo 3 >/proc/sys/vm/drop_caches
# ls -l /
file is logged to tmpfs, then copied
http://www.pcapr.net/view/afenkart/2014/6/3/8/nfs-client_v4.dump.html
> Note: I am a little surprised that there are no GETATTRs in that dump.
> Even if the readdir is cached, the client is supposed to revalidate
> the directory.
should be there now
Andreas
2014-07-10 19:51 GMT+02:00 J. Bruce Fields <[email protected]>:
> On Mon, Jun 23, 2014 at 12:07:22PM +0200, Andreas Fenkart wrote:
>> Found regression introduced by this patch
>>
>> 311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"
>>
>> # cd /bin
>> -sh: cd: /bin: Not a directory
>> root@192:/# pwd
>> /
>> root@192:/# ls /bin
>> ls: /bin
>> root@192:/# ls -l /bin/sh
>> lrwxrwxrwx 1 root root 9 Jun 16 13:59 /bin/sh -> /bin/bash
>>
>> the mount options of the client, an embedded system, mounting nfsroot:
>> nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
>> vers: 3.16.0-rc1
>>
>> the server exports:
>> /exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
>> vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux
>>
>> reverting above patch, dir listing with 'ls /' works again
>
> Is this still happening?
yes it is, I just reverted the aforementioned commit.
Hi Bruce,
http://snapshot.debian.org/package/linux/3.14.4-1/#linux-image-3.14-1-amd64_3.14.4-1
<- version when reporting bug
http://snapshot.debian.org/package/linux/3.14.15-2/#linux-image-3.14-2-amd64_3.14.15-2
<- current version
even when downgrading to the previous version, I can't reproduce the
bug anymore...
just realized that I didn't mention the tip of my embedded system
kernel, bad me, but it was some 3.16 rc,
/Andreas
2014-08-27 15:26 GMT+02:00 J. Bruce Fields <[email protected]>:
> On Wed, Aug 27, 2014 at 10:30:34AM +0200, Andreas Fenkart wrote:
>> Hi
>>
>> 2014-08-26 21:50 GMT+02:00 J. Bruce Fields <[email protected]>:
>> >>
>> >> Still not seeing the readdir, but the getattrs are there and all looks OK.
>> >> Question: the server is knfsd from a Linux 3.14.4 kernel, is that
>> >> correct? So it does have commit 83b19121522 (nfsd: revert v2 half of
>> >> "nfsd: don't return high mode bits")?
>> >
>> > Yeah, you can even see the high bits set (correctly) in the stat
>> > returns, so he has that patch.
>> >
>> > Andreas, are you still seeing this problem?
>>
>> No it's gone, but I upgraded my (server-)system meanwhile (debian unstable)
>> I also tried downgrading my kernel to the version I had back then, but
>> the problem doesn't show up anymore.
>
> Well, maybe there was a server bug in the old server that this change
> just happened to expose.
>
> What was the server upgrade, out of curiosity? (From what kernel
> version, to what kernel version?)
>
> --b.
Hi
2014-08-26 21:50 GMT+02:00 J. Bruce Fields <[email protected]>:
>>
>> Still not seeing the readdir, but the getattrs are there and all looks OK.
>> Question: the server is knfsd from a Linux 3.14.4 kernel, is that
>> correct? So it does have commit 83b19121522 (nfsd: revert v2 half of
>> "nfsd: don't return high mode bits")?
>
> Yeah, you can even see the high bits set (correctly) in the stat
> returns, so he has that patch.
>
> Andreas, are you still seeing this problem?
No it's gone, but I upgraded my (server-)system meanwhile (debian unstable)
I also tried downgrading my kernel to the version I had back then, but
the problem doesn't show up anymore.
sorry the noise
/Andreas
$ cat /proc/version
Linux version 3.14-1-amd64 ([email protected]) (gcc
version 4.8.2 (Debian 4.8.2-21) ) #1 SMP Debian 3.14.4-1 (2014-05-13)
http://snapshot.debian.org/archive/debian/20140514T103724Z/pool/main/l/linux/linux-image-3.14-1-amd64_3.14.4-1_i386.deb
On Wed, Aug 27, 2014 at 10:30:34AM +0200, Andreas Fenkart wrote:
> Hi
>
> 2014-08-26 21:50 GMT+02:00 J. Bruce Fields <[email protected]>:
> >>
> >> Still not seeing the readdir, but the getattrs are there and all looks OK.
> >> Question: the server is knfsd from a Linux 3.14.4 kernel, is that
> >> correct? So it does have commit 83b19121522 (nfsd: revert v2 half of
> >> "nfsd: don't return high mode bits")?
> >
> > Yeah, you can even see the high bits set (correctly) in the stat
> > returns, so he has that patch.
> >
> > Andreas, are you still seeing this problem?
>
> No it's gone, but I upgraded my (server-)system meanwhile (debian unstable)
> I also tried downgrading my kernel to the version I had back then, but
> the problem doesn't show up anymore.
Well, maybe there was a server bug in the old server that this change
just happened to expose.
What was the server upgrade, out of curiosity? (From what kernel
version, to what kernel version?)
--b.
On Fri, Jul 25, 2014 at 06:12:40PM -0400, Trond Myklebust wrote:
> On Wed, Jul 16, 2014 at 11:45 AM, Andreas Fenkart <[email protected]> wrote:
> > 2014-07-16 16:47 GMT+02:00 Trond Myklebust <[email protected]>:
> >> On Wed, Jul 16, 2014 at 1:21 AM, Andreas Fenkart <[email protected]> wrote:
> >
> >> That unfortunately only shows a few READ requests, probably because
> >> the readdirs etc are cached. Could you try again with a directory that
> >> is not in cache and/or try "echo 3 >/proc/sys/vm/drop_caches" before
> >> retrying the 'ls /'?
> >
> > # tcpdump -s 0 -w /tmp/nfs-client_v4.dump
> > # echo 3 >/proc/sys/vm/drop_caches
> > # ls -l /
> >
> > file is logged to tmpfs, then copied
> > http://www.pcapr.net/view/afenkart/2014/6/3/8/nfs-client_v4.dump.html
> >
> >> Note: I am a little surprised that there are no GETATTRs in that dump.
> >> Even if the readdir is cached, the client is supposed to revalidate
> >> the directory.
> >
> > should be there now
>
> Still not seeing the readdir, but the getattrs are there and all looks OK.
> Question: the server is knfsd from a Linux 3.14.4 kernel, is that
> correct? So it does have commit 83b19121522 (nfsd: revert v2 half of
> "nfsd: don't return high mode bits")?
Yeah, you can even see the high bits set (correctly) in the stat
returns, so he has that patch.
Andreas, are you still seeing this problem?
--b.