On Mar 23, 2011, at 10:30 AM, Belisko Marek wrote:
> On Wed, Mar 23, 2011 at 3:06 PM, Chuck Lever <[email protected]> wrote:
>>
>> On Mar 23, 2011, at 9:46 AM, Belisko Marek wrote:
>>
>>> On Wed, Mar 23, 2011 at 2:44 PM, Chuck Lever <[email protected]> wrote:
>>>>
>>>> What does "rpcinfo -p 10.146.1.21" output look like? Is NFS over UDP enabled on your server?
>>> program vers proto port
>>> 100000 2 tcp 111 portmapper
>>> 100000 2 udp 111 portmapper
>>> 100024 1 udp 40517 status
>>> 100024 1 tcp 51397 status
>>> 100021 1 udp 44516 nlockmgr
>>> 100021 3 udp 44516 nlockmgr
>>> 100021 4 udp 44516 nlockmgr
>>> 100021 1 tcp 55152 nlockmgr
>>> 100021 3 tcp 55152 nlockmgr
>>> 100021 4 tcp 55152 nlockmgr
>>> 100003 2 tcp 2049 nfs
>>> 100003 3 tcp 2049 nfs
>>> 100003 4 tcp 2049 nfs
>>> 100227 2 tcp 2049
>>> 100227 3 tcp 2049
>>> 100003 2 udp 2049 nfs
>>> 100003 3 udp 2049 nfs
>>> 100003 4 udp 2049 nfs
The server is advertising NFS over UDP. Why can't a client access your server via UDP? What happens if you perform a normal post-boot mount of this file system via UDP?
>>> 100227 2 udp 2049
>>> 100227 3 udp 2049
>>> 100005 1 udp 58278 mountd
>>> 100005 1 tcp 37178 mountd
>>> 100005 2 udp 58278 mountd
>>> 100005 2 tcp 37178 mountd
>>> 100005 3 udp 58278 mountd
>>> 100005 3 tcp 37178 mountd
>>
>> Can you boot if you specify either the "tcp" or "proto=tcp" NFSROOT mount options?
> When add proto=tcp to bootargs it boot fine
> (....nfsroot=10.146.1.21:/home/open-nandra/rootfs,proto=tcp....).
>> Perhaps a network trace would be probative. Capture on the server with "tcpdump -s0 -w /tmp/foo ip 10.146.1.199" (untested, but I think you get the idea) while the client is attempting to boot, and post.
> Log is attached in attachment (too big 4.8M). Correct form is: tcpdump
> -s0 -w /tmp/foo host 10.146.1.199
Received. I should have been clear: Please capture a non-working client boot attempt. To reduce the size of the attachment, strip the TFTP packets before sending, and please gzip the file.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Tue, Mar 29, 2011 at 04:24:08PM -0400, Will Simoneau wrote:
> Interestingly, we are both using DM9000 network chips. I wonder if there
> is a UDP-related problem on the DM9000. I will play with my
> configuration a little more (in particular testing proto=tcp and revert
> of 53d4737580535e073963b91ce87d4216e434fab5) but my guess is the DM9000
> driver/chip is to blame. The DM9000A on my board has 16K of on-chip SRAM
> used for its RX/TX buffers which may be playing a part in the problem.
It's more likely that the DM9000 is just slow and fragile. One thing to
note with a lot of the lower end network chips is that you end up
needing to specify rsize and wsize so that you don't get fragmentation -
about 1k tends to work well IME. Otherwise performance often degrades
to the point of unusability.
On 09:26 Fri 25 Mar , Chuck Lever wrote:
>
> According to the trace, the mount succeeds, and NFS requests are working. However, the server's replies to READ requests are not being seen by the client. Something on the client host or in the network is dropping them. I see successful 16KB reads, but 32KB reads do not work. Check your client for packet filtering (either intentional filtering, or accidental filtering due to configuration issues).
>
> I'll revisit the mount option defaults for NFSROOT again.
I have encountered the same problem as Belisko, using an FPGA board I am
working on (CPU is a mipsel r4k clone). 2.6.38-rc8 nfsroot works fine,
2.6.38 hangs while starting init. I can see read replies going out from
the NFS server but the client does not seem to receive them.
Interestingly, we are both using DM9000 network chips. I wonder if there
is a UDP-related problem on the DM9000. I will play with my
configuration a little more (in particular testing proto=tcp and revert
of 53d4737580535e073963b91ce87d4216e434fab5) but my guess is the DM9000
driver/chip is to blame. The DM9000A on my board has 16K of on-chip SRAM
used for its RX/TX buffers which may be playing a part in the problem.
On Mar 25, 2011, at 3:24 AM, Belisko Marek wrote:
> On Wed, Mar 23, 2011 at 3:54 PM, Chuck Lever <[email protected]> wrote:
>>
>> On Mar 23, 2011, at 10:30 AM, Belisko Marek wrote:
>>
>> The server is advertising NFS over UDP. Why can't a client access your server via UDP? What happens if you perform a normal post-boot mount of this file system via UDP?
> Strange (rootfs mounted from SD card and try to mount nfs always fail)
>
> mount -t nfs -o udp 10.146.1.21:/home/open-nandra/rootfs /mnt
> svc: failed to register lockdv1 RPC service (errno 111).
> lockd_up: makesock failed, error=-111
> mount: mounting 10.146.1.21:/home/open-nandra/rootfs on /mnt failed:
> Connection refused
> # mount -t nfs -o tcp 10.146.1.21:/home/open-nandra/rootfs /mnt
> svc: failed to register lockdv1 RPC service (errno 111).
> mount: mounting 10.146.1.21:/home/open-nandra/rootfs on /mnt failed:
> Connection refused
The client system isn't running a portmapper that its NFS client (specifically lockd) can talk to. That "connection refused" is due to an incorrect client configuration. A simple workaround might be to specify the "nolock" mount option for this test.
>
>>
>>>>> 100227 2 udp 2049
>>>>> 100227 3 udp 2049
>>>>> 100005 1 udp 58278 mountd
>>>>> 100005 1 tcp 37178 mountd
>>>>> 100005 2 udp 58278 mountd
>>>>> 100005 2 tcp 37178 mountd
>>>>> 100005 3 udp 58278 mountd
>>>>> 100005 3 tcp 37178 mountd
>>>>
>>>> Can you boot if you specify either the "tcp" or "proto=tcp" NFSROOT mount options?
>>> When add proto=tcp to bootargs it boot fine
>>> (....nfsroot=10.146.1.21:/home/open-nandra/rootfs,proto=tcp....).
>>>> Perhaps a network trace would be probative. Capture on the server with "tcpdump -s0 -w /tmp/foo ip 10.146.1.199" (untested, but I think you get the idea) while the client is attempting to boot, and post.
>>> Log is attached in attachment (too big 4.8M). Correct form is: tcpdump
>>> -s0 -w /tmp/foo host 10.146.1.199
>>
>> Received. I should have been clear: Please capture a non-working client boot attempt. To reduce the size of the attachment, strip the TFTP packets before sending, and please gzip the file.
> tcpdump log from wrong attemp in attachment.
According to the trace, the mount succeeds, and NFS requests are working. However, the server's replies to READ requests are not being seen by the client. Something on the client host or in the network is dropping them. I see successful 16KB reads, but 32KB reads do not work. Check your client for packet filtering (either intentional filtering, or accidental filtering due to configuration issues).
I'll revisit the mount option defaults for NFSROOT again.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Wed, Mar 23, 2011 at 3:54 PM, Chuck Lever <[email protected]> wrote:
>
> On Mar 23, 2011, at 10:30 AM, Belisko Marek wrote:
>
>> On Wed, Mar 23, 2011 at 3:06 PM, Chuck Lever <[email protected]> wrote:
>>>
>>> On Mar 23, 2011, at 9:46 AM, Belisko Marek wrote:
>>>
>>>> On Wed, Mar 23, 2011 at 2:44 PM, Chuck Lever <[email protected]> wrote:
>>>>>
>>>>> What does "rpcinfo -p 10.146.1.21" output look like? Is NFS over UDP enabled on your server?
>>>> program vers proto port
>>>> 100000 2 tcp 111 portmapper
>>>> 100000 2 udp 111 portmapper
>>>> 100024 1 udp 40517 status
>>>> 100024 1 tcp 51397 status
>>>> 100021 1 udp 44516 nlockmgr
>>>> 100021 3 udp 44516 nlockmgr
>>>> 100021 4 udp 44516 nlockmgr
>>>> 100021 1 tcp 55152 nlockmgr
>>>> 100021 3 tcp 55152 nlockmgr
>>>> 100021 4 tcp 55152 nlockmgr
>>>> 100003 2 tcp 2049 nfs
>>>> 100003 3 tcp 2049 nfs
>>>> 100003 4 tcp 2049 nfs
>>>> 100227 2 tcp 2049
>>>> 100227 3 tcp 2049
>>>> 100003 2 udp 2049 nfs
>>>> 100003 3 udp 2049 nfs
>>>> 100003 4 udp 2049 nfs
>
> The server is advertising NFS over UDP. Why can't a client access your server via UDP? What happens if you perform a normal post-boot mount of this file system via UDP?
Strange (rootfs mounted from SD card and try to mount nfs always fail)
mount -t nfs -o udp 10.146.1.21:/home/open-nandra/rootfs /mnt
svc: failed to register lockdv1 RPC service (errno 111).
lockd_up: makesock failed, error=-111
mount: mounting 10.146.1.21:/home/open-nandra/rootfs on /mnt failed:
Connection refused
# mount -t nfs -o tcp 10.146.1.21:/home/open-nandra/rootfs /mnt
svc: failed to register lockdv1 RPC service (errno 111).
mount: mounting 10.146.1.21:/home/open-nandra/rootfs on /mnt failed:
Connection refused
>
>>>> 100227 2 udp 2049
>>>> 100227 3 udp 2049
>>>> 100005 1 udp 58278 mountd
>>>> 100005 1 tcp 37178 mountd
>>>> 100005 2 udp 58278 mountd
>>>> 100005 2 tcp 37178 mountd
>>>> 100005 3 udp 58278 mountd
>>>> 100005 3 tcp 37178 mountd
>>>
>>> Can you boot if you specify either the "tcp" or "proto=tcp" NFSROOT mount options?
>> When add proto=tcp to bootargs it boot fine
>> (....nfsroot=10.146.1.21:/home/open-nandra/rootfs,proto=tcp....).
>>> Perhaps a network trace would be probative. Capture on the server with "tcpdump -s0 -w /tmp/foo ip 10.146.1.199" (untested, but I think you get the idea) while the client is attempting to boot, and post.
>> Log is attached in attachment (too big 4.8M). Correct form is: tcpdump
>> -s0 -w /tmp/foo host 10.146.1.199
>
> Received. I should have been clear: Please capture a non-working client boot attempt. To reduce the size of the attachment, strip the TFTP packets before sending, and please gzip the file.
tcpdump log from wrong attemp in attachment.
>
> --
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
>
>
>
>
>
regards,
marek
--
as simple and primitive as possible
-------------------------------------------------
Marek Belisko - OPEN-NANDRA
Freelance Developer
Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
Tel: +421 915 052 184
skype: marekwhite
icq: 290551086
web: http://open-nandra.com