2011-11-03 19:49:12

by Lukas Razik

[permalink] [raw]
Subject: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

Hello together!

My OS: Debian 6.0.3 (squeeze)
Machines: SUN Enterprise T5120 (USPARC64)
---
Issue description:

I've an NFS
server (cluster1=137.226.167.241) and a
client (cluster2=137.226.167.242) which should mount it's nfsroot from cluster1.

The linux-2.6.32 kernel on cluster2 shows this during startup:
[ 528.982985] IP-Config: Complete:
[ 528.983049] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
[ 528.983299] host=cluster2, domain=, nis-domain=(none),
[ 528.983383] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[ 528.983633] Looking up port of RPC 100003/2 on 137.226.167.241
[ 530.037059] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[ 530.056881] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 564.002113] rpcbind: server 137.226.167.241 not responding, timed out
[ 564.002295] Root-NFS: Unable to get nfsd port number from server, using default
[ 564.002412] Looking up port of RPC 100005/1 on 137.226.167.241
[ 564.104137] VFS: Mounted root (nfs filesystem) on device 0:15.

It can mount the nfsroot finally.

But if I use kernel linux-2.6.39.4 on cluster2 it can't mount it's nfsroot.
(I've added "nfsdebug" to the kernel arguments for more debug info):
[ 407.571521] IP-Config: Complete:
[ 407.571589] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
[ 407.571793] host=cluster2, domain=, nis-domain=(none),
[ 407.571907] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[ 407.572332] Root-NFS: nfsroot=/srv/nfs/cluster2
[ 407.572726] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
[ 407.572927] NFS: parsing nfs mount option 'udp'
[ 407.572995] NFS: parsing nfs mount option 'nolock'
[ 407.573071] NFS: parsing nfs mount option 'addr=137.226.167.241'
[ 407.573139] NFS: MNTPATH: '/srv/nfs/cluster2'
[ 407.573203] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[ 408.617894] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[ 408.638319] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 442.666622] NFS: failed to create MNT RPC client, status=-60
[ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
[ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
[ 442.667032] VFS: Insert root floppy and press ENTER


The NFS _server_ is always the same.
In 2.6.32, 2.6.33, 2.6.35, 2,6.36 and 2.6.36.4 an NFS mount for nfsroot works fine.
I've tested these kernels where the issue occurs: 2.6.37, 2.6.38, 2.6.39.4.
So there's a break between 2.6.36.4 and 2.6.37.

If you need more information - please ask!
The problem is that I need a kernel >= 2.6.37 because of some other issues...
So I'm very thankful for each help and I provide all needed information very fast.


Regards,
Lukas


PS:
My kernel-configs:
http://net.razik.de/linux/T5120/

And here the NFS parts:
linux-2.6.36.4# grep NFS .config
?CONFIG_NFS_FS=y
?CONFIG_NFS_V3=y
?CONFIG_NFS_V3_ACL=y
?CONFIG_NFS_V4=y
?# CONFIG_NFS_V4_1 is not set
?CONFIG_ROOT_NFS=y
?# CONFIG_NFS_USE_LEGACY_DNS is not set
?CONFIG_NFS_USE_KERNEL_DNS=y
?CONFIG_NFSD=m
?CONFIG_NFSD_V2_ACL=y
?CONFIG_NFSD_V3=y
?CONFIG_NFSD_V3_ACL=y
?CONFIG_NFSD_V4=y
?CONFIG_NFS_ACL_SUPPORT=y
?CONFIG_NFS_COMMON=y
?CONFIG_NCPFS_NFS_NS=y


linux-2.6.37# grep NFS .config
?CONFIG_NFS_FS=y
?CONFIG_NFS_V3=y
?CONFIG_NFS_V3_ACL=y
?CONFIG_NFS_V4=y
?# CONFIG_NFS_V4_1 is not set
?CONFIG_ROOT_NFS=y
?# CONFIG_NFS_USE_LEGACY_DNS is not set
?CONFIG_NFS_USE_KERNEL_DNS=y
?# CONFIG_NFS_USE_NEW_IDMAPPER is not set
?CONFIG_NFSD=m
?CONFIG_NFSD_DEPRECATED=y
?CONFIG_NFSD_V2_ACL=y
?CONFIG_NFSD_V3=y
?CONFIG_NFSD_V3_ACL=y
?CONFIG_NFSD_V4=y
?CONFIG_NFS_ACL_SUPPORT=y
?CONFIG_NFS_COMMON=y
?CONFIG_NCPFS_NFS_NS=y


2011-11-13 22:19:16

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> [ 1764.189975] ADDRCONF(NETDEV_UP): eth0: link is not ready

> [ 1765.212110] IP-Config: Complete:
> [ 1765.212180]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224,
> gw=137.226.167.225,
> [ 1765.212383]????? host=137.226.167.242, domain=, nis-domain=(none),
> [ 1765.212498]????? bootserver=255.255.255.255, rootserver=137.226.167.241,
> rootpath=
> [ 1765.212943] VFS: 1. attempt to mount root fs via NFS.
> [ 1765.212992] Root-NFS: nfsroot=/srv/nfs/cluster2
> [ 1765.213265] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
> [ 1765.213315] NFS:?? parsing nfs mount option 'udp'
> [ 1765.213353] NFS:?? parsing nfs mount option 'nolock'
> [ 1765.213399] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
> [ 1765.213467] NFS: MNTPATH: '/srv/nfs/cluster2'
> [ 1765.213501] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
> [ 1766.298480] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control:
> Rx
> [ 1766.318949] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 1800.284325] NFS: failed to create MNT RPC client, status=-60
> [ 1800.284435] NFS: unable to mount server 137.226.167.241, error -60
> [ 1800.284571] VFS: 2. attempt to mount root fs via NFS.
> [ 1800.284682] Root-NFS: nfsroot=/srv/nfs/cluster2
> [ 1800.285051] NFS: nfs mount
> opts='udp,nolock,addr=137.226.167.241,nolock,addr=137.226.167.241'
> [ 1800.285167] NFS:?? parsing nfs mount option 'udp'
> [ 1800.285233] NFS:?? parsing nfs mount option 'nolock'
> [ 1800.285337] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
> [ 1800.285455] NFS:?? parsing nfs mount option 'nolock'
> [ 1800.285531] NFS:?? parsing nfs mount option 'addr=137.226.167.241'

Chuck, I've seen that it's wrong to wrap mount_nfs_root() with while() because then we get a multiple occurrence of some mount options. Hence I've edited mount_nfs_root() itself so that the options will be set only once.
Maybe that's better?
--- linux-2.6.39.4/init/do_mounts.c??????? 2011-08-03 21:43:28.000000000 +0200
+++ linux-2.6.39.4-fix/init/do_mounts.c???? 2011-11-13 23:05:13.000000000 +0100
@@ -363,12 +363,18 @@
?static int __init mount_nfs_root(void)
?{
??????? char *root_dev, *root_data;
+?????? char attempt;
?
??????? if (nfs_root_data(&root_dev, &root_data) != 0)
??????????????? return 0;
-?????? if (do_mount_root(root_dev, "nfs", root_mountflags, root_data) != 0)
-?????????????? return 0;
-?????? return 1;
+
+?????? for(attempt = 1; attempt <= 3; attempt++) {
+?????????????? printk(KERN_INFO "VFS: %d. attempt to mount root fs via NFS.\n", attempt);
+?????????????? if (do_mount_root(root_dev, "nfs", root_mountflags, root_data) == 0)
+?????????????????????? return 1;
+?????? }
+
+?????? return 0;
?}
?#endif

--
Regards,
Lukas


PS: That's the new behaviour:
[ 4416.829621] ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4417.851657] IP-Config: Complete:
[ 4417.851729]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
[ 4417.851931]????? host=137.226.167.242, domain=, nis-domain=(none),
[ 4417.852045]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[ 4417.852493] Root-NFS: nfsroot=/srv/nfs/cluster2
[ 4417.852539] VFS: 1. attempt to mount root fs via NFS.
[ 4417.852814] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
[ 4417.852893] NFS:?? parsing nfs mount option 'udp'
[ 4417.852930] NFS:?? parsing nfs mount option 'nolock'
[ 4417.852977] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[ 4417.853073] NFS: MNTPATH: '/srv/nfs/cluster2'
[ 4417.853137] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[ 4418.933894] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[ 4418.954252] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 4452.967331] NFS: failed to create MNT RPC client, status=-60
[ 4452.967441] NFS: unable to mount server 137.226.167.241, error -60
[ 4452.967577] VFS: 2. attempt to mount root fs via NFS.
[ 4452.967978] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
[ 4452.968085] NFS:?? parsing nfs mount option 'udp'
[ 4452.968150] NFS:?? parsing nfs mount option 'nolock'
[ 4452.968197] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[ 4452.968333] NFS: MNTPATH: '/srv/nfs/cluster2'
[ 4452.968397] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[ 4452.974480] NFS: received 1 auth flavors
[ 4452.974543] NFS:?? auth flavor[0]: 1
[ 4452.974723] NFS: MNT request succeeded
[ 4452.974785] NFS: using auth flavor 1
[ 4452.979498] VFS: Mounted root (nfs filesystem) on device 0:13.
INIT: version 2.88 booting

2011-11-13 01:03:53

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

Chuck Lever <[email protected]> wrote:

> On Nov 12, 2011, at 1:49 PM, Jim Rees wrote:
>
>> The question for us is how long should an nfsroot client wait for the
> server
>> to reply.? It sounds like the client used to wait longer than it does now.
>
> Before, the client performed the GETPORT(NFS) step synchronously, first.? This
> took 30 seconds or so to timeout.? When it did, the client decided to proceed
> with port 2049.? Then it went on to do the other mount tasks, and at the point
> had waited long enough that these tasks did not time out while waiting for the
> switch port.
>
>> It seems to me the client should wait at least 90 seconds so that the
>> situation you're in (servers on non-portfast ports) will work.? I would
>> think they should wait indefinitely, since there's not much else they
> can
>> do.
>
> It should be simple to wrap the (MNT(mnt), NFS(getroot)) steps in a while(true)
> loop.? Would mount_root_nfs() be the right place for this?
>

I thought it would be harder and I had no time to look inside the kernel but now I wrote a patch:
The kernel tries to create the MNT RPC client not once as before but three times - then it gives up.
Third time lucky... ;-)
In my case the 2. MNT request is successful:
---
[?? 71.594744] ADDRCONF(NETDEV_UP): eth0: link is not ready
[?? 72.617007] IP-Config: Complete:
[?? 72.617077]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
[?? 72.617278]????? host=137.226.167.242, domain=, nis-domain=(none),
[?? 72.617393]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[?? 72.617741] Root-NFS: nfsroot=/srv/nfs/cluster2
[?? 72.618010] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
[?? 72.618147] NFS:?? parsing nfs mount option 'udp'
[?? 72.618187] NFS:?? parsing nfs mount option 'nolock'
[?? 72.618233] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[?? 72.618301] NFS: MNTPATH: '/srv/nfs/cluster2'
[?? 72.618335] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[?? 72.618383] NFS: 1. MNT request
[?? 73.691872] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[?? 73.711988] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[? 107.697332] NFS: 2. MNT request
[? 107.704591] NFS: received 1 auth flavors
[? 107.704653] NFS:?? auth flavor[0]: 1
[? 107.704834] NFS: MNT request succeeded
[? 107.704897] NFS: using auth flavor 1
[? 107.711857] VFS: Mounted root (nfs filesystem) on device 0:13.
INIT: version 2.88 booting
---

So many thanks again for your help and your very helpful hints!

Regards,
Lukas


PS: That's what I've done:
--- linux-2.6.39.4/fs/nfs/mount_clnt.c? 2011-08-03 21:43:28.000000000 +0200
+++ linux-2.6.39.4-fix/fs/nfs/mount_clnt.c????? 2011-11-13 01:58:13.000000000 +0100
@@ -164,6 +164,7 @@
??????? };
??????? struct rpc_clnt???????? *mnt_clnt;
??????? int???????????????????? status;
+?????? int???????????????????? attempt = 0;
?
??????? dprintk("NFS: sending MNT request for %s:%s\n",
??????????????? (info->hostname ? info->hostname : "server"),
@@ -172,7 +173,13 @@
??????? if (info->noresvport)
??????????????? args.flags |= RPC_CLNT_CREATE_NONPRIVPORT;
?
-?????? mnt_clnt = rpc_create(&args);
+?????? do {
+?????????????? attempt++;
+?????????????? dprintk("NFS: %d. MNT request\n", attempt);
+?????????????? mnt_clnt = rpc_create(&args);
+?????? } while (IS_ERR(mnt_clnt) && attempt < 3);
+
+
??????? if (IS_ERR(mnt_clnt))
??????????????? goto out_clnt_err;
--

2011-11-05 02:32:35

by Myklebust, Trond

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

On Sat, 2011-11-05 at 02:14 +0000, Lukas Razik wrote:
> ***IMPORTANT HINT***
>
> In the test belowI've tried to ping the NFS client (cluster2) as soon as I've got the
>
> ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> message.
>
>
> And at first I've got no answers from the client. But 10-20sec later I've got pongs back.
> So the NIC comes up some seconds later than the message "link becomes ready" is shown...
> In the case of 'mountproto=tcp' it comes up after the kernel panic. So that's much too late.
>

So, once again...

What kind of switch is this client connected to, and how is that switch
configured?

--
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
http://www.netapp.com


2011-11-05 02:32:34

by Myklebust, Trond

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

On Sat, 2011-11-05 at 02:14 +0000, Lukas Razik wrote:
> ***IMPORTANT HINT***
>
> In the test belowI've tried to ping the NFS client (cluster2) as soon as I've got the
>
> ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> message.
>
>
> And at first I've got no answers from the client. But 10-20sec later I've got pongs back.
> So the NIC comes up some seconds later than the message "link becomes ready" is shown...
> In the case of 'mountproto=tcp' it comes up after the kernel panic. So that's much too late.
>

So, once again...

What kind of switch is this client connected to, and how is that switch
configured?

--
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
http://www.netapp.com


2011-11-04 01:33:29

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64



> It agrees with our initial assessment that the first RPC request is failing.?
> The RPC client never gets the request through cluster2's network stack
> because the NIC hasn't re-initialized when the request is sent.
>
> It looks like your system does a PXE boot, which provides the IP configuration
> shown above.? But then the kernel resets the NIC.? During that reset, the kernel
> is attempting to contact the NFS server to mount the root file system.

This machine has no PXE boot but it has OpenBoot ;) and I can boot by fetching the kernel from a tftp.
But that's something we want to do in the future...
Now (for the sake of convenience) we load the kernel from hard disk but we want an NFSROOT.

Therefore I've set a static configuration :
root=/dev/nfs rw
nfsroot=137.226.167.241:/srv/nfs/cluster2
ip=137.226.167.242::137.226.167.225:255.255.255.224:cluster2:eth0:off
nfsrootdebug

Hence the NIC isn't "up" before the try to mount the NFSROOT.
Sorry, that I haven't mentioned it before my last email...


> We've set up NFSROOT to use UDP so that it will be relatively immune to
> these initialization order problems.? The RPC client should be retrying the lost
> request, but apparently it isn't.? What if you added "retrans=10"
> to cluster2's mount options?? (on the chance that mount option setting would
> be copied to the rpcbind client's RPC transport...)
> IMO the correct way to fix this is to provide proper serialization in the
> networking layer so that RPC requests are not even attempted until the NIC is
> ready to carry traffic.? That may be a pipe dream though.
>

As I wrote it in my last email the retrans option doesn nothing. I've tested it again.
That's the whole bootloader/kernel output with ",retrans=10,timeo=60":
---
{0} ok boot disk
Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@0? File and args:
SILO Version 1.4.14
boot: LinuxNET3-debug
Allocated 64 Megs of memory at 0x40000000 for kernel
Uncompressing image...
Loaded kernel version 2.6.39

[??? 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
[??? 0.000000] PROMLIB: Root node compatible: sun4v
[??? 0.000000] Initializing cgroup subsys cpuset
[??? 0.000000] Initializing cgroup subsys cpu
[??? 0.000000] Linux version 2.6.39.4-razik-2011-10-27 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #6 SMP Thu Nov 3 14:59:53 CET 2011
[??? 0.000000] bootconsole [earlyprom0] enabled
[??? 0.000000] ARCH: SUN4V
[??? 0.000000] Ethernet address: 00:14:4f:98:09:c2
[??? 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
[??? 0.000000] Remapping the kernel... done.
[??? 0.000000] OF stdout device is: /virtual-devices@100/console@1
[??? 0.000000] PROM: Built device tree with 123285 bytes of memory.
[??? 0.000000] MDESC: Size is 58672 bytes.
[??? 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
[??? 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
[??? 0.000000] PLATFORM: hostid [849809c2]
[??? 0.000000] PLATFORM: serial# [00ab4130]
[??? 0.000000] PLATFORM: stick-frequency [5458c3a0]
[??? 0.000000] PLATFORM: mac-address [144f9809c2]
[??? 0.000000] PLATFORM: watchdog-resolution [1000 ms]
[??? 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
[??? 0.000000] PLATFORM: max-cpus [64]
[??? 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c4000
[??? 0.000000] Memory hole size: 132MB
[??? 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
[??? 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
[??? 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
[??? 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
[??? 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
[??? 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
[??? 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
[??? 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
[??? 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
[??? 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
[??? 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
[??? 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
[??? 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
[??? 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
[??? 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
[??? 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
[??? 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
[??? 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
[??? 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
[??? 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
[??? 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
[??? 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
[??? 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
[??? 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
[??? 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
[??? 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
[??? 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
[??? 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
[??? 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
[??? 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
[??? 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
[??? 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
[??? 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
[??? 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
[??? 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
[??? 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
[??? 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
[??? 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
[??? 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
[??? 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
[??? 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
[??? 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
[??? 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
[??? 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
[??? 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
[??? 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
[??? 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
[??? 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
[??? 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
[??? 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
[??? 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
[??? 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
[??? 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
[??? 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
[??? 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
[??? 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
[??? 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
[??? 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
[??? 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
[??? 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
[??? 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
[??? 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
[??? 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
[??? 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
[??? 0.000000] Zone PFN ranges:
[??? 0.000000]?? Normal?? 0x00004200 -> 0x003ffd8e
[??? 0.000000] Movable zone start PFN for each node
[??? 0.000000] early_node_map[3] active PFN ranges
[??? 0.000000]???? 0: 0x00004200 -> 0x003ffd51
[??? 0.000000]???? 0: 0x003ffd59 -> 0x003ffd63
[??? 0.000000]???? 0: 0x003ffd87 -> 0x003ffd8e
[??? 0.000000] Booting Linux...
[??? 0.000000] PERCPU: Embedded 6 pages/cpu @fffff80019000000 s16256 r8192 d24704 u65536
[??? 0.000000] Built 1 zonelists in Zone order, mobility grouping on.? Total pages: 4144106
[??? 0.000000] Kernel command line: ro root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2,retrans=10,timeo=60 ip=137.226.167.242::137.226.167.225:255.255.255.224:cluster2:eth0:off nfsrootdebug
[??? 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
[??? 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
[??? 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
[??? 0.000000] Memory: 33090176k available (4192k kernel code, 1560k data, 232k init) [fffff80000000000,00000007ffb1c000]
[??? 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[??? 0.000000] Hierarchical RCU implementation.
[??? 0.000000]? CONFIG_RCU_FANOUT set to non-default value of 32
[??? 0.000000]? RCU-based detection of stalled CPUs is disabled.
[??? 0.000000] NR_IRQS:255
[??? 0.000000] clocksource: mult[5a73e7bb] shift[31]
[??? 0.000000] clockevent: mult[b5221ba3] shift[31]
[??? 0.000000] Console: colour dummy device 80x25
[??? 0.000000] console [tty0] enabled, bootconsole disabled
[??? 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
[??? 0.000000] PROMLIB: Root node compatible: sun4v
[??? 0.000000] Initializing cgroup subsys cpuset
[??? 0.000000] Initializing cgroup subsys cpu
[??? 0.000000] Linux version 2.6.39.4-razik-2011-10-27 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #6 SMP Thu Nov 3 14:59:53 CET 2011
[??? 0.000000] bootconsole [earlyprom0] enabled
[??? 0.000000] ARCH: SUN4V
[??? 0.000000] Ethernet address: 00:14:4f:98:09:c2
[??? 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
[??? 0.000000] Remapping the kernel... done.
[??? 0.000000] OF stdout device is: /virtual-devices@100/console@1
[??? 0.000000] PROM: Built device tree with 123285 bytes of memory.
[??? 0.000000] MDESC: Size is 58672 bytes.
[??? 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
[??? 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
[??? 0.000000] PLATFORM: hostid [849809c2]
[??? 0.000000] PLATFORM: serial# [00ab4130]
[??? 0.000000] PLATFORM: stick-frequency [5458c3a0]
[??? 0.000000] PLATFORM: mac-address [144f9809c2]
[??? 0.000000] PLATFORM: watchdog-resolution [1000 ms]
[??? 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
[??? 0.000000] PLATFORM: max-cpus [64]
[??? 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c4000
[??? 0.000000] Memory hole size: 132MB
[??? 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
[??? 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
[??? 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
[??? 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
[??? 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
[??? 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
[??? 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
[??? 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
[??? 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
[??? 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
[??? 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
[??? 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
[??? 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
[??? 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
[??? 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
[??? 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
[??? 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
[??? 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
[??? 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
[??? 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
[??? 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
[??? 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
[??? 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
[??? 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
[??? 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
[??? 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
[??? 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
[??? 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
[??? 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
[??? 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
[??? 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
[??? 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
[??? 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
[??? 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
[??? 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
[??? 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
[??? 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
[??? 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
[??? 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
[??? 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
[??? 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
[??? 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
[??? 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
[??? 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
[??? 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
[??? 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
[??? 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
[??? 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
[??? 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
[??? 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
[??? 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
[??? 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
[??? 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
[??? 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
[??? 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
[??? 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
[??? 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
[??? 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
[??? 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
[??? 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
[??? 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
[??? 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
[??? 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
[??? 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
[??? 0.000000] Zone PFN ranges:
[??? 0.000000]?? Normal?? 0x00004200 -> 0x003ffd8e
[??? 0.000000] Movable zone start PFN for each node
[??? 0.000000] early_node_map[3] active PFN ranges
[??? 0.000000]???? 0: 0x00004200 -> 0x003ffd51
[??? 0.000000]???? 0: 0x003ffd59 -> 0x003ffd63
[??? 0.000000]???? 0: 0x003ffd87 -> 0x003ffd8e
[??? 0.000000] Booting Linux...
[??? 0.000000] PERCPU: Embedded 6 pages/cpu @fffff80019000000 s16256 r8192 d24704 u65536
[??? 0.000000] Built 1 zonelists in Zone order, mobility grouping on.? Total pages: 4144106
[??? 0.000000] Kernel command line: ro root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2,retrans=10,timeo=60 ip=137.226.167.242::137.226.167.225:255.255.255.224:cluster2:eth0:off nfsrootdebug
[??? 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
[??? 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
[??? 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
[??? 0.000000] Memory: 33090176k available (4192k kernel code, 1560k data, 232k init) [fffff80000000000,00000007ffb1c000]
[??? 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[??? 0.000000] Hierarchical RCU implementation.
[??? 0.000000]? CONFIG_RCU_FANOUT set to non-default value of 32
[??? 0.000000]? RCU-based detection of stalled CPUs is disabled.
[??? 0.000000] NR_IRQS:255
[??? 0.000000] clocksource: mult[5a73e7bb] shift[31]
[??? 0.000000] clockevent: mult[b5221ba3] shift[31]
[??? 0.000000] Console: colour dummy device 80x25
[??? 0.000000] console [tty0] enabled, bootconsole disabled
[? 542.555558] Calibrating delay using timer specific routine.. 2832.87 BogoMIPS (lpj=5665753)
[? 542.555590] pid_max: default: 65536 minimum: 512
[? 542.555896] Security Framework initialized
[? 542.555929] SELinux:? Disabled at boot.
[? 542.556065] Mount-cache hash table entries: 512
[? 542.557246] Initializing cgroup subsys ns
[? 542.557266] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[? 542.557294] Initializing cgroup subsys cpuacct
[? 542.557389] Initializing cgroup subsys devices
[? 542.557407] Initializing cgroup subsys freezer
[? 542.557424] Initializing cgroup subsys net_cls
[? 542.557704] Performance events: Supported PMU type is 'niagara2'
[? 542.580874] Brought up 64 CPUs
[? 542.580934] Testing NMI watchdog ... OK.
[? 542.666730] devtmpfs: initialized
[? 542.667184] ldc.c:v1.1 (July 22, 2008)
[? 542.667793] print_constraints: dummy:
[? 542.668159] NET: Registered protocol family 16
[? 542.677406] VIO: Adding device channel-devices
[? 542.677484] VIO: Adding device vldc-port-3-0
[? 542.677543] VIO: Adding device vldc-port-3-1
[? 542.677603] VIO: Adding device vldc-port-3-2
[? 542.677670] VIO: Adding device vldc-port-3-3
[? 542.677745] VIO: Adding device vldc-port-3-4
[? 542.677811] VIO: Adding device vldc-port-3-5
[? 542.677877] VIO: Adding device vldc-port-2-0
[? 542.677944] VIO: Adding device vldc-port-0-0
[? 542.678011] VIO: Adding device vldc-port-0-1
[? 542.678079] VIO: Adding device vldc-port-0-2
[? 542.678147] VIO: Adding device vldc-port-1-0
[? 542.678215] VIO: Adding device vldc-port-3-7
[? 542.678287] VIO: Adding device vldc-port-3-8
[? 542.678369] VIO: Adding device ds-1
[? 542.678432] VIO: Adding device ds-0
[? 542.683082] pci_sun4v: Registered hvapi major[1] minor[0]
[? 542.683356] /pci@0: SUN4V PCI Bus Module
[? 542.683376] /pci@0: On NUMA node -1
[? 542.683397] /pci@0: PCI IO[c0f0000000] MEM[c100000000]
[? 542.683426] /pci@0: Unable to request IOMMU resource.
[? 542.727780] /pci@0: Imported 3 TSB entries from OBP
[? 542.729230] /pci@0: MSI Queue first[0] num[36] count[128] devino[0x18]
[? 542.729258] /pci@0: MSI first[0] num[256] mask[0xff] width[32]
[? 542.729283] /pci@0: MSI addr32[0x7fff0000:0x10000] addr64[0x3ffff0000:0x10000]
[? 542.729312] /pci@0: MSI queues at RA [00000007f8300000]
[? 542.729335] PCI: Scanning PBM /pci@0
[? 542.741097] ds.c:v1.0 (Jul 11, 2007)
[? 542.744664] bio: create slab <bio-0> at 0
[? 542.745277] vgaarb: loaded
[? 542.746083] Switching to clocksource stick
[? 542.746745] Switched to NOHz mode on CPU #35
[? 542.746834] Switched to NOHz mode on CPU #12
[? 542.746857] Switched to NOHz mode on CPU #58
[? 542.746907] Switched to NOHz mode on CPU #24
[? 542.746935] Switched to NOHz mode on CPU #47
[? 542.747082] Switched to NOHz mode on CPU #36
[? 542.747110] Switched to NOHz mode on CPU #1
[? 542.747169] Switched to NOHz mode on CPU #13
[? 542.747228] Switched to NOHz mode on CPU #59
[? 542.747253] Switched to NOHz mode on CPU #25
[? 542.747292] Switched to NOHz mode on CPU #48
[? 542.747460] Switched to NOHz mode on CPU #37
[? 542.747470] Switched to NOHz mode on CPU #0
[? 542.747495] Switched to NOHz mode on CPU #2
[? 542.747504] Switched to NOHz mode on CPU #14
[? 542.747587] Switched to NOHz mode on CPU #60
[? 542.747606] Switched to NOHz mode on CPU #26
[? 542.747651] Switched to NOHz mode on CPU #49
[? 542.747780] Switched to NOHz mode on CPU #38
[? 542.747829] Switched to NOHz mode on CPU #3
[? 542.747841] Switched to NOHz mode on CPU #15
[? 542.747939] Switched to NOHz mode on CPU #61
[? 542.747948] Switched to NOHz mode on CPU #27
[? 542.748000] Switched to NOHz mode on CPU #50
[? 542.748125] Switched to NOHz mode on CPU #39
[? 542.748163] Switched to NOHz mode on CPU #4
[? 542.748178] Switched to NOHz mode on CPU #16
[? 542.748281] Switched to NOHz mode on CPU #28
[? 542.748305] Switched to NOHz mode on CPU #62
[? 542.748349] Switched to NOHz mode on CPU #51
[? 542.748471] Switched to NOHz mode on CPU #40
[? 542.748504] Switched to NOHz mode on CPU #5
[? 542.748526] Switched to NOHz mode on CPU #17
[? 542.748633] Switched to NOHz mode on CPU #29
[? 542.748664] Switched to NOHz mode on CPU #63
[? 542.748697] Switched to NOHz mode on CPU #52
[? 542.748823] Switched to NOHz mode on CPU #41
[? 542.748839] Switched to NOHz mode on CPU #6
[? 542.748858] Switched to NOHz mode on CPU #18
[? 542.748979] Switched to NOHz mode on CPU #30
[? 542.749073] Switched to NOHz mode on CPU #53
[? 542.749170] Switched to NOHz mode on CPU #7
[? 542.749179] Switched to NOHz mode on CPU #42
[? 542.749192] Switched to NOHz mode on CPU #19
[? 542.749329] Switched to NOHz mode on CPU #31
[? 542.749439] Switched to NOHz mode on CPU #54
[? 542.749469] Switched to NOHz mode on CPU #8
[? 542.749526] Switched to NOHz mode on CPU #43
[? 542.749535] Switched to NOHz mode on CPU #20
[? 542.749681] Switched to NOHz mode on CPU #32
[? 542.749795] Switched to NOHz mode on CPU #55
[? 542.749804] Switched to NOHz mode on CPU #9
[? 542.749868] Switched to NOHz mode on CPU #44
[? 542.749879] Switched to NOHz mode on CPU #21
[? 542.750021] Switched to NOHz mode on CPU #33
[? 542.750143] Switched to NOHz mode on CPU #10
[? 542.750152] Switched to NOHz mode on CPU #56
[? 542.750217] Switched to NOHz mode on CPU #45
[? 542.750226] Switched to NOHz mode on CPU #22
[? 542.750364] Switched to NOHz mode on CPU #34
[? 542.750566] Switched to NOHz mode on CPU #23
[? 542.750576] Switched to NOHz mode on CPU #46
[? 542.750585] Switched to NOHz mode on CPU #57
[? 542.750594] Switched to NOHz mode on CPU #11
[? 542.762031] NET: Registered protocol family 2
[? 542.762624] IP route cache hash table entries: 524288 (order: 9, 4194304 bytes)
[? 542.763059] ds-1: Registered pri service.
[? 542.765208] TCP established hash table entries: 524288 (order: 10, 8388608 bytes)
[? 542.769250] ds-1: Registered var-config-backup service.
[? 542.790571] TCP bind hash table entries: 65536 (order: 7, 1048576 bytes)
[? 542.794314] TCP: Hash tables configured (established 524288 bind 65536)
[? 542.794340] TCP reno registered
[? 542.794379] UDP hash table entries: 16384 (order: 6, 524288 bytes)
[? 542.796402] UDP-Lite hash table entries: 16384 (order: 6, 524288 bytes)
[? 542.799835] NET: Registered protocol family 1
[? 542.800489] RPC: Registered udp transport module.
[? 542.800512] RPC: Registered tcp transport module.
[? 542.800532] RPC: Registered tcp NFSv4.1 backchannel transport module.
[? 542.801635] audit: initializing netlink socket (disabled)
[? 542.801684] type=2000 audit(0.316:1): initialized
[? 542.854020] HugeTLB registered 4 MB page size, pre-allocated 0 pages
[? 542.862444] VFS: Disk quotas dquot_6.5.2
[? 542.862958] Dquot-cache hash table entries: 1024 (order 0, 8192 bytes)
[? 542.864268] msgmni has been set to 32768
[? 542.865471] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[? 542.865505] io scheduler noop registered
[? 542.865524] io scheduler deadline registered
[? 542.865763] io scheduler cfq registered (default)
[? 542.867089] f027ab74: ttyS0 at I/O 0x0 (irq = 17) is a SUN4V HCONS
[? 543.075410] console [ttyHV0] enabled
[? 543.077100] f028af14: ttyS1 at MMIO 0xfff0ca0000 (irq = 24) is a 16550A
[? 543.077715] [drm] Initialized drm 1.1.0 20060810
[? 543.081079] Uniform Multi-Platform E-IDE driver
[? 543.081244] ide-gd driver 1.18
[? 543.081307] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[? 543.081356] e1000: Copyright (c) 1999-2006 Intel Corporation.
[? 543.081486] e1000e: Intel(R) PRO/1000 Network Driver - 1.3.10-k2
[? 543.081533] e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
[? 543.081631] e1000e 0000:08:00.0: Disabling ASPM? L1
[? 543.257986] e1000e 0000:08:00.0: eth0: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c2
[? 543.258192] e1000e 0000:08:00.0: eth0: Intel(R) PRO/1000 Network Connection
[? 543.258319] e1000e 0000:08:00.0: eth0: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 543.258400] e1000e 0000:08:00.1: Disabling ASPM? L1
[? 543.443816] e1000e 0000:08:00.1: eth1: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c3
[? 543.443937] e1000e 0000:08:00.1: eth1: Intel(R) PRO/1000 Network Connection
[? 543.444121] e1000e 0000:08:00.1: eth1: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 543.444248] e1000e 0000:09:00.0: Disabling ASPM? L1
[? 543.622266] e1000e 0000:09:00.0: eth2: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c4
[? 543.622387] e1000e 0000:09:00.0: eth2: Intel(R) PRO/1000 Network Connection
[? 543.622570] e1000e 0000:09:00.0: eth2: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 543.622697] e1000e 0000:09:00.1: Disabling ASPM? L1
[? 543.808183] e1000e 0000:09:00.1: eth3: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c5
[? 543.808304] e1000e 0000:09:00.1: eth3: Intel(R) PRO/1000 Network Connection
[? 543.808487] e1000e 0000:09:00.1: eth3: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 543.808655] Intel(R) Gigabit Ethernet Network Driver - version 3.0.6-k2
[? 543.808734] Copyright (c) 2007-2011 Intel Corporation.
[? 543.808854] Intel(R) Virtual Function Network Driver - version 1.0.8-k0
[? 543.808904] Copyright (c) 2009 - 2010 Intel Corporation.
[? 543.809548] mousedev: PS/2 mouse device common for all mice
[? 543.810734] rtc-sun4v rtc-sun4v: rtc core: registered sun4v as rtc0
[? 543.812500] TCP cubic registered
[? 543.813350] NET: Registered protocol family 10
[? 543.824976] Mobile IPv6
[? 543.825023] NET: Registered protocol family 17
[? 543.825107] Registering the dns_resolver key type
[? 543.825741] registered taskstats version 1
[? 543.828720] rtc-sun4v rtc-sun4v: setting system clock to 2011-11-04 01:12:04 UTC (1320369124)
[? 543.828779] Initializing network drop monitor service
[? 544.577265] ADDRCONF(NETDEV_UP): eth0: link is not ready
[? 545.599366] IP-Config: Complete:
[? 545.599435]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
[? 545.599638]????? host=cluster2, domain=, nis-domain=(none),
[? 545.599751]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[? 545.600198] Root-NFS: nfsroot=/srv/nfs/cluster2,retrans=10,timeo=60
[? 545.600521] NFS: nfs mount opts='udp,retrans=10,timeo=60,nolock,addr=137.226.167.241'
[? 545.600616] NFS:?? parsing nfs mount option 'udp'
[? 545.600654] NFS:?? parsing nfs mount option 'retrans=10'
[? 545.600710] NFS:?? parsing nfs mount option 'timeo=60'
[? 545.600760] NFS:?? parsing nfs mount option 'nolock'
[? 545.600807] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[? 545.600871] NFS: MNTPATH: '/srv/nfs/cluster2'
[? 545.600906] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[? 546.737859] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[? 546.758283] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[? 580.671263] NFS: failed to create MNT RPC client, status=-60
[? 580.671373] NFS: unable to mount server 137.226.167.241, error -60
[? 580.671508] VFS: Unable to mount root fs via NFS, trying floppy.
[? 580.671657] VFS: Insert root floppy and press ENTER

2011-11-03 21:52:17

by Chuck Lever III

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64


On Nov 3, 2011, at 5:37 PM, Lukas Razik wrote:

>> On Nov 3, 2011, at 5:11 PM, Jim Rees wrote:
>
>>
>>> Trond Myklebust wrote:
>>>
>>>> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
>>>> [ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
>>>> [ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
>>>> [ 442.667032] VFS: Insert root floppy and press ENTER
>>>>
>>> Error 60 is ETIMEDOUT on SPARC, so it seems that the problem is
>>> basically the same one that you see in your 2.6.32 trace (rpcbind:
>>> server 137.226.167.241 not responding, timed out) except that now it is
>>> a fatal error.
>>>
>>> Any idea why the first RPC calls might be failing here? A switch
>>> misconfiguration or something like that perhaps?
>>>
>>> Wasn't there a change in the way nfs mount options are handled by the
>> kernel
>>> for nfsroot about the time of 2.6.39? Something about changing from
>> default
>>> udp to tcp maybe?
>>
>> There was a change, but it was changed back to UDP because of problems like
>> this. Behavior in 3.0 or the latest 2.6.39 stable kernel may be improved.
>>
>
> I don't know if this was a tip to test newest 2.6.39 but as I wrote in my first email
> http://thread.gmane.org/gmane.linux.nfs/44596
> that's the output of linux-2.6.39.4 with "nfsdebug":
>
> [ 407.571521] IP-Config: Complete:
> [ 407.571589] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
> [ 407.571793] host=cluster2, domain=, nis-domain=(none),
> [ 407.571907] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
> [ 407.572332] Root-NFS: nfsroot=/srv/nfs/cluster2
> [ 407.572726] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
> [ 407.572927] NFS: parsing nfs mount option 'udp'
> [ 407.572995] NFS: parsing nfs mount option 'nolock'
> [ 407.573071] NFS: parsing nfs mount option 'addr=137.226.167.241'
> [ 407.573139] NFS: MNTPATH: '/srv/nfs/cluster2'
> [ 407.573203] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
> [ 408.617894] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
> [ 408.638319] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
> [ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
> [ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
> [ 442.667032] VFS: Insert root floppy and press ENTER
>
> And this behaviour is exactly the same as in all other 2.6.37 - 2.6.39.4 which I've tested.
> So if anybody of you all have an idea what I could try to do, I'll follow...

Find out why the very first RPC on your system always fails. As Trond says, the only reason this worked on the older kernels is because NFSROOT fell back to a default port for NFSD. This is also broken behavior, but in your case it happened to work so you never noticed it.

I seem to recall there's a way to set the NFS and RPC debugging flags on the kernel command line so more information can be captured during boot. But I don't see it under Documentation/.

You could add a line in fs/nfs/nfsroot.c:nfs_root_debug() to set flags also in the rpc_debug global variable to gather more information.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com





2011-11-04 22:55:51

by Chuck Lever III

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64


On Nov 4, 2011, at 11:46 AM, Lukas Razik wrote:

>> > As for a fix... we're trying to move away from udp transport
>
>> anyway. Maybe
>> > someone should figure out a way to get it to work with tcp? I have zero
>> > experience with nfs over udp, at least on linux. Just for fun, have you
>> > tried tcp transport (proto=tcp)?
>>
>> TCP is a real problem in this environment, because it deals poorly with
>> NIC initialization timing issues. UDP is still the best approach (as long
>> as it is retransmitting appropriately). To support TCP, ultimately what
>> we need to do is to introduce serialization to make the kernel wait for
>> the NIC to become ready before attempting network activity.
>>
>> Agreed, but we're grasping at straws here, and this guy just wants it to
>> work. It's something to try.
>
> I've tried that but "proto=tcp" is no way to change the protocol in linux-2.6.39.4.
> It uses the udp protocol and in "Documentation/filesystems/nfs/nfsroot.txt" there's no "proto" option (or something similar).
> [11140.866623] IP-Config: Complete:
> [11140.866692] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
> [11140.866895] host=cluster2, domain=, nis-domain=(none),
> [11140.867008] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
> [11140.867431] Root-NFS: nfsroot=/srv/nfs/cluster2,proto=tcp,retrans=10,timeo=60
> [11140.867728] NFS: nfs mount opts='udp,proto=tcp,retrans=10,timeo=60,nolock,addr=137.226.167.241'
> [11140.867797] NFS: parsing nfs mount option 'udp'
> [11140.867836] NFS: parsing nfs mount option 'proto=tcp'

"proto=tcp" worked; see below. NFS mount options are parsed left-to-right. "proto=tcp" overrides "udp", which is to its left.

> [11140.867895] NFS: parsing nfs mount option 'retrans=10'
> [11140.867946] NFS: parsing nfs mount option 'timeo=60'
> [11140.867996] NFS: parsing nfs mount option 'nolock'
> [11140.868043] NFS: parsing nfs mount option 'addr=137.226.167.241'
> [11140.868106] NFS: MNTPATH: '/srv/nfs/cluster2'
> [11140.868142] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
> [11141.912761] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
> [11141.933177] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [11143.873095] NFS: failed to create MNT RPC client, status=-65
> [11143.873205] NFS: unable to mount server 137.226.167.241, error -65

You got a different result: -65 is EHOSTUNREACH. And actually, I would have expected that error from the UDP case as well.

> [11143.873339] VFS: Unable to mount root fs via NFS, trying floppy.
> [11143.873494] VFS: Insert root floppy and press ENTER
>
> Now I'll try the other workaround idea: netboot from tftp (to bring eth0 up) and subsequent nfsroot mount.
> Therefore I need a customized (and rather small) kernel...
>
> Regards,
> Lukas

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com





2011-11-05 02:32:34

by Myklebust, Trond

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

On Sat, 2011-11-05 at 02:14 +0000, Lukas Razik wrote:
> ***IMPORTANT HINT***
>
> In the test belowI've tried to ping the NFS client (cluster2) as soon as I've got the
>
> ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> message.
>
>
> And at first I've got no answers from the client. But 10-20sec later I've got pongs back.
> So the NIC comes up some seconds later than the message "link becomes ready" is shown...
> In the case of 'mountproto=tcp' it comes up after the kernel panic. So that's much too late.
>

So, once again...

What kind of switch is this client connected to, and how is that switch
configured?

--
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
http://www.netapp.com


2011-11-04 15:18:41

by Myklebust, Trond

[permalink] [raw]
Subject: RE: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> -----Original Message-----
> From: Jim Rees [mailto:[email protected]]
> Sent: Friday, November 04, 2011 10:46 AM
> To: Chuck Lever
> Cc: Lukas Razik; Myklebust, Trond; Linux NFS Mailing List
> Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64
>
> Chuck Lever wrote:
>
> On Nov 4, 2011, at 9:20 AM, Jim Rees wrote:
> > As we move toward nfs4 someone will have to give some thought to
> nfsroot.
> > It's hard to imagine we could put enough nfs4 cruft into the
kernel (gssd,
> > idmapd) to make it work.
>
> A kernel-level basic id mapper is being considered. That would
allow
> NFSv4 with AUTH_SYS, if we can get the NIC problems squared away.
>
> Actually I wonder if you could get by with auth_sys, no gss, and no id
> mapping until you get to the point where the root is remounted by user
land.

Please read RFC3530-bis. This is a solved problem.

2011-11-04 01:06:45

by Chuck Lever III

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64


On Nov 3, 2011, at 7:09 PM, Lukas Razik wrote:

>
>
> Chuck Lever <[email protected]> wrote:
>>
>>
>> On Nov 3, 2011, at 5:37 PM, Lukas Razik wrote:
>>
>>>> On Nov 3, 2011, at 5:11 PM, Jim Rees wrote:
>>>
>>>>
>>>>> Trond Myklebust wrote:
>>>>>
>>>>>> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
>>>>>> [ 442.666732] NFS: unable to mount server 137.226.167.241,
>> error -60
>>>>>> [ 442.666868] VFS: Unable to mount root fs via NFS, trying
>> floppy.
>>>>>> [ 442.667032] VFS: Insert root floppy and press ENTER
>>>>>>
>>>>> Error 60 is ETIMEDOUT on SPARC, so it seems that the problem is
>>>>> basically the same one that you see in your 2.6.32 trace
>> (rpcbind:
>>>>> server 137.226.167.241 not responding, timed out) except that now
>> it is
>>>>> a fatal error.
>>>>>
>>>>> Any idea why the first RPC calls might be failing here? A switch
>>>>> misconfiguration or something like that perhaps?
>>>>>
>>>>> Wasn't there a change in the way nfs mount options are handled
>> by the
>>>> kernel
>>>>> for nfsroot about the time of 2.6.39? Something about changing
>> from
>>>> default
>>>>> udp to tcp maybe?
>>>>
>>>> There was a change, but it was changed back to UDP because of problems
>> like
>>>> this. Behavior in 3.0 or the latest 2.6.39 stable kernel may be
>> improved.
>>>>
>>>
>>> I don't know if this was a tip to test newest 2.6.39 but as I wrote in
>> my first email
>>> http://thread.gmane.org/gmane.linux.nfs/44596
>>> that's the output of linux-2.6.39.4 with "nfsdebug":
>>>
>>> [ 407.571521] IP-Config: Complete:
>>> [ 407.571589] device=eth0, addr=137.226.167.242, mask=255.255.255.224,
>> gw=137.226.167.225,
>>> [ 407.571793] host=cluster2, domain=, nis-domain=(none),
>>> [ 407.571907] bootserver=255.255.255.255, rootserver=137.226.167.241,
>> rootpath=
>>> [ 407.572332] Root-NFS: nfsroot=/srv/nfs/cluster2
>>> [ 407.572726] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
>>> [ 407.572927] NFS: parsing nfs mount option 'udp'
>>> [ 407.572995] NFS: parsing nfs mount option 'nolock'
>>> [ 407.573071] NFS: parsing nfs mount option 'addr=137.226.167.241'
>>> [ 407.573139] NFS: MNTPATH: '/srv/nfs/cluster2'
>>> [ 407.573203] NFS: sending MNT request for
>> 137.226.167.241:/srv/nfs/cluster2
>>> [ 408.617894] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
>> Control: Rx
>>> [ 408.638319] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>>> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
>>> [ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
>>> [ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
>>> [ 442.667032] VFS: Insert root floppy and press ENTER
>>>
>>> And this behaviour is exactly the same as in all other 2.6.37 - 2.6.39.4
>> which I've tested.
>>> So if anybody of you all have an idea what I could try to do, I'll
>> follow...
>>
>> Find out why the very first RPC on your system always fails. As Trond says, the
>> only reason this worked on the older kernels is because NFSROOT fell back to a
>> default port for NFSD. This is also broken behavior, but in your case it
>> happened to work so you never noticed it.
>>
>> I seem to recall there's a way to set the NFS and RPC debugging flags on the
>> kernel command line so more information can be captured during boot. But I
>> don't see it under Documentation/.
>>
>> You could add a line in fs/nfs/nfsroot.c:nfs_root_debug() to set flags also in
>> the rpc_debug global variable to gather more information.
>>
>
> OK
> I've watched wireshark on cluster1 during start up of cluster2 (with linux-2.6.32) which first tries 10003 and then 10005.
> The result is that cluster1 doesn't get a datagram for port 10003:
> http://net.razik.de/linux/T5120/cluster2_NFSROOT_MOUNT.png
>
> The first ARP request in the screenshot came _after_ the <tag> in this kernel log:
> [ 6492.807917] IP-Config: Complete:
> [ 6492.807978] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
> [ 6492.808227] host=cluster2, domain=, nis-domain=(none),
> [ 6492.808312] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
> [ 6492.808570] Looking up port of RPC 100003/2 on 137.226.167.241
> [ 6493.886014] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
> [ 6493.905840] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> <tag>
> [ 6527.827055] rpcbind: server 137.226.167.241 not responding, timed out
> [ 6527.827237] Root-NFS: Unable to get nfsd port number from server, using default
> [ 6527.827353] Looking up port of RPC 100005/1 on 137.226.167.241
> [ 6527.842212] VFS: Mounted root (nfs filesystem) on device 0:15.
>
>
> So I don't think that it's a problem of the hardware between the machines.
> There's no reason why I wouldn't see an ARP requests from cluster2 which would have been sent _before_ the <tag> if there would be one. I think: cluster2 never sends a request to port 10003.
> What do you think?

It agrees with our initial assessment that the first RPC request is failing. The RPC client never gets the request through cluster2's network stack because the NIC hasn't re-initialized when the request is sent.

It looks like your system does a PXE boot, which provides the IP configuration shown above. But then the kernel resets the NIC. During that reset, the kernel is attempting to contact the NFS server to mount the root file system.

We've set up NFSROOT to use UDP so that it will be relatively immune to these initialization order problems. The RPC client should be retrying the lost request, but apparently it isn't. What if you added "retrans=10" to cluster2's mount options? (on the chance that mount option setting would be copied to the rpcbind client's RPC transport...)

IMO the correct way to fix this is to provide proper serialization in the networking layer so that RPC requests are not even attempted until the NIC is ready to carry traffic. That may be a pipe dream though.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com





2011-11-03 21:16:55

by Chuck Lever III

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64


On Nov 3, 2011, at 5:11 PM, Jim Rees wrote:

> Trond Myklebust wrote:
>
>> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
>> [ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
>> [ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
>> [ 442.667032] VFS: Insert root floppy and press ENTER
>>
> Error 60 is ETIMEDOUT on SPARC, so it seems that the problem is
> basically the same one that you see in your 2.6.32 trace (rpcbind:
> server 137.226.167.241 not responding, timed out) except that now it is
> a fatal error.
>
> Any idea why the first RPC calls might be failing here? A switch
> misconfiguration or something like that perhaps?
>
> Wasn't there a change in the way nfs mount options are handled by the kernel
> for nfsroot about the time of 2.6.39? Something about changing from default
> udp to tcp maybe?

There was a change, but it was changed back to UDP because of problems like this. Behavior in 3.0 or the latest 2.6.39 stable kernel may be improved.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com





2011-11-03 23:59:29

by Jim Rees

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

I suspect some kind of timing problem, where the network interface hasn't
come up yet at the time the portmap request is made. What does your nfsroot
kernel option look like? What happens if you add ",port=2049" to the end of
your nfsroot kernel option?

2011-11-03 21:58:58

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> Lukas Razik wrote:

>
> ? The next thing is:
> ? Really all working kernels (<=2.6.36.4) first output
> ? ?Looking up port of RPC 100003/2 on 137.226.167.241
> ? then
> ? ?Looking up port of RPC 100005/1 on 137.226.167.241
> ? and then the mount is successful
> ? ?VFS: Mounted root (nfs filesystem) on device 0:15.
> ?
> ? So what about >=2.6.37?
> ? Why don't these kernels try other ports, too?
> ? Or why do the old kernels try more than one port?
> ? Why is there no output (even in the nfsdebug mode) that the kernel tries to
> connect to the RPC service?
> ? Is there a "easy" possibility to change port 100003 to 100005 in
>> =2.6.37?
>
> Those are the rpc numbers.? The kernel is trying to find the port numbers
> for those services.? 100003 is nfs, 100005 is mount.
> I guess at this point
> I would use wireshark to find out what requests are actually being made and
> responded to in both cases.? Are you sure portmap is working on the server
> and not being blocked by a firewall?
>

When I boot from hard disk and try to mount the same exports from cluster1 on cluster2 by
# mount -t nfs cluster1:[...]
then it works immediately and without errors.

There's no firewall. But I've done a test to check it:
---
root@cluster2:~# telnet cluster1 111
Trying 137.226.167.241...
Connected to 137.226.167.241.
Escape character is '^]'.
TEST
TEST
Connection closed by foreign host.
---
So it's no problem to connect to the portmapper of cluster1 from cluster2.

2011-11-04 16:56:29

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> If you boot via tftp, I bet this problem will go away because the network

> interface will be working by the time the NFSROOT mount is attempted.
>
> The NFSROOT code assumes that if kernel IP configuration worked, then the NIC is
> already up.? That is clearly not the case if you boot from your local disk.
>

This also doesn't work. The kernel doesn't recognize that the eth0 is already up by OpenBoot and tries to do that.
Hence the result is the same as before (see end of my email, please).
---
BTW:
OpenBoot gets the IP configuration for eth0 from a rarp server when I enter "boot net".
As soon as it has the IP (always 6 requests needed), it tries to download the kernel from a tftp server.
After the kernel is downloaded, it's started by OpenBoot with the same kernel line I used before (in silo.conf).
---

> As for a patch: no-one can write a patch unless we understand precisely why the
> first RPC fails.? I already explained how to add a line or two to
> fs/nfs/nfsroot.c to give us more information.? If you need a patch to do this, I
> can send one later today.

So now I'll try to add the lines you've mentioned in an older email...
But if you're faster, I'm also happy about a patch. Because you know better what you want to see...

Regards,
Lukas

PS:
That's the whole OpenBoot and linux-2.6.39.4 kernel log:
---
SPARC Enterprise T5120, No Keyboard
Copyright 2008 Sun Microsystems, Inc.? All rights reserved.
OpenBoot 4.30.0, 32640 MB memory available, Serial #77072834.
Ethernet address 0:14:4f:98:9:c2, Host ID: 849809c2.

{0} ok boot net
Boot device: /pci@0/pci@0/pci@1/pci@0/pci@2/network@0? File and args: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242::137.226.167.225:255.255.255.224:cluster2:eth0:off
1000 Mbps full duplex? Link up
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
/
[??? 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
[??? 0.000000] PROMLIB: Root node compatible: sun4v
[??? 0.000000] Initializing cgroup subsys cpuset
[??? 0.000000] Initializing cgroup subsys cpu
[??? 0.000000] Linux version 2.6.39.4-razik-2011-11-04 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #7 SMP Fri Nov 4 17:25:22 CET 2011
[??? 0.000000] bootconsole [earlyprom0] enabled
[??? 0.000000] ARCH: SUN4V
[??? 0.000000] Ethernet address: 00:14:4f:98:09:c2
[??? 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
[??? 0.000000] Remapping the kernel... done.
[??? 0.000000] OF stdout device is: /virtual-devices@100/console@1
[??? 0.000000] PROM: Built device tree with 123783 bytes of memory.
[??? 0.000000] MDESC: Size is 58832 bytes.
[??? 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
[??? 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
[??? 0.000000] PLATFORM: hostid [849809c2]
[??? 0.000000] PLATFORM: serial# [00ab4130]
[??? 0.000000] PLATFORM: stick-frequency [5458c3a0]
[??? 0.000000] PLATFORM: mac-address [144f9809c2]
[??? 0.000000] PLATFORM: watchdog-resolution [1000 ms]
[??? 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
[??? 0.000000] PLATFORM: max-cpus [64]
[??? 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c2000
[??? 0.000000] Memory hole size: 132MB
[??? 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
[??? 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
[??? 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
[??? 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
[??? 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
[??? 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
[??? 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
[??? 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
[??? 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
[??? 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
[??? 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
[??? 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
[??? 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
[??? 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
[??? 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
[??? 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
[??? 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
[??? 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
[??? 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
[??? 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
[??? 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
[??? 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
[??? 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
[??? 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
[??? 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
[??? 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
[??? 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
[??? 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
[??? 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
[??? 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
[??? 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
[??? 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
[??? 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
[??? 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
[??? 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
[??? 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
[??? 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
[??? 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
[??? 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
[??? 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
[??? 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
[??? 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
[??? 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
[??? 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
[??? 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
[??? 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
[??? 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
[??? 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
[??? 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
[??? 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
[??? 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
[??? 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
[??? 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
[??? 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
[??? 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
[??? 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
[??? 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
[??? 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
[??? 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
[??? 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
[??? 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
[??? 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
[??? 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
[??? 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
[??? 0.000000] Zone PFN ranges:
[??? 0.000000]?? Normal?? 0x00004200 -> 0x003ffd8e
[??? 0.000000] Movable zone start PFN for each node
[??? 0.000000] early_node_map[3] active PFN ranges
[??? 0.000000]???? 0: 0x00004200 -> 0x003ffd47
[??? 0.000000]???? 0: 0x003ffd50 -> 0x003ffd63
[??? 0.000000]???? 0: 0x003ffd87 -> 0x003ffd8e
[??? 0.000000] Booting Linux...
[??? 0.000000] PERCPU: Embedded 6 pages/cpu @fffff80019000000 s15616 r8192 d25344 u65536
[??? 0.000000] Built 1 zonelists in Zone order, mobility grouping on.? Total pages: 4144105
[??? 0.000000] Kernel command line: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242::137.226.167.225:255.255.255.224:cluster2:eth0:off
[??? 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
[??? 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
[??? 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
[??? 0.000000] Memory: 33091488k available (3528k kernel code, 1248k data, 208k init) [fffff80000000000,00000007ffb1c000]
[??? 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[??? 0.000000] Hierarchical RCU implementation.
[??? 0.000000]? CONFIG_RCU_FANOUT set to non-default value of 32
[??? 0.000000]? RCU-based detection of stalled CPUs is disabled.
[??? 0.000000] NR_IRQS:255
[??? 0.000000] clocksource: mult[5a73e7bb] shift[31]
[??? 0.000000] clockevent: mult[b5221ba3] shift[31]
[??? 0.000000] Console: colour dummy device 80x25
[??? 0.000000] console [tty0] enabled, bootconsole disabled
[??? 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
[??? 0.000000] PROMLIB: Root node compatible: sun4v
[??? 0.000000] Initializing cgroup subsys cpuset
[??? 0.000000] Initializing cgroup subsys cpu
[??? 0.000000] Linux version 2.6.39.4-razik-2011-11-04 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #7 SMP Fri Nov 4 17:25:22 CET 2011
[??? 0.000000] bootconsole [earlyprom0] enabled
[??? 0.000000] ARCH: SUN4V
[??? 0.000000] Ethernet address: 00:14:4f:98:09:c2
[??? 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
[??? 0.000000] Remapping the kernel... done.
[??? 0.000000] OF stdout device is: /virtual-devices@100/console@1
[??? 0.000000] PROM: Built device tree with 123783 bytes of memory.
[??? 0.000000] MDESC: Size is 58832 bytes.
[??? 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
[??? 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
[??? 0.000000] PLATFORM: hostid [849809c2]
[??? 0.000000] PLATFORM: serial# [00ab4130]
[??? 0.000000] PLATFORM: stick-frequency [5458c3a0]
[??? 0.000000] PLATFORM: mac-address [144f9809c2]
[??? 0.000000] PLATFORM: watchdog-resolution [1000 ms]
[??? 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
[??? 0.000000] PLATFORM: max-cpus [64]
[??? 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c2000
[??? 0.000000] Memory hole size: 132MB
[??? 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
[??? 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
[??? 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
[??? 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
[??? 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
[??? 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
[??? 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
[??? 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
[??? 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
[??? 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
[??? 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
[??? 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
[??? 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
[??? 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
[??? 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
[??? 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
[??? 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
[??? 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
[??? 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
[??? 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
[??? 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
[??? 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
[??? 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
[??? 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
[??? 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
[??? 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
[??? 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
[??? 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
[??? 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
[??? 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
[??? 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
[??? 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
[??? 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
[??? 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
[??? 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
[??? 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
[??? 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
[??? 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
[??? 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
[??? 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
[??? 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
[??? 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
[??? 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
[??? 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
[??? 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
[??? 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
[??? 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
[??? 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
[??? 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
[??? 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
[??? 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
[??? 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
[??? 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
[??? 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
[??? 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
[??? 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
[??? 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
[??? 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
[??? 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
[??? 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
[??? 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
[??? 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
[??? 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
[??? 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
[??? 0.000000] Zone PFN ranges:
[??? 0.000000]?? Normal?? 0x00004200 -> 0x003ffd8e
[??? 0.000000] Movable zone start PFN for each node
[??? 0.000000] early_node_map[3] active PFN ranges
[??? 0.000000]???? 0: 0x00004200 -> 0x003ffd47
[??? 0.000000]???? 0: 0x003ffd50 -> 0x003ffd63
[??? 0.000000]???? 0: 0x003ffd87 -> 0x003ffd8e
[??? 0.000000] Booting Linux...
[??? 0.000000] PERCPU: Embedded 6 pages/cpu @fffff80019000000 s15616 r8192 d25344 u65536
[??? 0.000000] Built 1 zonelists in Zone order, mobility grouping on.? Total pages: 4144105
[??? 0.000000] Kernel command line: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242::137.226.167.225:255.255.255.224:cluster2:eth0:off
[??? 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
[??? 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
[??? 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
[??? 0.000000] Memory: 33091488k available (3528k kernel code, 1248k data, 208k init) [fffff80000000000,00000007ffb1c000]
[??? 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[??? 0.000000] Hierarchical RCU implementation.
[??? 0.000000]? CONFIG_RCU_FANOUT set to non-default value of 32
[??? 0.000000]? RCU-based detection of stalled CPUs is disabled.
[??? 0.000000] NR_IRQS:255
[??? 0.000000] clocksource: mult[5a73e7bb] shift[31]
[??? 0.000000] clockevent: mult[b5221ba3] shift[31]
[??? 0.000000] Console: colour dummy device 80x25
[??? 0.000000] console [tty0] enabled, bootconsole disabled
[? 509.804744] Calibrating delay using timer specific routine.. 2832.71 BogoMIPS (lpj=5665433)
[? 509.804778] pid_max: default: 65536 minimum: 512
[? 509.805222] Mount-cache hash table entries: 512
[? 509.806382] Initializing cgroup subsys ns
[? 509.806402] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[? 509.806429] Initializing cgroup subsys cpuacct
[? 509.806536] Initializing cgroup subsys devices
[? 509.806554] Initializing cgroup subsys freezer
[? 509.806570] Initializing cgroup subsys net_cls
[? 509.806836] Performance events: Supported PMU type is 'niagara2'
[? 509.829343] Brought up 64 CPUs
[? 509.829440] Testing NMI watchdog ... OK.
[? 509.910785] devtmpfs: initialized
[? 509.911236] ldc.c:v1.1 (July 22, 2008)
[? 509.911508] print_constraints: dummy:
[? 509.911845] NET: Registered protocol family 16
[? 509.920950] VIO: Adding device channel-devices
[? 509.921027] VIO: Adding device vldc-port-3-0
[? 509.921089] VIO: Adding device vldc-port-3-1
[? 509.921149] VIO: Adding device vldc-port-3-2
[? 509.921209] VIO: Adding device vldc-port-3-3
[? 509.921279] VIO: Adding device vldc-port-3-4
[? 509.921346] VIO: Adding device vldc-port-3-5
[? 509.921415] VIO: Adding device vldc-port-2-0
[? 509.921482] VIO: Adding device vldc-port-0-0
[? 509.921548] VIO: Adding device vldc-port-0-1
[? 509.921615] VIO: Adding device vldc-port-0-2
[? 509.921692] VIO: Adding device vldc-port-1-0
[? 509.921764] VIO: Adding device vldc-port-3-7
[? 509.921836] VIO: Adding device vldc-port-3-8
[? 509.921918] VIO: Adding device ds-1
[? 509.921981] VIO: Adding device ds-0
[? 509.926530] pci_sun4v: Registered hvapi major[1] minor[0]
[? 509.926805] /pci@0: SUN4V PCI Bus Module
[? 509.926824] /pci@0: On NUMA node -1
[? 509.926844] /pci@0: PCI IO[c0f0000000] MEM[c100000000]
[? 509.926871] /pci@0: Unable to request IOMMU resource.
[? 509.971041] /pci@0: Imported 3 TSB entries from OBP
[? 509.972495] /pci@0: MSI Queue first[0] num[36] count[128] devino[0x18]
[? 509.972523] /pci@0: MSI first[0] num[256] mask[0xff] width[32]
[? 509.972547] /pci@0: MSI addr32[0x7fff0000:0x10000] addr64[0x3ffff0000:0x10000]
[? 509.972577] /pci@0: MSI queues at RA [00000007f8180000]
[? 509.972600] PCI: Scanning PBM /pci@0
[? 509.984309] ds.c:v1.0 (Jul 11, 2007)
[? 509.987359] bio: create slab <bio-0> at 0
[? 509.987974] vgaarb: loaded
[? 509.988772] Switching to clocksource stick
[? 509.989440] Switched to NOHz mode on CPU #41
[? 509.989554] Switched to NOHz mode on CPU #18
[? 509.989580] Switched to NOHz mode on CPU #6
[? 509.989616] Switched to NOHz mode on CPU #53
[? 509.989626] Switched to NOHz mode on CPU #30
[? 509.989759] Switched to NOHz mode on CPU #42
[? 509.989885] Switched to NOHz mode on CPU #19
[? 509.989911] Switched to NOHz mode on CPU #7
[? 509.989962] Switched to NOHz mode on CPU #31
[? 509.989971] Switched to NOHz mode on CPU #54
[? 509.990097] Switched to NOHz mode on CPU #43
[? 509.990216] Switched to NOHz mode on CPU #20
[? 509.990226] Switched to NOHz mode on CPU #8
[? 509.990306] Switched to NOHz mode on CPU #32
[? 509.990327] Switched to NOHz mode on CPU #55
[? 509.990444] Switched to NOHz mode on CPU #44
[? 509.990541] Switched to NOHz mode on CPU #9
[? 509.990557] Switched to NOHz mode on CPU #21
[? 509.990651] Switched to NOHz mode on CPU #33
[? 509.990671] Switched to NOHz mode on CPU #56
[? 509.990789] Switched to NOHz mode on CPU #45
[? 509.990876] Switched to NOHz mode on CPU #10
[? 509.990893] Switched to NOHz mode on CPU #22
[? 509.990985] Switched to NOHz mode on CPU #34
[? 509.991023] Switched to NOHz mode on CPU #57
[? 509.991136] Switched to NOHz mode on CPU #46
[? 509.991207] Switched to NOHz mode on CPU #11
[? 509.991239] Switched to NOHz mode on CPU #23
[? 509.991340] Switched to NOHz mode on CPU #35
[? 509.991367] Switched to NOHz mode on CPU #58
[? 509.991485] Switched to NOHz mode on CPU #47
[? 509.991536] Switched to NOHz mode on CPU #12
[? 509.991577] Switched to NOHz mode on CPU #24
[? 509.991675] Switched to NOHz mode on CPU #36
[? 509.991736] Switched to NOHz mode on CPU #59
[? 509.991835] Switched to NOHz mode on CPU #48
[? 509.991853] Switched to NOHz mode on CPU #1
[? 509.991863] Switched to NOHz mode on CPU #13
[? 509.991917] Switched to NOHz mode on CPU #25
[? 509.992033] Switched to NOHz mode on CPU #37
[? 509.992083] Switched to NOHz mode on CPU #60
[? 509.992188] Switched to NOHz mode on CPU #14
[? 509.992198] Switched to NOHz mode on CPU #49
[? 509.992254] Switched to NOHz mode on CPU #26
[? 509.992263] Switched to NOHz mode on CPU #2
[? 509.992369] Switched to NOHz mode on CPU #38
[? 509.992435] Switched to NOHz mode on CPU #61
[? 509.992527] Switched to NOHz mode on CPU #15
[? 509.992546] Switched to NOHz mode on CPU #50
[? 509.992579] Switched to NOHz mode on CPU #3
[? 509.992645] Switched to NOHz mode on CPU #0
[? 509.992655] Switched to NOHz mode on CPU #27
[? 509.992699] Switched to NOHz mode on CPU #39
[? 509.992787] Switched to NOHz mode on CPU #62
[? 509.992870] Switched to NOHz mode on CPU #16
[? 509.992893] Switched to NOHz mode on CPU #51
[? 509.992916] Switched to NOHz mode on CPU #4
[? 509.992933] Switched to NOHz mode on CPU #28
[? 509.993055] Switched to NOHz mode on CPU #40
[? 509.993180] Switched to NOHz mode on CPU #52
[? 509.993190] Switched to NOHz mode on CPU #29
[? 509.993199] Switched to NOHz mode on CPU #5
[? 509.993208] Switched to NOHz mode on CPU #17
[? 509.993218] Switched to NOHz mode on CPU #63
[? 509.997364] NET: Registered protocol family 2
[? 509.997817] IP route cache hash table entries: 524288 (order: 9, 4194304 bytes)
[? 509.999942] TCP established hash table entries: 524288 (order: 10, 8388608 bytes)
[? 510.006892] ds-1: Registered pri service.
[? 510.009264] ds-1: Registered var-config-backup service.
[? 510.025315] TCP bind hash table entries: 65536 (order: 7, 1048576 bytes)
[? 510.029037] TCP: Hash tables configured (established 524288 bind 65536)
[? 510.029062] TCP reno registered
[? 510.029106] UDP hash table entries: 16384 (order: 6, 524288 bytes)
[? 510.031128] UDP-Lite hash table entries: 16384 (order: 6, 524288 bytes)
[? 510.034532] NET: Registered protocol family 1
[? 510.035019] RPC: Registered udp transport module.
[? 510.035042] RPC: Registered tcp transport module.
[? 510.035063] RPC: Registered tcp NFSv4.1 backchannel transport module.
[? 510.036252] audit: initializing netlink socket (disabled)
[? 510.036298] type=2000 audit(0.304:1): initialized
[? 510.084013] HugeTLB registered 4 MB page size, pre-allocated 0 pages
[? 510.092493] VFS: Disk quotas dquot_6.5.2
[? 510.092950] Dquot-cache hash table entries: 1024 (order 0, 8192 bytes)
[? 510.094145] msgmni has been set to 32768
[? 510.095212] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[? 510.095245] io scheduler noop registered
[? 510.095264] io scheduler deadline registered
[? 510.095494] io scheduler cfq registered (default)
[? 510.096783] f027ab74: ttyS0 at I/O 0x0 (irq = 17) is a SUN4V HCONS
[? 510.155390] console [ttyHV0] enabled
[? 510.156988] f028af14: ttyS1 at MMIO 0xfff0ca0000 (irq = 24) is a 16550A
[? 510.157601] [drm] Initialized drm 1.1.0 20060810
[? 510.158744] e1000e: Intel(R) PRO/1000 Network Driver - 1.3.10-k2
[? 510.186832] e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
[? 510.186933] e1000e 0000:08:00.0: Disabling ASPM? L1
[? 510.364001] e1000e 0000:08:00.0: eth0: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c2
[? 510.364122] e1000e 0000:08:00.0: eth0: Intel(R) PRO/1000 Network Connection
[? 510.364306] e1000e 0000:08:00.0: eth0: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 510.364444] e1000e 0000:08:00.1: Disabling ASPM? L1
[? 510.545790] e1000e 0000:08:00.1: eth1: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c3
[? 510.545910] e1000e 0000:08:00.1: eth1: Intel(R) PRO/1000 Network Connection
[? 510.546093] e1000e 0000:08:00.1: eth1: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 510.546219] e1000e 0000:09:00.0: Disabling ASPM? L1
[? 510.724130] e1000e 0000:09:00.0: eth2: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c4
[? 510.724251] e1000e 0000:09:00.0: eth2: Intel(R) PRO/1000 Network Connection
[? 510.724434] e1000e 0000:09:00.0: eth2: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 510.724559] e1000e 0000:09:00.1: Disabling ASPM? L1
[? 510.905999] e1000e 0000:09:00.1: eth3: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c5
[? 510.906120] e1000e 0000:09:00.1: eth3: Intel(R) PRO/1000 Network Connection
[? 510.906303] e1000e 0000:09:00.1: eth3: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 510.906988] mousedev: PS/2 mouse device common for all mice
[? 510.907998] rtc-sun4v rtc-sun4v: rtc core: registered sun4v as rtc0
[? 510.908670] TCP cubic registered
[? 510.909540] NET: Registered protocol family 10
[? 510.914368] Mobile IPv6
[? 510.914442] NET: Registered protocol family 17
[? 510.914548] Registering the dns_resolver key type
[? 510.915076] registered taskstats version 1
[? 510.918076] rtc-sun4v rtc-sun4v: setting system clock to 2011-11-04 16:49:01 UTC (1320425341)
[? 511.667422] ADDRCONF(NETDEV_UP): eth0: link is not ready
[? 512.689540] IP-Config: Complete:
[? 512.689612]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
[? 512.689815]????? host=cluster2, domain=, nis-domain=(none),
[? 512.689927]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[? 513.772653] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[? 513.792638] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

2011-11-03 23:09:27

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64



Chuck Lever <[email protected]> wrote:
>
>
> On Nov 3, 2011, at 5:37 PM, Lukas Razik wrote:
>
>>> On Nov 3, 2011, at 5:11 PM, Jim Rees wrote:
>>
>>>
>>>> Trond Myklebust wrote:
>>>>
>>>>> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
>>>>> [ 442.666732] NFS: unable to mount server 137.226.167.241,
> error -60
>>>>> [ 442.666868] VFS: Unable to mount root fs via NFS, trying
> floppy.
>>>>> [ 442.667032] VFS: Insert root floppy and press ENTER
>>>>>
>>>> ? Error 60 is ETIMEDOUT on SPARC, so it seems that the problem is
>>>> ? basically the same one that you see in your 2.6.32 trace
> (rpcbind:
>>>> ? server 137.226.167.241 not responding, timed out) except that now
> it is
>>>> ? a fatal error.
>>>>
>>>> ? Any idea why the first RPC calls might be failing here? A switch
>>>> ? misconfiguration or something like that perhaps?
>>>>
>>>> Wasn't there a change in the way nfs mount options are handled
> by the
>>> kernel
>>>> for nfsroot about the time of 2.6.39?? Something about changing
> from
>>> default
>>>> udp to tcp maybe?
>>>
>>> There was a change, but it was changed back to UDP because of problems
> like
>>> this.? Behavior in 3.0 or the latest 2.6.39 stable kernel may be
> improved.
>>>
>>
>> I don't know if this was a tip to test newest 2.6.39 but as I wrote in
> my first email
>> ? http://thread.gmane.org/gmane.linux.nfs/44596
>> that's the output of linux-2.6.39.4 with "nfsdebug":
>>
>> [ 407.571521] IP-Config: Complete:
>> [ 407.571589] device=eth0, addr=137.226.167.242, mask=255.255.255.224,
> gw=137.226.167.225,
>> [ 407.571793] host=cluster2, domain=, nis-domain=(none),
>> [ 407.571907] bootserver=255.255.255.255, rootserver=137.226.167.241,
> rootpath=
>> [ 407.572332] Root-NFS: nfsroot=/srv/nfs/cluster2
>> [ 407.572726] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
>> [ 407.572927] NFS: parsing nfs mount option 'udp'
>> [ 407.572995] NFS: parsing nfs mount option 'nolock'
>> [ 407.573071] NFS: parsing nfs mount option 'addr=137.226.167.241'
>> [ 407.573139] NFS: MNTPATH: '/srv/nfs/cluster2'
>> [ 407.573203] NFS: sending MNT request for
> 137.226.167.241:/srv/nfs/cluster2
>> [ 408.617894] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: Rx
>> [ 408.638319] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
>> [ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
>> [ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
>> [ 442.667032] VFS: Insert root floppy and press ENTER
>>
>> And this behaviour is exactly the same as in all other 2.6.37 - 2.6.39.4
> which I've tested.
>> So if anybody of you all have an idea what I could try to do, I'll
> follow...
>
> Find out why the very first RPC on your system always fails.? As Trond says, the
> only reason this worked on the older kernels is because NFSROOT fell back to a
> default port for NFSD.? This is also broken behavior, but in your case it
> happened to work so you never noticed it.
>
> I seem to recall there's a way to set the NFS and RPC debugging flags on the
> kernel command line so more information can be captured during boot.? But I
> don't see it under Documentation/.
>
> You could add a line in fs/nfs/nfsroot.c:nfs_root_debug() to set flags also in
> the rpc_debug global variable to gather more information.
>

OK
I've watched wireshark on cluster1 during start up of cluster2 (with linux-2.6.32) which first tries 10003 and then 10005.
The result is that cluster1 doesn't get a datagram for port 10003:
http://net.razik.de/linux/T5120/cluster2_NFSROOT_MOUNT.png

The first ARP request in the screenshot came _after_ the <tag> in this kernel log:
[ 6492.807917] IP-Config: Complete:
[ 6492.807978]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
[ 6492.808227]????? host=cluster2, domain=, nis-domain=(none),
[ 6492.808312]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[ 6492.808570] Looking up port of RPC 100003/2 on 137.226.167.241
[ 6493.886014] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[ 6493.905840] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
<tag>
[ 6527.827055] rpcbind: server 137.226.167.241 not responding, timed out
[ 6527.827237] Root-NFS: Unable to get nfsd port number from server, using default
[ 6527.827353] Looking up port of RPC 100005/1 on 137.226.167.241
[ 6527.842212] VFS: Mounted root (nfs filesystem) on device 0:15.


So I don't think that it's a problem of the hardware between the machines.
There's no reason why I wouldn't see an ARP requests from cluster2 which would have been sent _before_ the <tag> if there would be one. I think: cluster2 never sends a request to port 10003.
What do you think?

2011-11-14 15:31:07

by Chuck Lever III

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

Yes, that's what I intended. Thanks for testing the idea.

Sent from my iPad

On Nov 13, 2011, at 5:19 PM, Lukas Razik <[email protected]> wrote:

>> [ 1764.189975] ADDRCONF(NETDEV_UP): eth0: link is not ready
>
>> [ 1765.212110] IP-Config: Complete:
>> [ 1765.212180] device=eth0, addr=137.226.167.242, mask=255.255.255.224,
>> gw=137.226.167.225,
>> [ 1765.212383] host=137.226.167.242, domain=, nis-domain=(none),
>> [ 1765.212498] bootserver=255.255.255.255, rootserver=137.226.167.241,
>> rootpath=
>> [ 1765.212943] VFS: 1. attempt to mount root fs via NFS.
>> [ 1765.212992] Root-NFS: nfsroot=/srv/nfs/cluster2
>> [ 1765.213265] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
>> [ 1765.213315] NFS: parsing nfs mount option 'udp'
>> [ 1765.213353] NFS: parsing nfs mount option 'nolock'
>> [ 1765.213399] NFS: parsing nfs mount option 'addr=137.226.167.241'
>> [ 1765.213467] NFS: MNTPATH: '/srv/nfs/cluster2'
>> [ 1765.213501] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
>> [ 1766.298480] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control:
>> Rx
>> [ 1766.318949] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 1800.284325] NFS: failed to create MNT RPC client, status=-60
>> [ 1800.284435] NFS: unable to mount server 137.226.167.241, error -60
>> [ 1800.284571] VFS: 2. attempt to mount root fs via NFS.
>> [ 1800.284682] Root-NFS: nfsroot=/srv/nfs/cluster2
>> [ 1800.285051] NFS: nfs mount
>> opts='udp,nolock,addr=137.226.167.241,nolock,addr=137.226.167.241'
>> [ 1800.285167] NFS: parsing nfs mount option 'udp'
>> [ 1800.285233] NFS: parsing nfs mount option 'nolock'
>> [ 1800.285337] NFS: parsing nfs mount option 'addr=137.226.167.241'
>> [ 1800.285455] NFS: parsing nfs mount option 'nolock'
>> [ 1800.285531] NFS: parsing nfs mount option 'addr=137.226.167.241'
>
> Chuck, I've seen that it's wrong to wrap mount_nfs_root() with while() because then we get a multiple occurrence of some mount options. Hence I've edited mount_nfs_root() itself so that the options will be set only once.
> Maybe that's better?
> --- linux-2.6.39.4/init/do_mounts.c 2011-08-03 21:43:28.000000000 +0200
> +++ linux-2.6.39.4-fix/init/do_mounts.c 2011-11-13 23:05:13.000000000 +0100
> @@ -363,12 +363,18 @@
> static int __init mount_nfs_root(void)
> {
> char *root_dev, *root_data;
> + char attempt;
>
> if (nfs_root_data(&root_dev, &root_data) != 0)
> return 0;
> - if (do_mount_root(root_dev, "nfs", root_mountflags, root_data) != 0)
> - return 0;
> - return 1;
> +
> + for(attempt = 1; attempt <= 3; attempt++) {
> + printk(KERN_INFO "VFS: %d. attempt to mount root fs via NFS.\n", attempt);
> + if (do_mount_root(root_dev, "nfs", root_mountflags, root_data) == 0)
> + return 1;
> + }
> +
> + return 0;
> }
> #endif
>
> --
> Regards,
> Lukas
>
>
> PS: That's the new behaviour:
> [ 4416.829621] ADDRCONF(NETDEV_UP): eth0: link is not ready
> [ 4417.851657] IP-Config: Complete:
> [ 4417.851729] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
> [ 4417.851931] host=137.226.167.242, domain=, nis-domain=(none),
> [ 4417.852045] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
> [ 4417.852493] Root-NFS: nfsroot=/srv/nfs/cluster2
> [ 4417.852539] VFS: 1. attempt to mount root fs via NFS.
> [ 4417.852814] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
> [ 4417.852893] NFS: parsing nfs mount option 'udp'
> [ 4417.852930] NFS: parsing nfs mount option 'nolock'
> [ 4417.852977] NFS: parsing nfs mount option 'addr=137.226.167.241'
> [ 4417.853073] NFS: MNTPATH: '/srv/nfs/cluster2'
> [ 4417.853137] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
> [ 4418.933894] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
> [ 4418.954252] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 4452.967331] NFS: failed to create MNT RPC client, status=-60
> [ 4452.967441] NFS: unable to mount server 137.226.167.241, error -60
> [ 4452.967577] VFS: 2. attempt to mount root fs via NFS.
> [ 4452.967978] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
> [ 4452.968085] NFS: parsing nfs mount option 'udp'
> [ 4452.968150] NFS: parsing nfs mount option 'nolock'
> [ 4452.968197] NFS: parsing nfs mount option 'addr=137.226.167.241'
> [ 4452.968333] NFS: MNTPATH: '/srv/nfs/cluster2'
> [ 4452.968397] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
> [ 4452.974480] NFS: received 1 auth flavors
> [ 4452.974543] NFS: auth flavor[0]: 1
> [ 4452.974723] NFS: MNT request succeeded
> [ 4452.974785] NFS: using auth flavor 1
> [ 4452.979498] VFS: Mounted root (nfs filesystem) on device 0:13.
> INIT: version 2.88 booting

2011-11-13 21:29:01

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

Chuck Lever <[email protected]>:

> This changes the behavior of normal user space NFS mounts as well.? User space
> mounts are already retried by the mount.nfs program.? We risk making the
> mount(2) system call hang by retrying here.
>
> I think we prefer a solution that changes the behavior of only NFSROOT mounts.?
> Have you tried wrapping mount_root_nfs() as I suggested?
>

No, because there's no "mount_root_nfs()" - hence I thought you've meant an older kernel and tried a work around - sorry for that.
Now I think you meant "mount_nfs_root()" and wrote the wrapper you've mentioned:
---
[ 1764.189975] ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1765.212110] IP-Config: Complete:
[ 1765.212180]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
[ 1765.212383]????? host=137.226.167.242, domain=, nis-domain=(none),
[ 1765.212498]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[ 1765.212943] VFS: 1. attempt to mount root fs via NFS.
[ 1765.212992] Root-NFS: nfsroot=/srv/nfs/cluster2
[ 1765.213265] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
[ 1765.213315] NFS:?? parsing nfs mount option 'udp'
[ 1765.213353] NFS:?? parsing nfs mount option 'nolock'
[ 1765.213399] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[ 1765.213467] NFS: MNTPATH: '/srv/nfs/cluster2'
[ 1765.213501] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[ 1766.298480] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[ 1766.318949] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1800.284325] NFS: failed to create MNT RPC client, status=-60
[ 1800.284435] NFS: unable to mount server 137.226.167.241, error -60
[ 1800.284571] VFS: 2. attempt to mount root fs via NFS.
[ 1800.284682] Root-NFS: nfsroot=/srv/nfs/cluster2
[ 1800.285051] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241,nolock,addr=137.226.167.241'
[ 1800.285167] NFS:?? parsing nfs mount option 'udp'
[ 1800.285233] NFS:?? parsing nfs mount option 'nolock'
[ 1800.285337] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[ 1800.285455] NFS:?? parsing nfs mount option 'nolock'
[ 1800.285531] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[ 1800.285635] NFS: MNTPATH: '/srv/nfs/cluster2'
[ 1800.285699] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[ 1800.290635] NFS: received 1 auth flavors
[ 1800.290697] NFS:?? auth flavor[0]: 1
[ 1800.290806] NFS: MNT request succeeded
[ 1800.290867] NFS: using auth flavor 1
[ 1800.295352] VFS: Mounted root (nfs filesystem) on device 0:13.
INIT: version 2.88 booting
---

I've used no while(true) because VFS tries to mount from floppy if NFS wasn't successful.
That's the patch:
--- linux-2.6.39.4/init/do_mounts.c??? 2011-08-03 21:43:28.000000000 +0200
+++ linux-2.6.39.4-fix/init/do_mounts.c??? 2011-11-13 22:22:29.000000000 +0100
@@ -405,9 +405,14 @@
?void __init mount_root(void)
?{
?#ifdef CONFIG_ROOT_NFS
+??? char attempt;
+
???? if (MAJOR(ROOT_DEV) == UNNAMED_MAJOR) {
-??? ??? if (mount_nfs_root())
-??? ??? ??? return;
+??? ??? for(attempt = 1; attempt <= 3; attempt++) {
+??? ??? ??? printk(KERN_INFO "VFS: %d. attempt to mount root fs via NFS.\n", attempt);
+??? ??? ??? if (mount_nfs_root())
+??? ??? ??? ??? return;
+??? ??? }
?
???? ??? printk(KERN_ERR "VFS: Unable to mount root fs via NFS, trying floppy.\n");
???? ??? ROOT_DEV = Root_FD0;
--
Regards,
Lukas

2011-11-04 14:46:25

by Jim Rees

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

Chuck Lever wrote:

On Nov 4, 2011, at 9:20 AM, Jim Rees wrote:

> As for a fix... we're trying to move away from udp transport anyway. Maybe
> someone should figure out a way to get it to work with tcp? I have zero
> experience with nfs over udp, at least on linux. Just for fun, have you
> tried tcp transport (proto=tcp)?

TCP is a real problem in this environment, because it deals poorly with
NIC initialization timing issues. UDP is still the best approach (as long
as it is retransmitting appropriately). To support TCP, ultimately what
we need to do is to introduce serialization to make the kernel wait for
the NIC to become ready before attempting network activity.

Agreed, but we're grasping at straws here, and this guy just wants it to
work. It's something to try.

> As we move toward nfs4 someone will have to give some thought to nfsroot.
> It's hard to imagine we could put enough nfs4 cruft into the kernel (gssd,
> idmapd) to make it work.

A kernel-level basic id mapper is being considered. That would allow
NFSv4 with AUTH_SYS, if we can get the NIC problems squared away.

Actually I wonder if you could get by with auth_sys, no gss, and no id
mapping until you get to the point where the root is remounted by user land.

2011-11-05 01:19:42

by Myklebust, Trond

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

On Fri, 2011-11-04 at 23:40 +0000, Lukas Razik wrote:
> > As for a patch: no-one can write a patch unless we understand precisely why the
>
> > first RPC fails. I already explained how to add a line or two to
> > fs/nfs/nfsroot.c to give us more information. If you need a patch to do this, I
> > can send one later today.
>
>
> Maybe that's also helpful for you because eth0 seems to be up before NFS tries to mount the NFSROOT.
> BTW: Thanks for forwarding to the right mailing list!
>
> The debug output (with NFSDBG_ALL and RPCDBG_ALL) of linux-3.1:
> ---
> {0} ok boot net
> Boot device: /pci@0/pci@0/pci@1/pci@0/pci@2/network@0 File and args: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
> 1000 Mbps full duplex Link up
> Requesting Internet Address for 0:14:4f:98:9:c2
> Requesting Internet Address for 0:14:4f:98:9:c2
> Requesting Internet Address for 0:14:4f:98:9:c2
> Requesting Internet Address for 0:14:4f:98:9:c2
> Requesting Internet Address for 0:14:4f:98:9:c2
> Requesting Internet Address for 0:14:4f:98:9:c2
> [ 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
> [ 0.000000] PROMLIB: Root node compatible: sun4v
> [ 0.000000] Initializing cgroup subsys cpuset
> [ 0.000000] Initializing cgroup subsys cpu
> [ 0.000000] Linux version 3.1.0-razik-2011-11-05 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #3 SMP Sat Nov 5 00:24:24 CET 2011
> [ 0.000000] bootconsole [earlyprom0] enabled
> [ 0.000000] ARCH: SUN4V
> [ 0.000000] Ethernet address: 00:14:4f:98:09:c2
> [ 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
> [ 0.000000] Remapping the kernel... done.
> [ 0.000000] OF stdout device is: /virtual-devices@100/console@1
> [ 0.000000] PROM: Built device tree with 123763 bytes of memory.
> [ 0.000000] MDESC: Size is 58816 bytes.
> [ 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
> [ 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
> [ 0.000000] PLATFORM: hostid [849809c2]
> [ 0.000000] PLATFORM: serial# [00ab4130]
> [ 0.000000] PLATFORM: stick-frequency [5458c3a0]
> [ 0.000000] PLATFORM: mac-address [144f9809c2]
> [ 0.000000] PLATFORM: watchdog-resolution [1000 ms]
> [ 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
> [ 0.000000] PLATFORM: max-cpus [64]
> [ 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c2000
> [ 0.000000] Memory hole size: 132MB
> [ 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
> [ 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
> [ 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
> [ 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
> [ 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
> [ 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
> [ 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
> [ 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
> [ 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
> [ 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
> [ 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
> [ 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
> [ 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
> [ 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
> [ 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
> [ 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
> [ 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
> [ 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
> [ 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
> [ 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
> [ 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
> [ 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
> [ 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
> [ 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
> [ 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
> [ 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
> [ 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
> [ 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
> [ 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
> [ 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
> [ 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
> [ 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
> [ 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
> [ 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
> [ 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
> [ 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
> [ 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
> [ 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
> [ 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
> [ 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
> [ 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
> [ 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
> [ 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
> [ 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
> [ 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
> [ 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
> [ 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
> [ 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
> [ 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
> [ 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
> [ 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
> [ 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
> [ 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
> [ 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
> [ 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
> [ 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
> [ 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
> [ 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
> [ 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
> [ 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
> [ 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
> [ 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
> [ 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
> [ 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
> [ 0.000000] Zone PFN ranges:
> [ 0.000000] Normal 0x00004200 -> 0x003ffd8e
> [ 0.000000] Movable zone start PFN for each node
> [ 0.000000] early_node_map[3] active PFN ranges
> [ 0.000000] 0: 0x00004200 -> 0x003ffd47
> [ 0.000000] 0: 0x003ffd50 -> 0x003ffd63
> [ 0.000000] 0: 0x003ffd87 -> 0x003ffd8e
> [ 0.000000] Booting Linux...
> [ 0.000000] CPU CAPS: [flush,stbar,swap,muldiv,v9,blkinit,n2,mul32]
> [ 0.000000] CPU CAPS: [div32,v8plus,popc,vis,vis2,ASIBlkInit]
> [ 0.000000] PERCPU: Embedded 5 pages/cpu @fffff80019000000 s10816 r8192 d21952 u65536
> [ 0.000000] SUN4V: Mondo queue sizes [cpu(8192) dev(16384) r(8192) nr(256)]
> [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 4144105
> [ 0.000000] Kernel command line: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
> [ 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
> [ 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
> [ 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
> [ 0.000000] Memory: 33091944k available (3568k kernel code, 1272k data, 216k init) [fffff80000000000,00000007ffb1c000]
> [ 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
> [ 0.000000] Hierarchical RCU implementation.
> [ 0.000000] CONFIG_RCU_FANOUT set to non-default value of 32
> [ 0.000000] NR_IRQS:255
> [ 0.000000] clocksource: mult[b4e7cf] shift[24]
> [ 0.000000] clockevent: mult[b5221ba3] shift[31]
> [ 0.000000] Console: colour dummy device 80x25
> [ 0.000000] console [tty0] enabled, bootconsole disabled
> [ 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
> [ 0.000000] PROMLIB: Root node compatible: sun4v
> [ 0.000000] Initializing cgroup subsys cpuset
> [ 0.000000] Initializing cgroup subsys cpu
> [ 0.000000] Linux version 3.1.0-razik-2011-11-05 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #3 SMP Sat Nov 5 00:24:24 CET 2011
> [ 0.000000] bootconsole [earlyprom0] enabled
> [ 0.000000] ARCH: SUN4V
> [ 0.000000] Ethernet address: 00:14:4f:98:09:c2
> [ 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
> [ 0.000000] Remapping the kernel... done.
> [ 0.000000] OF stdout device is: /virtual-devices@100/console@1
> [ 0.000000] PROM: Built device tree with 123763 bytes of memory.
> [ 0.000000] MDESC: Size is 58816 bytes.
> [ 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
> [ 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
> [ 0.000000] PLATFORM: hostid [849809c2]
> [ 0.000000] PLATFORM: serial# [00ab4130]
> [ 0.000000] PLATFORM: stick-frequency [5458c3a0]
> [ 0.000000] PLATFORM: mac-address [144f9809c2]
> [ 0.000000] PLATFORM: watchdog-resolution [1000 ms]
> [ 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
> [ 0.000000] PLATFORM: max-cpus [64]
> [ 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c2000
> [ 0.000000] Memory hole size: 132MB
> [ 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
> [ 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
> [ 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
> [ 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
> [ 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
> [ 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
> [ 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
> [ 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
> [ 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
> [ 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
> [ 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
> [ 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
> [ 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
> [ 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
> [ 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
> [ 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
> [ 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
> [ 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
> [ 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
> [ 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
> [ 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
> [ 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
> [ 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
> [ 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
> [ 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
> [ 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
> [ 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
> [ 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
> [ 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
> [ 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
> [ 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
> [ 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
> [ 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
> [ 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
> [ 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
> [ 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
> [ 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
> [ 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
> [ 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
> [ 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
> [ 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
> [ 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
> [ 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
> [ 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
> [ 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
> [ 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
> [ 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
> [ 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
> [ 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
> [ 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
> [ 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
> [ 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
> [ 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
> [ 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
> [ 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
> [ 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
> [ 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
> [ 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
> [ 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
> [ 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
> [ 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
> [ 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
> [ 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
> [ 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
> [ 0.000000] Zone PFN ranges:
> [ 0.000000] Normal 0x00004200 -> 0x003ffd8e
> [ 0.000000] Movable zone start PFN for each node
> [ 0.000000] early_node_map[3] active PFN ranges
> [ 0.000000] 0: 0x00004200 -> 0x003ffd47
> [ 0.000000] 0: 0x003ffd50 -> 0x003ffd63
> [ 0.000000] 0: 0x003ffd87 -> 0x003ffd8e
> [ 0.000000] Booting Linux...
> [ 0.000000] CPU CAPS: [flush,stbar,swap,muldiv,v9,blkinit,n2,mul32]
> [ 0.000000] CPU CAPS: [div32,v8plus,popc,vis,vis2,ASIBlkInit]
> [ 0.000000] PERCPU: Embedded 5 pages/cpu @fffff80019000000 s10816 r8192 d21952 u65536
> [ 0.000000] SUN4V: Mondo queue sizes [cpu(8192) dev(16384) r(8192) nr(256)]
> [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 4144105
> [ 0.000000] Kernel command line: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
> [ 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
> [ 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
> [ 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
> [ 0.000000] Memory: 33091944k available (3568k kernel code, 1272k data, 216k init) [fffff80000000000,00000007ffb1c000]
> [ 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
> [ 0.000000] Hierarchical RCU implementation.
> [ 0.000000] CONFIG_RCU_FANOUT set to non-default value of 32
> [ 0.000000] NR_IRQS:255
> [ 0.000000] clocksource: mult[b4e7cf] shift[24]
> [ 0.000000] clockevent: mult[b5221ba3] shift[31]
> [ 0.000000] Console: colour dummy device 80x25
> [ 0.000000] console [tty0] enabled, bootconsole disabled
> [ 1349.933990] Calibrating delay using timer specific routine.. 2832.78 BogoMIPS (lpj=5665561)
> [ 1349.934023] pid_max: default: 65536 minimum: 512
> [ 1349.934487] Mount-cache hash table entries: 512
> [ 1349.935761] Initializing cgroup subsys cpuacct
> [ 1349.935872] Initializing cgroup subsys devices
> [ 1349.935890] Initializing cgroup subsys freezer
> [ 1349.935907] Initializing cgroup subsys net_cls
> [ 1349.936185] Performance events: Supported PMU type is 'niagara2'
> [ 1349.958675] Brought up 64 CPUs
> [ 1349.958734] Testing NMI watchdog ... OK.
> [ 1350.041367] devtmpfs: initialized
> [ 1350.041863] ldc.c:v1.1 (July 22, 2008)
> [ 1350.042167] print_constraints: dummy:
> [ 1350.042552] NET: Registered protocol family 16
> [ 1350.051781] VIO: Adding device channel-devices
> [ 1350.051858] VIO: Adding device vldc-port-3-0
> [ 1350.051918] VIO: Adding device vldc-port-3-1
> [ 1350.051978] VIO: Adding device vldc-port-3-2
> [ 1350.052048] VIO: Adding device vldc-port-3-3
> [ 1350.052110] VIO: Adding device vldc-port-3-4
> [ 1350.052171] VIO: Adding device vldc-port-3-5
> [ 1350.052233] VIO: Adding device vldc-port-2-0
> [ 1350.052296] VIO: Adding device vldc-port-0-0
> [ 1350.052365] VIO: Adding device vldc-port-0-1
> [ 1350.052433] VIO: Adding device vldc-port-0-2
> [ 1350.052511] VIO: Adding device vldc-port-1-0
> [ 1350.052581] VIO: Adding device vldc-port-3-7
> [ 1350.052654] VIO: Adding device vldc-port-3-8
> [ 1350.052737] VIO: Adding device ds-1
> [ 1350.052801] VIO: Adding device ds-0
> [ 1350.057406] pci_sun4v: Registered hvapi major[1] minor[0]
> [ 1350.057671] /pci@0: SUN4V PCI Bus Module
> [ 1350.057690] /pci@0: On NUMA node -1
> [ 1350.057711] /pci@0: PCI IO[c0f0000000] MEM[c100000000]
> [ 1350.057739] /pci@0: Unable to request IOMMU resource.
> [ 1350.102075] /pci@0: Imported 3 TSB entries from OBP
> [ 1350.103612] /pci@0: MSI Queue first[0] num[36] count[128] devino[0x18]
> [ 1350.103640] /pci@0: MSI first[0] num[256] mask[0xff] width[32]
> [ 1350.103665] /pci@0: MSI addr32[0x7fff0000:0x10000] addr64[0x3ffff0000:0x10000]
> [ 1350.103694] /pci@0: MSI queues at RA [00000007fe580000]
> [ 1350.103718] PCI: Scanning PBM /pci@0
> [ 1350.115563] ds.c:v1.0 (Jul 11, 2007)
> [ 1350.118931] bio: create slab <bio-0> at 0
> [ 1350.119516] vgaarb: loaded
> [ 1350.120349] Switching to clocksource stick
> [ 1350.120981] Switched to NOHz mode on CPU #12
> [ 1350.120991] Switched to NOHz mode on CPU #24
> [ 1350.121098] Switched to NOHz mode on CPU #36
> [ 1350.121150] Switched to NOHz mode on CPU #59
> [ 1350.121253] Switched to NOHz mode on CPU #1
> [ 1350.121263] Switched to NOHz mode on CPU #48
> [ 1350.121272] Switched to NOHz mode on CPU #13
> [ 1350.121331] Switched to NOHz mode on CPU #25
> [ 1350.121457] Switched to NOHz mode on CPU #37
> [ 1350.121502] Switched to NOHz mode on CPU #60
> [ 1350.121608] Switched to NOHz mode on CPU #14
> [ 1350.121623] Switched to NOHz mode on CPU #49
> [ 1350.121658] Switched to NOHz mode on CPU #2
> [ 1350.121670] Switched to NOHz mode on CPU #26
> [ 1350.121798] Switched to NOHz mode on CPU #38
> [ 1350.121894] Switched to NOHz mode on CPU #61
> [ 1350.121903] Switched to NOHz mode on CPU #0
> [ 1350.121942] Switched to NOHz mode on CPU #15
> [ 1350.121960] Switched to NOHz mode on CPU #50
> [ 1350.121969] Switched to NOHz mode on CPU #3
> [ 1350.122011] Switched to NOHz mode on CPU #27
> [ 1350.122138] Switched to NOHz mode on CPU #39
> [ 1350.122204] Switched to NOHz mode on CPU #62
> [ 1350.122277] Switched to NOHz mode on CPU #16
> [ 1350.122298] Switched to NOHz mode on CPU #4
> [ 1350.122307] Switched to NOHz mode on CPU #51
> [ 1350.122351] Switched to NOHz mode on CPU #28
> [ 1350.122480] Switched to NOHz mode on CPU #40
> [ 1350.122564] Switched to NOHz mode on CPU #63
> [ 1350.122623] Switched to NOHz mode on CPU #17
> [ 1350.122640] Switched to NOHz mode on CPU #5
> [ 1350.122653] Switched to NOHz mode on CPU #52
> [ 1350.122697] Switched to NOHz mode on CPU #29
> [ 1350.122828] Switched to NOHz mode on CPU #41
> [ 1350.122954] Switched to NOHz mode on CPU #18
> [ 1350.122971] Switched to NOHz mode on CPU #6
> [ 1350.123029] Switched to NOHz mode on CPU #53
> [ 1350.123038] Switched to NOHz mode on CPU #30
> [ 1350.123175] Switched to NOHz mode on CPU #42
> [ 1350.123286] Switched to NOHz mode on CPU #19
> [ 1350.123298] Switched to NOHz mode on CPU #7
> [ 1350.123382] Switched to NOHz mode on CPU #31
> [ 1350.123400] Switched to NOHz mode on CPU #54
> [ 1350.123524] Switched to NOHz mode on CPU #43
> [ 1350.123602] Switched to NOHz mode on CPU #8
> [ 1350.123620] Switched to NOHz mode on CPU #20
> [ 1350.123729] Switched to NOHz mode on CPU #32
> [ 1350.123749] Switched to NOHz mode on CPU #55
> [ 1350.123860] Switched to NOHz mode on CPU #44
> [ 1350.123929] Switched to NOHz mode on CPU #9
> [ 1350.123971] Switched to NOHz mode on CPU #21
> [ 1350.124065] Switched to NOHz mode on CPU #33
> [ 1350.124095] Switched to NOHz mode on CPU #56
> [ 1350.124209] Switched to NOHz mode on CPU #45
> [ 1350.124268] Switched to NOHz mode on CPU #10
> [ 1350.124303] Switched to NOHz mode on CPU #22
> [ 1350.124402] Switched to NOHz mode on CPU #34
> [ 1350.124442] Switched to NOHz mode on CPU #57
> [ 1350.124564] Switched to NOHz mode on CPU #46
> [ 1350.124737] Switched to NOHz mode on CPU #58
> [ 1350.124747] Switched to NOHz mode on CPU #23
> [ 1350.124756] Switched to NOHz mode on CPU #47
> [ 1350.124766] Switched to NOHz mode on CPU #11
> [ 1350.124776] Switched to NOHz mode on CPU #35
> [ 1350.129462] NET: Registered protocol family 2
> [ 1350.129995] IP route cache hash table entries: 524288 (order: 9, 4194304 bytes)
> [ 1350.132183] TCP established hash table entries: 524288 (order: 10, 8388608 bytes)
> [ 1350.132224] ds-1: Registered pri service.
> [ 1350.132328] ds-1: Registered var-config-backup service.
> [ 1350.157752] TCP bind hash table entries: 65536 (order: 7, 1048576 bytes)
> [ 1350.161496] TCP: Hash tables configured (established 524288 bind 65536)
> [ 1350.161523] TCP reno registered
> [ 1350.161562] UDP hash table entries: 16384 (order: 6, 524288 bytes)
> [ 1350.163563] UDP-Lite hash table entries: 16384 (order: 6, 524288 bytes)
> [ 1350.166529] NET: Registered protocol family 1
> [ 1350.166613] RPC: creating workqueue rpciod
> [ 1350.166921] RPC: registering /proc/net/rpc
> [ 1350.167024] svc: Adding svc transport class 'tcp'
> [ 1350.167051] svc: Adding svc transport class 'udp'
> [ 1350.167121] RPC: Registered named UNIX socket transport module.
> [ 1350.167145] RPC: Registered udp transport module.
> [ 1350.167165] RPC: Registered tcp transport module.
> [ 1350.167185] RPC: Registered tcp NFSv4.1 backchannel transport module.
> [ 1350.168544] audit: initializing netlink socket (disabled)
> [ 1350.168591] type=2000 audit(0.304:1): initialized
> [ 1350.872401] HugeTLB registered 4 MB page size, pre-allocated 0 pages
> [ 1350.881157] VFS: Disk quotas dquot_6.5.2
> [ 1350.881602] Dquot-cache hash table entries: 1024 (order 0, 8192 bytes)
> [ 1350.882234] RPC: creating workqueue nfsiod
> [ 1350.883014] RPC: registering /proc/net/rpc/nfs
> [ 1350.883068] msgmni has been set to 32768
> [ 1350.884176] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
> [ 1350.884211] io scheduler noop registered
> [ 1350.884230] io scheduler deadline registered
> [ 1350.884466] io scheduler cfq registered (default)
> [ 1350.885978] f027ab74: ttyS0 at I/O 0x0 (irq = 17) is a SUN4V HCONS
> [ 1351.101139] console [ttyHV0] enabled
> [ 1351.103812] f028af14: ttyS1 at MMIO 0xfff0ca0000 (irq = 24) is a 16550A
> [ 1351.104504] [drm] Initialized drm 1.1.0 20060810
> [ 1351.105715] e1000e: Intel(R) PRO/1000 Network Driver - 1.4.4-k
> [ 1351.105850] e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
> [ 1351.105948] e1000e 0000:08:00.0: Disabling ASPM L1
> [ 1351.279997] e1000e 0000:08:00.0: eth0: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c2
> [ 1351.280059] e1000e 0000:08:00.0: eth0: Intel(R) PRO/1000 Network Connection
> [ 1351.280186] e1000e 0000:08:00.0: eth0: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
> [ 1351.280267] e1000e 0000:08:00.1: Disabling ASPM L1
> [ 1351.457748] e1000e 0000:08:00.1: eth1: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c3
> [ 1351.457809] e1000e 0000:08:00.1: eth1: Intel(R) PRO/1000 Network Connection
> [ 1351.457935] e1000e 0000:08:00.1: eth1: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
> [ 1351.458003] e1000e 0000:09:00.0: Disabling ASPM L1
> [ 1351.632096] e1000e 0000:09:00.0: eth2: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c4
> [ 1351.632157] e1000e 0000:09:00.0: eth2: Intel(R) PRO/1000 Network Connection
> [ 1351.632283] e1000e 0000:09:00.0: eth2: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
> [ 1351.632351] e1000e 0000:09:00.1: Disabling ASPM L1
> [ 1351.809926] e1000e 0000:09:00.1: eth3: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c5
> [ 1351.810047] e1000e 0000:09:00.1: eth3: Intel(R) PRO/1000 Network Connection
> [ 1351.810231] e1000e 0000:09:00.1: eth3: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
> [ 1351.810858] mousedev: PS/2 mouse device common for all mice
> [ 1351.811886] rtc-sun4v rtc-sun4v: rtc core: registered sun4v as rtc0
> [ 1351.812592] TCP cubic registered
> [ 1351.813469] NET: Registered protocol family 10
> [ 1351.815597] Mobile IPv6
> [ 1351.815633] NET: Registered protocol family 17
> [ 1351.815704] Registering the dns_resolver key type
> [ 1351.816174] registered taskstats version 1
> [ 1351.818649] rtc-sun4v rtc-sun4v: setting system clock to 2011-11-04 23:31:46 UTC (1320449506)
> [ 1352.067344] ADDRCONF(NETDEV_UP): eth0: link is not ready
> [ 1354.200067] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
> [ 1354.222397] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 1354.243385] IP-Config: Complete:
> [ 1354.243457] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=255.255.255.255,
> [ 1354.243659] host=137.226.167.242, domain=, nis-domain=(none),
> [ 1354.243774] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
> [ 1354.244257] Root-NFS: nfsroot=/srv/nfs/cluster2
> [ 1354.244620] NFS: nfs mount opts='vers=2,udp,rsize=4096,wsize=4096,nolock,addr=137.226.167.241'
> [ 1354.244710] NFS: parsing nfs mount option 'vers=2'
> [ 1354.244770] NFS: parsing nfs mount option 'udp'
> [ 1354.244834] NFS: parsing nfs mount option 'rsize=4096'
> [ 1354.244884] NFS: parsing nfs mount option 'wsize=4096'
> [ 1354.244934] NFS: parsing nfs mount option 'nolock'
> [ 1354.244981] NFS: parsing nfs mount option 'addr=137.226.167.241'
> [ 1354.245045] NFS: MNTPATH: '/srv/nfs/cluster2'
> [ 1354.245080] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
> [ 1354.245157] RPC: set up xprt to 137.226.167.241 (autobind) via udp
> [ 1354.245236] RPC: created transport fffff807fdf40800 with 16 slots
> [ 1354.245285] RPC: creating mount client for 137.226.167.241 (xprt fffff807fdf40800)
> [ 1354.245346] RPC: creating UNIX authenticator for client fffff807fdf1a200
> [ 1354.245411] RPC: new task initialized, procpid 1
> [ 1354.245456] RPC: allocated task fffff807fdf10000
> [ 1354.245504] RPC: 1 __rpc_execute flags=0x680
> [ 1354.245539] RPC: 1 call_start mount1 proc NULL (sync)
> [ 1354.245614] RPC: 1 call_reserve (status 0)
> [ 1354.245649] RPC: 1 reserved req fffff807fdf1a000 xid 51d47d10
> [ 1354.245698] RPC: wake_up_next(fffff807fdf40978 "xprt_sending")
> [ 1354.245747] RPC: 1 call_reserveresult (status 0)
> [ 1354.245792] RPC: 1 call_refresh (status 0)
> [ 1354.245825] RPC: 1 holding NULL cred 00000000008b8cf0
> [ 1354.245871] RPC: 1 refreshing NULL cred 00000000008b8cf0
> [ 1354.245917] RPC: 1 call_refreshresult (status 0)
> [ 1354.245962] RPC: 1 call_allocate (status 0)
> [ 1354.246025] RPC: 1 allocated buffer of size 96 at fffff807fdf41000
> [ 1354.246073] RPC: 1 call_bind (status 0)
> [ 1354.246108] RPC: 1 rpcb_getport_async(137.226.167.241, 100005, 1, 17)
> [ 1354.246157] RPC: 1 sleep_on(queue "xprt_binding" time 4294893393)
> [ 1354.246206] RPC: 1 added to queue fffff807fdf408c8 "xprt_binding"
> [ 1354.246253] RPC: 1 setting alarm for 60000 ms
> [ 1354.246291] RPC: 1 rpcb_getport_async: trying rpcbind version 2
> [ 1354.246366] RPC: set up xprt to 137.226.167.241 (port 111) via udp
> [ 1354.246445] RPC: created transport fffff807fdf41800 with 16 slots
> [ 1354.246495] RPC: creating rpcbind client for 137.226.167.241 (xprt fffff807fdf41800)
> [ 1354.246554] RPC: creating UNIX authenticator for client fffff807fdf1c400
> [ 1354.246612] RPC: new task initialized, procpid 1
> [ 1354.246656] RPC: allocated task fffff807fdf100e0
> [ 1354.246708] RPC: rpc_release_client(fffff807fdf1c400)
> [ 1354.246756] RPC: 1 sync task going to sleep
> [ 1354.246825] RPC: 2 __rpc_execute flags=0x681
> [ 1354.246860] RPC: 2 call_start rpcbind2 proc GETPORT (async)
> [ 1354.246921] RPC: 2 call_reserve (status 0)
> [ 1354.246957] RPC: 2 reserved req fffff807fdf1c200 xid 18460f63
> [ 1354.247005] RPC: wake_up_next(fffff807fdf41978 "xprt_sending")
> [ 1354.247054] RPC: 2 call_reserveresult (status 0)
> [ 1354.247098] RPC: 2 call_refresh (status 0)
> [ 1354.247130] RPC: 2 looking up UNIX cred
> [ 1354.247189] RPC: looking up UNIX cred
> [ 1354.247223] RPC: allocating UNIX cred for uid 0 gid 0
> [ 1354.247270] RPC: 2 refreshing UNIX cred fffff807fdefc900
> [ 1354.247317] RPC: 2 call_refreshresult (status 0)
> [ 1354.247361] RPC: 2 call_allocate (status 0)
> [ 1354.247395] RPC: 2 allocated buffer of size 416 at fffff807fdf42000
> [ 1354.247443] RPC: 2 call_bind (status 0)
> [ 1354.247476] RPC: 2 call_connect xprt fffff807fdf41800 is not connected
> [ 1354.247526] RPC: 2 xprt_connect xprt fffff807fdf41800 is not connected
> [ 1354.247604] RPC: 2 xprt_cwnd_limited cong = 0 cwnd = 256
> [ 1354.247651] RPC: 2 sleep_on(queue "xprt_pending" time 4294893394)
> [ 1354.247700] RPC: 2 added to queue fffff807fdf41a28 "xprt_pending"
> [ 1354.247747] RPC: 2 setting alarm for 5000 ms
> [ 1354.247780] RPC: xs_connect scheduled xprt fffff807fdf41800
> [ 1354.247922] RPC: xs_bind 0.0.0.0:0: ok (0)
> [ 1354.247988] RPC: worker connecting xprt fffff807fdf41800 via udp to 137.226.167.241 (port 111)
> [ 1354.248050] RPC: 2 __rpc_wake_up_task (now 4294893394)
> [ 1354.248094] RPC: 2 disabling timer
> [ 1354.248125] RPC: 2 removed from queue fffff807fdf41a28 "xprt_pending"
> [ 1354.248172] RPC: __rpc_wake_up_task done
> [ 1354.248205] RPC: 2 __rpc_execute flags=0x681
> [ 1354.248238] RPC: 2 xprt_connect_status: connection established
> [ 1354.248285] RPC: 2 call_connect_status (status 0)
> [ 1354.248329] RPC: 2 call_transmit (status 0)
> [ 1354.248389] RPC: 2 xprt_prepare_transmit
> [ 1354.248421] RPC: 2 rpc_xdr_encode (status 0)
> [ 1354.248455] RPC: 2 marshaling UNIX cred fffff807fdefc900
> [ 1354.248503] RPC: 2 using AUTH_UNIX cred fffff807fdefc900 to wrap rpc data
> [ 1354.248556] RPC: 2 encoding PMAP_GETPORT call (100005, 1, 17, 0)
> [ 1354.248603] RPC: 2 xprt_transmit(92)
> [ 1354.253576] RPC: xs_udp_send_request(92) = 92
> [ 1354.253611] RPC: 2 xmit complete
> [ 1354.253644] RPC: 2 sleep_on(queue "xprt_pending" time 4294893395)
> [ 1354.253692] RPC: 2 added to queue fffff807fdf41a28 "xprt_pending"
> [ 1354.253739] RPC: 2 setting alarm for 5000 ms
> [ 1359.265758] RPC: 2 timeout
...
<snip>
...
> [ 1389.332120] RPC: 1 rpcbind request timed out
> [ 1389.332129] RPC: 1 call_timeout (major)
> [ 1389.332138] RPC: 1 return 0, status -60
> [ 1389.332145] RPC: 1 release task
> [ 1389.332158] RPC: freeing buffer of size 96 at fffff807fdf41000
> [ 1389.332171] RPC: 1 release request fffff807fdf1a000
> [ 1389.332181] RPC: wake_up_next(fffff807fdf40ad8 "xprt_backlog")
> [ 1389.332194] RPC: rpc_release_client(fffff807fdf1a200)
> [ 1389.332203] RPC: 1 freeing task
> [ 1389.332213] RPC: shutting down mount client for 137.226.167.241
> [ 1389.332222] RPC: rpc_release_client(fffff807fdf1a200)
> [ 1389.332232] RPC: destroying UNIX authenticator 00000000008b8d78
> [ 1389.332246] RPC: destroying mount client for 137.226.167.241
> [ 1389.332259] NFS: failed to create MNT RPC client, status=-60
> [ 1389.332271] NFS: unable to mount server 137.226.167.241, error -60
> [ 1389.332306] VFS: Unable to mount root fs via NFS, trying floppy.
> [ 1389.332444] List of all partitions:
> [ 1389.332454] No filesystem could mount root, tried:
> [ 1389.332469] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
> [ 1389.332480] Call Trace:
> [ 1389.332509] [00000000008c6d18] mount_block_root+0x210/0x228
> [ 1389.332523] [00000000008c6df4] mount_root+0xc4/0xd4
> [ 1389.332537] [00000000008c6f80] prepare_namespace+0x17c/0x1c0
> [ 1389.332554] [00000000008c6250] kernel_init+0x140/0x158
> [ 1389.332574] [000000000042ac68] kernel_thread+0x30/0x48
> [ 1389.332593] [0000000000767840] rest_init+0x18/0x74
> [ 1389.332682] Press Stop-A (L1-A) to return to the boot prom
> [ 1389.333548] RPC: destroying transport fffff807fdf40800
> [ 1389.334301] RPC: xs_destroy xprt fffff807fdf40800
> [ 1389.334530] RPC: xs_close xprt fffff807fdf40800
> [ 1389.334576] RPC: disconnected transport fffff807fdf40800

So the NFS client is trying to establish contact with the server for a
period of 35 seconds, and there is no reply... Have you tried using
wireshark to see if any NFS traffic is arriving at the server from the
client?

Have you tried using the 'mountproto=tcp' option?

Trond

--
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
http://www.netapp.com


2011-11-03 21:11:04

by Jim Rees

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

Trond Myklebust wrote:

> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
> [ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
> [ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
> [ 442.667032] VFS: Insert root floppy and press ENTER
>
Error 60 is ETIMEDOUT on SPARC, so it seems that the problem is
basically the same one that you see in your 2.6.32 trace (rpcbind:
server 137.226.167.241 not responding, timed out) except that now it is
a fatal error.

Any idea why the first RPC calls might be failing here? A switch
misconfiguration or something like that perhaps?

Wasn't there a change in the way nfs mount options are handled by the kernel
for nfsroot about the time of 2.6.39? Something about changing from default
udp to tcp maybe?

2011-11-12 18:49:33

by Jim Rees

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

Lukas Razik wrote:

I've connected the systems directly (without a switch between them).
It seems that the problem came from the switch because now the newer kernels also mount their NFSROOTs during start up without problems.

Assuming that the ports of the switch really run in STP mode - two questions:
1. Is it unusual that the ports of a switch where computers are connected to run in STP mode?
2. If that's not unusual: Wouldn't it be useful if the new linux kernels could mount an NFSROOT even when the system is plugged into an STP port?

STP ports are normally won't start forwarding packets until they have
decided there are no loops. The usual way to configure a data center switch
is to leave the trunks in this configuration, and set portfast on the ports
that have end systems (like your server) on them. I think "portfast" is the
Cisco term for this, other vendors might call it something else. Portfast
ports are still running stp, they just start out in forwarding mode and only
switch to learning mode if they detect a problem.

The question for us is how long should an nfsroot client wait for the server
to reply. It sounds like the client used to wait longer than it does now.
It seems to me the client should wait at least 90 seconds so that the
situation you're in (servers on non-portfast ports) will work. I would
think they should wait indefinitely, since there's not much else they can
do.

2011-11-03 20:55:39

by Myklebust, Trond

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

On Thu, 2011-11-03 at 19:43 +0000, Lukas Razik wrote:
> Hello together!
>
> My OS: Debian 6.0.3 (squeeze)
> Machines: SUN Enterprise T5120 (USPARC64)
> ---
> Issue description:
>
> I've an NFS
> server (cluster1=137.226.167.241) and a
> client (cluster2=137.226.167.242) which should mount it's nfsroot from cluster1.
>
> The linux-2.6.32 kernel on cluster2 shows this during startup:
> [ 528.982985] IP-Config: Complete:
> [ 528.983049] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
> [ 528.983299] host=cluster2, domain=, nis-domain=(none),
> [ 528.983383] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
> [ 528.983633] Looking up port of RPC 100003/2 on 137.226.167.241
> [ 530.037059] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
> [ 530.056881] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 564.002113] rpcbind: server 137.226.167.241 not responding, timed out
> [ 564.002295] Root-NFS: Unable to get nfsd port number from server, using default
> [ 564.002412] Looking up port of RPC 100005/1 on 137.226.167.241
> [ 564.104137] VFS: Mounted root (nfs filesystem) on device 0:15.
>
> It can mount the nfsroot finally.
>
> But if I use kernel linux-2.6.39.4 on cluster2 it can't mount it's nfsroot.
> (I've added "nfsdebug" to the kernel arguments for more debug info):
> [ 407.571521] IP-Config: Complete:
> [ 407.571589] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
> [ 407.571793] host=cluster2, domain=, nis-domain=(none),
> [ 407.571907] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
> [ 407.572332] Root-NFS: nfsroot=/srv/nfs/cluster2
> [ 407.572726] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
> [ 407.572927] NFS: parsing nfs mount option 'udp'
> [ 407.572995] NFS: parsing nfs mount option 'nolock'
> [ 407.573071] NFS: parsing nfs mount option 'addr=137.226.167.241'
> [ 407.573139] NFS: MNTPATH: '/srv/nfs/cluster2'
> [ 407.573203] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
> [ 408.617894] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
> [ 408.638319] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
> [ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
> [ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
> [ 442.667032] VFS: Insert root floppy and press ENTER
>
Error 60 is ETIMEDOUT on SPARC, so it seems that the problem is
basically the same one that you see in your 2.6.32 trace (rpcbind:
server 137.226.167.241 not responding, timed out) except that now it is
a fatal error.

Any idea why the first RPC calls might be failing here? A switch
misconfiguration or something like that perhaps?

Trond
--
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
http://www.netapp.com


2011-11-04 14:09:43

by Myklebust, Trond

[permalink] [raw]
Subject: RE: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> -----Original Message-----
> From: Chuck Lever [mailto:[email protected]]
> Sent: Friday, November 04, 2011 10:02 AM
> To: Jim Rees
> Cc: Lukas Razik; Myklebust, Trond; Linux NFS Mailing List
> Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64
>
>
> On Nov 4, 2011, at 9:20 AM, Jim Rees wrote:
> > As we move toward nfs4 someone will have to give some thought to
> nfsroot.
> > It's hard to imagine we could put enough nfs4 cruft into the kernel
> > (gssd,
> > idmapd) to make it work.
>
> A kernel-level basic id mapper is being considered. That would allow
NFSv4
> with AUTH_SYS, if we can get the NIC problems squared away.

We already have a solution for NFSv4 and nfsroot: use auth_sys and set
the nfs4_disable_idmapping flag. That does require an rfc3530-bis
compatible server, but it shouldn't be long before we see those
appearing in the wild...

Trond

2011-11-05 02:14:15

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

***IMPORTANT HINT***

In the test belowI've tried to ping the NFS client (cluster2) as soon as I've got the

ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
message.


And at first I've got no answers from the client. But 10-20sec later I've got pongs back.
So the NIC comes up some seconds later than the message "link becomes ready" is shown...
In the case of 'mountproto=tcp' it comes up after the kernel panic. So that's much too late.



[? 126.134520] ADDRCONF(NETDEV_UP): eth0: link is not ready
[? 128.282991] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[? 128.305364] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[? 128.326199] IP-Config: Complete:
[? 128.326269]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=255.255.255.255,
[? 128.326471]????? host=137.226.167.242, domain=, nis-domain=(none),
[? 128.326586]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[? 128.327055] Root-NFS: nfsroot=/srv/nfs/cluster2,mountproto=tcp
[? 128.327404] NFS: nfs mount opts='vers=2,udp,rsize=4096,wsize=4096,mountproto=tcp,nolock,addr=137.226.167.241'
[? 128.327477] NFS:?? parsing nfs mount option 'vers=2'
[? 128.327538] NFS:?? parsing nfs mount option 'udp'
[? 128.327573] NFS:?? parsing nfs mount option 'rsize=4096'
[? 128.327623] NFS:?? parsing nfs mount option 'wsize=4096'
[? 128.327673] NFS:?? parsing nfs mount option 'mountproto=tcp'
[? 128.327728] NFS:?? parsing nfs mount option 'nolock'
[? 128.327774] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[? 128.327838] NFS: MNTPATH: '/srv/nfs/cluster2'
[? 128.327873] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[? 128.327942] RPC:?????? set up xprt to 137.226.167.241 (autobind) via tcp
[? 128.327992] RPC:?????? created transport fffff807fdec0800 with 128 slots
[? 128.328042] RPC:?????? creating mount client for 137.226.167.241 (xprt fffff807fdec0800)
[? 128.328103] RPC:?????? creating UNIX authenticator for client fffff807fdea0600
[? 128.328168] RPC:?????? new task initialized, procpid 1
[? 128.328213] RPC:?????? allocated task fffff807fe38e000
[? 128.328260] RPC:???? 1 __rpc_execute flags=0x680
[? 128.328295] RPC:???? 1 call_start mount1 proc NULL (sync)
[? 128.328341] RPC:???? 1 call_reserve (status 0)
[? 128.328376] RPC:???? 1 reserved req fffff807fdea0400 xid 39411b74
[? 128.328425] RPC:?????? wake_up_next(fffff807fdec0978 "xprt_sending")
[? 128.328474] RPC:???? 1 call_reserveresult (status 0)
[? 128.328518] RPC:???? 1 call_refresh (status 0)
[? 128.328551] RPC:???? 1 holding NULL cred 00000000008b8cf0
[? 128.328597] RPC:???? 1 refreshing NULL cred 00000000008b8cf0
[? 128.328643] RPC:???? 1 call_refreshresult (status 0)
[? 128.328687] RPC:???? 1 call_allocate (status 0)
[? 128.328722] RPC:???? 1 allocated buffer of size 96 at fffff807fdec1000
[? 128.328770] RPC:???? 1 call_bind (status 0)
[? 128.328804] RPC:???? 1 rpcb_getport_async(137.226.167.241, 100005, 1, 6)
[? 128.328854] RPC:???? 1 sleep_on(queue "xprt_binding" time 4294893514)
[? 128.328902] RPC:???? 1 added to queue fffff807fdec08c8 "xprt_binding"
[? 128.328949] RPC:???? 1 setting alarm for 60000 ms
[? 128.328986] RPC:???? 1 rpcb_getport_async: trying rpcbind version 2
[? 128.329046] RPC:?????? set up xprt to 137.226.167.241 (port 111) via tcp
[? 128.329096] RPC:?????? created transport fffff807fdec1800 with 128 slots
[? 128.329146] RPC:?????? creating rpcbind client for 137.226.167.241 (xprt fffff807fdec1800)
[? 128.329204] RPC:?????? creating UNIX authenticator for client fffff807fdea0c00
[? 128.329262] RPC:?????? new task initialized, procpid 1
[? 128.329306] RPC:?????? allocated task fffff807fe38e0e0
[? 128.329358] RPC:?????? rpc_release_client(fffff807fdea0c00)
[? 128.329406] RPC:???? 1 sync task going to sleep
[? 128.329446] RPC:???? 2 __rpc_execute flags=0x681
[? 128.329482] RPC:???? 2 call_start rpcbind2 proc GETPORT (async)
[? 128.329529] RPC:???? 2 call_reserve (status 0)
[? 128.329563] RPC:???? 2 reserved req fffff807fdea0a00 xid 54cf8ddb
[? 128.329610] RPC:?????? wake_up_next(fffff807fdec1978 "xprt_sending")
[? 128.329659] RPC:???? 2 call_reserveresult (status 0)
[? 128.329703] RPC:???? 2 call_refresh (status 0)
[? 128.329735] RPC:???? 2 looking up UNIX cred
[? 128.329765] RPC:?????? looking up UNIX cred
[? 128.329809] RPC:?????? allocating UNIX cred for uid 0 gid 0
[? 128.329859] RPC:???? 2 refreshing UNIX cred fffff807fe37c900
[? 128.329906] RPC:???? 2 call_refreshresult (status 0)
[? 128.329950] RPC:???? 2 call_allocate (status 0)
[? 128.329984] RPC:???? 2 allocated buffer of size 416 at fffff807fdec2000
[? 128.330032] RPC:???? 2 call_bind (status 0)
[? 128.330064] RPC:???? 2 call_connect xprt fffff807fdec1800 is not connected
[? 128.330114] RPC:???? 2 xprt_connect xprt fffff807fdec1800 is not connected
[? 128.330165] RPC:???? 2 sleep_on(queue "xprt_pending" time 4294893515)
[? 128.330213] RPC:???? 2 added to queue fffff807fdec1a28 "xprt_pending"
[? 128.330260] RPC:???? 2 setting alarm for 60000 ms
[? 128.330294] RPC:?????? xs_connect scheduled xprt fffff807fdec1800
[? 128.330453] RPC:?????? xs_bind 0.0.0.0:0: ok (0)
[? 128.330490] RPC:?????? worker connecting xprt fffff807fdec1800 via tcp to 137.226.167.241 (port 111)
[? 128.330650] RPC:?????? fffff807fdec1800 connect status 36 connected 0 sock state 2
[? 131.335657] RPC:?????? xs_error_report client fffff807fdec1800...
[? 131.335665] RPC:?????? error 65
[? 131.335822] RPC:???? 2 __rpc_wake_up_task (now 4294894266)
[? 131.335924] RPC:???? 2 disabling timer
[? 131.335985] RPC:???? 2 removed from queue fffff807fdec1a28 "xprt_pending"
[? 131.336061] RPC:?????? __rpc_wake_up_task done
[? 131.336097] RPC:?????? xs_tcp_state_change client fffff807fdec1800...
[? 131.336146] RPC:?????? state 7 conn 0 dead 0 zapped 1 sk_shutdown 3
[? 131.336194] RPC:?????? disconnected transport fffff807fdec1800
[? 131.336291] RPC:???? 2 __rpc_execute flags=0x681
[? 131.336355] RPC:???? 2 xprt_connect_status: retrying
[? 131.336399] RPC:???? 2 call_connect_status (status -11)
[? 131.336473] RPC:???? 2 call_transmit (status 0)
[? 131.336504] RPC:???? 2 xprt_prepare_transmit
[? 131.336565] RPC:???? 2 rpc_xdr_encode (status 0)
[? 131.336599] RPC:???? 2 marshaling UNIX cred fffff807fe37c900
[? 131.336676] RPC:???? 2 using AUTH_UNIX cred fffff807fe37c900 to wrap rpc data
[? 131.336757] RPC:???? 2 encoding PMAP_GETPORT call (100005, 1, 6, 0)
[? 131.336805] RPC:???? 2 xprt_transmit(96)
[? 131.336873] RPC:?????? xs_tcp_send_request(96) = -65
[? 131.336946] RPC:?????? sendmsg returned unrecognized error 65
[? 131.336993] RPC:?????? xs_tcp_state_change client fffff807fdec1800...
[? 131.337071] RPC:?????? state 7 conn 0 dead 0 zapped 1 sk_shutdown 3
[? 131.337146] RPC:?????? disconnected transport fffff807fdec1800
[? 131.337195] RPC:?????? wake_up_next(fffff807fdec1978 "xprt_sending")
[? 131.337273] RPC:?????? setting port for xprt fffff807fdec0800 to 0
[? 131.337352] RPC:???? 2 rpcb_getport_done(status -65, port 0)
[? 131.337398] RPC:???? 2 return 0, status -65
[? 131.337456] RPC:???? 2 release task
[? 131.337490] RPC:?????? freeing buffer of size 416 at fffff807fdec2000
[? 131.337568] RPC:???? 2 release request fffff807fdea0a00
[? 131.337642] RPC:?????? wake_up_next(fffff807fdec1ad8 "xprt_backlog")
[? 131.337690] RPC:?????? rpc_release_client(fffff807fdea0c00)
[? 131.337764] RPC:?????? destroying rpcbind client for 137.226.167.241
[? 131.337841] RPC:?????? destroying transport fffff807fdec1800
[? 131.337921] RPC:?????? xs_destroy xprt fffff807fdec1800
[? 131.338045] RPC:?????? xs_close xprt fffff807fdec1800
[? 131.338134] RPC:?????? disconnected transport fffff807fdec1800
[? 131.338184] RPC:???? 2 freeing task
[? 131.338245] RPC:???? 1 __rpc_wake_up_task (now 4294894266)
[? 131.338317] RPC:???? 1 disabling timer
[? 131.338348] RPC:???? 1 removed from queue fffff807fdec08c8 "xprt_binding"
[? 131.338429] RPC:?????? __rpc_wake_up_task done
[? 131.338443] RPC:???? 1 sync task resuming
[? 131.338454] RPC:???? 1 remote rpcbind unreachable: -65
[? 131.338466] RPC:???? 1 return 0, status -65
[? 131.338473] RPC:???? 1 release task
[? 131.338483] RPC:?????? freeing buffer of size 96 at fffff807fdec1000
[? 131.338495] RPC:???? 1 release request fffff807fdea0400
[? 131.338505] RPC:?????? wake_up_next(fffff807fdec0ad8 "xprt_backlog")
[? 131.338517] RPC:?????? rpc_release_client(fffff807fdea0600)
[? 131.338527] RPC:???? 1 freeing task
[? 131.338537] RPC:?????? shutting down mount client for 137.226.167.241
[? 131.338548] RPC:?????? rpc_release_client(fffff807fdea0600)
[? 131.338559] RPC:?????? destroying UNIX authenticator 00000000008b8d78
[? 131.338572] RPC:?????? destroying mount client for 137.226.167.241
[? 131.338584] NFS: failed to create MNT RPC client, status=-65
[? 131.338593] NFS: unable to mount server 137.226.167.241, error -65
[? 131.338627] VFS: Unable to mount root fs via NFS, trying floppy.
[? 131.338687] List of all partitions:
[? 131.338697] No filesystem could mount root, tried:
[? 131.338713] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
[? 131.338723] Call Trace:
[? 131.338752]? [00000000008c6d18] mount_block_root+0x210/0x228
[? 131.338767]? [00000000008c6df4] mount_root+0xc4/0xd4
[? 131.338781]? [00000000008c6f80] prepare_namespace+0x17c/0x1c0
[? 131.338795]? [00000000008c6250] kernel_init+0x140/0x158
[? 131.338816]? [000000000042ac68] kernel_thread+0x30/0x48
[? 131.338835]? [0000000000767840] rest_init+0x18/0x74
[? 131.338941] Press Stop-A (L1-A) to return to the boot prom
[? 131.339844] RPC:?????? destroying transport fffff807fdec0800
[? 131.339948] RPC:?????? xs_destroy xprt fffff807fdec0800
[? 131.340039] RPC:?????? xs_close xprt fffff807fdec0800
[? 131.340114] RPC:?????? disconnected transport fffff807fdec0800
[? 138.467087] eth0: no IPv6 routers present

2011-11-04 13:54:30

by Chuck Lever III

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64


On Nov 4, 2011, at 5:44 AM, Lukas Razik wrote:

>>> OK
>
>>> I've watched wireshark on cluster1 during start up of cluster2 (with
>> linux-2.6.32) which first tries 10003 and then 10005.
>>> The result is that cluster1 doesn't get a datagram for port 10003:
>>> http://net.razik.de/linux/T5120/cluster2_NFSROOT_MOUNT.png
>>>
>>> The first ARP request in the screenshot came _after_ the <tag> in
>> this kernel log:
>>> [ 6492.807917] IP-Config: Complete:
>>> [ 6492.807978] device=eth0, addr=137.226.167.242,
>> mask=255.255.255.224, gw=137.226.167.225,
>>> [ 6492.808227] host=cluster2, domain=, nis-domain=(none),
>>> [ 6492.808312] bootserver=255.255.255.255, rootserver=137.226.167.241,
>> rootpath=
>>> [ 6492.808570] Looking up port of RPC 100003/2 on 137.226.167.241
>>> [ 6493.886014] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
>> Control: Rx
>>> [ 6493.905840] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>>> <tag>
>>> [ 6527.827055] rpcbind: server 137.226.167.241 not responding, timed out
>>> [ 6527.827237] Root-NFS: Unable to get nfsd port number from server, using
>> default
>>> [ 6527.827353] Looking up port of RPC 100005/1 on 137.226.167.241
>>> [ 6527.842212] VFS: Mounted root (nfs filesystem) on device 0:15.
>>>
>>>
>>> So I don't think that it's a problem of the hardware between the
>> machines.
>>> There's no reason why I wouldn't see an ARP requests from cluster2
>> which would have been sent _before_ the <tag> if there would be one. I
>> think: cluster2 never sends a request to port 10003.
>>> What do you think?
>>
>> It agrees with our initial assessment that the first RPC request is failing.
>> The RPC client never gets the request through cluster2's network stack
>> because the NIC hasn't re-initialized when the request is sent.
>>
>> It looks like your system does a PXE boot, which provides the IP configuration
>> shown above. But then the kernel resets the NIC. During that reset, the kernel
>> is attempting to contact the NFS server to mount the root file system.
>>
>> We've set up NFSROOT to use UDP so that it will be relatively immune to
>> these initialization order problems. The RPC client should be retrying the lost
>> request, but apparently it isn't. What if you added "retrans=10"
>> to cluster2's mount options? (on the chance that mount option setting would
>> be copied to the rpcbind client's RPC transport...)
>>
>> IMO the correct way to fix this is to provide proper serialization in the
>> networking layer so that RPC requests are not even attempted until the NIC is
>> ready to carry traffic. That may be a pipe dream though.
>>
>
> I thank you three very much for your help! Now I'm sure that I haven't misconfigured anything...
> But I don't see a work around to get the NFSROOT mounted during start up of a kernel >=2.6.37 .
> It would be very sad with these nice Oracle (SUN) machines if no one could use them because of this bug.

If you boot via tftp, I bet this problem will go away because the network interface will be working by the time the NFSROOT mount is attempted.

The NFSROOT code assumes that if kernel IP configuration worked, then the NIC is already up. That is clearly not the case if you boot from your local disk.

> Do you know a kernel developer who maybe would try to write a patch for this problem?
> Or do you have another idea what I could do?

As for a patch: no-one can write a patch unless we understand precisely why the first RPC fails. I already explained how to add a line or two to fs/nfs/nfsroot.c to give us more information. If you need a patch to do this, I can send one later today.

I might be able to reproduce it here, now that I understand your set up, but it would require building a partial NFSROOT environment. I can't get to that until next week.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com





2011-11-04 14:02:23

by Chuck Lever III

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64


On Nov 4, 2011, at 9:20 AM, Jim Rees wrote:

> As for a fix... we're trying to move away from udp transport anyway. Maybe
> someone should figure out a way to get it to work with tcp? I have zero
> experience with nfs over udp, at least on linux. Just for fun, have you
> tried tcp transport (proto=tcp)?

TCP is a real problem in this environment, because it deals poorly with NIC initialization timing issues. UDP is still the best approach (as long as it is retransmitting appropriately). To support TCP, ultimately what we need to do is to introduce serialization to make the kernel wait for the NIC to become ready before attempting network activity.

> As we move toward nfs4 someone will have to give some thought to nfsroot.
> It's hard to imagine we could put enough nfs4 cruft into the kernel (gssd,
> idmapd) to make it work.

A kernel-level basic id mapper is being considered. That would allow NFSv4 with AUTH_SYS, if we can get the NIC problems squared away.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com





2011-11-04 15:46:44

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> ? > As for a fix...? we're trying to move away from udp transport

> anyway.? Maybe
> ? > someone should figure out a way to get it to work with tcp?? I have zero
> ? > experience with nfs over udp, at least on linux.? Just for fun, have you
> ? > tried tcp transport (proto=tcp)?
> ?
> ? TCP is a real problem in this environment, because it deals poorly with
> ? NIC initialization timing issues.? UDP is still the best approach (as long
> ? as it is retransmitting appropriately).? To support TCP, ultimately what
> ? we need to do is to introduce serialization to make the kernel wait for
> ? the NIC to become ready before attempting network activity.
>
> Agreed, but we're grasping at straws here, and this guy just wants it to
> work.? It's something to try.

I've tried that but "proto=tcp" is no way to change the protocol in linux-2.6.39.4.
It uses the udp protocol and in "Documentation/filesystems/nfs/nfsroot.txt" there's no "proto" option (or something similar).
[11140.866623] IP-Config: Complete:
[11140.866692]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
[11140.866895]????? host=cluster2, domain=, nis-domain=(none),
[11140.867008]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[11140.867431] Root-NFS: nfsroot=/srv/nfs/cluster2,proto=tcp,retrans=10,timeo=60
[11140.867728] NFS: nfs mount opts='udp,proto=tcp,retrans=10,timeo=60,nolock,addr=137.226.167.241'
[11140.867797] NFS:?? parsing nfs mount option 'udp'
[11140.867836] NFS:?? parsing nfs mount option 'proto=tcp'
[11140.867895] NFS:?? parsing nfs mount option 'retrans=10'
[11140.867946] NFS:?? parsing nfs mount option 'timeo=60'
[11140.867996] NFS:?? parsing nfs mount option 'nolock'
[11140.868043] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[11140.868106] NFS: MNTPATH: '/srv/nfs/cluster2'
[11140.868142] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[11141.912761] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[11141.933177] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[11143.873095] NFS: failed to create MNT RPC client, status=-65
[11143.873205] NFS: unable to mount server 137.226.167.241, error -65
[11143.873339] VFS: Unable to mount root fs via NFS, trying floppy.
[11143.873494] VFS: Insert root floppy and press ENTER

Now I'll try the other workaround idea: netboot from tftp (to bring eth0 up) and subsequent nfsroot mount.
Therefore I need a customized (and rather small) kernel...

Regards,
Lukas

2011-11-04 23:40:04

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> As for a patch: no-one can write a patch unless we understand precisely why the

> first RPC fails.? I already explained how to add a line or two to
> fs/nfs/nfsroot.c to give us more information.? If you need a patch to do this, I
> can send one later today.


Maybe that's also helpful for you because eth0 seems to be up before NFS tries to mount the NFSROOT.
BTW: Thanks for forwarding to the right mailing list!

The debug output (with NFSDBG_ALL and RPCDBG_ALL) of linux-3.1:
---
{0} ok boot net
Boot device: /pci@0/pci@0/pci@1/pci@0/pci@2/network@0? File and args: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
1000 Mbps full duplex? Link up
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
|
[??? 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
[??? 0.000000] PROMLIB: Root node compatible: sun4v
[??? 0.000000] Initializing cgroup subsys cpuset
[??? 0.000000] Initializing cgroup subsys cpu
[??? 0.000000] Linux version 3.1.0-razik-2011-11-05 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #3 SMP Sat Nov 5 00:24:24 CET 2011
[??? 0.000000] bootconsole [earlyprom0] enabled
[??? 0.000000] ARCH: SUN4V
[??? 0.000000] Ethernet address: 00:14:4f:98:09:c2
[??? 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
[??? 0.000000] Remapping the kernel... done.
[??? 0.000000] OF stdout device is: /virtual-devices@100/console@1
[??? 0.000000] PROM: Built device tree with 123763 bytes of memory.
[??? 0.000000] MDESC: Size is 58816 bytes.
[??? 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
[??? 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
[??? 0.000000] PLATFORM: hostid [849809c2]
[??? 0.000000] PLATFORM: serial# [00ab4130]
[??? 0.000000] PLATFORM: stick-frequency [5458c3a0]
[??? 0.000000] PLATFORM: mac-address [144f9809c2]
[??? 0.000000] PLATFORM: watchdog-resolution [1000 ms]
[??? 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
[??? 0.000000] PLATFORM: max-cpus [64]
[??? 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c2000
[??? 0.000000] Memory hole size: 132MB
[??? 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
[??? 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
[??? 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
[??? 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
[??? 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
[??? 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
[??? 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
[??? 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
[??? 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
[??? 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
[??? 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
[??? 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
[??? 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
[??? 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
[??? 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
[??? 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
[??? 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
[??? 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
[??? 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
[??? 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
[??? 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
[??? 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
[??? 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
[??? 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
[??? 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
[??? 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
[??? 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
[??? 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
[??? 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
[??? 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
[??? 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
[??? 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
[??? 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
[??? 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
[??? 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
[??? 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
[??? 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
[??? 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
[??? 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
[??? 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
[??? 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
[??? 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
[??? 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
[??? 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
[??? 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
[??? 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
[??? 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
[??? 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
[??? 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
[??? 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
[??? 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
[??? 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
[??? 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
[??? 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
[??? 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
[??? 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
[??? 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
[??? 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
[??? 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
[??? 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
[??? 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
[??? 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
[??? 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
[??? 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
[??? 0.000000] Zone PFN ranges:
[??? 0.000000]?? Normal?? 0x00004200 -> 0x003ffd8e
[??? 0.000000] Movable zone start PFN for each node
[??? 0.000000] early_node_map[3] active PFN ranges
[??? 0.000000]???? 0: 0x00004200 -> 0x003ffd47
[??? 0.000000]???? 0: 0x003ffd50 -> 0x003ffd63
[??? 0.000000]???? 0: 0x003ffd87 -> 0x003ffd8e
[??? 0.000000] Booting Linux...
[??? 0.000000] CPU CAPS: [flush,stbar,swap,muldiv,v9,blkinit,n2,mul32]
[??? 0.000000] CPU CAPS: [div32,v8plus,popc,vis,vis2,ASIBlkInit]
[??? 0.000000] PERCPU: Embedded 5 pages/cpu @fffff80019000000 s10816 r8192 d21952 u65536
[??? 0.000000] SUN4V: Mondo queue sizes [cpu(8192) dev(16384) r(8192) nr(256)]
[??? 0.000000] Built 1 zonelists in Zone order, mobility grouping on.? Total pages: 4144105
[??? 0.000000] Kernel command line: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
[??? 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
[??? 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
[??? 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
[??? 0.000000] Memory: 33091944k available (3568k kernel code, 1272k data, 216k init) [fffff80000000000,00000007ffb1c000]
[??? 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[??? 0.000000] Hierarchical RCU implementation.
[??? 0.000000]? CONFIG_RCU_FANOUT set to non-default value of 32
[??? 0.000000] NR_IRQS:255
[??? 0.000000] clocksource: mult[b4e7cf] shift[24]
[??? 0.000000] clockevent: mult[b5221ba3] shift[31]
[??? 0.000000] Console: colour dummy device 80x25
[??? 0.000000] console [tty0] enabled, bootconsole disabled
[??? 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
[??? 0.000000] PROMLIB: Root node compatible: sun4v
[??? 0.000000] Initializing cgroup subsys cpuset
[??? 0.000000] Initializing cgroup subsys cpu
[??? 0.000000] Linux version 3.1.0-razik-2011-11-05 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #3 SMP Sat Nov 5 00:24:24 CET 2011
[??? 0.000000] bootconsole [earlyprom0] enabled
[??? 0.000000] ARCH: SUN4V
[??? 0.000000] Ethernet address: 00:14:4f:98:09:c2
[??? 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
[??? 0.000000] Remapping the kernel... done.
[??? 0.000000] OF stdout device is: /virtual-devices@100/console@1
[??? 0.000000] PROM: Built device tree with 123763 bytes of memory.
[??? 0.000000] MDESC: Size is 58816 bytes.
[??? 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
[??? 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
[??? 0.000000] PLATFORM: hostid [849809c2]
[??? 0.000000] PLATFORM: serial# [00ab4130]
[??? 0.000000] PLATFORM: stick-frequency [5458c3a0]
[??? 0.000000] PLATFORM: mac-address [144f9809c2]
[??? 0.000000] PLATFORM: watchdog-resolution [1000 ms]
[??? 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
[??? 0.000000] PLATFORM: max-cpus [64]
[??? 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c2000
[??? 0.000000] Memory hole size: 132MB
[??? 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
[??? 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
[??? 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
[??? 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
[??? 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
[??? 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
[??? 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
[??? 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
[??? 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
[??? 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
[??? 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
[??? 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
[??? 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
[??? 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
[??? 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
[??? 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
[??? 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
[??? 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
[??? 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
[??? 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
[??? 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
[??? 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
[??? 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
[??? 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
[??? 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
[??? 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
[??? 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
[??? 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
[??? 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
[??? 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
[??? 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
[??? 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
[??? 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
[??? 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
[??? 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
[??? 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
[??? 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
[??? 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
[??? 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
[??? 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
[??? 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
[??? 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
[??? 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
[??? 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
[??? 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
[??? 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
[??? 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
[??? 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
[??? 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
[??? 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
[??? 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
[??? 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
[??? 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
[??? 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
[??? 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
[??? 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
[??? 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
[??? 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
[??? 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
[??? 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
[??? 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
[??? 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
[??? 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
[??? 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
[??? 0.000000] Zone PFN ranges:
[??? 0.000000]?? Normal?? 0x00004200 -> 0x003ffd8e
[??? 0.000000] Movable zone start PFN for each node
[??? 0.000000] early_node_map[3] active PFN ranges
[??? 0.000000]???? 0: 0x00004200 -> 0x003ffd47
[??? 0.000000]???? 0: 0x003ffd50 -> 0x003ffd63
[??? 0.000000]???? 0: 0x003ffd87 -> 0x003ffd8e
[??? 0.000000] Booting Linux...
[??? 0.000000] CPU CAPS: [flush,stbar,swap,muldiv,v9,blkinit,n2,mul32]
[??? 0.000000] CPU CAPS: [div32,v8plus,popc,vis,vis2,ASIBlkInit]
[??? 0.000000] PERCPU: Embedded 5 pages/cpu @fffff80019000000 s10816 r8192 d21952 u65536
[??? 0.000000] SUN4V: Mondo queue sizes [cpu(8192) dev(16384) r(8192) nr(256)]
[??? 0.000000] Built 1 zonelists in Zone order, mobility grouping on.? Total pages: 4144105
[??? 0.000000] Kernel command line: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
[??? 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
[??? 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
[??? 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
[??? 0.000000] Memory: 33091944k available (3568k kernel code, 1272k data, 216k init) [fffff80000000000,00000007ffb1c000]
[??? 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[??? 0.000000] Hierarchical RCU implementation.
[??? 0.000000]? CONFIG_RCU_FANOUT set to non-default value of 32
[??? 0.000000] NR_IRQS:255
[??? 0.000000] clocksource: mult[b4e7cf] shift[24]
[??? 0.000000] clockevent: mult[b5221ba3] shift[31]
[??? 0.000000] Console: colour dummy device 80x25
[??? 0.000000] console [tty0] enabled, bootconsole disabled
[ 1349.933990] Calibrating delay using timer specific routine.. 2832.78 BogoMIPS (lpj=5665561)
[ 1349.934023] pid_max: default: 65536 minimum: 512
[ 1349.934487] Mount-cache hash table entries: 512
[ 1349.935761] Initializing cgroup subsys cpuacct
[ 1349.935872] Initializing cgroup subsys devices
[ 1349.935890] Initializing cgroup subsys freezer
[ 1349.935907] Initializing cgroup subsys net_cls
[ 1349.936185] Performance events: Supported PMU type is 'niagara2'
[ 1349.958675] Brought up 64 CPUs
[ 1349.958734] Testing NMI watchdog ... OK.
[ 1350.041367] devtmpfs: initialized
[ 1350.041863] ldc.c:v1.1 (July 22, 2008)
[ 1350.042167] print_constraints: dummy:
[ 1350.042552] NET: Registered protocol family 16
[ 1350.051781] VIO: Adding device channel-devices
[ 1350.051858] VIO: Adding device vldc-port-3-0
[ 1350.051918] VIO: Adding device vldc-port-3-1
[ 1350.051978] VIO: Adding device vldc-port-3-2
[ 1350.052048] VIO: Adding device vldc-port-3-3
[ 1350.052110] VIO: Adding device vldc-port-3-4
[ 1350.052171] VIO: Adding device vldc-port-3-5
[ 1350.052233] VIO: Adding device vldc-port-2-0
[ 1350.052296] VIO: Adding device vldc-port-0-0
[ 1350.052365] VIO: Adding device vldc-port-0-1
[ 1350.052433] VIO: Adding device vldc-port-0-2
[ 1350.052511] VIO: Adding device vldc-port-1-0
[ 1350.052581] VIO: Adding device vldc-port-3-7
[ 1350.052654] VIO: Adding device vldc-port-3-8
[ 1350.052737] VIO: Adding device ds-1
[ 1350.052801] VIO: Adding device ds-0
[ 1350.057406] pci_sun4v: Registered hvapi major[1] minor[0]
[ 1350.057671] /pci@0: SUN4V PCI Bus Module
[ 1350.057690] /pci@0: On NUMA node -1
[ 1350.057711] /pci@0: PCI IO[c0f0000000] MEM[c100000000]
[ 1350.057739] /pci@0: Unable to request IOMMU resource.
[ 1350.102075] /pci@0: Imported 3 TSB entries from OBP
[ 1350.103612] /pci@0: MSI Queue first[0] num[36] count[128] devino[0x18]
[ 1350.103640] /pci@0: MSI first[0] num[256] mask[0xff] width[32]
[ 1350.103665] /pci@0: MSI addr32[0x7fff0000:0x10000] addr64[0x3ffff0000:0x10000]
[ 1350.103694] /pci@0: MSI queues at RA [00000007fe580000]
[ 1350.103718] PCI: Scanning PBM /pci@0
[ 1350.115563] ds.c:v1.0 (Jul 11, 2007)
[ 1350.118931] bio: create slab <bio-0> at 0
[ 1350.119516] vgaarb: loaded
[ 1350.120349] Switching to clocksource stick
[ 1350.120981] Switched to NOHz mode on CPU #12
[ 1350.120991] Switched to NOHz mode on CPU #24
[ 1350.121098] Switched to NOHz mode on CPU #36
[ 1350.121150] Switched to NOHz mode on CPU #59
[ 1350.121253] Switched to NOHz mode on CPU #1
[ 1350.121263] Switched to NOHz mode on CPU #48
[ 1350.121272] Switched to NOHz mode on CPU #13
[ 1350.121331] Switched to NOHz mode on CPU #25
[ 1350.121457] Switched to NOHz mode on CPU #37
[ 1350.121502] Switched to NOHz mode on CPU #60
[ 1350.121608] Switched to NOHz mode on CPU #14
[ 1350.121623] Switched to NOHz mode on CPU #49
[ 1350.121658] Switched to NOHz mode on CPU #2
[ 1350.121670] Switched to NOHz mode on CPU #26
[ 1350.121798] Switched to NOHz mode on CPU #38
[ 1350.121894] Switched to NOHz mode on CPU #61
[ 1350.121903] Switched to NOHz mode on CPU #0
[ 1350.121942] Switched to NOHz mode on CPU #15
[ 1350.121960] Switched to NOHz mode on CPU #50
[ 1350.121969] Switched to NOHz mode on CPU #3
[ 1350.122011] Switched to NOHz mode on CPU #27
[ 1350.122138] Switched to NOHz mode on CPU #39
[ 1350.122204] Switched to NOHz mode on CPU #62
[ 1350.122277] Switched to NOHz mode on CPU #16
[ 1350.122298] Switched to NOHz mode on CPU #4
[ 1350.122307] Switched to NOHz mode on CPU #51
[ 1350.122351] Switched to NOHz mode on CPU #28
[ 1350.122480] Switched to NOHz mode on CPU #40
[ 1350.122564] Switched to NOHz mode on CPU #63
[ 1350.122623] Switched to NOHz mode on CPU #17
[ 1350.122640] Switched to NOHz mode on CPU #5
[ 1350.122653] Switched to NOHz mode on CPU #52
[ 1350.122697] Switched to NOHz mode on CPU #29
[ 1350.122828] Switched to NOHz mode on CPU #41
[ 1350.122954] Switched to NOHz mode on CPU #18
[ 1350.122971] Switched to NOHz mode on CPU #6
[ 1350.123029] Switched to NOHz mode on CPU #53
[ 1350.123038] Switched to NOHz mode on CPU #30
[ 1350.123175] Switched to NOHz mode on CPU #42
[ 1350.123286] Switched to NOHz mode on CPU #19
[ 1350.123298] Switched to NOHz mode on CPU #7
[ 1350.123382] Switched to NOHz mode on CPU #31
[ 1350.123400] Switched to NOHz mode on CPU #54
[ 1350.123524] Switched to NOHz mode on CPU #43
[ 1350.123602] Switched to NOHz mode on CPU #8
[ 1350.123620] Switched to NOHz mode on CPU #20
[ 1350.123729] Switched to NOHz mode on CPU #32
[ 1350.123749] Switched to NOHz mode on CPU #55
[ 1350.123860] Switched to NOHz mode on CPU #44
[ 1350.123929] Switched to NOHz mode on CPU #9
[ 1350.123971] Switched to NOHz mode on CPU #21
[ 1350.124065] Switched to NOHz mode on CPU #33
[ 1350.124095] Switched to NOHz mode on CPU #56
[ 1350.124209] Switched to NOHz mode on CPU #45
[ 1350.124268] Switched to NOHz mode on CPU #10
[ 1350.124303] Switched to NOHz mode on CPU #22
[ 1350.124402] Switched to NOHz mode on CPU #34
[ 1350.124442] Switched to NOHz mode on CPU #57
[ 1350.124564] Switched to NOHz mode on CPU #46
[ 1350.124737] Switched to NOHz mode on CPU #58
[ 1350.124747] Switched to NOHz mode on CPU #23
[ 1350.124756] Switched to NOHz mode on CPU #47
[ 1350.124766] Switched to NOHz mode on CPU #11
[ 1350.124776] Switched to NOHz mode on CPU #35
[ 1350.129462] NET: Registered protocol family 2
[ 1350.129995] IP route cache hash table entries: 524288 (order: 9, 4194304 bytes)
[ 1350.132183] TCP established hash table entries: 524288 (order: 10, 8388608 bytes)
[ 1350.132224] ds-1: Registered pri service.
[ 1350.132328] ds-1: Registered var-config-backup service.
[ 1350.157752] TCP bind hash table entries: 65536 (order: 7, 1048576 bytes)
[ 1350.161496] TCP: Hash tables configured (established 524288 bind 65536)
[ 1350.161523] TCP reno registered
[ 1350.161562] UDP hash table entries: 16384 (order: 6, 524288 bytes)
[ 1350.163563] UDP-Lite hash table entries: 16384 (order: 6, 524288 bytes)
[ 1350.166529] NET: Registered protocol family 1
[ 1350.166613] RPC:?????? creating workqueue rpciod
[ 1350.166921] RPC:?????? registering /proc/net/rpc
[ 1350.167024] svc: Adding svc transport class 'tcp'
[ 1350.167051] svc: Adding svc transport class 'udp'
[ 1350.167121] RPC: Registered named UNIX socket transport module.
[ 1350.167145] RPC: Registered udp transport module.
[ 1350.167165] RPC: Registered tcp transport module.
[ 1350.167185] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 1350.168544] audit: initializing netlink socket (disabled)
[ 1350.168591] type=2000 audit(0.304:1): initialized
[ 1350.872401] HugeTLB registered 4 MB page size, pre-allocated 0 pages
[ 1350.881157] VFS: Disk quotas dquot_6.5.2
[ 1350.881602] Dquot-cache hash table entries: 1024 (order 0, 8192 bytes)
[ 1350.882234] RPC:?????? creating workqueue nfsiod
[ 1350.883014] RPC:?????? registering /proc/net/rpc/nfs
[ 1350.883068] msgmni has been set to 32768
[ 1350.884176] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[ 1350.884211] io scheduler noop registered
[ 1350.884230] io scheduler deadline registered
[ 1350.884466] io scheduler cfq registered (default)
[ 1350.885978] f027ab74: ttyS0 at I/O 0x0 (irq = 17) is a SUN4V HCONS
[ 1351.101139] console [ttyHV0] enabled
[ 1351.103812] f028af14: ttyS1 at MMIO 0xfff0ca0000 (irq = 24) is a 16550A
[ 1351.104504] [drm] Initialized drm 1.1.0 20060810
[ 1351.105715] e1000e: Intel(R) PRO/1000 Network Driver - 1.4.4-k
[ 1351.105850] e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
[ 1351.105948] e1000e 0000:08:00.0: Disabling ASPM? L1
[ 1351.279997] e1000e 0000:08:00.0: eth0: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c2
[ 1351.280059] e1000e 0000:08:00.0: eth0: Intel(R) PRO/1000 Network Connection
[ 1351.280186] e1000e 0000:08:00.0: eth0: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[ 1351.280267] e1000e 0000:08:00.1: Disabling ASPM? L1
[ 1351.457748] e1000e 0000:08:00.1: eth1: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c3
[ 1351.457809] e1000e 0000:08:00.1: eth1: Intel(R) PRO/1000 Network Connection
[ 1351.457935] e1000e 0000:08:00.1: eth1: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[ 1351.458003] e1000e 0000:09:00.0: Disabling ASPM? L1
[ 1351.632096] e1000e 0000:09:00.0: eth2: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c4
[ 1351.632157] e1000e 0000:09:00.0: eth2: Intel(R) PRO/1000 Network Connection
[ 1351.632283] e1000e 0000:09:00.0: eth2: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[ 1351.632351] e1000e 0000:09:00.1: Disabling ASPM? L1
[ 1351.809926] e1000e 0000:09:00.1: eth3: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c5
[ 1351.810047] e1000e 0000:09:00.1: eth3: Intel(R) PRO/1000 Network Connection
[ 1351.810231] e1000e 0000:09:00.1: eth3: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[ 1351.810858] mousedev: PS/2 mouse device common for all mice
[ 1351.811886] rtc-sun4v rtc-sun4v: rtc core: registered sun4v as rtc0
[ 1351.812592] TCP cubic registered
[ 1351.813469] NET: Registered protocol family 10
[ 1351.815597] Mobile IPv6
[ 1351.815633] NET: Registered protocol family 17
[ 1351.815704] Registering the dns_resolver key type
[ 1351.816174] registered taskstats version 1
[ 1351.818649] rtc-sun4v rtc-sun4v: setting system clock to 2011-11-04 23:31:46 UTC (1320449506)
[ 1352.067344] ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1354.200067] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[ 1354.222397] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1354.243385] IP-Config: Complete:
[ 1354.243457]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=255.255.255.255,
[ 1354.243659]????? host=137.226.167.242, domain=, nis-domain=(none),
[ 1354.243774]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[ 1354.244257] Root-NFS: nfsroot=/srv/nfs/cluster2
[ 1354.244620] NFS: nfs mount opts='vers=2,udp,rsize=4096,wsize=4096,nolock,addr=137.226.167.241'
[ 1354.244710] NFS:?? parsing nfs mount option 'vers=2'
[ 1354.244770] NFS:?? parsing nfs mount option 'udp'
[ 1354.244834] NFS:?? parsing nfs mount option 'rsize=4096'
[ 1354.244884] NFS:?? parsing nfs mount option 'wsize=4096'
[ 1354.244934] NFS:?? parsing nfs mount option 'nolock'
[ 1354.244981] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[ 1354.245045] NFS: MNTPATH: '/srv/nfs/cluster2'
[ 1354.245080] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[ 1354.245157] RPC:?????? set up xprt to 137.226.167.241 (autobind) via udp
[ 1354.245236] RPC:?????? created transport fffff807fdf40800 with 16 slots
[ 1354.245285] RPC:?????? creating mount client for 137.226.167.241 (xprt fffff807fdf40800)
[ 1354.245346] RPC:?????? creating UNIX authenticator for client fffff807fdf1a200
[ 1354.245411] RPC:?????? new task initialized, procpid 1
[ 1354.245456] RPC:?????? allocated task fffff807fdf10000
[ 1354.245504] RPC:???? 1 __rpc_execute flags=0x680
[ 1354.245539] RPC:???? 1 call_start mount1 proc NULL (sync)
[ 1354.245614] RPC:???? 1 call_reserve (status 0)
[ 1354.245649] RPC:???? 1 reserved req fffff807fdf1a000 xid 51d47d10
[ 1354.245698] RPC:?????? wake_up_next(fffff807fdf40978 "xprt_sending")
[ 1354.245747] RPC:???? 1 call_reserveresult (status 0)
[ 1354.245792] RPC:???? 1 call_refresh (status 0)
[ 1354.245825] RPC:???? 1 holding NULL cred 00000000008b8cf0
[ 1354.245871] RPC:???? 1 refreshing NULL cred 00000000008b8cf0
[ 1354.245917] RPC:???? 1 call_refreshresult (status 0)
[ 1354.245962] RPC:???? 1 call_allocate (status 0)
[ 1354.246025] RPC:???? 1 allocated buffer of size 96 at fffff807fdf41000
[ 1354.246073] RPC:???? 1 call_bind (status 0)
[ 1354.246108] RPC:???? 1 rpcb_getport_async(137.226.167.241, 100005, 1, 17)
[ 1354.246157] RPC:???? 1 sleep_on(queue "xprt_binding" time 4294893393)
[ 1354.246206] RPC:???? 1 added to queue fffff807fdf408c8 "xprt_binding"
[ 1354.246253] RPC:???? 1 setting alarm for 60000 ms
[ 1354.246291] RPC:???? 1 rpcb_getport_async: trying rpcbind version 2
[ 1354.246366] RPC:?????? set up xprt to 137.226.167.241 (port 111) via udp
[ 1354.246445] RPC:?????? created transport fffff807fdf41800 with 16 slots
[ 1354.246495] RPC:?????? creating rpcbind client for 137.226.167.241 (xprt fffff807fdf41800)
[ 1354.246554] RPC:?????? creating UNIX authenticator for client fffff807fdf1c400
[ 1354.246612] RPC:?????? new task initialized, procpid 1
[ 1354.246656] RPC:?????? allocated task fffff807fdf100e0
[ 1354.246708] RPC:?????? rpc_release_client(fffff807fdf1c400)
[ 1354.246756] RPC:???? 1 sync task going to sleep
[ 1354.246825] RPC:???? 2 __rpc_execute flags=0x681
[ 1354.246860] RPC:???? 2 call_start rpcbind2 proc GETPORT (async)
[ 1354.246921] RPC:???? 2 call_reserve (status 0)
[ 1354.246957] RPC:???? 2 reserved req fffff807fdf1c200 xid 18460f63
[ 1354.247005] RPC:?????? wake_up_next(fffff807fdf41978 "xprt_sending")
[ 1354.247054] RPC:???? 2 call_reserveresult (status 0)
[ 1354.247098] RPC:???? 2 call_refresh (status 0)
[ 1354.247130] RPC:???? 2 looking up UNIX cred
[ 1354.247189] RPC:?????? looking up UNIX cred
[ 1354.247223] RPC:?????? allocating UNIX cred for uid 0 gid 0
[ 1354.247270] RPC:???? 2 refreshing UNIX cred fffff807fdefc900
[ 1354.247317] RPC:???? 2 call_refreshresult (status 0)
[ 1354.247361] RPC:???? 2 call_allocate (status 0)
[ 1354.247395] RPC:???? 2 allocated buffer of size 416 at fffff807fdf42000
[ 1354.247443] RPC:???? 2 call_bind (status 0)
[ 1354.247476] RPC:???? 2 call_connect xprt fffff807fdf41800 is not connected
[ 1354.247526] RPC:???? 2 xprt_connect xprt fffff807fdf41800 is not connected
[ 1354.247604] RPC:???? 2 xprt_cwnd_limited cong = 0 cwnd = 256
[ 1354.247651] RPC:???? 2 sleep_on(queue "xprt_pending" time 4294893394)
[ 1354.247700] RPC:???? 2 added to queue fffff807fdf41a28 "xprt_pending"
[ 1354.247747] RPC:???? 2 setting alarm for 5000 ms
[ 1354.247780] RPC:?????? xs_connect scheduled xprt fffff807fdf41800
[ 1354.247922] RPC:?????? xs_bind 0.0.0.0:0: ok (0)
[ 1354.247988] RPC:?????? worker connecting xprt fffff807fdf41800 via udp to 137.226.167.241 (port 111)
[ 1354.248050] RPC:???? 2 __rpc_wake_up_task (now 4294893394)
[ 1354.248094] RPC:???? 2 disabling timer
[ 1354.248125] RPC:???? 2 removed from queue fffff807fdf41a28 "xprt_pending"
[ 1354.248172] RPC:?????? __rpc_wake_up_task done
[ 1354.248205] RPC:???? 2 __rpc_execute flags=0x681
[ 1354.248238] RPC:???? 2 xprt_connect_status: connection established
[ 1354.248285] RPC:???? 2 call_connect_status (status 0)
[ 1354.248329] RPC:???? 2 call_transmit (status 0)
[ 1354.248389] RPC:???? 2 xprt_prepare_transmit
[ 1354.248421] RPC:???? 2 rpc_xdr_encode (status 0)
[ 1354.248455] RPC:???? 2 marshaling UNIX cred fffff807fdefc900
[ 1354.248503] RPC:???? 2 using AUTH_UNIX cred fffff807fdefc900 to wrap rpc data
[ 1354.248556] RPC:???? 2 encoding PMAP_GETPORT call (100005, 1, 17, 0)
[ 1354.248603] RPC:???? 2 xprt_transmit(92)
[ 1354.253576] RPC:?????? xs_udp_send_request(92) = 92
[ 1354.253611] RPC:???? 2 xmit complete
[ 1354.253644] RPC:???? 2 sleep_on(queue "xprt_pending" time 4294893395)
[ 1354.253692] RPC:???? 2 added to queue fffff807fdf41a28 "xprt_pending"
[ 1354.253739] RPC:???? 2 setting alarm for 5000 ms
[ 1359.265758] RPC:???? 2 timeout
[ 1359.265818] RPC:???? 2 __rpc_wake_up_task (now 4294894648)
[ 1359.265920] RPC:???? 2 disabling timer
[ 1359.265981] RPC:???? 2 removed from queue fffff807fdf41a28 "xprt_pending"
[ 1359.266086] RPC:?????? __rpc_wake_up_task done
[ 1359.266172] RPC:???? 2 __rpc_execute flags=0xe81
[ 1359.266205] RPC:???? 2 xprt_timer
[ 1359.266237] RPC:?????? cong 256, cwnd was 256, now 256
[ 1359.266281] RPC:?????? wake_up_next(fffff807fdf41978 "xprt_sending")
[ 1359.266358] RPC:???? 2 call_status (status -60)
[ 1359.266391] RPC:???? 2 call_timeout (minor)
[ 1359.266452] RPC:???? 2 call_bind (status 0)
[ 1359.266484] RPC:???? 2 call_connect xprt fffff807fdf41800 is connected
[ 1359.266560] RPC:???? 2 call_transmit (status 0)
[ 1359.266591] RPC:???? 2 xprt_prepare_transmit
[ 1359.266651] RPC:???? 2 xprt_cwnd_limited cong = 0 cwnd = 256
[ 1359.266697] RPC:???? 2 rpc_xdr_encode (status 0)
[ 1359.266758] RPC:???? 2 marshaling UNIX cred fffff807fdefc900
[ 1359.266805] RPC:???? 2 using AUTH_UNIX cred fffff807fdefc900 to wrap rpc data
[ 1359.266886] RPC:???? 2 encoding PMAP_GETPORT call (100005, 1, 17, 0)
[ 1359.266961] RPC:???? 2 xprt_transmit(92)
[ 1359.267005] RPC:?????? xs_udp_send_request(92) = 92
[ 1359.267067] RPC:???? 2 xmit complete
[ 1359.267128] RPC:???? 2 sleep_on(queue "xprt_pending" time 4294894648)
[ 1359.267176] RPC:???? 2 added to queue fffff807fdf41a28 "xprt_pending"
[ 1359.267251] RPC:???? 2 setting alarm for 10000 ms
[ 1369.287139] RPC:???? 2 timeout
[ 1369.287199] RPC:???? 2 __rpc_wake_up_task (now 4294897152)
[ 1369.287301] RPC:???? 2 disabling timer
[ 1369.287361] RPC:???? 2 removed from queue fffff807fdf41a28 "xprt_pending"
[ 1369.287466] RPC:?????? __rpc_wake_up_task done
[ 1369.287552] RPC:???? 2 __rpc_execute flags=0xe81
[ 1369.287586] RPC:???? 2 xprt_timer
[ 1369.287616] RPC:?????? cong 256, cwnd was 256, now 256
[ 1369.287661] RPC:?????? wake_up_next(fffff807fdf41978 "xprt_sending")
[ 1369.287709] RPC:???? 2 call_status (status -60)
[ 1369.287770] RPC:???? 2 call_timeout (minor)
[ 1369.287801] RPC:???? 2 call_bind (status 0)
[ 1369.287833] RPC:???? 2 call_connect xprt fffff807fdf41800 is connected
[ 1369.287909] RPC:???? 2 call_transmit (status 0)
[ 1369.287969] RPC:???? 2 xprt_prepare_transmit
[ 1369.288001] RPC:???? 2 xprt_cwnd_limited cong = 0 cwnd = 256
[ 1369.288076] RPC:???? 2 rpc_xdr_encode (status 0)
[ 1369.288108] RPC:???? 2 marshaling UNIX cred fffff807fdefc900
[ 1369.288184] RPC:???? 2 using AUTH_UNIX cred fffff807fdefc900 to wrap rpc data
[ 1369.288236] RPC:???? 2 encoding PMAP_GETPORT call (100005, 1, 17, 0)
[ 1369.288311] RPC:???? 2 xprt_transmit(92)
[ 1369.288384] RPC:?????? xs_udp_send_request(92) = 92
[ 1369.288418] RPC:???? 2 xmit complete
[ 1369.288478] RPC:???? 2 sleep_on(queue "xprt_pending" time 4294897152)
[ 1369.288555] RPC:???? 2 added to queue fffff807fdf41a28 "xprt_pending"
[ 1369.288601] RPC:???? 2 setting alarm for 20000 ms
[ 1389.330272] RPC:???? 2 timeout
[ 1389.330333] RPC:???? 2 __rpc_wake_up_task (now 4294902160)
[ 1389.330434] RPC:???? 2 disabling timer
[ 1389.330495] RPC:???? 2 removed from queue fffff807fdf41a28 "xprt_pending"
[ 1389.330600] RPC:?????? __rpc_wake_up_task done
[ 1389.330686] RPC:???? 2 __rpc_execute flags=0xe81
[ 1389.330719] RPC:???? 2 xprt_timer
[ 1389.330750] RPC:?????? cong 256, cwnd was 256, now 256
[ 1389.330794] RPC:?????? wake_up_next(fffff807fdf41978 "xprt_sending")
[ 1389.330871] RPC:???? 2 call_status (status -60)
[ 1389.330903] RPC:???? 2 call_timeout (major)
[ 1389.330965] RPC:?????? setting port for xprt fffff807fdf40800 to 0
[ 1389.331015] RPC:???? 2 rpcb_getport_done(status -60, port 0)
[ 1389.331090] RPC:???? 2 return 0, status -60
[ 1389.331120] RPC:???? 2 release task
[ 1389.331182] RPC:?????? freeing buffer of size 416 at fffff807fdf42000
[ 1389.331261] RPC:???? 2 release request fffff807fdf1c200
[ 1389.331306] RPC:?????? wake_up_next(fffff807fdf41ad8 "xprt_backlog")
[ 1389.331382] RPC:?????? rpc_release_client(fffff807fdf1c400)
[ 1389.331456] RPC:?????? destroying rpcbind client for 137.226.167.241
[ 1389.331504] RPC:?????? destroying transport fffff807fdf41800
[ 1389.331612] RPC:?????? xs_destroy xprt fffff807fdf41800
[ 1389.331707] RPC:?????? xs_close xprt fffff807fdf41800
[ 1389.331795] RPC:?????? disconnected transport fffff807fdf41800
[ 1389.331879] RPC:???? 2 freeing task
[ 1389.331910] RPC:???? 1 __rpc_wake_up_task (now 4294902160)
[ 1389.331983] RPC:???? 1 disabling timer
[ 1389.332043] RPC:???? 1 removed from queue fffff807fdf408c8 "xprt_binding"
[ 1389.332095] RPC:?????? __rpc_wake_up_task done
[ 1389.332111] RPC:???? 1 sync task resuming
[ 1389.332120] RPC:???? 1 rpcbind request timed out
[ 1389.332129] RPC:???? 1 call_timeout (major)
[ 1389.332138] RPC:???? 1 return 0, status -60
[ 1389.332145] RPC:???? 1 release task
[ 1389.332158] RPC:?????? freeing buffer of size 96 at fffff807fdf41000
[ 1389.332171] RPC:???? 1 release request fffff807fdf1a000
[ 1389.332181] RPC:?????? wake_up_next(fffff807fdf40ad8 "xprt_backlog")
[ 1389.332194] RPC:?????? rpc_release_client(fffff807fdf1a200)
[ 1389.332203] RPC:???? 1 freeing task
[ 1389.332213] RPC:?????? shutting down mount client for 137.226.167.241
[ 1389.332222] RPC:?????? rpc_release_client(fffff807fdf1a200)
[ 1389.332232] RPC:?????? destroying UNIX authenticator 00000000008b8d78
[ 1389.332246] RPC:?????? destroying mount client for 137.226.167.241
[ 1389.332259] NFS: failed to create MNT RPC client, status=-60
[ 1389.332271] NFS: unable to mount server 137.226.167.241, error -60
[ 1389.332306] VFS: Unable to mount root fs via NFS, trying floppy.
[ 1389.332444] List of all partitions:
[ 1389.332454] No filesystem could mount root, tried:
[ 1389.332469] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
[ 1389.332480] Call Trace:
[ 1389.332509]? [00000000008c6d18] mount_block_root+0x210/0x228
[ 1389.332523]? [00000000008c6df4] mount_root+0xc4/0xd4
[ 1389.332537]? [00000000008c6f80] prepare_namespace+0x17c/0x1c0
[ 1389.332554]? [00000000008c6250] kernel_init+0x140/0x158
[ 1389.332574]? [000000000042ac68] kernel_thread+0x30/0x48
[ 1389.332593]? [0000000000767840] rest_init+0x18/0x74
[ 1389.332682] Press Stop-A (L1-A) to return to the boot prom
[ 1389.333548] RPC:?????? destroying transport fffff807fdf40800
[ 1389.334301] RPC:?????? xs_destroy xprt fffff807fdf40800
[ 1389.334530] RPC:?????? xs_close xprt fffff807fdf40800
[ 1389.334576] RPC:?????? disconnected transport fffff807fdf40800

2011-11-04 01:00:05

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> I suspect some kind of timing problem, where the network interface hasn't

> come up yet at the time the portmap request is made.? What does your nfsroot
> kernel option look like?? What happens if you add ",port=2049" to the
> end of
> your nfsroot kernel option?

That's my kernel line (and it's exactly the same for all kernel versions I used):
root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242::137.226.167.225:255.255.255.224:cluster2:eth0:off nfsrootdebug
(I know that nfsrootdebug isn't available in older kernels but it doesn't matter...)

With ",port=2049" I get this from linux-2.6.29.4:
[? 252.932407] IP-Config: Complete:
[? 252.932476]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
[? 252.932680]????? host=cluster2, domain=, nis-domain=(none),
[? 252.932794]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[? 252.933230] Root-NFS: nfsroot=/srv/nfs/cluster2,port=2049
[? 252.933523] NFS: nfs mount opts='udp,port=2049,nolock,addr=137.226.167.241'
[? 252.933583] NFS:?? parsing nfs mount option 'udp'
[? 252.933621] NFS:?? parsing nfs mount option 'port=2049'
[? 252.933675] NFS:?? parsing nfs mount option 'nolock'
[? 252.933722] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[? 252.933787] NFS: MNTPATH: '/srv/nfs/cluster2'
[? 252.933823] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[? 254.038874] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[? 254.059251] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[? 288.023684] NFS: failed to create MNT RPC client, status=-60
[? 288.023793] NFS: unable to mount server 137.226.167.241, error -60
[? 288.023929] VFS: Unable to mount root fs via NFS, trying floppy.
[? 288.024092] VFS: Insert root floppy and press ENTER

So the result is the same as before...
In the past I also tried higher "timeo" and I think higher "retrans" values but also without a positive effect...

2011-11-04 23:17:24

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

>> [11140.867895] NFS:? parsing nfs mount option 'retrans=10'

>> [11140.867946] NFS:? parsing nfs mount option 'timeo=60'
>> [11140.867996] NFS:? parsing nfs mount option 'nolock'
>> [11140.868043] NFS:? parsing nfs mount option
> 'addr=137.226.167.241'
>> [11140.868106] NFS: MNTPATH: '/srv/nfs/cluster2'
>> [11140.868142] NFS: sending MNT request for
> 137.226.167.241:/srv/nfs/cluster2
>> [11141.912761] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: Rx
>> [11141.933177] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [11143.873095] NFS: failed to create MNT RPC client, status=-65
>> [11143.873205] NFS: unable to mount server 137.226.167.241, error -65
>
> You got a different result: -65 is EHOSTUNREACH.? And actually, I would have
> expected that error from the UDP case as well.
>

Oh, sorry! I thought it wouldn't have used the option...
Thanks for this hint!

But I've another important fact:
I've tested linux-3.1 and the behaviour is the same as in linux-2.6.39.4 with the difference that eth0 seems to be up when NFS tries to mount NFSROOT. So I see no reason why the kernel can't send the mount request (I see no request in wireshark).

So now I'll enable more debug output in 3.1 and send you the result like I've done it in my last email for 2.6.39.4...

2011-11-04 17:55:43

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> As for a patch: no-one can write a patch unless we understand precisely why the

> first RPC fails.? I already explained how to add a line or two to
> fs/nfs/nfsroot.c to give us more information.? If you need a patch to do this, I
> can send one later today.
>
> I might be able to reproduce it here, now that I understand your set up, but it
> would require building a partial NFSROOT environment.? I can't get to that
> until next week.

OK, it was easier than I thought...
I've set NFSDBG_ALL where you wanted it and rpc_debug = RPCDBG_ALL .

That's the linux-2.6.39.4 output now. Hopefully you can see the problem now...
---
{0} ok boot net??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
Boot device: /pci@0/pci@0/pci@1/pci@0/pci@2/network@0? File and args: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
1000 Mbps full duplex? Link up
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
/
[??? 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
[??? 0.000000] PROMLIB: Root node compatible: sun4v
[??? 0.000000] Initializing cgroup subsys cpuset
[??? 0.000000] Initializing cgroup subsys cpu
[??? 0.000000] Linux version 2.6.39.4-razik-2011-11-04 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #8 SMP Fri Nov 4 18:40:53 CET 2011
[??? 0.000000] bootconsole [earlyprom0] enabled
[??? 0.000000] ARCH: SUN4V
[??? 0.000000] Ethernet address: 00:14:4f:98:09:c2
[??? 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
[??? 0.000000] Remapping the kernel... done.
[??? 0.000000] OF stdout device is: /virtual-devices@100/console@1
[??? 0.000000] PROM: Built device tree with 123763 bytes of memory.
[??? 0.000000] MDESC: Size is 58816 bytes.
[??? 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
[??? 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
[??? 0.000000] PLATFORM: hostid [849809c2]
[??? 0.000000] PLATFORM: serial# [00ab4130]
[??? 0.000000] PLATFORM: stick-frequency [5458c3a0]
[??? 0.000000] PLATFORM: mac-address [144f9809c2]
[??? 0.000000] PLATFORM: watchdog-resolution [1000 ms]
[??? 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
[??? 0.000000] PLATFORM: max-cpus [64]
[??? 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c2000
[??? 0.000000] Memory hole size: 132MB
[??? 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
[??? 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
[??? 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
[??? 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
[??? 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
[??? 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
[??? 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
[??? 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
[??? 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
[??? 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
[??? 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
[??? 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
[??? 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
[??? 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
[??? 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
[??? 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
[??? 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
[??? 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
[??? 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
[??? 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
[??? 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
[??? 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
[??? 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
[??? 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
[??? 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
[??? 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
[??? 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
[??? 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
[??? 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
[??? 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
[??? 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
[??? 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
[??? 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
[??? 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
[??? 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
[??? 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
[??? 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
[??? 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
[??? 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
[??? 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
[??? 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
[??? 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
[??? 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
[??? 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
[??? 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
[??? 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
[??? 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
[??? 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
[??? 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
[??? 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
[??? 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
[??? 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
[??? 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
[??? 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
[??? 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
[??? 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
[??? 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
[??? 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
[??? 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
[??? 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
[??? 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
[??? 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
[??? 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
[??? 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
[??? 0.000000] Zone PFN ranges:
[??? 0.000000]?? Normal?? 0x00004200 -> 0x003ffd8e
[??? 0.000000] Movable zone start PFN for each node
[??? 0.000000] early_node_map[3] active PFN ranges
[??? 0.000000]???? 0: 0x00004200 -> 0x003ffd47
[??? 0.000000]???? 0: 0x003ffd50 -> 0x003ffd63
[??? 0.000000]???? 0: 0x003ffd87 -> 0x003ffd8e
[??? 0.000000] Booting Linux...
[??? 0.000000] PERCPU: Embedded 6 pages/cpu @fffff80019000000 s15616 r8192 d25344 u65536
[??? 0.000000] Built 1 zonelists in Zone order, mobility grouping on.? Total pages: 4144105
[??? 0.000000] Kernel command line: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
[??? 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
[??? 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
[??? 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
[??? 0.000000] Memory: 33091488k available (3528k kernel code, 1248k data, 208k init) [fffff80000000000,00000007ffb1c000]
[??? 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[??? 0.000000] Hierarchical RCU implementation.
[??? 0.000000]? CONFIG_RCU_FANOUT set to non-default value of 32
[??? 0.000000]? RCU-based detection of stalled CPUs is disabled.
[??? 0.000000] NR_IRQS:255
[??? 0.000000] clocksource: mult[5a73e7bb] shift[31]
[??? 0.000000] clockevent: mult[b5221ba3] shift[31]
[??? 0.000000] Console: colour dummy device 80x25
[??? 0.000000] console [tty0] enabled, bootconsole disabled
[??? 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
[??? 0.000000] PROMLIB: Root node compatible: sun4v
[??? 0.000000] Initializing cgroup subsys cpuset
[??? 0.000000] Initializing cgroup subsys cpu
[??? 0.000000] Linux version 2.6.39.4-razik-2011-11-04 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #8 SMP Fri Nov 4 18:40:53 CET 2011
[??? 0.000000] bootconsole [earlyprom0] enabled
[??? 0.000000] ARCH: SUN4V
[??? 0.000000] Ethernet address: 00:14:4f:98:09:c2
[??? 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
[??? 0.000000] Remapping the kernel... done.
[??? 0.000000] OF stdout device is: /virtual-devices@100/console@1
[??? 0.000000] PROM: Built device tree with 123763 bytes of memory.
[??? 0.000000] MDESC: Size is 58816 bytes.
[??? 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
[??? 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
[??? 0.000000] PLATFORM: hostid [849809c2]
[??? 0.000000] PLATFORM: serial# [00ab4130]
[??? 0.000000] PLATFORM: stick-frequency [5458c3a0]
[??? 0.000000] PLATFORM: mac-address [144f9809c2]
[??? 0.000000] PLATFORM: watchdog-resolution [1000 ms]
[??? 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
[??? 0.000000] PLATFORM: max-cpus [64]
[??? 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c2000
[??? 0.000000] Memory hole size: 132MB
[??? 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
[??? 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
[??? 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
[??? 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
[??? 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
[??? 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
[??? 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
[??? 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
[??? 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
[??? 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
[??? 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
[??? 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
[??? 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
[??? 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
[??? 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
[??? 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
[??? 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
[??? 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
[??? 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
[??? 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
[??? 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
[??? 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
[??? 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
[??? 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
[??? 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
[??? 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
[??? 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
[??? 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
[??? 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
[??? 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
[??? 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
[??? 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
[??? 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
[??? 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
[??? 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
[??? 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
[??? 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
[??? 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
[??? 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
[??? 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
[??? 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
[??? 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
[??? 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
[??? 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
[??? 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
[??? 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
[??? 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
[??? 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
[??? 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
[??? 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
[??? 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
[??? 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
[??? 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
[??? 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
[??? 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
[??? 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
[??? 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
[??? 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
[??? 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
[??? 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
[??? 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
[??? 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
[??? 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
[??? 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
[??? 0.000000] Zone PFN ranges:
[??? 0.000000]?? Normal?? 0x00004200 -> 0x003ffd8e
[??? 0.000000] Movable zone start PFN for each node
[??? 0.000000] early_node_map[3] active PFN ranges
[??? 0.000000]???? 0: 0x00004200 -> 0x003ffd47
[??? 0.000000]???? 0: 0x003ffd50 -> 0x003ffd63
[??? 0.000000]???? 0: 0x003ffd87 -> 0x003ffd8e
[??? 0.000000] Booting Linux...
[??? 0.000000] PERCPU: Embedded 6 pages/cpu @fffff80019000000 s15616 r8192 d25344 u65536
[??? 0.000000] Built 1 zonelists in Zone order, mobility grouping on.? Total pages: 4144105
[??? 0.000000] Kernel command line: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2 ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
[??? 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
[??? 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
[??? 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
[??? 0.000000] Memory: 33091488k available (3528k kernel code, 1248k data, 208k init) [fffff80000000000,00000007ffb1c000]
[??? 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[??? 0.000000] Hierarchical RCU implementation.
[??? 0.000000]? CONFIG_RCU_FANOUT set to non-default value of 32
[??? 0.000000]? RCU-based detection of stalled CPUs is disabled.
[??? 0.000000] NR_IRQS:255
[??? 0.000000] clocksource: mult[5a73e7bb] shift[31]
[??? 0.000000] clockevent: mult[b5221ba3] shift[31]
[??? 0.000000] Console: colour dummy device 80x25
[??? 0.000000] console [tty0] enabled, bootconsole disabled
[? 120.892024] Calibrating delay using timer specific routine.. 2832.74 BogoMIPS (lpj=5665497)
[? 120.892057] pid_max: default: 65536 minimum: 512
[? 120.892502] Mount-cache hash table entries: 512
[? 120.893663] Initializing cgroup subsys ns
[? 120.893682] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[? 120.893710] Initializing cgroup subsys cpuacct
[? 120.893817] Initializing cgroup subsys devices
[? 120.893835] Initializing cgroup subsys freezer
[? 120.893851] Initializing cgroup subsys net_cls
[? 120.894117] Performance events: Supported PMU type is 'niagara2'
[? 120.916617] Brought up 64 CPUs
[? 120.916685] Testing NMI watchdog ... OK.
[? 120.998031] devtmpfs: initialized
[? 120.998482] ldc.c:v1.1 (July 22, 2008)
[? 120.998753] print_constraints: dummy:
[? 120.999091] NET: Registered protocol family 16
[? 121.008200] VIO: Adding device channel-devices
[? 121.008276] VIO: Adding device vldc-port-3-0
[? 121.008338] VIO: Adding device vldc-port-3-1
[? 121.008398] VIO: Adding device vldc-port-3-2
[? 121.008457] VIO: Adding device vldc-port-3-3
[? 121.008526] VIO: Adding device vldc-port-3-4
[? 121.008594] VIO: Adding device vldc-port-3-5
[? 121.008663] VIO: Adding device vldc-port-2-0
[? 121.008730] VIO: Adding device vldc-port-0-0
[? 121.008795] VIO: Adding device vldc-port-0-1
[? 121.008862] VIO: Adding device vldc-port-0-2
[? 121.008939] VIO: Adding device vldc-port-1-0
[? 121.009011] VIO: Adding device vldc-port-3-7
[? 121.009082] VIO: Adding device vldc-port-3-8
[? 121.009164] VIO: Adding device ds-1
[? 121.009227] VIO: Adding device ds-0
[? 121.013776] pci_sun4v: Registered hvapi major[1] minor[0]
[? 121.014050] /pci@0: SUN4V PCI Bus Module
[? 121.014069] /pci@0: On NUMA node -1
[? 121.014089] /pci@0: PCI IO[c0f0000000] MEM[c100000000]
[? 121.014117] /pci@0: Unable to request IOMMU resource.
[? 121.058300] /pci@0: Imported 3 TSB entries from OBP
[? 121.059736] /pci@0: MSI Queue first[0] num[36] count[128] devino[0x18]
[? 121.059764] /pci@0: MSI first[0] num[256] mask[0xff] width[32]
[? 121.059788] /pci@0: MSI addr32[0x7fff0000:0x10000] addr64[0x3ffff0000:0x10000]
[? 121.059817] /pci@0: MSI queues at RA [00000007f8180000]
[? 121.059841] PCI: Scanning PBM /pci@0
[? 121.071538] ds.c:v1.0 (Jul 11, 2007)
[? 121.074662] bio: create slab <bio-0> at 0
[? 121.075346] vgaarb: loaded
[? 121.076145] Switching to clocksource stick
[? 121.076886] Switched to NOHz mode on CPU #30
[? 121.077007] Switched to NOHz mode on CPU #42
[? 121.077128] Switched to NOHz mode on CPU #19
[? 121.077162] Switched to NOHz mode on CPU #7
[? 121.077206] Switched to NOHz mode on CPU #54
[? 121.077215] Switched to NOHz mode on CPU #31
[? 121.077346] Switched to NOHz mode on CPU #43
[? 121.077457] Switched to NOHz mode on CPU #20
[? 121.077468] Switched to NOHz mode on CPU #8
[? 121.077551] Switched to NOHz mode on CPU #32
[? 121.077563] Switched to NOHz mode on CPU #55
[? 121.077693] Switched to NOHz mode on CPU #44
[? 121.077791] Switched to NOHz mode on CPU #9
[? 121.077799] Switched to NOHz mode on CPU #21
[? 121.077897] Switched to NOHz mode on CPU #33
[? 121.077910] Switched to NOHz mode on CPU #56
[? 121.078038] Switched to NOHz mode on CPU #45
[? 121.078125] Switched to NOHz mode on CPU #10
[? 121.078134] Switched to NOHz mode on CPU #22
[? 121.078231] Switched to NOHz mode on CPU #34
[? 121.078262] Switched to NOHz mode on CPU #57
[? 121.078382] Switched to NOHz mode on CPU #46
[? 121.078457] Switched to NOHz mode on CPU #11
[? 121.078479] Switched to NOHz mode on CPU #23
[? 121.078586] Switched to NOHz mode on CPU #35
[? 121.078607] Switched to NOHz mode on CPU #58
[? 121.078730] Switched to NOHz mode on CPU #47
[? 121.078786] Switched to NOHz mode on CPU #12
[? 121.078819] Switched to NOHz mode on CPU #24
[? 121.078922] Switched to NOHz mode on CPU #36
[? 121.078973] Switched to NOHz mode on CPU #59
[? 121.079081] Switched to NOHz mode on CPU #48
[? 121.079103] Switched to NOHz mode on CPU #1
[? 121.079115] Switched to NOHz mode on CPU #13
[? 121.079158] Switched to NOHz mode on CPU #25
[? 121.079275] Switched to NOHz mode on CPU #37
[? 121.079323] Switched to NOHz mode on CPU #60
[? 121.079438] Switched to NOHz mode on CPU #14
[? 121.079447] Switched to NOHz mode on CPU #49
[? 121.079494] Switched to NOHz mode on CPU #26
[? 121.079504] Switched to NOHz mode on CPU #2
[? 121.079612] Switched to NOHz mode on CPU #38
[? 121.079674] Switched to NOHz mode on CPU #61
[? 121.079777] Switched to NOHz mode on CPU #15
[? 121.079788] Switched to NOHz mode on CPU #50
[? 121.079828] Switched to NOHz mode on CPU #3
[? 121.079843] Switched to NOHz mode on CPU #27
[? 121.079928] Switched to NOHz mode on CPU #0
[? 121.079946] Switched to NOHz mode on CPU #39
[? 121.080027] Switched to NOHz mode on CPU #62
[? 121.080115] Switched to NOHz mode on CPU #16
[? 121.080137] Switched to NOHz mode on CPU #51
[? 121.080163] Switched to NOHz mode on CPU #4
[? 121.080178] Switched to NOHz mode on CPU #28
[? 121.080300] Switched to NOHz mode on CPU #40
[? 121.080378] Switched to NOHz mode on CPU #63
[? 121.080631] Switched to NOHz mode on CPU #53
[? 121.080640] Switched to NOHz mode on CPU #52
[? 121.080649] Switched to NOHz mode on CPU #29
[? 121.080658] Switched to NOHz mode on CPU #18
[? 121.080666] Switched to NOHz mode on CPU #17
[? 121.080676] Switched to NOHz mode on CPU #6
[? 121.080685] Switched to NOHz mode on CPU #5
[? 121.080694] Switched to NOHz mode on CPU #41
[? 121.084586] NET: Registered protocol family 2
[? 121.085036] IP route cache hash table entries: 524288 (order: 9, 4194304 bytes)
[? 121.087098] TCP established hash table entries: 524288 (order: 10, 8388608 bytes)
[? 121.112468] TCP bind hash table entries: 65536 (order: 7, 1048576 bytes)
[? 121.116211] TCP: Hash tables configured (established 524288 bind 65536)
[? 121.116237] TCP reno registered
[? 121.116275] UDP hash table entries: 16384 (order: 6, 524288 bytes)
[? 121.118284] UDP-Lite hash table entries: 16384 (order: 6, 524288 bytes)
[? 121.121764] NET: Registered protocol family 1
[? 121.121844] RPC:?????? creating workqueue rpciod
[? 121.122191] RPC:?????? registering /proc/net/rpc
[? 121.122304] svc: Adding svc transport class 'tcp'
[? 121.122330] svc: Adding svc transport class 'udp'
[? 121.122417] RPC: Registered udp transport module.
[? 121.122439] RPC: Registered tcp transport module.
[? 121.122460] RPC: Registered tcp NFSv4.1 backchannel transport module.
[? 121.123725] audit: initializing netlink socket (disabled)
[? 121.123771] type=2000 audit(0.304:1): initialized
[? 121.138489] ds-1: Registered pri service.
[? 121.144197] ds-1: Registered var-config-backup service.
[? 121.172347] HugeTLB registered 4 MB page size, pre-allocated 0 pages
[? 121.180933] VFS: Disk quotas dquot_6.5.2
[? 121.181389] Dquot-cache hash table entries: 1024 (order 0, 8192 bytes)
[? 121.181957] RPC:?????? creating workqueue nfsiod
[? 121.182576] RPC:?????? registering /proc/net/rpc/nfs
[? 121.182629] msgmni has been set to 32768
[? 121.183725] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[? 121.183758] io scheduler noop registered
[? 121.183777] io scheduler deadline registered
[? 121.184006] io scheduler cfq registered (default)
[? 121.185364] f027ab74: ttyS0 at I/O 0x0 (irq = 17) is a SUN4V HCONS
[? 121.290493] console [ttyHV0] enabled
[? 121.296625] f028af14: ttyS1 at MMIO 0xfff0ca0000 (irq = 24) is a 16550A
[? 121.297233] [drm] Initialized drm 1.1.0 20060810
[? 121.298368] e1000e: Intel(R) PRO/1000 Network Driver - 1.3.10-k2
[? 121.298415] e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
[? 121.316970] e1000e 0000:08:00.0: Disabling ASPM? L1
[? 121.495481] e1000e 0000:08:00.0: eth0: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c2
[? 121.495542] e1000e 0000:08:00.0: eth0: Intel(R) PRO/1000 Network Connection
[? 121.495668] e1000e 0000:08:00.0: eth0: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 121.495749] e1000e 0000:08:00.1: Disabling ASPM? L1
[? 121.677262] e1000e 0000:08:00.1: eth1: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c3
[? 121.677322] e1000e 0000:08:00.1: eth1: Intel(R) PRO/1000 Network Connection
[? 121.677448] e1000e 0000:08:00.1: eth1: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 121.677516] e1000e 0000:09:00.0: Disabling ASPM? L1
[? 121.855616] e1000e 0000:09:00.0: eth2: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c4
[? 121.855736] e1000e 0000:09:00.0: eth2: Intel(R) PRO/1000 Network Connection
[? 121.855919] e1000e 0000:09:00.0: eth2: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 121.856045] e1000e 0000:09:00.1: Disabling ASPM? L1
[? 122.037491] e1000e 0000:09:00.1: eth3: (PCI Express:2.5GB/s:Width x4) 00:14:4f:98:09:c5
[? 122.037612] e1000e 0000:09:00.1: eth3: Intel(R) PRO/1000 Network Connection
[? 122.037795] e1000e 0000:09:00.1: eth3: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 122.038436] mousedev: PS/2 mouse device common for all mice
[? 122.039494] rtc-sun4v rtc-sun4v: rtc core: registered sun4v as rtc0
[? 122.040139] TCP cubic registered
[? 122.041005] NET: Registered protocol family 10
[? 122.045854] Mobile IPv6
[? 122.045927] NET: Registered protocol family 17
[? 122.046031] Registering the dns_resolver key type
[? 122.046545] registered taskstats version 1
[? 122.049569] rtc-sun4v rtc-sun4v: setting system clock to 2011-11-04 17:52:28 UTC (1320429148)
[? 122.798799] ADDRCONF(NETDEV_UP): eth0: link is not ready
[? 123.821111] IP-Config: Complete:
[? 123.821180]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=255.255.255.255,
[? 123.821382]????? host=137.226.167.242, domain=, nis-domain=(none),
[? 123.821497]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[? 123.821828] Root-NFS: nfsroot=/srv/nfs/cluster2
[? 123.822225] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
[? 123.822447] NFS:?? parsing nfs mount option 'udp'
[? 123.822514] NFS:?? parsing nfs mount option 'nolock'
[? 123.822589] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[? 123.822657] NFS: MNTPATH: '/srv/nfs/cluster2'
[? 123.822720] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[? 123.822816] RPC:?????? set up xprt to 137.226.167.241 (autobind) via udp
[? 123.822898] RPC:?????? created transport fffff807f7595000 with 16 slots
[? 123.822947] RPC:?????? creating mount client for 137.226.167.241 (xprt fffff807f7595000)
[? 123.823037] RPC:?????? creating UNIX authenticator for client fffff807f74f4c00
[? 123.823125] RPC:?????? new task initialized, procpid 1
[? 123.823198] RPC:?????? allocated task fffff807f8212700
[? 123.823274] RPC:???? 1 __rpc_execute flags=0x680
[? 123.823309] RPC:???? 1 call_start mount3 proc NULL (sync)
[? 123.823384] RPC:???? 1 call_reserve (status 0)
[? 123.823417] RPC:???? 1 reserved req fffff807f71c8000 xid 28f70b9f
[? 123.823493] RPC:???? 1 call_reserveresult (status 0)
[? 123.823566] RPC:???? 1 call_refresh (status 0)
[? 123.823599] RPC:???? 1 holding NULL cred 00000000008aa050
[? 123.823674] RPC:???? 1 refreshing NULL cred 00000000008aa050
[? 123.823719] RPC:???? 1 call_refreshresult (status 0)
[? 123.823792] RPC:???? 1 call_allocate (status 0)
[? 123.823856] RPC:???? 1 allocated buffer of size 96 at fffff807f7595800
[? 123.823904] RPC:???? 1 call_bind (status 0)
[? 123.823967] RPC:???? 1 rpcb_getport_async(137.226.167.241, 100005, 3, 17)
[? 123.824016] RPC:???? 1 sleep_on(queue "xprt_binding" time 4294893049)
[? 123.824093] RPC:???? 1 added to queue fffff807f75950c8 "xprt_binding"
[? 123.824169] RPC:???? 1 setting alarm for 60000 ms
[? 123.824206] RPC:???? 1 rpcb_getport_async: trying rpcbind version 2
[? 123.824296] RPC:?????? set up xprt to 137.226.167.241 (port 111) via udp
[? 123.824376] RPC:?????? created transport fffff807f7596000 with 16 slots
[? 123.824454] RPC:?????? creating rpcbind client for 137.226.167.241 (xprt fffff807f7596000)
[? 123.824513] RPC:?????? creating UNIX authenticator for client fffff807f74f4e00
[? 123.824599] RPC:?????? new task initialized, procpid 1
[? 123.824672] RPC:?????? allocated task fffff807f82127e0
[? 123.824752] RPC:?????? rpc_release_client(fffff807f74f4e00)
[? 123.824800] RPC:???? 1 sync task going to sleep
[? 123.824869] RPC:???? 2 __rpc_execute flags=0x681
[? 123.824933] RPC:???? 2 call_start rpcbind2 proc GETPORT (async)
[? 123.824980] RPC:???? 2 call_reserve (status 0)
[? 123.825042] RPC:???? 2 reserved req fffff807f71ca000 xid 61482335
[? 123.825088] RPC:???? 2 call_reserveresult (status 0)
[? 123.825161] RPC:???? 2 call_refresh (status 0)
[? 123.825221] RPC:???? 2 looking up UNIX cred
[? 123.825252] RPC:?????? looking up UNIX cred
[? 123.825286] RPC:?????? allocating UNIX cred for uid 0 gid 0
[? 123.825362] RPC:???? 2 refreshing UNIX cred fffff807f74035c0
[? 123.825437] RPC:???? 2 call_refreshresult (status 0)
[? 123.825481] RPC:???? 2 call_allocate (status 0)
[? 123.825543] RPC:???? 2 allocated buffer of size 416 at fffff807f7596800
[? 123.825620] RPC:???? 2 call_bind (status 0)
[? 123.825653] RPC:???? 2 call_connect xprt fffff807f7596000 is not connected
[? 123.825731] RPC:???? 2 xprt_connect xprt fffff807f7596000 is not connected
[? 123.825809] RPC:???? 2 xprt_cwnd_limited cong = 0 cwnd = 256
[? 123.825856] RPC:???? 2 sleep_on(queue "xprt_pending" time 4294893049)
[? 123.825933] RPC:???? 2 added to queue fffff807f75962d8 "xprt_pending"
[? 123.826008] RPC:???? 2 setting alarm for 5000 ms
[? 123.826042] RPC:?????? xs_connect scheduled xprt fffff807f7596000
[? 123.826167] RPC:?????? xs_bind 0.0.0.0:0: ok (0)
[? 123.826232] RPC:?????? worker connecting xprt fffff807f7596000 via udp to 137.226.167.241 (port 111)
[? 123.826323] RPC:???? 2 __rpc_wake_up_task (now 4294893050)
[? 123.826396] RPC:???? 2 disabling timer
[? 123.826428] RPC:???? 2 removed from queue fffff807f75962d8 "xprt_pending"
[? 123.826475] RPC:?????? __rpc_wake_up_task done
[? 123.826508] RPC:???? 2 __rpc_execute flags=0x681
[? 123.826541] RPC:???? 2 xprt_connect_status: connection established
[? 123.826588] RPC:???? 2 call_connect_status (status 0)
[? 123.826632] RPC:???? 2 call_transmit (status 0)
[? 123.826664] RPC:???? 2 xprt_prepare_transmit
[? 123.826696] RPC:???? 2 rpc_xdr_encode (status 0)
[? 123.826729] RPC:???? 2 marshaling UNIX cred fffff807f74035c0
[? 123.826777] RPC:???? 2 using AUTH_UNIX cred fffff807f74035c0 to wrap rpc data
[? 123.826830] RPC:???? 2 encoding PMAP_GETPORT call (100005, 3, 17, 0)
[? 123.826877] RPC:???? 2 xprt_transmit(92)
[? 123.826973] RPC:?????? xs_udp_send_request(92) = 92
[? 123.827007] RPC:???? 2 xmit complete
[? 123.827040] RPC:???? 2 sleep_on(queue "xprt_pending" time 4294893050)
[? 123.827088] RPC:???? 2 added to queue fffff807f75962d8 "xprt_pending"
[? 123.827135] RPC:???? 2 setting alarm for 5000 ms
[? 124.915844] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[? 124.936050] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[? 128.844738] RPC:???? 2 timeout
[? 128.844798] RPC:???? 2 __rpc_wake_up_task (now 4294894304)
[? 128.844901] RPC:???? 2 disabling timer
[? 128.844962] RPC:???? 2 removed from queue fffff807f75962d8 "xprt_pending"
[? 128.845068] RPC:?????? __rpc_wake_up_task done
[? 128.845172] RPC:???? 2 __rpc_execute flags=0xe81
[? 128.845206] RPC:???? 2 xprt_timer
[? 128.845238] RPC:?????? cong 256, cwnd was 256, now 256
[? 128.845312] RPC:?????? wake_up_next(fffff807f7596228 "xprt_resend")
[? 128.845360] RPC:?????? wake_up_next(fffff807f7596178 "xprt_sending")
[? 128.845437] RPC:???? 2 call_status (status -60)
[? 128.845470] RPC:???? 2 call_timeout (minor)
[? 128.845530] RPC:???? 2 call_bind (status 0)
[? 128.845563] RPC:???? 2 call_connect xprt fffff807f7596000 is connected
[? 128.845638] RPC:???? 2 call_transmit (status 0)
[? 128.845669] RPC:???? 2 xprt_prepare_transmit
[? 128.845731] RPC:???? 2 xprt_cwnd_limited cong = 0 cwnd = 256
[? 128.845776] RPC:???? 2 rpc_xdr_encode (status 0)
[? 128.845839] RPC:???? 2 marshaling UNIX cred fffff807f74035c0
[? 128.845915] RPC:???? 2 using AUTH_UNIX cred fffff807f74035c0 to wrap rpc data
[? 128.845968] RPC:???? 2 encoding PMAP_GETPORT call (100005, 3, 17, 0)
[? 128.846044] RPC:???? 2 xprt_transmit(92)
[? 128.846125] RPC:?????? xs_udp_send_request(92) = 92
[? 128.846159] RPC:???? 2 xmit complete
[? 128.846220] RPC:???? 2 sleep_on(queue "xprt_pending" time 4294894304)
[? 128.846297] RPC:???? 2 added to queue fffff807f75962d8 "xprt_pending"
[? 128.846344] RPC:???? 2 setting alarm for 10000 ms
[? 138.865672] RPC:???? 2 timeout
[? 138.865733] RPC:???? 2 __rpc_wake_up_task (now 4294896808)
[? 138.865834] RPC:???? 2 disabling timer
[? 138.865894] RPC:???? 2 removed from queue fffff807f75962d8 "xprt_pending"
[? 138.866002] RPC:?????? __rpc_wake_up_task done
[? 138.866101] RPC:???? 2 __rpc_execute flags=0xe81
[? 138.866134] RPC:???? 2 xprt_timer
[? 138.866165] RPC:?????? cong 256, cwnd was 256, now 256
[? 138.866238] RPC:?????? wake_up_next(fffff807f7596228 "xprt_resend")
[? 138.866285] RPC:?????? wake_up_next(fffff807f7596178 "xprt_sending")
[? 138.866362] RPC:???? 2 call_status (status -60)
[? 138.866394] RPC:???? 2 call_timeout (minor)
[? 138.866454] RPC:???? 2 call_bind (status 0)
[? 138.866486] RPC:???? 2 call_connect xprt fffff807f7596000 is connected
[? 138.866562] RPC:???? 2 call_transmit (status 0)
[? 138.866593] RPC:???? 2 xprt_prepare_transmit
[? 138.866654] RPC:???? 2 xprt_cwnd_limited cong = 0 cwnd = 256
[? 138.866700] RPC:???? 2 rpc_xdr_encode (status 0)
[? 138.866761] RPC:???? 2 marshaling UNIX cred fffff807f74035c0
[? 138.866837] RPC:???? 2 using AUTH_UNIX cred fffff807f74035c0 to wrap rpc data
[? 138.866889] RPC:???? 2 encoding PMAP_GETPORT call (100005, 3, 17, 0)
[? 138.866963] RPC:???? 2 xprt_transmit(92)
[? 138.867007] RPC:?????? xs_udp_send_request(92) = 92
[? 138.867070] RPC:???? 2 xmit complete
[? 138.867130] RPC:???? 2 sleep_on(queue "xprt_pending" time 4294896808)
[? 138.867179] RPC:???? 2 added to queue fffff807f75962d8 "xprt_pending"
[? 138.867254] RPC:???? 2 setting alarm for 20000 ms
[? 158.940429] RPC:???? 2 timeout
[? 158.940490] RPC:???? 2 __rpc_wake_up_task (now 4294901824)
[? 158.940591] RPC:???? 2 disabling timer
[? 158.940651] RPC:???? 2 removed from queue fffff807f75962d8 "xprt_pending"
[? 158.940756] RPC:?????? __rpc_wake_up_task done
[? 158.940902] RPC:???? 2 __rpc_execute flags=0xe81
[? 158.940965] RPC:???? 2 xprt_timer
[? 158.940995] RPC:?????? cong 256, cwnd was 256, now 256
[? 158.941039] RPC:?????? wake_up_next(fffff807f7596228 "xprt_resend")
[? 158.941115] RPC:?????? wake_up_next(fffff807f7596178 "xprt_sending")
[? 158.941163] RPC:???? 2 call_status (status -60)
[? 158.941225] RPC:???? 2 call_timeout (major)
[? 158.941258] RPC:?????? setting port for xprt fffff807f7595000 to 0
[? 158.941337] RPC:???? 2 rpcb_getport_done(status -60, port 0)
[? 158.941412] RPC:???? 2 return 0, status -60
[? 158.941442] RPC:???? 2 release task
[? 158.941504] RPC:?????? freeing buffer of size 416 at fffff807f7596800
[? 158.941554] RPC:???? 2 release request fffff807f71ca000
[? 158.941628] RPC:?????? wake_up_next(fffff807f7596388 "xprt_backlog")
[? 158.941705] RPC:?????? rpc_release_client(fffff807f74f4e00)
[? 158.941751] RPC:?????? destroying rpcbind client for 137.226.167.241
[? 158.941827] RPC:?????? destroying transport fffff807f7596000
[? 158.941937] RPC:?????? xs_destroy xprt fffff807f7596000
[? 158.942033] RPC:?????? xs_close xprt fffff807f7596000
[? 158.942125] RPC:?????? disconnected transport fffff807f7596000
[? 158.942205] RPC:???? 2 freeing task
[? 158.942238] RPC:???? 1 __rpc_wake_up_task (now 4294901824)
[? 158.942310] RPC:???? 1 disabling timer
[? 158.942371] RPC:???? 1 removed from queue fffff807f75950c8 "xprt_binding"
[? 158.942427] RPC:?????? __rpc_wake_up_task done
[? 158.942439] RPC:???? 1 sync task resuming
[? 158.942447] RPC:???? 1 rpcbind request timed out
[? 158.942456] RPC:???? 1 call_timeout (major)
[? 158.942465] RPC:???? 1 return 0, status -60
[? 158.942472] RPC:???? 1 release task
[? 158.942485] RPC:?????? freeing buffer of size 96 at fffff807f7595800
[? 158.942498] RPC:???? 1 release request fffff807f71c8000
[? 158.942508] RPC:?????? wake_up_next(fffff807f7595388 "xprt_backlog")
[? 158.942520] RPC:?????? rpc_release_client(fffff807f74f4c00)
[? 158.942531] RPC:???? 1 freeing task
[? 158.942541] RPC:?????? shutting down mount client for 137.226.167.241
[? 158.942550] RPC:?????? rpc_release_client(fffff807f74f4c00)
[? 158.942560] RPC:?????? destroying UNIX authenticator 00000000008aa0d8
[? 158.942575] RPC:?????? destroying mount client for 137.226.167.241
[? 158.942587] NFS: failed to create MNT RPC client, status=-60
[? 158.942599] NFS: unable to mount server 137.226.167.241, error -60
[? 158.942642] VFS: Unable to mount root fs via NFS, trying floppy.
[? 158.942804] List of all partitions:
[? 158.942815] No filesystem could mount root, tried:
[? 158.942829] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
[? 158.942840] Call Trace:
[? 158.942867]? [00000000008b8d18] mount_block_root+0x210/0x228
[? 158.942884]? [00000000008b8df4] mount_root+0xc4/0xd4
[? 158.942899]? [00000000008b8f80] prepare_namespace+0x17c/0x1c0
[? 158.942915]? [00000000008b8260] kernel_init+0x164/0x17c
[? 158.942934]? [000000000042ac74] kernel_thread+0x30/0x48
[? 158.942952]? [000000000075d1fc] rest_init+0x18/0x74
[? 158.943026] Press Stop-A (L1-A) to return to the boot prom
[? 158.943848] RPC:?????? destroying transport fffff807f7595000
[? 158.943913] RPC:?????? xs_destroy xprt fffff807f7595000
[? 158.944013] RPC:?????? xs_close xprt fffff807f7595000
[? 158.944087] RPC:?????? disconnected transport fffff807f7595000

2011-11-12 21:06:36

by Chuck Lever III

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64


On Nov 12, 2011, at 1:49 PM, Jim Rees wrote:

> The question for us is how long should an nfsroot client wait for the server
> to reply. It sounds like the client used to wait longer than it does now.

Before, the client performed the GETPORT(NFS) step synchronously, first. This took 30 seconds or so to timeout. When it did, the client decided to proceed with port 2049. Then it went on to do the other mount tasks, and at the point had waited long enough that these tasks did not time out while waiting for the switch port.

> It seems to me the client should wait at least 90 seconds so that the
> situation you're in (servers on non-portfast ports) will work. I would
> think they should wait indefinitely, since there's not much else they can
> do.

It should be simple to wrap the (MNT(mnt), NFS(getroot)) steps in a while(true) loop. Would mount_root_nfs() be the right place for this?

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com





2011-11-04 13:21:00

by Jim Rees

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

It's still a mystery to me because I use nfsroot all the time and it works
for me. But I was never able to get static addressing to work and have had
to depend on the in-kernel dhcp. And I have a different ethernet card that
maybe resets faster. I have run into problems later in the boot, when user
space remounts the root, but that's not what you're seeing.

As for a fix... we're trying to move away from udp transport anyway. Maybe
someone should figure out a way to get it to work with tcp? I have zero
experience with nfs over udp, at least on linux. Just for fun, have you
tried tcp transport (proto=tcp)?

As we move toward nfs4 someone will have to give some thought to nfsroot.
It's hard to imagine we could put enough nfs4 cruft into the kernel (gssd,
idmapd) to make it work.

2011-11-05 03:51:51

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

>> ***IMPORTANT HINT***

>>
>> In the test belowI've tried to ping the NFS client (cluster2) as soon
> as I've got the
>>
>> ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> message.
>>
>>
>> And at first I've got no answers from the client. But 10-20sec later
> I've got pongs back.
>> So the NIC comes up some seconds later than the message "link becomes
> ready" is shown...
>> In the case of 'mountproto=tcp' it comes up after the kernel panic.
> So that's much too late.
>>
>
> So, once again...
>
> What kind of switch is this client connected to, and how is that switch
> configured?

I can't answer this question because I have no physical access to the hardware.
Hence I think it's logical to stop here until the nodes are connected directly (without any hardware between them).
I'll write here as soon as I know if it works...

For the rest I thank all contributors for the fast help!

Regards,
Lukas

2011-11-03 21:37:23

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> On Nov 3, 2011, at 5:11 PM, Jim Rees wrote:

>
>> Trond Myklebust wrote:
>>
>>> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
>>> [ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
>>> [ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
>>> [ 442.667032] VFS: Insert root floppy and press ENTER
>>>
>> ? Error 60 is ETIMEDOUT on SPARC, so it seems that the problem is
>> ? basically the same one that you see in your 2.6.32 trace (rpcbind:
>> ? server 137.226.167.241 not responding, timed out) except that now it is
>> ? a fatal error.
>>
>> ? Any idea why the first RPC calls might be failing here? A switch
>> ? misconfiguration or something like that perhaps?
>>
>> Wasn't there a change in the way nfs mount options are handled by the
> kernel
>> for nfsroot about the time of 2.6.39?? Something about changing from
> default
>> udp to tcp maybe?
>
> There was a change, but it was changed back to UDP because of problems like
> this.? Behavior in 3.0 or the latest 2.6.39 stable kernel may be improved.
>

I don't know if this was a tip to test newest 2.6.39 but as I wrote in my first email
?http://thread.gmane.org/gmane.linux.nfs/44596
that's the output of linux-2.6.39.4 with "nfsdebug":

[ 407.571521] IP-Config: Complete:
[ 407.571589] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
[ 407.571793] host=cluster2, domain=, nis-domain=(none),
[ 407.571907] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[ 407.572332] Root-NFS: nfsroot=/srv/nfs/cluster2
[ 407.572726] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
[ 407.572927] NFS: parsing nfs mount option 'udp'
[ 407.572995] NFS: parsing nfs mount option 'nolock'
[ 407.573071] NFS: parsing nfs mount option 'addr=137.226.167.241'
[ 407.573139] NFS: MNTPATH: '/srv/nfs/cluster2'
[ 407.573203] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[ 408.617894] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[ 408.638319] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 442.666622] NFS: failed to create MNT RPC client, status=-60
[ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
[ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
[ 442.667032] VFS: Insert root floppy and press ENTER

And this behaviour is exactly the same as in all other 2.6.37 - 2.6.39.4 which I've tested.
So if anybody of you all have an idea what I could try to do, I'll follow...

Regards,
Lukas

2011-11-05 01:52:48

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> So the NFS client is trying to establish contact with the server for a

> period of 35 seconds, and there is no reply... Have you tried using
> wireshark to see if any NFS traffic is arriving at the server from the
> client?

Yes, the server gets no packets from the client between linux 2.6.37 and 3.1.
In linux <= 2.6.36 it also doesn't get packets from client to RPC port 10003 (not even ARP packets) but then the kernel tries 10005. This works (the server gets ARP, then SUNRPC packets and the client can mount NFSROOT).


> Have you tried using the 'mountproto=tcp' option?

Now yes. There were no ARP or SUNRPC packets.
That's the result of linux-3.1 with 'mountproto=tcp':
---
{0} ok boot net root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2,mountproto=tcp ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
Boot device: /pci@0/pci@0/pci@1/pci@0/pci@2/network@0? File and args: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2,mountproto=tcp ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
1000 Mbps full duplex? Link up
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
Requesting Internet Address for 0:14:4f:98:9:c2
|
[??? 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
[??? 0.000000] PROMLIB: Root node compatible: sun4v
[??? 0.000000] Initializing cgroup subsys cpuset
[??? 0.000000] Initializing cgroup subsys cpu
[??? 0.000000] Linux version 3.1.0-razik-2011-11-05 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #3 SMP Sat Nov 5 00:24:24 CET 2011
[??? 0.000000] bootconsole [earlyprom0] enabled
[??? 0.000000] ARCH: SUN4V
[??? 0.000000] Ethernet address: 00:14:4f:98:09:c2
[??? 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
[??? 0.000000] Remapping the kernel... done.
[??? 0.000000] OF stdout device is: /virtual-devices@100/console@1
[??? 0.000000] PROM: Built device tree with 123778 bytes of memory.
[??? 0.000000] MDESC: Size is 58816 bytes.
[??? 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
[??? 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
[??? 0.000000] PLATFORM: hostid [849809c2]
[??? 0.000000] PLATFORM: serial# [00ab4130]
[??? 0.000000] PLATFORM: stick-frequency [5458c3a0]
[??? 0.000000] PLATFORM: mac-address [144f9809c2]
[??? 0.000000] PLATFORM: watchdog-resolution [1000 ms]
[??? 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
[??? 0.000000] PLATFORM: max-cpus [64]
[??? 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c2000
[??? 0.000000] Memory hole size: 132MB
[??? 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
[??? 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
[??? 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
[??? 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
[??? 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
[??? 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
[??? 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
[??? 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
[??? 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
[??? 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
[??? 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
[??? 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
[??? 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
[??? 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
[??? 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
[??? 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
[??? 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
[??? 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
[??? 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
[??? 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
[??? 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
[??? 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
[??? 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
[??? 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
[??? 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
[??? 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
[??? 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
[??? 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
[??? 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
[??? 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
[??? 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
[??? 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
[??? 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
[??? 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
[??? 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
[??? 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
[??? 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
[??? 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
[??? 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
[??? 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
[??? 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
[??? 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
[??? 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
[??? 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
[??? 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
[??? 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
[??? 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
[??? 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
[??? 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
[??? 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
[??? 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
[??? 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
[??? 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
[??? 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
[??? 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
[??? 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
[??? 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
[??? 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
[??? 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
[??? 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
[??? 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
[??? 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
[??? 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
[??? 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
[??? 0.000000] Zone PFN ranges:
[??? 0.000000]?? Normal?? 0x00004200 -> 0x003ffd8e
[??? 0.000000] Movable zone start PFN for each node
[??? 0.000000] early_node_map[3] active PFN ranges
[??? 0.000000]???? 0: 0x00004200 -> 0x003ffd47
[??? 0.000000]???? 0: 0x003ffd50 -> 0x003ffd63
[??? 0.000000]???? 0: 0x003ffd87 -> 0x003ffd8e
[??? 0.000000] Booting Linux...
[??? 0.000000] CPU CAPS: [flush,stbar,swap,muldiv,v9,blkinit,n2,mul32]
[??? 0.000000] CPU CAPS: [div32,v8plus,popc,vis,vis2,ASIBlkInit]
[??? 0.000000] PERCPU: Embedded 5 pages/cpu @fffff80019000000 s10816 r8192 d21952 u65536
[??? 0.000000] SUN4V: Mondo queue sizes [cpu(8192) dev(16384) r(8192) nr(256)]
[??? 0.000000] Built 1 zonelists in Zone order, mobility grouping on.? Total pages: 4144105
[??? 0.000000] Kernel command line: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2,mountproto=tcp ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
[??? 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
[??? 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
[??? 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
[??? 0.000000] Memory: 33091944k available (3568k kernel code, 1272k data, 216k init) [fffff80000000000,00000007ffb1c000]
[??? 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[??? 0.000000] Hierarchical RCU implementation.
[??? 0.000000]? CONFIG_RCU_FANOUT set to non-default value of 32
[??? 0.000000] NR_IRQS:255
[??? 0.000000] clocksource: mult[b4e7cf] shift[24]
[??? 0.000000] clockevent: mult[b5221ba3] shift[31]
[??? 0.000000] Console: colour dummy device 80x25
[??? 0.000000] console [tty0] enabled, bootconsole disabled
[??? 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.30.0 2008/12/11 12:15'
[??? 0.000000] PROMLIB: Root node compatible: sun4v
[??? 0.000000] Initializing cgroup subsys cpuset
[??? 0.000000] Initializing cgroup subsys cpu
[??? 0.000000] Linux version 3.1.0-razik-2011-11-05 (root@cluster1) (gcc version 4.4.5 (Debian 4.4.5-8) ) #3 SMP Sat Nov 5 00:24:24 CET 2011
[??? 0.000000] bootconsole [earlyprom0] enabled
[??? 0.000000] ARCH: SUN4V
[??? 0.000000] Ethernet address: 00:14:4f:98:09:c2
[??? 0.000000] Kernel: Using 2 locked TLB entries for main kernel image.
[??? 0.000000] Remapping the kernel... done.
[??? 0.000000] OF stdout device is: /virtual-devices@100/console@1
[??? 0.000000] PROM: Built device tree with 123778 bytes of memory.
[??? 0.000000] MDESC: Size is 58816 bytes.
[??? 0.000000] PLATFORM: banner-name [SPARC Enterprise T5120]
[??? 0.000000] PLATFORM: name [SUNW,SPARC-Enterprise-T5120]
[??? 0.000000] PLATFORM: hostid [849809c2]
[??? 0.000000] PLATFORM: serial# [00ab4130]
[??? 0.000000] PLATFORM: stick-frequency [5458c3a0]
[??? 0.000000] PLATFORM: mac-address [144f9809c2]
[??? 0.000000] PLATFORM: watchdog-resolution [1000 ms]
[??? 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
[??? 0.000000] PLATFORM: max-cpus [64]
[??? 0.000000] Top of RAM: 0x7ffb1c000, Total RAM: 0x7f76c2000
[??? 0.000000] Memory hole size: 132MB
[??? 0.000000] [0000010000000000-fffff80009000000] page_structs=131072 node=0 entry=0/8192
[??? 0.000000] [0000010000000000-fffff80009400000] page_structs=131072 node=0 entry=1/8192
[??? 0.000000] [0000010000800000-fffff80009800000] page_structs=131072 node=0 entry=2/8192
[??? 0.000000] [0000010000800000-fffff80009c00000] page_structs=131072 node=0 entry=3/8192
[??? 0.000000] [0000010001000000-fffff8000a000000] page_structs=131072 node=0 entry=4/8192
[??? 0.000000] [0000010001000000-fffff8000a400000] page_structs=131072 node=0 entry=5/8192
[??? 0.000000] [0000010001800000-fffff8000a800000] page_structs=131072 node=0 entry=6/8192
[??? 0.000000] [0000010001800000-fffff8000ac00000] page_structs=131072 node=0 entry=7/8192
[??? 0.000000] [0000010002000000-fffff8000b000000] page_structs=131072 node=0 entry=8/8192
[??? 0.000000] [0000010002000000-fffff8000b400000] page_structs=131072 node=0 entry=9/8192
[??? 0.000000] [0000010002800000-fffff8000b800000] page_structs=131072 node=0 entry=10/8192
[??? 0.000000] [0000010002800000-fffff8000bc00000] page_structs=131072 node=0 entry=11/8192
[??? 0.000000] [0000010003000000-fffff8000c000000] page_structs=131072 node=0 entry=12/8192
[??? 0.000000] [0000010003000000-fffff8000c400000] page_structs=131072 node=0 entry=13/8192
[??? 0.000000] [0000010003800000-fffff8000c800000] page_structs=131072 node=0 entry=14/8192
[??? 0.000000] [0000010003800000-fffff8000cc00000] page_structs=131072 node=0 entry=15/8192
[??? 0.000000] [0000010004000000-fffff8000d000000] page_structs=131072 node=0 entry=16/8192
[??? 0.000000] [0000010004000000-fffff8000d400000] page_structs=131072 node=0 entry=17/8192
[??? 0.000000] [0000010004800000-fffff8000d800000] page_structs=131072 node=0 entry=18/8192
[??? 0.000000] [0000010004800000-fffff8000dc00000] page_structs=131072 node=0 entry=19/8192
[??? 0.000000] [0000010005000000-fffff8000e000000] page_structs=131072 node=0 entry=20/8192
[??? 0.000000] [0000010005000000-fffff8000e400000] page_structs=131072 node=0 entry=21/8192
[??? 0.000000] [0000010005800000-fffff8000e800000] page_structs=131072 node=0 entry=22/8192
[??? 0.000000] [0000010005800000-fffff8000ec00000] page_structs=131072 node=0 entry=23/8192
[??? 0.000000] [0000010006000000-fffff8000f000000] page_structs=131072 node=0 entry=24/8192
[??? 0.000000] [0000010006000000-fffff8000f400000] page_structs=131072 node=0 entry=25/8192
[??? 0.000000] [0000010006800000-fffff8000f800000] page_structs=131072 node=0 entry=26/8192
[??? 0.000000] [0000010006800000-fffff8000fc00000] page_structs=131072 node=0 entry=27/8192
[??? 0.000000] [0000010007000000-fffff80010000000] page_structs=131072 node=0 entry=28/8192
[??? 0.000000] [0000010007000000-fffff80010400000] page_structs=131072 node=0 entry=29/8192
[??? 0.000000] [0000010007800000-fffff80010800000] page_structs=131072 node=0 entry=30/8192
[??? 0.000000] [0000010007800000-fffff80010c00000] page_structs=131072 node=0 entry=31/8192
[??? 0.000000] [0000010008000000-fffff80011000000] page_structs=131072 node=0 entry=32/8192
[??? 0.000000] [0000010008000000-fffff80011400000] page_structs=131072 node=0 entry=33/8192
[??? 0.000000] [0000010008800000-fffff80011800000] page_structs=131072 node=0 entry=34/8192
[??? 0.000000] [0000010008800000-fffff80011c00000] page_structs=131072 node=0 entry=35/8192
[??? 0.000000] [0000010009000000-fffff80012000000] page_structs=131072 node=0 entry=36/8192
[??? 0.000000] [0000010009000000-fffff80012400000] page_structs=131072 node=0 entry=37/8192
[??? 0.000000] [0000010009800000-fffff80012800000] page_structs=131072 node=0 entry=38/8192
[??? 0.000000] [0000010009800000-fffff80012c00000] page_structs=131072 node=0 entry=39/8192
[??? 0.000000] [000001000a000000-fffff80013000000] page_structs=131072 node=0 entry=40/8192
[??? 0.000000] [000001000a000000-fffff80013400000] page_structs=131072 node=0 entry=41/8192
[??? 0.000000] [000001000a800000-fffff80013800000] page_structs=131072 node=0 entry=42/8192
[??? 0.000000] [000001000a800000-fffff80013c00000] page_structs=131072 node=0 entry=43/8192
[??? 0.000000] [000001000b000000-fffff80014000000] page_structs=131072 node=0 entry=44/8192
[??? 0.000000] [000001000b000000-fffff80014400000] page_structs=131072 node=0 entry=45/8192
[??? 0.000000] [000001000b800000-fffff80014800000] page_structs=131072 node=0 entry=46/8192
[??? 0.000000] [000001000b800000-fffff80014c00000] page_structs=131072 node=0 entry=47/8192
[??? 0.000000] [000001000c000000-fffff80015000000] page_structs=131072 node=0 entry=48/8192
[??? 0.000000] [000001000c000000-fffff80015400000] page_structs=131072 node=0 entry=49/8192
[??? 0.000000] [000001000c800000-fffff80015800000] page_structs=131072 node=0 entry=50/8192
[??? 0.000000] [000001000c800000-fffff80015c00000] page_structs=131072 node=0 entry=51/8192
[??? 0.000000] [000001000d000000-fffff80016000000] page_structs=131072 node=0 entry=52/8192
[??? 0.000000] [000001000d000000-fffff80016400000] page_structs=131072 node=0 entry=53/8192
[??? 0.000000] [000001000d800000-fffff80016800000] page_structs=131072 node=0 entry=54/8192
[??? 0.000000] [000001000d800000-fffff80016c00000] page_structs=131072 node=0 entry=55/8192
[??? 0.000000] [000001000e000000-fffff80017000000] page_structs=131072 node=0 entry=56/8192
[??? 0.000000] [000001000e000000-fffff80017400000] page_structs=131072 node=0 entry=57/8192
[??? 0.000000] [000001000e800000-fffff80017800000] page_structs=131072 node=0 entry=58/8192
[??? 0.000000] [000001000e800000-fffff80017c00000] page_structs=131072 node=0 entry=59/8192
[??? 0.000000] [000001000f000000-fffff80018000000] page_structs=131072 node=0 entry=60/8192
[??? 0.000000] [000001000f000000-fffff80018400000] page_structs=131072 node=0 entry=61/8192
[??? 0.000000] [000001000f800000-fffff80018800000] page_structs=131072 node=0 entry=62/8192
[??? 0.000000] [000001000f800000-fffff80018c00000] page_structs=131072 node=0 entry=63/8192
[??? 0.000000] Zone PFN ranges:
[??? 0.000000]?? Normal?? 0x00004200 -> 0x003ffd8e
[??? 0.000000] Movable zone start PFN for each node
[??? 0.000000] early_node_map[3] active PFN ranges
[??? 0.000000]???? 0: 0x00004200 -> 0x003ffd47
[??? 0.000000]???? 0: 0x003ffd50 -> 0x003ffd63
[??? 0.000000]???? 0: 0x003ffd87 -> 0x003ffd8e
[??? 0.000000] Booting Linux...
[??? 0.000000] CPU CAPS: [flush,stbar,swap,muldiv,v9,blkinit,n2,mul32]
[??? 0.000000] CPU CAPS: [div32,v8plus,popc,vis,vis2,ASIBlkInit]
[??? 0.000000] PERCPU: Embedded 5 pages/cpu @fffff80019000000 s10816 r8192 d21952 u65536
[??? 0.000000] SUN4V: Mondo queue sizes [cpu(8192) dev(16384) r(8192) nr(256)]
[??? 0.000000] Built 1 zonelists in Zone order, mobility grouping on.? Total pages: 4144105
[??? 0.000000] Kernel command line: root=/dev/nfs rw nfsroot=137.226.167.241:/srv/nfs/cluster2,mountproto=tcp ip=137.226.167.242:::255.255.255.224::eth0:off nfsrootdebug
[??? 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
[??? 0.000000] Dentry cache hash table entries: 4194304 (order: 12, 33554432 bytes)
[??? 0.000000] Inode-cache hash table entries: 2097152 (order: 11, 16777216 bytes)
[??? 0.000000] Memory: 33091944k available (3568k kernel code, 1272k data, 216k init) [fffff80000000000,00000007ffb1c000]
[??? 0.000000] SLUB: Genslabs=16, HWalign=32, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[??? 0.000000] Hierarchical RCU implementation.
[??? 0.000000]? CONFIG_RCU_FANOUT set to non-default value of 32
[??? 0.000000] NR_IRQS:255
[??? 0.000000] clocksource: mult[b4e7cf] shift[24]
[??? 0.000000] clockevent: mult[b5221ba3] shift[31]
[??? 0.000000] Console: colour dummy device 80x25
[??? 0.000000] console [tty0] enabled, bootconsole disabled
[? 273.822931] Calibrating delay using timer specific routine.. 2832.85 BogoMIPS (lpj=5665715)
[? 273.822964] pid_max: default: 65536 minimum: 512
[? 273.823429] Mount-cache hash table entries: 512
[? 273.824703] Initializing cgroup subsys cpuacct
[? 273.824814] Initializing cgroup subsys devices
[? 273.824832] Initializing cgroup subsys freezer
[? 273.824848] Initializing cgroup subsys net_cls
[? 273.825126] Performance events: Supported PMU type is 'niagara2'
[? 273.847590] Brought up 64 CPUs
[? 273.847649] Testing NMI watchdog ... OK.
[? 273.930292] devtmpfs: initialized
[? 273.930789] ldc.c:v1.1 (July 22, 2008)
[? 273.931092] print_constraints: dummy:
[? 273.931479] NET: Registered protocol family 16
[? 273.940690] VIO: Adding device channel-devices
[? 273.940767] VIO: Adding device vldc-port-3-0
[? 273.940828] VIO: Adding device vldc-port-3-1
[? 273.940888] VIO: Adding device vldc-port-3-2
[? 273.940958] VIO: Adding device vldc-port-3-3
[? 273.941020] VIO: Adding device vldc-port-3-4
[? 273.941082] VIO: Adding device vldc-port-3-5
[? 273.941144] VIO: Adding device vldc-port-2-0
[? 273.941207] VIO: Adding device vldc-port-0-0
[? 273.941276] VIO: Adding device vldc-port-0-1
[? 273.941344] VIO: Adding device vldc-port-0-2
[? 273.941422] VIO: Adding device vldc-port-1-0
[? 273.941492] VIO: Adding device vldc-port-3-7
[? 273.941565] VIO: Adding device vldc-port-3-8
[? 273.941647] VIO: Adding device ds-1
[? 273.941711] VIO: Adding device ds-0
[? 273.946313] pci_sun4v: Registered hvapi major[1] minor[0]
[? 273.946578] /pci@0: SUN4V PCI Bus Module
[? 273.946598] /pci@0: On NUMA node -1
[? 273.946619] /pci@0: PCI IO[c0f0000000] MEM[c100000000]
[? 273.946649] /pci@0: Unable to request IOMMU resource.
[? 273.990964] /pci@0: Imported 3 TSB entries from OBP
[? 273.992499] /pci@0: MSI Queue first[0] num[36] count[128] devino[0x18]
[? 273.992527] /pci@0: MSI first[0] num[256] mask[0xff] width[32]
[? 273.992552] /pci@0: MSI addr32[0x7fff0000:0x10000] addr64[0x3ffff0000:0x10000]
[? 273.992581] /pci@0: MSI queues at RA [00000007fe580000]
[? 273.992605] PCI: Scanning PBM /pci@0
[? 274.004444] ds.c:v1.0 (Jul 11, 2007)
[? 274.007744] bio: create slab <bio-0> at 0
[? 274.008323] vgaarb: loaded
[? 274.009098] Switching to clocksource stick
[? 274.009749] Switched to NOHz mode on CPU #47
[? 274.009801] Switched to NOHz mode on CPU #12
[? 274.009829] Switched to NOHz mode on CPU #24
[? 274.009923] Switched to NOHz mode on CPU #36
[? 274.009968] Switched to NOHz mode on CPU #59
[? 274.010093] Switched to NOHz mode on CPU #48
[? 274.010106] Switched to NOHz mode on CPU #1
[? 274.010131] Switched to NOHz mode on CPU #13
[? 274.010170] Switched to NOHz mode on CPU #25
[? 274.010282] Switched to NOHz mode on CPU #37
[? 274.010316] Switched to NOHz mode on CPU #60
[? 274.010449] Switched to NOHz mode on CPU #49
[? 274.010463] Switched to NOHz mode on CPU #14
[? 274.010510] Switched to NOHz mode on CPU #2
[? 274.010519] Switched to NOHz mode on CPU #26
[? 274.010624] Switched to NOHz mode on CPU #38
[? 274.010669] Switched to NOHz mode on CPU #61
[? 274.010830] Switched to NOHz mode on CPU #3
[? 274.010838] Switched to NOHz mode on CPU #0
[? 274.010848] Switched to NOHz mode on CPU #50
[? 274.010857] Switched to NOHz mode on CPU #27
[? 274.010967] Switched to NOHz mode on CPU #39
[? 274.011024] Switched to NOHz mode on CPU #62
[? 274.011124] Switched to NOHz mode on CPU #16
[? 274.011133] Switched to NOHz mode on CPU #51
[? 274.011157] Switched to NOHz mode on CPU #4
[? 274.011178] Switched to NOHz mode on CPU #28
[? 274.011307] Switched to NOHz mode on CPU #40
[? 274.011375] Switched to NOHz mode on CPU #63
[? 274.011465] Switched to NOHz mode on CPU #17
[? 274.011476] Switched to NOHz mode on CPU #52
[? 274.011492] Switched to NOHz mode on CPU #5
[? 274.011526] Switched to NOHz mode on CPU #29
[? 274.011656] Switched to NOHz mode on CPU #41
[? 274.011796] Switched to NOHz mode on CPU #18
[? 274.011827] Switched to NOHz mode on CPU #6
[? 274.011847] Switched to NOHz mode on CPU #53
[? 274.011864] Switched to NOHz mode on CPU #30
[? 274.012006] Switched to NOHz mode on CPU #42
[? 274.012126] Switched to NOHz mode on CPU #19
[? 274.012155] Switched to NOHz mode on CPU #7
[? 274.012205] Switched to NOHz mode on CPU #54
[? 274.012215] Switched to NOHz mode on CPU #31
[? 274.012355] Switched to NOHz mode on CPU #43
[? 274.012457] Switched to NOHz mode on CPU #20
[? 274.012466] Switched to NOHz mode on CPU #8
[? 274.012554] Switched to NOHz mode on CPU #32
[? 274.012563] Switched to NOHz mode on CPU #55
[? 274.012692] Switched to NOHz mode on CPU #44
[? 274.012786] Switched to NOHz mode on CPU #9
[? 274.012806] Switched to NOHz mode on CPU #21
[? 274.012891] Switched to NOHz mode on CPU #33
[? 274.012902] Switched to NOHz mode on CPU #56
[? 274.013039] Switched to NOHz mode on CPU #45
[? 274.013126] Switched to NOHz mode on CPU #10
[? 274.013139] Switched to NOHz mode on CPU #22
[? 274.013226] Switched to NOHz mode on CPU #34
[? 274.013248] Switched to NOHz mode on CPU #57
[? 274.013491] Switched to NOHz mode on CPU #58
[? 274.013500] Switched to NOHz mode on CPU #23
[? 274.013510] Switched to NOHz mode on CPU #46
[? 274.013519] Switched to NOHz mode on CPU #35
[? 274.013531] Switched to NOHz mode on CPU #11
[? 274.014829] Switched to NOHz mode on CPU #15
[? 274.018228] NET: Registered protocol family 2
[? 274.018764] IP route cache hash table entries: 524288 (order: 9, 4194304 bytes)
[? 274.020959] TCP established hash table entries: 524288 (order: 10, 8388608 bytes)
[? 274.043706] ds-1: Registered pri service.
[? 274.046488] TCP bind hash table entries: 65536 (order: 7, 1048576 bytes)
[? 274.049897] ds-1: Registered var-config-backup service.
[? 274.050231] TCP: Hash tables configured (established 524288 bind 65536)
[? 274.050256] TCP reno registered
[? 274.050295] UDP hash table entries: 16384 (order: 6, 524288 bytes)
[? 274.052307] UDP-Lite hash table entries: 16384 (order: 6, 524288 bytes)
[? 274.055271] NET: Registered protocol family 1
[? 274.055356] RPC:?????? creating workqueue rpciod
[? 274.055660] RPC:?????? registering /proc/net/rpc
[? 274.055762] svc: Adding svc transport class 'tcp'
[? 274.055790] svc: Adding svc transport class 'udp'
[? 274.055860] RPC: Registered named UNIX socket transport module.
[? 274.055884] RPC: Registered udp transport module.
[? 274.055904] RPC: Registered tcp transport module.
[? 274.055923] RPC: Registered tcp NFSv4.1 backchannel transport module.
[? 274.057280] audit: initializing netlink socket (disabled)
[? 274.057327] type=2000 audit(0.304:1): initialized
[? 274.763711] HugeTLB registered 4 MB page size, pre-allocated 0 pages
[? 274.772451] VFS: Disk quotas dquot_6.5.2
[? 274.772895] Dquot-cache hash table entries: 1024 (order 0, 8192 bytes)
[? 274.773526] RPC:?????? creating workqueue nfsiod
[? 274.774321] RPC:?????? registering /proc/net/rpc/nfs
[? 274.774375] msgmni has been set to 32768
[? 274.775486] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[? 274.775520] io scheduler noop registered
[? 274.775539] io scheduler deadline registered
[? 274.775775] io scheduler cfq registered (default)
[? 274.777268] f027ab74: ttyS0 at I/O 0x0 (irq = 17) is a SUN4V HCONS
[? 274.840726] console [ttyHV0] enabled
[? 274.842372] f028af14: ttyS1 at MMIO 0xfff0ca0000 (irq = 24) is a 16550A
[? 274.843094] [drm] Initialized drm 1.1.0 20060810
[? 274.844259] e1000e: Intel(R) PRO/1000 Network Driver - 1.4.4-k
[? 274.844307] e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
[? 274.844405] e1000e 0000:08:00.0: Disabling ASPM? L1
[? 275.024685] e1000e 0000:08:00.0: eth0: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c2
[? 275.024746] e1000e 0000:08:00.0: eth0: Intel(R) PRO/1000 Network Connection
[? 275.024873] e1000e 0000:08:00.0: eth0: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 275.025011] e1000e 0000:08:00.1: Disabling ASPM? L1
[? 275.202428] e1000e 0000:08:00.1: eth1: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c3
[? 275.202549] e1000e 0000:08:00.1: eth1: Intel(R) PRO/1000 Network Connection
[? 275.202732] e1000e 0000:08:00.1: eth1: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 275.202858] e1000e 0000:09:00.0: Disabling ASPM? L1
[? 275.376782] e1000e 0000:09:00.0: eth2: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c4
[? 275.376902] e1000e 0000:09:00.0: eth2: Intel(R) PRO/1000 Network Connection
[? 275.377086] e1000e 0000:09:00.0: eth2: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 275.377212] e1000e 0000:09:00.1: Disabling ASPM? L1
[? 275.554605] e1000e 0000:09:00.1: eth3: (PCI Express:2.5GT/s:Width x4) 00:14:4f:98:09:c5
[? 275.554726] e1000e 0000:09:00.1: eth3: Intel(R) PRO/1000 Network Connection
[? 275.554909] e1000e 0000:09:00.1: eth3: MAC: 0, PHY: 4, PBA No: FFFFFF-0FF
[? 275.555535] mousedev: PS/2 mouse device common for all mice
[? 275.556563] rtc-sun4v rtc-sun4v: rtc core: registered sun4v as rtc0
[? 275.557267] TCP cubic registered
[? 275.558144] NET: Registered protocol family 10
[? 275.560267] Mobile IPv6
[? 275.560303] NET: Registered protocol family 17
[? 275.560374] Registering the dns_resolver key type
[? 275.560844] registered taskstats version 1
[? 275.563318] rtc-sun4v rtc-sun4v: setting system clock to 2011-11-05 01:46:43 UTC (1320457603)
[? 275.812032] ADDRCONF(NETDEV_UP): eth0: link is not ready
[? 277.936740] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
[? 277.959073] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[? 277.980042] IP-Config: Complete:
[? 277.980115]????? device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=255.255.255.255,
[? 277.980317]????? host=137.226.167.242, domain=, nis-domain=(none),
[? 277.980432]????? bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
[? 277.980910] Root-NFS: nfsroot=/srv/nfs/cluster2,mountproto=tcp
[? 277.981405] NFS: nfs mount opts='vers=2,udp,rsize=4096,wsize=4096,mountproto=tcp,nolock,addr=137.226.167.241'
[? 277.981651] NFS:?? parsing nfs mount option 'vers=2'
[? 277.981739] NFS:?? parsing nfs mount option 'udp'
[? 277.981803] NFS:?? parsing nfs mount option 'rsize=4096'
[? 277.981853] NFS:?? parsing nfs mount option 'wsize=4096'
[? 277.981932] NFS:?? parsing nfs mount option 'mountproto=tcp'
[? 277.982016] NFS:?? parsing nfs mount option 'nolock'
[? 277.982062] NFS:?? parsing nfs mount option 'addr=137.226.167.241'
[? 277.982155] NFS: MNTPATH: '/srv/nfs/cluster2'
[? 277.982218] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
[? 277.982316] RPC:?????? set up xprt to 137.226.167.241 (autobind) via tcp
[? 277.982394] RPC:?????? created transport fffff807fdf40800 with 128 slots
[? 277.982444] RPC:?????? creating mount client for 137.226.167.241 (xprt fffff807fdf40800)
[? 277.982534] RPC:?????? creating UNIX authenticator for client fffff807fdf18600
[? 277.982627] RPC:?????? new task initialized, procpid 1
[? 277.982700] RPC:?????? allocated task fffff807fdf10000
[? 277.982748] RPC:???? 1 __rpc_execute flags=0x680
[? 277.982813] RPC:???? 1 call_start mount1 proc NULL (sync)
[? 277.982887] RPC:???? 1 call_reserve (status 0)
[? 277.982922] RPC:???? 1 reserved req fffff807fdf18400 xid 945eb419
[? 277.983000] RPC:?????? wake_up_next(fffff807fdf40978 "xprt_sending")
[? 277.983078] RPC:???? 1 call_reserveresult (status 0)
[? 277.983122] RPC:???? 1 call_refresh (status 0)
[? 277.983185] RPC:???? 1 holding NULL cred 00000000008b8cf0
[? 277.983230] RPC:???? 1 refreshing NULL cred 00000000008b8cf0
[? 277.983305] RPC:???? 1 call_refreshresult (status 0)
[? 277.983378] RPC:???? 1 call_allocate (status 0)
[? 277.983413] RPC:???? 1 allocated buffer of size 96 at fffff807fdf41000
[? 277.983490] RPC:???? 1 call_bind (status 0)
[? 277.983525] RPC:???? 1 rpcb_getport_async(137.226.167.241, 100005, 1, 6)
[? 277.983617] RPC:???? 1 sleep_on(queue "xprt_binding" time 4294893356)
[? 277.983696] RPC:???? 1 added to queue fffff807fdf408c8 "xprt_binding"
[? 277.983773] RPC:???? 1 setting alarm for 60000 ms
[? 277.983810] RPC:???? 1 rpcb_getport_async: trying rpcbind version 2
[? 277.983899] RPC:?????? set up xprt to 137.226.167.241 (port 111) via tcp
[? 277.983978] RPC:?????? created transport fffff807fdf41800 with 128 slots
[? 277.984057] RPC:?????? creating rpcbind client for 137.226.167.241 (xprt fffff807fdf41800)
[? 277.984115] RPC:?????? creating UNIX authenticator for client fffff807fdf18c00
[? 277.984202] RPC:?????? new task initialized, procpid 1
[? 277.984274] RPC:?????? allocated task fffff807fdf100e0
[? 277.984355] RPC:?????? rpc_release_client(fffff807fdf18c00)
[? 277.984403] RPC:???? 1 sync task going to sleep
[? 277.984472] RPC:???? 2 __rpc_execute flags=0x681
[? 277.984536] RPC:???? 2 call_start rpcbind2 proc GETPORT (async)
[? 277.984583] RPC:???? 2 call_reserve (status 0)
[? 277.984646] RPC:???? 2 reserved req fffff807fdf18a00 xid e3ac2074
[? 277.984693] RPC:?????? wake_up_next(fffff807fdf41978 "xprt_sending")
[? 277.984771] RPC:???? 2 call_reserveresult (status 0)
[? 277.984814] RPC:???? 2 call_refresh (status 0)
[? 277.984847] RPC:???? 2 looking up UNIX cred
[? 277.984877] RPC:?????? looking up UNIX cred
[? 277.984910] RPC:?????? allocating UNIX cred for uid 0 gid 0
[? 277.984958] RPC:???? 2 refreshing UNIX cred fffff807fdefc900
[? 277.985004] RPC:???? 2 call_refreshresult (status 0)
[? 277.985048] RPC:???? 2 call_allocate (status 0)
[? 277.985082] RPC:???? 2 allocated buffer of size 416 at fffff807fdf42000
[? 277.985130] RPC:???? 2 call_bind (status 0)
[? 277.985163] RPC:???? 2 call_connect xprt fffff807fdf41800 is not connected
[? 277.985213] RPC:???? 2 xprt_connect xprt fffff807fdf41800 is not connected
[? 277.985263] RPC:???? 2 sleep_on(queue "xprt_pending" time 4294893356)
[? 277.985312] RPC:???? 2 added to queue fffff807fdf41a28 "xprt_pending"
[? 277.985358] RPC:???? 2 setting alarm for 60000 ms
[? 277.985393] RPC:?????? xs_connect scheduled xprt fffff807fdf41800
[? 277.985553] RPC:?????? xs_bind 0.0.0.0:0: ok (0)
[? 277.985590] RPC:?????? worker connecting xprt fffff807fdf41800 via tcp to 137.226.167.241 (port 111)
[? 277.985750] RPC:?????? fffff807fdf41800 connect status 36 connected 0 sock state 2
[? 280.989403] RPC:?????? xs_error_report client fffff807fdf41800...
[? 280.989411] RPC:?????? error 65
[? 280.989568] RPC:???? 2 __rpc_wake_up_task (now 4294894107)
[? 280.989670] RPC:???? 2 disabling timer
[? 280.989731] RPC:???? 2 removed from queue fffff807fdf41a28 "xprt_pending"
[? 280.989808] RPC:?????? __rpc_wake_up_task done
[? 280.989843] RPC:?????? xs_tcp_state_change client fffff807fdf41800...
[? 280.989893] RPC:?????? state 7 conn 0 dead 0 zapped 1 sk_shutdown 3
[? 280.989968] RPC:?????? disconnected transport fffff807fdf41800
[? 280.990037] RPC:???? 2 __rpc_execute flags=0x681
[? 280.990101] RPC:???? 2 xprt_connect_status: retrying
[? 280.990174] RPC:???? 2 call_connect_status (status -11)
[? 280.990219] RPC:???? 2 call_transmit (status 0)
[? 280.990279] RPC:???? 2 xprt_prepare_transmit
[? 280.990312] RPC:???? 2 rpc_xdr_encode (status 0)
[? 280.990374] RPC:???? 2 marshaling UNIX cred fffff807fdefc900
[? 280.990423] RPC:???? 2 using AUTH_UNIX cred fffff807fdefc900 to wrap rpc data
[? 280.990504] RPC:???? 2 encoding PMAP_GETPORT call (100005, 1, 6, 0)
[? 280.990580] RPC:???? 2 xprt_transmit(96)
[? 280.990620] RPC:?????? xs_tcp_send_request(96) = -65
[? 280.990692] RPC:?????? sendmsg returned unrecognized error 65
[? 280.990768] RPC:?????? xs_tcp_state_change client fffff807fdf41800...
[? 280.990817] RPC:?????? state 7 conn 0 dead 0 zapped 1 sk_shutdown 3
[? 280.990892] RPC:?????? disconnected transport fffff807fdf41800
[? 280.990970] RPC:?????? wake_up_next(fffff807fdf41978 "xprt_sending")
[? 280.991020] RPC:?????? setting port for xprt fffff807fdf40800 to 0
[? 280.991098] RPC:???? 2 rpcb_getport_done(status -65, port 0)
[? 280.991173] RPC:???? 2 return 0, status -65
[? 280.991203] RPC:???? 2 release task
[? 280.991265] RPC:?????? freeing buffer of size 416 at fffff807fdf42000
[? 280.991315] RPC:???? 2 release request fffff807fdf18a00
[? 280.991389] RPC:?????? wake_up_next(fffff807fdf41ad8 "xprt_backlog")
[? 280.991465] RPC:?????? rpc_release_client(fffff807fdf18c00)
[? 280.991539] RPC:?????? destroying rpcbind client for 137.226.167.241
[? 280.991587] RPC:?????? destroying transport fffff807fdf41800
[? 280.991695] RPC:?????? xs_destroy xprt fffff807fdf41800
[? 280.991791] RPC:?????? xs_close xprt fffff807fdf41800
[? 280.991880] RPC:?????? disconnected transport fffff807fdf41800
[? 280.991958] RPC:???? 2 freeing task
[? 280.991989] RPC:???? 1 __rpc_wake_up_task (now 4294894107)
[? 280.992062] RPC:???? 1 disabling timer
[? 280.992094] RPC:???? 1 removed from queue fffff807fdf408c8 "xprt_binding"
[? 280.992174] RPC:?????? __rpc_wake_up_task done
[? 280.992189] RPC:???? 1 sync task resuming
[? 280.992199] RPC:???? 1 remote rpcbind unreachable: -65
[? 280.992208] RPC:???? 1 return 0, status -65
[? 280.992216] RPC:???? 1 release task
[? 280.992227] RPC:?????? freeing buffer of size 96 at fffff807fdf41000
[? 280.992240] RPC:???? 1 release request fffff807fdf18400
[? 280.992250] RPC:?????? wake_up_next(fffff807fdf40ad8 "xprt_backlog")
[? 280.992263] RPC:?????? rpc_release_client(fffff807fdf18600)
[? 280.992273] RPC:???? 1 freeing task
[? 280.992283] RPC:?????? shutting down mount client for 137.226.167.241
[? 280.992292] RPC:?????? rpc_release_client(fffff807fdf18600)
[? 280.992301] RPC:?????? destroying UNIX authenticator 00000000008b8d78
[? 280.992314] RPC:?????? destroying mount client for 137.226.167.241
[? 280.992328] NFS: failed to create MNT RPC client, status=-65
[? 280.992339] NFS: unable to mount server 137.226.167.241, error -65
[? 280.992373] VFS: Unable to mount root fs via NFS, trying floppy.
[? 280.992433] List of all partitions:
[? 280.992443] No filesystem could mount root, tried:
[? 280.992458] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
[? 280.992469] Call Trace:
[? 280.992498]? [00000000008c6d18] mount_block_root+0x210/0x228
[? 280.992512]? [00000000008c6df4] mount_root+0xc4/0xd4
[? 280.992525]? [00000000008c6f80] prepare_namespace+0x17c/0x1c0
[? 280.992542]? [00000000008c6250] kernel_init+0x140/0x158
[? 280.992562]? [000000000042ac68] kernel_thread+0x30/0x48
[? 280.992580]? [0000000000767840] rest_init+0x18/0x74
[? 280.992659] Press Stop-A (L1-A) to return to the boot prom
[? 280.993582] RPC:?????? destroying transport fffff807fdf40800
[? 280.993715] RPC:?????? xs_destroy xprt fffff807fdf40800
[? 280.993806] RPC:?????? xs_close xprt fffff807fdf40800
[? 280.993881] RPC:?????? disconnected transport fffff807fdf40800
[? 288.549074] eth0: no IPv6 routers present

2011-11-04 14:25:18

by J. Bruce Fields

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

On Fri, Nov 04, 2011 at 07:09:41AM -0700, Myklebust, Trond wrote:
> > -----Original Message-----
> > From: Chuck Lever [mailto:[email protected]]
> > Sent: Friday, November 04, 2011 10:02 AM
> > To: Jim Rees
> > Cc: Lukas Razik; Myklebust, Trond; Linux NFS Mailing List
> > Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64
> >
> >
> > On Nov 4, 2011, at 9:20 AM, Jim Rees wrote:
> > > As we move toward nfs4 someone will have to give some thought to
> > nfsroot.
> > > It's hard to imagine we could put enough nfs4 cruft into the kernel
> > > (gssd,
> > > idmapd) to make it work.
> >
> > A kernel-level basic id mapper is being considered. That would allow
> NFSv4
> > with AUTH_SYS, if we can get the NIC problems squared away.
>
> We already have a solution for NFSv4 and nfsroot: use auth_sys and set
> the nfs4_disable_idmapping flag. That does require an rfc3530-bis
> compatible server, but it shouldn't be long before we see those
> appearing in the wild...

And I *do* want to get back to that, I just haven't had the time.
(Patches welcome if someone else is motivated.)

--b.

2011-11-03 21:11:00

by Chuck Lever III

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64


On Nov 3, 2011, at 4:54 PM, Trond Myklebust wrote:

> On Thu, 2011-11-03 at 19:43 +0000, Lukas Razik wrote:
>> Hello together!
>>
>> My OS: Debian 6.0.3 (squeeze)
>> Machines: SUN Enterprise T5120 (USPARC64)
>> ---
>> Issue description:
>>
>> I've an NFS
>> server (cluster1=137.226.167.241) and a
>> client (cluster2=137.226.167.242) which should mount it's nfsroot from cluster1.
>>
>> The linux-2.6.32 kernel on cluster2 shows this during startup:
>> [ 528.982985] IP-Config: Complete:
>> [ 528.983049] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
>> [ 528.983299] host=cluster2, domain=, nis-domain=(none),
>> [ 528.983383] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
>> [ 528.983633] Looking up port of RPC 100003/2 on 137.226.167.241
>> [ 530.037059] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
>> [ 530.056881] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 564.002113] rpcbind: server 137.226.167.241 not responding, timed out
>> [ 564.002295] Root-NFS: Unable to get nfsd port number from server, using default
>> [ 564.002412] Looking up port of RPC 100005/1 on 137.226.167.241
>> [ 564.104137] VFS: Mounted root (nfs filesystem) on device 0:15.
>>
>> It can mount the nfsroot finally.
>>
>> But if I use kernel linux-2.6.39.4 on cluster2 it can't mount it's nfsroot.
>> (I've added "nfsdebug" to the kernel arguments for more debug info):
>> [ 407.571521] IP-Config: Complete:
>> [ 407.571589] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
>> [ 407.571793] host=cluster2, domain=, nis-domain=(none),
>> [ 407.571907] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
>> [ 407.572332] Root-NFS: nfsroot=/srv/nfs/cluster2
>> [ 407.572726] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
>> [ 407.572927] NFS: parsing nfs mount option 'udp'
>> [ 407.572995] NFS: parsing nfs mount option 'nolock'
>> [ 407.573071] NFS: parsing nfs mount option 'addr=137.226.167.241'
>> [ 407.573139] NFS: MNTPATH: '/srv/nfs/cluster2'
>> [ 407.573203] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
>> [ 408.617894] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
>> [ 408.638319] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
>> [ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
>> [ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
>> [ 442.667032] VFS: Insert root floppy and press ENTER
>>
> Error 60 is ETIMEDOUT on SPARC, so it seems that the problem is
> basically the same one that you see in your 2.6.32 trace (rpcbind:
> server 137.226.167.241 not responding, timed out) except that now it is
> a fatal error.
>
> Any idea why the first RPC calls might be failing here? A switch
> misconfiguration or something like that perhaps?

Yeah, I'm not clear how the system can do kernel IP configuration with the NIC not yet initialized.

In any event, these RPC requests are supposed to be over UDP, and they should be retransmitted, making the timing of NIC readiness immaterial. That's the design, anyway.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com





2011-11-03 21:38:36

by Jim Rees

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

Lukas Razik wrote:

The next thing is:
Really all working kernels (<=2.6.36.4) first output
?Looking up port of RPC 100003/2 on 137.226.167.241
then
?Looking up port of RPC 100005/1 on 137.226.167.241
and then the mount is successful
?VFS: Mounted root (nfs filesystem) on device 0:15.

So what about >=2.6.37?
Why don't these kernels try other ports, too?
Or why do the old kernels try more than one port?
Why is there no output (even in the nfsdebug mode) that the kernel tries to connect to the RPC service?
Is there a "easy" possibility to change port 100003 to 100005 in >=2.6.37?

Those are the rpc numbers. The kernel is trying to find the port numbers
for those services. 100003 is nfs, 100005 is mount. I guess at this point
I would use wireshark to find out what requests are actually being made and
responded to in both cases. Are you sure portmap is working on the server
and not being blocked by a firewall?

2011-11-04 15:02:35

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> Chuck Lever wrote:

>
> ? On Nov 4, 2011, at 9:20 AM, Jim Rees wrote:
> ?
> ? > As for a fix...? we're trying to move away from udp transport
> anyway.? Maybe
> ? > someone should figure out a way to get it to work with tcp?? I have zero
> ? > experience with nfs over udp, at least on linux.? Just for fun, have you
> ? > tried tcp transport (proto=tcp)?
> ?
> ? TCP is a real problem in this environment, because it deals poorly with
> ? NIC initialization timing issues.? UDP is still the best approach (as long
> ? as it is retransmitting appropriately).? To support TCP, ultimately what
> ? we need to do is to introduce serialization to make the kernel wait for
> ? the NIC to become ready before attempting network activity.
>
> Agreed, but we're grasping at straws here, and this guy just wants it to
> work.? It's something to try.

Yes, this could be a fast workaround. I'll test it in some minutes.
Thanks for that and till later then...

2011-11-12 11:38:09

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> Jim Rees <[email protected]> wrote:

> Lukas Razik wrote:
>
> ? > So, once again...
> ? >
> ? > What kind of switch is this client connected to, and how is that switch
> ? > configured?
> ?
> ? I can't answer this question because I have no physical access to the
> ? hardware.? Hence I think it's logical to stop here until the nodes are
> ? connected directly (without any hardware between them).? I'll write here
> ? as soon as I know if it works...
> ?
> ? For the rest I thank all contributors for the fast help!
>
> I think the implication here is that you're plugged in to a port that's
> running spanning tree (stp) and is not set to portfast.? Such a port can
> take up to a minute to start forwarding packets.? If so, you should fix
> that, but it would also be nice if nfsroot could be made to work in this
> setup.
>
> By the way, if you want to send me that machine with its 64 processors and
> 32GB of memory, I'd be glad to investigate.
>

Hello again!

I've connected the systems directly (without a switch between them).
It seems that the problem came from the switch because now the newer kernels also mount their NFSROOTs during start up without problems.

Assuming that the ports of the switch really run in STP mode - two questions:
1. Is it unusual that the ports of a switch where computers are connected to run in STP mode?
2. If that's not unusual: Wouldn't it be useful if the new linux kernels could mount an NFSROOT even when the system is plugged into an STP port?

The problem is that we'll have 16 such systems which can't be connected directly - they must communicate through the switch. And I've no access to the switch.

Best regards,
Lukas

2011-11-05 13:05:26

by Jim Rees

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

Lukas Razik wrote:

> So, once again...
>
> What kind of switch is this client connected to, and how is that switch
> configured?

I can't answer this question because I have no physical access to the
hardware. Hence I think it's logical to stop here until the nodes are
connected directly (without any hardware between them). I'll write here
as soon as I know if it works...

For the rest I thank all contributors for the fast help!

I think the implication here is that you're plugged in to a port that's
running spanning tree (stp) and is not set to portfast. Such a port can
take up to a minute to start forwarding packets. If so, you should fix
that, but it would also be nice if nfsroot could be made to work in this
setup.

By the way, if you want to send me that machine with its 64 processors and
32GB of memory, I'd be glad to investigate.

2011-11-04 15:04:31

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

>> It would be very sad with these nice Oracle (SUN) machines if no one could

> use them because of this bug.
>
> If you boot via tftp, I bet this problem will go away because the network
> interface will be working by the time the NFSROOT mount is attempted.

Yes this could be a workaround but there is another problem with boot via tftp (and OpenBoot) which another team member is trying to solve...


> The NFSROOT code assumes that if kernel IP configuration worked, then the NIC is
> already up.? That is clearly not the case if you boot from your local disk.

Yes, that's surely not the standard way: To mount nfsroot although you have a local disk...


>> Do you know a kernel developer who maybe would try to write a patch for
> this problem?
>> Or do you have another idea what I could do?
>
> As for a patch: no-one can write a patch unless we understand precisely why the
> first RPC fails.? I already explained how to add a line or two to
> fs/nfs/nfsroot.c to give us more information.? If you need a patch to do this, I
> can send one later today.
>
> I might be able to reproduce it here, now that I understand your set up, but it
> would require building a partial NFSROOT environment.? I can't get to that
> until next week.
>

If you have the time for the patch, it would be very nice! Otherwise I'll try it as soon as I can.
The problem is that the system should work as soon as possible. So it would be nice to have a work around.
In parallel, I'll give you all information I have and test all patches you give me as long as you all are interested.

So now I'm trying the proposed workaroundsand after that I'll try to add the lines you wrote about (ASAP)...

Regards,
Lukas

2011-11-03 21:24:55

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

> On Thu, 2011-11-03 at 19:43 +0000, Lukas Razik wrote:

>> Hello together!
>>
>> My OS: Debian 6.0.3 (squeeze)
>> Machines: SUN Enterprise T5120 (USPARC64)
>> ---
>> Issue description:
>>
>> I've an NFS
>> server (cluster1=137.226.167.241) and a
>> client (cluster2=137.226.167.242) which should mount it's nfsroot from
> cluster1.
>>
>> The linux-2.6.32 kernel on cluster2 shows this during startup:
>> [ 528.982985] IP-Config: Complete:
>> [ 528.983049] device=eth0, addr=137.226.167.242, mask=255.255.255.224,
> gw=137.226.167.225,
>> [ 528.983299] host=cluster2, domain=, nis-domain=(none),
>> [ 528.983383] bootserver=255.255.255.255, rootserver=137.226.167.241,
> rootpath=
>> [ 528.983633] Looking up port of RPC 100003/2 on 137.226.167.241
>> [ 530.037059] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: Rx
>> [ 530.056881] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 564.002113] rpcbind: server 137.226.167.241 not responding, timed out
>> [ 564.002295] Root-NFS: Unable to get nfsd port number from server, using
> default
>> [ 564.002412] Looking up port of RPC 100005/1 on 137.226.167.241
>> [ 564.104137] VFS: Mounted root (nfs filesystem) on device 0:15.
>>
>> It can mount the nfsroot finally.
>>
>> But if I use kernel linux-2.6.39.4 on cluster2 it can't mount it's
> nfsroot.
>> (I've added "nfsdebug" to the kernel arguments for more debug
> info):
>> [ 407.571521] IP-Config: Complete:
>> [ 407.571589] device=eth0, addr=137.226.167.242, mask=255.255.255.224,
> gw=137.226.167.225,
>> [ 407.571793] host=cluster2, domain=, nis-domain=(none),
>> [ 407.571907] bootserver=255.255.255.255, rootserver=137.226.167.241,
> rootpath=
>> [ 407.572332] Root-NFS: nfsroot=/srv/nfs/cluster2
>> [ 407.572726] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
>> [ 407.572927] NFS: parsing nfs mount option 'udp'
>> [ 407.572995] NFS: parsing nfs mount option 'nolock'
>> [ 407.573071] NFS: parsing nfs mount option 'addr=137.226.167.241'
>> [ 407.573139] NFS: MNTPATH: '/srv/nfs/cluster2'
>> [ 407.573203] NFS: sending MNT request for
> 137.226.167.241:/srv/nfs/cluster2
>> [ 408.617894] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: Rx
>> [ 408.638319] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 442.666622] NFS: failed to create MNT RPC client, status=-60
>> [ 442.666732] NFS: unable to mount server 137.226.167.241, error -60
>> [ 442.666868] VFS: Unable to mount root fs via NFS, trying floppy.
>> [ 442.667032] VFS: Insert root floppy and press ENTER
>>
> Error 60 is ETIMEDOUT on SPARC, so it seems that the problem is
> basically the same one that you see in your 2.6.32 trace (rpcbind:
> server 137.226.167.241 not responding, timed out) except that now it is
> a fatal error.
>
> Any idea why the first RPC calls might be failing here? A switch
> misconfiguration or something like that perhaps?
>

Honestly, I must state that I also thought of any hardware between the nodes etc. in our computing centre which could cause this fault. Therefore I want to connect the nodes directly but this will take some days (because of bureaucracy)... :(

The next thing is:
Really all working kernels (<=2.6.36.4) first output
?Looking up port of RPC 100003/2 on 137.226.167.241
then
?Looking up port of RPC 100005/1 on 137.226.167.241
and then the mount is successful
?VFS: Mounted root (nfs filesystem) on device 0:15.

So what about >=2.6.37?
Why don't these kernels try other ports, too?
Or why do the old kernels try more than one port?
Why is there no output (even in the nfsdebug mode) that the kernel tries to connect to the RPC service?
Is there a "easy" possibility to change port 100003 to 100005 in >=2.6.37?

Many thanks for your fast answer!

Regards,
Lukas

2011-11-13 19:32:47

by Chuck Lever III

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64


On Nov 12, 2011, at 8:03 PM, Lukas Razik wrote:

> Chuck Lever <[email protected]> wrote:
>
>> On Nov 12, 2011, at 1:49 PM, Jim Rees wrote:
>>
>>> The question for us is how long should an nfsroot client wait for the
>> server
>>> to reply. It sounds like the client used to wait longer than it does now.
>>
>> Before, the client performed the GETPORT(NFS) step synchronously, first. This
>> took 30 seconds or so to timeout. When it did, the client decided to proceed
>> with port 2049. Then it went on to do the other mount tasks, and at the point
>> had waited long enough that these tasks did not time out while waiting for the
>> switch port.
>>
>>> It seems to me the client should wait at least 90 seconds so that the
>>> situation you're in (servers on non-portfast ports) will work. I would
>>> think they should wait indefinitely, since there's not much else they
>> can
>>> do.
>>
>> It should be simple to wrap the (MNT(mnt), NFS(getroot)) steps in a while(true)
>> loop. Would mount_root_nfs() be the right place for this?
>>
>
> I thought it would be harder and I had no time to look inside the kernel but now I wrote a patch:
> The kernel tries to create the MNT RPC client not once as before but three times - then it gives up.

This changes the behavior of normal user space NFS mounts as well. User space mounts are already retried by the mount.nfs program. We risk making the mount(2) system call hang by retrying here.

I think we prefer a solution that changes the behavior of only NFSROOT mounts. Have you tried wrapping mount_root_nfs() as I suggested?

> Third time lucky... ;-)
> In my case the 2. MNT request is successful:
> ---
> [ 71.594744] ADDRCONF(NETDEV_UP): eth0: link is not ready
> [ 72.617007] IP-Config: Complete:
> [ 72.617077] device=eth0, addr=137.226.167.242, mask=255.255.255.224, gw=137.226.167.225,
> [ 72.617278] host=137.226.167.242, domain=, nis-domain=(none),
> [ 72.617393] bootserver=255.255.255.255, rootserver=137.226.167.241, rootpath=
> [ 72.617741] Root-NFS: nfsroot=/srv/nfs/cluster2
> [ 72.618010] NFS: nfs mount opts='udp,nolock,addr=137.226.167.241'
> [ 72.618147] NFS: parsing nfs mount option 'udp'
> [ 72.618187] NFS: parsing nfs mount option 'nolock'
> [ 72.618233] NFS: parsing nfs mount option 'addr=137.226.167.241'
> [ 72.618301] NFS: MNTPATH: '/srv/nfs/cluster2'
> [ 72.618335] NFS: sending MNT request for 137.226.167.241:/srv/nfs/cluster2
> [ 72.618383] NFS: 1. MNT request
> [ 73.691872] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx
> [ 73.711988] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 107.697332] NFS: 2. MNT request
> [ 107.704591] NFS: received 1 auth flavors
> [ 107.704653] NFS: auth flavor[0]: 1
> [ 107.704834] NFS: MNT request succeeded
> [ 107.704897] NFS: using auth flavor 1
> [ 107.711857] VFS: Mounted root (nfs filesystem) on device 0:13.
> INIT: version 2.88 booting
> ---
>
> So many thanks again for your help and your very helpful hints!
>
> Regards,
> Lukas
>
>
> PS: That's what I've done:
> --- linux-2.6.39.4/fs/nfs/mount_clnt.c 2011-08-03 21:43:28.000000000 +0200
> +++ linux-2.6.39.4-fix/fs/nfs/mount_clnt.c 2011-11-13 01:58:13.000000000 +0100
> @@ -164,6 +164,7 @@
> };
> struct rpc_clnt *mnt_clnt;
> int status;
> + int attempt = 0;
>
> dprintk("NFS: sending MNT request for %s:%s\n",
> (info->hostname ? info->hostname : "server"),
> @@ -172,7 +173,13 @@
> if (info->noresvport)
> args.flags |= RPC_CLNT_CREATE_NONPRIVPORT;
>
> - mnt_clnt = rpc_create(&args);
> + do {
> + attempt++;
> + dprintk("NFS: %d. MNT request\n", attempt);
> + mnt_clnt = rpc_create(&args);
> + } while (IS_ERR(mnt_clnt) && attempt < 3);
> +
> +
> if (IS_ERR(mnt_clnt))
> goto out_clnt_err;
> --

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com





2011-11-04 09:44:20

by Lukas Razik

[permalink] [raw]
Subject: Re: [BUG?] Maybe NFS bug since 2.6.37 on SPARC64

>> OK

>> I've watched wireshark on cluster1 during start up of cluster2 (with
> linux-2.6.32) which first tries 10003 and then 10005.
>> The result is that cluster1 doesn't get a datagram for port 10003:
>> http://net.razik.de/linux/T5120/cluster2_NFSROOT_MOUNT.png
>>
>> The first ARP request in the screenshot came _after_ the <tag> in
> this kernel log:
>> [ 6492.807917] IP-Config: Complete:
>> [ 6492.807978]? ? ? device=eth0, addr=137.226.167.242,
> mask=255.255.255.224, gw=137.226.167.225,
>> [ 6492.808227]? ? ? host=cluster2, domain=, nis-domain=(none),
>> [ 6492.808312]? ? ? bootserver=255.255.255.255, rootserver=137.226.167.241,
> rootpath=
>> [ 6492.808570] Looking up port of RPC 100003/2 on 137.226.167.241
>> [ 6493.886014] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: Rx
>> [ 6493.905840] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> <tag>
>> [ 6527.827055] rpcbind: server 137.226.167.241 not responding, timed out
>> [ 6527.827237] Root-NFS: Unable to get nfsd port number from server, using
> default
>> [ 6527.827353] Looking up port of RPC 100005/1 on 137.226.167.241
>> [ 6527.842212] VFS: Mounted root (nfs filesystem) on device 0:15.
>>
>>
>> So I don't think that it's a problem of the hardware between the
> machines.
>> There's no reason why I wouldn't see an ARP requests from cluster2
> which would have been sent _before_ the <tag> if there would be one. I
> think: cluster2 never sends a request to port 10003.
>> What do you think?
>
> It agrees with our initial assessment that the first RPC request is failing.?
> The RPC client never gets the request through cluster2's network stack
> because the NIC hasn't re-initialized when the request is sent.
>
> It looks like your system does a PXE boot, which provides the IP configuration
> shown above.? But then the kernel resets the NIC.? During that reset, the kernel
> is attempting to contact the NFS server to mount the root file system.
>
> We've set up NFSROOT to use UDP so that it will be relatively immune to
> these initialization order problems.? The RPC client should be retrying the lost
> request, but apparently it isn't.? What if you added "retrans=10"
> to cluster2's mount options?? (on the chance that mount option setting would
> be copied to the rpcbind client's RPC transport...)
>
> IMO the correct way to fix this is to provide proper serialization in the
> networking layer so that RPC requests are not even attempted until the NIC is
> ready to carry traffic.? That may be a pipe dream though.
>

I thank you three very much for your help! Now I'm sure that I haven't misconfigured anything...
But I don't see a work around to get the NFSROOT mounted during start up of a kernel >=2.6.37 .
It would be very sad with these nice Oracle (SUN) machines if no one could use them because of this bug.

Do you know a kernel developer who maybe would try to write a patch for this problem?
Or do you have another idea what I could do?

Regards,
Lukas