DQpIaSBhbGwsDQoNCk15IGVtYmVkZGVkIGRldmVsb3BtZW50IGJveCBmYWlscyB0byBORlMtYm9v
dCB3aXRoIE5GUyBzZXJ2ZXIgd2hpY2ggdXNlcyByZWNlbnQga2VybmVsLg0KDQpVc2luZyBnaXQg
YmlzZWN0LCBJIGZvdW5kIGl0IGlzIGNhdXNlZCBieSBjb21taXQgNGJkYzMzZWQgKCJORlNEdjQu
MjogQWRkIE5GUyB2NC4yIHN1cHBvcnQgdG8gdGhlIE5GUyBzZXJ2ZXIiKS4NCg0KDQoxLiBkbWVz
ZyAoTkZTIGJvb3QgZmFpbHVyZSBjYXNlKQ0KDQouLi4NClsgICAgMi4wNDA4OTNdIEFERFJDT05G
KE5FVERFVl9VUCk6IGV0aDA6IGxpbmsgaXMgbm90IHJlYWR5DQpbICAgIDIuMDQ2MjA3XSBlMTAw
MDogZXRoMCBOSUMgTGluayBpcyBVcCAxMDAwIE1icHMgRnVsbCBEdXBsZXgsIEZsb3cgQ29udHJv
bDogUlgNClsgICAgMi4wNTM1NzBdIEFERFJDT05GKE5FVERFVl9DSEFOR0UpOiBldGgwOiBsaW5r
IGJlY29tZXMgcmVhZHkNClsgICAgMy4wNTUwMjNdIElQLUNvbmZpZzogR3Vlc3NpbmcgbmV0bWFz
ayAyNTUuMjU1LjAuMA0KWyAgICAzLjA1OTk3OV0gSVAtQ29uZmlnOiBHYXRld2F5IG5vdCBvbiBk
aXJlY3RseSBjb25uZWN0ZWQgbmV0d29yay4NClsgICAgMy4wNjYzMzBdIExvb2tpbmcgdXAgcG9y
dCBvZiBSUEMgMTAwMDAzLzIgb24gMTY1LjIxMy44OC4yNDkNClsgICAgMy4wNzQwMDFdIExvb2tp
bmcgdXAgcG9ydCBvZiBSUEMgMTAwMDA1LzEgb24gMTY1LjIxMy44OC4yNDkNClsgICAgMy4xMjI4
NzhdIFZGUzogVW5hYmxlIHRvIG1vdW50IHJvb3QgZnMgdmlhIE5GUywgdHJ5aW5nIGZsb3BweS4N
ClsgICAgMy4xMjkxMzRdIFZGUzogQ2Fubm90IG9wZW4gcm9vdCBkZXZpY2UgIm5mcyIgb3IgdW5r
bm93bi1ibG9jaygyLDApDQpbICAgIDMuMTM1NDc4XSBQbGVhc2UgYXBwZW5kIGEgY29ycmVjdCAi
cm9vdD0iIGJvb3Qgb3B0aW9uOyBoZXJlIGFyZSB0aGUgYXZhaWxhYmxlIHBhcnRpdGlvbnM6DQpb
ICAgIDMuMTQzODMxXSAxZjAwICAgICAgICAgICAgMzA3MiBtdGRibG9jazAgKGRyaXZlcj8pDQpb
ICAgIDMuMTQ4Nzk4XSAxZjAxICAgICAgICAgICAgICA2NCBtdGRibG9jazEgKGRyaXZlcj8pDQpb
ICAgIDMuMTUzNzU4XSAxZjAyICAgICAgICAgICAgICA2NCBtdGRibG9jazIgKGRyaXZlcj8pDQpb
ICAgIDMuMTU4NzE5XSAxZjAzICAgICAgICAgICAgICA2NCBtdGRibG9jazMgKGRyaXZlcj8pDQpb
ICAgIDMuMTYzNjgyXSAxZjA0ICAgICAgICAgICAgICA2NCBtdGRibG9jazQgKGRyaXZlcj8pDQpb
ICAgIDMuMTY4NjQ0XSAxZjA1ICAgICAgICAgICAgICA2NCBtdGRibG9jazUgKGRyaXZlcj8pDQpb
ICAgIDMuMTczNjA3XSAxZjA2ICAgICAgICAgICAgICA2NCBtdGRibG9jazYgKGRyaXZlcj8pDQpb
ICAgIDMuMTc4NTY4XSAwODAwICAgICAgIDQ4ODM4NjU4NCBzZGEgZHJpdmVyOiBzZA0KWyAgICAz
LjE4MzA5OV0gICAwODAxICAgICAgICAgIDUwNjAxNiBzZGExDQpbICAgIDMuMTg2OTI3XSAgIDA4
MDIgICAgICAgICA0MDA4MjE3IHNkYTINClsgICAgMy4xOTA3NTVdICAgMDgwMyAgICAgICA0ODM4
Njk3Njcgc2RhMw0KWyAgICAzLjE5NDU4NF0gYjMwMCAgICAgICAgIDE4ODAwNjQgbW1jYmxrMCBk
cml2ZXI6IG1tY2Jsaw0KWyAgICAzLjE5OTgwMl0gICBiMzAxICAgICAgICAgICAgNDA5NiBtbWNi
bGswcDENClsgICAgMy4yMDQwNjNdICAgYjMwMiAgICAgICAgICAxMDI0MDAgbW1jYmxrMHAyDQpb
ICAgIDMuMjA4MzMwXSAgIGIzMDMgICAgICAgICAgICA0MDk2IG1tY2JsazBwMw0KWyAgICAzLjIx
MjU5NF0gICBiMzA0ICAgICAgICAgICAgICAgMSBtbWNibGswcDQNClsgICAgMy4yMTY4NTVdICAg
YjMwNSAgICAgICAgICAgIDIwNDggbW1jYmxrMHA1DQpbICAgIDMuMjIxMTE2XSAgIGIzMDYgICAg
ICAgICAgICAyMDQ4IG1tY2JsazBwNg0KWyAgICAzLjIyNTM4Ml0gICBiMzA3ICAgICAgICAgICAg
MjA0OCBtbWNibGswcDcNClsgICAgMy4yMjk2NDRdICAgYjMwOCAgICAgICAgICAgIDQwOTYgbW1j
YmxrMHA4DQpbICAgIDMuMjMzOTA2XSAgIGIzMDkgICAgICAgICAgIDEyMjg4IG1tY2JsazBwOQ0K
WyAgICAzLjIzODE3Nl0gICBiMzBhICAgICAgICAgICAxNjM4NCBtbWNibGswcDEwDQpbICAgIDMu
MjQyNTI0XSAgIGIzMGIgICAgICAgICAgMTQyMzM2IG1tY2JsazBwMTENClsgICAgMy4yNDY4Njld
ICAgYjMwYyAgICAgICAgIDE1NzI4NjQgbW1jYmxrMHAxMg0KWyAgICAzLjI1MTIxOV0gYjMyMCAg
ICAgICAgICAgMTIyODggbW1jYmxrMGdwMSAoZHJpdmVyPykNClsgICAgMy4yNTYyNzJdIGIzMTAg
ICAgICAgICAgIDEyMjg4IG1tY2JsazBncDAgKGRyaXZlcj8pDQpbICAgIDMuMjYxMzIwXSBLZXJu
ZWwgcGFuaWMgLSBub3Qgc3luY2luZzogVkZTOiBVbmFibGUgdG8gbW91bnQgcm9vdCBmcyBvbiB1
bmtub3duLWJsb2NrKDIsMCkNClsgICAgMy4yNjk1NjZdIFBpZDogMSwgY29tbTogc3dhcHBlciBO
b3QgdGFpbnRlZCAyLjYuMzUgIzENClsgICAgMy4yNzQ3NzZdIENhbGwgVHJhY2U6DQpbICAgIDMu
Mjc3MjMyXSAgWzw4MGQwZGI1Yj5dID8gcHJpbnRrKzB4MWUvMHgyMA0KWyAgICAzLjI4MTQ5Ml0g
IFs8ODBkMGRhZDE+XSBwYW5pYysweDY1LzB4ZDENClsgICAgMy4yODU0OTVdICBbPDgwZWI5Y2Uz
Pl0gbW91bnRfYmxvY2tfcm9vdCsweDEyNS8weDFiZQ0KWyAgICAzLjI5MDYzMV0gIFs8ODA5ZDFm
NmQ+XSA/IHN5c19ta25vZCsweDJkLzB4MzANClsgICAgMy4yOTUxNTZdICBbPDgwZWI5ZjZkPl0g
bW91bnRfcm9vdCsweGQwLzB4ZjINClsgICAgMy4yOTk1OTFdICBbPDgwZWJhMGQ5Pl0gcHJlcGFy
ZV9uYW1lc3BhY2UrMHgxNGEvMHgxODQNClsgICAgMy4zMDQ4MDNdICBbPDgwOWM0NGY2Pl0gPyBz
eXNfYWNjZXNzKzB4MjYvMHgzMA0KWyAgICAzLjMwOTQxMV0gIFs8ODBlYjlhNGU+XSBrZXJuZWxf
aW5pdCsweDI1ZS8weDI2ZQ0KWyAgICAzLjMxNDEwNV0gIFs8ODBlYjk3ZjA+XSA/IGtlcm5lbF9p
bml0KzB4MC8weDI2ZQ0KWyAgICAzLjMxODgwMF0gIFs8ODA5MDMyNDI+XSBrZXJuZWxfdGhyZWFk
X2hlbHBlcisweDYvMHgxMA0KDQoNCjIuIENsaWVudCAobXkgZW1iZWRkZWQgYm94KSBjb25maWd1
cmF0aW9uDQogIEl0J3Mga2VybmVsIDIuNi4zNSBiYXNlZCwgYW5kIGhhcyBmb2xsb3dpbmcgTkZT
IGtlcm5lbCBjb25maWdzLg0KDQojIGdyZXAgTkZTIC5jb25maWcNCkNPTkZJR19ORlNfRlM9eQ0K
Q09ORklHX05GU19WMz15DQpDT05GSUdfTkZTX1YzX0FDTD15DQpDT05GSUdfTkZTX1Y0PXkNCiMg
Q09ORklHX05GU19WNF8xIGlzIG5vdCBzZXQNCkNPTkZJR19ST09UX05GUz15DQojIENPTkZJR19O
RlNEIGlzIG5vdCBzZXQNCkNPTkZJR19ORlNfQUNMX1NVUFBPUlQ9eQ0KQ09ORklHX05GU19DT01N
T049eQ0KDQoNCjMuIFNlcnZlciAoTkZTRCkgY29uZmlndXJhdGlvbg0KICAgRmVkb3JhIDE5ICsg
bGF0ZXN0IGxpbnVzIGdpdCBrZXJuZWwgMy4xMi4wLXJjMisgKGNvbW1pdCAyMjM1NmY0NCwgbW06
IFBsYWNlIHByZWVtcHRpb24gcG9pbnQgaW4gZG9fbWxvY2thbGwoKSBsb29wKQ0KDQoNCjQuIHdv
cmthcm91bmQNCg0KUmV2ZXJ0aW5nIHRoZSBjb21taXQgNGJkYzMzZWQgcmVzb2x2ZXMgbXkgaXNz
dWUsIE5GUyBib290IGlzIHdvcmtpbmcgdGhlbi4NCkkndmUgZG9uZSBnaXQgYmlzZWN0LCBidXQg
bG9zdCB0aGUgcmVzdWx0aW5nIGJpc2VjdCBsb2cgZHVlIHRvIHN1ZGRlbiBwb3dlciBsb3NzIDoo
Lg0KDQpCZXN0IHJlZ2FyZHMsDQpKb25nbWFuIEhlbw0K
Hi Jongman,
Is the panic on your client or server? I don't see how the patch your
bisect led you to could cause the problem, since all it does is expand
the minor version array on the server. Your client doesn't have NFSD
enabled, so this code shouldn't even be affecting it.
A few questions: what is your /etc/exports on the server? What
version of NFS are you using for nfsroot?
Thanks!
Anna
On Wed, Sep 25, 2013 at 1:19 AM, Jongman Heo <[email protected]> wrote:
>
> Hi all,
>
> My embedded development box fails to NFS-boot with NFS server which uses recent kernel.
>
> Using git bisect, I found it is caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server").
>
>
> 1. dmesg (NFS boot failure case)
>
> ...
> [ 2.040893] ADDRCONF(NETDEV_UP): eth0: link is not ready
> [ 2.046207] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
> [ 2.053570] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 3.055023] IP-Config: Guessing netmask 255.255.0.0
> [ 3.059979] IP-Config: Gateway not on directly connected network.
> [ 3.066330] Looking up port of RPC 100003/2 on 165.213.88.249
> [ 3.074001] Looking up port of RPC 100005/1 on 165.213.88.249
> [ 3.122878] VFS: Unable to mount root fs via NFS, trying floppy.
> [ 3.129134] VFS: Cannot open root device "nfs" or unknown-block(2,0)
> [ 3.135478] Please append a correct "root=" boot option; here are the available partitions:
> [ 3.143831] 1f00 3072 mtdblock0 (driver?)
> [ 3.148798] 1f01 64 mtdblock1 (driver?)
> [ 3.153758] 1f02 64 mtdblock2 (driver?)
> [ 3.158719] 1f03 64 mtdblock3 (driver?)
> [ 3.163682] 1f04 64 mtdblock4 (driver?)
> [ 3.168644] 1f05 64 mtdblock5 (driver?)
> [ 3.173607] 1f06 64 mtdblock6 (driver?)
> [ 3.178568] 0800 488386584 sda driver: sd
> [ 3.183099] 0801 506016 sda1
> [ 3.186927] 0802 4008217 sda2
> [ 3.190755] 0803 483869767 sda3
> [ 3.194584] b300 1880064 mmcblk0 driver: mmcblk
> [ 3.199802] b301 4096 mmcblk0p1
> [ 3.204063] b302 102400 mmcblk0p2
> [ 3.208330] b303 4096 mmcblk0p3
> [ 3.212594] b304 1 mmcblk0p4
> [ 3.216855] b305 2048 mmcblk0p5
> [ 3.221116] b306 2048 mmcblk0p6
> [ 3.225382] b307 2048 mmcblk0p7
> [ 3.229644] b308 4096 mmcblk0p8
> [ 3.233906] b309 12288 mmcblk0p9
> [ 3.238176] b30a 16384 mmcblk0p10
> [ 3.242524] b30b 142336 mmcblk0p11
> [ 3.246869] b30c 1572864 mmcblk0p12
> [ 3.251219] b320 12288 mmcblk0gp1 (driver?)
> [ 3.256272] b310 12288 mmcblk0gp0 (driver?)
> [ 3.261320] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
> [ 3.269566] Pid: 1, comm: swapper Not tainted 2.6.35 #1
> [ 3.274776] Call Trace:
> [ 3.277232] [<80d0db5b>] ? printk+0x1e/0x20
> [ 3.281492] [<80d0dad1>] panic+0x65/0xd1
> [ 3.285495] [<80eb9ce3>] mount_block_root+0x125/0x1be
> [ 3.290631] [<809d1f6d>] ? sys_mknod+0x2d/0x30
> [ 3.295156] [<80eb9f6d>] mount_root+0xd0/0xf2
> [ 3.299591] [<80eba0d9>] prepare_namespace+0x14a/0x184
> [ 3.304803] [<809c44f6>] ? sys_access+0x26/0x30
> [ 3.309411] [<80eb9a4e>] kernel_init+0x25e/0x26e
> [ 3.314105] [<80eb97f0>] ? kernel_init+0x0/0x26e
> [ 3.318800] [<80903242>] kernel_thread_helper+0x6/0x10
>
>
> 2. Client (my embedded box) configuration
> It's kernel 2.6.35 based, and has following NFS kernel configs.
>
> # grep NFS .config
> CONFIG_NFS_FS=y
> CONFIG_NFS_V3=y
> CONFIG_NFS_V3_ACL=y
> CONFIG_NFS_V4=y
> # CONFIG_NFS_V4_1 is not set
> CONFIG_ROOT_NFS=y
> # CONFIG_NFSD is not set
> CONFIG_NFS_ACL_SUPPORT=y
> CONFIG_NFS_COMMON=y
>
>
> 3. Server (NFSD) configuration
> Fedora 19 + latest linus git kernel 3.12.0-rc2+ (commit 22356f44, mm: Place preemption point in do_mlockall() loop)
>
>
> 4. workaround
>
> Reverting the commit 4bdc33ed resolves my issue, NFS boot is working then.
> I've done git bisect, but lost the resulting bisect log due to sudden power loss :(.
>
> Best regards,
> Jongman Heo
On Wed, Sep 25, 2013 at 05:19:50AM +0000, Jongman Heo wrote:
> My embedded development box fails to NFS-boot with NFS server which uses recent kernel.
>
> Using git bisect, I found it is caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server").
>
>
> 1. dmesg (NFS boot failure case)
>
> ...
> [ 2.040893] ADDRCONF(NETDEV_UP): eth0: link is not ready
> [ 2.046207] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
> [ 2.053570] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 3.055023] IP-Config: Guessing netmask 255.255.0.0
> [ 3.059979] IP-Config: Gateway not on directly connected network.
> [ 3.066330] Looking up port of RPC 100003/2 on 165.213.88.249
> [ 3.074001] Looking up port of RPC 100005/1 on 165.213.88.249
> [ 3.122878] VFS: Unable to mount root fs via NFS, trying floppy.
> [ 3.129134] VFS: Cannot open root device "nfs" or unknown-block(2,0)
> [ 3.135478] Please append a correct "root=" boot option; here are the available partitions:
> [ 3.143831] 1f00 3072 mtdblock0 (driver?)
> [ 3.148798] 1f01 64 mtdblock1 (driver?)
> [ 3.153758] 1f02 64 mtdblock2 (driver?)
> [ 3.158719] 1f03 64 mtdblock3 (driver?)
> [ 3.163682] 1f04 64 mtdblock4 (driver?)
> [ 3.168644] 1f05 64 mtdblock5 (driver?)
> [ 3.173607] 1f06 64 mtdblock6 (driver?)
> [ 3.178568] 0800 488386584 sda driver: sd
> [ 3.183099] 0801 506016 sda1
> [ 3.186927] 0802 4008217 sda2
> [ 3.190755] 0803 483869767 sda3
> [ 3.194584] b300 1880064 mmcblk0 driver: mmcblk
> [ 3.199802] b301 4096 mmcblk0p1
> [ 3.204063] b302 102400 mmcblk0p2
> [ 3.208330] b303 4096 mmcblk0p3
> [ 3.212594] b304 1 mmcblk0p4
> [ 3.216855] b305 2048 mmcblk0p5
> [ 3.221116] b306 2048 mmcblk0p6
> [ 3.225382] b307 2048 mmcblk0p7
> [ 3.229644] b308 4096 mmcblk0p8
> [ 3.233906] b309 12288 mmcblk0p9
> [ 3.238176] b30a 16384 mmcblk0p10
> [ 3.242524] b30b 142336 mmcblk0p11
> [ 3.246869] b30c 1572864 mmcblk0p12
> [ 3.251219] b320 12288 mmcblk0gp1 (driver?)
> [ 3.256272] b310 12288 mmcblk0gp0 (driver?)
> [ 3.261320] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
> [ 3.269566] Pid: 1, comm: swapper Not tainted 2.6.35 #1
> [ 3.274776] Call Trace:
> [ 3.277232] [<80d0db5b>] ? printk+0x1e/0x20
> [ 3.281492] [<80d0dad1>] panic+0x65/0xd1
> [ 3.285495] [<80eb9ce3>] mount_block_root+0x125/0x1be
> [ 3.290631] [<809d1f6d>] ? sys_mknod+0x2d/0x30
> [ 3.295156] [<80eb9f6d>] mount_root+0xd0/0xf2
> [ 3.299591] [<80eba0d9>] prepare_namespace+0x14a/0x184
> [ 3.304803] [<809c44f6>] ? sys_access+0x26/0x30
> [ 3.309411] [<80eb9a4e>] kernel_init+0x25e/0x26e
> [ 3.314105] [<80eb97f0>] ? kernel_init+0x0/0x26e
> [ 3.318800] [<80903242>] kernel_thread_helper+0x6/0x10
>
>
> 2. Client (my embedded box) configuration
> It's kernel 2.6.35 based, and has following NFS kernel configs.
>
> # grep NFS .config
> CONFIG_NFS_FS=y
> CONFIG_NFS_V3=y
> CONFIG_NFS_V3_ACL=y
> CONFIG_NFS_V4=y
> # CONFIG_NFS_V4_1 is not set
> CONFIG_ROOT_NFS=y
> # CONFIG_NFSD is not set
> CONFIG_NFS_ACL_SUPPORT=y
> CONFIG_NFS_COMMON=y
>
>
> 3. Server (NFSD) configuration
> Fedora 19 + latest linus git kernel 3.12.0-rc2+ (commit 22356f44, mm: Place preemption point in do_mlockall() loop)
>
>
> 4. workaround
>
> Reverting the commit 4bdc33ed resolves my issue, NFS boot is working then.
> I've done git bisect, but lost the resulting bisect log due to sudden power loss :(.
So when you say you revert that commit, you mean you revert it on your
*server*, right? You're not changing the client at all throughout these
tests?
A network trace might be interesting: so, on the server, run
tcpdump -s0 -wtmp.pcap -ieth0
(replace eth0 by the right network interface), then try booting the
client and after the client fails, kill tcpdump and send us a copy of
tmp.pcap.
(And also you might want to fire up "wireshark tmp.pcap" and take a look
yourself--you'll probably see something like a version mismatch error in
the network traffic.)
--b.
Hi,
>
>------- Original Message -------
>Sender : J. Bruce Fields<[email protected]>
>Date : 2013-09-25 23:05 (GMT+09:00)
>Title : Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
>
>On Wed, Sep 25, 2013 at 05:19:50AM +0000, Jongman Heo wrote:
>> My embedded development box fails to NFS-boot with NFS server which uses recent kernel.
>>
>> Using git bisect, I found it is caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server").
>>
>>
>> 1. dmesg (NFS boot failure case)
>>
>> ...
>> [ 2.040893] ADDRCONF(NETDEV_UP): eth0: link is not ready
>> [ 2.046207] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
>> [ 2.053570] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 3.055023] IP-Config: Guessing netmask 255.255.0.0
>> [ 3.059979] IP-Config: Gateway not on directly connected network.
>> [ 3.066330] Looking up port of RPC 100003/2 on 165.213.88.249
>> [ 3.074001] Looking up port of RPC 100005/1 on 165.213.88.249
>> [ 3.122878] VFS: Unable to mount root fs via NFS, trying floppy.
>> [ 3.129134] VFS: Cannot open root device "nfs" or unknown-block(2,0)
>> [ 3.135478] Please append a correct "root=" boot option; here are the available partitions:
>> [ 3.143831] 1f00 3072 mtdblock0 (driver?)
>> [ 3.148798] 1f01 64 mtdblock1 (driver?)
>> [ 3.153758] 1f02 64 mtdblock2 (driver?)
>> [ 3.158719] 1f03 64 mtdblock3 (driver?)
>> [ 3.163682] 1f04 64 mtdblock4 (driver?)
>> [ 3.168644] 1f05 64 mtdblock5 (driver?)
>> [ 3.173607] 1f06 64 mtdblock6 (driver?)
>> [ 3.178568] 0800 488386584 sda driver: sd
>> [ 3.183099] 0801 506016 sda1
>> [ 3.186927] 0802 4008217 sda2
>> [ 3.190755] 0803 483869767 sda3
>> [ 3.194584] b300 1880064 mmcblk0 driver: mmcblk
>> [ 3.199802] b301 4096 mmcblk0p1
>> [ 3.204063] b302 102400 mmcblk0p2
>> [ 3.208330] b303 4096 mmcblk0p3
>> [ 3.212594] b304 1 mmcblk0p4
>> [ 3.216855] b305 2048 mmcblk0p5
>> [ 3.221116] b306 2048 mmcblk0p6
>> [ 3.225382] b307 2048 mmcblk0p7
>> [ 3.229644] b308 4096 mmcblk0p8
>> [ 3.233906] b309 12288 mmcblk0p9
>> [ 3.238176] b30a 16384 mmcblk0p10
>> [ 3.242524] b30b 142336 mmcblk0p11
>> [ 3.246869] b30c 1572864 mmcblk0p12
>> [ 3.251219] b320 12288 mmcblk0gp1 (driver?)
>> [ 3.256272] b310 12288 mmcblk0gp0 (driver?)
>> [ 3.261320] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
>> [ 3.269566] Pid: 1, comm: swapper Not tainted 2.6.35 #1
>> [ 3.274776] Call Trace:
>> [ 3.277232] [<80d0db5b>] ? printk+0x1e/0x20
>> [ 3.281492] [<80d0dad1>] panic+0x65/0xd1
>> [ 3.285495] [<80eb9ce3>] mount_block_root+0x125/0x1be
>> [ 3.290631] [<809d1f6d>] ? sys_mknod+0x2d/0x30
>> [ 3.295156] [<80eb9f6d>] mount_root+0xd0/0xf2
>> [ 3.299591] [<80eba0d9>] prepare_namespace+0x14a/0x184
>> [ 3.304803] [<809c44f6>] ? sys_access+0x26/0x30
>> [ 3.309411] [<80eb9a4e>] kernel_init+0x25e/0x26e
>> [ 3.314105] [<80eb97f0>] ? kernel_init+0x0/0x26e
>> [ 3.318800] [<80903242>] kernel_thread_helper+0x6/0x10
>>
>>
>> 2. Client (my embedded box) configuration
>> It's kernel 2.6.35 based, and has following NFS kernel configs.
>>
>> # grep NFS .config
>> CONFIG_NFS_FS=y
>> CONFIG_NFS_V3=y
>> CONFIG_NFS_V3_ACL=y
>> CONFIG_NFS_V4=y
>> # CONFIG_NFS_V4_1 is not set
>> CONFIG_ROOT_NFS=y
>> # CONFIG_NFSD is not set
>> CONFIG_NFS_ACL_SUPPORT=y
>> CONFIG_NFS_COMMON=y
>>
>>
>> 3. Server (NFSD) configuration
>> Fedora 19 + latest linus git kernel 3.12.0-rc2+ (commit 22356f44, mm: Place preemption point in do_mlockall() loop)
>>
>>
>> 4. workaround
>>
>> Reverting the commit 4bdc33ed resolves my issue, NFS boot is working then.
>> I've done git bisect, but lost the resulting bisect log due to sudden power loss :(.
>
>So when you say you revert that commit, you mean you revert it on your
>*server*, right? You're not changing the client at all throughout these
>tests?
Right. I reverted the commit on my server, while client is same throughout the tests.
>
>A network trace might be interesting: so, on the server, run
>
>tcpdump -s0 -wtmp.pcap -ieth0
>
>(replace eth0 by the right network interface), then try booting the
>client and after the client fails, kill tcpdump and send us a copy of
>tmp.pcap.
>
>(And also you might want to fire up "wireshark tmp.pcap" and take a look
>yourself--you'll probably see something like a version mismatch error in
>the network traffic.)
>
>--b.
I've attached two tcpdump files.
In the dump, 165.213.88.238 is IP address for NFS client (embedded box with 2.6.35 kernel), and 192.168.64.128 is for NFS server (running latest git kernel with and without the commit revert)
* tmp_good_filtered.pcap
- latest linus git tree + commit 4bdc33ed reverted
- NFS boot is working
* tmp_bad_filtered.pcap
- latest linus git tree
- NFS boot doesn't work
In error case, I can see following message from wireshark packet window ;
Accept State: remote can't support version # (2)
Program Version (Minimum): 3
Program Version (Maximum): 4
And I forgot to attach my config of NFS server. Here it is.
# grep NFS .config
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
# CONFIG_NFS_V4_2 is not set
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
# CONFIG_NFSD_V4_SECURITY_LABEL is not set
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
# rpm -qa|grep nfs
nfs-utils-1.2.8-4.0.fc19.i686
libnfsidmap-0.25-5.fc19.i686
# cat /etc/exports
/home/NFSROOT_mmc/ 165.213.88.238(rw,no_root_squash,sync)
Thanks,
Jongman Heo.
On Thu, Sep 26, 2013 at 04:22:48AM +0000, Jongman Heo wrote:
>
> Hi,
>
> >
> >------- Original Message -------
> >Sender : J. Bruce Fields<[email protected]>
> >Date : 2013-09-25 23:05 (GMT+09:00)
> >Title : Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
> >
> >On Wed, Sep 25, 2013 at 05:19:50AM +0000, Jongman Heo wrote:
> >> My embedded development box fails to NFS-boot with NFS server which uses recent kernel.
> >>
> >> Using git bisect, I found it is caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server").
> >>
> >>
> >> 1. dmesg (NFS boot failure case)
> >>
> >> ...
> >> [ 2.040893] ADDRCONF(NETDEV_UP): eth0: link is not ready
> >> [ 2.046207] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
> >> [ 2.053570] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> >> [ 3.055023] IP-Config: Guessing netmask 255.255.0.0
> >> [ 3.059979] IP-Config: Gateway not on directly connected network.
> >> [ 3.066330] Looking up port of RPC 100003/2 on 165.213.88.249
> >> [ 3.074001] Looking up port of RPC 100005/1 on 165.213.88.249
> >> [ 3.122878] VFS: Unable to mount root fs via NFS, trying floppy.
> >> [ 3.129134] VFS: Cannot open root device "nfs" or unknown-block(2,0)
> >> [ 3.135478] Please append a correct "root=" boot option; here are the available partitions:
> >> [ 3.143831] 1f00 3072 mtdblock0 (driver?)
> >> [ 3.148798] 1f01 64 mtdblock1 (driver?)
> >> [ 3.153758] 1f02 64 mtdblock2 (driver?)
> >> [ 3.158719] 1f03 64 mtdblock3 (driver?)
> >> [ 3.163682] 1f04 64 mtdblock4 (driver?)
> >> [ 3.168644] 1f05 64 mtdblock5 (driver?)
> >> [ 3.173607] 1f06 64 mtdblock6 (driver?)
> >> [ 3.178568] 0800 488386584 sda driver: sd
> >> [ 3.183099] 0801 506016 sda1
> >> [ 3.186927] 0802 4008217 sda2
> >> [ 3.190755] 0803 483869767 sda3
> >> [ 3.194584] b300 1880064 mmcblk0 driver: mmcblk
> >> [ 3.199802] b301 4096 mmcblk0p1
> >> [ 3.204063] b302 102400 mmcblk0p2
> >> [ 3.208330] b303 4096 mmcblk0p3
> >> [ 3.212594] b304 1 mmcblk0p4
> >> [ 3.216855] b305 2048 mmcblk0p5
> >> [ 3.221116] b306 2048 mmcblk0p6
> >> [ 3.225382] b307 2048 mmcblk0p7
> >> [ 3.229644] b308 4096 mmcblk0p8
> >> [ 3.233906] b309 12288 mmcblk0p9
> >> [ 3.238176] b30a 16384 mmcblk0p10
> >> [ 3.242524] b30b 142336 mmcblk0p11
> >> [ 3.246869] b30c 1572864 mmcblk0p12
> >> [ 3.251219] b320 12288 mmcblk0gp1 (driver?)
> >> [ 3.256272] b310 12288 mmcblk0gp0 (driver?)
> >> [ 3.261320] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
> >> [ 3.269566] Pid: 1, comm: swapper Not tainted 2.6.35 #1
> >> [ 3.274776] Call Trace:
> >> [ 3.277232] [<80d0db5b>] ? printk+0x1e/0x20
> >> [ 3.281492] [<80d0dad1>] panic+0x65/0xd1
> >> [ 3.285495] [<80eb9ce3>] mount_block_root+0x125/0x1be
> >> [ 3.290631] [<809d1f6d>] ? sys_mknod+0x2d/0x30
> >> [ 3.295156] [<80eb9f6d>] mount_root+0xd0/0xf2
> >> [ 3.299591] [<80eba0d9>] prepare_namespace+0x14a/0x184
> >> [ 3.304803] [<809c44f6>] ? sys_access+0x26/0x30
> >> [ 3.309411] [<80eb9a4e>] kernel_init+0x25e/0x26e
> >> [ 3.314105] [<80eb97f0>] ? kernel_init+0x0/0x26e
> >> [ 3.318800] [<80903242>] kernel_thread_helper+0x6/0x10
> >>
> >>
> >> 2. Client (my embedded box) configuration
> >> It's kernel 2.6.35 based, and has following NFS kernel configs.
> >>
> >> # grep NFS .config
> >> CONFIG_NFS_FS=y
> >> CONFIG_NFS_V3=y
> >> CONFIG_NFS_V3_ACL=y
> >> CONFIG_NFS_V4=y
> >> # CONFIG_NFS_V4_1 is not set
> >> CONFIG_ROOT_NFS=y
> >> # CONFIG_NFSD is not set
> >> CONFIG_NFS_ACL_SUPPORT=y
> >> CONFIG_NFS_COMMON=y
> >>
> >>
> >> 3. Server (NFSD) configuration
> >> Fedora 19 + latest linus git kernel 3.12.0-rc2+ (commit 22356f44, mm: Place preemption point in do_mlockall() loop)
> >>
> >>
> >> 4. workaround
> >>
> >> Reverting the commit 4bdc33ed resolves my issue, NFS boot is working then.
> >> I've done git bisect, but lost the resulting bisect log due to sudden power loss :(.
> >
> >So when you say you revert that commit, you mean you revert it on your
> >*server*, right? You're not changing the client at all throughout these
> >tests?
>
> Right. I reverted the commit on my server, while client is same throughout the tests.
>
> >
> >A network trace might be interesting: so, on the server, run
> >
> >tcpdump -s0 -wtmp.pcap -ieth0
> >
> >(replace eth0 by the right network interface), then try booting the
> >client and after the client fails, kill tcpdump and send us a copy of
> >tmp.pcap.
> >
> >(And also you might want to fire up "wireshark tmp.pcap" and take a look
> >yourself--you'll probably see something like a version mismatch error in
> >the network traffic.)
> >
> >--b.
>
> I've attached two tcpdump files.
> In the dump, 165.213.88.238 is IP address for NFS client (embedded box with 2.6.35 kernel), and 192.168.64.128 is for NFS server (running latest git kernel with and without the commit revert)
>
> * tmp_good_filtered.pcap
> - latest linus git tree + commit 4bdc33ed reverted
> - NFS boot is working
>
> * tmp_bad_filtered.pcap
> - latest linus git tree
> - NFS boot doesn't work
>
> In error case, I can see following message from wireshark packet window ;
>
> Accept State: remote can't support version # (2)
> Program Version (Minimum): 3
> Program Version (Maximum): 4
This is pretty weird--it's not at all obvious how that patch would
affect this.
You're absolutely positive that the *only* thing you're changing on the
server between the "good" and "bad" cases is that one kernel patch?
You're not changing anything in userspace?
What does "cat /proc/fs/nfsd/versions" report in the good and bad cases?
(BTW, out of curiosity: what kind of client is this that only supports
NFSv2 and NFSv3? Even for an embedded system that's a bit surprising.)
--b.
On Thu, Sep 26, 2013 at 10:47 AM, J. Bruce Fields <[email protected]> wrote:
> (BTW, out of curiosity: what kind of client is this that only supports
> NFSv2 and NFSv3? Even for an embedded system that's a bit surprising.)
>
There are few floating around - at least I'm working on one at this moment.
Look to me the upgrade issue is with user space packages (e.g. where
"mount" lives). Embedded systems can take their user space packages
from certain "distribution"s that are not necessarily sync well with
individual linux upstream tool distribution, even the kernel itself is
reasonably updated.
And don't shoot a messenger :) .. I'm just describing the practices I
have been observed.
-- Wendy
DQpIaSxCcnVjZSwNCg0KPg0KPi0tLS0tLS0gT3JpZ2luYWwgTWVzc2FnZSAtLS0tLS0tDQo+U2Vu
ZGVyIDogSi4gQnJ1Y2UgRmllbGRzPGJmaWVsZHNAZmllbGRzZXMub3JnPg0KPkRhdGUgOiAyMDEz
LTA5LTI3IDAyOjQ3IChHTVQrMDk6MDApDQo+VGl0bGUgOiBSZTogUmU6IFJlZ3Jlc3Npb24gY2F1
c2VkIGJ5IGNvbW1pdCA0YmRjMzNlZCAoIk5GU0R2NC4yOiBBZGQgTkZTIHY0LjIgc3VwcG9ydCB0
byB0aGUgTkZTIHNlcnZlciIpDQo+DQo+T24gVGh1LCBTZXAgMjYsIDIwMTMgYXQgMDQ6MjI6NDhB
TSArMDAwMCwgSm9uZ21hbiBIZW8gd3JvdGU6DQo+PiANCj4+IEhpLA0KPj4gDQo+PiA+DQo+PiA+
LS0tLS0tLSBPcmlnaW5hbCBNZXNzYWdlIC0tLS0tLS0NCj4+ID5TZW5kZXIgOiBKLiBCcnVjZSBG
aWVsZHMNCj4+ID5EYXRlIDogMjAxMy0wOS0yNSAyMzowNSAoR01UKzA5OjAwKQ0KPj4gPlRpdGxl
IDogUmU6IFJlZ3Jlc3Npb24gY2F1c2VkIGJ5IGNvbW1pdCA0YmRjMzNlZCAoIk5GU0R2NC4yOiBB
ZGQgTkZTIHY0LjIgc3VwcG9ydCB0byB0aGUgTkZTIHNlcnZlciIpDQo+PiA+DQo+PiA+T24gV2Vk
LCBTZXAgMjUsIDIwMTMgYXQgMDU6MTk6NTBBTSArMDAwMCwgSm9uZ21hbiBIZW8gd3JvdGU6DQo+
PiA+PiBNeSBlbWJlZGRlZCBkZXZlbG9wbWVudCBib3ggZmFpbHMgdG8gTkZTLWJvb3Qgd2l0aCBO
RlMgc2VydmVyIHdoaWNoIHVzZXMgcmVjZW50IGtlcm5lbC4NCj4+ID4+IA0KPj4gPj4gVXNpbmcg
Z2l0IGJpc2VjdCwgSSBmb3VuZCBpdCBpcyBjYXVzZWQgYnkgY29tbWl0IDRiZGMzM2VkICgiTkZT
RHY0LjI6IEFkZCBORlMgdjQuMiBzdXBwb3J0IHRvIHRoZSBORlMgc2VydmVyIikuDQo+PiA+PiAN
Cj4+ID4+IA0KPj4gPj4gMS4gZG1lc2cgKE5GUyBib290IGZhaWx1cmUgY2FzZSkNCj4+ID4+IA0K
Pj4gPj4gLi4uDQo+PiA+PiBbICAgIDIuMDQwODkzXSBBRERSQ09ORihORVRERVZfVVApOiBldGgw
OiBsaW5rIGlzIG5vdCByZWFkeQ0KPj4gPj4gWyAgICAyLjA0NjIwN10gZTEwMDA6IGV0aDAgTklD
IExpbmsgaXMgVXAgMTAwMCBNYnBzIEZ1bGwgRHVwbGV4LCBGbG93IENvbnRyb2w6IFJYDQo+PiA+
PiBbICAgIDIuMDUzNTcwXSBBRERSQ09ORihORVRERVZfQ0hBTkdFKTogZXRoMDogbGluayBiZWNv
bWVzIHJlYWR5DQo+PiA+PiBbICAgIDMuMDU1MDIzXSBJUC1Db25maWc6IEd1ZXNzaW5nIG5ldG1h
c2sgMjU1LjI1NS4wLjANCj4+ID4+IFsgICAgMy4wNTk5NzldIElQLUNvbmZpZzogR2F0ZXdheSBu
b3Qgb24gZGlyZWN0bHkgY29ubmVjdGVkIG5ldHdvcmsuDQo+PiA+PiBbICAgIDMuMDY2MzMwXSBM
b29raW5nIHVwIHBvcnQgb2YgUlBDIDEwMDAwMy8yIG9uIDE2NS4yMTMuODguMjQ5DQo+PiA+PiBb
ICAgIDMuMDc0MDAxXSBMb29raW5nIHVwIHBvcnQgb2YgUlBDIDEwMDAwNS8xIG9uIDE2NS4yMTMu
ODguMjQ5DQo+PiA+PiBbICAgIDMuMTIyODc4XSBWRlM6IFVuYWJsZSB0byBtb3VudCByb290IGZz
IHZpYSBORlMsIHRyeWluZyBmbG9wcHkuDQo+PiA+PiBbICAgIDMuMTI5MTM0XSBWRlM6IENhbm5v
dCBvcGVuIHJvb3QgZGV2aWNlICJuZnMiIG9yIHVua25vd24tYmxvY2soMiwwKQ0KPj4gPj4gWyAg
ICAzLjEzNTQ3OF0gUGxlYXNlIGFwcGVuZCBhIGNvcnJlY3QgInJvb3Q9IiBib290IG9wdGlvbjsg
aGVyZSBhcmUgdGhlIGF2YWlsYWJsZSBwYXJ0aXRpb25zOg0KPj4gPj4gWyAgICAzLjE0MzgzMV0g
MWYwMCAgICAgICAgICAgIDMwNzIgbXRkYmxvY2swIChkcml2ZXI/KQ0KPj4gPj4gWyAgICAzLjE0
ODc5OF0gMWYwMSAgICAgICAgICAgICAgNjQgbXRkYmxvY2sxIChkcml2ZXI/KQ0KPj4gPj4gWyAg
ICAzLjE1Mzc1OF0gMWYwMiAgICAgICAgICAgICAgNjQgbXRkYmxvY2syIChkcml2ZXI/KQ0KPj4g
Pj4gWyAgICAzLjE1ODcxOV0gMWYwMyAgICAgICAgICAgICAgNjQgbXRkYmxvY2szIChkcml2ZXI/
KQ0KPj4gPj4gWyAgICAzLjE2MzY4Ml0gMWYwNCAgICAgICAgICAgICAgNjQgbXRkYmxvY2s0IChk
cml2ZXI/KQ0KPj4gPj4gWyAgICAzLjE2ODY0NF0gMWYwNSAgICAgICAgICAgICAgNjQgbXRkYmxv
Y2s1IChkcml2ZXI/KQ0KPj4gPj4gWyAgICAzLjE3MzYwN10gMWYwNiAgICAgICAgICAgICAgNjQg
bXRkYmxvY2s2IChkcml2ZXI/KQ0KPj4gPj4gWyAgICAzLjE3ODU2OF0gMDgwMCAgICAgICA0ODgz
ODY1ODQgc2RhIGRyaXZlcjogc2QNCj4+ID4+IFsgICAgMy4xODMwOTldICAgMDgwMSAgICAgICAg
ICA1MDYwMTYgc2RhMQ0KPj4gPj4gWyAgICAzLjE4NjkyN10gICAwODAyICAgICAgICAgNDAwODIx
NyBzZGEyDQo+PiA+PiBbICAgIDMuMTkwNzU1XSAgIDA4MDMgICAgICAgNDgzODY5NzY3IHNkYTMN
Cj4+ID4+IFsgICAgMy4xOTQ1ODRdIGIzMDAgICAgICAgICAxODgwMDY0IG1tY2JsazAgZHJpdmVy
OiBtbWNibGsNCj4+ID4+IFsgICAgMy4xOTk4MDJdICAgYjMwMSAgICAgICAgICAgIDQwOTYgbW1j
YmxrMHAxDQo+PiA+PiBbICAgIDMuMjA0MDYzXSAgIGIzMDIgICAgICAgICAgMTAyNDAwIG1tY2Js
azBwMg0KPj4gPj4gWyAgICAzLjIwODMzMF0gICBiMzAzICAgICAgICAgICAgNDA5NiBtbWNibGsw
cDMNCj4+ID4+IFsgICAgMy4yMTI1OTRdICAgYjMwNCAgICAgICAgICAgICAgIDEgbW1jYmxrMHA0
DQo+PiA+PiBbICAgIDMuMjE2ODU1XSAgIGIzMDUgICAgICAgICAgICAyMDQ4IG1tY2JsazBwNQ0K
Pj4gPj4gWyAgICAzLjIyMTExNl0gICBiMzA2ICAgICAgICAgICAgMjA0OCBtbWNibGswcDYNCj4+
ID4+IFsgICAgMy4yMjUzODJdICAgYjMwNyAgICAgICAgICAgIDIwNDggbW1jYmxrMHA3DQo+PiA+
PiBbICAgIDMuMjI5NjQ0XSAgIGIzMDggICAgICAgICAgICA0MDk2IG1tY2JsazBwOA0KPj4gPj4g
WyAgICAzLjIzMzkwNl0gICBiMzA5ICAgICAgICAgICAxMjI4OCBtbWNibGswcDkNCj4+ID4+IFsg
ICAgMy4yMzgxNzZdICAgYjMwYSAgICAgICAgICAgMTYzODQgbW1jYmxrMHAxMA0KPj4gPj4gWyAg
ICAzLjI0MjUyNF0gICBiMzBiICAgICAgICAgIDE0MjMzNiBtbWNibGswcDExDQo+PiA+PiBbICAg
IDMuMjQ2ODY5XSAgIGIzMGMgICAgICAgICAxNTcyODY0IG1tY2JsazBwMTINCj4+ID4+IFsgICAg
My4yNTEyMTldIGIzMjAgICAgICAgICAgIDEyMjg4IG1tY2JsazBncDEgKGRyaXZlcj8pDQo+PiA+
PiBbICAgIDMuMjU2MjcyXSBiMzEwICAgICAgICAgICAxMjI4OCBtbWNibGswZ3AwIChkcml2ZXI/
KQ0KPj4gPj4gWyAgICAzLjI2MTMyMF0gS2VybmVsIHBhbmljIC0gbm90IHN5bmNpbmc6IFZGUzog
VW5hYmxlIHRvIG1vdW50IHJvb3QgZnMgb24gdW5rbm93bi1ibG9jaygyLDApDQo+PiA+PiBbICAg
IDMuMjY5NTY2XSBQaWQ6IDEsIGNvbW06IHN3YXBwZXIgTm90IHRhaW50ZWQgMi42LjM1ICMxDQo+
PiA+PiBbICAgIDMuMjc0Nzc2XSBDYWxsIFRyYWNlOg0KPj4gPj4gWyAgICAzLjI3NzIzMl0gIFs8
ODBkMGRiNWI+XSA/IHByaW50aysweDFlLzB4MjANCj4+ID4+IFsgICAgMy4yODE0OTJdICBbPDgw
ZDBkYWQxPl0gcGFuaWMrMHg2NS8weGQxDQo+PiA+PiBbICAgIDMuMjg1NDk1XSAgWzw4MGViOWNl
Mz5dIG1vdW50X2Jsb2NrX3Jvb3QrMHgxMjUvMHgxYmUNCj4+ID4+IFsgICAgMy4yOTA2MzFdICBb
PDgwOWQxZjZkPl0gPyBzeXNfbWtub2QrMHgyZC8weDMwDQo+PiA+PiBbICAgIDMuMjk1MTU2XSAg
Wzw4MGViOWY2ZD5dIG1vdW50X3Jvb3QrMHhkMC8weGYyDQo+PiA+PiBbICAgIDMuMjk5NTkxXSAg
Wzw4MGViYTBkOT5dIHByZXBhcmVfbmFtZXNwYWNlKzB4MTRhLzB4MTg0DQo+PiA+PiBbICAgIDMu
MzA0ODAzXSAgWzw4MDljNDRmNj5dID8gc3lzX2FjY2VzcysweDI2LzB4MzANCj4+ID4+IFsgICAg
My4zMDk0MTFdICBbPDgwZWI5YTRlPl0ga2VybmVsX2luaXQrMHgyNWUvMHgyNmUNCj4+ID4+IFsg
ICAgMy4zMTQxMDVdICBbPDgwZWI5N2YwPl0gPyBrZXJuZWxfaW5pdCsweDAvMHgyNmUNCj4+ID4+
IFsgICAgMy4zMTg4MDBdICBbPDgwOTAzMjQyPl0ga2VybmVsX3RocmVhZF9oZWxwZXIrMHg2LzB4
MTANCj4+ID4+IA0KPj4gPj4gDQo+PiA+PiAyLiBDbGllbnQgKG15IGVtYmVkZGVkIGJveCkgY29u
ZmlndXJhdGlvbg0KPj4gPj4gICBJdCdzIGtlcm5lbCAyLjYuMzUgYmFzZWQsIGFuZCBoYXMgZm9s
bG93aW5nIE5GUyBrZXJuZWwgY29uZmlncy4NCj4+ID4+IA0KPj4gPj4gIyBncmVwIE5GUyAuY29u
ZmlnDQo+PiA+PiBDT05GSUdfTkZTX0ZTPXkNCj4+ID4+IENPTkZJR19ORlNfVjM9eQ0KPj4gPj4g
Q09ORklHX05GU19WM19BQ0w9eQ0KPj4gPj4gQ09ORklHX05GU19WND15DQo+PiA+PiAjIENPTkZJ
R19ORlNfVjRfMSBpcyBub3Qgc2V0DQo+PiA+PiBDT05GSUdfUk9PVF9ORlM9eQ0KPj4gPj4gIyBD
T05GSUdfTkZTRCBpcyBub3Qgc2V0DQo+PiA+PiBDT05GSUdfTkZTX0FDTF9TVVBQT1JUPXkNCj4+
ID4+IENPTkZJR19ORlNfQ09NTU9OPXkNCj4+ID4+IA0KPj4gPj4gDQo+PiA+PiAzLiBTZXJ2ZXIg
KE5GU0QpIGNvbmZpZ3VyYXRpb24NCj4+ID4+ICAgIEZlZG9yYSAxOSArIGxhdGVzdCBsaW51cyBn
aXQga2VybmVsIDMuMTIuMC1yYzIrIChjb21taXQgMjIzNTZmNDQsIG1tOiBQbGFjZSBwcmVlbXB0
aW9uIHBvaW50IGluIGRvX21sb2NrYWxsKCkgbG9vcCkNCj4+ID4+IA0KPj4gPj4gDQo+PiA+PiA0
LiB3b3JrYXJvdW5kDQo+PiA+PiANCj4+ID4+IFJldmVydGluZyB0aGUgY29tbWl0IDRiZGMzM2Vk
IHJlc29sdmVzIG15IGlzc3VlLCBORlMgYm9vdCBpcyB3b3JraW5nIHRoZW4uDQo+PiA+PiBJJ3Zl
IGRvbmUgZ2l0IGJpc2VjdCwgYnV0IGxvc3QgdGhlIHJlc3VsdGluZyBiaXNlY3QgbG9nIGR1ZSB0
byBzdWRkZW4gcG93ZXIgbG9zcyA6KC4NCj4+ID4NCj4+ID5TbyB3aGVuIHlvdSBzYXkgeW91IHJl
dmVydCB0aGF0IGNvbW1pdCwgeW91IG1lYW4geW91IHJldmVydCBpdCBvbiB5b3VyDQo+PiA+KnNl
cnZlciosIHJpZ2h0PyAgWW91J3JlIG5vdCBjaGFuZ2luZyB0aGUgY2xpZW50IGF0IGFsbCB0aHJv
dWdob3V0IHRoZXNlDQo+PiA+dGVzdHM/DQo+PiANCj4+IFJpZ2h0LiBJIHJldmVydGVkIHRoZSBj
b21taXQgb24gbXkgc2VydmVyLCB3aGlsZSBjbGllbnQgaXMgc2FtZSB0aHJvdWdob3V0IHRoZSB0
ZXN0cy4NCj4+IA0KPj4gPg0KPj4gPkEgbmV0d29yayB0cmFjZSBtaWdodCBiZSBpbnRlcmVzdGlu
Zzogc28sIG9uIHRoZSBzZXJ2ZXIsIHJ1bg0KPj4gPg0KPj4gPnRjcGR1bXAgLXMwIC13dG1wLnBj
YXAgLWlldGgwDQo+PiA+DQo+PiA+KHJlcGxhY2UgZXRoMCBieSB0aGUgcmlnaHQgbmV0d29yayBp
bnRlcmZhY2UpLCB0aGVuIHRyeSBib290aW5nIHRoZQ0KPj4gPmNsaWVudCBhbmQgYWZ0ZXIgdGhl
IGNsaWVudCBmYWlscywga2lsbCB0Y3BkdW1wIGFuZCBzZW5kIHVzIGEgY29weSBvZg0KPj4gPnRt
cC5wY2FwLg0KPj4gPg0KPj4gPihBbmQgYWxzbyB5b3UgbWlnaHQgd2FudCB0byBmaXJlIHVwICJ3
aXJlc2hhcmsgdG1wLnBjYXAiIGFuZCB0YWtlIGEgbG9vaw0KPj4gPnlvdXJzZWxmLS15b3UnbGwg
cHJvYmFibHkgc2VlIHNvbWV0aGluZyBsaWtlIGEgdmVyc2lvbiBtaXNtYXRjaCBlcnJvciBpbg0K
Pj4gPnRoZSBuZXR3b3JrIHRyYWZmaWMuKQ0KPj4gPg0KPj4gPi0tYi4NCj4+IA0KPj4gSSd2ZSBh
dHRhY2hlZCB0d28gdGNwZHVtcCBmaWxlcy4gDQo+PiBJbiB0aGUgZHVtcCwgMTY1LjIxMy44OC4y
MzggaXMgSVAgYWRkcmVzcyBmb3IgTkZTIGNsaWVudCAoZW1iZWRkZWQgYm94IHdpdGggMi42LjM1
IGtlcm5lbCksIGFuZCAxOTIuMTY4LjY0LjEyOCBpcyBmb3IgTkZTIHNlcnZlciAocnVubmluZyBs
YXRlc3QgZ2l0IGtlcm5lbCB3aXRoIGFuZCB3aXRob3V0IHRoZSBjb21taXQgcmV2ZXJ0KQ0KPj4g
DQo+PiAgKiB0bXBfZ29vZF9maWx0ZXJlZC5wY2FwDQo+PiAgICAtIGxhdGVzdCBsaW51cyBnaXQg
dHJlZSArIGNvbW1pdCA0YmRjMzNlZCByZXZlcnRlZA0KPj4gICAgLSBORlMgYm9vdCBpcyB3b3Jr
aW5nDQo+PiANCj4+ICAqIHRtcF9iYWRfZmlsdGVyZWQucGNhcA0KPj4gICAgLSBsYXRlc3QgbGlu
dXMgZ2l0IHRyZWUNCj4+ICAgIC0gTkZTIGJvb3QgZG9lc24ndCB3b3JrDQo+PiAgDQo+PiBJbiBl
cnJvciBjYXNlLCBJIGNhbiBzZWUgZm9sbG93aW5nIG1lc3NhZ2UgZnJvbSB3aXJlc2hhcmsgcGFj
a2V0IHdpbmRvdyA7DQo+PiANCj4+ICAgICBBY2NlcHQgU3RhdGU6IHJlbW90ZSBjYW4ndCBzdXBw
b3J0IHZlcnNpb24gIyAoMikNCj4+ICAgICBQcm9ncmFtIFZlcnNpb24gKE1pbmltdW0pOiAzDQo+
PiAgICAgUHJvZ3JhbSBWZXJzaW9uIChNYXhpbXVtKTogNA0KPg0KPlRoaXMgaXMgcHJldHR5IHdl
aXJkLS1pdCdzIG5vdCBhdCBhbGwgb2J2aW91cyBob3cgdGhhdCBwYXRjaCB3b3VsZA0KPmFmZmVj
dCB0aGlzLg0KPg0KPllvdSdyZSBhYnNvbHV0ZWx5IHBvc2l0aXZlIHRoYXQgdGhlICpvbmx5KiB0
aGluZyB5b3UncmUgY2hhbmdpbmcgb24gdGhlDQo+c2VydmVyIGJldHdlZW4gdGhlICJnb29kIiBh
bmQgImJhZCIgY2FzZXMgaXMgdGhhdCBvbmUga2VybmVsIHBhdGNoPw0KPllvdSdyZSBub3QgY2hh
bmdpbmcgYW55dGhpbmcgaW4gdXNlcnNwYWNlPw0KPg0KDQpZZXMsIHByZXR0eSBzdXJlLg0KDQo+
V2hhdCBkb2VzICJjYXQgL3Byb2MvZnMvbmZzZC92ZXJzaW9ucyIgcmVwb3J0IGluIHRoZSBnb29k
IGFuZCBiYWQgY2FzZXM/DQo+DQo+KEJUVywgb3V0IG9mIGN1cmlvc2l0eTogd2hhdCBraW5kIG9m
IGNsaWVudCBpcyB0aGlzIHRoYXQgb25seSBzdXBwb3J0cw0KPk5GU3YyIGFuZCBORlN2Mz8gIEV2
ZW4gZm9yIGFuIGVtYmVkZGVkIHN5c3RlbSB0aGF0J3MgYSBiaXQgc3VycHJpc2luZy4pDQo+DQo+
LS1iLg0KPg0KDQpIZXJlIGFyZSAvcHJvYy9mcy9uZnNkL3ZlcnNpb25zIGluZm9ybWF0aW9uIGZv
ciBnb29kIGFuZCBiYWQgY2FzZXMgOw0KDQpnb29kIChjb21taXQgNGJkYzMzZWQgcmV2ZXJ0ZWQp
DQoNCiMgY2F0IC9wcm9jL2ZzL25mc2QvdmVyc2lvbnMgDQorMiArMyArNCArNC4xDQoNCg0KYmFk
IChjdXJyZW50IGxpbnVzIGdpdCkNCg0KIyBjYXQgL3Byb2MvZnMvbmZzZC92ZXJzaW9ucyAgDQot
MiArMyArNCArNC4xIC00LjINCg0KDQpJIGRvbid0IGtub3cgd2h5IHRoZSBjb21taXQgNGJkYzMz
ZWQgbWFrZXMgdGhpcyBkaWZmZXJlbmNlICggZnJvbSArMiB0byAtMiApLg0KDQpNeSBORlMgc2Vy
dmVyIGp1c3QgdXNlcyBGZWRvcmEgMTkgKyBsYXRlc3Qga2VybmVsICh3aGljaCBpcyBub3QgYSBy
YXJlIHNldHVwLi4uKSwgDQpzbyBJIHRoaW5rIHNvbWUgcGVvcGxlIGNhbiB2ZXJpZnkgaWYgdGhp
cyB2ZXJzaW9uIGluZm9ybWF0aW9uIGNoYW5nZSBoYXBwZW5zIHcvIGFuZCB3L28gdGhlIGNvbW1p
dCByZXZlcnQuDQoNCkRvbid0IGtub3cgdGhlIGRldGFpbCBvZiBORlMgcHJvdG9jb2wsIGJ1dCBv
dXIgTkZTIGNsaWVudCBzZWVtcyBub3QgdG8gdHJ5IHdpdGggdjMgYW5kIHY0IGluIGNhc2UgdjIg
ZmFpbHMuLi4NCklzIHRoaXMgYW4gdW5leHBlY3RlZCAoYnVnZ3kpIGJlaGF2aW9yIG9mIG15IG9s
ZCBlbWJlZGRlZCBib3ggKE5GUyBjbGllbnQgb2Yga2VybmVsIDIuNi4zNSksIG9yIGV4cGVjdGVk
IG9uZSBmcm9tIHRoZSBORlMgcHJvdG9jb2w/DQoNClRoYW5rcywNCkpvbmdtYW4gSGVvLg==
Hi,
>
>------- Original Message -------
>Sender : J. Bruce Fields<[email protected]>
>Date : 2013-09-25 23:05 (GMT+09:00)
>Title : Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
>
>On Wed, Sep 25, 2013 at 05:19:50AM +0000, Jongman Heo wrote:
>> My embedded development box fails to NFS-boot with NFS server which uses recent kernel.
>>
>> Using git bisect, I found it is caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server").
>>
>>
>> 1. dmesg (NFS boot failure case)
>>
>> ...
>> [ 2.040893] ADDRCONF(NETDEV_UP): eth0: link is not ready
>> [ 2.046207] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
>> [ 2.053570] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 3.055023] IP-Config: Guessing netmask 255.255.0.0
>> [ 3.059979] IP-Config: Gateway not on directly connected network.
>> [ 3.066330] Looking up port of RPC 100003/2 on 165.213.88.249
>> [ 3.074001] Looking up port of RPC 100005/1 on 165.213.88.249
>> [ 3.122878] VFS: Unable to mount root fs via NFS, trying floppy.
>> [ 3.129134] VFS: Cannot open root device "nfs" or unknown-block(2,0)
>> [ 3.135478] Please append a correct "root=" boot option; here are the available partitions:
>> [ 3.143831] 1f00 3072 mtdblock0 (driver?)
>> [ 3.148798] 1f01 64 mtdblock1 (driver?)
>> [ 3.153758] 1f02 64 mtdblock2 (driver?)
>> [ 3.158719] 1f03 64 mtdblock3 (driver?)
>> [ 3.163682] 1f04 64 mtdblock4 (driver?)
>> [ 3.168644] 1f05 64 mtdblock5 (driver?)
>> [ 3.173607] 1f06 64 mtdblock6 (driver?)
>> [ 3.178568] 0800 488386584 sda driver: sd
>> [ 3.183099] 0801 506016 sda1
>> [ 3.186927] 0802 4008217 sda2
>> [ 3.190755] 0803 483869767 sda3
>> [ 3.194584] b300 1880064 mmcblk0 driver: mmcblk
>> [ 3.199802] b301 4096 mmcblk0p1
>> [ 3.204063] b302 102400 mmcblk0p2
>> [ 3.208330] b303 4096 mmcblk0p3
>> [ 3.212594] b304 1 mmcblk0p4
>> [ 3.216855] b305 2048 mmcblk0p5
>> [ 3.221116] b306 2048 mmcblk0p6
>> [ 3.225382] b307 2048 mmcblk0p7
>> [ 3.229644] b308 4096 mmcblk0p8
>> [ 3.233906] b309 12288 mmcblk0p9
>> [ 3.238176] b30a 16384 mmcblk0p10
>> [ 3.242524] b30b 142336 mmcblk0p11
>> [ 3.246869] b30c 1572864 mmcblk0p12
>> [ 3.251219] b320 12288 mmcblk0gp1 (driver?)
>> [ 3.256272] b310 12288 mmcblk0gp0 (driver?)
>> [ 3.261320] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
>> [ 3.269566] Pid: 1, comm: swapper Not tainted 2.6.35 #1
>> [ 3.274776] Call Trace:
>> [ 3.277232] [<80d0db5b>] ? printk+0x1e/0x20
>> [ 3.281492] [<80d0dad1>] panic+0x65/0xd1
>> [ 3.285495] [<80eb9ce3>] mount_block_root+0x125/0x1be
>> [ 3.290631] [<809d1f6d>] ? sys_mknod+0x2d/0x30
>> [ 3.295156] [<80eb9f6d>] mount_root+0xd0/0xf2
>> [ 3.299591] [<80eba0d9>] prepare_namespace+0x14a/0x184
>> [ 3.304803] [<809c44f6>] ? sys_access+0x26/0x30
>> [ 3.309411] [<80eb9a4e>] kernel_init+0x25e/0x26e
>> [ 3.314105] [<80eb97f0>] ? kernel_init+0x0/0x26e
>> [ 3.318800] [<80903242>] kernel_thread_helper+0x6/0x10
>>
>>
>> 2. Client (my embedded box) configuration
>> It's kernel 2.6.35 based, and has following NFS kernel configs.
>>
>> # grep NFS .config
>> CONFIG_NFS_FS=y
>> CONFIG_NFS_V3=y
>> CONFIG_NFS_V3_ACL=y
>> CONFIG_NFS_V4=y
>> # CONFIG_NFS_V4_1 is not set
>> CONFIG_ROOT_NFS=y
>> # CONFIG_NFSD is not set
>> CONFIG_NFS_ACL_SUPPORT=y
>> CONFIG_NFS_COMMON=y
>>
>>
>> 3. Server (NFSD) configuration
>> Fedora 19 + latest linus git kernel 3.12.0-rc2+ (commit 22356f44, mm: Place preemption point in do_mlockall() loop)
>>
>>
>> 4. workaround
>>
>> Reverting the commit 4bdc33ed resolves my issue, NFS boot is working then.
>> I've done git bisect, but lost the resulting bisect log due to sudden power loss :(.
>
>So when you say you revert that commit, you mean you revert it on your
>*server*, right? You're not changing the client at all throughout these
>tests?
Right. I reverted the commit on my server, while client is same throughout the tests.
>
>A network trace might be interesting: so, on the server, run
>
>tcpdump -s0 -wtmp.pcap -ieth0
>
>(replace eth0 by the right network interface), then try booting the
>client and after the client fails, kill tcpdump and send us a copy of
>tmp.pcap.
>
>(And also you might want to fire up "wireshark tmp.pcap" and take a look
>yourself--you'll probably see something like a version mismatch error in
>the network traffic.)
>
>--b.
I've attached two tcpdump files.
In the dump, 165.213.88.238 is IP address for NFS client (embedded box with 2.6.35 kernel), and 192.168.64.128 is for NFS server (running latest git kernel with and without the commit revert)
* tmp_good_filtered.pcap
- latest linus git tree + commit 4bdc33ed reverted
- NFS boot is working
* tmp_bad_filtered.pcap
- latest linus git tree
- NFS boot doesn't work
In error case, I can see following message from wireshark packet window ;
Accept State: remote can't support version # (2)
Program Version (Minimum): 3
Program Version (Maximum): 4
And I forgot to attach my config of NFS server. Here it is.
# grep NFS .config
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
# CONFIG_NFS_V4_2 is not set
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
# CONFIG_NFSD_V4_SECURITY_LABEL is not set
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
# rpm -qa|grep nfs
nfs-utils-1.2.8-4.0.fc19.i686
libnfsidmap-0.25-5.fc19.i686
# cat /etc/exports
/home/NFSROOT_mmc/ 165.213.88.238(rw,no_root_squash,sync)
Thanks,
Jongman Heo.
On Thu, Sep 26, 2013 at 11:35:46AM -0700, Wendy Cheng wrote:
> On Thu, Sep 26, 2013 at 10:47 AM, J. Bruce Fields <[email protected]> wrote:
>
> > (BTW, out of curiosity: what kind of client is this that only supports
> > NFSv2 and NFSv3? Even for an embedded system that's a bit surprising.)
> >
>
> There are few floating around - at least I'm working on one at this moment.
>
> Look to me the upgrade issue is with user space packages (e.g. where
> "mount" lives). Embedded systems can take their user space packages
> from certain "distribution"s that are not necessarily sync well with
> individual linux upstream tool distribution, even the kernel itself is
> reasonably updated.
>
> And don't shoot a messenger :) .. I'm just describing the practices I
> have been observed.
OK. I'm not sure what standard to hold this sort of change to. I'm
inclined to use the same "it's OK if and only if nobody notices"
standard as for normal kernel interfaces. Which means we've already got
enough evidence to put off dropping NFSv2 support for a few more years,
at least on the server side....
--b.