The information included below is per the URL:
http://www.kernel.org/pub/linux/docs/lkml/reporting-bugs.html
If different or additional information is needed, please let me know.
Kernel version:
Linux martin 3.5.2-gentoo #1 SMP Fri Aug 24 09:35:54 EDT 2012 x86_64 Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz GenuineIntel GNU/Linux
Linux version 3.5.2-gentoo (root@martin) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.2) ) #1 SMP Fri Aug 24 09:35:54 EDT 2012
Problem also occurred in 3.5.1-gentoo.
Modules:
Module Size Used by
vboxnetadp 17542 0
vboxnetflt 14893 0
vboxdrv 1775582 2 vboxnetadp,vboxnetflt
Bug trigger:
Solaris 11/11 (patched as of July, 2012) is serving NFS V4 to a Gentoo Linux system. IDMAP is being used to manage users/groups. "/etc/idmap.conf" is fairly generic except for the domain name.
The Gentoo "portage" tree is shared via NFS. I initiate an 'emerge --sync' and the condition occurs. NFS no longer works. The system will not restart or shut down due to the hang in NFS. The system has to be hard powered off to clear the condition. The bug does not trigger reliably. The Solaris 11/11 NFS server & Gentoo Linux NFS client have been configured this way for more than a year.
Kernel log:
[160372.303310] ------------[ cut here ]------------
[160372.303353] kernel BUG at fs/nfs/idmap.c:684!
[160372.303392] invalid opcode: 0000 [#1] SMP
[160372.303431] CPU 2
[160372.303437] Modules linked in: vboxnetadp(O) vboxnetflt(O) vboxdrv(O)
[160372.303541]
[160372.303572] Pid: 12852, comm: mount.nfs Tainted: G O 3.5.2-gentoo #1 Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3
[160372.303655] RIP: 0010:[<ffffffff81187e03>] [<ffffffff81187e03>] nfs_idmap_legacy_upcall+0x323/0x330
[160372.303736] RSP: 0018:ffff8802421af5f8 EFLAGS: 00010282
[160372.303777] RAX: 0000000000000015 RBX: ffff8802ed9e5000 RCX: 0000000000000000
[160372.303847] RDX: 0000000000000005 RSI: ffff88026af757b9 RDI: ffff8802ed9e5017
[160372.303917] RBP: ffff88040d524740 R08: 000000000000003a R09: ffffffff81187b35
[160372.303987] R10: 00000000139c5e77 R11: 000000006138df6f R12: ffff88040b568410
[160372.304056] R13: ffff88040c3e5700 R14: 0000000000000015 R15: 0000000000000000
[160372.304128] FS: 00007f35102ca700(0000) GS:ffff88041fb00000(0000) knlGS:0000000000000000
[160372.304200] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[160372.304240] CR2: 00007f37a2b09880 CR3: 0000000236eae000 CR4: 00000000000407e0
[160372.304311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[160372.304382] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[160372.304453] Process mount.nfs (pid: 12852, threadinfo ffff8802421ae000, task ffff8803e789be90)
[160372.304525] Stack:
[160372.304558] ffff88026af757a4 ffff88026af757b9 ffff88040b568410 ffff8802ed9e59c0
[160372.304632] ffff8802ed9e5540 ffffffff81526c0e 0000000000000000 ffffffff811ad2de
[160372.304707] 0000000000000000 ffff88040d53ae40 ffff88040c3e5700 ffff88040d53ae40
[160372.304781] Call Trace:
[160372.304819] [<ffffffff811ad2de>] ? request_key_and_link+0x31e/0x450
[160372.304864] [<ffffffff811ad465>] ? request_key_with_auxdata+0x15/0x60
[160372.304908] [<ffffffff81187eeb>] ? nfs_idmap_request_key+0xdb/0x1c0
[160372.304951] [<ffffffff811881a0>] ? nfs_idmap_lookup_id+0xe0/0x100
[160372.304995] [<ffffffff811883a9>] ? nfs_map_string_to_numeric+0x29/0x90
[160372.305039] [<ffffffff81182fcc>] ? decode_getfattr_attrs+0xb2c/0xb50
[160372.305082] [<ffffffff8118307d>] ? decode_getfattr_generic.constprop.96+0x8d/0xe0
[160372.305155] [<ffffffff811834b0>] ? nfs4_xdr_dec_link+0xf0/0xf0
[160372.305197] [<ffffffff81183529>] ? nfs4_xdr_dec_lookup_root+0x79/0x80
[160372.305240] [<ffffffff811834b0>] ? nfs4_xdr_dec_link+0xf0/0xf0
[160372.305285] [<ffffffff81414d68>] ? rpcauth_unwrap_resp+0x58/0x60
[160372.305328] [<ffffffff8140bed3>] ? call_decode+0x333/0x430
[160372.305370] [<ffffffff81413776>] ? __rpc_execute+0x46/0x1e0
[160372.305412] [<ffffffff810571f8>] ? wake_up_bit+0x18/0x40
[160372.305454] [<ffffffff8140d529>] ? rpc_run_task+0x69/0x90
[160372.305496] [<ffffffff8140d65f>] ? rpc_call_sync+0x3f/0x70
[160372.306028] [<ffffffff81174e5e>] ? _nfs4_lookup_root.isra.42+0xae/0xd0
[160372.306072] [<ffffffff81174ec7>] ? nfs4_lookup_root+0x47/0x80
[160372.306115] [<ffffffff81179c10>] ? nfs4_proc_get_rootfh+0x30/0xd0
[160372.306158] [<ffffffff8115e148>] ? nfs4_get_rootfh+0x28/0xc0
[160372.306200] [<ffffffff8140cd21>] ? rpc_register_client+0x41/0x70
[160372.306243] [<ffffffff81413fdb>] ? rpciod_up+0xb/0x20
[160372.306283] [<ffffffff8140ce97>] ? rpc_clone_client+0x147/0x1c0
[160372.306326] [<ffffffff81158290>] ? nfs4_server_common_setup+0x90/0x170
[160372.306369] [<ffffffff811594fe>] ? nfs4_create_server+0x14e/0x2b0
[160372.306413] [<ffffffff811625df>] ? nfs4_remote_mount+0x3f/0x90
[160372.306457] [<ffffffff810d17da>] ? mount_fs+0x1a/0xd0
[160372.306500] [<ffffffff810e92a3>] ? vfs_kern_mount+0x73/0x120
[160372.306542] [<ffffffff81162720>] ? nfs_do_root_mount+0x90/0xe0
[160372.306584] [<ffffffff81163bdf>] ? nfs_fs_mount+0x75f/0x970
[160372.306626] [<ffffffff81163240>] ? nfs_fill_super+0x1d0/0x1d0
[160372.306669] [<ffffffff81160720>] ? nfs_destroy_inode+0x20/0x20
[160372.306712] [<ffffffff810d17da>] ? mount_fs+0x1a/0xd0
[160372.306753] [<ffffffff810e92a3>] ? vfs_kern_mount+0x73/0x120
[160372.306795] [<ffffffff810e9a93>] ? do_kern_mount+0x53/0x120
[160372.306838] [<ffffffff810eb662>] ? do_mount+0x532/0x8a0
[160372.306880] [<ffffffff810eb05a>] ? copy_mount_options+0xca/0x160
[160372.306922] [<ffffffff810ebb0a>] ? sys_mount+0x9a/0x100
[160372.306965] [<ffffffff8143e922>] ? system_call_fastpath+0x16/0x1b
[160372.307006] Code: b2 2f e9 c5 fd ff ff 90 66 c7 07 00 00 83 ea 02 48 83 c7 02 e9 72 fd ff ff 0f 1f 80 00 00 00 00 41 be f4 ff ff ff e9 22 fe ff ff <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec 58 48 89 7c 24
[160372.307191] RIP [<ffffffff81187e03>] nfs_idmap_legacy_upcall+0x323/0x330
[160372.307236] RSP <ffff8802421af5f8>
[160372.307672] ---[ end trace fefc21185a0edeb6 ]
On Aug 28, 2012, at 1:43 PM, Jim Rees <[email protected]> wrote:
> Frank Nicholas wrote:
>
> Thanks for the quick reply.
>
> The patches applied cleanly. I've rebuilt my kernel. The reboot will
> have to wait until I have physical access to the machine to do a hard
> power off? Unless you know of some way to clean up the hung NFS items so
> I can do a reboot. Currently if I try a 'shutdown -r now', the system
> hangs on trying to clean up the NFS items.
>
> Did you try "reboot -f"?
Yep - Remembering back to the old Solaris days, 'sync ; sync ; sync ; reboot -f'.
That got it. The 'emerge --sync' ran fine & the VM's that are using stores from NFS shares are working ok. So far no NFS issues.
Thanks.
Thanks for the quick reply.
The patches applied cleanly. I've rebuilt my kernel. The reboot will have to wait until I have physical access to the machine to do a hard power off? Unless you know of some way to clean up the hung NFS items so I can do a reboot. Currently if I try a 'shutdown -r now', the system hangs on trying to clean up the NFS items.
Thanks.
On Aug 28, 2012, at 10:43 AM, Bryan Schumaker <[email protected]> wrote:
> Thanks for the info, this looks like a problem we've already seen and (hopefully) fixed. Can you try the attached patches? They were recently added to stable, so I think that means they'll be in linux 3.5.4.
>
> - Bryan
>
> On 08/28/2012 08:49 AM, Frank Nicholas wrote:
>> Additional information I probably should have included:
>>
>> Linux martin 3.5.2-gentoo #1 SMP Fri Aug 24 09:35:54 EDT 2012 x86_64 Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz GenuineIntel GNU/Linux
>>
>> Gnu C 4.6.3
>> Gnu make 3.82
>> binutils 2.22.90.20120727
>> util-linux 2.21.2
>> mount support
>> module-init-tools 9
>> e2fsprogs 1.42.5
>> reiserfsprogs 3.6.21
>> Linux C Library 2.15
>> Dynamic linker (ldd) 2.15
>> Procps UNKNOWN
>> Net-tools 1.60_p20120127084908
>> Kbd 1.15.3wip
>> Sh-utils 8.19
>> Modules Loaded vboxnetadp vboxnetflt vboxdrv
>>
>> On Aug 28, 2012, at 8:42 AM, Frank Nicholas <[email protected]> wrote:
>>
>>> The information included below is per the URL:
>>> http://www.kernel.org/pub/linux/docs/lkml/reporting-bugs.html
>>>
>>> If different or additional information is needed, please let me know.
>>>
>>> Kernel version:
>>> Linux martin 3.5.2-gentoo #1 SMP Fri Aug 24 09:35:54 EDT 2012 x86_64 Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz GenuineIntel GNU/Linux
>>> Linux version 3.5.2-gentoo (root@martin) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.2) ) #1 SMP Fri Aug 24 09:35:54 EDT 2012
>>> Problem also occurred in 3.5.1-gentoo.
>>>
>>> Modules:
>>> Module Size Used by
>>> vboxnetadp 17542 0
>>> vboxnetflt 14893 0
>>> vboxdrv 1775582 2 vboxnetadp,vboxnetflt
>>>
>>> Bug trigger:
>>> Solaris 11/11 (patched as of July, 2012) is serving NFS V4 to a Gentoo Linux system. IDMAP is being used to manage users/groups. "/etc/idmap.conf" is fairly generic except for the domain name.
>>> The Gentoo "portage" tree is shared via NFS. I initiate an 'emerge --sync' and the condition occurs. NFS no longer works. The system will not restart or shut down due to the hang in NFS. The system has to be hard powered off to clear the condition. The bug does not trigger reliably. The Solaris 11/11 NFS server & Gentoo Linux NFS client have been configured this way for more than a year.
>>>
>>> Kernel log:
>>> [160372.303310] ------------[ cut here ]------------
>>> [160372.303353] kernel BUG at fs/nfs/idmap.c:684!
>>> [160372.303392] invalid opcode: 0000 [#1] SMP
>>> [160372.303431] CPU 2
>>> [160372.303437] Modules linked in: vboxnetadp(O) vboxnetflt(O) vboxdrv(O)
>>> [160372.303541]
>>> [160372.303572] Pid: 12852, comm: mount.nfs Tainted: G O 3.5.2-gentoo #1 Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3
>>> [160372.303655] RIP: 0010:[<ffffffff81187e03>] [<ffffffff81187e03>] nfs_idmap_legacy_upcall+0x323/0x330
>>> [160372.303736] RSP: 0018:ffff8802421af5f8 EFLAGS: 00010282
>>> [160372.303777] RAX: 0000000000000015 RBX: ffff8802ed9e5000 RCX: 0000000000000000
>>> [160372.303847] RDX: 0000000000000005 RSI: ffff88026af757b9 RDI: ffff8802ed9e5017
>>> [160372.303917] RBP: ffff88040d524740 R08: 000000000000003a R09: ffffffff81187b35
>>> [160372.303987] R10: 00000000139c5e77 R11: 000000006138df6f R12: ffff88040b568410
>>> [160372.304056] R13: ffff88040c3e5700 R14: 0000000000000015 R15: 0000000000000000
>>> [160372.304128] FS: 00007f35102ca700(0000) GS:ffff88041fb00000(0000) knlGS:0000000000000000
>>> [160372.304200] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> [160372.304240] CR2: 00007f37a2b09880 CR3: 0000000236eae000 CR4: 00000000000407e0
>>> [160372.304311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> [160372.304382] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> [160372.304453] Process mount.nfs (pid: 12852, threadinfo ffff8802421ae000, task ffff8803e789be90)
>>> [160372.304525] Stack:
>>> [160372.304558] ffff88026af757a4 ffff88026af757b9 ffff88040b568410 ffff8802ed9e59c0
>>> [160372.304632] ffff8802ed9e5540 ffffffff81526c0e 0000000000000000 ffffffff811ad2de
>>> [160372.304707] 0000000000000000 ffff88040d53ae40 ffff88040c3e5700 ffff88040d53ae40
>>> [160372.304781] Call Trace:
>>> [160372.304819] [<ffffffff811ad2de>] ? request_key_and_link+0x31e/0x450
>>> [160372.304864] [<ffffffff811ad465>] ? request_key_with_auxdata+0x15/0x60
>>> [160372.304908] [<ffffffff81187eeb>] ? nfs_idmap_request_key+0xdb/0x1c0
>>> [160372.304951] [<ffffffff811881a0>] ? nfs_idmap_lookup_id+0xe0/0x100
>>> [160372.304995] [<ffffffff811883a9>] ? nfs_map_string_to_numeric+0x29/0x90
>>> [160372.305039] [<ffffffff81182fcc>] ? decode_getfattr_attrs+0xb2c/0xb50
>>> [160372.305082] [<ffffffff8118307d>] ? decode_getfattr_generic.constprop.96+0x8d/0xe0
>>> [160372.305155] [<ffffffff811834b0>] ? nfs4_xdr_dec_link+0xf0/0xf0
>>> [160372.305197] [<ffffffff81183529>] ? nfs4_xdr_dec_lookup_root+0x79/0x80
>>> [160372.305240] [<ffffffff811834b0>] ? nfs4_xdr_dec_link+0xf0/0xf0
>>> [160372.305285] [<ffffffff81414d68>] ? rpcauth_unwrap_resp+0x58/0x60
>>> [160372.305328] [<ffffffff8140bed3>] ? call_decode+0x333/0x430
>>> [160372.305370] [<ffffffff81413776>] ? __rpc_execute+0x46/0x1e0
>>> [160372.305412] [<ffffffff810571f8>] ? wake_up_bit+0x18/0x40
>>> [160372.305454] [<ffffffff8140d529>] ? rpc_run_task+0x69/0x90
>>> [160372.305496] [<ffffffff8140d65f>] ? rpc_call_sync+0x3f/0x70
>>> [160372.306028] [<ffffffff81174e5e>] ? _nfs4_lookup_root.isra.42+0xae/0xd0
>>> [160372.306072] [<ffffffff81174ec7>] ? nfs4_lookup_root+0x47/0x80
>>> [160372.306115] [<ffffffff81179c10>] ? nfs4_proc_get_rootfh+0x30/0xd0
>>> [160372.306158] [<ffffffff8115e148>] ? nfs4_get_rootfh+0x28/0xc0
>>> [160372.306200] [<ffffffff8140cd21>] ? rpc_register_client+0x41/0x70
>>> [160372.306243] [<ffffffff81413fdb>] ? rpciod_up+0xb/0x20
>>> [160372.306283] [<ffffffff8140ce97>] ? rpc_clone_client+0x147/0x1c0
>>> [160372.306326] [<ffffffff81158290>] ? nfs4_server_common_setup+0x90/0x170
>>> [160372.306369] [<ffffffff811594fe>] ? nfs4_create_server+0x14e/0x2b0
>>> [160372.306413] [<ffffffff811625df>] ? nfs4_remote_mount+0x3f/0x90
>>> [160372.306457] [<ffffffff810d17da>] ? mount_fs+0x1a/0xd0
>>> [160372.306500] [<ffffffff810e92a3>] ? vfs_kern_mount+0x73/0x120
>>> [160372.306542] [<ffffffff81162720>] ? nfs_do_root_mount+0x90/0xe0
>>> [160372.306584] [<ffffffff81163bdf>] ? nfs_fs_mount+0x75f/0x970
>>> [160372.306626] [<ffffffff81163240>] ? nfs_fill_super+0x1d0/0x1d0
>>> [160372.306669] [<ffffffff81160720>] ? nfs_destroy_inode+0x20/0x20
>>> [160372.306712] [<ffffffff810d17da>] ? mount_fs+0x1a/0xd0
>>> [160372.306753] [<ffffffff810e92a3>] ? vfs_kern_mount+0x73/0x120
>>> [160372.306795] [<ffffffff810e9a93>] ? do_kern_mount+0x53/0x120
>>> [160372.306838] [<ffffffff810eb662>] ? do_mount+0x532/0x8a0
>>> [160372.306880] [<ffffffff810eb05a>] ? copy_mount_options+0xca/0x160
>>> [160372.306922] [<ffffffff810ebb0a>] ? sys_mount+0x9a/0x100
>>> [160372.306965] [<ffffffff8143e922>] ? system_call_fastpath+0x16/0x1b
>>> [160372.307006] Code: b2 2f e9 c5 fd ff ff 90 66 c7 07 00 00 83 ea 02 48 83 c7 02 e9 72 fd ff ff 0f 1f 80 00 00 00 00 41 be f4 ff ff ff e9 22 fe ff ff <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec 58 48 89 7c 24
>>> [160372.307191] RIP [<ffffffff81187e03>] nfs_idmap_legacy_upcall+0x323/0x330
>>> [160372.307236] RSP <ffff8802421af5f8>
>>> [160372.307672] ---[ end trace fefc21185a0edeb6 ]---
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
> <0001-NFS-Clear-key-construction-data-if-the-idmap-upcall-.patch><0002-NFS-return-ENOKEY-when-the-upcall-fails-to-map-the-n.patch>
Frank Nicholas wrote:
Thanks for the quick reply.
The patches applied cleanly. I've rebuilt my kernel. The reboot will
have to wait until I have physical access to the machine to do a hard
power off… Unless you know of some way to clean up the hung NFS items so
I can do a reboot. Currently if I try a 'shutdown -r now', the system
hangs on trying to clean up the NFS items.
Did you try "reboot -f"?
Thanks for the info, this looks like a problem we've already seen and (hopefully) fixed. Can you try the attached patches? They were recently added to stable, so I think that means they'll be in linux 3.5.4.
- Bryan
On 08/28/2012 08:49 AM, Frank Nicholas wrote:
> Additional information I probably should have included:
>
> Linux martin 3.5.2-gentoo #1 SMP Fri Aug 24 09:35:54 EDT 2012 x86_64 Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz GenuineIntel GNU/Linux
>
> Gnu C 4.6.3
> Gnu make 3.82
> binutils 2.22.90.20120727
> util-linux 2.21.2
> mount support
> module-init-tools 9
> e2fsprogs 1.42.5
> reiserfsprogs 3.6.21
> Linux C Library 2.15
> Dynamic linker (ldd) 2.15
> Procps UNKNOWN
> Net-tools 1.60_p20120127084908
> Kbd 1.15.3wip
> Sh-utils 8.19
> Modules Loaded vboxnetadp vboxnetflt vboxdrv
>
> On Aug 28, 2012, at 8:42 AM, Frank Nicholas <[email protected]> wrote:
>
>> The information included below is per the URL:
>> http://www.kernel.org/pub/linux/docs/lkml/reporting-bugs.html
>>
>> If different or additional information is needed, please let me know.
>>
>> Kernel version:
>> Linux martin 3.5.2-gentoo #1 SMP Fri Aug 24 09:35:54 EDT 2012 x86_64 Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz GenuineIntel GNU/Linux
>> Linux version 3.5.2-gentoo (root@martin) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.2) ) #1 SMP Fri Aug 24 09:35:54 EDT 2012
>> Problem also occurred in 3.5.1-gentoo.
>>
>> Modules:
>> Module Size Used by
>> vboxnetadp 17542 0
>> vboxnetflt 14893 0
>> vboxdrv 1775582 2 vboxnetadp,vboxnetflt
>>
>> Bug trigger:
>> Solaris 11/11 (patched as of July, 2012) is serving NFS V4 to a Gentoo Linux system. IDMAP is being used to manage users/groups. "/etc/idmap.conf" is fairly generic except for the domain name.
>> The Gentoo "portage" tree is shared via NFS. I initiate an 'emerge --sync' and the condition occurs. NFS no longer works. The system will not restart or shut down due to the hang in NFS. The system has to be hard powered off to clear the condition. The bug does not trigger reliably. The Solaris 11/11 NFS server & Gentoo Linux NFS client have been configured this way for more than a year.
>>
>> Kernel log:
>> [160372.303310] ------------[ cut here ]------------
>> [160372.303353] kernel BUG at fs/nfs/idmap.c:684!
>> [160372.303392] invalid opcode: 0000 [#1] SMP
>> [160372.303431] CPU 2
>> [160372.303437] Modules linked in: vboxnetadp(O) vboxnetflt(O) vboxdrv(O)
>> [160372.303541]
>> [160372.303572] Pid: 12852, comm: mount.nfs Tainted: G O 3.5.2-gentoo #1 Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3
>> [160372.303655] RIP: 0010:[<ffffffff81187e03>] [<ffffffff81187e03>] nfs_idmap_legacy_upcall+0x323/0x330
>> [160372.303736] RSP: 0018:ffff8802421af5f8 EFLAGS: 00010282
>> [160372.303777] RAX: 0000000000000015 RBX: ffff8802ed9e5000 RCX: 0000000000000000
>> [160372.303847] RDX: 0000000000000005 RSI: ffff88026af757b9 RDI: ffff8802ed9e5017
>> [160372.303917] RBP: ffff88040d524740 R08: 000000000000003a R09: ffffffff81187b35
>> [160372.303987] R10: 00000000139c5e77 R11: 000000006138df6f R12: ffff88040b568410
>> [160372.304056] R13: ffff88040c3e5700 R14: 0000000000000015 R15: 0000000000000000
>> [160372.304128] FS: 00007f35102ca700(0000) GS:ffff88041fb00000(0000) knlGS:0000000000000000
>> [160372.304200] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [160372.304240] CR2: 00007f37a2b09880 CR3: 0000000236eae000 CR4: 00000000000407e0
>> [160372.304311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [160372.304382] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [160372.304453] Process mount.nfs (pid: 12852, threadinfo ffff8802421ae000, task ffff8803e789be90)
>> [160372.304525] Stack:
>> [160372.304558] ffff88026af757a4 ffff88026af757b9 ffff88040b568410 ffff8802ed9e59c0
>> [160372.304632] ffff8802ed9e5540 ffffffff81526c0e 0000000000000000 ffffffff811ad2de
>> [160372.304707] 0000000000000000 ffff88040d53ae40 ffff88040c3e5700 ffff88040d53ae40
>> [160372.304781] Call Trace:
>> [160372.304819] [<ffffffff811ad2de>] ? request_key_and_link+0x31e/0x450
>> [160372.304864] [<ffffffff811ad465>] ? request_key_with_auxdata+0x15/0x60
>> [160372.304908] [<ffffffff81187eeb>] ? nfs_idmap_request_key+0xdb/0x1c0
>> [160372.304951] [<ffffffff811881a0>] ? nfs_idmap_lookup_id+0xe0/0x100
>> [160372.304995] [<ffffffff811883a9>] ? nfs_map_string_to_numeric+0x29/0x90
>> [160372.305039] [<ffffffff81182fcc>] ? decode_getfattr_attrs+0xb2c/0xb50
>> [160372.305082] [<ffffffff8118307d>] ? decode_getfattr_generic.constprop.96+0x8d/0xe0
>> [160372.305155] [<ffffffff811834b0>] ? nfs4_xdr_dec_link+0xf0/0xf0
>> [160372.305197] [<ffffffff81183529>] ? nfs4_xdr_dec_lookup_root+0x79/0x80
>> [160372.305240] [<ffffffff811834b0>] ? nfs4_xdr_dec_link+0xf0/0xf0
>> [160372.305285] [<ffffffff81414d68>] ? rpcauth_unwrap_resp+0x58/0x60
>> [160372.305328] [<ffffffff8140bed3>] ? call_decode+0x333/0x430
>> [160372.305370] [<ffffffff81413776>] ? __rpc_execute+0x46/0x1e0
>> [160372.305412] [<ffffffff810571f8>] ? wake_up_bit+0x18/0x40
>> [160372.305454] [<ffffffff8140d529>] ? rpc_run_task+0x69/0x90
>> [160372.305496] [<ffffffff8140d65f>] ? rpc_call_sync+0x3f/0x70
>> [160372.306028] [<ffffffff81174e5e>] ? _nfs4_lookup_root.isra.42+0xae/0xd0
>> [160372.306072] [<ffffffff81174ec7>] ? nfs4_lookup_root+0x47/0x80
>> [160372.306115] [<ffffffff81179c10>] ? nfs4_proc_get_rootfh+0x30/0xd0
>> [160372.306158] [<ffffffff8115e148>] ? nfs4_get_rootfh+0x28/0xc0
>> [160372.306200] [<ffffffff8140cd21>] ? rpc_register_client+0x41/0x70
>> [160372.306243] [<ffffffff81413fdb>] ? rpciod_up+0xb/0x20
>> [160372.306283] [<ffffffff8140ce97>] ? rpc_clone_client+0x147/0x1c0
>> [160372.306326] [<ffffffff81158290>] ? nfs4_server_common_setup+0x90/0x170
>> [160372.306369] [<ffffffff811594fe>] ? nfs4_create_server+0x14e/0x2b0
>> [160372.306413] [<ffffffff811625df>] ? nfs4_remote_mount+0x3f/0x90
>> [160372.306457] [<ffffffff810d17da>] ? mount_fs+0x1a/0xd0
>> [160372.306500] [<ffffffff810e92a3>] ? vfs_kern_mount+0x73/0x120
>> [160372.306542] [<ffffffff81162720>] ? nfs_do_root_mount+0x90/0xe0
>> [160372.306584] [<ffffffff81163bdf>] ? nfs_fs_mount+0x75f/0x970
>> [160372.306626] [<ffffffff81163240>] ? nfs_fill_super+0x1d0/0x1d0
>> [160372.306669] [<ffffffff81160720>] ? nfs_destroy_inode+0x20/0x20
>> [160372.306712] [<ffffffff810d17da>] ? mount_fs+0x1a/0xd0
>> [160372.306753] [<ffffffff810e92a3>] ? vfs_kern_mount+0x73/0x120
>> [160372.306795] [<ffffffff810e9a93>] ? do_kern_mount+0x53/0x120
>> [160372.306838] [<ffffffff810eb662>] ? do_mount+0x532/0x8a0
>> [160372.306880] [<ffffffff810eb05a>] ? copy_mount_options+0xca/0x160
>> [160372.306922] [<ffffffff810ebb0a>] ? sys_mount+0x9a/0x100
>> [160372.306965] [<ffffffff8143e922>] ? system_call_fastpath+0x16/0x1b
>> [160372.307006] Code: b2 2f e9 c5 fd ff ff 90 66 c7 07 00 00 83 ea 02 48 83 c7 02 e9 72 fd ff ff 0f 1f 80 00 00 00 00 41 be f4 ff ff ff e9 22 fe ff ff <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec 58 48 89 7c 24
>> [160372.307191] RIP [<ffffffff81187e03>] nfs_idmap_legacy_upcall+0x323/0x330
>> [160372.307236] RSP <ffff8802421af5f8>
>> [160372.307672] ---[ end trace fefc21185a0edeb6 ]---
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
On 08/28/2012 02:01 PM, Frank Nicholas wrote:
> On Aug 28, 2012, at 1:46 PM, Frank Nicholas <[email protected]> wrote:
>
>>
>> On Aug 28, 2012, at 1:43 PM, Jim Rees <[email protected]> wrote:
>>
>>> Frank Nicholas wrote:
>>>
>>> Thanks for the quick reply.
>>>
>>> The patches applied cleanly. I've rebuilt my kernel. The reboot will
>>> have to wait until I have physical access to the machine to do a hard
>>> power off? Unless you know of some way to clean up the hung NFS items so
>>> I can do a reboot. Currently if I try a 'shutdown -r now', the system
>>> hangs on trying to clean up the NFS items.
>>>
>>> Did you try "reboot -f"?
>>
>>
>> Yep - Remembering back to the old Solaris days, 'sync ; sync ; sync ; reboot -f'.
>>
>> That got it. The 'emerge --sync' ran fine & the VM's that are using stores from NFS shares are working ok. So far no NFS issues.
>>
>> Thanks.
>>
>
> After the reboot, doing an 'emerge --sync' & starting a VM, I have the following in my syslog:
>
> Aug 28 13:52:09 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>
> I believe each group of these (by date/time) were when I did the 'emerge --sync' & when I started the VM who has a virtual drive on an NFS share. I don't remember seeing anything like this before. Is this due to the patch? Should I be concerned? I know this is a pipe, but I've confirmed there is plenty of free disk space everywhere. The pipe does exist on the file system. There is plenty of free memory. Any suggestions?
>
Do you have rpc.idmapd running? Alternatively, if you have keyutils installed you could configure /etc/request-key.conf following the instructions in `man 8 nfsidmap`.
- Bryan
> Thanks.
>
On Aug 28, 2012, at 1:46 PM, Frank Nicholas <[email protected]> wrote:
>
> On Aug 28, 2012, at 1:43 PM, Jim Rees <[email protected]> wrote:
>
>> Frank Nicholas wrote:
>>
>> Thanks for the quick reply.
>>
>> The patches applied cleanly. I've rebuilt my kernel. The reboot will
>> have to wait until I have physical access to the machine to do a hard
>> power off? Unless you know of some way to clean up the hung NFS items so
>> I can do a reboot. Currently if I try a 'shutdown -r now', the system
>> hangs on trying to clean up the NFS items.
>>
>> Did you try "reboot -f"?
>
>
> Yep - Remembering back to the old Solaris days, 'sync ; sync ; sync ; reboot -f'.
>
> That got it. The 'emerge --sync' ran fine & the VM's that are using stores from NFS shares are working ok. So far no NFS issues.
>
> Thanks.
>
After the reboot, doing an 'emerge --sync' & starting a VM, I have the following in my syslog:
Aug 28 13:52:09 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
I believe each group of these (by date/time) were when I did the 'emerge --sync' & when I started the VM who has a virtual drive on an NFS share. I don't remember seeing anything like this before. Is this due to the patch? Should I be concerned? I know this is a pipe, but I've confirmed there is plenty of free disk space everywhere. The pipe does exist on the file system. There is plenty of free memory. Any suggestions?
Thanks.
On 08/28/2012 02:06 PM, Myklebust, Trond wrote:
> On Tue, 2012-08-28 at 14:01 -0400, Frank Nicholas wrote:
>> On Aug 28, 2012, at 1:46 PM, Frank Nicholas <[email protected]> wrote:
>>
>>>
>>> On Aug 28, 2012, at 1:43 PM, Jim Rees <[email protected]> wrote:
>>>
>>>> Frank Nicholas wrote:
>>>>
>>>> Thanks for the quick reply.
>>>>
>>>> The patches applied cleanly. I've rebuilt my kernel. The reboot will
>>>> have to wait until I have physical access to the machine to do a hard
>>>> power off… Unless you know of some way to clean up the hung NFS items so
>>>> I can do a reboot. Currently if I try a 'shutdown -r now', the system
>>>> hangs on trying to clean up the NFS items.
>>>>
>>>> Did you try "reboot -f"?
>>>
>>>
>>> Yep - Remembering back to the old Solaris days, 'sync ; sync ; sync ; reboot -f'.
>>>
>>> That got it. The 'emerge --sync' ran fine & the VM's that are using stores from NFS shares are working ok. So far no NFS issues.
>>>
>>> Thanks.
>>>
>>
>> After the reboot, doing an 'emerge --sync' & starting a VM, I have the following in my syslog:
>>
>> Aug 28 13:52:09 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>
>> I believe each group of these (by date/time) were when I did the 'emerge --sync' & when I started the VM who has a virtual drive on an NFS share. I don't remember seeing anything like this before. Is this due to the patch? Should I be concerned? I know this is a pipe, but I've confirmed there is plenty of free disk space everywhere. The pipe does exist on the file system. There is plenty of free memory. Any suggestions?
>
> An early version of those patches did indeed cause problems such as the
> above. Are you sure that you applied the patches that actually went
> upstream in
No, I don't think he did. I forgot you fixed up a few things, so I sent him the last copy I had.
- Bryan
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git&a=commitdiff&h=c5066945b7ea346a11424dbeb7830b7d7d00c206
> and
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git&a=commitdiff&h=12dfd080556124088ed61a292184947711b46cbe
>
On Aug 28, 2012, at 2:47 PM, Frank Nicholas <[email protected]> wrote:
> On Aug 28, 2012, at 2:08 PM, Bryan Schumaker <[email protected]> wrote:
>
>> On 08/28/2012 02:06 PM, Myklebust, Trond wrote:
>>> On Tue, 2012-08-28 at 14:01 -0400, Frank Nicholas wrote:
>>>
>>> An early version of those patches did indeed cause problems such as the
>>> above. Are you sure that you applied the patches that actually went
>>> upstream in
>>
>> No, I don't think he did. I forgot you fixed up a few things, so I sent him the last copy I had.
>>
>> - Bryan
>>
>>>
>>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git&a=commitdiff&h=c5066945b7ea346a11424dbeb7830b7d7d00c206
>>> and
>>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git&a=commitdiff&h=12dfd080556124088ed61a292184947711b46cbe
>>>
>>
>
>
> rpc.idmapd is running.
>
> I applied the patches Bryan e-mailed me. I'll apply the linked patches & confirm they clear up the log entries.
>
> Thanks.
Linked patches have been applied. syslog messages are gone and all seems to be good.
What version of the kernel will include these patches?
Thanks everyone for the quick replies & solution!
T24gVHVlLCAyMDEyLTA4LTI4IGF0IDE0OjAxIC0wNDAwLCBGcmFuayBOaWNob2xhcyB3cm90ZToN
Cj4gT24gQXVnIDI4LCAyMDEyLCBhdCAxOjQ2IFBNLCBGcmFuayBOaWNob2xhcyA8ZnJhbmtAbmlj
aG9sYXNmYW1pbHljZW50cmFsLmNvbT4gd3JvdGU6DQo+IA0KPiA+IA0KPiA+IE9uIEF1ZyAyOCwg
MjAxMiwgYXQgMTo0MyBQTSwgSmltIFJlZXMgPHJlZXNAdW1pY2guZWR1PiB3cm90ZToNCj4gPiAN
Cj4gPj4gRnJhbmsgTmljaG9sYXMgd3JvdGU6DQo+ID4+IA0KPiA+PiBUaGFua3MgZm9yIHRoZSBx
dWljayByZXBseS4NCj4gPj4gDQo+ID4+IFRoZSBwYXRjaGVzIGFwcGxpZWQgY2xlYW5seS4gIEkn
dmUgcmVidWlsdCBteSBrZXJuZWwuICBUaGUgcmVib290IHdpbGwNCj4gPj4gaGF2ZSB0byB3YWl0
IHVudGlsIEkgaGF2ZSBwaHlzaWNhbCBhY2Nlc3MgdG8gdGhlIG1hY2hpbmUgdG8gZG8gYSBoYXJk
DQo+ID4+IHBvd2VyIG9mZuKApiAgVW5sZXNzIHlvdSBrbm93IG9mIHNvbWUgd2F5IHRvIGNsZWFu
IHVwIHRoZSBodW5nIE5GUyBpdGVtcyBzbw0KPiA+PiBJIGNhbiBkbyBhIHJlYm9vdC4gIEN1cnJl
bnRseSBpZiBJIHRyeSBhICdzaHV0ZG93biAtciBub3cnLCB0aGUgc3lzdGVtDQo+ID4+IGhhbmdz
IG9uIHRyeWluZyB0byBjbGVhbiB1cCB0aGUgTkZTIGl0ZW1zLg0KPiA+PiANCj4gPj4gRGlkIHlv
dSB0cnkgInJlYm9vdCAtZiI/DQo+ID4gDQo+ID4gDQo+ID4gWWVwIC0gUmVtZW1iZXJpbmcgYmFj
ayB0byB0aGUgb2xkIFNvbGFyaXMgZGF5cywgJ3N5bmMgOyBzeW5jIDsgc3luYyA7IHJlYm9vdCAt
ZicuDQo+ID4gDQo+ID4gVGhhdCBnb3QgaXQuICBUaGUgJ2VtZXJnZSAtLXN5bmMnIHJhbiBmaW5l
ICYgdGhlIFZNJ3MgdGhhdCBhcmUgdXNpbmcgc3RvcmVzIGZyb20gTkZTIHNoYXJlcyBhcmUgd29y
a2luZyBvay4gIFNvIGZhciBubyBORlMgaXNzdWVzLiAgDQo+ID4gDQo+ID4gVGhhbmtzLg0KPiA+
IA0KPiANCj4gQWZ0ZXIgdGhlIHJlYm9vdCwgZG9pbmcgYW4gJ2VtZXJnZSAtLXN5bmMnICYgc3Rh
cnRpbmcgYSBWTSwgSSBoYXZlIHRoZSBmb2xsb3dpbmcgaW4gbXkgc3lzbG9nOg0KPiANCj4gQXVn
IDI4IDEzOjUyOjA5IG1hcnRpbiBycGMuaWRtYXBkWzI4OTldOiBuZnNjYjogd3JpdGUoL3Zhci9s
aWIvbmZzL3JwY19waXBlZnMvL25mcy9jbG50MC9pZG1hcCk6IE5vIHNwYWNlIGxlZnQgb24gZGV2
aWNlDQo+IEF1ZyAyOCAxMzo1MjoxOSBtYXJ0aW4gcnBjLmlkbWFwZFsyODk5XTogbmZzY2I6IHdy
aXRlKC92YXIvbGliL25mcy9ycGNfcGlwZWZzLy9uZnMvY2xudDAvaWRtYXApOiBObyBzcGFjZSBs
ZWZ0IG9uIGRldmljZQ0KPiBBdWcgMjggMTM6NTI6MTkgbWFydGluIHJwYy5pZG1hcGRbMjg5OV06
IG5mc2NiOiB3cml0ZSgvdmFyL2xpYi9uZnMvcnBjX3BpcGVmcy8vbmZzL2NsbnQwL2lkbWFwKTog
Tm8gc3BhY2UgbGVmdCBvbiBkZXZpY2UNCj4gQXVnIDI4IDEzOjUyOjE5IG1hcnRpbiBycGMuaWRt
YXBkWzI4OTldOiBuZnNjYjogd3JpdGUoL3Zhci9saWIvbmZzL3JwY19waXBlZnMvL25mcy9jbG50
MC9pZG1hcCk6IE5vIHNwYWNlIGxlZnQgb24gZGV2aWNlDQo+IEF1ZyAyOCAxMzo1MjoxOSBtYXJ0
aW4gcnBjLmlkbWFwZFsyODk5XTogbmZzY2I6IHdyaXRlKC92YXIvbGliL25mcy9ycGNfcGlwZWZz
Ly9uZnMvY2xudDAvaWRtYXApOiBObyBzcGFjZSBsZWZ0IG9uIGRldmljZQ0KPiBBdWcgMjggMTM6
NTI6MTkgbWFydGluIHJwYy5pZG1hcGRbMjg5OV06IG5mc2NiOiB3cml0ZSgvdmFyL2xpYi9uZnMv
cnBjX3BpcGVmcy8vbmZzL2NsbnQwL2lkbWFwKTogTm8gc3BhY2UgbGVmdCBvbiBkZXZpY2UNCj4g
QXVnIDI4IDEzOjUyOjE5IG1hcnRpbiBycGMuaWRtYXBkWzI4OTldOiBuZnNjYjogd3JpdGUoL3Zh
ci9saWIvbmZzL3JwY19waXBlZnMvL25mcy9jbG50MC9pZG1hcCk6IE5vIHNwYWNlIGxlZnQgb24g
ZGV2aWNlDQo+IEF1ZyAyOCAxMzo1MjoxOSBtYXJ0aW4gcnBjLmlkbWFwZFsyODk5XTogbmZzY2I6
IHdyaXRlKC92YXIvbGliL25mcy9ycGNfcGlwZWZzLy9uZnMvY2xudDAvaWRtYXApOiBObyBzcGFj
ZSBsZWZ0IG9uIGRldmljZQ0KPiBBdWcgMjggMTM6NTQ6MDUgbWFydGluIHJwYy5pZG1hcGRbMjg5
OV06IG5mc2NiOiB3cml0ZSgvdmFyL2xpYi9uZnMvcnBjX3BpcGVmcy8vbmZzL2NsbnQwL2lkbWFw
KTogTm8gc3BhY2UgbGVmdCBvbiBkZXZpY2UNCj4gQXVnIDI4IDEzOjU0OjA1IG1hcnRpbiBycGMu
aWRtYXBkWzI4OTldOiBuZnNjYjogd3JpdGUoL3Zhci9saWIvbmZzL3JwY19waXBlZnMvL25mcy9j
bG50MC9pZG1hcCk6IE5vIHNwYWNlIGxlZnQgb24gZGV2aWNlDQo+IEF1ZyAyOCAxMzo1NDowNSBt
YXJ0aW4gcnBjLmlkbWFwZFsyODk5XTogbmZzY2I6IHdyaXRlKC92YXIvbGliL25mcy9ycGNfcGlw
ZWZzLy9uZnMvY2xudDAvaWRtYXApOiBObyBzcGFjZSBsZWZ0IG9uIGRldmljZQ0KPiBBdWcgMjgg
MTM6NTQ6MDUgbWFydGluIHJwYy5pZG1hcGRbMjg5OV06IG5mc2NiOiB3cml0ZSgvdmFyL2xpYi9u
ZnMvcnBjX3BpcGVmcy8vbmZzL2NsbnQwL2lkbWFwKTogTm8gc3BhY2UgbGVmdCBvbiBkZXZpY2UN
Cj4gQXVnIDI4IDEzOjU0OjA1IG1hcnRpbiBycGMuaWRtYXBkWzI4OTldOiBuZnNjYjogd3JpdGUo
L3Zhci9saWIvbmZzL3JwY19waXBlZnMvL25mcy9jbG50MC9pZG1hcCk6IE5vIHNwYWNlIGxlZnQg
b24gZGV2aWNlDQo+IEF1ZyAyOCAxMzo1NDowNSBtYXJ0aW4gcnBjLmlkbWFwZFsyODk5XTogbmZz
Y2I6IHdyaXRlKC92YXIvbGliL25mcy9ycGNfcGlwZWZzLy9uZnMvY2xudDAvaWRtYXApOiBObyBz
cGFjZSBsZWZ0IG9uIGRldmljZQ0KPiANCj4gSSBiZWxpZXZlIGVhY2ggZ3JvdXAgb2YgdGhlc2Ug
KGJ5IGRhdGUvdGltZSkgd2VyZSB3aGVuIEkgZGlkIHRoZSAnZW1lcmdlIC0tc3luYycgJiB3aGVu
IEkgc3RhcnRlZCB0aGUgVk0gd2hvIGhhcyBhIHZpcnR1YWwgZHJpdmUgb24gYW4gTkZTIHNoYXJl
LiAgSSBkb24ndCByZW1lbWJlciBzZWVpbmcgYW55dGhpbmcgbGlrZSB0aGlzIGJlZm9yZS4gIElz
IHRoaXMgZHVlIHRvIHRoZSBwYXRjaD8gIFNob3VsZCBJIGJlIGNvbmNlcm5lZD8gIEkga25vdyB0
aGlzIGlzIGEgcGlwZSwgYnV0IEkndmUgY29uZmlybWVkIHRoZXJlIGlzIHBsZW50eSBvZiBmcmVl
IGRpc2sgc3BhY2UgZXZlcnl3aGVyZS4gIFRoZSBwaXBlIGRvZXMgZXhpc3Qgb24gdGhlIGZpbGUg
c3lzdGVtLiAgVGhlcmUgaXMgcGxlbnR5IG9mIGZyZWUgbWVtb3J5LiAgQW55IHN1Z2dlc3Rpb25z
Pw0KDQpBbiBlYXJseSB2ZXJzaW9uIG9mIHRob3NlIHBhdGNoZXMgZGlkIGluZGVlZCBjYXVzZSBw
cm9ibGVtcyBzdWNoIGFzIHRoZQ0KYWJvdmUuIEFyZSB5b3Ugc3VyZSB0aGF0IHlvdSBhcHBsaWVk
IHRoZSBwYXRjaGVzIHRoYXQgYWN0dWFsbHkgd2VudA0KdXBzdHJlYW0gaW4NCg0KaHR0cDovL2dp
dC5rZXJuZWwub3JnLz9wPWxpbnV4L2tlcm5lbC9naXQvdG9ydmFsZHMvbGludXgtMi42LmdpdCZh
PWNvbW1pdGRpZmYmaD1jNTA2Njk0NWI3ZWEzNDZhMTE0MjRkYmViNzgzMGI3ZDdkMDBjMjA2DQph
bmQNCmh0dHA6Ly9naXQua2VybmVsLm9yZy8/cD1saW51eC9rZXJuZWwvZ2l0L3RvcnZhbGRzL2xp
bnV4LTIuNi5naXQmYT1jb21taXRkaWZmJmg9MTJkZmQwODA1NTYxMjQwODhlZDYxYTI5MjE4NDk0
NzcxMWI0NmNiZQ0KDQotLSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWlu
dGFpbmVyDQoNCk5ldEFwcA0KVHJvbmQuTXlrbGVidXN0QG5ldGFwcC5jb20NCnd3dy5uZXRhcHAu
Y29tDQoNCg==
On Aug 28, 2012, at 2:08 PM, Bryan Schumaker <[email protected]> wrote:
> On 08/28/2012 02:06 PM, Myklebust, Trond wrote:
>> On Tue, 2012-08-28 at 14:01 -0400, Frank Nicholas wrote:
>>> On Aug 28, 2012, at 1:46 PM, Frank Nicholas <[email protected]> wrote:
>>>
>>>>
>>>> On Aug 28, 2012, at 1:43 PM, Jim Rees <[email protected]> wrote:
>>>>
>>>>> Frank Nicholas wrote:
>>>>>
>>>>> Thanks for the quick reply.
>>>>>
>>>>> The patches applied cleanly. I've rebuilt my kernel. The reboot will
>>>>> have to wait until I have physical access to the machine to do a hard
>>>>> power off? Unless you know of some way to clean up the hung NFS items so
>>>>> I can do a reboot. Currently if I try a 'shutdown -r now', the system
>>>>> hangs on trying to clean up the NFS items.
>>>>>
>>>>> Did you try "reboot -f"?
>>>>
>>>>
>>>> Yep - Remembering back to the old Solaris days, 'sync ; sync ; sync ; reboot -f'.
>>>>
>>>> That got it. The 'emerge --sync' ran fine & the VM's that are using stores from NFS shares are working ok. So far no NFS issues.
>>>>
>>>> Thanks.
>>>>
>>>
>>> After the reboot, doing an 'emerge --sync' & starting a VM, I have the following in my syslog:
>>>
>>> Aug 28 13:52:09 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:52:19 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>> Aug 28 13:54:05 martin rpc.idmapd[2899]: nfscb: write(/var/lib/nfs/rpc_pipefs//nfs/clnt0/idmap): No space left on device
>>>
>>> I believe each group of these (by date/time) were when I did the 'emerge --sync' & when I started the VM who has a virtual drive on an NFS share. I don't remember seeing anything like this before. Is this due to the patch? Should I be concerned? I know this is a pipe, but I've confirmed there is plenty of free disk space everywhere. The pipe does exist on the file system. There is plenty of free memory. Any suggestions?
>>
>> An early version of those patches did indeed cause problems such as the
>> above. Are you sure that you applied the patches that actually went
>> upstream in
>
> No, I don't think he did. I forgot you fixed up a few things, so I sent him the last copy I had.
>
> - Bryan
>
>>
>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git&a=commitdiff&h=c5066945b7ea346a11424dbeb7830b7d7d00c206
>> and
>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git&a=commitdiff&h=12dfd080556124088ed61a292184947711b46cbe
>>
>
rpc.idmapd is running.
I applied the patches Bryan e-mailed me. I'll apply the linked patches & confirm they clear up the log entries.
Thanks.
Additional information I probably should have included:
Linux martin 3.5.2-gentoo #1 SMP Fri Aug 24 09:35:54 EDT 2012 x86_64 Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz GenuineIntel GNU/Linux
Gnu C 4.6.3
Gnu make 3.82
binutils 2.22.90.20120727
util-linux 2.21.2
mount support
module-init-tools 9
e2fsprogs 1.42.5
reiserfsprogs 3.6.21
Linux C Library 2.15
Dynamic linker (ldd) 2.15
Procps UNKNOWN
Net-tools 1.60_p20120127084908
Kbd 1.15.3wip
Sh-utils 8.19
Modules Loaded vboxnetadp vboxnetflt vboxdrv
On Aug 28, 2012, at 8:42 AM, Frank Nicholas <[email protected]> wrote:
> The information included below is per the URL:
> http://www.kernel.org/pub/linux/docs/lkml/reporting-bugs.html
>
> If different or additional information is needed, please let me know.
>
> Kernel version:
> Linux martin 3.5.2-gentoo #1 SMP Fri Aug 24 09:35:54 EDT 2012 x86_64 Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz GenuineIntel GNU/Linux
> Linux version 3.5.2-gentoo (root@martin) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.2) ) #1 SMP Fri Aug 24 09:35:54 EDT 2012
> Problem also occurred in 3.5.1-gentoo.
>
> Modules:
> Module Size Used by
> vboxnetadp 17542 0
> vboxnetflt 14893 0
> vboxdrv 1775582 2 vboxnetadp,vboxnetflt
>
> Bug trigger:
> Solaris 11/11 (patched as of July, 2012) is serving NFS V4 to a Gentoo Linux system. IDMAP is being used to manage users/groups. "/etc/idmap.conf" is fairly generic except for the domain name.
> The Gentoo "portage" tree is shared via NFS. I initiate an 'emerge --sync' and the condition occurs. NFS no longer works. The system will not restart or shut down due to the hang in NFS. The system has to be hard powered off to clear the condition. The bug does not trigger reliably. The Solaris 11/11 NFS server & Gentoo Linux NFS client have been configured this way for more than a year.
>
> Kernel log:
> [160372.303310] ------------[ cut here ]------------
> [160372.303353] kernel BUG at fs/nfs/idmap.c:684!
> [160372.303392] invalid opcode: 0000 [#1] SMP
> [160372.303431] CPU 2
> [160372.303437] Modules linked in: vboxnetadp(O) vboxnetflt(O) vboxdrv(O)
> [160372.303541]
> [160372.303572] Pid: 12852, comm: mount.nfs Tainted: G O 3.5.2-gentoo #1 Gigabyte Technology Co., Ltd. Z68A-D3H-B3/Z68A-D3H-B3
> [160372.303655] RIP: 0010:[<ffffffff81187e03>] [<ffffffff81187e03>] nfs_idmap_legacy_upcall+0x323/0x330
> [160372.303736] RSP: 0018:ffff8802421af5f8 EFLAGS: 00010282
> [160372.303777] RAX: 0000000000000015 RBX: ffff8802ed9e5000 RCX: 0000000000000000
> [160372.303847] RDX: 0000000000000005 RSI: ffff88026af757b9 RDI: ffff8802ed9e5017
> [160372.303917] RBP: ffff88040d524740 R08: 000000000000003a R09: ffffffff81187b35
> [160372.303987] R10: 00000000139c5e77 R11: 000000006138df6f R12: ffff88040b568410
> [160372.304056] R13: ffff88040c3e5700 R14: 0000000000000015 R15: 0000000000000000
> [160372.304128] FS: 00007f35102ca700(0000) GS:ffff88041fb00000(0000) knlGS:0000000000000000
> [160372.304200] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [160372.304240] CR2: 00007f37a2b09880 CR3: 0000000236eae000 CR4: 00000000000407e0
> [160372.304311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [160372.304382] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [160372.304453] Process mount.nfs (pid: 12852, threadinfo ffff8802421ae000, task ffff8803e789be90)
> [160372.304525] Stack:
> [160372.304558] ffff88026af757a4 ffff88026af757b9 ffff88040b568410 ffff8802ed9e59c0
> [160372.304632] ffff8802ed9e5540 ffffffff81526c0e 0000000000000000 ffffffff811ad2de
> [160372.304707] 0000000000000000 ffff88040d53ae40 ffff88040c3e5700 ffff88040d53ae40
> [160372.304781] Call Trace:
> [160372.304819] [<ffffffff811ad2de>] ? request_key_and_link+0x31e/0x450
> [160372.304864] [<ffffffff811ad465>] ? request_key_with_auxdata+0x15/0x60
> [160372.304908] [<ffffffff81187eeb>] ? nfs_idmap_request_key+0xdb/0x1c0
> [160372.304951] [<ffffffff811881a0>] ? nfs_idmap_lookup_id+0xe0/0x100
> [160372.304995] [<ffffffff811883a9>] ? nfs_map_string_to_numeric+0x29/0x90
> [160372.305039] [<ffffffff81182fcc>] ? decode_getfattr_attrs+0xb2c/0xb50
> [160372.305082] [<ffffffff8118307d>] ? decode_getfattr_generic.constprop.96+0x8d/0xe0
> [160372.305155] [<ffffffff811834b0>] ? nfs4_xdr_dec_link+0xf0/0xf0
> [160372.305197] [<ffffffff81183529>] ? nfs4_xdr_dec_lookup_root+0x79/0x80
> [160372.305240] [<ffffffff811834b0>] ? nfs4_xdr_dec_link+0xf0/0xf0
> [160372.305285] [<ffffffff81414d68>] ? rpcauth_unwrap_resp+0x58/0x60
> [160372.305328] [<ffffffff8140bed3>] ? call_decode+0x333/0x430
> [160372.305370] [<ffffffff81413776>] ? __rpc_execute+0x46/0x1e0
> [160372.305412] [<ffffffff810571f8>] ? wake_up_bit+0x18/0x40
> [160372.305454] [<ffffffff8140d529>] ? rpc_run_task+0x69/0x90
> [160372.305496] [<ffffffff8140d65f>] ? rpc_call_sync+0x3f/0x70
> [160372.306028] [<ffffffff81174e5e>] ? _nfs4_lookup_root.isra.42+0xae/0xd0
> [160372.306072] [<ffffffff81174ec7>] ? nfs4_lookup_root+0x47/0x80
> [160372.306115] [<ffffffff81179c10>] ? nfs4_proc_get_rootfh+0x30/0xd0
> [160372.306158] [<ffffffff8115e148>] ? nfs4_get_rootfh+0x28/0xc0
> [160372.306200] [<ffffffff8140cd21>] ? rpc_register_client+0x41/0x70
> [160372.306243] [<ffffffff81413fdb>] ? rpciod_up+0xb/0x20
> [160372.306283] [<ffffffff8140ce97>] ? rpc_clone_client+0x147/0x1c0
> [160372.306326] [<ffffffff81158290>] ? nfs4_server_common_setup+0x90/0x170
> [160372.306369] [<ffffffff811594fe>] ? nfs4_create_server+0x14e/0x2b0
> [160372.306413] [<ffffffff811625df>] ? nfs4_remote_mount+0x3f/0x90
> [160372.306457] [<ffffffff810d17da>] ? mount_fs+0x1a/0xd0
> [160372.306500] [<ffffffff810e92a3>] ? vfs_kern_mount+0x73/0x120
> [160372.306542] [<ffffffff81162720>] ? nfs_do_root_mount+0x90/0xe0
> [160372.306584] [<ffffffff81163bdf>] ? nfs_fs_mount+0x75f/0x970
> [160372.306626] [<ffffffff81163240>] ? nfs_fill_super+0x1d0/0x1d0
> [160372.306669] [<ffffffff81160720>] ? nfs_destroy_inode+0x20/0x20
> [160372.306712] [<ffffffff810d17da>] ? mount_fs+0x1a/0xd0
> [160372.306753] [<ffffffff810e92a3>] ? vfs_kern_mount+0x73/0x120
> [160372.306795] [<ffffffff810e9a93>] ? do_kern_mount+0x53/0x120
> [160372.306838] [<ffffffff810eb662>] ? do_mount+0x532/0x8a0
> [160372.306880] [<ffffffff810eb05a>] ? copy_mount_options+0xca/0x160
> [160372.306922] [<ffffffff810ebb0a>] ? sys_mount+0x9a/0x100
> [160372.306965] [<ffffffff8143e922>] ? system_call_fastpath+0x16/0x1b
> [160372.307006] Code: b2 2f e9 c5 fd ff ff 90 66 c7 07 00 00 83 ea 02 48 83 c7 02 e9 72 fd ff ff 0f 1f 80 00 00 00 00 41 be f4 ff ff ff e9 22 fe ff ff <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec 58 48 89 7c 24
> [160372.307191] RIP [<ffffffff81187e03>] nfs_idmap_legacy_upcall+0x323/0x330
> [160372.307236] RSP <ffff8802421af5f8>
> [160372.307672] ---[ end trace fefc21185a0edeb6 ]---
On 08/28/2012 03:55 PM, Frank Nicholas wrote:
> On Aug 28, 2012, at 2:47 PM, Frank Nicholas <[email protected]> wrote:
>
>> On Aug 28, 2012, at 2:08 PM, Bryan Schumaker <[email protected]> wrote:
>>
>>> On 08/28/2012 02:06 PM, Myklebust, Trond wrote:
>>>> On Tue, 2012-08-28 at 14:01 -0400, Frank Nicholas wrote:
>>>>
>>>> An early version of those patches did indeed cause problems such as the
>>>> above. Are you sure that you applied the patches that actually went
>>>> upstream in
>>>
>>> No, I don't think he did. I forgot you fixed up a few things, so I sent him the last copy I had.
>>>
>>> - Bryan
>>>
>>>>
>>>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git&a=commitdiff&h=c5066945b7ea346a11424dbeb7830b7d7d00c206
>>>> and
>>>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git&a=commitdiff&h=12dfd080556124088ed61a292184947711b46cbe
>>>>
>>>
>>
>>
>> rpc.idmapd is running.
>>
>> I applied the patches Bryan e-mailed me. I'll apply the linked patches & confirm they clear up the log entries.
>>
>> Thanks.
>
> Linked patches have been applied. syslog messages are gone and all seems to be good.
>
> What version of the kernel will include these patches?
I think 3.5.4, Greg K-H sent out email yesterday saying that they had just been added to the stable tree.
- Bryan
>
> Thanks everyone for the quick replies & solution!
>