2009-04-03 01:20:10

by Yinghai Lu

[permalink] [raw]
Subject: nfs mount fail on linus 20090402 git

mpk12-3214-189-158:~ # cat x
mount -t nfs 10.6.75.100:/data/shared/pxeboot /x
mpk12-3214-189-158:~ # sh x
[ 63.198629] ------------[ cut here ]------------
[ 63.202589] kernel BUG at mm/slub.c:2753!
[ 63.202589] invalid opcode: 0000 [#1] SMP
[ 63.202589] last sysfs file: /sys/devices/virtual/net/sit0/type
[ 63.202589] CPU 0
[ 63.202589] Modules linked in:
[ 63.202589] Pid: 10027, comm: mount.nfs Not tainted 2.6.29-07100-g833bb30 #21 Sun Fire X4440
[ 63.202589] RIP: 0010:[<ffffffff802e0015>] [<ffffffff802e0015>] kfree+0x5a/0xcd
[ 63.202589] RSP: 0018:ffff882042ceb9f8 EFLAGS: 00010246
[ 63.202589] RAX: 0200000000000000 RBX: 0000000000000005 RCX: ffffffff80a7dc1f
[ 63.202589] RDX: ffffe20000000000 RSI: ffffc2000000f470 RDI: ffffe2001c018950
[ 63.202589] RBP: ffff882042ceba18 R08: 0000000000000000 R09: ffffffff811019c0
[ 63.202589] R10: 000000004262ce02 R11: ffff882042ceba18 R12: ffff880800706475
[ 63.202589] R13: ffff882042886000 R14: ffff882042cebbd8 R15: ffff882042cebbf0
[ 63.202589] FS: 00007fac729ed6f0(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000
[ 63.202589] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 63.202589] CR2: 00007fac72c12000 CR3: 0000001841cbb000 CR4: 00000000000006e0
[ 63.202589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 63.202589] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 63.202589] Process mount.nfs (pid: 10027, threadinfo ffff882042cea000, task ffff8820434dc290)
[ 63.202589] Stack:
[ 63.202589] ffff882042ceba18 000000004262ce02 0000000000000005 ffff882042886028
[ 63.202589] ffff882042ceba58 ffffffff80a7dc1f 000000004262ce02 ffff882042886000
[ 63.202589] 000000004262ce02 ffff882042886000 ffffffff80a7b4a6 ffff882042c9ee18
[ 63.202589] Call Trace:
[ 63.202589] [<ffffffff80a7dc1f>] xs_destroy+0x67/0xac
[ 63.202589] [<ffffffff80a7b4a6>] ? xprt_destroy+0x0/0xa7
[ 63.202589] [<ffffffff80a7b532>] xprt_destroy+0x8c/0xa7
[ 63.202589] [<ffffffff80a823b2>] ? put_rpccred+0x112/0x131
[ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
[ 63.202589] [<ffffffff80a7a9a9>] ? rpc_free_client+0x0/0xf9
[ 63.202589] [<ffffffff80a7b490>] xprt_put+0x23/0x39
[ 63.202589] [<ffffffff80a7aa7a>] rpc_free_client+0xd1/0xf9
[ 63.202589] [<ffffffff80a83345>] ? unx_destroy+0x3c/0x57
[ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
[ 63.202589] [<ffffffff80a7aaa2>] ? rpc_free_auth+0x0/0x69
[ 63.202589] [<ffffffff80a7aaf0>] rpc_free_auth+0x4e/0x69
[ 63.202589] [<ffffffff8025b827>] ? __wake_up+0x52/0x75
[ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
[ 63.202589] [<ffffffff80a7a98e>] rpc_release_client+0x64/0x7f
[ 63.202589] [<ffffffff80a8061c>] ? rpc_put_task+0xb0/0xcb
[ 63.202589] [<ffffffff80a7abe0>] rpc_shutdown_client+0xd5/0xf8
[ 63.202589] [<ffffffff80a7a893>] ? rpc_call_sync+0x63/0x80
[ 63.202589] [<ffffffff803fc4ab>] nfs_mount+0x11f/0x1bf
[ 63.202589] [<ffffffff803f3036>] nfs_get_sb+0x4ac/0x82a
[ 63.202589] [<ffffffff802e8f24>] vfs_kern_mount+0x61/0xbf
[ 63.202589] [<ffffffff802fea1d>] ? get_fs_type+0x58/0xc5
[ 63.202589] [<ffffffff802e9015>] do_kern_mount+0x56/0x108
[ 63.202589] [<ffffffff80302195>] do_mount+0x729/0x788
[ 63.202589] [<ffffffff80300025>] ? copy_mount_options+0xdf/0x155
[ 63.202589] [<ffffffff8030228c>] sys_mount+0x98/0xf8
[ 63.202589] [<ffffffff80230d6b>] system_call_fastpath+0x16/0x1b
[ 63.202589] Code: 0c 48 ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48 8d 3c 10 48 8b 07 f6 c4 40 74 04 48 8b 7f 10 48 8b 07 84 c0 78 10 f6 c4 60 75 04 <0f> 0b eb fe e8 90 75 fd ff eb 4c 48 8b 4d 08 4c 8b 4f 10 9c 5b
[ 63.202589] RIP [<ffffffff802e0015>] kfree+0x5a/0xcd
[ 63.202589] RSP <ffff882042ceb9f8>
[ 63.524555] ---[ end trace cd0d38e02ad11d61 ]---


2009-04-03 05:52:38

by Pekka Enberg

[permalink] [raw]
Subject: Re: nfs mount fail on linus 20090402 git

On Fri, Apr 3, 2009 at 4:19 AM, Yinghai Lu <[email protected]> wrote:
> mpk12-3214-189-158:~ # cat x
> mount -t nfs 10.6.75.100:/data/shared/pxeboot /x
> mpk12-3214-189-158:~ # sh x
> [ ? 63.198629] ------------[ cut here ]------------
> [ ? 63.202589] kernel BUG at mm/slub.c:2753!
> [ ? 63.202589] invalid opcode: 0000 [#1] SMP
> [ ? 63.202589] last sysfs file: /sys/devices/virtual/net/sit0/type
> [ ? 63.202589] CPU 0
> [ ? 63.202589] Modules linked in:
> [ ? 63.202589] Pid: 10027, comm: mount.nfs Not tainted 2.6.29-07100-g833bb30 #21 Sun Fire X4440
> [ ? 63.202589] RIP: 0010:[<ffffffff802e0015>] ?[<ffffffff802e0015>] kfree+0x5a/0xcd
> [ ? 63.202589] RSP: 0018:ffff882042ceb9f8 ?EFLAGS: 00010246
> [ ? 63.202589] RAX: 0200000000000000 RBX: 0000000000000005 RCX: ffffffff80a7dc1f
> [ ? 63.202589] RDX: ffffe20000000000 RSI: ffffc2000000f470 RDI: ffffe2001c018950
> [ ? 63.202589] RBP: ffff882042ceba18 R08: 0000000000000000 R09: ffffffff811019c0
> [ ? 63.202589] R10: 000000004262ce02 R11: ffff882042ceba18 R12: ffff880800706475
> [ ? 63.202589] R13: ffff882042886000 R14: ffff882042cebbd8 R15: ffff882042cebbf0
> [ ? 63.202589] FS: ?00007fac729ed6f0(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000
> [ ? 63.202589] CS: ?0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ ? 63.202589] CR2: 00007fac72c12000 CR3: 0000001841cbb000 CR4: 00000000000006e0
> [ ? 63.202589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ ? 63.202589] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ ? 63.202589] Process mount.nfs (pid: 10027, threadinfo ffff882042cea000, task ffff8820434dc290)
> [ ? 63.202589] Stack:
> [ ? 63.202589] ?ffff882042ceba18 000000004262ce02 0000000000000005 ffff882042886028
> [ ? 63.202589] ?ffff882042ceba58 ffffffff80a7dc1f 000000004262ce02 ffff882042886000
> [ ? 63.202589] ?000000004262ce02 ffff882042886000 ffffffff80a7b4a6 ffff882042c9ee18
> [ ? 63.202589] Call Trace:
> [ ? 63.202589] ?[<ffffffff80a7dc1f>] xs_destroy+0x67/0xac
> [ ? 63.202589] ?[<ffffffff80a7b4a6>] ? xprt_destroy+0x0/0xa7
> [ ? 63.202589] ?[<ffffffff80a7b532>] xprt_destroy+0x8c/0xa7
> [ ? 63.202589] ?[<ffffffff80a823b2>] ? put_rpccred+0x112/0x131
> [ ? 63.202589] ?[<ffffffff8051cdd5>] kref_put+0x65/0x87
> [ ? 63.202589] ?[<ffffffff80a7a9a9>] ? rpc_free_client+0x0/0xf9
> [ ? 63.202589] ?[<ffffffff80a7b490>] xprt_put+0x23/0x39
> [ ? 63.202589] ?[<ffffffff80a7aa7a>] rpc_free_client+0xd1/0xf9
> [ ? 63.202589] ?[<ffffffff80a83345>] ? unx_destroy+0x3c/0x57
> [ ? 63.202589] ?[<ffffffff8051cdd5>] kref_put+0x65/0x87
> [ ? 63.202589] ?[<ffffffff80a7aaa2>] ? rpc_free_auth+0x0/0x69
> [ ? 63.202589] ?[<ffffffff80a7aaf0>] rpc_free_auth+0x4e/0x69
> [ ? 63.202589] ?[<ffffffff8025b827>] ? __wake_up+0x52/0x75
> [ ? 63.202589] ?[<ffffffff8051cdd5>] kref_put+0x65/0x87
> [ ? 63.202589] ?[<ffffffff80a7a98e>] rpc_release_client+0x64/0x7f
> [ ? 63.202589] ?[<ffffffff80a8061c>] ? rpc_put_task+0xb0/0xcb
> [ ? 63.202589] ?[<ffffffff80a7abe0>] rpc_shutdown_client+0xd5/0xf8
> [ ? 63.202589] ?[<ffffffff80a7a893>] ? rpc_call_sync+0x63/0x80
> [ ? 63.202589] ?[<ffffffff803fc4ab>] nfs_mount+0x11f/0x1bf
> [ ? 63.202589] ?[<ffffffff803f3036>] nfs_get_sb+0x4ac/0x82a
> [ ? 63.202589] ?[<ffffffff802e8f24>] vfs_kern_mount+0x61/0xbf
> [ ? 63.202589] ?[<ffffffff802fea1d>] ? get_fs_type+0x58/0xc5
> [ ? 63.202589] ?[<ffffffff802e9015>] do_kern_mount+0x56/0x108
> [ ? 63.202589] ?[<ffffffff80302195>] do_mount+0x729/0x788
> [ ? 63.202589] ?[<ffffffff80300025>] ? copy_mount_options+0xdf/0x155
> [ ? 63.202589] ?[<ffffffff8030228c>] sys_mount+0x98/0xf8
> [ ? 63.202589] ?[<ffffffff80230d6b>] system_call_fastpath+0x16/0x1b
> [ ? 63.202589] Code: 0c 48 ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48 8d 3c 10 48 8b 07 f6 c4 40 74 04 48 8b 7f 10 48 8b 07 84 c0 78 10 f6 c4 60 75 04 <0f> 0b eb fe e8 90 75 fd ff eb 4c 48 8b 4d 08 4c 8b 4f 10 9c 5b
> [ ? 63.202589] RIP ?[<ffffffff802e0015>] kfree+0x5a/0xcd
> [ ? 63.202589] ?RSP <ffff882042ceb9f8>
> [ ? 63.524555] ---[ end trace cd0d38e02ad11d61 ]---

Looks like a bogus pointer passed to kfree(). Lets cc some NFS folks.

2009-04-03 07:09:27

by Ingo Molnar

[permalink] [raw]
Subject: Re: nfs mount fail on linus 20090402 git


* Pekka Enberg <[email protected]> wrote:

> On Fri, Apr 3, 2009 at 4:19 AM, Yinghai Lu <[email protected]> wrote:
> > mpk12-3214-189-158:~ # cat x
> > mount -t nfs 10.6.75.100:/data/shared/pxeboot /x
> > mpk12-3214-189-158:~ # sh x
> > [ ? 63.198629] ------------[ cut here ]------------
> > [ ? 63.202589] kernel BUG at mm/slub.c:2753!
> > [ ? 63.202589] invalid opcode: 0000 [#1] SMP
> > [ ? 63.202589] last sysfs file: /sys/devices/virtual/net/sit0/type
> > [ ? 63.202589] CPU 0
> > [ ? 63.202589] Modules linked in:
> > [ ? 63.202589] Pid: 10027, comm: mount.nfs Not tainted 2.6.29-07100-g833bb30 #21 Sun Fire X4440
> > [ ? 63.202589] RIP: 0010:[<ffffffff802e0015>] ?[<ffffffff802e0015>] kfree+0x5a/0xcd
> > [ ? 63.202589] RSP: 0018:ffff882042ceb9f8 ?EFLAGS: 00010246
> > [ ? 63.202589] RAX: 0200000000000000 RBX: 0000000000000005 RCX: ffffffff80a7dc1f
> > [ ? 63.202589] RDX: ffffe20000000000 RSI: ffffc2000000f470 RDI: ffffe2001c018950
> > [ ? 63.202589] RBP: ffff882042ceba18 R08: 0000000000000000 R09: ffffffff811019c0
> > [ ? 63.202589] R10: 000000004262ce02 R11: ffff882042ceba18 R12: ffff880800706475
> > [ ? 63.202589] R13: ffff882042886000 R14: ffff882042cebbd8 R15: ffff882042cebbf0
> > [ ? 63.202589] FS: ?00007fac729ed6f0(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000
> > [ ? 63.202589] CS: ?0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [ ? 63.202589] CR2: 00007fac72c12000 CR3: 0000001841cbb000 CR4: 00000000000006e0
> > [ ? 63.202589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [ ? 63.202589] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [ ? 63.202589] Process mount.nfs (pid: 10027, threadinfo ffff882042cea000, task ffff8820434dc290)
> > [ ? 63.202589] Stack:
> > [ ? 63.202589] ?ffff882042ceba18 000000004262ce02 0000000000000005 ffff882042886028
> > [ ? 63.202589] ?ffff882042ceba58 ffffffff80a7dc1f 000000004262ce02 ffff882042886000
> > [ ? 63.202589] ?000000004262ce02 ffff882042886000 ffffffff80a7b4a6 ffff882042c9ee18
> > [ ? 63.202589] Call Trace:
> > [ ? 63.202589] ?[<ffffffff80a7dc1f>] xs_destroy+0x67/0xac
> > [ ? 63.202589] ?[<ffffffff80a7b4a6>] ? xprt_destroy+0x0/0xa7
> > [ ? 63.202589] ?[<ffffffff80a7b532>] xprt_destroy+0x8c/0xa7
> > [ ? 63.202589] ?[<ffffffff80a823b2>] ? put_rpccred+0x112/0x131
> > [ ? 63.202589] ?[<ffffffff8051cdd5>] kref_put+0x65/0x87
> > [ ? 63.202589] ?[<ffffffff80a7a9a9>] ? rpc_free_client+0x0/0xf9
> > [ ? 63.202589] ?[<ffffffff80a7b490>] xprt_put+0x23/0x39
> > [ ? 63.202589] ?[<ffffffff80a7aa7a>] rpc_free_client+0xd1/0xf9
> > [ ? 63.202589] ?[<ffffffff80a83345>] ? unx_destroy+0x3c/0x57
> > [ ? 63.202589] ?[<ffffffff8051cdd5>] kref_put+0x65/0x87
> > [ ? 63.202589] ?[<ffffffff80a7aaa2>] ? rpc_free_auth+0x0/0x69
> > [ ? 63.202589] ?[<ffffffff80a7aaf0>] rpc_free_auth+0x4e/0x69
> > [ ? 63.202589] ?[<ffffffff8025b827>] ? __wake_up+0x52/0x75
> > [ ? 63.202589] ?[<ffffffff8051cdd5>] kref_put+0x65/0x87
> > [ ? 63.202589] ?[<ffffffff80a7a98e>] rpc_release_client+0x64/0x7f
> > [ ? 63.202589] ?[<ffffffff80a8061c>] ? rpc_put_task+0xb0/0xcb
> > [ ? 63.202589] ?[<ffffffff80a7abe0>] rpc_shutdown_client+0xd5/0xf8
> > [ ? 63.202589] ?[<ffffffff80a7a893>] ? rpc_call_sync+0x63/0x80
> > [ ? 63.202589] ?[<ffffffff803fc4ab>] nfs_mount+0x11f/0x1bf
> > [ ? 63.202589] ?[<ffffffff803f3036>] nfs_get_sb+0x4ac/0x82a
> > [ ? 63.202589] ?[<ffffffff802e8f24>] vfs_kern_mount+0x61/0xbf
> > [ ? 63.202589] ?[<ffffffff802fea1d>] ? get_fs_type+0x58/0xc5
> > [ ? 63.202589] ?[<ffffffff802e9015>] do_kern_mount+0x56/0x108
> > [ ? 63.202589] ?[<ffffffff80302195>] do_mount+0x729/0x788
> > [ ? 63.202589] ?[<ffffffff80300025>] ? copy_mount_options+0xdf/0x155
> > [ ? 63.202589] ?[<ffffffff8030228c>] sys_mount+0x98/0xf8
> > [ ? 63.202589] ?[<ffffffff80230d6b>] system_call_fastpath+0x16/0x1b
> > [ ? 63.202589] Code: 0c 48 ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48 8d 3c 10 48 8b 07 f6 c4 40 74 04 48 8b 7f 10 48 8b 07 84 c0 78 10 f6 c4 60 75 04 <0f> 0b eb fe e8 90 75 fd ff eb 4c 48 8b 4d 08 4c 8b 4f 10 9c 5b
> > [ ? 63.202589] RIP ?[<ffffffff802e0015>] kfree+0x5a/0xcd
> > [ ? 63.202589] ?RSP <ffff882042ceb9f8>
> > [ ? 63.524555] ---[ end trace cd0d38e02ad11d61 ]---
>
> Looks like a bogus pointer passed to kfree(). Lets cc some NFS folks.

a quick look suggests that this fresh commit moves around a kfree()
and a destructor call:

a67d18f: NFS: load the rpc/rdma transport module automatically

the commit log does not mention why the kfree was moved which
suggests that it might have been done by mistake. (And it might be
totally correct even - i have not checked the details.) I'll send a
patch undoing those bits in the next mail.

Ingo

2009-04-03 07:16:43

by Yinghai Lu

[permalink] [raw]
Subject: Re: nfs mount fail on linus 20090402 git

Ingo Molnar wrote:
> * Pekka Enberg <[email protected]> wrote:
>
>> On Fri, Apr 3, 2009 at 4:19 AM, Yinghai Lu <[email protected]> wrote:
>>> mpk12-3214-189-158:~ # cat x
>>> mount -t nfs 10.6.75.100:/data/shared/pxeboot /x
>>> mpk12-3214-189-158:~ # sh x
>>> [ 63.198629] ------------[ cut here ]------------
>>> [ 63.202589] kernel BUG at mm/slub.c:2753!
>>> [ 63.202589] invalid opcode: 0000 [#1] SMP
>>> [ 63.202589] last sysfs file: /sys/devices/virtual/net/sit0/type
>>> [ 63.202589] CPU 0
>>> [ 63.202589] Modules linked in:
>>> [ 63.202589] Pid: 10027, comm: mount.nfs Not tainted 2.6.29-07100-g833bb30 #21 Sun Fire X4440
>>> [ 63.202589] RIP: 0010:[<ffffffff802e0015>] [<ffffffff802e0015>] kfree+0x5a/0xcd
>>> [ 63.202589] RSP: 0018:ffff882042ceb9f8 EFLAGS: 00010246
>>> [ 63.202589] RAX: 0200000000000000 RBX: 0000000000000005 RCX: ffffffff80a7dc1f
>>> [ 63.202589] RDX: ffffe20000000000 RSI: ffffc2000000f470 RDI: ffffe2001c018950
>>> [ 63.202589] RBP: ffff882042ceba18 R08: 0000000000000000 R09: ffffffff811019c0
>>> [ 63.202589] R10: 000000004262ce02 R11: ffff882042ceba18 R12: ffff880800706475
>>> [ 63.202589] R13: ffff882042886000 R14: ffff882042cebbd8 R15: ffff882042cebbf0
>>> [ 63.202589] FS: 00007fac729ed6f0(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000
>>> [ 63.202589] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> [ 63.202589] CR2: 00007fac72c12000 CR3: 0000001841cbb000 CR4: 00000000000006e0
>>> [ 63.202589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> [ 63.202589] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> [ 63.202589] Process mount.nfs (pid: 10027, threadinfo ffff882042cea000, task ffff8820434dc290)
>>> [ 63.202589] Stack:
>>> [ 63.202589] ffff882042ceba18 000000004262ce02 0000000000000005 ffff882042886028
>>> [ 63.202589] ffff882042ceba58 ffffffff80a7dc1f 000000004262ce02 ffff882042886000
>>> [ 63.202589] 000000004262ce02 ffff882042886000 ffffffff80a7b4a6 ffff882042c9ee18
>>> [ 63.202589] Call Trace:
>>> [ 63.202589] [<ffffffff80a7dc1f>] xs_destroy+0x67/0xac
>>> [ 63.202589] [<ffffffff80a7b4a6>] ? xprt_destroy+0x0/0xa7
>>> [ 63.202589] [<ffffffff80a7b532>] xprt_destroy+0x8c/0xa7
>>> [ 63.202589] [<ffffffff80a823b2>] ? put_rpccred+0x112/0x131
>>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
>>> [ 63.202589] [<ffffffff80a7a9a9>] ? rpc_free_client+0x0/0xf9
>>> [ 63.202589] [<ffffffff80a7b490>] xprt_put+0x23/0x39
>>> [ 63.202589] [<ffffffff80a7aa7a>] rpc_free_client+0xd1/0xf9
>>> [ 63.202589] [<ffffffff80a83345>] ? unx_destroy+0x3c/0x57
>>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
>>> [ 63.202589] [<ffffffff80a7aaa2>] ? rpc_free_auth+0x0/0x69
>>> [ 63.202589] [<ffffffff80a7aaf0>] rpc_free_auth+0x4e/0x69
>>> [ 63.202589] [<ffffffff8025b827>] ? __wake_up+0x52/0x75
>>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
>>> [ 63.202589] [<ffffffff80a7a98e>] rpc_release_client+0x64/0x7f
>>> [ 63.202589] [<ffffffff80a8061c>] ? rpc_put_task+0xb0/0xcb
>>> [ 63.202589] [<ffffffff80a7abe0>] rpc_shutdown_client+0xd5/0xf8
>>> [ 63.202589] [<ffffffff80a7a893>] ? rpc_call_sync+0x63/0x80
>>> [ 63.202589] [<ffffffff803fc4ab>] nfs_mount+0x11f/0x1bf
>>> [ 63.202589] [<ffffffff803f3036>] nfs_get_sb+0x4ac/0x82a
>>> [ 63.202589] [<ffffffff802e8f24>] vfs_kern_mount+0x61/0xbf
>>> [ 63.202589] [<ffffffff802fea1d>] ? get_fs_type+0x58/0xc5
>>> [ 63.202589] [<ffffffff802e9015>] do_kern_mount+0x56/0x108
>>> [ 63.202589] [<ffffffff80302195>] do_mount+0x729/0x788
>>> [ 63.202589] [<ffffffff80300025>] ? copy_mount_options+0xdf/0x155
>>> [ 63.202589] [<ffffffff8030228c>] sys_mount+0x98/0xf8
>>> [ 63.202589] [<ffffffff80230d6b>] system_call_fastpath+0x16/0x1b
>>> [ 63.202589] Code: 0c 48 ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48 8d 3c 10 48 8b 07 f6 c4 40 74 04 48 8b 7f 10 48 8b 07 84 c0 78 10 f6 c4 60 75 04 <0f> 0b eb fe e8 90 75 fd ff eb 4c 48 8b 4d 08 4c 8b 4f 10 9c 5b
>>> [ 63.202589] RIP [<ffffffff802e0015>] kfree+0x5a/0xcd
>>> [ 63.202589] RSP <ffff882042ceb9f8>
>>> [ 63.524555] ---[ end trace cd0d38e02ad11d61 ]---
>> Looks like a bogus pointer passed to kfree(). Lets cc some NFS folks.
>
> a quick look suggests that this fresh commit moves around a kfree()
> and a destructor call:
>
> a67d18f: NFS: load the rpc/rdma transport module automatically
>
> the commit log does not mention why the kfree was moved which
> suggests that it might have been done by mistake. (And it might be
> totally correct even - i have not checked the details.) I'll send a
> patch undoing those bits in the next mail.
>

got two kfree(string), could be some merging problem.

string = match_strdup(args);
if (string == NULL)
goto out_nomem;
token = match_token(string,
nfs_xprt_protocol_tokens, args);
kfree(string);

switch (token) {
case Opt_xprt_udp:
mnt->flags &= ~NFS_MOUNT_TCP;
mnt->nfs_server.protocol = XPRT_TRANSPORT_UDP;
break;
case Opt_xprt_tcp:
mnt->flags |= NFS_MOUNT_TCP;
mnt->nfs_server.protocol = XPRT_TRANSPORT_TCP;
break;
case Opt_xprt_rdma:
/* vector side protocols to TCP */
mnt->flags |= NFS_MOUNT_TCP;
mnt->nfs_server.protocol = XPRT_TRANSPORT_RDMA;
xprt_load_transport(string);
break;
default:
errors++;
dfprintk(MOUNT, "NFS: unrecognized "
"transport protocol\n");
}
kfree(string);

2009-04-03 07:17:20

by Ingo Molnar

[permalink] [raw]
Subject: [PATCH] nfs: fix nfs_parse_mount_options() double kfree()


Impact: fix crash

Yinghai Lu reported the following crash:

> mpk12-3214-189-158:~ # sh x
> [ ? 63.198629] ------------[ cut here ]------------
> [ ? 63.202589] kernel BUG at mm/slub.c:2753!
> [ ? 63.202589] invalid opcode: 0000 [#1] SMP
> [ ? 63.202589] last sysfs file: /sys/devices/virtual/net/sit0/type
> [ ? 63.202589] CPU 0
> [ ? 63.202589] Modules linked in:
> [ ? 63.202589] Pid: 10027, comm: mount.nfs Not tainted 2.6.29-07100-g833bb30 #21 Sun Fire X4440
> [ ? 63.202589] RIP: 0010:[<ffffffff802e0015>] ?[<ffffffff802e0015>] kfree+0x5a/0xcd
> [ ? 63.202589] RSP: 0018:ffff882042ceb9f8 ?EFLAGS: 00010246
> [ ? 63.202589] RAX: 0200000000000000 RBX: 0000000000000005 RCX: ffffffff80a7dc1f
> [ ? 63.202589] RDX: ffffe20000000000 RSI: ffffc2000000f470 RDI: ffffe2001c018950
> [ ? 63.202589] RBP: ffff882042ceba18 R08: 0000000000000000 R09: ffffffff811019c0
> [ ? 63.202589] R10: 000000004262ce02 R11: ffff882042ceba18 R12: ffff880800706475
> [ ? 63.202589] R13: ffff882042886000 R14: ffff882042cebbd8 R15: ffff882042cebbf0
> [ ? 63.202589] FS: ?00007fac729ed6f0(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000
> [ ? 63.202589] CS: ?0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ ? 63.202589] CR2: 00007fac72c12000 CR3: 0000001841cbb000 CR4: 00000000000006e0
> [ ? 63.202589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ ? 63.202589] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ ? 63.202589] Process mount.nfs (pid: 10027, threadinfo ffff882042cea000, task ffff8820434dc290)
> [ ? 63.202589] Stack:
> [ ? 63.202589] ?ffff882042ceba18 000000004262ce02 0000000000000005 ffff882042886028
> [ ? 63.202589] ?ffff882042ceba58 ffffffff80a7dc1f 000000004262ce02 ffff882042886000
> [ ? 63.202589] ?000000004262ce02 ffff882042886000 ffffffff80a7b4a6 ffff882042c9ee18
> [ ? 63.202589] Call Trace:
> [ ? 63.202589] ?[<ffffffff80a7dc1f>] xs_destroy+0x67/0xac
> [ ? 63.202589] ?[<ffffffff80a7b4a6>] ? xprt_destroy+0x0/0xa7
> [ ? 63.202589] ?[<ffffffff80a7b532>] xprt_destroy+0x8c/0xa7
> [ ? 63.202589] ?[<ffffffff80a823b2>] ? put_rpccred+0x112/0x131
> [ ? 63.202589] ?[<ffffffff8051cdd5>] kref_put+0x65/0x87
> [ ? 63.202589] ?[<ffffffff80a7a9a9>] ? rpc_free_client+0x0/0xf9
> [ ? 63.202589] ?[<ffffffff80a7b490>] xprt_put+0x23/0x39
> [ ? 63.202589] ?[<ffffffff80a7aa7a>] rpc_free_client+0xd1/0xf9
> [ ? 63.202589] ?[<ffffffff80a83345>] ? unx_destroy+0x3c/0x57
> [ ? 63.202589] ?[<ffffffff8051cdd5>] kref_put+0x65/0x87
> [ ? 63.202589] ?[<ffffffff80a7aaa2>] ? rpc_free_auth+0x0/0x69
> [ ? 63.202589] ?[<ffffffff80a7aaf0>] rpc_free_auth+0x4e/0x69
> [ ? 63.202589] ?[<ffffffff8025b827>] ? __wake_up+0x52/0x75
> [ ? 63.202589] ?[<ffffffff8051cdd5>] kref_put+0x65/0x87
> [ ? 63.202589] ?[<ffffffff80a7a98e>] rpc_release_client+0x64/0x7f
> [ ? 63.202589] ?[<ffffffff80a8061c>] ? rpc_put_task+0xb0/0xcb
> [ ? 63.202589] ?[<ffffffff80a7abe0>] rpc_shutdown_client+0xd5/0xf8
> [ ? 63.202589] ?[<ffffffff80a7a893>] ? rpc_call_sync+0x63/0x80
> [ ? 63.202589] ?[<ffffffff803fc4ab>] nfs_mount+0x11f/0x1bf
> [ ? 63.202589] ?[<ffffffff803f3036>] nfs_get_sb+0x4ac/0x82a
> [ ? 63.202589] ?[<ffffffff802e8f24>] vfs_kern_mount+0x61/0xbf
> [ ? 63.202589] ?[<ffffffff802fea1d>] ? get_fs_type+0x58/0xc5
> [ ? 63.202589] ?[<ffffffff802e9015>] do_kern_mount+0x56/0x108
> [ ? 63.202589] ?[<ffffffff80302195>] do_mount+0x729/0x788
> [ ? 63.202589] ?[<ffffffff80300025>] ? copy_mount_options+0xdf/0x155
> [ ? 63.202589] ?[<ffffffff8030228c>] sys_mount+0x98/0xf8
> [ ? 63.202589] ?[<ffffffff80230d6b>] system_call_fastpath+0x16/0x1b
> [ ? 63.202589] Code: 0c 48 ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48 8d 3c 10 48 8b 07 f6 c4 40 74 04 48 8b 7f 10 48 8b 07 84 c0 78 10 f6 c4 60 75 04 <0f> 0b eb fe e8 90 75 fd ff eb 4c 48 8b 4d 08 4c 8b 4f 10 9c 5b
> [ ? 63.202589] RIP ?[<ffffffff802e0015>] kfree+0x5a/0xcd
> [ ? 63.202589] ?RSP <ffff882042ceb9f8>
> [ ? 63.524555] ---[ end trace cd0d38e02ad11d61 ]---

Pekka observed that a bogus pointer was passed to kfree().

This commit:

a67d18f: NFS: load the rpc/rdma transport module automatically

Moved a kfree() of the options strings in nfs_parse_mount_options()
inadvertently and introduced a double kfree(). Fix it.

Reported-by: Yinghai Lu <[email protected]>
Analyzed-by: Pekka Enberg <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
---
fs/nfs/super.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/fs/nfs/super.c b/fs/nfs/super.c
index 0942fcb..4e7f879 100644
--- a/fs/nfs/super.c
+++ b/fs/nfs/super.c
@@ -1191,7 +1191,6 @@ static int nfs_parse_mount_options(char *raw,
goto out_nomem;
token = match_token(string,
nfs_xprt_protocol_tokens, args);
- kfree(string);

switch (token) {
case Opt_xprt_udp:
@@ -1221,6 +1220,7 @@ static int nfs_parse_mount_options(char *raw,
goto out_nomem;
token = match_token(string,
nfs_xprt_protocol_tokens, args);
+ kfree(string);

switch (token) {
case Opt_xprt_udp:

2009-04-03 07:19:45

by Ingo Molnar

[permalink] [raw]
Subject: Re: nfs mount fail on linus 20090402 git


* Yinghai Lu <[email protected]> wrote:

> Ingo Molnar wrote:
> > * Pekka Enberg <[email protected]> wrote:
> >
> >> On Fri, Apr 3, 2009 at 4:19 AM, Yinghai Lu <[email protected]> wrote:
> >>> mpk12-3214-189-158:~ # cat x
> >>> mount -t nfs 10.6.75.100:/data/shared/pxeboot /x
> >>> mpk12-3214-189-158:~ # sh x
> >>> [ 63.198629] ------------[ cut here ]------------
> >>> [ 63.202589] kernel BUG at mm/slub.c:2753!
> >>> [ 63.202589] invalid opcode: 0000 [#1] SMP
> >>> [ 63.202589] last sysfs file: /sys/devices/virtual/net/sit0/type
> >>> [ 63.202589] CPU 0
> >>> [ 63.202589] Modules linked in:
> >>> [ 63.202589] Pid: 10027, comm: mount.nfs Not tainted 2.6.29-07100-g833bb30 #21 Sun Fire X4440
> >>> [ 63.202589] RIP: 0010:[<ffffffff802e0015>] [<ffffffff802e0015>] kfree+0x5a/0xcd
> >>> [ 63.202589] RSP: 0018:ffff882042ceb9f8 EFLAGS: 00010246
> >>> [ 63.202589] RAX: 0200000000000000 RBX: 0000000000000005 RCX: ffffffff80a7dc1f
> >>> [ 63.202589] RDX: ffffe20000000000 RSI: ffffc2000000f470 RDI: ffffe2001c018950
> >>> [ 63.202589] RBP: ffff882042ceba18 R08: 0000000000000000 R09: ffffffff811019c0
> >>> [ 63.202589] R10: 000000004262ce02 R11: ffff882042ceba18 R12: ffff880800706475
> >>> [ 63.202589] R13: ffff882042886000 R14: ffff882042cebbd8 R15: ffff882042cebbf0
> >>> [ 63.202589] FS: 00007fac729ed6f0(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000
> >>> [ 63.202589] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> >>> [ 63.202589] CR2: 00007fac72c12000 CR3: 0000001841cbb000 CR4: 00000000000006e0
> >>> [ 63.202589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> >>> [ 63.202589] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> >>> [ 63.202589] Process mount.nfs (pid: 10027, threadinfo ffff882042cea000, task ffff8820434dc290)
> >>> [ 63.202589] Stack:
> >>> [ 63.202589] ffff882042ceba18 000000004262ce02 0000000000000005 ffff882042886028
> >>> [ 63.202589] ffff882042ceba58 ffffffff80a7dc1f 000000004262ce02 ffff882042886000
> >>> [ 63.202589] 000000004262ce02 ffff882042886000 ffffffff80a7b4a6 ffff882042c9ee18
> >>> [ 63.202589] Call Trace:
> >>> [ 63.202589] [<ffffffff80a7dc1f>] xs_destroy+0x67/0xac
> >>> [ 63.202589] [<ffffffff80a7b4a6>] ? xprt_destroy+0x0/0xa7
> >>> [ 63.202589] [<ffffffff80a7b532>] xprt_destroy+0x8c/0xa7
> >>> [ 63.202589] [<ffffffff80a823b2>] ? put_rpccred+0x112/0x131
> >>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
> >>> [ 63.202589] [<ffffffff80a7a9a9>] ? rpc_free_client+0x0/0xf9
> >>> [ 63.202589] [<ffffffff80a7b490>] xprt_put+0x23/0x39
> >>> [ 63.202589] [<ffffffff80a7aa7a>] rpc_free_client+0xd1/0xf9
> >>> [ 63.202589] [<ffffffff80a83345>] ? unx_destroy+0x3c/0x57
> >>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
> >>> [ 63.202589] [<ffffffff80a7aaa2>] ? rpc_free_auth+0x0/0x69
> >>> [ 63.202589] [<ffffffff80a7aaf0>] rpc_free_auth+0x4e/0x69
> >>> [ 63.202589] [<ffffffff8025b827>] ? __wake_up+0x52/0x75
> >>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
> >>> [ 63.202589] [<ffffffff80a7a98e>] rpc_release_client+0x64/0x7f
> >>> [ 63.202589] [<ffffffff80a8061c>] ? rpc_put_task+0xb0/0xcb
> >>> [ 63.202589] [<ffffffff80a7abe0>] rpc_shutdown_client+0xd5/0xf8
> >>> [ 63.202589] [<ffffffff80a7a893>] ? rpc_call_sync+0x63/0x80
> >>> [ 63.202589] [<ffffffff803fc4ab>] nfs_mount+0x11f/0x1bf
> >>> [ 63.202589] [<ffffffff803f3036>] nfs_get_sb+0x4ac/0x82a
> >>> [ 63.202589] [<ffffffff802e8f24>] vfs_kern_mount+0x61/0xbf
> >>> [ 63.202589] [<ffffffff802fea1d>] ? get_fs_type+0x58/0xc5
> >>> [ 63.202589] [<ffffffff802e9015>] do_kern_mount+0x56/0x108
> >>> [ 63.202589] [<ffffffff80302195>] do_mount+0x729/0x788
> >>> [ 63.202589] [<ffffffff80300025>] ? copy_mount_options+0xdf/0x155
> >>> [ 63.202589] [<ffffffff8030228c>] sys_mount+0x98/0xf8
> >>> [ 63.202589] [<ffffffff80230d6b>] system_call_fastpath+0x16/0x1b
> >>> [ 63.202589] Code: 0c 48 ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48 8d 3c 10 48 8b 07 f6 c4 40 74 04 48 8b 7f 10 48 8b 07 84 c0 78 10 f6 c4 60 75 04 <0f> 0b eb fe e8 90 75 fd ff eb 4c 48 8b 4d 08 4c 8b 4f 10 9c 5b
> >>> [ 63.202589] RIP [<ffffffff802e0015>] kfree+0x5a/0xcd
> >>> [ 63.202589] RSP <ffff882042ceb9f8>
> >>> [ 63.524555] ---[ end trace cd0d38e02ad11d61 ]---
> >> Looks like a bogus pointer passed to kfree(). Lets cc some NFS folks.
> >
> > a quick look suggests that this fresh commit moves around a kfree()
> > and a destructor call:
> >
> > a67d18f: NFS: load the rpc/rdma transport module automatically
> >
> > the commit log does not mention why the kfree was moved which
> > suggests that it might have been done by mistake. (And it might be
> > totally correct even - i have not checked the details.) I'll send a
> > patch undoing those bits in the next mail.
> >
>
> got two kfree(string), could be some merging problem.

Yes - i just sent the fix for that, could you please check whether
it fixes the bug for you?

Ingo

2009-04-03 07:23:23

by Pekka Enberg

[permalink] [raw]
Subject: Re: [PATCH] nfs: fix nfs_parse_mount_options() double kfree()

On Fri, 2009-04-03 at 09:16 +0200, Ingo Molnar wrote:
> Impact: fix crash
>
> Yinghai Lu reported the following crash:
>
> > mpk12-3214-189-158:~ # sh x
> > [ 63.198629] ------------[ cut here ]------------
> > [ 63.202589] kernel BUG at mm/slub.c:2753!
> > [ 63.202589] invalid opcode: 0000 [#1] SMP
> > [ 63.202589] last sysfs file: /sys/devices/virtual/net/sit0/type
> > [ 63.202589] CPU 0
> > [ 63.202589] Modules linked in:
> > [ 63.202589] Pid: 10027, comm: mount.nfs Not tainted 2.6.29-07100-g833bb30 #21 Sun Fire X4440
> > [ 63.202589] RIP: 0010:[<ffffffff802e0015>] [<ffffffff802e0015>] kfree+0x5a/0xcd
> > [ 63.202589] RSP: 0018:ffff882042ceb9f8 EFLAGS: 00010246
> > [ 63.202589] RAX: 0200000000000000 RBX: 0000000000000005 RCX: ffffffff80a7dc1f
> > [ 63.202589] RDX: ffffe20000000000 RSI: ffffc2000000f470 RDI: ffffe2001c018950
> > [ 63.202589] RBP: ffff882042ceba18 R08: 0000000000000000 R09: ffffffff811019c0
> > [ 63.202589] R10: 000000004262ce02 R11: ffff882042ceba18 R12: ffff880800706475
> > [ 63.202589] R13: ffff882042886000 R14: ffff882042cebbd8 R15: ffff882042cebbf0
> > [ 63.202589] FS: 00007fac729ed6f0(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000
> > [ 63.202589] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [ 63.202589] CR2: 00007fac72c12000 CR3: 0000001841cbb000 CR4: 00000000000006e0
> > [ 63.202589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [ 63.202589] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [ 63.202589] Process mount.nfs (pid: 10027, threadinfo ffff882042cea000, task ffff8820434dc290)
> > [ 63.202589] Stack:
> > [ 63.202589] ffff882042ceba18 000000004262ce02 0000000000000005 ffff882042886028
> > [ 63.202589] ffff882042ceba58 ffffffff80a7dc1f 000000004262ce02 ffff882042886000
> > [ 63.202589] 000000004262ce02 ffff882042886000 ffffffff80a7b4a6 ffff882042c9ee18
> > [ 63.202589] Call Trace:
> > [ 63.202589] [<ffffffff80a7dc1f>] xs_destroy+0x67/0xac
> > [ 63.202589] [<ffffffff80a7b4a6>] ? xprt_destroy+0x0/0xa7
> > [ 63.202589] [<ffffffff80a7b532>] xprt_destroy+0x8c/0xa7
> > [ 63.202589] [<ffffffff80a823b2>] ? put_rpccred+0x112/0x131
> > [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
> > [ 63.202589] [<ffffffff80a7a9a9>] ? rpc_free_client+0x0/0xf9
> > [ 63.202589] [<ffffffff80a7b490>] xprt_put+0x23/0x39
> > [ 63.202589] [<ffffffff80a7aa7a>] rpc_free_client+0xd1/0xf9
> > [ 63.202589] [<ffffffff80a83345>] ? unx_destroy+0x3c/0x57
> > [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
> > [ 63.202589] [<ffffffff80a7aaa2>] ? rpc_free_auth+0x0/0x69
> > [ 63.202589] [<ffffffff80a7aaf0>] rpc_free_auth+0x4e/0x69
> > [ 63.202589] [<ffffffff8025b827>] ? __wake_up+0x52/0x75
> > [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
> > [ 63.202589] [<ffffffff80a7a98e>] rpc_release_client+0x64/0x7f
> > [ 63.202589] [<ffffffff80a8061c>] ? rpc_put_task+0xb0/0xcb
> > [ 63.202589] [<ffffffff80a7abe0>] rpc_shutdown_client+0xd5/0xf8
> > [ 63.202589] [<ffffffff80a7a893>] ? rpc_call_sync+0x63/0x80
> > [ 63.202589] [<ffffffff803fc4ab>] nfs_mount+0x11f/0x1bf
> > [ 63.202589] [<ffffffff803f3036>] nfs_get_sb+0x4ac/0x82a
> > [ 63.202589] [<ffffffff802e8f24>] vfs_kern_mount+0x61/0xbf
> > [ 63.202589] [<ffffffff802fea1d>] ? get_fs_type+0x58/0xc5
> > [ 63.202589] [<ffffffff802e9015>] do_kern_mount+0x56/0x108
> > [ 63.202589] [<ffffffff80302195>] do_mount+0x729/0x788
> > [ 63.202589] [<ffffffff80300025>] ? copy_mount_options+0xdf/0x155
> > [ 63.202589] [<ffffffff8030228c>] sys_mount+0x98/0xf8
> > [ 63.202589] [<ffffffff80230d6b>] system_call_fastpath+0x16/0x1b
> > [ 63.202589] Code: 0c 48 ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48 8d 3c 10 48 8b 07 f6 c4 40 74 04 48 8b 7f 10 48 8b 07 84 c0 78 10 f6 c4 60 75 04 <0f> 0b eb fe e8 90 75 fd ff eb 4c 48 8b 4d 08 4c 8b 4f 10 9c 5b
> > [ 63.202589] RIP [<ffffffff802e0015>] kfree+0x5a/0xcd
> > [ 63.202589] RSP <ffff882042ceb9f8>
> > [ 63.524555] ---[ end trace cd0d38e02ad11d61 ]---
>
> Pekka observed that a bogus pointer was passed to kfree().
>
> This commit:
>
> a67d18f: NFS: load the rpc/rdma transport module automatically
>
> Moved a kfree() of the options strings in nfs_parse_mount_options()
> inadvertently and introduced a double kfree(). Fix it.
>
> Reported-by: Yinghai Lu <[email protected]>
> Analyzed-by: Pekka Enberg <[email protected]>
> Signed-off-by: Ingo Molnar <[email protected]>

Looks good!

Reviewed-by: Pekka Enberg <[email protected]>

2009-04-03 07:50:46

by Yinghai Lu

[permalink] [raw]
Subject: Re: [PATCH] nfs: fix nfs_parse_mount_options() double kfree()

Pekka Enberg wrote:
> On Fri, 2009-04-03 at 09:16 +0200, Ingo Molnar wrote:
>> Impact: fix crash
>>
>> Yinghai Lu reported the following crash:
>>
>>> mpk12-3214-189-158:~ # sh x
>>> [ 63.198629] ------------[ cut here ]------------
>>> [ 63.202589] kernel BUG at mm/slub.c:2753!
>>> [ 63.202589] invalid opcode: 0000 [#1] SMP
>>> [ 63.202589] last sysfs file: /sys/devices/virtual/net/sit0/type
>>> [ 63.202589] CPU 0
>>> [ 63.202589] Modules linked in:
>>> [ 63.202589] Pid: 10027, comm: mount.nfs Not tainted 2.6.29-07100-g833bb30 #21 Sun Fire X4440
>>> [ 63.202589] RIP: 0010:[<ffffffff802e0015>] [<ffffffff802e0015>] kfree+0x5a/0xcd
>>> [ 63.202589] RSP: 0018:ffff882042ceb9f8 EFLAGS: 00010246
>>> [ 63.202589] RAX: 0200000000000000 RBX: 0000000000000005 RCX: ffffffff80a7dc1f
>>> [ 63.202589] RDX: ffffe20000000000 RSI: ffffc2000000f470 RDI: ffffe2001c018950
>>> [ 63.202589] RBP: ffff882042ceba18 R08: 0000000000000000 R09: ffffffff811019c0
>>> [ 63.202589] R10: 000000004262ce02 R11: ffff882042ceba18 R12: ffff880800706475
>>> [ 63.202589] R13: ffff882042886000 R14: ffff882042cebbd8 R15: ffff882042cebbf0
>>> [ 63.202589] FS: 00007fac729ed6f0(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000
>>> [ 63.202589] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> [ 63.202589] CR2: 00007fac72c12000 CR3: 0000001841cbb000 CR4: 00000000000006e0
>>> [ 63.202589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> [ 63.202589] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> [ 63.202589] Process mount.nfs (pid: 10027, threadinfo ffff882042cea000, task ffff8820434dc290)
>>> [ 63.202589] Stack:
>>> [ 63.202589] ffff882042ceba18 000000004262ce02 0000000000000005 ffff882042886028
>>> [ 63.202589] ffff882042ceba58 ffffffff80a7dc1f 000000004262ce02 ffff882042886000
>>> [ 63.202589] 000000004262ce02 ffff882042886000 ffffffff80a7b4a6 ffff882042c9ee18
>>> [ 63.202589] Call Trace:
>>> [ 63.202589] [<ffffffff80a7dc1f>] xs_destroy+0x67/0xac
>>> [ 63.202589] [<ffffffff80a7b4a6>] ? xprt_destroy+0x0/0xa7
>>> [ 63.202589] [<ffffffff80a7b532>] xprt_destroy+0x8c/0xa7
>>> [ 63.202589] [<ffffffff80a823b2>] ? put_rpccred+0x112/0x131
>>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
>>> [ 63.202589] [<ffffffff80a7a9a9>] ? rpc_free_client+0x0/0xf9
>>> [ 63.202589] [<ffffffff80a7b490>] xprt_put+0x23/0x39
>>> [ 63.202589] [<ffffffff80a7aa7a>] rpc_free_client+0xd1/0xf9
>>> [ 63.202589] [<ffffffff80a83345>] ? unx_destroy+0x3c/0x57
>>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
>>> [ 63.202589] [<ffffffff80a7aaa2>] ? rpc_free_auth+0x0/0x69
>>> [ 63.202589] [<ffffffff80a7aaf0>] rpc_free_auth+0x4e/0x69
>>> [ 63.202589] [<ffffffff8025b827>] ? __wake_up+0x52/0x75
>>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
>>> [ 63.202589] [<ffffffff80a7a98e>] rpc_release_client+0x64/0x7f
>>> [ 63.202589] [<ffffffff80a8061c>] ? rpc_put_task+0xb0/0xcb
>>> [ 63.202589] [<ffffffff80a7abe0>] rpc_shutdown_client+0xd5/0xf8
>>> [ 63.202589] [<ffffffff80a7a893>] ? rpc_call_sync+0x63/0x80
>>> [ 63.202589] [<ffffffff803fc4ab>] nfs_mount+0x11f/0x1bf
>>> [ 63.202589] [<ffffffff803f3036>] nfs_get_sb+0x4ac/0x82a
>>> [ 63.202589] [<ffffffff802e8f24>] vfs_kern_mount+0x61/0xbf
>>> [ 63.202589] [<ffffffff802fea1d>] ? get_fs_type+0x58/0xc5
>>> [ 63.202589] [<ffffffff802e9015>] do_kern_mount+0x56/0x108
>>> [ 63.202589] [<ffffffff80302195>] do_mount+0x729/0x788
>>> [ 63.202589] [<ffffffff80300025>] ? copy_mount_options+0xdf/0x155
>>> [ 63.202589] [<ffffffff8030228c>] sys_mount+0x98/0xf8
>>> [ 63.202589] [<ffffffff80230d6b>] system_call_fastpath+0x16/0x1b
>>> [ 63.202589] Code: 0c 48 ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48 8d 3c 10 48 8b 07 f6 c4 40 74 04 48 8b 7f 10 48 8b 07 84 c0 78 10 f6 c4 60 75 04 <0f> 0b eb fe e8 90 75 fd ff eb 4c 48 8b 4d 08 4c 8b 4f 10 9c 5b
>>> [ 63.202589] RIP [<ffffffff802e0015>] kfree+0x5a/0xcd
>>> [ 63.202589] RSP <ffff882042ceb9f8>
>>> [ 63.524555] ---[ end trace cd0d38e02ad11d61 ]---
>> Pekka observed that a bogus pointer was passed to kfree().
>>
>> This commit:
>>
>> a67d18f: NFS: load the rpc/rdma transport module automatically
>>
>> Moved a kfree() of the options strings in nfs_parse_mount_options()
>> inadvertently and introduced a double kfree(). Fix it.
>>
>> Reported-by: Yinghai Lu <[email protected]>
>> Analyzed-by: Pekka Enberg <[email protected]>
>> Signed-off-by: Ingo Molnar <[email protected]>
>
> Looks good!
>
> Reviewed-by: Pekka Enberg <[email protected]>

thanks. it fixs the problem.

YH

2009-04-03 12:16:24

by Tom Talpey

[permalink] [raw]
Subject: Re: [PATCH] nfs: fix nfs_parse_mount_options() double kfree()

At 03:16 AM 4/3/2009, Ingo Molnar wrote:
>
>Impact: fix crash
>
>Yinghai Lu reported the following crash:

This has to have been a merge problem, I moved a single kfree()
down a few lines in order to use the pointer in a second context.
With the moved line, it had been running successfully in Trond's
tree and linux-next for several weeks.

Thanks for fixing!

Tom.

>
>> mpk12-3214-189-158:~ # sh x
>> [ 63.198629] ------------[ cut here ]------------
>> [ 63.202589] kernel BUG at mm/slub.c:2753!
>> [ 63.202589] invalid opcode: 0000 [#1] SMP
>> [ 63.202589] last sysfs file: /sys/devices/virtual/net/sit0/type
>> [ 63.202589] CPU 0
>> [ 63.202589] Modules linked in:
>> [ 63.202589] Pid: 10027, comm: mount.nfs Not tainted
>2.6.29-07100-g833bb30 #21 Sun Fire X4440
>> [ 63.202589] RIP: 0010:[<ffffffff802e0015>] [<ffffffff802e0015>]
>kfree+0x5a/0xcd
>> [ 63.202589] RSP: 0018:ffff882042ceb9f8 EFLAGS: 00010246
>> [ 63.202589] RAX: 0200000000000000 RBX: 0000000000000005 RCX:
>ffffffff80a7dc1f
>> [ 63.202589] RDX: ffffe20000000000 RSI: ffffc2000000f470 RDI:
>ffffe2001c018950
>> [ 63.202589] RBP: ffff882042ceba18 R08: 0000000000000000 R09:
>ffffffff811019c0
>> [ 63.202589] R10: 000000004262ce02 R11: ffff882042ceba18 R12:
>ffff880800706475
>> [ 63.202589] R13: ffff882042886000 R14: ffff882042cebbd8 R15:
>ffff882042cebbf0
>> [ 63.202589] FS: 00007fac729ed6f0(0000) GS:ffffc20000000000(0000)
>knlGS:0000000000000000
>> [ 63.202589] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [ 63.202589] CR2: 00007fac72c12000 CR3: 0000001841cbb000 CR4:
>00000000000006e0
>> [ 63.202589] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>0000000000000000
>> [ 63.202589] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>0000000000000400
>> [ 63.202589] Process mount.nfs (pid: 10027, threadinfo
>ffff882042cea000, task ffff8820434dc290)
>> [ 63.202589] Stack:
>> [ 63.202589] ffff882042ceba18 000000004262ce02 0000000000000005
>ffff882042886028
>> [ 63.202589] ffff882042ceba58 ffffffff80a7dc1f 000000004262ce02
>ffff882042886000
>> [ 63.202589] 000000004262ce02 ffff882042886000 ffffffff80a7b4a6
>ffff882042c9ee18
>> [ 63.202589] Call Trace:
>> [ 63.202589] [<ffffffff80a7dc1f>] xs_destroy+0x67/0xac
>> [ 63.202589] [<ffffffff80a7b4a6>] ? xprt_destroy+0x0/0xa7
>> [ 63.202589] [<ffffffff80a7b532>] xprt_destroy+0x8c/0xa7
>> [ 63.202589] [<ffffffff80a823b2>] ? put_rpccred+0x112/0x131
>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
>> [ 63.202589] [<ffffffff80a7a9a9>] ? rpc_free_client+0x0/0xf9
>> [ 63.202589] [<ffffffff80a7b490>] xprt_put+0x23/0x39
>> [ 63.202589] [<ffffffff80a7aa7a>] rpc_free_client+0xd1/0xf9
>> [ 63.202589] [<ffffffff80a83345>] ? unx_destroy+0x3c/0x57
>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
>> [ 63.202589] [<ffffffff80a7aaa2>] ? rpc_free_auth+0x0/0x69
>> [ 63.202589] [<ffffffff80a7aaf0>] rpc_free_auth+0x4e/0x69
>> [ 63.202589] [<ffffffff8025b827>] ? __wake_up+0x52/0x75
>> [ 63.202589] [<ffffffff8051cdd5>] kref_put+0x65/0x87
>> [ 63.202589] [<ffffffff80a7a98e>] rpc_release_client+0x64/0x7f
>> [ 63.202589] [<ffffffff80a8061c>] ? rpc_put_task+0xb0/0xcb
>> [ 63.202589] [<ffffffff80a7abe0>] rpc_shutdown_client+0xd5/0xf8
>> [ 63.202589] [<ffffffff80a7a893>] ? rpc_call_sync+0x63/0x80
>> [ 63.202589] [<ffffffff803fc4ab>] nfs_mount+0x11f/0x1bf
>> [ 63.202589] [<ffffffff803f3036>] nfs_get_sb+0x4ac/0x82a
>> [ 63.202589] [<ffffffff802e8f24>] vfs_kern_mount+0x61/0xbf
>> [ 63.202589] [<ffffffff802fea1d>] ? get_fs_type+0x58/0xc5
>> [ 63.202589] [<ffffffff802e9015>] do_kern_mount+0x56/0x108
>> [ 63.202589] [<ffffffff80302195>] do_mount+0x729/0x788
>> [ 63.202589] [<ffffffff80300025>] ? copy_mount_options+0xdf/0x155
>> [ 63.202589] [<ffffffff8030228c>] sys_mount+0x98/0xf8
>> [ 63.202589] [<ffffffff80230d6b>] system_call_fastpath+0x16/0x1b
>> [ 63.202589] Code: 0c 48 ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48
>8d 3c 10 48 8b 07 f6 c4 40 74 04 48 8b 7f 10 48 8b 07 84 c0 78 10 f6
>c4 60 75 04 <0f> 0b eb fe e8 90 75 fd ff eb 4c 48 8b 4d 08 4c 8b 4f 10 9c 5b
>> [ 63.202589] RIP [<ffffffff802e0015>] kfree+0x5a/0xcd
>> [ 63.202589] RSP <ffff882042ceb9f8>
>> [ 63.524555] ---[ end trace cd0d38e02ad11d61 ]---
>
>Pekka observed that a bogus pointer was passed to kfree().
>
>This commit:
>
> a67d18f: NFS: load the rpc/rdma transport module automatically
>
>Moved a kfree() of the options strings in nfs_parse_mount_options()
>inadvertently and introduced a double kfree(). Fix it.
>
>Reported-by: Yinghai Lu <[email protected]>
>Analyzed-by: Pekka Enberg <[email protected]>
>Signed-off-by: Ingo Molnar <[email protected]>
>---
> fs/nfs/super.c | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
>diff --git a/fs/nfs/super.c b/fs/nfs/super.c
>index 0942fcb..4e7f879 100644
>--- a/fs/nfs/super.c
>+++ b/fs/nfs/super.c
>@@ -1191,7 +1191,6 @@ static int nfs_parse_mount_options(char *raw,
> goto out_nomem;
> token = match_token(string,
> nfs_xprt_protocol_tokens, args);
>- kfree(string);
>
> switch (token) {
> case Opt_xprt_udp:
>@@ -1221,6 +1220,7 @@ static int nfs_parse_mount_options(char *raw,
> goto out_nomem;
> token = match_token(string,
> nfs_xprt_protocol_tokens, args);
>+ kfree(string);
>
> switch (token) {
> case Opt_xprt_udp:

2009-04-03 13:08:27

by Ingo Molnar

[permalink] [raw]
Subject: Re: [PATCH] nfs: fix nfs_parse_mount_options() double kfree()


* Tom Talpey <[email protected]> wrote:

> At 03:16 AM 4/3/2009, Ingo Molnar wrote:
> >
> >Impact: fix crash
> >
> >Yinghai Lu reported the following crash:
>
> This has to have been a merge problem, I moved a single kfree()
> down a few lines in order to use the pointer in a second context.
> With the moved line, it had been running successfully in Trond's
> tree and linux-next for several weeks.

Maybe the tree got rebased and that introduced this merge artifact?

> Thanks for fixing!

You are welcome!

Ingo

2009-04-03 13:26:12

by Myklebust, Trond

[permalink] [raw]
Subject: Re: [PATCH] nfs: fix nfs_parse_mount_options() double kfree()

On Fri, 2009-04-03 at 15:07 +0200, Ingo Molnar wrote:
> * Tom Talpey <[email protected]> wrote:
>
> > At 03:16 AM 4/3/2009, Ingo Molnar wrote:
> > >
> > >Impact: fix crash
> > >
> > >Yinghai Lu reported the following crash:
> >
> > This has to have been a merge problem, I moved a single kfree()
> > down a few lines in order to use the pointer in a second context.
> > With the moved line, it had been running successfully in Trond's
> > tree and linux-next for several weeks.
>
> Maybe the tree got rebased and that introduced this merge artifact?

No, but the linux-next tree also contained cachefs, which hasn't been
merged yet. Let's keep an eye out when it does...

--
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
http://www.netapp.com

2009-04-03 18:52:54

by Yinghai Lu

[permalink] [raw]
Subject: Re: [PATCH] nfs: fix nfs_parse_mount_options() double kfree()

Trond Myklebust wrote:
> On Fri, 2009-04-03 at 15:07 +0200, Ingo Molnar wrote:
>> * Tom Talpey <[email protected]> wrote:
>>
>>> At 03:16 AM 4/3/2009, Ingo Molnar wrote:
>>>> Impact: fix crash
>>>>
>>>> Yinghai Lu reported the following crash:
>>> This has to have been a merge problem, I moved a single kfree()
>>> down a few lines in order to use the pointer in a second context.
>>> With the moved line, it had been running successfully in Trond's
>>> tree and linux-next for several weeks.
>> Maybe the tree got rebased and that introduced this merge artifact?
>
> No, but the linux-next tree also contained cachefs, which hasn't been
> merged yet. Let's keep an eye out when it does...

could be merging sequence.

YH