Hello,
with recent kernels I'm seeing this BUG - triggered by racoon - at boot:
NET: Registered protocol family 15
------------[ cut here ]------------
kernel BUG at /home/kronos/src/linux-2.6.git/include/net/netns/generic.h:43!
invalid opcode: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/kernel/uevent_seqnum
CPU 1
Pid: 1941, comm: racoon Not tainted 2.6.33-rc5-00271-gbe8cde8-dirty #238 F3Sa /F3Sa
RIP: 0010:[<ffffffffa03035be>] [<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]
RSP: 0018:ffff88013ddebe98 EFLAGS: 00010246
RAX: ffff88013f894480 RBX: ffff88013f44de40 RCX: 0000000000000000
RDX: 0000000000000002 RSI: ffff88013f44de40 RDI: 0000000000000001
RBP: ffff88013ddebeb8 R08: ffff88013ddea000 R09: dead000000200200
R10: dead000000100100 R11: ffff88013ddebd80 R12: 0000000000000000
R13: ffffffff81771860 R14: 0000000000000002 R15: 0000000000000000
FS: 00007f02c3e1f710(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000794f38 CR3: 000000013d98d000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process racoon (pid: 1941, threadinfo ffff88013ddea000, task ffff88013dffd880)
Stack:
000000000000000f ffff88013f44de40 0000000000000003 ffffffffa0304a80
<0> ffff88013ddebf28 ffffffff81220802 ffff88013ddebee8 00007f02c3a22e80
<0> ffffffff00000001 ffff88013ddebf60 ffffffff81771860 0000000200000001
Call Trace:
[<ffffffff81220802>] __sock_create+0x242/0x3dd
[<ffffffff812209e9>] sock_create+0x2b/0x2d
[<ffffffff81220b91>] sys_socket+0x26/0x57
[<ffffffff8129fe0f>] ? page_fault+0x1f/0x30
[<ffffffff81002a2b>] system_call_fastpath+0x16/0x1b
Code: 49 89 fd 41 54 bf 01 00 00 00 53 44 8b 25 27 19 00 00 48 89 f3 e8 4f f1 f9 e0 49 8b 85 48 08 00 00 45 85 e4 74 05 44 3b 20 76 04 <0f> 0b eb fe 4d 63 e4 4e 8b 64 e0 10 bf 01 00 00 00 e8 aa f0 f9
RIP [<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]
RSP <ffff88013ddebe98>
---[ end trace 78cabe73779ec9df ]---
note: racoon[1941] exited with preempt_count 1
It looks like a bug I reported a while ago[1] and which was fixed by Eric. The
fix is still in place, but the bug has resurfaced recently, probably in .33.
It's quiet elusive, i.e. it happens maybe once every 5 boots; restarting racoon
doesn't seem to trigger it...
Luca
[1] http://bugzilla.kernel.org/show_bug.cgi?id=13838
On Fri, Jan 29, 2010 at 11:48 AM, Luca Tettamanti <[email protected]> wrote:
> with recent kernels I'm seeing this BUG - triggered by racoon - at boot:
>
> NET: Registered protocol family 15
> ------------[ cut here ]------------
> kernel BUG at /home/kronos/src/linux-2.6.git/include/net/netns/generic.h:43!
> invalid opcode: 0000 [#1] PREEMPT SMP
> last sysfs file: /sys/kernel/uevent_seqnum
> CPU 1
> Pid: 1941, comm: racoon Not tainted 2.6.33-rc5-00271-gbe8cde8-dirty #238 F3Sa ? ? ?/F3Sa
> RIP: 0010:[<ffffffffa03035be>] ?[<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]
Does it triggers after succesfull boot if you do
rmmod af_key; modprobe af_key
a couple of times?
Post .config, just in case.
Le vendredi 29 janvier 2010 à 12:17 +0200, Alexey Dobriyan a écrit :
> On Fri, Jan 29, 2010 at 11:48 AM, Luca Tettamanti <[email protected]> wrote:
> > with recent kernels I'm seeing this BUG - triggered by racoon - at boot:
> >
> > NET: Registered protocol family 15
> > ------------[ cut here ]------------
> > kernel BUG at /home/kronos/src/linux-2.6.git/include/net/netns/generic.h:43!
> > invalid opcode: 0000 [#1] PREEMPT SMP
> > last sysfs file: /sys/kernel/uevent_seqnum
> > CPU 1
> > Pid: 1941, comm: racoon Not tainted 2.6.33-rc5-00271-gbe8cde8-dirty #238 F3Sa /F3Sa
> > RIP: 0010:[<ffffffffa03035be>] [<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]
>
> Does it triggers after succesfull boot if you do
>
> rmmod af_key; modprobe af_key
>
> a couple of times?
>
> Post .config, just in case.
I am looking at ipsec_pfkey_init()
We call sock_register(&pfkey_family_ops) before pfkey_net_id being
initialized (by the call to register_pernet_subsys(&pfkey_net_ops);
As soon as sock_register(&pfkey_family_ops) is done, another thread can
open a socket and call pfkey_create() -> crash
We should change order of initializations somehow
Le vendredi 29 janvier 2010 à 16:22 +0100, Eric Dumazet a écrit :
> Le vendredi 29 janvier 2010 à 12:17 +0200, Alexey Dobriyan a écrit :
> > On Fri, Jan 29, 2010 at 11:48 AM, Luca Tettamanti <[email protected]> wrote:
> > > with recent kernels I'm seeing this BUG - triggered by racoon - at boot:
> > >
> > > NET: Registered protocol family 15
> > > ------------[ cut here ]------------
> > > kernel BUG at /home/kronos/src/linux-2.6.git/include/net/netns/generic.h:43!
> > > invalid opcode: 0000 [#1] PREEMPT SMP
> > > last sysfs file: /sys/kernel/uevent_seqnum
> > > CPU 1
> > > Pid: 1941, comm: racoon Not tainted 2.6.33-rc5-00271-gbe8cde8-dirty #238 F3Sa /F3Sa
> > > RIP: 0010:[<ffffffffa03035be>] [<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]
> >
> > Does it triggers after succesfull boot if you do
> >
> > rmmod af_key; modprobe af_key
> >
> > a couple of times?
> >
> > Post .config, just in case.
>
> I am looking at ipsec_pfkey_init()
>
> We call sock_register(&pfkey_family_ops) before pfkey_net_id being
> initialized (by the call to register_pernet_subsys(&pfkey_net_ops);
>
> As soon as sock_register(&pfkey_family_ops) is done, another thread can
> open a socket and call pfkey_create() -> crash
>
> We should change order of initializations somehow
>
Something like this (compiled but not tested) patch ?
Should probably be sent to stable team...
[PATCH] xfrm: Change initializations order in ipsec_pfkey_init()
Before allowing other threads to create PF_KEY sockets, we must make
sure pfkey_net_id is properly initialized.
That means calling register_pernet_subsys(&pfkey_net_ops) before
sock_register(&pfkey_family_ops)
Reported-by: Luca Tettamanti <[email protected]>
Signed-off-by: Eric Dumazet <[email protected]>
---
net/key/af_key.c | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/net/key/af_key.c b/net/key/af_key.c
index 76fa6fe..e399ddf 100644
--- a/net/key/af_key.c
+++ b/net/key/af_key.c
@@ -3807,21 +3807,24 @@ static int __init ipsec_pfkey_init(void)
if (err != 0)
goto out;
- err = sock_register(&pfkey_family_ops);
- if (err != 0)
- goto out_unregister_key_proto;
err = xfrm_register_km(&pfkeyv2_mgr);
if (err != 0)
- goto out_sock_unregister;
+ goto out_unregister_key_proto;
+
err = register_pernet_subsys(&pfkey_net_ops);
if (err != 0)
goto out_xfrm_unregister_km;
+
+ err = sock_register(&pfkey_family_ops);
+ if (err != 0)
+ goto out_unregister_pernet;
out:
return err;
+
+out_unregister_pernet:
+ unregister_pernet_subsys(&pfkey_net_ops);
out_xfrm_unregister_km:
xfrm_unregister_km(&pfkeyv2_mgr);
-out_sock_unregister:
- sock_unregister(PF_KEY);
out_unregister_key_proto:
proto_unregister(&key_proto);
goto out;
On Fri, Jan 29, 2010 at 05:33:26PM +0100, Eric Dumazet wrote:
> @@ -3807,21 +3807,24 @@ static int __init ipsec_pfkey_init(void)
> if (err != 0)
> goto out;
>
> - err = sock_register(&pfkey_family_ops);
> - if (err != 0)
> - goto out_unregister_key_proto;
> err = xfrm_register_km(&pfkeyv2_mgr);
> if (err != 0)
> - goto out_sock_unregister;
> + goto out_unregister_key_proto;
> +
> err = register_pernet_subsys(&pfkey_net_ops);
> if (err != 0)
> goto out_xfrm_unregister_km;
> +
> + err = sock_register(&pfkey_family_ops);
> + if (err != 0)
> + goto out_unregister_pernet;
> out:
> return err;
> +
> +out_unregister_pernet:
> + unregister_pernet_subsys(&pfkey_net_ops);
> out_xfrm_unregister_km:
> xfrm_unregister_km(&pfkeyv2_mgr);
> -out_sock_unregister:
> - sock_unregister(PF_KEY);
> out_unregister_key_proto:
> proto_unregister(&key_proto);
> goto out;
ACK analysis, except this is not enough.
Here is patch which survived netns start/stop/modprobe/rmmod cycles.
Alexey, who still doesn't get why bug reproduces so easily for bug reporter.
Luca, please confirm.
Audit for the rest of the modules pending.
[PATCH] af_key: fix netns ops ordering on module load/unload
1. After sock_register() returns, it's possible to create sockets,
even if module still not initialized fully (blame generic module code
for that!)
2. Consequently, pfkey_create() can be called with pfkey_net_id still not
initialized which will BUG_ON in net_generic():
kernel BUG at include/net/netns/generic.h:43!
3. During netns shutdown, netns ops should be unregistered after
key manager unregistered because key manager calls can be triggered
from xfrm_user module:
general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
pfkey_broadcast+0x111/0x210 [af_key]
pfkey_send_notify+0x16a/0x300 [af_key]
km_state_notify+0x41/0x70
xfrm_flush_sa+0x75/0x90 [xfrm_user]
4. Unregister netns ops after socket ops just in case and for symmetry.
Reported by Luca Tettamanti.
Signed-off-by: Alexey Dobriyan <[email protected]>
---
net/key/af_key.c | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
--- a/net/key/af_key.c
+++ b/net/key/af_key.c
@@ -3794,9 +3794,9 @@ static struct pernet_operations pfkey_net_ops = {
static void __exit ipsec_pfkey_exit(void)
{
- unregister_pernet_subsys(&pfkey_net_ops);
xfrm_unregister_km(&pfkeyv2_mgr);
sock_unregister(PF_KEY);
+ unregister_pernet_subsys(&pfkey_net_ops);
proto_unregister(&key_proto);
}
@@ -3807,21 +3807,22 @@ static int __init ipsec_pfkey_init(void)
if (err != 0)
goto out;
- err = sock_register(&pfkey_family_ops);
+ err = register_pernet_subsys(&pfkey_net_ops);
if (err != 0)
goto out_unregister_key_proto;
+ err = sock_register(&pfkey_family_ops);
+ if (err != 0)
+ goto out_unregister_pernet;
err = xfrm_register_km(&pfkeyv2_mgr);
if (err != 0)
goto out_sock_unregister;
- err = register_pernet_subsys(&pfkey_net_ops);
- if (err != 0)
- goto out_xfrm_unregister_km;
out:
return err;
-out_xfrm_unregister_km:
- xfrm_unregister_km(&pfkeyv2_mgr);
+
out_sock_unregister:
sock_unregister(PF_KEY);
+out_unregister_pernet:
+ unregister_pernet_subsys(&pfkey_net_ops);
out_unregister_key_proto:
proto_unregister(&key_proto);
goto out;
On Fri, Jan 29, 2010 at 11:17 AM, Alexey Dobriyan <[email protected]> wrote:
> On Fri, Jan 29, 2010 at 11:48 AM, Luca Tettamanti <[email protected]> wrote:
>> with recent kernels I'm seeing this BUG - triggered by racoon - at boot:
>>
>> NET: Registered protocol family 15
>> ------------[ cut here ]------------
>> kernel BUG at /home/kronos/src/linux-2.6.git/include/net/netns/generic.h:43!
>> invalid opcode: 0000 [#1] PREEMPT SMP
>> last sysfs file: /sys/kernel/uevent_seqnum
>> CPU 1
>> Pid: 1941, comm: racoon Not tainted 2.6.33-rc5-00271-gbe8cde8-dirty #238 F3Sa /F3Sa
>> RIP: 0010:[<ffffffffa03035be>] [<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]
>
> Does it triggers after succesfull boot if you do
>
> rmmod af_key; modprobe af_key
>
> a couple of times?
No. Neither does:
racoon stop; rmmod af_key; racoon start
It might be connected to the fact that I'm using Debian startpar to
run multiple init scripts in parallel.
I'm now testing Eric's patch and I haven't seen the BUG after a few reboots.
> Post .config, just in case.
Attached.
Luca
On Sat, Jan 30, 2010 at 1:53 PM, Alexey Dobriyan <[email protected]> wrote:
> On Fri, Jan 29, 2010 at 05:33:26PM +0100, Eric Dumazet wrote:
>> @@ -3807,21 +3807,24 @@ static int __init ipsec_pfkey_init(void)
>> if (err != 0)
>> goto out;
>>
>> - err = sock_register(&pfkey_family_ops);
>> - if (err != 0)
>> - goto out_unregister_key_proto;
>> err = xfrm_register_km(&pfkeyv2_mgr);
>> if (err != 0)
>> - goto out_sock_unregister;
>> + goto out_unregister_key_proto;
>> +
>> err = register_pernet_subsys(&pfkey_net_ops);
>> if (err != 0)
>> goto out_xfrm_unregister_km;
>> +
>> + err = sock_register(&pfkey_family_ops);
>> + if (err != 0)
>> + goto out_unregister_pernet;
>> out:
>> return err;
>> +
>> +out_unregister_pernet:
>> + unregister_pernet_subsys(&pfkey_net_ops);
>> out_xfrm_unregister_km:
>> xfrm_unregister_km(&pfkeyv2_mgr);
>> -out_sock_unregister:
>> - sock_unregister(PF_KEY);
>> out_unregister_key_proto:
>> proto_unregister(&key_proto);
>> goto out;
>
> ACK analysis, except this is not enough.
>
> Here is patch which survived netns start/stop/modprobe/rmmod cycles.
>
> Alexey, who still doesn't get why bug reproduces so easily for bug reporter.
>
> Luca, please confirm.
Seems to work fine.
> [PATCH] af_key: fix netns ops ordering on module load/unload
>
> 1. After sock_register() returns, it's possible to create sockets,
> even if module still not initialized fully (blame generic module code
> for that!)
> 2. Consequently, pfkey_create() can be called with pfkey_net_id still not
> initialized which will BUG_ON in net_generic():
> kernel BUG at include/net/netns/generic.h:43!
> 3. During netns shutdown, netns ops should be unregistered after
> key manager unregistered because key manager calls can be triggered
> from xfrm_user module:
>
> general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> pfkey_broadcast+0x111/0x210 [af_key]
> pfkey_send_notify+0x16a/0x300 [af_key]
> km_state_notify+0x41/0x70
> xfrm_flush_sa+0x75/0x90 [xfrm_user]
> 4. Unregister netns ops after socket ops just in case and for symmetry.
>
> Reported by Luca Tettamanti.
>
> Signed-off-by: Alexey Dobriyan <[email protected]>
Tested-by: Luca Tettamanti <[email protected]>
thanks,
Luca
Le lundi 01 février 2010 à 14:50 +0100, Luca Tettamanti a écrit :
> On Sat, Jan 30, 2010 at 1:53 PM, Alexey Dobriyan <[email protected]> wrote:
> > [PATCH] af_key: fix netns ops ordering on module load/unload
> >
> > 1. After sock_register() returns, it's possible to create sockets,
> > even if module still not initialized fully (blame generic module code
> > for that!)
> > 2. Consequently, pfkey_create() can be called with pfkey_net_id still not
> > initialized which will BUG_ON in net_generic():
> > kernel BUG at include/net/netns/generic.h:43!
> > 3. During netns shutdown, netns ops should be unregistered after
> > key manager unregistered because key manager calls can be triggered
> > from xfrm_user module:
> >
> > general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> > pfkey_broadcast+0x111/0x210 [af_key]
> > pfkey_send_notify+0x16a/0x300 [af_key]
> > km_state_notify+0x41/0x70
> > xfrm_flush_sa+0x75/0x90 [xfrm_user]
> > 4. Unregister netns ops after socket ops just in case and for symmetry.
> >
> > Reported by Luca Tettamanti.
> >
> > Signed-off-by: Alexey Dobriyan <[email protected]>
>
> Tested-by: Luca Tettamanti <[email protected]>
Signed-off-by: Eric Dumazet <[email protected]>
From: Eric Dumazet <[email protected]>
Date: Mon, 01 Feb 2010 14:56:30 +0100
> Le lundi 01 f?vrier 2010 ? 14:50 +0100, Luca Tettamanti a ?crit :
>> On Sat, Jan 30, 2010 at 1:53 PM, Alexey Dobriyan <[email protected]> wrote:
>
>> > [PATCH] af_key: fix netns ops ordering on module load/unload
>> >
>> > 1. After sock_register() returns, it's possible to create sockets,
>> > even if module still not initialized fully (blame generic module code
>> > for that!)
>> > 2. Consequently, pfkey_create() can be called with pfkey_net_id still not
>> > initialized which will BUG_ON in net_generic():
>> > kernel BUG at include/net/netns/generic.h:43!
>> > 3. During netns shutdown, netns ops should be unregistered after
>> > key manager unregistered because key manager calls can be triggered
>> > from xfrm_user module:
>> >
>> > general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
>> > pfkey_broadcast+0x111/0x210 [af_key]
>> > pfkey_send_notify+0x16a/0x300 [af_key]
>> > km_state_notify+0x41/0x70
>> > xfrm_flush_sa+0x75/0x90 [xfrm_user]
>> > 4. Unregister netns ops after socket ops just in case and for symmetry.
>> >
>> > Reported by Luca Tettamanti.
>> >
>> > Signed-off-by: Alexey Dobriyan <[email protected]>
>>
>> Tested-by: Luca Tettamanti <[email protected]>
>
> Signed-off-by: Eric Dumazet <[email protected]>
Applied, thanks everyone!