2012-05-24 01:41:27

by kun.jiang

[permalink] [raw]
Subject: [PATCH V2] ipv4: fix the rcu race between free_fib_info and ip_route_output_slow

From: Yanmin Zhang <[email protected]>

We hit a kernel OOPS.

<3>[23898.789643] BUG: sleeping function called from invalid context at
/data/buildbot/workdir/ics/hardware/intel/linux-2.6/arch/x86/mm/fault.c:1103
<3>[23898.862215] in_atomic(): 0, irqs_disabled(): 0, pid: 10526, name:
Thread-6683
<4>[23898.967805] HSU serial 0000:00:05.1: 0000:00:05.2:HSU serial prevented me
to suspend...
<4>[23899.258526] Pid: 10526, comm: Thread-6683 Tainted: G W
3.0.8-137685-ge7742f9 #1
<4>[23899.357404] HSU serial 0000:00:05.1: 0000:00:05.2:HSU serial prevented me
to suspend...
<4>[23899.904225] Call Trace:
<4>[23899.989209] [<c1227f50>] ? pgtable_bad+0x130/0x130
<4>[23900.000416] [<c1238c2a>] __might_sleep+0x10a/0x110
<4>[23900.007357] [<c1228021>] do_page_fault+0xd1/0x3c0
<4>[23900.013764] [<c18e9ba9>] ? restore_all+0xf/0xf
<4>[23900.024024] [<c17c007b>] ? napi_complete+0x8b/0x690
<4>[23900.029297] [<c1227f50>] ? pgtable_bad+0x130/0x130
<4>[23900.123739] [<c1227f50>] ? pgtable_bad+0x130/0x130
<4>[23900.128955] [<c18ea0c3>] error_code+0x5f/0x64
<4>[23900.133466] [<c1227f50>] ? pgtable_bad+0x130/0x130
<4>[23900.138450] [<c17f6298>] ? __ip_route_output_key+0x698/0x7c0
<4>[23900.144312] [<c17f5f8d>] ? __ip_route_output_key+0x38d/0x7c0
<4>[23900.150730] [<c17f63df>] ip_route_output_flow+0x1f/0x60
<4>[23900.156261] [<c181de58>] ip4_datagram_connect+0x188/0x2b0
<4>[23900.161960] [<c18e981f>] ? _raw_spin_unlock_bh+0x1f/0x30
<4>[23900.167834] [<c18298d6>] inet_dgram_connect+0x36/0x80
<4>[23900.173224] [<c14f9e88>] ? _copy_from_user+0x48/0x140
<4>[23900.178817] [<c17ab9da>] sys_connect+0x9a/0xd0
<4>[23900.183538] [<c132e93c>] ? alloc_file+0xdc/0x240
<4>[23900.189111] [<c123925d>] ? sub_preempt_count+0x3d/0x50

Function free_fib_info resets nexthop_nh->nh_dev to NULL before releasing
fi. Other cpu might be accessing fi. Fixing it by delaying the releasing.

With the patch, we ran MTBF testing on Android mobile for 12 hours
and didn't trigger the issue.

Thank Eric for very detailed review/checking the issue.

Signed-off-by: Yanmin Zhang <[email protected]>
Signed-off-by: Kun Jiang <[email protected]>
---
net/ipv4/fib_semantics.c | 12 ++++++------
1 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c
index 5063fa3..8861f91 100644
--- a/net/ipv4/fib_semantics.c
+++ b/net/ipv4/fib_semantics.c
@@ -145,6 +145,12 @@ static void free_fib_info_rcu(struct rcu_head *head)
{
struct fib_info *fi = container_of(head, struct fib_info, rcu);

+ change_nexthops(fi) {
+ if (nexthop_nh->nh_dev)
+ dev_put(nexthop_nh->nh_dev);
+ } endfor_nexthops(fi);
+
+ release_net(fi->fib_net);
if (fi->fib_metrics != (u32 *) dst_default_metrics)
kfree(fi->fib_metrics);
kfree(fi);
@@ -156,13 +162,7 @@ void free_fib_info(struct fib_info *fi)
pr_warn("Freeing alive fib_info %p\n", fi);
return;
}
- change_nexthops(fi) {
- if (nexthop_nh->nh_dev)
- dev_put(nexthop_nh->nh_dev);
- nexthop_nh->nh_dev = NULL;
- } endfor_nexthops(fi);
fib_info_cnt--;
- release_net(fi->fib_net);
call_rcu(&fi->rcu, free_fib_info_rcu);
}

--
1.7.1


2012-05-24 02:42:48

by Eric Dumazet

[permalink] [raw]
Subject: Re: [PATCH V2] ipv4: fix the rcu race between free_fib_info and ip_route_output_slow

On Thu, 2012-05-24 at 09:39 +0800, kun.jiang wrote:
> From: Yanmin Zhang <[email protected]>
>
> We hit a kernel OOPS.
>
...
>
> Function free_fib_info resets nexthop_nh->nh_dev to NULL before releasing
> fi. Other cpu might be accessing fi. Fixing it by delaying the releasing.
>
> With the patch, we ran MTBF testing on Android mobile for 12 hours
> and didn't trigger the issue.
>
> Thank Eric for very detailed review/checking the issue.
>
> Signed-off-by: Yanmin Zhang <[email protected]>
> Signed-off-by: Kun Jiang <[email protected]>
> ---
> net/ipv4/fib_semantics.c | 12 ++++++------
> 1 files changed, 6 insertions(+), 6 deletions(-)

Acked-by: Eric Dumazet <[email protected]>

Thanks !

2012-05-24 04:27:35

by David Miller

[permalink] [raw]
Subject: Re: [PATCH V2] ipv4: fix the rcu race between free_fib_info and ip_route_output_slow

From: Eric Dumazet <[email protected]>
Date: Thu, 24 May 2012 04:42:42 +0200

> On Thu, 2012-05-24 at 09:39 +0800, kun.jiang wrote:
>> From: Yanmin Zhang <[email protected]>
>>
>> We hit a kernel OOPS.
>>
> ...
>>
>> Function free_fib_info resets nexthop_nh->nh_dev to NULL before releasing
>> fi. Other cpu might be accessing fi. Fixing it by delaying the releasing.
>>
>> With the patch, we ran MTBF testing on Android mobile for 12 hours
>> and didn't trigger the issue.
>>
>> Thank Eric for very detailed review/checking the issue.
>>
>> Signed-off-by: Yanmin Zhang <[email protected]>
>> Signed-off-by: Kun Jiang <[email protected]>
>> ---
>> net/ipv4/fib_semantics.c | 12 ++++++------
>> 1 files changed, 6 insertions(+), 6 deletions(-)
>
> Acked-by: Eric Dumazet <[email protected]>

Applied, thanks everyone.