LinuxLists.cc - Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

[ 5955.558356] BUG: unable to handle kernel paging request at virtual address 182b10f7
[ 5955.558362] printing eip:
[ 5955.558363] 182b10f7
[ 5955.558365] *pde = 00000000
[ 5955.558367] Oops: 0000 [#1]
[ 5955.558369] PREEMPT
[ 5955.558370] Modules linked in: eeprom i2c_viapro i2c_core pcspkr k8temp hwmon eth1394
[ 5955.558377] CPU: 0
[ 5955.558378] EIP: 0060:[<182b10f7>] Not tainted VLI
[ 5955.558379] EFLAGS: 00010202 (2.6.21.1 #1)
[ 5955.558382] EIP is at 0x182b10f7
[ 5955.558385] eax: 80c7dcc1 ebx: f786bc00 ecx: 182b10f7 edx: 00000002
[ 5955.558388] esi: f7836200 edi: c2669f6c ebp: f780c88f esp: c2669f34
[ 5955.558390] ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068
[ 5955.558393] Process pdflush (pid: 192, ti=c2669000 task=c268da90 task.ti=c2669000)
[ 5955.558395] Stack: c108a454 f7836284 00158443 0000024a f7836200 f783623c c2669f6c c147774c
[ 5955.558401] c108aa8b c2669fb8 00158925 0000bc01 c1053030 c1052c44 00000000 00000000
[ 5955.558406] c2669f94 000001b6 0000024a 00000000 00000000 00000000 00000000 00000025
[ 5955.558411] Call Trace:
[ 5955.558413] [<c108a454>] sync_sb_inodes+0x74/0x280
[ 5955.558419] [<c108aa8b>] writeback_inodes+0xab/0x110
[ 5955.558423] [<c1053030>] pdflush+0x0/0x220
[ 5955.558426] [<c1052c44>] wb_kupdate+0x74/0xe0
[ 5955.558430] [<c1053144>] pdflush+0x114/0x220
[ 5955.558433] [<c1052bd0>] wb_kupdate+0x0/0xe0
[ 5955.558436] [<c102fdd8>] kthread+0xa8/0xe0
[ 5955.558439] [<c102fd30>] kthread+0x0/0xe0
[ 5955.558442] [<c1004bff>] kernel_thread_helper+0x7/0x18
[ 5955.558446] =======================
[ 5955.558447] Code: Bad EIP value.
[ 5955.558449] EIP: [<182b10f7>] 0x182b10f7 SS:ESP 0068:c2669f34
[ 5955.558455] note: pdflush[192] exited with preempt_count 1

Attachments:

ooops.20070516-01 (1.72 kB)
signature.asc (252.00 B)
OpenPGP digital signature Download all attachments

2007-05-16 01:58:33

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

On Wed, 16 May 2007 09:24:54 +0900 Clemens Schwaighofer <[email protected]> wrote:

> I have re-occurring oopses and panics in those above kernels. The error
> is always the same. I have the last Kernel Panic as a picture here:
>
> http://dev.tequila.jp/clemens/R0010172.JPG
>
> The oops have the same error style like this Panic. I tried to capture
> one, but right after copying it into vim, I got a Panic. So next time I
> try to.
>
> I think it started with 2.6.19.2, I cannot remember I had any of those
> problems before. The box can work fine for about a week or more, or it
> looks up several times a day. I run a memtest for 10 h, but I had no errors.

shrink_dcache_memory->...sysfs_d_iput->BUG

BUG_ON(sd->s_dentry != dentry);

a number of people have hit that, on and off.

We were close to having a fix, I think, but then we decided that great
chunks of sysfs needed rewriting and I believe that we believe that this
great rewrite will fix this bug.

But alas, it's all too late for 2.6.22.

How frequently do you see these failures? If it's repeatable with any reliability
then it'd be great if you could test a patchset for us. It's at:

http://userweb.kernel.org/~akpm/cs.gz

that's a single patch against 2.6.21-rc1, containing the following patches, which
are from the forthcoming 2.6.21-rc1-mm1 lineup:

origin
gregkh-driver-uio
gregkh-driver-uio-documentation
gregkh-driver-uio-hilscher-cif-card-driver
gregkh-driver-idr-fix-obscure-bug-in-allocation-path
gregkh-driver-idr-separate-out-idr_mark_full
gregkh-driver-ida-implement-idr-based-id-allocator
gregkh-driver-sysfs-move-release_sysfs_dirent-to-dirc
gregkh-driver-sysfs-allocate-inode-number-using-ida
gregkh-driver-sysfs-make-sysfs_put-ignore-null-sd
gregkh-driver-sysfs-fix-error-handling-in-binattr-write
gregkh-driver-sysfs-flatten-cleanup-paths-in-sysfs_add_link-and-create_dir
gregkh-driver-sysfs-flatten-and-fix-sysfs_rename_dir-error-handling
gregkh-driver-sysfs-consolidate-sysfs_dirent-creation-functions
gregkh-driver-sysfs-add-sysfs_dirent-s_parent
gregkh-driver-sysfs-add-sysfs_dirent-s_name
gregkh-driver-sysfs-make-sysfs_dirent-s_element-a-union
gregkh-driver-sysfs-implement-kobj_sysfs_assoc_lock
gregkh-driver-sysfs-reimplement-symlink-using-sysfs_dirent-tree
gregkh-driver-sysfs-implement-bin_buffer
gregkh-driver-sysfs-implement-sysfs_dirent-active-reference-and-immediate-disconnect
gregkh-driver-sysfs-kill-attribute-file-orphaning
gregkh-driver-sysfs-separate-out-sysfs_attach_dentry
gregkh-driver-sysfs-reimplement-syfs_drop_dentry
gregkh-driver-sysfs-kill-unnecessary-attribute-owner
gregkh-driver-driver-core-make-devt_attr-and-uevent_attr-static
gregkh-driver-put_device-might_sleep
gregkh-driver-kobject-warn
gregkh-driver-warn-when-statically-allocated-kobjects-are-used
gregkh-driver-nozomi
fix-gregkh-driver-sysfs-fix-error-handling-in-binattr-write

Thanks.

2007-05-16 02:02:41

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

On 05/16/2007 10:53 AM, Andrew Morton wrote:
>
>> I think it started with 2.6.19.2, I cannot remember I had any of those
>> problems before. The box can work fine for about a week or more, or it
>> looks up several times a day. I run a memtest for 10 h, but I had no errors.
>
> shrink_dcache_memory->...sysfs_d_iput->BUG
>
> BUG_ON(sd->s_dentry != dentry);
>
> a number of people have hit that, on and off.
>
> We were close to having a fix, I think, but then we decided that great
> chunks of sysfs needed rewriting and I believe that we believe that this
> great rewrite will fix this bug.
>
> But alas, it's all too late for 2.6.22.

Well, there is always hope for 2.6.23 :)

> How frequently do you see these failures? If it's repeatable with any reliability
> then it'd be great if you could test a patchset for us. It's at:
>
> http://userweb.kernel.org/~akpm/cs.gz
>
> that's a single patch against 2.6.21-rc1, containing the following patches, which
> are from the forthcoming 2.6.21-rc1-mm1 lineup:

I get this very frequently recently. I just got hit by another PANIC
which was probably the same issue. I will get this patch and try it out
and see if it helps for me.

--
[ Clemens Schwaighofer -----=====:::::~ ]
[ TEQUILA\ Japan IT Group ]
[ 6-17-2 Ginza Chuo-ku, Tokyo 104-8167, JAPAN ]
[ Tel: +81-(0)3-3545-7703 Fax: +81-(0)3-3545-7343 ]
[ http://www.tequila.co.jp ]

Attachments:

signature.asc (252.00 B)
OpenPGP digital signature

2007-05-16 02:46:17

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

On 05/16/2007 10:53 AM, Andrew Morton wrote:

> How frequently do you see these failures? If it's repeatable with any reliability
> then it'd be great if you could test a patchset for us. It's at:
>
> http://userweb.kernel.org/~akpm/cs.gz
>
> that's a single patch against 2.6.21-rc1, containing the following patches, which
> are from the forthcoming 2.6.21-rc1-mm1 lineup:

(and those above are 2.6.22-rc1 of course)

well, I tried to apply those patches and when I compile I get this error:

CC net/ipv6/exthdrs.o
net/ipv6/exthdrs.c: In function ‘ipv6_rthdr_rcv’:
net/ipv6/exthdrs.c:390: error: ‘struct sk_buff’ has no member named ‘h’
net/ipv6/exthdrs.c:391: error: ‘struct sk_buff’ has no member named ‘h’
net/ipv6/exthdrs.c:391: error: ‘struct sk_buff’ has no member named ‘h’
net/ipv6/exthdrs.c:398: error: ‘struct sk_buff’ has no member named ‘h’
make[2]: *** [net/ipv6/exthdrs.o] Error 1
make[1]: *** [net/ipv6] Error 2
make: *** [net] Error 2

It's probably totally unrelated but sadly a showstopper for testing the
new sysfs patches,

--
[ Clemens Schwaighofer -----=====:::::~ ]
[ TEQUILA\ Japan IT Group ]
[ 6-17-2 Ginza Chuo-ku, Tokyo 104-8167, JAPAN ]
[ Tel: +81-(0)3-3545-7703 Fax: +81-(0)3-3545-7343 ]
[ http://www.tequila.co.jp ]

Attachments:

signature.asc (252.00 B)
OpenPGP digital signature

2007-05-16 03:23:23

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

On Wed, 16 May 2007 11:46:00 +0900 Clemens Schwaighofer <[email protected]> wrote:

> On 05/16/2007 10:53 AM, Andrew Morton wrote:
>
> > How frequently do you see these failures? If it's repeatable with any reliability
> > then it'd be great if you could test a patchset for us. It's at:
> >
> > http://userweb.kernel.org/~akpm/cs.gz
> >
> > that's a single patch against 2.6.21-rc1, containing the following patches, which
> > are from the forthcoming 2.6.21-rc1-mm1 lineup:
>
> (and those above are 2.6.22-rc1 of course)
>
> well, I tried to apply those patches and when I compile I get this error:
>
> CC net/ipv6/exthdrs.o
> net/ipv6/exthdrs.c: In function ‘ipv6_rthdr_rcv’:
> net/ipv6/exthdrs.c:390: error: ‘struct sk_buff’ has no member named ‘h’
> net/ipv6/exthdrs.c:391: error: ‘struct sk_buff’ has no member named ‘h’
> net/ipv6/exthdrs.c:391: error: ‘struct sk_buff’ has no member named ‘h’
> net/ipv6/exthdrs.c:398: error: ‘struct sk_buff’ has no member named ‘h’
> make[2]: *** [net/ipv6/exthdrs.o] Error 1
> make[1]: *** [net/ipv6] Error 2
> make: *** [net] Error 2
>
> It's probably totally unrelated but sadly a showstopper for testing the
> new sysfs patches,

oh crap, sorry. Oh well.

Please test 2.6.22-rc1-mm1 which has all the fixes and is a damn fine
kernel. It's at
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.22-rc1/2.6.22-rc1-mm1/

The core patch is at ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.22-rc1/2.6.22-rc1-mm1/2.6.22-rc1-mm1.gz, and it's against 2.6.22-rc1.

Thanks.

2007-05-16 11:05:44

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

Hello,

Andrew Morton wrote:
> On Wed, 16 May 2007 09:24:54 +0900 Clemens Schwaighofer <[email protected]> wrote:
>
>> I have re-occurring oopses and panics in those above kernels. The error
>> is always the same. I have the last Kernel Panic as a picture here:
>>
>> http://dev.tequila.jp/clemens/R0010172.JPG
>>
>> The oops have the same error style like this Panic. I tried to capture
>> one, but right after copying it into vim, I got a Panic. So next time I
>> try to.
>>
>> I think it started with 2.6.19.2, I cannot remember I had any of those
>> problems before. The box can work fine for about a week or more, or it
>> looks up several times a day. I run a memtest for 10 h, but I had no errors.
>
> shrink_dcache_memory->...sysfs_d_iput->BUG
>
> BUG_ON(sd->s_dentry != dentry);
>
> a number of people have hit that, on and off.

Yeah, I've been seeing that one. It should have been fixed with the big
fat patchset.

> We were close to having a fix, I think, but then we decided that great
> chunks of sysfs needed rewriting and I believe that we believe that this
> great rewrite will fix this bug.

How were we gonna fix it? If it isn't too complex, I can cook up a
patch for -stable series. The safest approach I can think of is making
dentries for attributes unreclaimable but those are made reclaimable for
good reasons. :-(

Thanks.

--
tejun

2007-05-16 15:34:31

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

On Wed, 16 May 2007 13:05:19 +0200 Tejun Heo <[email protected]> wrote:

> Hello,
>
> Andrew Morton wrote:
> > On Wed, 16 May 2007 09:24:54 +0900 Clemens Schwaighofer <[email protected]> wrote:
> >
> >> I have re-occurring oopses and panics in those above kernels. The error
> >> is always the same. I have the last Kernel Panic as a picture here:
> >>
> >> http://dev.tequila.jp/clemens/R0010172.JPG
> >>
> >> The oops have the same error style like this Panic. I tried to capture
> >> one, but right after copying it into vim, I got a Panic. So next time I
> >> try to.
> >>
> >> I think it started with 2.6.19.2, I cannot remember I had any of those
> >> problems before. The box can work fine for about a week or more, or it
> >> looks up several times a day. I run a memtest for 10 h, but I had no errors.
> >
> > shrink_dcache_memory->...sysfs_d_iput->BUG
> >
> > BUG_ON(sd->s_dentry != dentry);
> >
> > a number of people have hit that, on and off.
>
> Yeah, I've been seeing that one. It should have been fixed with the big
> fat patchset.

Great - fingers crossed.

> > We were close to having a fix, I think, but then we decided that great
> > chunks of sysfs needed rewriting and I believe that we believe that this
> > great rewrite will fix this bug.
>
> How were we gonna fix it? If it isn't too complex, I can cook up a
> patch for -stable series.

Do we actually understand the causes?

> The safest approach I can think of is making
> dentries for attributes unreclaimable but those are made reclaimable for
> good reasons. :-(

Yeah, that was the google workaround. It's OK unless you happen to have
thousands of disks on an ia32 box.

2007-05-16 15:41:44

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

Andrew Morton wrote:
>>> a number of people have hit that, on and off.
>> Yeah, I've been seeing that one. It should have been fixed with the big
>> fat patchset.
>
> Great - fingers crossed.
>
>>> We were close to having a fix, I think, but then we decided that great
>>> chunks of sysfs needed rewriting and I believe that we believe that this
>>> great rewrite will fix this bug.
>> How were we gonna fix it? If it isn't too complex, I can cook up a
>> patch for -stable series.
>
> Do we actually understand the causes?

Yeah, I think I do. Basically, the problem is that on-demand attach and
reclamation update sd->s_dentry but accesses to it aren't synchronized
properly. In the big fat patchset, first I tried to fix it by removing
sd->s_dentry completely which didn't work because of shadow nodes, so
the second try was to fix the synchronization which is in -mm now.

>> The safest approach I can think of is making
>> dentries for attributes unreclaimable but those are made reclaimable for
>> good reasons. :-(
>
> Yeah, that was the google workaround. It's OK unless you happen to have
> thousands of disks on an ia32 box.

I see. I thought there was different approach on fixing the problem.
I'll try to backport the synchronization fix but am afraid it can be too
risky for -stable. If it seems too risky, I'll send a patch to disable
reclamation.

--
tejun

2007-05-16 16:08:18

by Chuck Ebbert

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

Tejun Heo wrote:
>>> The safest approach I can think of is making
>>> dentries for attributes unreclaimable but those are made reclaimable for
>>> good reasons. :-(
>> Yeah, that was the google workaround. It's OK unless you happen to have
>> thousands of disks on an ia32 box.
>
> I see. I thought there was different approach on fixing the problem.
> I'll try to backport the synchronization fix but am afraid it can be too
> risky for -stable. If it seems too risky, I'll send a patch to disable
> reclamation.
>

Realistically, how can disabling the reclamation be worse than what's
there now?

2007-05-16 16:18:27

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

On Wed, 16 May 2007 17:40:53 +0200 Tejun Heo <[email protected]> wrote:

> I see. I thought there was different approach on fixing the problem.
> I'll try to backport the synchronization fix but am afraid it can be too
> risky for -stable. If it seems too risky, I'll send a patch to disable
> reclamation.

OK. Sad. Maybe we add /proc/sys/fs/i-have-lots-of-disks-and-dont-mind-if-it-oopses
to enable the old behaviour.

2007-05-16 18:31:20

[permalink] [raw]

Subject: [PATCH -stable] sysfs: disable reclamation by default

sd->s_dentry updates made by dentry/inode reclamation are racy and can
lead to BUG() or oops. This is already fixed in -mm and the fix is
scheduled to be merged into upstream for 2.6.23 but the fix
reimplements sysfs dentry dropping and is too risky for -stable
kernels.

This is an interim solution for -stable kernels. sysfs reclamation is
disabled by default and can be enabled by using sysfs.enable_reclaim
kernel parameter. Note that dentries are still created on demand, so
attribute and symlinks nodes aren't allocated on creation. They're
allocated on first lookup and deallocated when the sysfs node is
removed.

Signed-off-by: Tejun Heo <[email protected]>
---

Okay, I wasn't sure about the backport, so this is what I've come up
with. It's ugly but, as Chuck pointed out, anything is better than
oopsing.

Clemens, can you please test this one too?

If anyone has any better idea, feel free to override this one.

Thanks.

Documentation/kernel-parameters.txt | 8 ++++++++
fs/sysfs/Makefile | 3 ++-
fs/sysfs/dir.c | 11 +++++++++++
fs/sysfs/inode.c | 3 +++
fs/sysfs/sysfs.h | 1 +
5 files changed, 25 insertions(+), 1 deletion(-)

Index: tree0/fs/sysfs/dir.c
===================================================================
--- tree0.orig/fs/sysfs/dir.c
+++ tree0/fs/sysfs/dir.c
@@ -14,6 +14,13 @@

DECLARE_RWSEM(sysfs_rename_sem);

+/*
+ * sysfs dentry/inode reclaming has race conditions around
+ * sd->s_dentry. Disable it by default.
+ */
+int sysfs_enable_reclaim;
+module_param_named(enable_reclaim, sysfs_enable_reclaim, bool, 0444);
+
static void sysfs_d_iput(struct dentry * dentry, struct inode * inode)
{
struct sysfs_dirent * sd = dentry->d_fsdata;
@@ -289,6 +296,10 @@ static struct dentry * sysfs_lookup(stru
err = sysfs_attach_link(sd, dentry);
else
err = sysfs_attach_attr(sd, dentry);
+
+ if (err == 0 && !sysfs_enable_reclaim)
+ dget(dentry);
+
break;
}
}
Index: tree0/fs/sysfs/inode.c
===================================================================
--- tree0.orig/fs/sysfs/inode.c
+++ tree0/fs/sysfs/inode.c
@@ -266,6 +266,9 @@ void sysfs_drop_dentry(struct sysfs_dire
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
}
+
+ if (!sysfs_enable_reclaim)
+ dput(dentry);
}
}

Index: tree0/fs/sysfs/Makefile
===================================================================
--- tree0.orig/fs/sysfs/Makefile
+++ tree0/fs/sysfs/Makefile
@@ -2,5 +2,6 @@
# Makefile for the sysfs virtual filesystem
#

-obj-y := inode.o file.o dir.o symlink.o mount.o bin.o \
+obj-y := sysfs.o
+sysfs-objs := inode.o file.o dir.o symlink.o mount.o bin.o \
group.o
Index: tree0/Documentation/kernel-parameters.txt
===================================================================
--- tree0.orig/Documentation/kernel-parameters.txt
+++ tree0/Documentation/kernel-parameters.txt
@@ -1721,6 +1721,14 @@ and is between 256 and 4096 characters.
sym53c416= [HW,SCSI]
See header of drivers/scsi/sym53c416.c.

+ sysfs.enable_reclaim
+ Enable dentry and inode reclamation for
+ attribute and symlink nodes. This is buggy
+ and can cause oops or trigger BUG. Do not
+ enable unless you know what you're doing.
+ This kernel parameter will be removed after
+ reclamation is fixed.
+
sysrq_always_enabled
[KNL]
Ignore sysrq setting - this boot parameter will
Index: tree0/fs/sysfs/sysfs.h
===================================================================
--- tree0.orig/fs/sysfs/sysfs.h
+++ tree0/fs/sysfs/sysfs.h
@@ -12,6 +12,7 @@ struct sysfs_dirent {

extern struct vfsmount * sysfs_mount;
extern struct kmem_cache *sysfs_dir_cachep;
+extern int sysfs_enable_reclaim;

extern void sysfs_delete_inode(struct inode *inode);
extern struct inode * sysfs_new_inode(mode_t mode, struct sysfs_dirent *);

2007-05-17 12:06:19

by Greg KH

[permalink] [raw]

Subject: Re: [PATCH -stable] sysfs: disable reclamation by default

On Wed, May 16, 2007 at 08:31:00PM +0200, Tejun Heo wrote:
> sd->s_dentry updates made by dentry/inode reclamation are racy and can
> lead to BUG() or oops. This is already fixed in -mm and the fix is
> scheduled to be merged into upstream for 2.6.23 but the fix
> reimplements sysfs dentry dropping and is too risky for -stable
> kernels.
>
> This is an interim solution for -stable kernels. sysfs reclamation is
> disabled by default and can be enabled by using sysfs.enable_reclaim
> kernel parameter. Note that dentries are still created on demand, so
> attribute and symlinks nodes aren't allocated on creation. They're
> allocated on first lookup and deallocated when the sysfs node is
> removed.

Ick, this is going to kill memory on big boxes (s390 and others) and I
don't really want to apply this it if at all possible.

Maneesh, any other thoughts?

thanks,

greg k-h

2007-05-17 17:39:21

[permalink] [raw]

Subject: Re: [PATCH -stable] sysfs: disable reclamation by default

On Thu, May 17, 2007 at 05:04:23AM -0700, Greg KH wrote:
> On Wed, May 16, 2007 at 08:31:00PM +0200, Tejun Heo wrote:
> > sd->s_dentry updates made by dentry/inode reclamation are racy and can
> > lead to BUG() or oops. This is already fixed in -mm and the fix is
> > scheduled to be merged into upstream for 2.6.23 but the fix
> > reimplements sysfs dentry dropping and is too risky for -stable
> > kernels.
> >

But was the synchronization fix tested by people facing the race? I still
don't understand the racy code path. The last google problem I saw had
s_dentry field as NULL.

> > This is an interim solution for -stable kernels. sysfs reclamation is
> > disabled by default and can be enabled by using sysfs.enable_reclaim
> > kernel parameter. Note that dentries are still created on demand, so
> > attribute and symlinks nodes aren't allocated on creation. They're
> > allocated on first lookup and deallocated when the sysfs node is
> > removed.
>
> Ick, this is going to kill memory on big boxes (s390 and others) and I
> don't really want to apply this it if at all possible.
>
At least not make it default. This might create boot issues with these
boxes.

> Maneesh, any other thoughts?
>
I actually wanted to investigate this oops but left it considering the
rework being done by Tejun. If this still make sense we can have some
more debug code stuffed there or get a crashdump (kdump) to get better
understanding of the race.

Thanks
Maneesh

--
Maneesh Soni
Linux Technology Center,
IBM India Systems and Technology Lab,
Bangalore, India

2007-05-17 17:50:01

[permalink] [raw]

Subject: Re: [PATCH -stable] sysfs: disable reclamation by default

Maneesh Soni wrote:
> On Thu, May 17, 2007 at 05:04:23AM -0700, Greg KH wrote:
>> On Wed, May 16, 2007 at 08:31:00PM +0200, Tejun Heo wrote:
>>> sd->s_dentry updates made by dentry/inode reclamation are racy and can
>>> lead to BUG() or oops. This is already fixed in -mm and the fix is
>>> scheduled to be merged into upstream for 2.6.23 but the fix
>>> reimplements sysfs dentry dropping and is too risky for -stable
>>> kernels.
>>>
>
> But was the synchronization fix tested by people facing the race? I still
> don't understand the racy code path. The last google problem I saw had
> s_dentry field as NULL.

Please take a look at the following message.

http://article.gmane.org/gmane.linux.kernel/521729

I could reproduce both races on my test machine fairly reliable with
parallel find, cat, mount/mount while repeatedly ins/rmmoding a libata
driver.

>>> This is an interim solution for -stable kernels. sysfs reclamation is
>>> disabled by default and can be enabled by using sysfs.enable_reclaim
>>> kernel parameter. Note that dentries are still created on demand, so
>>> attribute and symlinks nodes aren't allocated on creation. They're
>>> allocated on first lookup and deallocated when the sysfs node is
>>> removed.
>> Ick, this is going to kill memory on big boxes (s390 and others) and I
>> don't really want to apply this it if at all possible.
>>
> At least not make it default. This might create boot issues with these
> boxes.

Which makes oopsing the default. Fun! :-)

>> Maneesh, any other thoughts?
>>
> I actually wanted to investigate this oops but left it considering the
> rework being done by Tejun. If this still make sense we can have some
> more debug code stuffed there or get a crashdump (kdump) to get better
> understanding of the race.

The above message contains analysis of both races. I just ported the
fixes. I have a different test machine now and can't reproduce the
races with this one yet so I couldn't verify whether the patches
actually fix the problem. I'll post the patches anyway. If anyone can
reproduce these races, please verify the posted patches fix the problem.

Thanks.

--
tejun

2007-05-17 17:53:19

[permalink] [raw]

Subject: [PATCH 1/2] sysfs: fix condition check in sysfs_drop_dentry()

The condition check doesn't make much sense as it basically always
succeeds. This causes NULL dereferencing on certain cases. It seems
that parentheses are put in the wrong place. Fix it.

DON'T APPLY JUST YET
---

Maneesh, is this correct?

fs/sysfs/inode.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Index: work/fs/sysfs/inode.c
===================================================================
--- work.orig/fs/sysfs/inode.c
+++ work/fs/sysfs/inode.c
@@ -250,7 +250,7 @@ void sysfs_drop_dentry(struct sysfs_dire
if (dentry) {
spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
- if (!(d_unhashed(dentry) && dentry->d_inode)) {
+ if (!d_unhashed(dentry) && dentry->d_inode) {
inode = dentry->d_inode;
spin_lock(&inode->i_lock);
__iget(inode);

2007-05-17 17:59:58

[permalink] [raw]

Subject: [PATCH 2/2] sysfs: fix race condition around sd->s_dentry

Allowing attribute and symlink dentries to be reclaimed means
sd->s_dentry can change dynamically. However, updates to the field
are unsynchronized leading to race conditions. This patch adds
sysfs_lock and use it to synchronize updates to sd->s_dentry.

Due to the locking around ->d_iput, the check in sysfs_drop_dentry()
is complex. sysfs_lock only protect sd->s_dentry pointer itself. The
validity of the dentry is protected by dcache_lock, so whether dentry
is alive or not can only be tested while holding both locks.

This is minimal backport of sysfs_drop_dentry() rewrite in devel
branch.

DONT APPLY JUST YET
---

Okay, here's minimal backport of synchronization fix. kobj
directories are left alone. sysfs_drop_dentry() modifications can be
made more efficient but are left that way on purpose.

fs/sysfs/dir.c | 29 ++++++++++++++++++++++++++---
fs/sysfs/inode.c | 28 ++++++++++++++++++++++++----
fs/sysfs/sysfs.h | 1 +
3 files changed, 51 insertions(+), 7 deletions(-)

Index: work/fs/sysfs/dir.c
===================================================================
--- work.orig/fs/sysfs/dir.c
+++ work/fs/sysfs/dir.c
@@ -13,14 +13,26 @@
#include "sysfs.h"

DECLARE_RWSEM(sysfs_rename_sem);
+spinlock_t sysfs_lock = SPIN_LOCK_UNLOCKED;

static void sysfs_d_iput(struct dentry * dentry, struct inode * inode)
{
struct sysfs_dirent * sd = dentry->d_fsdata;

if (sd) {
- BUG_ON(sd->s_dentry != dentry);
- sd->s_dentry = NULL;
+ /* sd->s_dentry is protected with sysfs_lock. This
+ * allows sysfs_drop_dentry() to dereference it.
+ */
+ spin_lock(&sysfs_lock);
+
+ /* The dentry might have been deleted or another
+ * lookup could have happened updating sd->s_dentry to
+ * point the new dentry. Ignore if it isn't pointing
+ * to this dentry.
+ */
+ if (sd->s_dentry == dentry)
+ sd->s_dentry = NULL;
+ spin_unlock(&sysfs_lock);
sysfs_put(sd);
}
iput(inode);
@@ -238,7 +250,10 @@ static int sysfs_attach_attr(struct sysf
}

dentry->d_fsdata = sysfs_get(sd);
+ /* protect sd->s_dentry against sysfs_d_iput */
+ spin_lock(&sysfs_lock);
sd->s_dentry = dentry;
+ spin_unlock(&sysfs_lock);
error = sysfs_create(dentry, (attr->mode & S_IALLUGO) | S_IFREG, init);
if (error) {
sysfs_put(sd);
@@ -260,7 +275,10 @@ static int sysfs_attach_link(struct sysf
int err = 0;

dentry->d_fsdata = sysfs_get(sd);
+ /* protect sd->s_dentry against sysfs_d_iput */
+ spin_lock(&sysfs_lock);
sd->s_dentry = dentry;
+ spin_unlock(&sysfs_lock);
err = sysfs_create(dentry, S_IFLNK|S_IRWXUGO, init_symlink);
if (!err) {
dentry->d_op = &sysfs_dentry_ops;
@@ -329,6 +347,7 @@ void sysfs_remove_subdir(struct dentry *

static void __sysfs_remove_dir(struct dentry *dentry)
{
+ LIST_HEAD(removed);
struct sysfs_dirent * parent_sd;
struct sysfs_dirent * sd, * tmp;

@@ -342,11 +361,15 @@ static void __sysfs_remove_dir(struct de
list_for_each_entry_safe(sd, tmp, &parent_sd->s_children, s_sibling) {
if (!sd->s_element || !(sd->s_type & SYSFS_NOT_PINNED))
continue;
+ list_move(&sd->s_sibling, &removed);
+ }
+ mutex_unlock(&dentry->d_inode->i_mutex);
+
+ list_for_each_entry_safe(sd, tmp, &removed, s_sibling) {
list_del_init(&sd->s_sibling);
sysfs_drop_dentry(sd, dentry);
sysfs_put(sd);
}
- mutex_unlock(&dentry->d_inode->i_mutex);

remove_dir(dentry);
/**
Index: work/fs/sysfs/inode.c
===================================================================
--- work.orig/fs/sysfs/inode.c
+++ work/fs/sysfs/inode.c
@@ -244,9 +244,23 @@ static inline void orphan_all_buffers(st
*/
void sysfs_drop_dentry(struct sysfs_dirent * sd, struct dentry * parent)
{
- struct dentry * dentry = sd->s_dentry;
+ struct dentry *dentry = NULL;
struct inode *inode;

+ /* We're not holding a reference to ->s_dentry dentry but the
+ * field will stay valid as long as sysfs_lock is held.
+ */
+ spin_lock(&sysfs_lock);
+ spin_lock(&dcache_lock);
+
+ /* dget dentry if it's still alive */
+ if (sd->s_dentry && sd->s_dentry->d_inode)
+ dentry = dget_locked(sd->s_dentry);
+
+ spin_unlock(&dcache_lock);
+ spin_unlock(&sysfs_lock);
+
+ /* drop dentry */
if (dentry) {
spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
@@ -266,6 +280,8 @@ void sysfs_drop_dentry(struct sysfs_dire
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
}
+
+ dput(dentry);
}
}

@@ -289,13 +305,17 @@ int sysfs_hash_and_remove(struct dentry
continue;
if (!strcmp(sysfs_get_name(sd), name)) {
list_del_init(&sd->s_sibling);
- sysfs_drop_dentry(sd, dir);
- sysfs_put(sd);
found = 1;
break;
}
}
mutex_unlock(&dir->d_inode->i_mutex);

- return found ? 0 : -ENOENT;
+ if (!found)
+ return -ENOENT;
+
+ sysfs_drop_dentry(sd, dir);
+ sysfs_put(sd);
+
+ return 0;
}
Index: work/fs/sysfs/sysfs.h
===================================================================
--- work.orig/fs/sysfs/sysfs.h
+++ work/fs/sysfs/sysfs.h
@@ -32,6 +32,7 @@ extern const unsigned char * sysfs_get_n
extern void sysfs_drop_dentry(struct sysfs_dirent *sd, struct dentry *parent);
extern int sysfs_setattr(struct dentry *dentry, struct iattr *iattr);

+extern spinlock_t sysfs_lock;
extern struct rw_semaphore sysfs_rename_sem;
extern struct super_block * sysfs_sb;
extern const struct file_operations sysfs_dir_operations;

2007-05-17 18:16:31

[permalink] [raw]

Subject: [PATCH 2/2] sysfs: fix race condition around sd->s_dentry, take#2

Allowing attribute and symlink dentries to be reclaimed means
sd->s_dentry can change dynamically. However, updates to the field
are unsynchronized leading to race conditions. This patch adds
sysfs_lock and use it to synchronize updates to sd->s_dentry.

Due to the locking around ->d_iput, the check in sysfs_drop_dentry()
is complex. sysfs_lock only protect sd->s_dentry pointer itself. The
validity of the dentry is protected by dcache_lock, so whether dentry
is alive or not can only be tested while holding both locks.

This is minimal backport of sysfs_drop_dentry() rewrite in devel
branch.

DONT APPLY JUST YET
---
Moving sysfs_drop_dentry() and sysfs_put() calls out of mutex isn't
necessary, so this is the minimal one but there shouldn't be
any difference functionality-wise.

fs/sysfs/dir.c | 22 ++++++++++++++++++++--
fs/sysfs/inode.c | 18 +++++++++++++++++-
fs/sysfs/sysfs.h | 1 +
3 files changed, 38 insertions(+), 3 deletions(-)

Index: work/fs/sysfs/dir.c
===================================================================
--- work.orig/fs/sysfs/dir.c
+++ work/fs/sysfs/dir.c
@@ -13,14 +13,26 @@
#include "sysfs.h"

DECLARE_RWSEM(sysfs_rename_sem);
+spinlock_t sysfs_lock = SPIN_LOCK_UNLOCKED;

static void sysfs_d_iput(struct dentry * dentry, struct inode * inode)
{
struct sysfs_dirent * sd = dentry->d_fsdata;

if (sd) {
- BUG_ON(sd->s_dentry != dentry);
- sd->s_dentry = NULL;
+ /* sd->s_dentry is protected with sysfs_lock. This
+ * allows sysfs_drop_dentry() to dereference it.
+ */
+ spin_lock(&sysfs_lock);
+
+ /* The dentry might have been deleted or another
+ * lookup could have happened updating sd->s_dentry to
+ * point the new dentry. Ignore if it isn't pointing
+ * to this dentry.
+ */
+ if (sd->s_dentry == dentry)
+ sd->s_dentry = NULL;
+ spin_unlock(&sysfs_lock);
sysfs_put(sd);
}
iput(inode);
@@ -238,7 +250,10 @@ static int sysfs_attach_attr(struct sysf
}

dentry->d_fsdata = sysfs_get(sd);
+ /* protect sd->s_dentry against sysfs_d_iput */
+ spin_lock(&sysfs_lock);
sd->s_dentry = dentry;
+ spin_unlock(&sysfs_lock);
error = sysfs_create(dentry, (attr->mode & S_IALLUGO) | S_IFREG, init);
if (error) {
sysfs_put(sd);
@@ -260,7 +275,10 @@ static int sysfs_attach_link(struct sysf
int err = 0;

dentry->d_fsdata = sysfs_get(sd);
+ /* protect sd->s_dentry against sysfs_d_iput */
+ spin_lock(&sysfs_lock);
sd->s_dentry = dentry;
+ spin_unlock(&sysfs_lock);
err = sysfs_create(dentry, S_IFLNK|S_IRWXUGO, init_symlink);
if (!err) {
dentry->d_op = &sysfs_dentry_ops;
Index: work/fs/sysfs/inode.c
===================================================================
--- work.orig/fs/sysfs/inode.c
+++ work/fs/sysfs/inode.c
@@ -244,9 +244,23 @@ static inline void orphan_all_buffers(st
*/
void sysfs_drop_dentry(struct sysfs_dirent * sd, struct dentry * parent)
{
- struct dentry * dentry = sd->s_dentry;
+ struct dentry *dentry = NULL;
struct inode *inode;

+ /* We're not holding a reference to ->s_dentry dentry but the
+ * field will stay valid as long as sysfs_lock is held.
+ */
+ spin_lock(&sysfs_lock);
+ spin_lock(&dcache_lock);
+
+ /* dget dentry if it's still alive */
+ if (sd->s_dentry && sd->s_dentry->d_inode)
+ dentry = dget_locked(sd->s_dentry);
+
+ spin_unlock(&dcache_lock);
+ spin_unlock(&sysfs_lock);
+
+ /* drop dentry */
if (dentry) {
spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
@@ -266,6 +280,8 @@ void sysfs_drop_dentry(struct sysfs_dire
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
}
+
+ dput(dentry);
}
}

Index: work/fs/sysfs/sysfs.h
===================================================================
--- work.orig/fs/sysfs/sysfs.h
+++ work/fs/sysfs/sysfs.h
@@ -32,6 +32,7 @@ extern const unsigned char * sysfs_get_n
extern void sysfs_drop_dentry(struct sysfs_dirent *sd, struct dentry *parent);
extern int sysfs_setattr(struct dentry *dentry, struct iattr *iattr);

+extern spinlock_t sysfs_lock;
extern struct rw_semaphore sysfs_rename_sem;
extern struct super_block * sysfs_sb;
extern const struct file_operations sysfs_dir_operations;

2007-05-17 18:56:39

by Eric Sandeen

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

Andrew Morton wrote:
> On Wed, 16 May 2007 17:40:53 +0200 Tejun Heo <[email protected]> wrote:
>
>> I see. I thought there was different approach on fixing the problem.
>> I'll try to backport the synchronization fix but am afraid it can be too
>> risky for -stable. If it seems too risky, I'll send a patch to disable
>> reclamation.
>
> OK. Sad. Maybe we add /proc/sys/fs/i-have-lots-of-disks-and-dont-mind-if-it-oopses
> to enable the old behaviour.

Out of curiosity, is there a decent reproducer for this problem, or is
it just a few lucky individuals? :)

-Eric

2007-05-21 04:35:26

[permalink] [raw]

Subject: Re: [PATCH 1/2] sysfs: fix condition check in sysfs_drop_dentry()

On Thu, May 17, 2007 at 07:52:53PM +0200, Tejun Heo wrote:
> The condition check doesn't make much sense as it basically always
> succeeds. This causes NULL dereferencing on certain cases. It seems
> that parentheses are put in the wrong place. Fix it.
>
> DON'T APPLY JUST YET
> ---
>
> Maneesh, is this correct?
>
Yes.

> fs/sysfs/inode.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> Index: work/fs/sysfs/inode.c
> ===================================================================
> --- work.orig/fs/sysfs/inode.c
> +++ work/fs/sysfs/inode.c
> @@ -250,7 +250,7 @@ void sysfs_drop_dentry(struct sysfs_dire
> if (dentry) {
> spin_lock(&dcache_lock);
> spin_lock(&dentry->d_lock);
> - if (!(d_unhashed(dentry) && dentry->d_inode)) {
> + if (!d_unhashed(dentry) && dentry->d_inode) {
> inode = dentry->d_inode;
> spin_lock(&inode->i_lock);
> __iget(inode);

Ack'ed

Thanks
Maneesh

--
Maneesh Soni
Linux Technology Center,
IBM India Systems and Technology Lab,
Bangalore, India

2007-05-21 04:38:40

[permalink] [raw]

Subject: Re: [PATCH -stable] sysfs: disable reclamation by default

On Thu, May 17, 2007 at 07:49:31PM +0200, Tejun Heo wrote:
> Maneesh Soni wrote:
> > On Thu, May 17, 2007 at 05:04:23AM -0700, Greg KH wrote:
> >> On Wed, May 16, 2007 at 08:31:00PM +0200, Tejun Heo wrote:
> >>> sd->s_dentry updates made by dentry/inode reclamation are racy and can
> >>> lead to BUG() or oops. This is already fixed in -mm and the fix is
> >>> scheduled to be merged into upstream for 2.6.23 but the fix
> >>> reimplements sysfs dentry dropping and is too risky for -stable
> >>> kernels.
> >>>
> >
> > But was the synchronization fix tested by people facing the race? I still
> > don't understand the racy code path. The last google problem I saw had
> > s_dentry field as NULL.
>
> Please take a look at the following message.
>
> http://article.gmane.org/gmane.linux.kernel/521729
>
> I could reproduce both races on my test machine fairly reliable with
> parallel find, cat, mount/mount while repeatedly ins/rmmoding a libata
> driver.
>
Thanks for the pointer.. earlier it got buried in the fat rework..

> >>> This is an interim solution for -stable kernels. sysfs reclamation is
> >>> disabled by default and can be enabled by using sysfs.enable_reclaim
> >>> kernel parameter. Note that dentries are still created on demand, so
> >>> attribute and symlinks nodes aren't allocated on creation. They're
> >>> allocated on first lookup and deallocated when the sysfs node is
> >>> removed.
> >> Ick, this is going to kill memory on big boxes (s390 and others) and I
> >> don't really want to apply this it if at all possible.
> >>
> > At least not make it default. This might create boot issues with these
> > boxes.
>
> Which makes oopsing the default. Fun! :-)
>
but.. avoid oops by not booting at all is more fun !! ;-)

> >> Maneesh, any other thoughts?
> >>
> > I actually wanted to investigate this oops but left it considering the
> > rework being done by Tejun. If this still make sense we can have some
> > more debug code stuffed there or get a crashdump (kdump) to get better
> > understanding of the race.
>
> The above message contains analysis of both races. I just ported the
> fixes. I have a different test machine now and can't reproduce the
> races with this one yet so I couldn't verify whether the patches
> actually fix the problem. I'll post the patches anyway. If anyone can
> reproduce these races, please verify the posted patches fix the problem.
>

I would prefer fixing the race instead of making attributes non-reclaimable.

Thanks
Maneesh

--
Maneesh Soni
Linux Technology Center,
IBM India Systems and Technology Lab,
Bangalore, India

2007-05-21 05:00:43

[permalink] [raw]

Subject: Re: [PATCH 2/2] sysfs: fix race condition around sd->s_dentry, take#2

On Thu, May 17, 2007 at 08:16:10PM +0200, Tejun Heo wrote:
> Allowing attribute and symlink dentries to be reclaimed means
> sd->s_dentry can change dynamically. However, updates to the field
> are unsynchronized leading to race conditions. This patch adds
> sysfs_lock and use it to synchronize updates to sd->s_dentry.
>
> Due to the locking around ->d_iput, the check in sysfs_drop_dentry()
> is complex. sysfs_lock only protect sd->s_dentry pointer itself. The
> validity of the dentry is protected by dcache_lock, so whether dentry
> is alive or not can only be tested while holding both locks.
>
> This is minimal backport of sysfs_drop_dentry() rewrite in devel
> branch.
>
> DONT APPLY JUST YET

Looks ok to me.. I have tested it it but unfortunately I couldn't
recreate the race without the patch also. It would be helpful if
people actually seeing the race, provide the test results.

Greg, please merge this one once we have some test results.

Regards,
Maneesh

> ---
> Moving sysfs_drop_dentry() and sysfs_put() calls out of mutex isn't
> necessary, so this is the minimal one but there shouldn't be
> any difference functionality-wise.
>
> fs/sysfs/dir.c | 22 ++++++++++++++++++++--
> fs/sysfs/inode.c | 18 +++++++++++++++++-
> fs/sysfs/sysfs.h | 1 +
> 3 files changed, 38 insertions(+), 3 deletions(-)
>
> Index: work/fs/sysfs/dir.c
> ===================================================================
> --- work.orig/fs/sysfs/dir.c
> +++ work/fs/sysfs/dir.c
> @@ -13,14 +13,26 @@
> #include "sysfs.h"
>
> DECLARE_RWSEM(sysfs_rename_sem);
> +spinlock_t sysfs_lock = SPIN_LOCK_UNLOCKED;
>
> static void sysfs_d_iput(struct dentry * dentry, struct inode * inode)
> {
> struct sysfs_dirent * sd = dentry->d_fsdata;
>
> if (sd) {
> - BUG_ON(sd->s_dentry != dentry);
> - sd->s_dentry = NULL;
> + /* sd->s_dentry is protected with sysfs_lock. This
> + * allows sysfs_drop_dentry() to dereference it.
> + */
> + spin_lock(&sysfs_lock);
> +
> + /* The dentry might have been deleted or another
> + * lookup could have happened updating sd->s_dentry to
> + * point the new dentry. Ignore if it isn't pointing
> + * to this dentry.
> + */
> + if (sd->s_dentry == dentry)
> + sd->s_dentry = NULL;
> + spin_unlock(&sysfs_lock);
> sysfs_put(sd);
> }
> iput(inode);
> @@ -238,7 +250,10 @@ static int sysfs_attach_attr(struct sysf
> }
>
> dentry->d_fsdata = sysfs_get(sd);
> + /* protect sd->s_dentry against sysfs_d_iput */
> + spin_lock(&sysfs_lock);
> sd->s_dentry = dentry;
> + spin_unlock(&sysfs_lock);
> error = sysfs_create(dentry, (attr->mode & S_IALLUGO) | S_IFREG, init);
> if (error) {
> sysfs_put(sd);
> @@ -260,7 +275,10 @@ static int sysfs_attach_link(struct sysf
> int err = 0;
>
> dentry->d_fsdata = sysfs_get(sd);
> + /* protect sd->s_dentry against sysfs_d_iput */
> + spin_lock(&sysfs_lock);
> sd->s_dentry = dentry;
> + spin_unlock(&sysfs_lock);
> err = sysfs_create(dentry, S_IFLNK|S_IRWXUGO, init_symlink);
> if (!err) {
> dentry->d_op = &sysfs_dentry_ops;
> Index: work/fs/sysfs/inode.c
> ===================================================================
> --- work.orig/fs/sysfs/inode.c
> +++ work/fs/sysfs/inode.c
> @@ -244,9 +244,23 @@ static inline void orphan_all_buffers(st
> */
> void sysfs_drop_dentry(struct sysfs_dirent * sd, struct dentry * parent)
> {
> - struct dentry * dentry = sd->s_dentry;
> + struct dentry *dentry = NULL;
> struct inode *inode;
>
> + /* We're not holding a reference to ->s_dentry dentry but the
> + * field will stay valid as long as sysfs_lock is held.
> + */
> + spin_lock(&sysfs_lock);
> + spin_lock(&dcache_lock);
> +
> + /* dget dentry if it's still alive */
> + if (sd->s_dentry && sd->s_dentry->d_inode)
> + dentry = dget_locked(sd->s_dentry);
> +
> + spin_unlock(&dcache_lock);
> + spin_unlock(&sysfs_lock);
> +
> + /* drop dentry */
> if (dentry) {
> spin_lock(&dcache_lock);
> spin_lock(&dentry->d_lock);
> @@ -266,6 +280,8 @@ void sysfs_drop_dentry(struct sysfs_dire
> spin_unlock(&dentry->d_lock);
> spin_unlock(&dcache_lock);
> }
> +
> + dput(dentry);
> }
> }
>
> Index: work/fs/sysfs/sysfs.h
> ===================================================================
> --- work.orig/fs/sysfs/sysfs.h
> +++ work/fs/sysfs/sysfs.h
> @@ -32,6 +32,7 @@ extern const unsigned char * sysfs_get_n
> extern void sysfs_drop_dentry(struct sysfs_dirent *sd, struct dentry *parent);
> extern int sysfs_setattr(struct dentry *dentry, struct iattr *iattr);
>
> +extern spinlock_t sysfs_lock;
> extern struct rw_semaphore sysfs_rename_sem;
> extern struct super_block * sysfs_sb;
> extern const struct file_operations sysfs_dir_operations;

--
Maneesh Soni
Linux Technology Center,
IBM India Systems and Technology Lab,
Bangalore, India

2007-05-21 16:10:10

by Eric Sandeen

[permalink] [raw]

Subject: Re: [PATCH 2/2] sysfs: fix race condition around sd->s_dentry, take#2

Maneesh Soni wrote:

> Looks ok to me.. I have tested it it but unfortunately I couldn't
> recreate the race without the patch also. It would be helpful if
> people actually seeing the race, provide the test results.
>
> Greg, please merge this one once we have some test results.

I merged this back to RHEL4-era (2.6.9) kernels due to a report of this
problem on RHEL4. The tester could hit the sysfs_d_iput bug fairly
easily before, and has not seen it with this patch in place.

The other bug they were seeing even more often was the sysfs_readdir()
oops as we try to extract the inode number for regular sysfs files,
again due to transient vnodes. Is any fix for that problem being
considered for -stable? I had backported a simpler version of Tejun's
s_ino fix for the problem, without the ida complexity - and brought a
couple other small upstream fixes back to support that. If nothing has
yet been proposed for -stable, I can post that here.

Thanks,

-Eric

2007-05-21 16:15:29

[permalink] [raw]

Subject: Re: [PATCH 2/2] sysfs: fix race condition around sd->s_dentry, take#2

Eric Sandeen wrote:
> Maneesh Soni wrote:
>
>> Looks ok to me.. I have tested it it but unfortunately I couldn't
>> recreate the race without the patch also. It would be helpful if
>> people actually seeing the race, provide the test results.
>>
>> Greg, please merge this one once we have some test results.
>
> I merged this back to RHEL4-era (2.6.9) kernels due to a report of this
> problem on RHEL4. The tester could hit the sysfs_d_iput bug fairly
> easily before, and has not seen it with this patch in place.
>
> The other bug they were seeing even more often was the sysfs_readdir()
> oops as we try to extract the inode number for regular sysfs files,
> again due to transient vnodes. Is any fix for that problem being
> considered for -stable? I had backported a simpler version of Tejun's
> s_ino fix for the problem, without the ida complexity - and brought a
> couple other small upstream fixes back to support that. If nothing has
> yet been proposed for -stable, I can post that here.

I don't think any other fix has been proposed yet. Please post them.
That definitely needs fixing.

--
tejun

2007-05-22 22:40:57

by Greg KH

[permalink] [raw]

Subject: Re: [PATCH 2/2] sysfs: fix race condition around sd->s_dentry, take#2

On Mon, May 21, 2007 at 10:31:01AM +0530, Maneesh Soni wrote:
> On Thu, May 17, 2007 at 08:16:10PM +0200, Tejun Heo wrote:
> > Allowing attribute and symlink dentries to be reclaimed means
> > sd->s_dentry can change dynamically. However, updates to the field
> > are unsynchronized leading to race conditions. This patch adds
> > sysfs_lock and use it to synchronize updates to sd->s_dentry.
> >
> > Due to the locking around ->d_iput, the check in sysfs_drop_dentry()
> > is complex. sysfs_lock only protect sd->s_dentry pointer itself. The
> > validity of the dentry is protected by dcache_lock, so whether dentry
> > is alive or not can only be tested while holding both locks.
> >
> > This is minimal backport of sysfs_drop_dentry() rewrite in devel
> > branch.
> >
> > DONT APPLY JUST YET
>
> Looks ok to me.. I have tested it it but unfortunately I couldn't
> recreate the race without the patch also. It would be helpful if
> people actually seeing the race, provide the test results.
>
> Greg, please merge this one once we have some test results.

Can someone just resend it after those test results are in, with the
proper signed-off-by, so I know it's safe to apply?

thanks,

greg k-h

2007-05-23 08:21:53

[permalink] [raw]

Subject: Re: [PATCH 2/2] sysfs: fix race condition around sd->s_dentry, take#2

Greg KH wrote:
> On Mon, May 21, 2007 at 10:31:01AM +0530, Maneesh Soni wrote:
>> On Thu, May 17, 2007 at 08:16:10PM +0200, Tejun Heo wrote:
>>> Allowing attribute and symlink dentries to be reclaimed means
>>> sd->s_dentry can change dynamically. However, updates to the field
>>> are unsynchronized leading to race conditions. This patch adds
>>> sysfs_lock and use it to synchronize updates to sd->s_dentry.
>>>
>>> Due to the locking around ->d_iput, the check in sysfs_drop_dentry()
>>> is complex. sysfs_lock only protect sd->s_dentry pointer itself. The
>>> validity of the dentry is protected by dcache_lock, so whether dentry
>>> is alive or not can only be tested while holding both locks.
>>>
>>> This is minimal backport of sysfs_drop_dentry() rewrite in devel
>>> branch.
>>>
>>> DONT APPLY JUST YET
>> Looks ok to me.. I have tested it it but unfortunately I couldn't
>> recreate the race without the patch also. It would be helpful if
>> people actually seeing the race, provide the test results.
>>
>> Greg, please merge this one once we have some test results.
>
> Can someone just resend it after those test results are in, with the
> proper signed-off-by, so I know it's safe to apply?

I'll resend with S-O-B after someone verifies it fixes the problem.

--
tejun

2007-06-08 14:35:43

[permalink] [raw]

Subject: Re: [PATCH 2/2] sysfs: fix race condition around sd->s_dentry, take#2

Tejun Heo wrote:
> Greg KH wrote:
>> On Mon, May 21, 2007 at 10:31:01AM +0530, Maneesh Soni wrote:
>>> On Thu, May 17, 2007 at 08:16:10PM +0200, Tejun Heo wrote:
>>>> Allowing attribute and symlink dentries to be reclaimed means
>>>> sd->s_dentry can change dynamically. However, updates to the field
>>>> are unsynchronized leading to race conditions. This patch adds
>>>> sysfs_lock and use it to synchronize updates to sd->s_dentry.
>>>>
>>>> Due to the locking around ->d_iput, the check in sysfs_drop_dentry()
>>>> is complex. sysfs_lock only protect sd->s_dentry pointer itself. The
>>>> validity of the dentry is protected by dcache_lock, so whether dentry
>>>> is alive or not can only be tested while holding both locks.
>>>>
>>>> This is minimal backport of sysfs_drop_dentry() rewrite in devel
>>>> branch.
>>>>
>>>> DONT APPLY JUST YET
>>> Looks ok to me.. I have tested it it but unfortunately I couldn't
>>> recreate the race without the patch also. It would be helpful if
>>> people actually seeing the race, provide the test results.
>>>
>>> Greg, please merge this one once we have some test results.
>> Can someone just resend it after those test results are in, with the
>> proper signed-off-by, so I know it's safe to apply?
>
> I'll resend with S-O-B after someone verifies it fixes the problem.

Have been trying to reproduce the problem on the latest -rc but haven't
succeeded yet. It seems I lost my magic recipe for these races. Did
anyone succeed?

--
tejun

2007-06-09 06:50:18

[permalink] [raw]

Subject: Re: [PATCH 2/2] sysfs: fix race condition around sd->s_dentry, take#2

Tejun Heo wrote:
> Tejun Heo wrote:
>> Greg KH wrote:
>>> On Mon, May 21, 2007 at 10:31:01AM +0530, Maneesh Soni wrote:
>>>> On Thu, May 17, 2007 at 08:16:10PM +0200, Tejun Heo wrote:
>>>>> Allowing attribute and symlink dentries to be reclaimed means
>>>>> sd->s_dentry can change dynamically. However, updates to the field
>>>>> are unsynchronized leading to race conditions. This patch adds
>>>>> sysfs_lock and use it to synchronize updates to sd->s_dentry.
>>>>>
>>>>> Due to the locking around ->d_iput, the check in sysfs_drop_dentry()
>>>>> is complex. sysfs_lock only protect sd->s_dentry pointer itself. The
>>>>> validity of the dentry is protected by dcache_lock, so whether dentry
>>>>> is alive or not can only be tested while holding both locks.
>>>>>
>>>>> This is minimal backport of sysfs_drop_dentry() rewrite in devel
>>>>> branch.
>>>>>
>>>>> DONT APPLY JUST YET
>>>> Looks ok to me.. I have tested it it but unfortunately I couldn't
>>>> recreate the race without the patch also. It would be helpful if
>>>> people actually seeing the race, provide the test results.
>>>>
>>>> Greg, please merge this one once we have some test results.
>>> Can someone just resend it after those test results are in, with the
>>> proper signed-off-by, so I know it's safe to apply?
>> I'll resend with S-O-B after someone verifies it fixes the problem.
>
> Have been trying to reproduce the problem on the latest -rc but haven't
> succeeded yet. It seems I lost my magic recipe for these races. Did
> anyone succeed?

Alright, overnight test finally hit the race condition. It takes me
much longer now but the race condition is definitely still there. I'll
verify each proposed patch and send it for 2.6.22 inclusion and if
they're accepted to -stable.

Thanks.

--
tejun

2007-06-10 16:16:41

by Greg KH

[permalink] [raw]

Subject: Re: [PATCH 2/2] sysfs: fix race condition around sd->s_dentry, take#2

On Sat, Jun 09, 2007 at 03:49:46PM +0900, Tejun Heo wrote:
> Tejun Heo wrote:
> > Tejun Heo wrote:
> >> Greg KH wrote:
> >>> On Mon, May 21, 2007 at 10:31:01AM +0530, Maneesh Soni wrote:
> >>>> On Thu, May 17, 2007 at 08:16:10PM +0200, Tejun Heo wrote:
> >>>>> Allowing attribute and symlink dentries to be reclaimed means
> >>>>> sd->s_dentry can change dynamically. However, updates to the field
> >>>>> are unsynchronized leading to race conditions. This patch adds
> >>>>> sysfs_lock and use it to synchronize updates to sd->s_dentry.
> >>>>>
> >>>>> Due to the locking around ->d_iput, the check in sysfs_drop_dentry()
> >>>>> is complex. sysfs_lock only protect sd->s_dentry pointer itself. The
> >>>>> validity of the dentry is protected by dcache_lock, so whether dentry
> >>>>> is alive or not can only be tested while holding both locks.
> >>>>>
> >>>>> This is minimal backport of sysfs_drop_dentry() rewrite in devel
> >>>>> branch.
> >>>>>
> >>>>> DONT APPLY JUST YET
> >>>> Looks ok to me.. I have tested it it but unfortunately I couldn't
> >>>> recreate the race without the patch also. It would be helpful if
> >>>> people actually seeing the race, provide the test results.
> >>>>
> >>>> Greg, please merge this one once we have some test results.
> >>> Can someone just resend it after those test results are in, with the
> >>> proper signed-off-by, so I know it's safe to apply?
> >> I'll resend with S-O-B after someone verifies it fixes the problem.
> >
> > Have been trying to reproduce the problem on the latest -rc but haven't
> > succeeded yet. It seems I lost my magic recipe for these races. Did
> > anyone succeed?
>
> Alright, overnight test finally hit the race condition. It takes me
> much longer now but the race condition is definitely still there. I'll
> verify each proposed patch and send it for 2.6.22 inclusion and if
> they're accepted to -stable.

Ok, thanks for doing this.

greg k-h

2007-06-29 03:51:20

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

On 05/16/2007 09:24 AM, Clemens Schwaighofer wrote:
> Hi,

I had my system running up for about one month without any issues, and
then it happened again, same kernel oops, panic, end.

So I have upgraded to 2.6.22-rc4-mm2 in hope it might fix it, but I just
got another oops (uptime 4d) [see attached file]

my config hasn't changed in any way to the previous kernels.

--
[ Clemens Schwaighofer -----=====:::::~ ]
[ TEQUILA\ Japan IT Group ]
[ 6-17-2 Ginza Chuo-ku, Tokyo 104-8167, JAPAN ]
[ Tel: +81-(0)3-3545-7703 Fax: +81-(0)3-3545-7343 ]
[ http://www.tequila.co.jp ]

Attachments:

2.6.22-rc4-mm2.20070629.oops.log (7.84 kB)
signature.asc (252.00 B)
OpenPGP digital signature Download all attachments

2007-06-29 06:12:59

by Satyam Sharma

[permalink] [raw]

Subject: Re: Oops and Panics in 2.6.21.1, 2.6.20.6 and 2.6.19.2

Hi Clemens,

[ Cc:'ing Andrew, original thread at http://lkml.org/lkml/2007/5/15/354 ]

On 6/29/07, Clemens Schwaighofer <[email protected]> wrote:
> On 05/16/2007 09:24 AM, Clemens Schwaighofer wrote:
> > Hi,
>
> I had my system running up for about one month without any issues, and
> then it happened again, same kernel oops, panic, end.
>
> So I have upgraded to 2.6.22-rc4-mm2 in hope it might fix it, but I just
> got another oops (uptime 4d) [see attached file]

You "upgraded" from -stable series kernels (2.6.19.2 / 2.6.20.6 /
2.6.21.1) to a -mm kernel, which is anything but :-)

On the one hand, I really like that we're getting testers for -mm
kernels, but on the other hand, my good and honest side would
recommend you to install stable kernels (2.6.x.y versions) on
production systems, if you really care about uptimes.

> my config hasn't changed in any way to the previous kernels.

This doesn't look like the same oops you were getting persistently
with 2.6.21.1 ... you could try upgrading to 2.6.22-rc6 (without -mm)
too, if the oops in 2.6.21.1 was occurring too frequently in your setup;
possibly it has been resolved in the 22-rc series.

Satyam

[ Clemens' 2.6.22-rc4-mm2 oops below. ]

Jun 29 11:25:08 saturn kernel: [348308.690154] BUG: unable to handle
kernel NULL pointer dereference at virtual address 00000001
Jun 29 11:25:08 saturn kernel: [348308.690160] printing eip:
Jun 29 11:25:08 saturn kernel: [348308.690162] c108887c
Jun 29 11:25:08 saturn kernel: [348308.690163] *pde = 00000000
Jun 29 11:25:08 saturn kernel: [348308.690166] Oops: 0000 [#2]
Jun 29 11:25:08 saturn kernel: [348308.690167] PREEMPT
Jun 29 11:25:08 saturn kernel: [348308.690169] Modules linked in:
eeprom pcspkr i2c_viapro k8temp hwmon i2c_core
Jun 29 11:25:08 saturn kernel: [348308.690177] CPU: 0
Jun 29 11:25:08 saturn kernel: [348308.690177] EIP:
0060:[__d_lookup+108/336] Not tainted VLI
Jun 29 11:25:08 saturn kernel: [348308.690179] EFLAGS: 00010202
(2.6.22-rc4-mm2 #1)
Jun 29 11:25:08 saturn kernel: [348308.690185] EIP is at __d_lookup+0x6c/0x150
Jun 29 11:25:08 saturn kernel: [348308.690187] eax: 00000001 ebx:
00000001 ecx: 00000001 edx: 089c1579
Jun 29 11:25:08 saturn kernel: [348308.690190] esi: c6840ee8 edi:
c301d734 ebp: f786f080 esp: c6840e84
Jun 29 11:25:08 saturn kernel: [348308.690193] ds: 007b es: 007b
fs: 0000 gs: 0033 ss: 0068
Jun 29 11:25:08 saturn kernel: [348308.690196] Process nfsd (pid:
30536, ti=c6840000 task=c62bce90 task.ti=c6840000)
Jun 29 11:25:08 saturn kernel: [348308.690198] Stack: c301d734
089c1579 c6840edb 00000002 c6840ee8 ffffffff 00000005 c6840edb
Jun 29 11:25:08 saturn kernel: [348308.690205] 0000f9ec
c6840ee8 c301d734 c471ba84 c1088976 c7bbb090 c7bbb090 00000000
Jun 29 11:25:08 saturn kernel: [348308.690211] c10b49fc
c6840edb 0000000d c14914fb 00007793 332bce90 31313630 c5469900
Jun 29 11:25:08 saturn kernel: [348308.690217] Call Trace:
Jun 29 11:25:08 saturn kernel: [348308.690220] [d_lookup+22/64]
d_lookup+0x16/0x40
Jun 29 11:25:08 saturn kernel: [348308.690224]
[proc_flush_task+76/496] proc_flush_task+0x4c/0x1f0
Jun 29 11:25:08 saturn kernel: [348308.690229] [release_task+612/880]
release_task+0x264/0x370
Jun 29 11:25:08 saturn kernel: [348308.690234] [do_wait+1850/3072]
do_wait+0x73a/0xc00
Jun 29 11:25:08 saturn kernel: [348308.690239]
[_spin_unlock_irq+38/64] _spin_unlock_irq+0x26/0x40
Jun 29 11:25:08 saturn kernel: [348308.690243]
[default_wake_function+0/16] default_wake_function+0x0/0x10
Jun 29 11:25:08 saturn kernel: [348308.690247] [sys_wait4+49/64]
sys_wait4+0x31/0x40
Jun 29 11:25:08 saturn kernel: [348308.690251] [sys_waitpid+39/48]
sys_waitpid+0x27/0x30
Jun 29 11:25:08 saturn kernel: [348308.690255] [syscall_call+7/11]
syscall_call+0x7/0xb
Jun 29 11:25:08 saturn kernel: [348308.690259] =======================
Jun 29 11:25:08 saturn kernel: [348308.690260] INFO: lockdep is turned off.
Jun 29 11:25:08 saturn kernel: [348308.690262] Code: d3 e8 31 c3 23 1d
b4 d8 51 c1 b8 01 00 00 00 c1 e3 02 03 1d bc d8 51 c1 e8 e2 24 f9 ff
8b 1b 85 db 75 08 eb 44 85 c0 89 c3 74 3e <8b> 03 0f 18 00 90 8d 6b d8
8b 54 24 04 3b 55 34 75 e8 8b 34 24
Jun 29 11:25:08 saturn kernel: [348308.690289] EIP:
[__d_lookup+108/336] __d_lookup+0x6c/0x150 SS:ESP 0068:c6840e84
Jun 29 11:25:08 saturn kernel: [348308.690296] note: nfsd[30536]
exited with preempt_count 1
Jun 29 11:25:08 saturn kernel: [348308.690303] BUG: scheduling while
atomic: nfsd/0x10000002/30536
Jun 29 11:25:08 saturn kernel: [348308.690305] INFO: lockdep is turned off.
Jun 29 11:25:08 saturn kernel: [348308.690307] [schedule+1490/1744]
schedule+0x5d2/0x6d0
Jun 29 11:25:08 saturn kernel: [348308.690311]
[vt_console_print+106/688] vt_console_print+0x6a/0x2b0
Jun 29 11:25:08 saturn kernel: [348308.690316] [__cond_resched+18/48]
__cond_resched+0x12/0x30
Jun 29 11:25:08 saturn kernel: [348308.690319] [cond_resched+42/64]
cond_resched+0x2a/0x40
Jun 29 11:25:08 saturn kernel: [348308.690322] [unmap_vmas+1116/1184]
unmap_vmas+0x45c/0x4a0
Jun 29 11:25:08 saturn kernel: [348308.690327] [exit_mmap+105/256]
exit_mmap+0x69/0x100
Jun 29 11:25:08 saturn kernel: [348308.690331] [mmput+68/256] mmput+0x44/0x100
Jun 29 11:25:08 saturn kernel: [348308.690335] [do_exit+301/2224]
do_exit+0x12d/0x8b0
Jun 29 11:25:08 saturn kernel: [348308.690339] [__wake_up+56/80]
__wake_up+0x38/0x50
Jun 29 11:25:08 saturn kernel: [348308.690342] [die+574/576] die+0x23e/0x240
Jun 29 11:25:08 saturn kernel: [348308.690346]
[do_page_fault+724/1520] do_page_fault+0x2d4/0x5f0
Jun 29 11:25:08 saturn kernel: [348308.690349] [tcp_ack+2742/6960]
tcp_ack+0xab6/0x1b30
Jun 29 11:25:08 saturn kernel: [348308.690354] [do_page_fault+0/1520]
do_page_fault+0x0/0x5f0
Jun 29 11:25:08 saturn kernel: [348308.690357] [error_code+106/112]
error_code+0x6a/0x70
Jun 29 11:25:08 saturn kernel: [348308.690360] [__d_lookup+108/336]
__d_lookup+0x6c/0x150
Jun 29 11:25:08 saturn kernel: [348308.690364] [d_lookup+22/64]
d_lookup+0x16/0x40
Jun 29 11:25:08 saturn kernel: [348308.690367]
[proc_flush_task+76/496] proc_flush_task+0x4c/0x1f0
Jun 29 11:25:08 saturn kernel: [348308.690372] [release_task+612/880]
release_task+0x264/0x370
Jun 29 11:25:08 saturn kernel: [348308.690376] [do_wait+1850/3072]
do_wait+0x73a/0xc00
Jun 29 11:25:08 saturn kernel: [348308.690380]
[_spin_unlock_irq+38/64] _spin_unlock_irq+0x26/0x40
Jun 29 11:25:08 saturn kernel: [348308.690383]
[default_wake_function+0/16] default_wake_function+0x0/0x10
Jun 29 11:25:08 saturn kernel: [348308.690387] [sys_wait4+49/64]
sys_wait4+0x31/0x40
Jun 29 11:25:08 saturn kernel: [348308.690390] [sys_waitpid+39/48]
sys_waitpid+0x27/0x30
Jun 29 11:25:08 saturn kernel: [348308.690394] [syscall_call+7/11]
syscall_call+0x7/0xb
Jun 29 11:25:08 saturn kernel: [348308.690398] =======================
Jun 29 11:25:08 saturn kernel: [348308.690472] BUG: scheduling while
atomic: nfsd/0x10000002/30536
Jun 29 11:25:08 saturn kernel: [348308.690474] INFO: lockdep is turned off.
Jun 29 11:25:08 saturn kernel: [348308.690476] [schedule+1490/1744]
schedule+0x5d2/0x6d0
Jun 29 11:25:08 saturn kernel: [348308.690480]
[_atomic_dec_and_lock+22/96] _atomic_dec_and_lock+0x16/0x60
Jun 29 11:25:08 saturn kernel: [348308.690485]
[_atomic_dec_and_lock+49/96] _atomic_dec_and_lock+0x31/0x60
Jun 29 11:25:08 saturn kernel: [348308.690489] [__cond_resched+18/48]
__cond_resched+0x12/0x30
Jun 29 11:25:08 saturn kernel: [348308.690492] [cond_resched+42/64]
cond_resched+0x2a/0x40
Jun 29 11:25:08 saturn kernel: [348308.690495]
[put_files_struct+166/208] put_files_struct+0xa6/0xd0
Jun 29 11:25:08 saturn kernel: [348308.690499] [do_exit+363/2224]
do_exit+0x16b/0x8b0
Jun 29 11:25:08 saturn kernel: [348308.690503] [__wake_up+56/80]
__wake_up+0x38/0x50
Jun 29 11:25:08 saturn kernel: [348308.690506] [die+574/576] die+0x23e/0x240
Jun 29 11:25:08 saturn kernel: [348308.690510]
[do_page_fault+724/1520] do_page_fault+0x2d4/0x5f0
Jun 29 11:25:08 saturn kernel: [348308.690513] [tcp_ack+2742/6960]
tcp_ack+0xab6/0x1b30
Jun 29 11:25:08 saturn kernel: [348308.690517] [do_page_fault+0/1520]
do_page_fault+0x0/0x5f0
Jun 29 11:25:08 saturn kernel: [348308.690520] [error_code+106/112]
error_code+0x6a/0x70
Jun 29 11:25:08 saturn kernel: [348308.690524] [__d_lookup+108/336]
__d_lookup+0x6c/0x150
Jun 29 11:25:08 saturn kernel: [348308.690528] [d_lookup+22/64]
d_lookup+0x16/0x40
Jun 29 11:25:08 saturn kernel: [348308.690531]
[proc_flush_task+76/496] proc_flush_task+0x4c/0x1f0
Jun 29 11:25:08 saturn kernel: [348308.690535] [release_task+612/880]
release_task+0x264/0x370
Jun 29 11:25:08 saturn kernel: [348308.690539] [do_wait+1850/3072]
do_wait+0x73a/0xc00
Jun 29 11:25:08 saturn kernel: [348308.690543]
[_spin_unlock_irq+38/64] _spin_unlock_irq+0x26/0x40
Jun 29 11:25:08 saturn kernel: [348308.690547]
[default_wake_function+0/16] default_wake_function+0x0/0x10
Jun 29 11:25:08 saturn kernel: [348308.690551] [sys_wait4+49/64]
sys_wait4+0x31/0x40
Jun 29 11:25:08 saturn kernel: [348308.690554] [sys_waitpid+39/48]
sys_waitpid+0x27/0x30
Jun 29 11:25:08 saturn kernel: [348308.690558] [syscall_call+7/11]
syscall_call+0x7/0xb
Jun 29 11:25:08 saturn kernel: [348308.690561] =======================

2007-06-29 06:18:23