Hi,
I'm tripping over a race condition involving file handling.
I can consistently crash here:
TASK: c0000000fd6dc040[10448] 'ifdown' THREAD: c0000000f3310000
Call Trace: Sep 22 14:29:21 marulp1 kernel: [c0000000f3313b10] [c0000000000506c4] .copy_files+0x400/0x414 (unreliable)
[c0000000f3313bd0] [c00000000005161c] .copy_process+0x660/0x12bc
[c0000000f3313ce0] [c000000000052318] .do_fork+0xa0/0x25c
[c0000000f3313dc0] [c0000000000159c8] .sys_clone+0x5c/0x74
[c0000000f3313e30] [c000000000010a88] .ppc_clone+0x8/0xc
The problem seems to be that one of the file pointers is breifly
set to (int32)-1 even on a 64-bit machine. The part of copy_process()
that gets mashed by this is:
for (i = open_files; i != 0; i--) {
struct file *f = *old_fds++;
if (f)
get_file(f); <== derefs f, which is -1
*new_fds++ = f;
}
By inserting if(f==(void*)0xffffffffUL) printk ...
I can find out that i=230 is the one with the problem and open_files=256!
I haven't yet found who set struct file * to a -1.
I'm generting this behaviour with a hotplug event that is causing ifdown
and ifup to run simultaneously. (The device driver was shut down and
restarted, causing simultaneous hotplug events). Although the above
stack shows ifdown getting clobbered, I've also seen pci.agent be
the process that suffers.
The problem goes away if I insert a sleep of about half-a-second or more
between the device driver shutdown and startup.
Affected machine is a ppc64 power4 box. I've seen the problem
for a long time (months?), including monday's bk clone of
bkbits of 2.6.9-rc2; waiting for this bug "to fix itself" doesn't
seem to be working.
--linas
p.s. am I supposed to be using the OSDL bugzilla to report & track bugs
like this?
More info...
On Wed, Sep 22, 2004 at 03:14:01PM -0500, Linas Vepstas was heard to remark:
>
> I'm tripping over a race condition involving file handling.
> I can consistently crash here:
>
> TASK: c0000000fd6dc040[10448] 'ifdown' THREAD: c0000000f3310000
> Call Trace: Sep 22 14:29:21 marulp1 kernel: [c0000000f3313b10] [c0000000000506c4] .copy_files+0x400/0x414 (unreliable)
> [c0000000f3313bd0] [c00000000005161c] .copy_process+0x660/0x12bc
> [c0000000f3313ce0] [c000000000052318] .do_fork+0xa0/0x25c
> [c0000000f3313dc0] [c0000000000159c8] .sys_clone+0x5c/0x74
> [c0000000f3313e30] [c000000000010a88] .ppc_clone+0x8/0xc
>
> The problem seems to be that one of the file pointers is breifly
> set to (int32)-1 even on a 64-bit machine. The part of copy_process()
> that gets mashed by this is:
>
> for (i = open_files; i != 0; i--) {
> struct file *f = *old_fds++;
> if (f)
> get_file(f); <== derefs f, which is -1
> *new_fds++ = f;
> }
>
> By inserting if(f==(void*)0xffffffffUL) printk ...
> I can find out that i=230 is the one with the problem and open_files=256!
> I haven't yet found who set struct file * to a -1.
>
> I'm generting this behaviour with a hotplug event that is causing ifdown
> and ifup to run simultaneously. (The device driver was shut down and
> restarted, causing simultaneous hotplug events). Although the above
> stack shows ifdown getting clobbered, I've also seen pci.agent be
> the process that suffers.
>
> The problem goes away if I insert a sleep of about half-a-second or more
> between the device driver shutdown and startup.
>
> Affected machine is a ppc64 power4 box. I've seen the problem
> for a long time (months?), including monday's bk clone of
> bkbits of 2.6.9-rc2; waiting for this bug "to fix itself" doesn't
> seem to be working.
By adding appropriate printk's to copy_files(), it appears that
many many processes have a -1 in the highest non-zero fd pointer.
The crash that I see seems to happen when there is another nearby
fd that is open, (nearby==within the same 8-bit bitmask) so that the
"open_files" value is larger than/equal-to the highest non-zero
fd pointer (which is set to -1), thus leading to crash. The "race"
is that these high fd's normally close pretty quickly, and thus,
"open_files" usually has a much smaller value.
--linas