2004-06-09 21:06:17

by stian

[permalink] [raw]
Subject: timer + fpu stuff locks my console race

Please keep me in CC as I'm not on the mailinglist. I'm currently on a
vaccation, so I can't hook my linux-box to the Internet, but I came across
a race condition in the "old" 2.4.26-rc1 vanilla kernel.

I'm doing some code tests when I came across problems with my program
locking my console (even X if I'm using a xterm).

I think first of all gcc triggers the problem, so the full report is here:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15905

For more details about versions and other information needed, please let
me know if needed. It triggers at every attempt at my box currently (and
I'm lacking Internet connection at the time-being on my machine).



Stian Skjelstad


2004-06-10 19:02:10

by Lars Age Kamfjord

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

ACK on 2.6.5 (fedora core 2 vanilla)

Totally locked my X window system.

Lars Age Kamfjord

> Please keep me in CC as I'm not on the mailinglist. I'm currently on a
> vaccation, so I can't hook my linux-box to the Internet, but I came
across
> a race condition in the "old" 2.4.26-rc1 vanilla kernel.

> I'm doing some code tests when I came across problems with my program
> locking my console (even X if I'm using a xterm).

> I think first of all gcc triggers the problem, so the full report is
here:
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15905

> Stian Skjelstad

2004-06-10 19:21:34

by Lars Age Kamfjord

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

Throwing in a ACK on 2.4.18-bf2.4 (debian woody vanilla) from ssh

The guy I crashed probably hates me now, but I warned him last week he
should give away shellaccounts to everyone he knows.......

Lars Age Kamfjord
BOFH

Lars Age Kamfjord wrote:

> ACK on 2.6.5 (fedora core 2 vanilla)
>
> Totally locked my X window system.
>
> Lars Age Kamfjord
>
> > Please keep me in CC as I'm not on the mailinglist. I'm currently on a
> > vaccation, so I can't hook my linux-box to the Internet, but I came
> across
> > a race condition in the "old" 2.4.26-rc1 vanilla kernel.
>
> > I'm doing some code tests when I came across problems with my program
> > locking my console (even X if I'm using a xterm).
>
> > I think first of all gcc triggers the problem, so the full report is
> here:
> > http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15905
>
> > Stian Skjelstad
> -
> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/


2004-06-10 19:28:34

by Bård Kalbakk

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

ACK on 2.6.7-rc2 singel CPU.

But, with 2.4.23 SMP it seems to be okay. I can't kill the process or attach to it with strace, but it doesn't lock the machine.

B?rd Kalbakk

2004-06-10 21:01:03

by Matias Hermanrud Fjeld

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

ACK

mhf@bilbo:~$ uname -a
Linux bilbo 2.6.6-1-k7 #1 Wed May 12 18:19:40 EST 2004 i686 GNU/Linux

--
Matias Hermanrud Fjeld
http://www.hex.no/mhf


Attachments:
(No filename) (149.00 B)
signature.asc (189.00 B)
Digital signature
Download all attachments

2004-06-11 06:08:50

by Lars Age Kamfjord

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

This bug seems VERY serious.... Every machine I've tested with so far
has crashed totally; and it happens with every version of 2.4 and
2.6-kernels. I've tested with a 2.2.19-kernel, and that didn't crash, so
it seems to be a bug in 2.4 and later..... Somebody really should look
at this......


Lars Age Kamfjord

2004-06-11 12:10:54

by stian

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

UML seems to not be affected, but it produces Floating Point Exception and
kills the program. Better respons than what happens when running on the
host (x86).

Seems like the kernel is still alive, but doesn't want to context switch
in user-space programs any more and io-schedules also stops.


Stian Skjelstad

2004-06-11 12:20:11

by Gard Spreemann

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

ACK on kernel 2.6.6 single CPU.
This seems scaringly serious!

-- Gard

2004-06-12 02:53:57

by Rik van Riel

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

On Wed, 9 Jun 2004 [email protected] wrote:

> I'm doing some code tests when I came across problems with my program
> locking my console (even X if I'm using a xterm).

Reproduced here, on my test system running a 2.6 kernel.
I did get a kernel backtrace over serial console, though ;)

Pid: 19752, comm: kernel-hang-bz1
EIP: 0060:[<ffff345c>] CPU: 0
EIP is at 0xffff345c
EFLAGS: 00000202 Not tainted (2.6.5-1.332)
EAX: 00000001 EBX: 12005870 ECX: fef32ea8 EDX: 1958f000
ESI: 1958f000 EDI: fef32ea8 EBP: fef32e48 DS: 007b ES: 007b
CR0: 80050033 CR2: 00c4b720 CR3: 003ab000 CR4: 000006d0
Call Trace:
[<0210dcda>] restore_i387_fxsave+0x18/0x60
[<0210dd38>] restore_i387+0x16/0x65
[<021059e5>] restore_sigcontext+0xf2/0x10c
[<0215b737>] get_user_size+0x30/0x57
[<02105c13>] sys_sigreturn+0x214/0x23a


--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

2004-06-12 03:50:30

by Rik van Riel

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

On Fri, 11 Jun 2004, Rik van Riel wrote:

> Reproduced here, on my test system running a 2.6 kernel.
> I did get a kernel backtrace over serial console, though ;)

With a 2.4 kernel I get a similar stack trace (also
on alt-sysrq-p) output:

Pid/TGid: 3815/3815, comm: kernel-hang-bz1
EIP: 0060:[<c03ec1cc>] CPU: 0
EIP is at coprocessor_error [kernel] 0x0 (2.4.21-15.5.ELsmp)
ESP: 0060:c0113d14 EFLAGS: 00000206 Not tainted
EAX: 00100000 EBX: bfffc888 ECX: bfffc888 EDX: d9818000
ESI: bfffc888 EDI: d9819fb0 EBP: bfffc830 DS: 0068 ES: 0068 FS: 0000
GS: 0033
CR0: 80050033 CR2: b7566720 CR3: 02553380 CR4: 000006f0
Call Trace: [<c0113d14>] restore_i387_fxsave [kernel] 0x24 (0xd9819ee4)
[<c0113de8>] restore_i387 [kernel] 0x78 (0xd9819f04)
[<c010b40e>] restore_sigcontext [kernel] 0x10e (0xd9819f18)
[<c010b51d>] sys_sigreturn [kernel] 0xed (0xd9819f94)

Now I'm not sure if the process is actually stuck in kernel
space or if it's looping tightly through both kernel and
user space...

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

2004-06-12 04:36:00

by Matt Mackall

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

On Fri, Jun 11, 2004 at 10:53:48PM -0400, Rik van Riel wrote:
> On Wed, 9 Jun 2004 [email protected] wrote:
>
> > I'm doing some code tests when I came across problems with my program
> > locking my console (even X if I'm using a xterm).
>
> Reproduced here, on my test system running a 2.6 kernel.
> I did get a kernel backtrace over serial console, though ;)

I stuck some strategic printks in the kernel. The example code's bogus
asm is generating an FPU fault in frstor in its signal handler, that's
bumping us into math_error -> force_sig_info ->
specific_send_sig_info. Then we hit:

if (LEGACY_QUEUE(&t->pending, sig))

which decides we don't need to send the signal after all and we bail
all the way back out and recurse.

--
Mathematics is the supreme nostalgia of our time.

2004-06-12 12:26:59

by stian

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

So far I have found out this:

if you ptrace is with for instace the strace program, it runs perfectly.
No signs at all of the fpu exception, and every thing runs happy

it also happens if you for instance if you trigger the exception inside a
SIGSEGV handler

But I'm not able to trigger other FPU errors. For instance
float a=1.0;
float b=0.0;
float c;
c=a/b;
does not generate a signal, but gives (inf) (isn't this configuration
option on the fpu?). So my question is then, does the FPU-exception
handler work at all since it appears to be rarely used?

A very _VERY_ nasty quick-fix (for those who are scared) is to exit the
process if we want to send a signal SIGFPE and is it already in the queue
and perhaps do a printk() about user trying to exploit known kernel-bug.
Works atleast for me currently at my 2.4.26-rc1 box.


Stian Skjelstad

2004-06-12 13:14:13

by stian

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

Can somebody test if this does the job for atleast the 2.4.x series?
Perhaps something alike for the 2.6.x aswell. (Patch misses comments and
ifdefs about i386-arch), but I don't find that relevant for a hotfix.

Stian Skjelstad

diff -ur linux-2.4.26/kernel/signal.c linux-2.4.26-fpuhotfix/kernel/signal.c
--- linux-2.4.26/kernel/signal.c 2004-02-18 14:36:32.000000000 +0100
+++ linux-2.4.26-fpuhotfix/kernel/signal.c 2004-06-12
15:11:07.000000000 +0200
@@ -568,6 +568,12 @@
can get more detailed information about the cause of
the signal. */
if (sig < SIGRTMIN && sigismember(&t->pending.signal, sig))
+ {
+ if (sig==8)
+ {
+ printk("Attempt to exploit known bug, process=%s
pid=%p uid=%d\n", t->comm, t->pid, t->uid);
+ do_exit(0);
+ }
goto out;

ret = deliver_signal(sig, info, t);



2004-06-12 13:28:51

by stian

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

Forgot to update the diff file after I fixed some bogus stuff. This patch
file compiles. Please report if it works or not for 2.4.26 (I'm lacking
that damn Internett connection on my linux box). So much for vaccation.

Stian Skjelstad

diff -ur linux-2.4.26/kernel/signal.c linux-2.4.26-fpuhotfix/kernel/signal.c
--- linux-2.4.26/kernel/signal.c 2004-02-18 14:36:32.000000000 +0100
+++ linux-2.4.26-fpuhotfix/kernel/signal.c 2004-06-12
15:26:10.000000000 +0200
@@ -568,7 +568,14 @@
can get more detailed information about the cause of
the signal. */
if (sig < SIGRTMIN && sigismember(&t->pending.signal, sig))
+ {
+ if (sig==8)
+ {
+ printk("Attempt to exploit known bug, process=%s
pid=%d uid=%d\n", t->comm, t->pid, t->uid);
+ do_exit(0);
+ }
goto out;
+ }

ret = deliver_signal(sig, info, t);
out:

2004-06-12 13:44:39

by Sergey Vlasov

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

On Fri, 11 Jun 2004 23:50:25 -0400, Rik van Riel wrote:

> On Fri, 11 Jun 2004, Rik van Riel wrote:
>
>> Reproduced here, on my test system running a 2.6 kernel.
>> I did get a kernel backtrace over serial console, though ;)
>
> Now I'm not sure if the process is actually stuck in kernel
> space or if it's looping tightly through both kernel and
> user space...

Here is the culprit (include/asm-i386/i387.h):

#define __clear_fpu( tsk ) \
do { \
if ((tsk)->thread_info->status & TS_USEDFPU) { \
asm volatile("fwait"); \
(tsk)->thread_info->status &= ~TS_USEDFPU; \
stts(); \
} \
} while (0)

This is called in flush_thread() (which is used in flush_old_exec()
and therefore in sys_execve() path) and in restore_i387_fsave(),
restore_i387_fxsave() (which are reached from sys_sigreturn() and
sys_rt_sigreturn()).

The buggy code in the Stian's program corrupts the FPU state - in
particular, it results in some exception bits being set in the FPU
status word. In this state the next FP command (except non-waiting
commands, like fnsave and fninit) will raise the FP error exception
(trap 16). The "fwait" above happens to be that next command.

The FP error handler do_coprocessor_error() calls math_error() for
real work (both in arch/i386/traps.c). math_error() calls
save_init_fpu(), which saves the FPU state in current->thread.i387 and
sets the TS flag; then math_error() queues a SIGFPE to the task and
returns. If the fault comes from userspace, this is enough - on the
return path the pending signal will be noticed and delivered.
However, in this case the fault happens in the kernel code, therefore
execution just resumes at the same point - trying to reexecute that
fwait again.

At this time, however, the TS flag is set, so we get another trap -
trap 7, device_not_available. The trap handler calls
math_state_restore(), which clears the TS flag and reloads the FP
state from current->thread.i387. Then it returns, and the faulting
instruction is restarted again. But it gets the same FP error
exception as at the first time...

So the CPU is stuck handling endless faults in kernel mode.

How to fix this? A quick and dirty fix is to remove the problematic
fwait from __clear_fpu(); 2.2.x kernels did not have it - probably it
was added in some 2.3.x.

--- linux-2.6.6/include/asm-i386/i387.h.fp-lockup 2004-05-10 06:33:06 +0400
+++ linux-2.6.6/include/asm-i386/i387.h 2004-06-12 17:25:56 +0400
@@ -51,7 +51,6 @@
#define __clear_fpu( tsk ) \
do { \
if ((tsk)->thread_info->status & TS_USEDFPU) { \
- asm volatile("fwait"); \
(tsk)->thread_info->status &= ~TS_USEDFPU; \
stts(); \
} \

In this case we will ignore a pending FP exception at execve() or
sigreturn() instead of raising SIGFPE (which was probably intended by
whoever put an fwait there).

If we want to be pedantic and care about such pending exceptions, we
should add a check for kernel addresses to do_coprocessor_error() and
add fixup_exception there, like we do for protection faults, so that
the handler will not attempt to restart the failing instruction again.


Attachments:
(No filename) (3.06 kB)
(No filename) (189.00 B)
Download all attachments

2004-06-12 13:50:18

by Kalin KOZHUHAROV

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

[email protected] wrote:
> Forgot to update the diff file after I fixed some bogus stuff. This patch
> file compiles. Please report if it works or not for 2.4.26 (I'm lacking
> that damn Internett connection on my linux box). So much for vaccation.
>
> Stian Skjelstad
>
> diff -ur linux-2.4.26/kernel/signal.c linux-2.4.26-fpuhotfix/kernel/signal.c
> --- linux-2.4.26/kernel/signal.c 2004-02-18 14:36:32.000000000 +0100
> +++ linux-2.4.26-fpuhotfix/kernel/signal.c 2004-06-12
> 15:26:10.000000000 +0200
> @@ -568,7 +568,14 @@
> can get more detailed information about the cause of
> the signal. */
> if (sig < SIGRTMIN && sigismember(&t->pending.signal, sig))
> + {
> + if (sig==8)
> + {
> + printk("Attempt to exploit known bug, process=%s
> pid=%d uid=%d\n", t->comm, t->pid, t->uid);
> + do_exit(0);
> + }
> goto out;
> + }
>
> ret = deliver_signal(sig, info, t);
> out:

Does this work for 2.6.{6,7} ?

Kalin.

--
||///_ o *****************************
||//'_/> WWW: http://ThinRope.net/
|||\/<"
|||\\ '
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2004-06-12 13:57:53

by stian

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

> --- linux-2.6.6/include/asm-i386/i387.h.fp-lockup 2004-05-10 06:33:06
> +0400
> +++ linux-2.6.6/include/asm-i386/i387.h 2004-06-12 17:25:56 +0400
> @@ -51,7 +51,6 @@
> #define __clear_fpu( tsk ) \
> do { \
> if ((tsk)->thread_info->status & TS_USEDFPU) { \
> - asm volatile("fwait"); \
> (tsk)->thread_info->status &= ~TS_USEDFPU; \
> stts(); \
> } \

But what about task-switching and fpu-exceptions that comes in late? I
know that the kernel does not use FPU in general, and the places it does,
fsave, fwait and frstor embeddes it all in kernel-space.


Stian Skjelstad

2004-06-12 13:58:20

by Manuel Arostegui Ramirez

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

El S?bado 12 Junio 2004 15:28, [email protected] escribi?:
> Forgot to update the diff file after I fixed some bogus stuff. This patch
> file compiles. Please report if it works or not for 2.4.26 (I'm lacking
> that damn Internett connection on my linux box). So much for vaccation.
>
> Stian Skjelstad
>
> diff -ur linux-2.4.26/kernel/signal.c
> linux-2.4.26-fpuhotfix/kernel/signal.c --- linux-2.4.26/kernel/signal.c
> 2004-02-18 14:36:32.000000000 +0100 +++
> linux-2.4.26-fpuhotfix/kernel/signal.c 2004-06-12
> 15:26:10.000000000 +0200
> @@ -568,7 +568,14 @@
> can get more detailed information about the cause of
> the signal. */
> if (sig < SIGRTMIN && sigismember(&t->pending.signal, sig))
> + {
> + if (sig==8)
> + {
> + printk("Attempt to exploit known bug, process=%s
> pid=%d uid=%d\n", t->comm, t->pid, t->uid);
> + do_exit(0);
> + }
> goto out;
> + }
>
> ret = deliver_signal(sig, info, t);
> out:

I'm going to try the patch on a 2.4.20-8 in about one hour.
Thanks

--
Manuel Arostegui Ramirez #Linux Registered User 200896

2004-06-12 14:28:20

by Sergey Vlasov

[permalink] [raw]
Subject: Re: timer + fpu stuff locks my console race

On Sat, Jun 12, 2004 at 03:57:42PM +0200, [email protected] wrote:
> > --- linux-2.6.6/include/asm-i386/i387.h.fp-lockup 2004-05-10 06:33:06
> > +0400
> > +++ linux-2.6.6/include/asm-i386/i387.h 2004-06-12 17:25:56 +0400
> > @@ -51,7 +51,6 @@
> > #define __clear_fpu( tsk ) \
> > do { \
> > if ((tsk)->thread_info->status & TS_USEDFPU) { \
> > - asm volatile("fwait"); \
> > (tsk)->thread_info->status &= ~TS_USEDFPU; \
> > stts(); \
> > } \
>
> But what about task-switching and fpu-exceptions that comes in late? I
> know that the kernel does not use FPU in general, and the places it does,
> fsave, fwait and frstor embeddes it all in kernel-space.

Kernel code which uses FPU should call kernel_fpu_begin() before it
and kernel_fpu_end() after. kernel_fpu_begin() is safe - it uses
fnsave or fxsave, both of which don't raise pending FPU exceptions.
Also fnsave performs implicit fninit, and fxsave is followed by
fnclex, which clears pending exceptions.

However, raid6_before_mmx() [drivers/md/raid6x86.h] seems to be buggy:

static inline void raid6_before_mmx(raid6_mmx_save_t *s)
{
s->cr0 = raid6_get_fpu();
asm volatile("fsave %0 ; fwait" : "=m" (s->fsave[0]));
}

fsave will raise pending exceptions (unlike fnsave).


Attachments:
(No filename) (1.24 kB)
(No filename) (189.00 B)
Download all attachments

2004-06-12 14:26:03

by Alexander Nyberg

[permalink] [raw]
Subject: Re: timer + fpu stuff locks up computer

On Sat, 2004-06-12 at 15:44, Sergey Vlasov wrote:
> On Fri, 11 Jun 2004 23:50:25 -0400, Rik van Riel wrote:
>
> > On Fri, 11 Jun 2004, Rik van Riel wrote:
> >
> >> Reproduced here, on my test system running a 2.6 kernel.
> >> I did get a kernel backtrace over serial console, though ;)
> >
> > Now I'm not sure if the process is actually stuck in kernel
> > space or if it's looping tightly through both kernel and
> > user space...
>
> --- linux-2.6.6/include/asm-i386/i387.h.fp-lockup 2004-05-10 06:33:06 +0400
> +++ linux-2.6.6/include/asm-i386/i387.h 2004-06-12 17:25:56 +0400
> @@ -51,7 +51,6 @@
> #define __clear_fpu( tsk ) \
> do { \
> if ((tsk)->thread_info->status & TS_USEDFPU) { \
> - asm volatile("fwait"); \
> (tsk)->thread_info->status &= ~TS_USEDFPU; \
> stts(); \
> } \

Sorry for this extremely informative mail but, doesn't work.

Looks like the problem is only being delayed:

Pid: 431, comm: sshd
EIP: 0060:[<c0119f98>] CPU: 0
EIP is at force_sig_info+0x48/0x80
EFLAGS: 00000286 Not tainted (2.6.7-rc3-mm1)
EAX: 00000000 EBX: de96d7d0 ECX: 00000007 EDX: 00000008
ESI: 00000008 EDI: 00000286 EBP: de9e3dd4 DS: 007b ES: 007b
CR0: 8005003b CR2: 080b2664 CR3: 1f48f000 CR4: 000002d0
[<c0105560>] do_coprocessor_error+0x0/0x20
[<c01054f2>] math_error+0xb2/0x120
[<c01d2bb8>] fast_clear_page+0x8/0x50
[<c0105de3>] do_IRQ+0x113/0x150
[<c0105de3>] do_IRQ+0x113/0x150
[<c0105de3>] do_IRQ+0x113/0x150
[<c0104398>] common_interrupt+0x18/0x20
[<c0109ed5>] restore_fpu+0x15/0x20
[<c0104435>] error_code+0x2d/0x38
[<c01d2bb8>] fast_clear_page+0x8/0x50
[<c013286e>] do_anonymous_page+0x8e/0x140
[<c0132979>] do_no_page+0x59/0x290
[<c0132d5e>] handle_mm_fault+0xbe/0x120
[<c010e5b4>] do_page_fault+0x134/0x506
[<c010fd90>] default_wake_function+0x0/0x10
[<c01f4f6a>] tty_read+0xaa/0xf0
[<c014dd3d>] sys_select+0x22d/0x490
[<c013e583>] vfs_read+0xc3/0x100
[<c011b0ac>] sigprocmask+0x4c/0xb0
[<c010e480>] do_page_fault+0x0/0x506
[<c0104435>] error_code+0x2d/0x38

2004-06-12 14:42:47

by stian

[permalink] [raw]
Subject: Re: timer + fpu stuff locks up computer

> Sorry for this extremely informative mail but, doesn't work.
>
> Looks like the problem is only being delayed:

Makes sense, since fwait is done in kernel-mode and it takes some time for
the exception to rise, since this is a slow instruction. So the problem
gets delayed. What do you think Sergey?

Does the other dirty nasty patch work for you?


Stian Skjelstad

2004-06-12 15:14:42

by Sergey Vlasov

[permalink] [raw]
Subject: Re: timer + fpu stuff locks up computer

On Sat, Jun 12, 2004 at 04:25:51PM +0200, Alexander Nyberg wrote:
> > --- linux-2.6.6/include/asm-i386/i387.h.fp-lockup 2004-05-10 06:33:06 +0400
> > +++ linux-2.6.6/include/asm-i386/i387.h 2004-06-12 17:25:56 +0400
> > @@ -51,7 +51,6 @@
> > #define __clear_fpu( tsk ) \
> > do { \
> > if ((tsk)->thread_info->status & TS_USEDFPU) { \
> > - asm volatile("fwait"); \
> > (tsk)->thread_info->status &= ~TS_USEDFPU; \
> > stts(); \
> > } \
>
> Sorry for this extremely informative mail but, doesn't work.
>
> Looks like the problem is only being delayed:
>
> Pid: 431, comm: sshd
> EIP: 0060:[<c0119f98>] CPU: 0
> EIP is at force_sig_info+0x48/0x80
> EFLAGS: 00000286 Not tainted (2.6.7-rc3-mm1)
> EAX: 00000000 EBX: de96d7d0 ECX: 00000007 EDX: 00000008
> ESI: 00000008 EDI: 00000286 EBP: de9e3dd4 DS: 007b ES: 007b
> CR0: 8005003b CR2: 080b2664 CR3: 1f48f000 CR4: 000002d0
> [<c0105560>] do_coprocessor_error+0x0/0x20
> [<c01054f2>] math_error+0xb2/0x120
> [<c01d2bb8>] fast_clear_page+0x8/0x50
...

Grrr. I was testing on a fairly generic kernel configuration which
did not include fast_clear_page()...

If the FPU state belong to the userspace process, kernel_fpu_begin()
is safe even if some exceptions are pending. However, after
__clear_fpu() the FPU is "orphaned", and kernel_fpu_begin() does
nothing with it.

Replacing fwait with fnclex instead of removing it completely should
avoid the fault later. However, looks like we really need the proper
fix - teach do_coprocessor_error() to recognize kernel mode faults and
fixup them.


Attachments:
(No filename) (1.57 kB)
(No filename) (189.00 B)
Download all attachments

2004-06-12 15:21:07

by martin capitanio

[permalink] [raw]
Subject: Re: timer + fpu stuff locks up computer

On Saturday 12 June 2004 16:42, [email protected] wrote:
>
> Does the other dirty nasty patch work for you?

ACK for 2.6.7-rc4-mm1 (gcc-Version 3.3.3)
user$ ./evil
completely freeze

--- linux-2.6.6-rc3-mm1/kernel/signal.c 2004-06-09 18:36:12.000000000 +0200
+++ linux-2.6.6-rc3-mm1-fpuhotfix/kernel/signal.c 2004-06-12 18:10:31.573001808 +0200
@@ -799,7 +799,15 @@
can get more detailed information about the cause of
the signal. */
if (LEGACY_QUEUE(&t->pending, sig))
+ {
+ if (sig==8)
+ {
+ printk("Attempt to exploit known bug, process=%s pid=%p uid=%d\n",
+ t->comm, t->pid, t->uid);
+ do_exit(0);
+ }
goto out;
+ }

ret = send_signal(sig, info, t, &t->pending);
if (!ret && !sigismember(&t->blocked, sig))

2.6.7-rc4-mm1-fpuhotfix:
user$ ./evil
........................*...............................................
......................*
Attempt to exploit known bug, process=evil pid=00000aa6 uid=1000
note: evil[2726] exited with preempt_count 2
bad: scheduling while atomic!
[<c032a045>] schedule+0x4b5/0x4c0
[<c01435cb>] zap_pmd_range+0x4b/0x70
[<c014362d>] unmap_page_range+0x3d/0x70
[<c014380b>] unmap_vmas+0x1ab/0x1c0
[<c0147639>] exit_mmap+0x79/0x150
[<c01184ee>] mmput+0x5e/0xa0
[<c011c523>] do_exit+0x153/0x3e0
[<c0122e6f>] specific_send_sig_info+0xff/0x100
[<c0122eb2>] force_sig_info+0x42/0x90
[<c0105be0>] do_coprocessor_error+0x0/0x20
[<c0105b5e>] math_error+0xde/0x160
[<c010b0f6>] restore_i387_fxsave+0x26/0xa0
[<c0222c8c>] write_chan+0x18c/0x250
[<c01170e0>] default_wake_function+0x0/0x10
[<c01170e0>] default_wake_function+0x0/0x10
[<c0104a05>] error_code+0x2d/0x38
[<c010b0f6>] restore_i387_fxsave+0x26/0xa0
[<c010b1fc>] restore_i387+0x8c/0x90
[<c0103434>] restore_sigcontext+0x114/0x130
[<c0103503>] sys_sigreturn+0xb3/0xd0
[<c0103f6b>] syscall_call+0x7/0xb

but it keeps the kernel alive :-)

martin

2004-06-12 16:15:43

by stian

[permalink] [raw]
Subject: Re: timer + fpu stuff locks up computer

>> Does the other dirty nasty patch work for you?
> --- linux-2.6.6-rc3-mm1/kernel/signal.c 2004-06-09 18:36:12.000000000
> +0200
> +++ linux-2.6.6-rc3-mm1-fpuhotfix/kernel/signal.c 2004-06-12
> 18:10:31.573001808 +0200
> @@ -799,7 +799,15 @@
> can get more detailed information about the cause of
> the signal. */
> if (LEGACY_QUEUE(&t->pending, sig))
> + {
> + if (sig==8)
> + {
> + printk("Attempt to exploit known bug, process=%s pid=%p
> uid=%d\n",
> + t->comm, t->pid, t->uid);
> + do_exit(0);
> + }
> goto out;
> + }
>
> ret = send_signal(sig, info, t, &t->pending);
> if (!ret && !sigismember(&t->blocked, sig))
>
> 2.6.7-rc4-mm1-fpuhotfix:
> user$ ./evil
> ........................*...............................................
> ......................*
> Attempt to exploit known bug, process=evil pid=00000aa6 uid=1000
> note: evil[2726] exited with preempt_count 2
> bad: scheduling while atomic!
> [<c032a045>] schedule+0x4b5/0x4c0
> [<c01435cb>] zap_pmd_range+0x4b/0x70
> [<c014362d>] unmap_page_range+0x3d/0x70
> [<c014380b>] unmap_vmas+0x1ab/0x1c0
> [<c0147639>] exit_mmap+0x79/0x150
> [<c01184ee>] mmput+0x5e/0xa0
> [<c011c523>] do_exit+0x153/0x3e0
> [<c0122e6f>] specific_send_sig_info+0xff/0x100
> [<c0122eb2>] force_sig_info+0x42/0x90
> [<c0105be0>] do_coprocessor_error+0x0/0x20
> [<c0105b5e>] math_error+0xde/0x160
> [<c010b0f6>] restore_i387_fxsave+0x26/0xa0
> [<c0222c8c>] write_chan+0x18c/0x250
> [<c01170e0>] default_wake_function+0x0/0x10
> [<c01170e0>] default_wake_function+0x0/0x10
> [<c0104a05>] error_code+0x2d/0x38
> [<c010b0f6>] restore_i387_fxsave+0x26/0xa0
> [<c010b1fc>] restore_i387+0x8c/0x90
> [<c0103434>] restore_sigcontext+0x114/0x130
> [<c0103503>] sys_sigreturn+0xb3/0xd0
> [<c0103f6b>] syscall_call+0x7/0xb
>
> but it keeps the kernel alive :-)

The hotfix should probably me moved to arch/i386/traps.c before we start
to due atomic locks, sinse it is beond dirty to kill the process here when
we have locked down resources. But the best would be to fix the
problem-source, since this is just a workaround.


Stian Skjelstad

2004-06-12 18:45:29

by Sergey Vlasov

[permalink] [raw]
Subject: Re: timer + fpu stuff locks up computer

On Sat, Jun 12, 2004 at 07:14:22PM +0400, Sergey Vlasov wrote:
> If the FPU state belong to the userspace process, kernel_fpu_begin()
> is safe even if some exceptions are pending. However, after
> __clear_fpu() the FPU is "orphaned", and kernel_fpu_begin() does
> nothing with it.
>
> Replacing fwait with fnclex instead of removing it completely should
> avoid the fault later.

Yes, it seems to be enough. Another case where it looks like FPU
might be "orphaned" is exit(); however, it is handled as a normal task
switch, __switch_to() calls __unlazy_fpu(), which clears pending
exceptions.

I'm still not sure what to do about possibly lost FP exceptions. This
can happen in two cases:

1) Program calls execve() while an FP exception is pending.

In this case clear_fpu() is called when the original executable is
already destroyed. Even if we generate a SIGFPE in this case, it
would be delivered to the new executable.

2) Program returns from a signal handler while an FP exception is
pending.

In this case at clear_fpu() time restore_sigcontext() has already
wiped out all state of the signal handler, so the SIGFPE would
appear to be raised from the program code at the point where it was
interrupted by the handled signal.

Signed-Off-By: Sergey Vlasov <[email protected]>

--- linux-2.6.6/include/asm-i386/i387.h.fp-lockup 2004-05-10 06:33:06 +0400
+++ linux-2.6.6/include/asm-i386/i387.h 2004-06-12 22:02:58 +0400
@@ -48,10 +48,17 @@
save_init_fpu( tsk ); \
} while (0)

+/*
+ * There might be some pending exceptions in the FP state at this point.
+ * However, it is too late to report them: this code is called during execve()
+ * (when the original executable is already gone) and during sigreturn() (when
+ * the signal handler context is already lost). So just clear them to prevent
+ * problems later.
+ */
#define __clear_fpu( tsk ) \
do { \
if ((tsk)->thread_info->status & TS_USEDFPU) { \
- asm volatile("fwait"); \
+ asm volatile("fnclex"); \
(tsk)->thread_info->status &= ~TS_USEDFPU; \
stts(); \
} \


Attachments:
(No filename) (2.05 kB)
(No filename) (189.00 B)
Download all attachments

2004-06-12 20:27:38

by Alexander Nyberg

[permalink] [raw]
Subject: Re: timer + fpu stuff locks up computer

On Sat, 2004-06-12 at 20:45, Sergey Vlasov wrote:
> On Sat, Jun 12, 2004 at 07:14:22PM +0400, Sergey Vlasov wrote:
> > If the FPU state belong to the userspace process, kernel_fpu_begin()
> > is safe even if some exceptions are pending. However, after
> > __clear_fpu() the FPU is "orphaned", and kernel_fpu_begin() does
> > nothing with it.
> >
> > Replacing fwait with fnclex instead of removing it completely should
> > avoid the fault later.
>
> Yes, it seems to be enough. Another case where it looks like FPU
> might be "orphaned" is exit(); however, it is handled as a normal task
> switch, __switch_to() calls __unlazy_fpu(), which clears pending
> exceptions.
>
> I'm still not sure what to do about possibly lost FP exceptions. This
> can happen in two cases:
>
> 1) Program calls execve() while an FP exception is pending.
>
> In this case clear_fpu() is called when the original executable is
> already destroyed. Even if we generate a SIGFPE in this case, it
> would be delivered to the new executable.
>
> 2) Program returns from a signal handler while an FP exception is
> pending.
>
> In this case at clear_fpu() time restore_sigcontext() has already
> wiped out all state of the signal handler, so the SIGFPE would
> appear to be raised from the program code at the point where it was
> interrupted by the handled signal.
>
> Signed-Off-By: Sergey Vlasov <[email protected]>
>
> --- linux-2.6.6/include/asm-i386/i387.h.fp-lockup 2004-05-10 06:33:06 +0400
> +++ linux-2.6.6/include/asm-i386/i387.h 2004-06-12 22:02:58 +0400
> @@ -48,10 +48,17 @@
> save_init_fpu( tsk ); \
> } while (0)
>
> +/*
> + * There might be some pending exceptions in the FP state at this point.
> + * However, it is too late to report them: this code is called during execve()
> + * (when the original executable is already gone) and during sigreturn() (when
> + * the signal handler context is already lost). So just clear them to prevent
> + * problems later.
> + */
> #define __clear_fpu( tsk ) \
> do { \
> if ((tsk)->thread_info->status & TS_USEDFPU) { \
> - asm volatile("fwait"); \
> + asm volatile("fnclex"); \
> (tsk)->thread_info->status &= ~TS_USEDFPU; \
> stts(); \
> } \
>

This works, tested also on a box with md and things looked fine.


Alex