2006-10-11 16:54:53

by Günther Starnberger

[permalink] [raw]
Subject: Userspace process may be able to DoS kernel

[I'm not subscribed on this list - please CC answers to me.]

Hello,

It seems that the latest version of Skype may exhibit a problem in the
kernel where a non-root userspace process is able to block the whole
system for durations of up to several minutes. If someone is able to
reproduce the steps which cause the problem he may be able to DoS a
system by consecutively causing soft lockups.

There were some reports of this problem on other lists before, but
mostly on tainted systems. I was able to reproduce this problem on a
non-tainted mostly vanilla 2.6.17.6 kernel (it includes the suspend2
patches). As most users who reported this problem are using Ubuntu,
the problem may be related to one of the settings in Ubuntu's kernel
config. The configuration of my kernel is also based on the Ubuntu
kernel config. As I am not using the patched kernel by Ubuntu I hope
that the LKML is the right place to report this issue.

The lockup usually occurs when Skype 1.3.x for Linux (I'm using
1.3.0.53) sits around idle for some time and then (presumably) uses
the sound device (i.e. for me it happens when I call a contact -
others reported this problem occurs for incoming messages [there may
be an audio notification of the messages enabled]). The lockup can
take from several seconds (where it is not detected by the kernel) up
to some minutes. The whole system seems to be blocked - i.e. there is
not even disk IO.

dmesg reports the following:
BUG: soft lockup detected on CPU#0!
<c01562cd> softlockup_tick+0xad/0xf0 <c012e609> update_process_times+0x39/0x90
<c011600b> smp_apic_timer_interrupt+0x5b/0x70 <c0110037>
get_offset_pmtmr+0x97/0x1060
<c0103d20> apic_timer_interrupt+0x1c/0x24 <c013d390> hrtimer_get_res+0x0/0x60
<c0110037> get_offset_pmtmr+0x97/0x1060 <c0106b9f> do_gettimeofday+0x1f/0xd0
<c0129654> getnstimeofday+0x14/0x40 <c01398d1> sys_clock_gettime+0x31/0xb0
<c01031e7> sysenter_past_esp+0x54/0x75

A copy of my kernel config is available at:
http://virtual.sysfrog.org/~gst/config.txt

The hardware where this problem occurs here is a X41 Thinkpad (Pentium
M Dothan). There are also reports on other non-Intel CPUs e.g. AMD
Turion 64.

Please see the more extensive documentation of this problem on
https://launchpad.net/distros/ubuntu/+source/linux-source-2.6.15/+bug/53216
(although some of the people there use tainted kernels).

I will upgrade my 2.6.17.6 kernel to 2.6.18 and try to reproduce the
problem there in the following days (but I fear that I won't have much
time before the weekend).

bye,
/gst


2006-10-12 06:03:06

by Joerg Platte

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

Am Mittwoch, 11. Oktober 2006 18:54 schrieb Günther Starnberger:

> I will upgrade my 2.6.17.6 kernel to 2.6.18 and try to reproduce the
> problem there in the following days (but I fear that I won't have much
> time before the weekend).

Using 2.6.18 does not solve the problem. I can see exactly the same behavior
with a vanilla and not tainted 2.6.18 kernel.

regards,
Jörg

2006-10-12 06:49:23

by Willy Tarreau

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

On Thu, Oct 12, 2006 at 08:02:58AM +0200, Joerg Platte wrote:
> Am Mittwoch, 11. Oktober 2006 18:54 schrieb G?nther Starnberger:
>
> > I will upgrade my 2.6.17.6 kernel to 2.6.18 and try to reproduce the
> > problem there in the following days (but I fear that I won't have much
> > time before the weekend).
>
> Using 2.6.18 does not solve the problem. I can see exactly the same behavior
> with a vanilla and not tainted 2.6.18 kernel.

Just out of curiosity, does the system still respond to ping during this
period, and does the time still progress during the lock up ? Not that it
could help me find out what's happening, but it might help troubleshooting
the problem.

Regards,
Willy

2006-10-12 10:54:11

by Joerg Platte

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

Am Donnerstag, 12. Oktober 2006 08:49 schrieb Willy Tarreau:
> On Thu, Oct 12, 2006 at 08:02:58AM +0200, Joerg Platte wrote:
> > Using 2.6.18 does not solve the problem. I can see exactly the same
> > behavior with a vanilla and not tainted 2.6.18 kernel.
>
> Just out of curiosity, does the system still respond to ping during this
> period, and does the time still progress during the lock up ? Not that it
> could help me find out what's happening, but it might help troubleshooting
> the problem.

Pinging the system works and the the time is still correct. But this time I
was not able to trigger a kernel warning, because I was not able to generate
a long lockup. Maybe I'll have to wait a few more hours before trying it
again, because the duration of the lockup is proportional to the time skype
is running.

regards,
J?rg

2006-10-12 11:30:16

by Pekka Enberg

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

Hi Joerg,

On 10/12/06, Joerg Platte <[email protected]> wrote:
> Using 2.6.18 does not solve the problem. I can see exactly the same behavior
> with a vanilla and not tainted 2.6.18 kernel.

Do you see this with 2.6.16 also or is it new to 2.6.17?

Pekka

2006-10-12 11:42:04

by Joerg Platte

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

Am Donnerstag, 12. Oktober 2006 13:30 schrieb Pekka Enberg:
Hi,

> On 10/12/06, Joerg Platte <[email protected]> wrote:
> > Using 2.6.18 does not solve the problem. I can see exactly the same
> > behavior with a vanilla and not tainted 2.6.18 kernel.
>
> Do you see this with 2.6.16 also or is it new to 2.6.17?

Hmmm, I deleted all my 2.6.16 kernels and I can't test a newly compiled 2.6.16
kernel before the weekend. But if I remember correctly, on previous kernel
versions skype just generates 100% system load when using the sound device
after some time (especially after a resume) and stuttered audio but no system
lockups. Hence, it worked much better than now from the kernel point of view
but was not usable from the skype users point of view. It was a userspace
only problem.

regards,
J?rg

2006-10-12 11:57:40

by Pekka Enberg

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

On 10/12/06, Joerg Platte <[email protected]> wrote:
> Hmmm, I deleted all my 2.6.16 kernels and I can't test a newly compiled 2.6.16
> kernel before the weekend. But if I remember correctly, on previous kernel
> versions skype just generates 100% system load when using the sound device
> after some time (especially after a resume) and stuttered audio but no system
> lockups. Hence, it worked much better than now from the kernel point of view
> but was not usable from the skype users point of view. It was a userspace
> only problem.

If that is the case, you can do git bisect to help us narrow down the
cause. See http://www.kernel.org/pub/software/scm/git/docs/howto/isolate-bugs-with-bisect.txt
for further details.

Pekka

2006-10-12 15:50:36

by Lee Revell

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

On Wed, 2006-10-11 at 12:54 -0400, G?nther Starnberger wrote:
> The lockup usually occurs when Skype 1.3.x for Linux (I'm using
> 1.3.0.53) sits around idle for some time and then (presumably) uses
> the sound device (i.e. for me it happens when I call a contact -
> others reported this problem occurs for incoming messages [there may
> be an audio notification of the messages enabled]). The lockup can
> take from several seconds (where it is not detected by the kernel) up
> to some minutes. The whole system seems to be blocked - i.e. there is
> not even disk IO.

Do you get the same behavior using the old OSS drivers that you get with
ALSA's OSS emulation?

Lee

2006-10-12 15:55:12

by Lee Revell

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

On Wed, 2006-10-11 at 12:54 -0400, G?nther Starnberger wrote:
> As most users who reported this problem are using Ubuntu,
> the problem may be related to one of the settings in Ubuntu's kernel
> config. The configuration of my kernel is also based on the Ubuntu
> kernel config. As I am not using the patched kernel by Ubuntu I hope
> that the LKML is the right place to report this issue.

IIRC Ubuntu is the only major distro that ships with CONFIG_PREEMPT=y.
Any difference if you disable it?

Lee

2006-10-12 16:11:31

by Jan Engelhardt

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel


>> As most users who reported this problem are using Ubuntu,
>> the problem may be related to one of the settings in Ubuntu's kernel
>> config. The configuration of my kernel is also based on the Ubuntu
>> kernel config. As I am not using the patched kernel by Ubuntu I hope
>> that the LKML is the right place to report this issue.
>
>IIRC Ubuntu is the only major distro that ships with CONFIG_PREEMPT=y.
>Any difference if you disable it?

SUSE uses CONFIG_PREEMPT(?) Voluntary Preemption too.


-`J'
--

2006-10-12 16:18:35

by Lee Revell

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

On Thu, 2006-10-12 at 18:10 +0200, Jan Engelhardt wrote:
> >> As most users who reported this problem are using Ubuntu,
> >> the problem may be related to one of the settings in Ubuntu's kernel
> >> config. The configuration of my kernel is also based on the Ubuntu
> >> kernel config. As I am not using the patched kernel by Ubuntu I hope
> >> that the LKML is the right place to report this issue.
> >
> >IIRC Ubuntu is the only major distro that ships with CONFIG_PREEMPT=y.
> >Any difference if you disable it?
>
> SUSE uses CONFIG_PREEMPT(?) Voluntary Preemption too.

CONFIG_PREEMPT_VOLUNTARY != CONFIG_PREEMPT

2006-10-12 16:55:09

by Günther Starnberger

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

On 10/12/06, Lee Revell <[email protected]> wrote:

> Do you get the same behavior using the old OSS drivers that you get with
> ALSA's OSS emulation?

No - not yet. The problem occurs when I use ALSA directly as well as
with ALSA's OSS emulation.

I will check if there is an OSS driver for my soundcard so that I can
try this out.

bye,
/gst

2006-10-12 17:05:07

by Lee Revell

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

On Thu, 2006-10-12 at 12:55 -0400, G?nther Starnberger wrote:
> On 10/12/06, Lee Revell <[email protected]> wrote:
>
> > Do you get the same behavior using the old OSS drivers that you get with
> > ALSA's OSS emulation?
>
> No - not yet. The problem occurs when I use ALSA directly as well as
> with ALSA's OSS emulation.
>

Ah, OK, I did not realize that Skype had (FINALLY) released a version
with ALSA support...

> I will check if there is an OSS driver for my soundcard so that I can
> try this out.
>


2006-10-12 20:11:23

by Joerg Platte

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

Am Donnerstag, 12. Oktober 2006 13:57 schrieb Pekka Enberg:
> If that is the case, you can do git bisect to help us narrow down the
> cause. See
> http://www.kernel.org/pub/software/scm/git/docs/howto/isolate-bugs-with-bis
>ect.txt for further details.

Not that easy, since it takes a few hours to be able to trigger the bug. I
tried to record the system calls with strace but without success. Skype did
not cause any lockups and then crashes... Maybe the timing is too different
with strace.

regards,
J?rg

2006-10-12 20:25:59

by Günther Starnberger

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

On 10/12/06, Joerg Platte <[email protected]> wrote:

> Not that easy, since it takes a few hours to be able to trigger the bug. I
> tried to record the system calls with strace but without success. Skype did
> not cause any lockups and then crashes... Maybe the timing is too different
> with strace.

According to [1] there are several anti-debugging techniques used in
Skype. I.e. if it notices some sort of debugger it will crash (on
purpose).

bye,
/gst

[1] http://www.blackhat.com/presentations/bh-europe-06/bh-eu-06-biondi/bh-eu-06-biondi-up.pdf

2006-10-12 20:30:07

by Günther Starnberger

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

On 10/12/06, Lee Revell <[email protected]> wrote:

> Do you get the same behavior using the old OSS drivers that you get with
> ALSA's OSS emulation?

Yes. I've rmmod'ed ALSA and used the i810_audio OSS module instead.
Same problem.

bye,
/gst

2006-10-12 20:36:37

by Lee Revell

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

On Thu, 2006-10-12 at 16:30 -0400, G?nther Starnberger wrote:
> On 10/12/06, Lee Revell <[email protected]> wrote:
>
> > Do you get the same behavior using the old OSS drivers that you get with
> > ALSA's OSS emulation?
>
> Yes. I've rmmod'ed ALSA and used the i810_audio OSS module instead.
> Same problem.

OK, so the sound subsystem has been ruled out. That just leaves...
everything else ;-)

Lee

2006-10-12 22:04:40

by Jan Engelhardt

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel


>> >IIRC Ubuntu is the only major distro that ships with CONFIG_PREEMPT=y.
>> >Any difference if you disable it?
>>
>> SUSE uses CONFIG_PREEMPT(?) Voluntary Preemption too.
>
>CONFIG_PREEMPT_VOLUNTARY != CONFIG_PREEMPT

But modinfo (vermagic) prints PREEMPT at least the former case too,
which is a little bit misleading.


-`J'
--

2006-10-13 13:24:37

by Joerg Platte

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

Am Donnerstag, 12. Oktober 2006 22:25 schrieb Günther Starnberger:
Hi,

> According to [1] there are several anti-debugging techniques used in
> Skype. I.e. if it notices some sort of debugger it will crash (on
> purpose).

I know. I just wanted to know if I'm able to catch a blocking system call...

> [1]
> http://www.blackhat.com/presentations/bh-europe-06/bh-eu-06-biondi/bh-eu-06-biondi
>-up.pdf -

Here is my output from a longer lockup today. I can confirm, that pings and
system time are not affected by the lockup. I didn't try to log in via ssh
but I think that will not work, too.

Oct 13 15:11:08 localhost kernel: BUG: soft lockup detected on CPU#0!
Oct 13 15:11:08 localhost kernel: [<c012f9d5>] softlockup_tick+0x89/0xa4
Oct 13 15:11:08 localhost kernel: [<c011b458>] update_process_times+0x35/0x57
Oct 13 15:11:08 localhost kernel: [<c01052c9>] timer_interrupt+0x35/0x64
Oct 13 15:11:08 localhost kernel: [<c012fc13>] handle_IRQ_event+0x23/0x49
Oct 13 15:11:08 localhost kernel: [<c012fe45>] __do_IRQ+0x7b/0xde
Oct 13 15:11:08 localhost kernel: [<c010438a>] do_IRQ+0x40/0x4d
Oct 13 15:11:08 localhost kernel: [<c0102d82>] common_interrupt+0x1a/0x20
Oct 13 15:11:08 localhost kernel: [<c01e3f0e>] acpi_pm_read+0x7/0xf
Oct 13 15:11:09 localhost kernel: [<c011a53d>] getnstimeofday+0x2b/0xac
Oct 13 15:11:09 localhost kernel: [<c01226f4>] sys_clock_gettime+0x42/0x7e
Oct 13 15:11:09 localhost kernel: [<c0102bfb>] syscall_call+0x7/0xb
Oct 13 15:11:09 localhost kernel: BUG: soft lockup detected on CPU#0!
Oct 13 15:11:09 localhost kernel: [<c012f9d5>] softlockup_tick+0x89/0xa4
Oct 13 15:11:09 localhost kernel: [<c011b458>] update_process_times+0x35/0x57
Oct 13 15:11:09 localhost kernel: [<c01052c9>] timer_interrupt+0x35/0x64
Oct 13 15:11:09 localhost kernel: [<c012fc13>] handle_IRQ_event+0x23/0x49
Oct 13 15:11:09 localhost kernel: [<c012fe45>] __do_IRQ+0x7b/0xde
Oct 13 15:11:09 localhost kernel: [<c010438a>] do_IRQ+0x40/0x4d
Oct 13 15:11:09 localhost kernel: [<c0102d82>] common_interrupt+0x1a/0x20
Oct 13 15:11:09 localhost kernel: [<c01e3f0e>] acpi_pm_read+0x7/0xf
Oct 13 15:11:09 localhost kernel: [<c011a53d>] getnstimeofday+0x2b/0xac
Oct 13 15:11:09 localhost kernel: [<c01226f4>] sys_clock_gettime+0x42/0x7e
Oct 13 15:11:09 localhost kernel: [<c0102bfb>] syscall_call+0x7/0xb

Typically I have two lockups, a first longer one and then a shorter one. Then
everything runs without any problems.

regards,
Jörg

--
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605

2006-11-11 12:29:25

by Joerg Platte

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

Am Freitag, 10. November 2006 08:19 schrieb Andrew Morton:

> OK, thanks.
>
> It'd be useful if you could grab a kernel profile when the system load is
> high:

> Or, if oprofile is working:
>
>
> #!/bin/sh
> sudo opcontrol --stop
> sudo opcontrol --shutdown
> sudo rm -rf /var/lib/oprofile
> sudo opcontrol --vmlinux=/boot/vmlinux-$(uname -r)
> sudo opcontrol --start-daemon
> sudo opcontrol --start
> sleep 10
> sudo opcontrol --stop
> sudo opcontrol --shutdown
> sudo opreport -l /boot/vmlinux-$(uname -r) | head -50

Here is the oprofile log. Seems to be acpi related?

CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
samples % symbol name
709 44.2848 acpi_pm_read
232 14.4909 schedule
164 10.2436 system_call
61 3.8101 __wake_up
48 2.9981 __copy_to_user_ll
42 2.6234 do_futex
29 1.8114 futex_wake
29 1.8114 sys_futex
26 1.6240 hash_futex
25 1.5615 getnstimeofday
20 1.2492 preempt_schedule
20 1.2492 sys_clock_gettime
18 1.1243 copy_to_user
17 1.0618 __copy_from_user_ll
17 1.0618 get_futex_key
16 0.9994 futex_requeue
15 0.9369 schedule_timeout
11 0.6871 __mod_timer
9 0.5621 do_gettimeofday
9 0.5621 sys_ioctl
9 0.5621 syscall_exit
8 0.4997 fget_light
7 0.4372 find_extend_vma
6 0.3748 find_vma
5 0.3123 copy_from_user
5 0.3123 lock_timer_base
5 0.3123 sys_gettimeofday
4 0.2498 do_ioctl
4 0.2498 up_read
4 0.2498 vfs_ioctl
4 0.2498 wake_futex
3 0.1874 add_wait_queue
3 0.1874 down_read
2 0.1249 memcpy
2 0.1249 profile_hit
1 0.0625 csum_partial
1 0.0625 do_page_fault
1 0.0625 dummy_file_ioctl
1 0.0625 fput
1 0.0625 handle_IRQ_event
1 0.0625 ip_append_data
1 0.0625 memcmp
1 0.0625 netif_receive_skb
1 0.0625 permission
1 0.0625 sched_clock
1 0.0625 syscall_call
1 0.0625 unmap_vmas


I captured this on my IBM Thinkpad T40p. Here is the system configuration:

Linux ibm 2.6.19-rc5 #1 PREEMPT Wed Nov 8 08:06:17 CET 2006 i686 GNU/Linux

Module Size Used by
sg 32156 0
sr_mod 14820 0
lt_hotswap 10888 0
oprofile 18400 1
radeon 109728 2
drm 69524 3 radeon
binfmt_misc 10696 1
ieee80211_crypt_ccmp 6912 3
cpufreq_userspace 3860 0
cpufreq_powersave 1792 0
rfcomm 34716 1
l2cap 21700 5 rfcomm
bluetooth 47780 4 rfcomm,l2cap
nfs 210280 0
nfsd 198320 17
exportfs 5440 1 nfsd
lockd 57480 3 nfs,nfsd
sunrpc 146108 12 nfs,nfsd,lockd
nsc_ircc 17296 0
uinput 8704 1
af_packet 19848 6
autofs4 19588 2
video 15172 0
sbs 14496 0
i2c_ec 4928 1 sbs
dock 7240 0
button 6544 0
container 4352 0
ac 5060 0
battery 9860 0
ipt_MASQUERADE 3328 3
iptable_nat 6724 1
ip_nat 17004 2 ipt_MASQUERADE,iptable_nat
xt_state 2112 9
ipt_LOG 6080 8
xt_limit 2624 8
ipt_REJECT 4288 2
xt_mark 1856 2
xt_tcpudp 2880 10
xt_mac 1856 29
iptable_filter 2880 1
xt_MARK 2240 3
xt_multiport 3008 8
iptable_mangle 2752 1
ip_tables 12552 3 iptable_nat,iptable_filter,iptable_mangle
x_tables 14276 12
ipt_MASQUERADE,iptable_nat,xt_state,ipt_LOG,xt_limit,ipt_REJECT,xt_mark,xt_tcpudp,xt_mac,xt_MARK,xt_multiport,ip_tables
ip_conntrack_ftp 7376 0
ip_conntrack 48524 5
ipt_MASQUERADE,iptable_nat,ip_nat,xt_state,ip_conntrack_ftp
nfnetlink 6360 2 ip_nat,ip_conntrack
deflate 3712 0
zlib_deflate 18072 1 deflate
zlib_inflate 13632 1 deflate
twofish 8384 0
twofish_common 35904 1 twofish
serpent 18816 0
aes 27968 3
blowfish 9280 0
des 17344 0
cbc 4288 0
ecb 3456 0
blkcipher 5504 2 cbc,ecb
sha256 11008 0
sha1 2560 0
crypto_null 2496 0
af_key 31696 2
nls_utf8 2048 1
ntfs 92788 1
nls_base 7168 2 nls_utf8,ntfs
ext2 59464 1
dm_snapshot 15328 0
dm_mirror 17936 0
dm_mod 50520 2 dm_snapshot,dm_mirror
deadline_iosched 5440 0
as_iosched 12616 1
cfq_iosched 16208 1
cdc_acm 14048 0
capability 4744 0
commoncap 6848 1 capability
ircomm_tty 22664 0
ircomm 13060 1 ircomm_tty
tun 10368 1
nvram 8072 1
ibm_acpi 25792 0
sd_mod 18576 0
8250_pci 19904 0
irtty_sir 6016 0
sir_dev 13956 1 irtty_sir
joydev 9024 0
snd_intel8x0m 16524 4
snd_seq_oss 28992 0
snd_seq_midi 8160 0
snd_rawmidi 22432 1 snd_seq_midi
snd_seq_midi_event 6784 2 snd_seq_oss,snd_seq_midi
snd_seq 44688 5 snd_seq_oss,snd_seq_midi,snd_seq_midi_event
tsdev 7424 0
snd_intel8x0 31004 2
snd_ac97_codec 89188 2 snd_intel8x0m,snd_intel8x0
snd_ac97_bus 2240 1 snd_ac97_codec
usb_storage 56576 0
snd_pcm_oss 38432 0
snd_mixer_oss 15424 1 snd_pcm_oss
snd_seq_device 7628 4 snd_seq_oss,snd_seq_midi,snd_rawmidi,snd_seq
ipw2200 136584 0
libusual 16016 1 usb_storage
snd_pcm 71048 6
snd_intel8x0m,snd_intel8x0,snd_ac97_codec,snd_pcm_oss
snd_timer 20676 2 snd_seq,snd_pcm
yenta_socket 24780 2
rsrc_nonstatic 12032 1 yenta_socket
pcmcia 34788 0
psmouse 34376 0
irda 106040 4 nsc_ircc,ircomm_tty,ircomm,sir_dev
i2c_i801 7308 0
usbhid 47008 0
8250_pnp 9088 0
8250 20004 2 8250_pci,8250_pnp
serial_core 19584 1 8250
ieee80211 29832 1 ipw2200
ieee80211_crypt 5824 2 ieee80211_crypt_ccmp,ieee80211
parport_pc 35940 0
parport 33288 1 parport_pc
iTCO_wdt 10016 0
pcspkr 2816 0
evdev 9152 3
intel_agp 22236 1
agpgart 29744 2 drm,intel_agp
serio_raw 6468 0
crc_ccitt 2112 1 irda
snd 47972 21
snd_intel8x0m,snd_seq_oss,snd_rawmidi,snd_seq,snd_intel8x0,snd_ac97_codec,snd_pcm_oss,snd_mixer_oss,snd_seq_device,snd_pcm,snd_timer
soundcore 7776 1 snd
snd_page_alloc 9608 3 snd_intel8x0m,snd_intel8x0,snd_pcm
ff_memless 5128 1 usbhid
rtc 12340 0
pcmcia_core 37520 3 yenta_socket,rsrc_nonstatic,pcmcia
firmware_class 9664 2 ipw2200,pcmcia
ext3 120904 2
jbd 53736 1 ext3
mbcache 8324 2 ext2,ext3
ide_cd 36192 0
cdrom 32992 2 sr_mod,ide_cd
ide_disk 15232 6
ata_piix 15368 0
libata 96276 1 ata_piix
scsi_mod 128588 5 sg,sr_mod,sd_mod,usb_storage,libata
piix 9156 0 [permanent]
uhci_hcd 21256 0
ehci_hcd 27976 0
usbcore 121924 7
cdc_acm,usb_storage,libusual,usbhid,uhci_hcd,ehci_hcd
generic 5316 0 [permanent]
ide_core 109532 6
lt_hotswap,usb_storage,ide_cd,ide_disk,piix,generic
e1000 108672 0
thermal 13640 0
fan 4612 0
unix 25328 1098
cpufreq_conservative 6368 0
cpufreq_ondemand 7168 1
speedstep_centrino 7120 1
freq_table 4292 2 cpufreq_ondemand,speedstep_centrino
processor 23276 2 thermal,speedstep_centrino
fbcon 38304 73
tileblit 2496 1 fbcon
crc32 4288 3 tun,pcmcia,fbcon
font 8256 1 fbcon
bitblit 5120 1 fbcon
softcursor 2240 1 bitblit
radeonfb 94144 1
fb 43688 5 fbcon,tileblit,bitblit,softcursor,radeonfb
fb_ddc 2560 1 radeonfb
i2c_algo_bit 7560 1 radeonfb
i2c_core 20688 4 i2c_ec,i2c_i801,fb_ddc,i2c_algo_bit
cfbcopyarea 3520 1 radeonfb
cfbimgblt 2816 1 radeonfb
cfbfillrect 3520 1 radeonfb


00:00.0 Host bridge: Intel Corporation 82855PM Processor to I/O Controller
(rev 03)
00:01.0 PCI bridge: Intel Corporation 82855PM Processor to AGP Controller (rev
03)
00:1d.0 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M)
USB UHCI Controller #1 (rev 01)
00:1d.1 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M)
USB UHCI Controller #2 (rev 01)
00:1d.2 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M)
USB UHCI Controller #3 (rev 01)
00:1d.7 USB Controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI
Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 81)
00:1f.0 ISA bridge: Intel Corporation 82801DBM (ICH4-M) LPC Interface Bridge
(rev 01)
00:1f.1 IDE interface: Intel Corporation 82801DBM (ICH4-M) IDE Controller (rev
01)
00:1f.3 SMBus: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) SMBus
Controller (rev 01)
00:1f.5 Multimedia audio controller: Intel Corporation 82801DB/DBL/DBM
(ICH4/ICH4-L/ICH4-M) AC'97 Audio Controller (rev 01)
00:1f.6 Modem: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) AC'97
Modem Controller (rev 01)
01:00.0 VGA compatible controller: ATI Technologies Inc Radeon R250 [Mobility
FireGL 9000] (rev 02)
02:00.0 CardBus bridge: Texas Instruments PCI1520 PC card Cardbus Controller
(rev 01)
02:00.1 CardBus bridge: Texas Instruments PCI1520 PC card Cardbus Controller
(rev 01)
02:01.0 Ethernet controller: Intel Corporation 82540EP Gigabit Ethernet
Controller (Mobile) (rev 03)
02:02.0 Network controller: Intel Corporation PRO/Wireless 2200BG Network
Connection (rev 05)

regards,
J?rg

2006-11-11 12:39:37

by Arjan van de Ven

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

On Sat, 2006-11-11 at 13:29 +0100, Joerg Platte wrote:
> Am Freitag, 10. November 2006 08:19 schrieb Andrew Morton:
>
> > OK, thanks.
> >
> > It'd be useful if you could grab a kernel profile when the system load is
> > high:
>
> > Or, if oprofile is working:
> >
> >
> > #!/bin/sh
> > sudo opcontrol --stop
> > sudo opcontrol --shutdown
> > sudo rm -rf /var/lib/oprofile
> > sudo opcontrol --vmlinux=/boot/vmlinux-$(uname -r)
> > sudo opcontrol --start-daemon
> > sudo opcontrol --start
> > sleep 10
> > sudo opcontrol --stop
> > sudo opcontrol --shutdown
> > sudo opreport -l /boot/vmlinux-$(uname -r) | head -50
>
> Here is the oprofile log. Seems to be acpi related?
>
> CPU: CPU with timer interrupt, speed 0 MHz (estimated)
> Profiling through timer interrupt
> samples % symbol name
> 709 44.2848 acpi_pm_read



this isn't per se acpi related: This is reading the PM timer from your
chipset. The PMTimer is a clock on your chipset that the kernel can use
to read a stable incrementing clock to find out what time it is right
now, usually as part of userspace asking the kernel what time it is via
the gettimeofday() system call. ACPI is just the component that does the
actual (slow) hardware access... eg the messenger.

Normally systems have better/faster clocks than the pmtimer, but there
are circumstances where those can't be used.

1) HPET. The HPET is a lot faster than pmtimer, and very reliable. Most
of the systems sold in the last 3 years have an hpet, but unfortunately,
many bioses turn this off by default. If your BOOS has a "Multimedia
timer" setting, make sure it's set to "On".
2) TSC. This is a super fast method of finding how much time has passed,
since it's inside the CPU. However there are many reasons why this
method may be unreliable, for example certain powermanagement features
on laptops cause this clock to stop when idle (not useful), or to vary
in frequency (also not useful if you want to find out what time it is).
Also on AMD Opteron SMP systems or extreme Intel big honking NUMA
systems, this timer is not synchronized between the various processors
and that breaks the current time keeping in Linux, and so Linux doesn't
use it in that case.

So my advice is
1) Check the bios to see if you have the HPET enabled. If not, enable
it.
2) Check the kernel config to see if you have HPET enabled there, if not
enable it.
3) Check dmesg to see if there's a reason the kernel doesn't use TSC;
there is probably nothing you can do but at least you know why :)


--
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

2006-11-11 13:15:39

by Joerg Platte

[permalink] [raw]
Subject: Re: Userspace process may be able to DoS kernel

Am Samstag, 11. November 2006 13:39 schrieb Arjan van de Ven:

> this isn't per se acpi related: This is reading the PM timer from your
> chipset. The PMTimer is a clock on your chipset that the kernel can use
> to read a stable incrementing clock to find out what time it is right
> now, usually as part of userspace asking the kernel what time it is via
> the gettimeofday() system call. ACPI is just the component that does the
> actual (slow) hardware access... eg the messenger.

OK.

> Normally systems have better/faster clocks than the pmtimer, but there
> are circumstances where those can't be used.
>
> 1) HPET. The HPET is a lot faster than pmtimer, and very reliable. Most
> of the systems sold in the last 3 years have an hpet, but unfortunately,
> many bioses turn this off by default. If your BOOS has a "Multimedia
> timer" setting, make sure it's set to "On".

My computer is 3,5 years old (one of the first centrino notebooks). Maybe it
does not have a HPET timer. I can't find HPET somewhere in the kernel.log
file and no option in the BIOS. But it is enabled in my kernel config.

> 2) TSC. This is a super fast method of finding how much time has passed,
> since it's inside the CPU. However there are many reasons why this
> method may be unreliable, for example certain powermanagement features
> on laptops cause this clock to stop when idle (not useful), or to vary
> in frequency (also not useful if you want to find out what time it is).
> Also on AMD Opteron SMP systems or extreme Intel big honking NUMA
> systems, this timer is not synchronized between the various processors
> and that breaks the current time keeping in Linux, and so Linux doesn't
> use it in that case.

I'm using frequency scaling. Maybe that's a reason for not using TSC in each
case.

> So my advice is
> 1) Check the bios to see if you have the HPET enabled. If not, enable
> it.
> 2) Check the kernel config to see if you have HPET enabled there, if not
> enable it.
> 3) Check dmesg to see if there's a reason the kernel doesn't use TSC;
> there is probably nothing you can do but at least you know why :)

The kernel semm to use TSC. I can't find another message stating that TSC has
been disabled.

localhost kernel: Time: tsc clocksource has been installed.

There seem to be some clock drift. Each time when starting skype everything
works perfect for a couple of hours. Then, skype behaves strange by causing
this high system load.

regards,
J?rg