Hi,
I have a system with an "AMD64 Phenom 9500" quad core cpu, 4GB RAM,
"ASUS M3A32 MVP Deluxe wifi" motherboard with latest vendor BIOS (0801).
I tried stock debian etch kernel (Debian 2.6.18.dfsg.1-18etch1), machine
froze with no message, debian etch backport kernel same, and then
Debian 2.6.24-4 from unstable and I got some messages: machine
is not frozen but some userland processes are (ps says "Dl" state
with child in "Zs" state) and "events/3" is taking 100% cpu
according to top:
18 root 15 -5 0 0 0 R 100 0.0 74:59.46 events/3
Got to the same state with ubuntu hardy 2.6.24-8-server kernel. All
kernels are untainted, no X running anyway.
It takes a few hours of doing some stuff, in my case bootstraping or
testing GCC at -j 4, and then the problem happens.
I did 32 hours of memtest without issue on this system, temperatures
are very low and the case has plenty of airflow, making memory
issue less likely.
Here is what I found by looking around:
"BUG: soft lockup with kernel 2.6.23.12 (x86-64)"
http://lkml.org/lkml/2008/1/11/98
"[BUG] 2.6.24-git6 soft lockup detected while running libhugetlbfs"
http://lkml.org/lkml/2008/1/30/30
(this one has some discussion)
Bug I reported in ubuntu BTS "BUG: soft lockup - CPU#3 stuck for 11s!
[events/3:18]"
https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/197252
"BUG: soft lockup - CPU#0 stuck for 11s! [swapper:0]"
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=464387
Let me know if you need more information or if there's something
I should try. ssh and root access to this box is possible for known
developpers (send me your ssh public key), there's nothing of value on
it, I'm trying to make this system stable for use in the GCC compile
farm (which is open for free software projects):
http://gcc.gnu.org/wiki/CompileFarm
Thanks in advance,
Laurent
Mar 8 21:36:28 gcc04 kernel: BUG: soft lockup - CPU#3 stuck for 11s! [events/3:18]
Mar 8 21:36:28 gcc04 kernel: CPU 3:
Mar 8 21:36:28 gcc04 kernel: Modules linked in: tun nfs nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs ac battery ipv6 loop ath5k mac80211 serio_raw snd_hda_intel snd_pcm snd_timer snd soundcore snd_page_alloc floppy psmouse pcspkr cfg80211 button i2c_piix4 i2c_core sky2 evdev ext3 jbd mbcache dm_mirror dm_snapshot dm_mod ata_generic sd_mod usbhid hid ahci pata_marvell firewire_ohci firewire_core crc_itu_t libata r8169 atiixp ehci_hcd ohci_hcd scsi_mod generic ide_core thermal processor fan
Mar 8 21:36:28 gcc04 kernel: Pid: 18, comm: events/3 Not tainted 2.6.24-1-amd64 #1
Mar 8 21:36:28 gcc04 kernel: RIP: 0010:[<ffffffff8021bc75>] [<ffffffff8021bc75>] __smp_call_function_mask+0x9b/0xbf
Mar 8 21:36:28 gcc04 kernel: RSP: 0000:ffff81012b773e00 EFLAGS: 00000297
Mar 8 21:36:28 gcc04 kernel: RAX: 00000000000008fc RBX: 0000000000000003 RCX: 0000000000000001
Mar 8 21:36:28 gcc04 kernel: RDX: 00000000000008fc RSI: 00000000000000fc RDI: 0000000000000007
Mar 8 21:36:28 gcc04 kernel: RBP: ffff81000103c8c0 R08: ffff81012b772000 R09: ffff810001040ee0
Mar 8 21:36:28 gcc04 kernel: R10: ffff81010484c800 R11: 0000000000000000 R12: 0000000280428760
Mar 8 21:36:28 gcc04 kernel: R13: 0000000000000000 R14: ffff81012b773d90 R15: ffff81012b773e24
Mar 8 21:36:28 gcc04 kernel: FS: 00002b97e47336d0(0000) GS:ffff81012b6b58c0(0000) knlGS:0000000000000000
Mar 8 21:36:28 gcc04 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Mar 8 21:36:28 gcc04 kernel: CR2: 00002b97e4c6d008 CR3: 0000000000201000 CR4: 00000000000006e0
Mar 8 21:36:28 gcc04 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 8 21:36:28 gcc04 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 8 21:36:28 gcc04 kernel:
Mar 8 21:36:28 gcc04 kernel: Call Trace:
Mar 8 21:36:28 gcc04 kernel: [<ffffffff8021611c>] mcheck_check_cpu+0x0/0x37
Mar 8 21:36:28 gcc04 kernel: [<ffffffff8021611c>] mcheck_check_cpu+0x0/0x37
Mar 8 21:36:28 gcc04 kernel: [<ffffffff8021bcf7>] smp_call_function_mask+0x5e/0x70
Mar 8 21:36:28 gcc04 kernel: [<ffffffff802159c2>] mcheck_timer+0x0/0x7c
Mar 8 21:36:28 gcc04 kernel: [<ffffffff8021611c>] mcheck_check_cpu+0x0/0x37
Mar 8 21:36:28 gcc04 kernel: [<ffffffff8023a0cb>] on_each_cpu+0x10/0x22
Mar 8 21:36:28 gcc04 kernel: [<ffffffff802159df>] mcheck_timer+0x1d/0x7c
Mar 8 21:36:28 gcc04 kernel: [<ffffffff8027cb05>] vmstat_update+0x0/0x31
Mar 8 21:36:28 gcc04 kernel: [<ffffffff80244695>] run_workqueue+0x7f/0x10b
Mar 8 21:36:28 gcc04 kernel: [<ffffffff80244fa7>] worker_thread+0x0/0xe4
Mar 8 21:36:28 gcc04 kernel: [<ffffffff80245081>] worker_thread+0xda/0xe4
Mar 8 21:36:28 gcc04 kernel: [<ffffffff80247ff2>] autoremove_wake_function+0x0/0x2e
Mar 8 21:36:28 gcc04 kernel: [<ffffffff80247ed3>] kthread+0x47/0x74
Mar 8 21:36:28 gcc04 kernel: [<ffffffff8020cc48>] child_rip+0xa/0x12
Mar 8 21:36:28 gcc04 kernel: [<ffffffff80247e8c>] kthread+0x0/0x74
Mar 8 21:36:28 gcc04 kernel: [<ffffffff8020cc3e>] child_rip+0x0/0x12
Mar 8 21:36:28 gcc04 kernel:
Mar 8 21:36:40 gcc04 kernel: BUG: soft lockup - CPU#3 stuck for 11s! [events/3:18]
(repeated about every 11s)
# lspci
00:00.0 Host bridge: ATI Technologies Inc RD790 Northbridge only dual slot PCI-e_GFX and HT3 K8 part
00:02.0 PCI bridge: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx0 port A)
00:04.0 PCI bridge: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port A)
00:06.0 PCI bridge: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port C)
00:07.0 PCI bridge: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port D)
00:09.0 PCI bridge: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port E)
00:12.0 SATA controller: ATI Technologies Inc SB600 Non-Raid-5 SATA
00:13.0 USB Controller: ATI Technologies Inc SB600 USB (OHCI0)
00:13.1 USB Controller: ATI Technologies Inc SB600 USB (OHCI1)
00:13.2 USB Controller: ATI Technologies Inc SB600 USB (OHCI2)
00:13.3 USB Controller: ATI Technologies Inc SB600 USB (OHCI3)
00:13.4 USB Controller: ATI Technologies Inc SB600 USB (OHCI4)
00:13.5 USB Controller: ATI Technologies Inc SB600 USB Controller (EHCI)
00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 14)
00:14.1 IDE interface: ATI Technologies Inc SB600 IDE
00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia
00:14.3 ISA bridge: ATI Technologies Inc SB600 PCI to LPC Bridge
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Link Control
01:00.0 VGA compatible controller: ATI Technologies Inc RV515 [Radeon X1600]
01:00.1 Display controller: ATI Technologies Inc Unknown device 7160
02:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6121 SATA II Controller (rev b1)
03:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 12)
04:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6121 SATA II Controller (rev b2)
05:00.0 Ethernet controller: Atheros Communications, Inc. AR242x 802.11abg Wireless PCI Express Adapter (rev 01)
06:08.0 FireWire (IEEE 1394): Agere Systems FW323 (rev 70)
On Sat, 2008-03-08 at 23:00 +0100, Laurent GUERBY wrote:
> Hi,
>
> I have a system with an "AMD64 Phenom 9500" quad core cpu, 4GB RAM,
> "ASUS M3A32 MVP Deluxe wifi" motherboard with latest vendor BIOS
> (0801).
>
> I tried stock debian etch kernel (Debian 2.6.18.dfsg.1-18etch1),
> machine
> froze with no message, debian etch backport kernel same, and then
> Debian 2.6.24-4 from unstable and I got some messages: machine
> is not frozen but some userland processes are (ps says "Dl" state
> with child in "Zs" state) and "events/3" is taking 100% cpu
> according to top:
>
> 18 root 15 -5 0 0 0 R 100 0.0 74:59.46
> events/3
>
> Got to the same state with ubuntu hardy 2.6.24-8-server kernel. All
> kernels are untainted, no X running anyway.
>
> It takes a few hours of doing some stuff, in my case bootstraping or
> testing GCC at -j 4, and then the problem happens.
On 2.6.24-1-amd64 (Debian 2.6.24-4) I got a slightly different
backtrace in /var/log/messages after a few hours of stressing the
machine with compilations, see below. The given process
was stuck and unkillable.
Any idea on what to do/try?
Thanks in advance,
Laurent
BUG: soft lockup - CPU#3 stuck for 11s! [sh:7652]
CPU 3:
Modules linked in: nfs tun nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs ac battery ipv6 it87 hwmon_vid eeprom loop snd_hda_intel snd_pcm snd_timer snd serio_raw i2c_piix4 soundcore snd_page_alloc ath5k pcspkr psmouse i2c_core mac80211 cfg80211 button sky2 evdev ext3 jbd mbcache dm_mirror dm_snapshot dm_mod sd_mod ata_generic firewire_ohci firewire_core atiixp crc_itu_t ehci_hcd ahci pata_marvell ohci_hcd libata scsi_mod generic ide_core thermal processor fan
Pid: 7652, comm: sh Not tainted 2.6.24-1-amd64 #1
RIP: 0010:[<ffffffff8021bc79>] [<ffffffff8021bc79>] __smp_call_function_mask+0x9f/0xbf
RSP: 0018:ffff81000131bd68 EFLAGS: 00000297
RAX: 00000000000008fc RBX: 0000000000000003 RCX: 0000000000000001
RDX: 00000000000008fc RSI: 00000000000000fc RDI: 0000000000000007
RBP: ffff81000000e000 R08: ffff81011bab5982 R09: ffff81000131bce8
R10: ffff81000131bce8 R11: 0000000000000062 R12: 0000000000000000
R13: 00000000000000d0 R14: ffff81000131bcb8 R15: ffff81000131bcb8
FS: 00002b61abbe4ae0(0000) GS:ffff81011bab58c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000005e1808 CR3: 0000000001303000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8021bc64>] __smp_call_function_mask+0x8a/0xbf
[<ffffffff80275939>] smp_drain_local_pages+0x0/0x5
[<ffffffff80275939>] smp_drain_local_pages+0x0/0x5
[<ffffffff8021bcf7>] smp_call_function_mask+0x5e/0x70
[<ffffffff80276c3a>] __alloc_pages+0x1ef/0x309
[<ffffffff8020be2e>] system_call+0x7e/0x83
[<ffffffff802762a4>] __get_free_pages+0xe/0x32
[<ffffffff802334bb>] copy_process+0xc1/0x148f
[<ffffffff8022d224>] set_next_entity+0x18/0x3a
[<ffffffff8022d2a3>] pick_next_task_fair+0x2d/0x3f
[<ffffffff8020be2e>] system_call+0x7e/0x83
[<ffffffff80234985>] do_fork+0x70/0x204
[<ffffffff8023f29a>] recalc_sigpending+0xe/0x3c
[<ffffffff8023f366>] sigprocmask+0x9e/0xc0
[<ffffffff8020c147>] ptregscall_common+0x67/0xb0
guerby@gcc04:~$ cat /proc/7652/stat
7652 (sh) R 7633 3857 3069 34816 3069 4194368 114 0 0 0 0 294472 0 0 20 0 1 0 2369557 8212480 332 18446744073709551615 4194304 4924460 140737476255664 18446744073709551615 47698491444962 262400 65538 0 65538 0 0 0 17 3 0 0 0 0 0
guerby@gcc04:~$ cat /proc/7652/status
Name: sh
State: R (running)
Tgid: 7652
Pid: 7652
PPid: 7633
TracerPid: 27382
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
FDSize: 256
Groups: 1000
VmPeak: 8020 kB
VmSize: 8020 kB
VmLck: 0 kB
VmHWM: 1328 kB
VmRSS: 1328 kB
VmData: 1168 kB
VmStk: 92 kB
VmExe: 716 kB
VmLib: 1560 kB
VmPTE: 24 kB
Threads: 1
SigQ: 12/18446744073709551615
SigPnd: 0000000000040100
ShdPnd: 00000000000f4102
SigBlk: 0000000000010002
SigIgn: 0000000000000000
SigCgt: 0000000000010002
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
Cpus_allowed: 0000000f
Mems_allowed: 00000000,00000001
voluntary_ctxt_switches: 0
nonvoluntary_ctxt_switches: 1
On Sun, 2008-03-16 at 15:43 +0100, Laurent GUERBY wrote:
> On Sat, 2008-03-08 at 23:00 +0100, Laurent GUERBY wrote:
> > Hi,
> >
> > I have a system with an "AMD64 Phenom 9500" quad core cpu, 4GB RAM,
> > "ASUS M3A32 MVP Deluxe wifi" motherboard with latest vendor BIOS
> > (0801).
> >
> > I tried stock debian etch kernel (Debian 2.6.18.dfsg.1-18etch1),
> > machine
> > froze with no message, debian etch backport kernel same, and then
> > Debian 2.6.24-4 from unstable and I got some messages: machine
> > is not frozen but some userland processes are (ps says "Dl" state
> > with child in "Zs" state) and "events/3" is taking 100% cpu
> > according to top:
> >
> > 18 root 15 -5 0 0 0 R 100 0.0 74:59.46
> > events/3
> >
> > Got to the same state with ubuntu hardy 2.6.24-8-server kernel. All
> > kernels are untainted, no X running anyway.
> >
> > It takes a few hours of doing some stuff, in my case bootstraping or
> > testing GCC at -j 4, and then the problem happens.
>
> On 2.6.24-1-amd64 (Debian 2.6.24-4) I got a slightly different
> backtrace in /var/log/messages after a few hours of stressing the
> machine with compilations, see below. The given process
> was stuck and unkillable.
>
> Any idea on what to do/try?
I changed motherboard and went for a (way cheaper but older) ASUS M2A-VM
with is based on the AMD 690G chipset with the exact same
kernel/phenom/memory/disk/box and installed the latest vendor BIOS
(1604). It took longer (25 hours) to get a stuck and unkillable process
but it did happen, but I got nothing in /var/log/kern.log this time.
In order to rule out a motherboard or memory issue I bought an Athlon X2
4400+ EE and put in replacement of the phenom with the exact same
kenel/memory/disk/box and my stress test has been running for 72 hours
without any issue so far.
So in the end it seems to be a problem specific to phenom 9500 with the
linux kernel.
Did anyone succeed in getting a stable linux box based on a phenom 9500
processor ? "Stable" defined as being able to survive a few days
compiling at -j4 (this is for the GCC compile farm after all :).
If so I'm interested by the exact motherboard/bios version/kernel
version/distro used.
As proposed in my first email, ssh root access is possible to my machine
(with either motherboard).
Thanks in advance,
Laurent
Laurent,
you may have triggered the L2 eviction bug (E298).
Please try
hexdump -s 0xc0010015 -n 8 -C /dev/cpu/0/msr
Output is little-endian, so the left-most byte must have bit 3 enabled meaning
TLB caching is disabled.
-Peter
Laurent GUERBY schrieb:
> On Sun, 2008-03-16 at 15:43 +0100, Laurent GUERBY wrote:
>> On Sat, 2008-03-08 at 23:00 +0100, Laurent GUERBY wrote:
>>> Hi,
>>>
>>> I have a system with an "AMD64 Phenom 9500" quad core cpu, 4GB RAM,
>>> "ASUS M3A32 MVP Deluxe wifi" motherboard with latest vendor BIOS
>>> (0801).
>>>
>>> I tried stock debian etch kernel (Debian 2.6.18.dfsg.1-18etch1),
>>> machine
>>> froze with no message, debian etch backport kernel same, and then
>>> Debian 2.6.24-4 from unstable and I got some messages: machine
>>> is not frozen but some userland processes are (ps says "Dl" state
>>> with child in "Zs" state) and "events/3" is taking 100% cpu
>>> according to top:
>>>
>>> 18 root 15 -5 0 0 0 R 100 0.0 74:59.46
>>> events/3
>>>
>>> Got to the same state with ubuntu hardy 2.6.24-8-server kernel. All
>>> kernels are untainted, no X running anyway.
>>>
>>> It takes a few hours of doing some stuff, in my case bootstraping or
>>> testing GCC at -j 4, and then the problem happens.
>> On 2.6.24-1-amd64 (Debian 2.6.24-4) I got a slightly different
>> backtrace in /var/log/messages after a few hours of stressing the
>> machine with compilations, see below. The given process
>> was stuck and unkillable.
>>
>> Any idea on what to do/try?
>
> I changed motherboard and went for a (way cheaper but older) ASUS M2A-VM
> with is based on the AMD 690G chipset with the exact same
> kernel/phenom/memory/disk/box and installed the latest vendor BIOS
> (1604). It took longer (25 hours) to get a stuck and unkillable process
> but it did happen, but I got nothing in /var/log/kern.log this time.
>
> In order to rule out a motherboard or memory issue I bought an Athlon X2
> 4400+ EE and put in replacement of the phenom with the exact same
> kenel/memory/disk/box and my stress test has been running for 72 hours
> without any issue so far.
>
> So in the end it seems to be a problem specific to phenom 9500 with the
> linux kernel.
>
> Did anyone succeed in getting a stable linux box based on a phenom 9500
> processor ? "Stable" defined as being able to survive a few days
> compiling at -j4 (this is for the GCC compile farm after all :).
>
> If so I'm interested by the exact motherboard/bios version/kernel
> version/distro used.
>
> As proposed in my first email, ssh root access is possible to my machine
> (with either motherboard).
>
> Thanks in advance,
>
> Laurent
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
>
--
| AMD Saxony Limited Liability Company & Co. KG
Operating | Wilschdorfer Landstr. 101, 01109 Dresden, Germany
System | Register Court Dresden: HRA 4896
Research | General Partner authorized to represent:
Center | AMD Saxony LLC (Wilmington, Delaware, US)
| General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy
On Thu, 2008-03-20 at 14:45 +0100, Peter Oruba wrote:
> Laurent,
>
> you may have triggered the L2 eviction bug (E298).
>
> Please try
>
> hexdump -s 0xc0010015 -n 8 -C /dev/cpu/0/msr
>
> Output is little-endian, so the left-most byte must have bit 3 enabled meaning
> TLB caching is disabled.
Hi Peter,
Since I did not have /dev/cpu I did after some searching:
for i in 0 1 2 3; do
mkdir -p /dev/cpu/$i
mknod -m 444 /dev/cpu/$i/msr c 202 $i
mknod -m 444 /dev/cpu/$i/cpuid c 203 $i
done
modprobe msr
Then:
# hexdump -s 0xc0010015 -n 8 -C /dev/cpu/0/msr
# echo $?
0
# cat /proc/version
Linux version 2.6.24-1-amd64 (Debian 2.6.24-4) ([email protected]) (gcc
version 4.1.3 20080114 (prerelease) (Debian 4.1.2-19)) #1 SMP Mon Feb 11
13:47:43 UTC 2008
When I look at hexdump -v it's only zeroes for as long as I run it.
What am I doing wrong?
Thanks for your help!
Laurent
# lsmod
Module Size Used by
msr 8968 0
nfs 258800 0
tun 16512 1
cpufreq_userspace 9604 4
nfsd 261416 17
lockd 73136 3 nfs,nfsd
nfs_acl 8192 2 nfs,nfsd
auth_rpcgss 53152 1 nfsd
sunrpc 201096 11 nfs,nfsd,lockd,nfs_acl,auth_rpcgss
exportfs 9472 1 nfsd
ac 11400 0
battery 19976 0
ipv6 286248 28
it87 28432 0
hwmon_vid 7424 1 it87
k8temp 10496 0
eeprom 12688 0
powernow_k8 18464 0
freq_table 9728 1 powernow_k8
loop 22788 0
snd_hda_intel 362024 0
parport_pc 42408 0
parport 44812 1 parport_pc
serio_raw 11908 0
snd_pcm 89352 1 snd_hda_intel
snd_timer 28552 1 snd_pcm
snd 65896 3 snd_hda_intel,snd_pcm,snd_timer
soundcore 13216 1 snd
snd_page_alloc 15248 2 snd_hda_intel,snd_pcm
pcspkr 7808 0
i2c_piix4 13708 0
psmouse 45724 0
i2c_core 29824 2 eeprom,i2c_piix4
shpchp 38428 0
button 13984 0
pci_hotplug 35504 1 shpchp
evdev 17024 0
ext3 139024 1
jbd 55976 1 ext3
mbcache 13952 1 ext3
dm_mirror 27008 0
dm_snapshot 21960 0
dm_mod 67832 2 dm_mirror,dm_snapshot
ata_generic 13060 0
sd_mod 33408 3
generic 9732 0 [permanent]
usbhid 35168 0
hid 41792 1 usbhid
atiixp 9488 0 [permanent]
ahci 34436 2
ide_core 144920 2 generic,atiixp
libata 162736 2 ata_generic,ahci
ehci_hcd 39692 0
ohci_hcd 28164 0
scsi_mod 170296 2 sd_mod,libata
r8169 36740 0
thermal 22688 0
processor 45032 2 powernow_k8,thermal
fan 9864 0