2008-01-18 01:20:35

by Tetsuo Handa

[permalink] [raw]
Subject: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.

Hello.

I found the CONFIG_DEBUG_LOCKING_API_SELFTESTS hangs
at (A-A deadlock)-(wlock) pair.

2.6.24-rc8 doesn't hang.
So, I think the bug is in 2.6.24-rc8-mm1.bz2 .

The kernel configs are at

http://I-love.SAKURA.ne.jp/tmp/config-2.6.24-rc8
http://I-love.SAKURA.ne.jp/tmp/config-2.6.24-rc8-mm1

Regards.

---------- 2.6.24-rc8-mm1 ----------
Linux version 2.6.24-rc8-mm1 (root@tomoyo) (gcc version 3.3.5 (Debian 1:3.3.5-13)) #2 SMP Fri Jan 18 10:13:09 JST 2008
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000001fef0000 (usable)
BIOS-e820: 000000001fef0000 - 000000001feff000 (ACPI data)
BIOS-e820: 000000001feff000 - 000000001ff00000 (ACPI NVS)
BIOS-e820: 000000001ff00000 - 0000000020000000 (usable)
BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved)
0MB HIGHMEM available.
512MB LOWMEM available.
found SMP MP-table at 000f6c90
Zone PFN ranges:
DMA 0 -> 4096
Normal 4096 -> 131072
HighMem 131072 -> 131072
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
0: 0 -> 131072
DMI present.
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: INTEL Product ID: 440BX APIC at: 0xFEE00000
Processor #0 6:15 APIC version 17
Processor #1 6:15 APIC version 17
I/O APIC #2 Version 17 at 0xFEC00000.
Enabling APIC mode: Flat. Using 1 I/O APICs
Processors: 2
Allocating PCI resources starting at 30000000 (gap: 20000000:dec00000)
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: root=/dev/sda1 ro noapic nolapic console=ttyS0,115200n8
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c0523000 soft=c0521000
PID hash table entries: 2048 (order: 11, 8192 bytes)
Detected 1993.105 MHz processor.
Console: colour VGA+ 80x25
console [ttyS0] enabled
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES: 8
... MAX_LOCK_DEPTH: 30
... MAX_LOCKDEP_KEYS: 2048
... CLASSHASH_SIZE: 1024
... MAX_LOCKDEP_ENTRIES: 8192
... MAX_LOCKDEP_CHAINS: 16384
... CHAINHASH_SIZE: 8192
memory used by lock dependency info: 1024 kB
per task-struct memory footprint: 1680 bytes
------------------------
| Locking API testsuite:
----------------------------------------------------------------------------
| spin |wlock |rlock |mutex | wsem | rsem |
--------------------------------------------------------------------------
A-A deadlock: ok |

---------- 2.6.24-rc8 ----------
Linux version 2.6.24-rc8 (root@tomoyo) (gcc version 3.3.5 (Debian 1:3.3.5-13)) #1 SMP Fri Jan 18 09:51:40 JST 2008
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000001fef0000 (usable)
BIOS-e820: 000000001fef0000 - 000000001feff000 (ACPI data)
BIOS-e820: 000000001feff000 - 000000001ff00000 (ACPI NVS)
BIOS-e820: 000000001ff00000 - 0000000020000000 (usable)
BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved)
0MB HIGHMEM available.
512MB LOWMEM available.
found SMP MP-table at 000f6c90
Zone PFN ranges:
DMA 0 -> 4096
Normal 4096 -> 131072
HighMem 131072 -> 131072
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
0: 0 -> 131072
DMI present.
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: INTEL Product ID: 440BX APIC at: 0xFEE00000
Processor #0 6:15 APIC version 17
Processor #1 6:15 APIC version 17
I/O APIC #2 Version 17 at 0xFEC00000.
Enabling APIC mode: Flat. Using 1 I/O APICs
Processors: 2
Allocating PCI resources starting at 30000000 (gap: 20000000:dec00000)
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: root=/dev/sda1 ro noapic nolapic console=ttyS0,115200n8
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c052c000 soft=c052a000
PID hash table entries: 2048 (order: 11, 8192 bytes)
Detected 1994.813 MHz processor.
Console: colour VGA+ 80x25
console [ttyS0] enabled
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES: 8
... MAX_LOCK_DEPTH: 30
... MAX_LOCKDEP_KEYS: 2048
... CLASSHASH_SIZE: 1024
... MAX_LOCKDEP_ENTRIES: 8192
... MAX_LOCKDEP_CHAINS: 16384
... CHAINHASH_SIZE: 8192
memory used by lock dependency info: 1024 kB
per task-struct memory footprint: 1680 bytes
------------------------
| Locking API testsuite:
----------------------------------------------------------------------------
| spin |wlock |rlock |mutex | wsem | rsem |
--------------------------------------------------------------------------
A-A deadlock: ok | ok | ok | ok | ok | ok |


2008-01-18 02:19:51

by Andrew Morton

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.

On Fri, 18 Jan 2008 10:20:11 +0900 Tetsuo Handa <[email protected]> wrote:

> Hello.
>
> I found the CONFIG_DEBUG_LOCKING_API_SELFTESTS hangs
> at (A-A deadlock)-(wlock) pair.
>
> 2.6.24-rc8 doesn't hang.
> So, I think the bug is in 2.6.24-rc8-mm1.bz2 .
>
> The kernel configs are at
>
> http://I-love.SAKURA.ne.jp/tmp/config-2.6.24-rc8
> http://I-love.SAKURA.ne.jp/tmp/config-2.6.24-rc8-mm1
>

Yes, Zan hit the same bug.

>
> ---------- 2.6.24-rc8-mm1 ----------
> Linux version 2.6.24-rc8-mm1 (root@tomoyo) (gcc version 3.3.5 (Debian 1:3.3.5-13)) #2 SMP Fri Jan 18 10:13:09 JST 2008
> BIOS-provided physical RAM map:
> BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
> BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
> BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
> BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
> BIOS-e820: 0000000000100000 - 000000001fef0000 (usable)
> BIOS-e820: 000000001fef0000 - 000000001feff000 (ACPI data)
> BIOS-e820: 000000001feff000 - 000000001ff00000 (ACPI NVS)
> BIOS-e820: 000000001ff00000 - 0000000020000000 (usable)
> BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
> BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
> BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved)
> 0MB HIGHMEM available.
> 512MB LOWMEM available.
> found SMP MP-table at 000f6c90
> Zone PFN ranges:
> DMA 0 -> 4096
> Normal 4096 -> 131072
> HighMem 131072 -> 131072
> Movable zone start PFN for each node
> early_node_map[1] active PFN ranges
> 0: 0 -> 131072
> DMI present.
> Intel MultiProcessor Specification v1.4
> Virtual Wire compatibility mode.
> OEM ID: INTEL Product ID: 440BX APIC at: 0xFEE00000
> Processor #0 6:15 APIC version 17
> Processor #1 6:15 APIC version 17
> I/O APIC #2 Version 17 at 0xFEC00000.
> Enabling APIC mode: Flat. Using 1 I/O APICs
> Processors: 2
> Allocating PCI resources starting at 30000000 (gap: 20000000:dec00000)
> Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
> Kernel command line: root=/dev/sda1 ro noapic nolapic console=ttyS0,115200n8
> Enabling fast FPU save and restore... done.
> Enabling unmasked SIMD FPU exception support... done.
> Initializing CPU#0
> CPU 0 irqstacks, hard=c0523000 soft=c0521000
> PID hash table entries: 2048 (order: 11, 8192 bytes)
> Detected 1993.105 MHz processor.
> Console: colour VGA+ 80x25
> console [ttyS0] enabled
> Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
> ... MAX_LOCKDEP_SUBCLASSES: 8
> ... MAX_LOCK_DEPTH: 30
> ... MAX_LOCKDEP_KEYS: 2048
> ... CLASSHASH_SIZE: 1024
> ... MAX_LOCKDEP_ENTRIES: 8192
> ... MAX_LOCKDEP_CHAINS: 16384
> ... CHAINHASH_SIZE: 8192
> memory used by lock dependency info: 1024 kB
> per task-struct memory footprint: 1680 bytes
> ------------------------
> | Locking API testsuite:
> ----------------------------------------------------------------------------
> | spin |wlock |rlock |mutex | wsem | rsem |
> --------------------------------------------------------------------------
> A-A deadlock: ok |

So does this mean that we got stuck in the first write_lock() test?

I'd be suspecting git-sched, so in lieu of a full bisection search it would
be great if one of you could apply
http://userweb.kernel.org/~akpm/git-sched.patch to 2.6.24-rc8 and see if it
fails in the same way.

2008-01-18 03:20:26

by Tetsuo Handa

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.

Hello.

Andrew Morton wrote:
> I'd be suspecting git-sched, so in lieu of a full bisection search it would
> be great if one of you could apply
> http://userweb.kernel.org/~akpm/git-sched.patch to 2.6.24-rc8 and see if it
> fails in the same way.
Too bad. It didn't fail.

Regards.

2008-01-18 04:43:00

by Andrew Morton

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.

On Fri, 18 Jan 2008 12:19:56 +0900 Tetsuo Handa <[email protected]> wrote:

> Hello.
>
> Andrew Morton wrote:
> > I'd be suspecting git-sched, so in lieu of a full bisection search it would
> > be great if one of you could apply
> > http://userweb.kernel.org/~akpm/git-sched.patch to 2.6.24-rc8 and see if it
> > fails in the same way.
> Too bad. It didn't fail.
>

I used your exact .config with the current mm lineup (I don't thinlk
anything relevant has changed) and it booted OK on my old PIII machine.

It could be compiler version dependent. I used gcc-4.1.0. Which version
were you and Zan using please?

2008-01-18 06:41:51

by Tetsuo Handa

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.

Hello.

Andrew Morton wrote:
> It could be compiler version dependent. I used gcc-4.1.0. Which version
> were you and Zan using please?

I'm using gcc 3.3.5 on Debian/Sarge .

By the way, 2.6.24-rc8-mm1 doesn't hang with a kernel config

# make -s allnoconfig
# make -s menuconfig

Set [*] to all entries in "Kernel hacking --->" section
(except "KGDB: kernel debugging with remote gdb").

# make -s && make -s install

Regards.

2008-01-18 06:59:43

by Dave Young

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.

On Jan 18, 2008 2:40 PM, Tetsuo Handa
<[email protected]> wrote:
> Hello.
>
> Andrew Morton wrote:
> > It could be compiler version dependent. I used gcc-4.1.0. Which version
> > were you and Zan using please?

I have the same problem, gcc 3.4.6

>
> I'm using gcc 3.3.5 on Debian/Sarge .
>
> By the way, 2.6.24-rc8-mm1 doesn't hang with a kernel config
>
> # make -s allnoconfig
> # make -s menuconfig
>
> Set [*] to all entries in "Kernel hacking --->" section
> (except "KGDB: kernel debugging with remote gdb").
>
> # make -s && make -s install
>
> Regards.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2008-01-18 08:22:06

by Andrew Morton

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.

On Fri, 18 Jan 2008 14:59:32 +0800 "Dave Young" <[email protected]> wrote:

> On Jan 18, 2008 2:40 PM, Tetsuo Handa
> <[email protected]> wrote:
> > Hello.
> >
> > Andrew Morton wrote:
> > > It could be compiler version dependent. I used gcc-4.1.0. Which version
> > > were you and Zan using please?
>
> I have the same problem, gcc 3.4.6
>
> >
> > I'm using gcc 3.3.5 on Debian/Sarge .
> >
> > By the way, 2.6.24-rc8-mm1 doesn't hang with a kernel config
> >
> > # make -s allnoconfig
> > # make -s menuconfig
> >
> > Set [*] to all entries in "Kernel hacking --->" section
> > (except "KGDB: kernel debugging with remote gdb").
> >
> > # make -s && make -s install

The only machine I have with gcc-3.x takes an amazing amount of time to
compile a quite minimal kernel then barfs in headers_install with

CHK include/linux/version.h
make[1]: `scripts/unifdef' is up to date.
make[1]: *** No rule to make target `|', needed by `asm-generic'. Stop.

which is a mainline bug, too. ISTR someone else reported it but I don't
recall it getting fixed.

I think I'll give up on this. Either someone else fixes it or we merge the
bug into mainline.

2008-01-18 08:56:18

by Sam Ravnborg

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.

On Fri, Jan 18, 2008 at 12:21:52AM -0800, Andrew Morton wrote:
> On Fri, 18 Jan 2008 14:59:32 +0800 "Dave Young" <[email protected]> wrote:
>
> > On Jan 18, 2008 2:40 PM, Tetsuo Handa
> > <[email protected]> wrote:
> > > Hello.
> > >
> > > Andrew Morton wrote:
> > > > It could be compiler version dependent. I used gcc-4.1.0. Which version
> > > > were you and Zan using please?
> >
> > I have the same problem, gcc 3.4.6
> >
> > >
> > > I'm using gcc 3.3.5 on Debian/Sarge .
> > >
> > > By the way, 2.6.24-rc8-mm1 doesn't hang with a kernel config
> > >
> > > # make -s allnoconfig
> > > # make -s menuconfig
> > >
> > > Set [*] to all entries in "Kernel hacking --->" section
> > > (except "KGDB: kernel debugging with remote gdb").
> > >
> > > # make -s && make -s install
>
> The only machine I have with gcc-3.x takes an amazing amount of time to
> compile a quite minimal kernel then barfs in headers_install with
>
> CHK include/linux/version.h
> make[1]: `scripts/unifdef' is up to date.
> make[1]: *** No rule to make target `|', needed by `asm-generic'. Stop.
>
> which is a mainline bug, too. ISTR someone else reported it but I don't
> recall it getting fixed.

I know about this bug and I plan to fix it one day.

I have not considered it important because the bug is only triggered
by following connditions:
1) In kernel hacking the check headers options is selected
2) make is older than version 3.80

Documentation/changes do not require us to use make 3.80.

> I think I'll give up on this. Either someone else fixes it or we merge the
> bug into mainline.

One way to fix this is to disable the config option like this.

Sam

[PATCH] kbuild: disable headers_check

The headers_check command requires make version 3.80 due to use
of ordered dependencies. But kernel build does otherwise
not require this recent make.
Disable the option until it get properly fixed.

Signed-off-by: Sam Ravnborg <[email protected]>
---
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index a601093..98b2431 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -67,7 +67,7 @@ config DEBUG_FS

config HEADERS_CHECK
bool "Run 'make headers_check' when building vmlinux"
- depends on !UML
+ depends on !UML || BROKEN
help
This option will extract the user-visible kernel headers whenever
building the kernel, and will run basic sanity checks on them to

2008-01-18 09:18:44

by Ingo Molnar

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.


* Dave Young <[email protected]> wrote:

> On Jan 18, 2008 2:40 PM, Tetsuo Handa
> <[email protected]> wrote:
> > Hello.
> >
> > Andrew Morton wrote:
> > > It could be compiler version dependent. I used gcc-4.1.0. Which version
> > > were you and Zan using please?
>
> I have the same problem, gcc 3.4.6

is the patch below already in -rc8-mm1? hmm ... it appears it's not.
Could you give it a try?

Ingo

------------->
commit 5a26db5bd25cf4bf32ae9fa9f6136b6b6d5b45c5
Author: Nick Piggin <[email protected]>
Date: Wed Jan 16 09:51:58 2008 +0100

lockdep: fix internal double unlock during self-test
Lockdep, during self-test (when it was simulating double unlocks) was
sometimes unconditionally unlocking a spinlock when it had not been
locked. This won't work for ticket locks.

Signed-off-by: Nick Piggin <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
Signed-off-by: Peter Zijlstra <[email protected]>

diff --git a/kernel/lockdep.c b/kernel/lockdep.c
index 723bd9f..4335f12 100644
--- a/kernel/lockdep.c
+++ b/kernel/lockdep.c
@@ -2943,9 +2943,10 @@ void lockdep_free_key_range(void *start, unsigned long size)
struct list_head *head;
unsigned long flags;
int i;
+ int locked;

raw_local_irq_save(flags);
- graph_lock();
+ locked = graph_lock();

/*
* Unhash all classes that were created by this module:
@@ -2959,7 +2960,8 @@ void lockdep_free_key_range(void *start, unsigned long size)
zap_class(class);
}

- graph_unlock();
+ if (locked)
+ graph_unlock();
raw_local_irq_restore(flags);
}

@@ -2969,6 +2971,7 @@ void lockdep_reset_lock(struct lockdep_map *lock)
struct list_head *head;
unsigned long flags;
int i, j;
+ int locked;

raw_local_irq_save(flags);

@@ -2987,7 +2990,7 @@ void lockdep_reset_lock(struct lockdep_map *lock)
* Debug check: in the end all mapped classes should
* be gone.
*/
- graph_lock();
+ locked = graph_lock();
for (i = 0; i < CLASSHASH_SIZE; i++) {
head = classhash_table + i;
if (list_empty(head))
@@ -3000,7 +3003,8 @@ void lockdep_reset_lock(struct lockdep_map *lock)
}
}
}
- graph_unlock();
+ if (locked)
+ graph_unlock();

out_restore:
raw_local_irq_restore(flags);

2008-01-18 09:19:54

by Tetsuo Handa

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.

Hello.

I tried gcc 4.1.2 on Fedora 8 using
http://I-love.SAKURA.ne.jp/tmp/config-2.6.24-rc8-mm1 .
Same result as gcc 3.3.5 on Debian/Sarge.

It seems kernel config (and possibly hardware) dependent
rather than gcc version dependent.

This is VMware workstation 6.0.0 on Thinkpad X60 (Core 2 Duo).

Regards.

2008-01-18 09:39:18

by Tetsuo Handa

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.

Hello.

Ingo Molnar wrote:
> is the patch below already in -rc8-mm1? hmm ... it appears it's not.
> Could you give it a try?

This patch solved this bug.

Thank you.

2008-01-18 09:49:23

by Dave Young

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.

On Jan 18, 2008 5:38 PM, Tetsuo Handa
<[email protected]> wrote:
> Hello.
>
> Ingo Molnar wrote:
> > is the patch below already in -rc8-mm1? hmm ... it appears it's not.
> > Could you give it a try?

Yes , It fixes that problem.

>
> This patch solved this bug.

So quick :)

>
> Thank you.
>

2008-01-18 16:29:18

by Zan Lynx

[permalink] [raw]
Subject: Re: [2.6.24-rc8-mm1] Locking API boot-time self-tests hangs.


On Thu, 2008-01-17 at 20:42 -0800, Andrew Morton wrote:
> On Fri, 18 Jan 2008 12:19:56 +0900 Tetsuo Handa <[email protected]> wrote:
>
> > Hello.
> >
> > Andrew Morton wrote:
> > > I'd be suspecting git-sched, so in lieu of a full bisection search it would
> > > be great if one of you could apply
> > > http://userweb.kernel.org/~akpm/git-sched.patch to 2.6.24-rc8 and see if it
> > > fails in the same way.
> > Too bad. It didn't fail.
> >
>
> I used your exact .config with the current mm lineup (I don't thinlk
> anything relevant has changed) and it booted OK on my old PIII machine.
>
> It could be compiler version dependent. I used gcc-4.1.0. Which version
> were you and Zan using please?

From my -rc6-mm1 boot.

$ scripts/ver_linux
If some fields are empty or look unusual you may have an old version.
Compare to the current minimal requirements in Documentation/Changes.

Linux zephyr 2.6.24-rc6-mm1 #1 SMP Tue Dec 25 17:53:03 MST 2007 x86_64
AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux

Gnu C 4.2.2
Gnu make 3.81
binutils 2.18.50.0.1.20070908
util-linux 2.13.1
mount 2.13.1
module-init-tools 3.4
e2fsprogs 1.40.4
reiserfsprogs 3.6.19
reiser4progs 1.0.6
pcmciautils 014
pcmcia-cs 3.2.9
PPP 2.4.4
nfs-utils 1.0.7
Linux C Library Dynamic linker (ldd) 2.7
Procps 3.2.7
Net-tools 1.60
Kbd 1.13
oprofile 0.9.3
Sh-utils 6.9
udev 118
wireless-tools 29
Modules Loaded nls_cp437 vfat fat nls_iso8859_1 isofs nls_base
nouveau drm snd_pcm_oss snd_mixer_oss snd_seq_dummy snd_seq_oss
snd_seq_midi_event snd_seq snd_seq_device rfcomm l2cap usbhid hid
hci_usb bluetooth usb_storage libusual joydev nvidiafb fb fb_ddc
i2c_algo_bit cfbcopyarea vgastate i2c_core cfbimgblt cfbfillrect
snd_intel8x0m snd_intel8x0 snd_ac97_codec ehci_hcd psmouse ohci_hcd
serio_raw ac97_bus snd_pcm snd_timer usbcore evdev snd sg snd_page_alloc

--
Zan Lynx <[email protected]>


Attachments:
signature.asc (189.00 B)
This is a digitally signed message part