2005-05-24 09:08:56

by Nagesh Sharyathi

[permalink] [raw]
Subject: kdump test update

These I have tested on the kernel 2.6.12-rc4-mm1 with the following test
suites , with kdump enabled

Once test suites PASS/SUCCESS, force the machine to hang(lock up) by
disabling irqs with the attached SPINLOCK test module from Badari
Pulavarthy,
try to take dump either with sysrq key or nmi_watchdog=2 kernel parameter.

Test Suite:
-----------
LTP Runall, FSracer(race condition in file system) with LVM partitions
(over ext2, ext3, JFS, XFS), FS stress, Mem Test/Bash Memory, Cerberus,
KernBench, NetPerf.

System Info:
------------
Distro: SLES 9 SP1

Software/kernel variables:
--------------------------
1. kernel - linux-2.6.12-rc4-mm1
2. kexec-tools-1.101 + kdump patches
3. kernel.sysrq=1

Command line parameters for first kernel:
-----------------------------------------
root = <> vga=0x31a selinux=0 splash=silent resume=<> elevator=cfq
showpts
crashkernel=48M@16M console=tty0 console=ttyS0,38400n1

Hardwares on which is test cases are run:
-----------------------------------------

A) 1way, Pentium IV 2.8GHz, 2G RAM
- Network Interface (e1000)
- Disk I/O: SCSI storage controller: Adaptec Ultra320

o Ran test suite KERNBENCH and CERBERUS test ran successfully. Forced
system hang by inserting spinlock test module and tried to invoke
panic with sysrq+c, but it failed to force Panic. I failed to take
the
dump as sysrq keys failed to respond during hang.

o Booted with nmi_watchdog=2 and ran similar tests and then forced
system
hang by inserting spinlock module, nmi_watchdog caused kernel panic
and
the system booted to panic kernel. I was able to take dump. The first

kernel was rebuilt after applying Vivek's following patch to fix
kexec
on panic with nmi watchdog enabled

http://marc.theaimsgroup.com/?l=linux-kernel&m=111631994607762&w=2

B) SMP, 2way, Pentium III (Coppermine) 1 GHz, 1.3G RAM
- Network Interface (e100)
- Disk I/O: SCSI storage controller: Adaptec Ultra160

o Ran test suite MemTest and Bash Memory, after running the test for
some
time (< 1hr.), forced system hang by inserting spinlock test module
and tried to invoke kernel panic with sysrq-c, but it failed to force

panic and hence couldn't initiate kdump.

o Booted with nmi_watchdog=2 and ran similar tests and then forced
system
hang by inserting spinlock module, nmi_watchdog caused kernel panic
and
the system booted to panic kernel. I was able to take dump. The first

kernel was rebuilt after applying Vivek's following patch to fix
kexec
on panic with nmi watchdog enabled

http://marc.theaimsgroup.com/?l=linux-kernel&m=111631994607762&w=2

o Ran the same test suite, after running the test for more than 10hours

kernel OOps have occured (bugme 4653) but kdump failed to boot to
secondary kernel as there was no kernel panic and just an kernel
oops.

Now I have set kernel sysctl parameter kernel.panic_on_oops=1,
testing in
progress to capture the oops in kdump

o While testing the functionality of kdump, encountered this driver
hardening
issue (bugme:4631) also.


C) SMP, 2way, Xeon TM 2.8GHz, 1.5G RAM
- Network Interface (Tigon3)
- Disk I/O: SCSI storage controller: IBM Serve RAID

o Ran test suite FSracer over LVM partition and the test ran without
failures. Forced system hang by inserting spinlock test module and
then
invoked kernel panic with sysrq-c. Secondary kernel booted properly
without any issues. I was able to take the dump using sysrq keys.


For more details please mail me

Attachment:


Attachments:
spinlock.c (285.00 B)

2005-05-24 13:21:32

by Alexander Nyberg

[permalink] [raw]
Subject: Re: kdump test update

tis 2005-05-24 klockan 14:38 +0530 skrev Nagesh Sharyathi:
> These I have tested on the kernel 2.6.12-rc4-mm1 with the following test
> suites , with kdump enabled
>
> Once test suites PASS/SUCCESS, force the machine to hang(lock up) by
> disabling irqs with the attached SPINLOCK test module from Badari
> Pulavarthy,
> try to take dump either with sysrq key or nmi_watchdog=2 kernel parameter.
>
> Test Suite:
> -----------
> LTP Runall, FSracer(race condition in file system) with LVM partitions
> (over ext2, ext3, JFS, XFS), FS stress, Mem Test/Bash Memory, Cerberus,
> KernBench, NetPerf.
>
> System Info:
> ------------
> Distro: SLES 9 SP1
>
> Software/kernel variables:
> --------------------------
> 1. kernel - linux-2.6.12-rc4-mm1
> 2. kexec-tools-1.101 + kdump patches
> 3. kernel.sysrq=1
>
> Command line parameters for first kernel:
> -----------------------------------------
> root = <> vga=0x31a selinux=0 splash=silent resume=<> elevator=cfq
> showpts
> crashkernel=48M@16M console=tty0 console=ttyS0,38400n1
>
> Hardwares on which is test cases are run:
> -----------------------------------------
>
> A) 1way, Pentium IV 2.8GHz, 2G RAM
> - Network Interface (e1000)
> - Disk I/O: SCSI storage controller: Adaptec Ultra320
>
> o Ran test suite KERNBENCH and CERBERUS test ran successfully. Forced
> system hang by inserting spinlock test module and tried to invoke
> panic with sysrq+c, but it failed to force Panic. I failed to take
> the
> dump as sysrq keys failed to respond during hang.

This is expected, your spinlock module disables interrupts so
sysrq-crashdump has not a chance to get through. NMI watchdog is highly
necessary. (this is even more reason to have NMI watchdog on by default
on at least newer cpus even on x86 if kdump enters mainline and wants to
be useful).


2005-05-25 07:04:22

by Maneesh Soni

[permalink] [raw]
Subject: Re: [Fastboot] kdump test update

On Tue, May 24, 2005 at 02:38:42PM +0530, Nagesh Sharyathi wrote:
> These I have tested on the kernel 2.6.12-rc4-mm1 with the following test
> suites , with kdump enabled
>

Thanks for doing this, but it will be nice if you could also verify the
dumps using gdb. I have anyway added these test reportd on kdump test webpage at

http://lse.sourceforge.net/kdump/kdump-test.html

Thanks
Maneesh

--
Maneesh Soni
Linux Technology Center,
IBM India Software Labs,
Bangalore, India
email: [email protected]
Phone: 91-80-25044990