Hi all,
I've been trying to upgrade to something newer than 4.2.6 since I want
to use LVM Cache on my home NFS fileserver, KVM server, test server,
etc. So when it goes down, I lose all my other systems which mount
stuff from it.
Right now I'm trying to figure out how to use Netconsole to grab a
dump of the oops, but it's not working well. But let me describe the
situation as I've found it so far.
When the system boots up, it first starts with eth0 on the network,
then switches to br0 since I have a KVM bridge setup so my VMs can
run on the same home network, 192.168.1.0/24 which is pretty
standard. The system is an AMD Phenom(tm) II X4 945 Processor,
running at a max of 3Ghz, with 16gb of RAM, mpt2 LSI PCI-E 8 port sata
controller, on an ASUS motherboard. I can get details if you like.
It's an older box, but still runs really well, so why change?
Anyway, if I try to boot up anything past the 4.2.6 kernel, the system
locks up pretty quickly with an oops message that scrolls off the
screen too far. I've got some pictures which I'll attach in a bit,
maybe they'll help. So at first I thought it was something to do with
bad kworker threads, or SCSI or SATA interactions, but as I tried to
configure Netconsole to log to my beaglebone black SBC, I found out
that if I compiled and installed 4.4-rc3, started the bridge up (br0),
even started KVM, but did NOT start my VMs, the system was stable.
And if I didn't start br0, I could start a VM, but the system wouldn't
crash. The VM wasn't on the network... but the system didn't crash.
So I think I've found a wierd interaction here. My KVMs are both
Debian images, with 1-2gb of RAM and 1 CPU each. Nothing strange. My
network config is:
> cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# Bridge for VMs
auto br0
iface br0 inet static
address 192.168.1.6
netmask 255.255.255.0
network 192.168.1.0
gateway 192.168.1.254
bridge_ports eth0
bridge_stp on
bridge_maxwait 0
bridge_fd 0
# Old setup
# auto eth0
# iface eth0 inet static
# address 192.168.1.6
# netmask 255.255.255.0
# gateway 192.168.1.254
The currently running system version is:
> cat /proc/version
Linux version 4.4.0-rc3 (john@quad) (gcc version 4.9.2 (Debian 4.9.2-10) ) #1 SMP Thu Dec 3 12:13:30 EST 2015
And more detailed CPU info
> cat /proc/cpuinfo
.....
processor : 3
vendor_id : AuthenticAMD
cpu family : 16
model : 4
model name : AMD Phenom(tm) II X4 945 Processor
stepping : 3
microcode : 0x10000b6
cpu MHz : 800.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl
nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm
extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit
wdt hw_pstate npt lbrv svm_lock nrip_save vmmcall
bugs : tlb_mmatch apic_c1e fxsave_leak sysret_ss_attrs
bogomips : 6027.13
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
Here's my bootup messages, unfortunately I don't have any oops
messages. For whatever reason, it kicks in so quickly, that I can't
get anything out over the network. I'm going to see if I can stuff
another network card in there and use that to send traffic, instead of
over the brige.
My next step is going to be to try and disable some of the bridge
settings, like bridge_stp, bridge_maxwait and bridge_fd to just accept
the defaults. I set this up under Debian Wheezy a long time ago and
never touched it since.
My network config is:
quad:~> ifconfig -a
br0 Link encap:Ethernet HWaddr 20:cf:30:95:5f:2f
inet addr:192.168.1.6 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: 2002:42bd:1ac0:1:22cf:30ff:fe95:5f2f/64 Scope:Global
inet6 addr: fe80::22cf:30ff:fe95:5f2f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:24154 errors:0 dropped:0 overruns:0 frame:0
TX packets:16103 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:68682293 (65.5 MiB) TX bytes:2563964 (2.4 MiB)
eth0 Link encap:Ethernet HWaddr 20:cf:30:95:5f:2f
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:66460 errors:0 dropped:0 overruns:0 frame:0
TX packets:18157 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:71819217 (68.4 MiB) TX bytes:2782126 (2.6 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:7308 errors:0 dropped:0 overruns:0 frame:0
TX packets:7308 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1539613 (1.4 MiB) TX bytes:1539613 (1.4 MiB)
Any suggestions on what else I can do to help debug this issue? It's amazing how quickly the system locks up when I have all three steps taken.
John
On Fri, Dec 04, 2015 at 11:28:33PM -0500, John Stoffel wrote:
>
> Hi all,
>
> I've been trying to upgrade to something newer than 4.2.6 since I want
> to use LVM Cache on my home NFS fileserver, KVM server, test server,
> etc. So when it goes down, I lose all my other systems which mount
> stuff from it.
>
> Right now I'm trying to figure out how to use Netconsole to grab a
> dump of the oops, but it's not working well. But let me describe the
> situation as I've found it so far.
>
> When the system boots up, it first starts with eth0 on the network,
> then switches to br0 since I have a KVM bridge setup so my VMs can
> run on the same home network, 192.168.1.0/24 which is pretty
> standard. The system is an AMD Phenom(tm) II X4 945 Processor,
> running at a max of 3Ghz, with 16gb of RAM, mpt2 LSI PCI-E 8 port sata
> controller, on an ASUS motherboard. I can get details if you like.
> It's an older box, but still runs really well, so why change?
>
> Anyway, if I try to boot up anything past the 4.2.6 kernel, the system
> locks up pretty quickly with an oops message that scrolls off the
> screen too far. I've got some pictures which I'll attach in a bit,
> maybe they'll help. So at first I thought it was something to do with
> bad kworker threads, or SCSI or SATA interactions, but as I tried to
> configure Netconsole to log to my beaglebone black SBC, I found out
> that if I compiled and installed 4.4-rc3, started the bridge up (br0),
> even started KVM, but did NOT start my VMs, the system was stable.
>
> And if I didn't start br0, I could start a VM, but the system wouldn't
> crash. The VM wasn't on the network... but the system didn't crash.
> So I think I've found a wierd interaction here. My KVMs are both
> Debian images, with 1-2gb of RAM and 1 CPU each. Nothing strange. My
> network config is:
>
> > cat /etc/network/interfaces
> # This file describes the network interfaces available on your system
> # and how to activate them. For more information, see interfaces(5).
>
> # The loopback network interface
> auto lo
> iface lo inet loopback
>
> # Bridge for VMs
> auto br0
>
> iface br0 inet static
> address 192.168.1.6
> netmask 255.255.255.0
> network 192.168.1.0
> gateway 192.168.1.254
> bridge_ports eth0
> bridge_stp on
> bridge_maxwait 0
> bridge_fd 0
>
> # Old setup
> # auto eth0
>
> # iface eth0 inet static
> # address 192.168.1.6
> # netmask 255.255.255.0
> # gateway 192.168.1.254
>
> The currently running system version is:
>
> > cat /proc/version
> Linux version 4.4.0-rc3 (john@quad) (gcc version 4.9.2 (Debian 4.9.2-10) ) #1 SMP Thu Dec 3 12:13:30 EST 2015
>
> And more detailed CPU info
>
> > cat /proc/cpuinfo
> .....
>
> processor : 3
> vendor_id : AuthenticAMD
> cpu family : 16
> model : 4
> model name : AMD Phenom(tm) II X4 945 Processor
> stepping : 3
> microcode : 0x10000b6
> cpu MHz : 800.000
> cache size : 512 KB
> physical id : 0
> siblings : 4
> core id : 3
> cpu cores : 4
> apicid : 3
> initial apicid : 3
> fpu : yes
> fpu_exception : yes
> cpuid level : 5
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
> fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl
> nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm
> extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit
> wdt hw_pstate npt lbrv svm_lock nrip_save vmmcall
> bugs : tlb_mmatch apic_c1e fxsave_leak sysret_ss_attrs
> bogomips : 6027.13
> TLB size : 1024 4K pages
> clflush size : 64
> cache_alignment : 64
> address sizes : 48 bits physical, 48 bits virtual
> power management: ts ttp tm stc 100mhzsteps hwpstate
>
>
> Here's my bootup messages, unfortunately I don't have any oops
> messages. For whatever reason, it kicks in so quickly, that I can't
> get anything out over the network. I'm going to see if I can stuff
> another network card in there and use that to send traffic, instead of
> over the brige.
>
> My next step is going to be to try and disable some of the bridge
> settings, like bridge_stp, bridge_maxwait and bridge_fd to just accept
> the defaults. I set this up under Debian Wheezy a long time ago and
> never touched it since.
>
> My network config is:
>
> quad:~> ifconfig -a
> br0 Link encap:Ethernet HWaddr 20:cf:30:95:5f:2f
> inet addr:192.168.1.6 Bcast:192.168.1.255 Mask:255.255.255.0
> inet6 addr: 2002:42bd:1ac0:1:22cf:30ff:fe95:5f2f/64 Scope:Global
> inet6 addr: fe80::22cf:30ff:fe95:5f2f/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:24154 errors:0 dropped:0 overruns:0 frame:0
> TX packets:16103 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:68682293 (65.5 MiB) TX bytes:2563964 (2.4 MiB)
>
> eth0 Link encap:Ethernet HWaddr 20:cf:30:95:5f:2f
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:66460 errors:0 dropped:0 overruns:0 frame:0
> TX packets:18157 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:71819217 (68.4 MiB) TX bytes:2782126 (2.6 MiB)
>
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> inet6 addr: ::1/128 Scope:Host
> UP LOOPBACK RUNNING MTU:65536 Metric:1
> RX packets:7308 errors:0 dropped:0 overruns:0 frame:0
> TX packets:7308 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:1539613 (1.4 MiB) TX bytes:1539613 (1.4 MiB)
>
>
> Any suggestions on what else I can do to help debug this issue? It's amazing how quickly the system locks up when I have all three steps taken.
I've found a crash dump picture that I took, and I might be off on the wrong track, but I
really don't know what else to think here. As a quick test, I'm making a new kernel with
more modules compiled in, and more debugging options turned on. Here's my by-hand copy of
the crash dump message I was able to take a picture of from my self compiled 4.4-rc3 kernel.
Sorry for any typos:
Workqueue: kblockd_cfq_kick_queue
task: ffff8800cf9f3180 ti: ffff8800cf940000 task.ti ffff800cf940000
RIP: 0010: [<ffffffff8136adaf>] [<ffffffff8136adaf>] scsi_init_sgtable+0x3e/0x59
RSP: 0018:ffff8800cf943cc0 EFLAGS: 00010002
RAX: 0000000000000006 RBX: ffff88040eaf2760 RCX: 0000000000000007
RDX: 0000000000000006 RSI: 00000000ffffffff RDI: 0000000000000009
RBP: ffff8800cf988010 R08: 0000000000000000 R09: 0000000000000000
R10: 0065706f63732e74 R11: ffffffff81c4ae00 R12: 0000000000000000
R13: ffff8800cf988010 R14: ffff88040cbff800 R15: ffff8800cf988010
FS: 00007ff020f1b980(0000) GS: ffff88041fc40000(0000) knlGS: 0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000008005003b
CR2: 000000000000809000 CR3: 00000000c9e72000 CR4: 00000000000006e0
Stack:
ffff88040eaf2680 ffff88040eaf2680 0000000000040000 ffffffff8136b42f
ffff88040cbff800 ffff88040eaf2680 00000000b605e950 0000000000400000
ffff88040c4b5000 ffff88040cbff800 ffff8800cf988010 ffffffff81391076
Call Trace:
[<ffffffff8136b42f>] ? scsi_init_io+0x41/0x19d
[<ffffffff81391076>] ? sd_init_command+0x3df/0xbaa
[<ffffffff81365952>] ? scsi_host_alloc_command+0x3e/0ca3
[<ffffffff810881e9>] ? init_timer_key+0xc/0x49
[<ffffffff8136b738>] ? scsi_prep_fn+0xa1/0x132
[<ffffffff81256481>] ? blk_peek_request+0x167/0x206
[<ffffffff81252daf>] ? __blk_run_queue_uncond+0x1e/0x26
[<ffffffff8126e12e>] ? cfq_kick_queue+0x24/0x32
[<ffffffff81057f89>] ? process_one_work+0x154/0x27e
[<ffffffff81058658>] ? worker_thread+0x1d5/0x278
[<ffffffff81058483>] ? rescuer_thread+0x277/0x277
[<ffffffff8105c04c?] ? kthread+0xa7/0xaf
[<ffffffff8105bfa5>] ? kthread_parkme+0x16/0x16
And the rest is off the screen. I guess I'll have to start a git bisect and see where I
end up, but I was hoping to find something in the lkml archives, no such luck.
Any suggestions on how to make Netconsole work better over eth0 then br0 so I can try to catch
these crash dumps? I guess I'll setup another ethernet card on there and try that too...
John
>>>>> "John" == John Stoffel <[email protected]> writes:
John> On Fri, Dec 04, 2015 at 11:28:33PM -0500, John Stoffel wrote:
>>
>> Hi all,
>> Anyway, if I try to boot up anything past the 4.2.6 kernel, the system
>> locks up pretty quickly with an oops message that scrolls off the
>> screen too far. I've got some pictures which I'll attach in a bit,
>> maybe they'll help. So at first I thought it was something to do with
>> bad kworker threads, or SCSI or SATA interactions, but as I tried to
>> configure Netconsole to log to my beaglebone black SBC, I found out
>> that if I compiled and installed 4.4-rc3, started the bridge up (br0),
>> even started KVM, but did NOT start my VMs, the system was stable.
I've now figured out that I can disable all my VMs from autostart, and
the system will come up properly. Then I can setup netconsole to use
the br0 interface, do an "echo t > sysrq" to confirm it's working,
and start up the VMs.
On my most recent bootup, I thought it was ok, since the VMs worked
for a while (10 minutes) and I was starting to re-compile the kernel
again to make more modules compiled in. No luck, I got the following
crash dump (partial) on my netconsole box.
[ 1434.266524] ------------[ cut here ]------------
[ 1434.266643] WARNING: CPU: 2 PID: 179 at block/blk-merge.c:435 blk_rq_map_sg+0x2d9/0x2eb()
[ 1434.266739] Modules linked in: vhost_net vhost macvtap macvlan tun binfmt_misc cpufreq_stats cpuf
req_powersave cpufreq_conservative cpufreq_userspace loop snd_pcm_oss snd_mixer_oss snd_pcm snd_time
r snd soundcore pcspkr serio_raw edac_mce_amd k10temp edac_core sp5100_tco i2c_piix4 asus_atk0110 wm
i shpchp evdev acpi_cpufreq netconsole configfs dm_mod raid1 usbhid md_mod
[ 1434.267691] CPU: 2 PID: 179 Comm: kworker/2:1H Not tainted 4.4.0-rc3 #3
[ 1434.267754] Hardware name: System manufacturer System Product Name/M4A88TD-V EVO/USB3, BIOS 1401
06/11/2010
[ 1434.267851] Workqueue: kblockd cfq_kick_queue
[ 1434.267927] 0000000000000000 ffff88040ba57b78 ffffffff812ded80 0000000000000000
[ 1434.268103] ffff88040ba57bb0 ffffffff81071184 ffffffff812c4cba ffff88034aecee60
[ 1434.268270] 0000000000000000 0000000000000002 ffff88040bd4b7c8 ffff88040ba57bc0
[ 1434.268440] Call Trace:
[ 1434.268501] [<ffffffff812ded80>] dump_stack+0x44/0x55
[ 1434.268565] [<ffffffff81071184>] warn_slowpath_common+0x95/0xae
[ 1434.268628] [<ffffffff812c4cba>] ? blk_rq_map_sg+0x2d9/0x2eb
[ 1434.268688] [<ffffffff81071241>] warn_slowpath_null+0x15/0x17
[ 1434.268749] [<ffffffff812c4cba>] blk_rq_map_sg+0x2d9/0x2eb
[ 1434.268814] [<ffffffff814fe816>] scsi_init_sgtable+0x3f/0x63
[ 1434.268876] [<ffffffff814fec2a>] scsi_init_io+0x47/0x1ab
[ 1434.268937] [<ffffffff81535109>] sd_init_command+0x3e5/0xba6
[ 1434.268997] [<ffffffff814f91d9>] ? scsi_host_alloc_command+0x48/0xb0
[ 1434.269060] [<ffffffff814fee14>] scsi_setup_cmnd+0x86/0x109
[ 1434.269123] [<ffffffff814fef3e>] scsi_prep_fn+0xa7/0x139
[ 1434.269185] [<ffffffff812c0ddd>] blk_peek_request+0x169/0x1de
[ 1434.269246] [<ffffffff81500269>] scsi_request_fn+0x26/0x2a2
[ 1434.269308] [<ffffffff8102f9c4>] ? __switch_to+0x1e9/0x3f1
[ 1434.269372] [<ffffffff812bde39>] __blk_run_queue_uncond+0x22/0x2b
[ 1434.269433] [<ffffffff812bde56>] __blk_run_queue+0x14/0x16
[ 1434.269494] [<ffffffff812d950f>] cfq_kick_queue+0x2a/0x3a
[ 1434.269554] [<ffffffff81082a4e>] process_one_work+0x144/0x217
[ 1434.269618] [<ffffffff81082f9e>] worker_thread+0x1e3/0x28c
[ 1434.269678] [<ffffffff81082dbb>] ? rescuer_thread+0x270/0x270
[ 1434.269738] [<ffffffff81082dbb>] ? rescuer_thread+0x270/0x270
[ 1434.269800] [<ffffffff81086a75>] kthread+0xb2/0xba
[ 1434.269864] [<ffffffff810869c3>] ? kthread_parkme+0x1f/0x1f
[ 1434.269925] [<ffffffff816efc5f>] ret_from_fork+0x3f/0x70
And it stops and the system locks hard, it won't respond to
magic-sysrq at all and I have to hit the reset button. Is there
anything I can provide for more details, or config options I can add
to do better debugging?
So now I'm doing yet another re-compile, but I'm making deadline be my
default scheduler. My system is pretty simple in setup, it's mostly
triple mirrored RAID1 devices:
quad:/sys/devices# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdg1[0] sdc1[3] sde1[1]
976628736 blocks super 1.2 [3/3] [UUU]
bitmap: 0/8 pages [0KB], 65536KB chunk
md4 : active raid1 sdf1[3] sdd1[1] sda1[2]
1953380736 blocks super 1.2 [3/3] [UUU]
bitmap: 0/15 pages [0KB], 65536KB chunk
md0 : active raid1 sdh2[0] sdj2[3] sdi2[4]
185545656 blocks super 1.2 [3/3] [UUU]
bitmap: 1/2 pages [4KB], 65536KB chunk
unused devices: <none>
And once this new kernel is compiled and installed, I'll also change
my disks to deadline scheduler and fire up the VMs to see what
happens.
On 12/05/2015 10:31 AM, John Stoffel wrote:
>>>>>> "John" == John Stoffel <[email protected]> writes:
>
> John> On Fri, Dec 04, 2015 at 11:28:33PM -0500, John Stoffel wrote:
>>>
>>> Hi all,
>
>>> Anyway, if I try to boot up anything past the 4.2.6 kernel, the system
>>> locks up pretty quickly with an oops message that scrolls off the
>>> screen too far. I've got some pictures which I'll attach in a bit,
>>> maybe they'll help. So at first I thought it was something to do with
>>> bad kworker threads, or SCSI or SATA interactions, but as I tried to
>>> configure Netconsole to log to my beaglebone black SBC, I found out
>>> that if I compiled and installed 4.4-rc3, started the bridge up (br0),
>>> even started KVM, but did NOT start my VMs, the system was stable.
>
> I've now figured out that I can disable all my VMs from autostart, and
> the system will come up properly. Then I can setup netconsole to use
> the br0 interface, do an "echo t > sysrq" to confirm it's working,
> and start up the VMs.
>
> On my most recent bootup, I thought it was ok, since the VMs worked
> for a while (10 minutes) and I was starting to re-compile the kernel
> again to make more modules compiled in. No luck, I got the following
> crash dump (partial) on my netconsole box.
>
> [ 1434.266524] ------------[ cut here ]------------
> [ 1434.266643] WARNING: CPU: 2 PID: 179 at block/blk-merge.c:435 blk_rq_map_sg+0x2d9/0x2eb()
This is fixed in current -git, as of a few days ago.
--
Jens Axboe
>>>>> "Jens" == Jens Axboe <[email protected]> writes:
Jens> On 12/05/2015 10:31 AM, John Stoffel wrote:
>>>>>>> "John" == John Stoffel <[email protected]> writes:
>>
John> On Fri, Dec 04, 2015 at 11:28:33PM -0500, John Stoffel wrote:
>>>>
>> On my most recent bootup, I thought it was ok, since the VMs worked
>> for a while (10 minutes) and I was starting to re-compile the kernel
>> again to make more modules compiled in. No luck, I got the following
>> crash dump (partial) on my netconsole box.
>>
>> [ 1434.266524] ------------[ cut here ]------------
>> [ 1434.266643] WARNING: CPU: 2 PID: 179 at block/blk-merge.c:435 blk_rq_map_sg+0x2d9/0x2eb()
Jens> This is fixed in current -git, as of a few days ago.
Thanks! I'll try that out and see how stable it is. I assume it
doesn't matter if it's a deadline or cfq scheduler?
In any case, I see the pull in Linus' tree and I'm building a kernel
now to play with.
Thanks!
John