I am running 2.6.23.1 with kvm built from that tree as a module. My
system is running Debian/testing on a Tyan board with two dual-core
Opteron 2216 processors; each socket has 4G of RAM. I have attached the
serial console dump including a bunch of output from SysRq (gzipped,
because it was 300k otherwise).
I have ran multiple passes of memtest, and I can build the kernel with
-j8, but when I run kvm the host freezes.
A bit about the system:
quark# cat /proc/version
Linux version 2.6.23.1 (root@quark) (gcc version 4.2.1 (Debian 4.2.1-4)) #5 SMP Fri Oct 19 14:10:30 EDT 2007
quark# kvm --help
QEMU PC emulator version 0.9.0, Copyright (c) 2003-2007 Fabrice Bellard
My kvm is started as follows; the guest is a 64bit version of RHEL5
(redhatized 2.6.18 kernel).
kvm -m 512 -vnc quark:12 -k en-us -usbdevice tablet -serial vc -k en-us -boot c -hda rhel5.img -net nic,vlan=0 -net user,vlan=0 -redir tcp:2222::22 -redir tcp:8080::8080 -redir udp:8080::8080
I added --no-acpi with no improvement.
I also tried running with taskset -c 0 with no improvement.
Once the system is booted, I attached using vnc, then I ssh in and ran
'svn update'... and the host machine froze.
The last messages I on my serial console are:
kvm: unhandled rdmsr: 0x417
kvm: unhandled rdmsr: 0xc0000400
kvm: unhandled rdmsr: 0xc0000401
kvm: unhandled rdmsr: 0xc0000402
kvm: unhandled rdmsr: 0xc0000403
kvm: unhandled rdmsr: 0xc0000404
kvm: unhandled rdmsr: 0xc0000405
kvm: unhandled rdmsr: 0xc0000406
kvm: unhandled rdmsr: 0xc0000407
And then freeze.
As this is pretty easy to reproduce, I am quite happy to run any patches
or configurations that people come up with :)
I noticed that linus merged a bunch of KVM changes last week. I will
try those out next.
Thanks in advance for looking at this.
-Bart
--
WebSig: http://www.jukie.net/~bart/sig/
* Bart Trojanowski <[email protected]> [071019 17:00]:
<snip>
>
> Once the system is booted, I attached using vnc, then I ssh in and ran
> 'svn update'... and the host machine froze.
>
> The last messages I on my serial console are:
>
> kvm: unhandled rdmsr: 0x417
> kvm: unhandled rdmsr: 0xc0000400
<snip>
> I noticed that linus merged a bunch of KVM changes last week. I will
> try those out next.
It looks like Linus' tree has the fix already.
-Bart
--
WebSig: http://www.jukie.net/~bart/sig/
* Bart Trojanowski <[email protected]> [071019 20:03]:
> * Bart Trojanowski <[email protected]> [071019 17:00]:
> <snip>
> >
> > Once the system is booted, I attached using vnc, then I ssh in and ran
> > 'svn update'... and the host machine froze.
> >
> > The last messages I on my serial console are:
> >
> > kvm: unhandled rdmsr: 0x417
> > kvm: unhandled rdmsr: 0xc0000400
> <snip>
> > I noticed that linus merged a bunch of KVM changes last week. I will
> > try those out next.
>
> It looks like Linus' tree has the fix already.
I spoke too soon. The latest from Linus' tree also freezes... but it
took overnight to get there. There was no particular load on it.
This system was previously running 2.6.22.y w/o problems with a similar
kvm pattern.
This time it froze w/o showing any kvm rdmsr messages. And while it
froze sometime over night, I cannot be sure when. Here is the SysRq's
"showPc" output:
SysRq : Show Regs
CPU 2:
Modules linked in: kvm_amd kvm rtc_cmos rtc_core rtc_lib
cpufreq_userspace cpufreq_powersave cpufreq_conservative ipv6 bridge
ext2 mbcache loop pcspkr button dm_mirror dm_snapshot dm_mod ide_generic
ide_cd pata_acpi ata_generic serverworks generic ide_core
Pid: 0, comm: swapper Not tainted 2.6.23-git-kvm-gc4ec2071 #1
RIP: 0010:[<ffffffff8020b0db>] [<ffffffff8020b0db>] default_idle+0x29/0x3d
RSP: 0018:ffff810140a55f30 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff8066a040 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff805c3350
RBP: ffffffff8066fa00 R08: ffff810140040560 R09: ffff8101006e7e70
R10: ffff810143e0a900 R11: ffffffff8066fa00 R12: ffffffff8024bc8e
R13: ffff810140a55ec8 R14: 000000000d6ed51a R15: ffffffff8066fa00
FS: 00002aaea92f0bf0(0000) GS:ffff810140a1d300(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002add9e236140 CR3: 0000000143def000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8020b182>] cpu_idle+0x93/0xbb
Are there things I should be looking for?
-Bart
--
WebSig: http://www.jukie.net/~bart/sig/
Bart Trojanowski wrote:
> * Bart Trojanowski <[email protected]> [071019 20:03]:
>
>> * Bart Trojanowski <[email protected]> [071019 17:00]:
>> <snip>
>>
>>> Once the system is booted, I attached using vnc, then I ssh in and ran
>>> 'svn update'... and the host machine froze.
>>>
>>> The last messages I on my serial console are:
>>>
>>> kvm: unhandled rdmsr: 0x417
>>> kvm: unhandled rdmsr: 0xc0000400
>>>
>> <snip>
>>
>>> I noticed that linus merged a bunch of KVM changes last week. I will
>>> try those out next.
>>>
>> It looks like Linus' tree has the fix already.
>>
>
> I spoke too soon. The latest from Linus' tree also freezes... but it
> took overnight to get there. There was no particular load on it.
>
> This system was previously running 2.6.22.y w/o problems with a similar
> kvm pattern.
>
> This time it froze w/o showing any kvm rdmsr messages. And while it
> froze sometime over night, I cannot be sure when. Here is the SysRq's
> "showPc" output:
>
Does 2.6.23 hang immediately? If so, these may be two separate problems
and it's worth bisecting to find out what's the fix in 2.6.24-rc.
Of course we'll have to address the later hang as well.
--
error compiling committee.c: too many arguments to function
* Avi Kivity <[email protected]> [071021 07:23]:
> Does 2.6.23 hang immediately? If so, these may be two separate problems
> and it's worth bisecting to find out what's the fix in 2.6.24-rc.
>
> Of course we'll have to address the later hang as well.
Avi,
it does not hang immediately. It seems to wait until I go to bed to
hang -- so it's going to be really fun to rebase. It could be triggered
by something that runs off cron in the guest or host. I am not sure.
At this point I know that 2.6.23 and 2.6.24-rc0 hung, and 2.6.22.10
(with kvm-userspace.git) ran overnight.
I will try some of the 23-rc's next.
-Bart
--
WebSig: http://www.jukie.net/~bart/sig/
Bart Trojanowski wrote:
> * Avi Kivity <[email protected]> [071021 07:23]:
>
>> Does 2.6.23 hang immediately? If so, these may be two separate problems
>> and it's worth bisecting to find out what's the fix in 2.6.24-rc.
>>
>> Of course we'll have to address the later hang as well.
>>
>
> Avi,
>
> it does not hang immediately. It seems to wait until I go to bed to
> hang -- so it's going to be really fun to rebase. It could be triggered
> by something that runs off cron in the guest or host. I am not sure.
>
> At this point I know that 2.6.23 and 2.6.24-rc0 hung, and 2.6.22.10
> (with kvm-userspace.git) ran overnight.
>
> I will try some of the 23-rc's next.
>
I'm not sure that's useful -- very little changed after 2.6.23-rc1 (10
patches).
There were 92 kvm patches in 2.6.23, so a bisect should take about a
week worst case.
--
error compiling committee.c: too many arguments to function
* Avi Kivity <[email protected]> [071022 09:42]:
> I'm not sure that's useful -- very little changed after 2.6.23-rc1 (10
> patches).
>
> There were 92 kvm patches in 2.6.23, so a bisect should take about a
> week worst case.
I'll get started tonight.
-Bart
--
WebSig: http://www.jukie.net/~bart/sig/
Bart Trojanowski wrote:
> * Avi Kivity <[email protected]> [071022 09:42]:
>
>> I'm not sure that's useful -- very little changed after 2.6.23-rc1 (10
>> patches).
>>
>> There were 92 kvm patches in 2.6.23, so a bisect should take about a
>> week worst case.
>>
>
> I'll get started tonight.
>
>
Thanks, that will be most helpful. If userspace won't compile against
some intermediate version, let me know the commit hash and I'll add a fixup.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
* Avi Kivity <[email protected]> [071023 10:21]:
> Thanks, that will be most helpful. If userspace won't compile against
> some intermediate version, let me know the commit hash and I'll add a fixup.
A bisect didn't find anything between 2.6.22 and 2.6.23. Which made me
very confused. I think, I must have used the Debian userland tools...
which is still bad and I will double check that with 2.6.23.
-Bart
Bart Trojanowski wrote:
> * Avi Kivity <[email protected]> [071023 10:21]:
>
>> Thanks, that will be most helpful. If userspace won't compile against
>> some intermediate version, let me know the commit hash and I'll add a fixup.
>>
>
> A bisect didn't find anything between 2.6.22 and 2.6.23. Which made me
> very confused. I think, I must have used the Debian userland tools...
> which is still bad and I will double check that with 2.6.23.
>
Yes. You can use the same tools for all versions from .22 and up.
Another thing to try is whether the lockup reproduces with 'taskset 1'.
Maybe it is preemption related.
Thanks for the effort.
--
Any sufficiently difficult bug is indistinguishable from a feature.