It seems for some reason, some linux kernel with KAISER/KPTI patch
can't work with qemu + haxm.
The mainline kernel from Linus is fine. But the patch to 4.4/4.10 doesn't work.
I am not familiar with HAXM and KPTI either. so not sure if this is a
qemu bug or KPTI bug or haxm bug.
The same kernel works fine under qemu + kvm.
This is the way to reproduce it:
1. Download qemu for windows, follow instructions here:
https://www.qemu.org/2017/11/22/haxm-usage-windows/
2. Build a kernel with KAISER/KPTI, I am using kernel here:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/log/?h=linux-4.4.y
Or follow instruction here to build a 4.10 kernel:
https://github.com/IAIK/KAISER/tree/master/KAISER
3. Build an ext2 image which has a simple init:
dd of=img count=8192 bs=4096 if=/dev/zero
mke2fs img
gcc --static -o init init.c
debugfs -R "write init init" -w img
cat init.c
#include <stdio.h>
#include <time.h>
int main()
{
while(1) {
printf("This is init %d\n", time(NULL));
sleep(3600);
}
}
4. copy kernel and disk image generated from 3 to windows and run it:
qemu-system-x86_64.exe -kernel bzImage -hda img -append "init=/init
root=/dev/sda" -serial stdio -accel hax
You will see kernel panic or "vpu shutdown reqeust" to qemu.
I tried some debug, it seems it crashed after switch CR3:
I tried 2 different kernel, so actual crash points are different, but
they have same pattern. It crashed when trying to pop %rax after
switch CR3:
for 4.10 with kaiser patch: it crashed at this point:
Dump of assembler code for function ret_from_fork:
0xffffffff8190ad00 <+0>: push %rbp
0xffffffff8190ad01 <+1>: mov %rsp,%rbp
0xffffffff8190ad04 <+4>: mov %rax,%rdi
0xffffffff8190ad07 <+7>: callq 0xffffffff81083670 <schedule_tail>
0xffffffff8190ad0c <+12>: test %rbx,%rbx
0xffffffff8190ad0f <+15>: jne 0xffffffff8190ad32 <ret_from_fork+50>
0xffffffff8190ad11 <+17>: lea 0x8(%rsp),%rdi
0xffffffff8190ad16 <+22>: callq 0xffffffff810027a0
<syscall_return_slowpath>
0xffffffff8190ad1b <+27>: push %rax
0xffffffff8190ad1c <+28>: mov %cr3,%rax
0xffffffff8190ad1f <+31>: or $0x1000,%rax
0xffffffff8190ad25 <+37>: mov %rax,%cr3
0xffffffff8190ad28 <+40>: pop %rax
<----------------------------------- crashed here
0xffffffff8190ad29 <+41>: swapgs
0xffffffff8190ad2c <+44>: pop %rbp
For 4.4.110:
Dump of assembler code for function opportunistic_sysret_failed:
0xffffffff81887165 <+0>: jmp 0xffffffff81887181
<opportunistic_sysret_failed+28>
0xffffffff81887167 <+2>: mov %cr3,%rax
0xffffffff8188716a <+5>: or %gs:0xf0c0,%rax
0xffffffff81887173 <+14>: js 0xffffffff8188717d
<opportunistic_sysret_failed+24>
0xffffffff81887175 <+16>: mov %al,%gs:0xf0c7
0xffffffff8188717d <+24>: mov %rax,%cr3
0xffffffff81887180 <+27>: pop %rax
<---------------------------------------- crashed here
0xffffffff81887181 <+28>: swapgs
0xffffffff81887184 <+31>: jmpq 0xffffffff81887ab0 <restore_c_regs_and_iret>
0xffffffff81887189 <+36>: nopl 0x0(%rax)
On Thu, Jan 4, 2018 at 2:13 PM, lepton <[email protected]> wrote:
> It seems for some reason, some linux kernel with KAISER/KPTI patch
> can't work with qemu + haxm.
> The mainline kernel from Linus is fine. But the patch to 4.4/4.10 doesn't work.
>
> I am not familiar with HAXM and KPTI either. so not sure if this is a
> qemu bug or KPTI bug or haxm bug.
>
> The same kernel works fine under qemu + kvm.
>
> This is the way to reproduce it:
>
> 1. Download qemu for windows, follow instructions here:
>
> https://www.qemu.org/2017/11/22/haxm-usage-windows/
>
> 2. Build a kernel with KAISER/KPTI, I am using kernel here:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/log/?h=linux-4.4.y
> Or follow instruction here to build a 4.10 kernel:
> https://github.com/IAIK/KAISER/tree/master/KAISER
>
> 3. Build an ext2 image which has a simple init:
> dd of=img count=8192 bs=4096 if=/dev/zero
> mke2fs img
> gcc --static -o init init.c
> debugfs -R "write init init" -w img
> cat init.c
> #include <stdio.h>
> #include <time.h>
>
> int main()
> {
> while(1) {
> printf("This is init %d\n", time(NULL));
> sleep(3600);
> }
> }
>
> 4. copy kernel and disk image generated from 3 to windows and run it:
> qemu-system-x86_64.exe -kernel bzImage -hda img -append "init=/init
> root=/dev/sda" -serial stdio -accel hax
>
> You will see kernel panic or "vpu shutdown reqeust" to qemu.
FYI, this was fixed by https://www.spinics.net/lists/stable/msg209612.html
On Tue, Jan 9, 2018 at 5:36 PM, lepton <[email protected]> wrote:
> I tried some debug, it seems it crashed after switch CR3:
>
>
> I tried 2 different kernel, so actual crash points are different, but
> they have same pattern. It crashed when trying to pop %rax after
> switch CR3:
>
> for 4.10 with kaiser patch: it crashed at this point:
>
> Dump of assembler code for function ret_from_fork:
> 0xffffffff8190ad00 <+0>: push %rbp
> 0xffffffff8190ad01 <+1>: mov %rsp,%rbp
> 0xffffffff8190ad04 <+4>: mov %rax,%rdi
> 0xffffffff8190ad07 <+7>: callq 0xffffffff81083670 <schedule_tail>
> 0xffffffff8190ad0c <+12>: test %rbx,%rbx
> 0xffffffff8190ad0f <+15>: jne 0xffffffff8190ad32 <ret_from_fork+50>
> 0xffffffff8190ad11 <+17>: lea 0x8(%rsp),%rdi
> 0xffffffff8190ad16 <+22>: callq 0xffffffff810027a0
> <syscall_return_slowpath>
> 0xffffffff8190ad1b <+27>: push %rax
> 0xffffffff8190ad1c <+28>: mov %cr3,%rax
> 0xffffffff8190ad1f <+31>: or $0x1000,%rax
> 0xffffffff8190ad25 <+37>: mov %rax,%cr3
> 0xffffffff8190ad28 <+40>: pop %rax
> <----------------------------------- crashed here
> 0xffffffff8190ad29 <+41>: swapgs
> 0xffffffff8190ad2c <+44>: pop %rbp
>
>
> For 4.4.110:
>
> Dump of assembler code for function opportunistic_sysret_failed:
> 0xffffffff81887165 <+0>: jmp 0xffffffff81887181
> <opportunistic_sysret_failed+28>
> 0xffffffff81887167 <+2>: mov %cr3,%rax
> 0xffffffff8188716a <+5>: or %gs:0xf0c0,%rax
> 0xffffffff81887173 <+14>: js 0xffffffff8188717d
> <opportunistic_sysret_failed+24>
> 0xffffffff81887175 <+16>: mov %al,%gs:0xf0c7
> 0xffffffff8188717d <+24>: mov %rax,%cr3
> 0xffffffff81887180 <+27>: pop %rax
> <---------------------------------------- crashed here
> 0xffffffff81887181 <+28>: swapgs
> 0xffffffff81887184 <+31>: jmpq 0xffffffff81887ab0 <restore_c_regs_and_iret>
> 0xffffffff81887189 <+36>: nopl 0x0(%rax)
>
> On Thu, Jan 4, 2018 at 2:13 PM, lepton <[email protected]> wrote:
>> It seems for some reason, some linux kernel with KAISER/KPTI patch
>> can't work with qemu + haxm.
>> The mainline kernel from Linus is fine. But the patch to 4.4/4.10 doesn't work.
>>
>> I am not familiar with HAXM and KPTI either. so not sure if this is a
>> qemu bug or KPTI bug or haxm bug.
>>
>> The same kernel works fine under qemu + kvm.
>>
>> This is the way to reproduce it:
>>
>> 1. Download qemu for windows, follow instructions here:
>>
>> https://www.qemu.org/2017/11/22/haxm-usage-windows/
>>
>> 2. Build a kernel with KAISER/KPTI, I am using kernel here:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/log/?h=linux-4.4.y
>> Or follow instruction here to build a 4.10 kernel:
>> https://github.com/IAIK/KAISER/tree/master/KAISER
>>
>> 3. Build an ext2 image which has a simple init:
>> dd of=img count=8192 bs=4096 if=/dev/zero
>> mke2fs img
>> gcc --static -o init init.c
>> debugfs -R "write init init" -w img
>> cat init.c
>> #include <stdio.h>
>> #include <time.h>
>>
>> int main()
>> {
>> while(1) {
>> printf("This is init %d\n", time(NULL));
>> sleep(3600);
>> }
>> }
>>
>> 4. copy kernel and disk image generated from 3 to windows and run it:
>> qemu-system-x86_64.exe -kernel bzImage -hda img -append "init=/init
>> root=/dev/sda" -serial stdio -accel hax
>>
>> You will see kernel panic or "vpu shutdown reqeust" to qemu.