Using 2.6.29 or 2.6.28 as a 64-bit Xen domU, a number of ptrace()
users seem to have issues with unexpected breakpoints. ltrace and gdb
both seem to be affected, under both 64-bit and 32-bit userspace.
32-bit kernels do not seem to be affected. Typical symptoms look like:
x86 li63-205:/# ltrace /bin/true
unexpected breakpoint at 0xf7e6d89f
unexpected breakpoint at 0xf7e60a3f
unexpected breakpoint at 0xf7e6464f
unexpected breakpoint at 0x804933f
unexpected breakpoint at 0xf7ea509f
unexpected breakpoint at 0xf7e9b1ff
unexpected breakpoint at 0xf7efef66
+++ exited (status 0) +++
x64 li63-205:~/linux-2.6# ltrace true
unexpected breakpoint at 0x7f3379878f1f
unexpected breakpoint at 0x7f337986ca3f
unexpected breakpoint at 0x7f337986fd3f
unexpected breakpoint at 0x402bdf
unexpected breakpoint at 0x7f33798b07bf
unexpected breakpoint at 0x7f33798a696f
unexpected breakpoint at 0x7f3379905e3f
+++ exited (status 0) +++
li63-205:~# gdb /bin/true
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(no debugging symbols found)
(gdb) run
Starting program: /bin/true
(no debugging symbols found)
(no debugging symbols found)
Program received signal SIGTRAP, Trace/breakpoint trap.
0x00007f640680e4e7 in ?? () from /lib64/ld-linux-x86-64.so.2
(gdb)
In the specific case of the above gdb session, here is /proc/pid/maps
from the tracee:
00400000-00407000 r-xp 00000000 ca:00 9531
/bin/true
00607000-00608000 rw-p 00007000 ca:00 9531
/bin/true
7f64067f9000-7f6406816000 r-xp 00000000 ca:00 1678
/lib/ld-2.9.so
7f6406a12000-7f6406a15000 rw-p 7f6406a12000 00:00 0
7f6406a15000-7f6406a17000 rw-p 0001c000 ca:00 1678
/lib/ld-2.9.so
7fff0ea01000-7fff0ea16000 rw-p 7ffffffea000 00:00 0 [stack]
7fff0ebfe000-7fff0ebff000 r-xp 7fff0ebfe000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
strace seems(?) to be unaffected.
I've attached my .config, dmesg, and strace output for running ltrace
(32-bit) on /bin/true on the affected system. Unfortunately, I'm not
enough of a ptrace guru to interpret what the cause might be.
Bryan Donlan wrote:
> Using 2.6.29 or 2.6.28 as a 64-bit Xen domU, a number of ptrace()
> users seem to have issues with unexpected breakpoints. ltrace and gdb
> both seem to be affected, under both 64-bit and 32-bit userspace.
> 32-bit kernels do not seem to be affected. Typical symptoms look like:
>
Thanks for the reminder. I'd noticed it in passing, but haven't
investigated it.
J
> x86 li63-205:/# ltrace /bin/true
> unexpected breakpoint at 0xf7e6d89f
> unexpected breakpoint at 0xf7e60a3f
> unexpected breakpoint at 0xf7e6464f
> unexpected breakpoint at 0x804933f
> unexpected breakpoint at 0xf7ea509f
> unexpected breakpoint at 0xf7e9b1ff
> unexpected breakpoint at 0xf7efef66
> +++ exited (status 0) +++
>
> x64 li63-205:~/linux-2.6# ltrace true
> unexpected breakpoint at 0x7f3379878f1f
> unexpected breakpoint at 0x7f337986ca3f
> unexpected breakpoint at 0x7f337986fd3f
> unexpected breakpoint at 0x402bdf
> unexpected breakpoint at 0x7f33798b07bf
> unexpected breakpoint at 0x7f33798a696f
> unexpected breakpoint at 0x7f3379905e3f
> +++ exited (status 0) +++
>
> li63-205:~# gdb /bin/true
> GNU gdb 6.8-debian
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu"...
> (no debugging symbols found)
> (gdb) run
> Starting program: /bin/true
> (no debugging symbols found)
> (no debugging symbols found)
>
> Program received signal SIGTRAP, Trace/breakpoint trap.
> 0x00007f640680e4e7 in ?? () from /lib64/ld-linux-x86-64.so.2
> (gdb)
>
> In the specific case of the above gdb session, here is /proc/pid/maps
> from the tracee:
> 00400000-00407000 r-xp 00000000 ca:00 9531
> /bin/true
> 00607000-00608000 rw-p 00007000 ca:00 9531
> /bin/true
> 7f64067f9000-7f6406816000 r-xp 00000000 ca:00 1678
> /lib/ld-2.9.so
> 7f6406a12000-7f6406a15000 rw-p 7f6406a12000 00:00 0
> 7f6406a15000-7f6406a17000 rw-p 0001c000 ca:00 1678
> /lib/ld-2.9.so
> 7fff0ea01000-7fff0ea16000 rw-p 7ffffffea000 00:00 0 [stack]
> 7fff0ebfe000-7fff0ebff000 r-xp 7fff0ebfe000 00:00 0 [vdso]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
> [vsyscall]
>
> strace seems(?) to be unaffected.
>
> I've attached my .config, dmesg, and strace output for running ltrace
> (32-bit) on /bin/true on the affected system. Unfortunately, I'm not
> enough of a ptrace guru to interpret what the cause might be.
Bryan Donlan wrote:
> Using 2.6.29 or 2.6.28 as a 64-bit Xen domU, a number of ptrace()
> users seem to have issues with unexpected breakpoints. ltrace and gdb
> both seem to be affected, under both 64-bit and 32-bit userspace.
> 32-bit kernels do not seem to be affected. Typical symptoms look like:
>
It looks like this is because the kernel sets up int3 (breakpoint) and
debug (watchpoints, etc) to be on a separate debug stack in the tss.
Xen doesn't do this (and doesn't appear to have a mechanism to do so),
so I guess the on-stack format isn't what the kernel expects. Does the
patch below work?
J
From: Jeremy Fitzhardinge <[email protected]>
Date: Sun, 29 Mar 2009 19:56:29 -0700
Subject: [PATCH] xen/x86-64: fix breakpoints and hardware watchpoints
Native x86-64 uses the IST mechanism to run int3 and debug traps on
an alternative stack. Xen does not do this, and so the frames were
being misinterpreted by the ptrace code. This change special-cases
these two exceptions by using Xen variants which run on the normal
kernel stack properly.
Signed-off-by: Jeremy Fitzhardinge <[email protected]>
diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h
index 0d53425..d696185 100644
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -13,6 +13,8 @@ asmlinkage void divide_error(void);
asmlinkage void debug(void);
asmlinkage void nmi(void);
asmlinkage void int3(void);
+asmlinkage void xen_debug(void);
+asmlinkage void xen_int3(void);
asmlinkage void overflow(void);
asmlinkage void bounds(void);
asmlinkage void invalid_op(void);
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index 3f129d9..6cde382 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -1383,6 +1383,10 @@ END(xen_failsafe_callback)
paranoidzeroentry_ist debug do_debug DEBUG_STACK
paranoidzeroentry_ist int3 do_int3 DEBUG_STACK
+#ifdef CONFIG_XEN
+zeroentry xen_debug do_debug
+zeroentry xen_int3 do_int3
+#endif
paranoiderrorentry stack_segment do_stack_segment
errorentry general_protection do_general_protection
errorentry page_fault do_page_fault
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 02b169e..f98124f 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -20,6 +20,7 @@
#include <linux/delay.h>
#include <linux/start_kernel.h>
#include <linux/sched.h>
+#include <linux/kprobes.h>
#include <linux/bootmem.h>
#include <linux/module.h>
#include <linux/mm.h>
@@ -44,6 +45,7 @@
#include <asm/processor.h>
#include <asm/proto.h>
#include <asm/msr-index.h>
+#include <asm/traps.h>
#include <asm/setup.h>
#include <asm/desc.h>
#include <asm/pgtable.h>
@@ -435,11 +437,24 @@ static void xen_write_ldt_entry(struct desc_struct *dt, int entrynum,
static int cvt_gate_to_trap(int vector, const gate_desc *val,
struct trap_info *info)
{
+ unsigned long addr;
+
if (val->type != GATE_TRAP && val->type != GATE_INTERRUPT)
return 0;
info->vector = vector;
- info->address = gate_offset(*val);
+
+ addr = gate_offset(*val);
+#ifdef CONFIG_X86_64
+ if (addr == (unsigned long)debug)
+ addr = (unsigned long)xen_debug;
+ else if (addr == (unsigned long)int3)
+ addr = (unsigned long)xen_int3;
+ else
+ WARN_ON(val->ist != 0);
+#endif /* CONFIG_X86_64 */
+ info->address = addr;
+
info->cs = gate_segment(*val);
info->flags = val->dpl;
/* interrupt gates clear IF */
On Sun, Mar 29, 2009 at 11:01 PM, Jeremy Fitzhardinge <[email protected]> wrote:
> Bryan Donlan wrote:
>>
>> Using 2.6.29 or 2.6.28 as a 64-bit Xen domU, a number of ptrace()
>> users seem to have issues with unexpected breakpoints. ltrace and gdb
>> both seem to be affected, under both 64-bit and 32-bit userspace.
>> 32-bit kernels do not seem to be affected. Typical symptoms look like:
>>
>
> It looks like this is because the kernel sets up int3 (breakpoint) and debug
> (watchpoints, etc) to be on a separate debug stack in the tss. ?Xen doesn't
> do this (and doesn't appear to have a mechanism to do so), so I guess the
> on-stack format isn't what the kernel expects. ?Does the patch below work?
Hi,
Sorry for the late reply; this message never made it to my inbox for
some reason...
I did try the patch, and saw similar results to Mark; ptrace works,
but lots of warnings:
------------[ cut here ]------------
WARNING: at arch/x86/xen/enlighten.c:447 cvt_gate_to_trap+0xe6/0xf0()
Modules linked in:
Pid: 0, comm: swapper Tainted: G W
2.6.30-rc3-ptracefix-00330-g6d03473 #4
Call Trace:
[<ffffffff8075e9b0>] ? stack_segment+0x0/0x30
[<ffffffff8075e9b0>] ? stack_segment+0x0/0x30
[<ffffffff8023ec4a>] ? warn_slowpath+0xea/0x160
[<ffffffff8020dd79>] ? xen_force_evtchn_callback+0x9/0x10
[<ffffffff8020e512>] ? check_events+0x12/0x20
[<ffffffff8020dd79>] ? xen_force_evtchn_callback+0x9/0x10
[<ffffffff8020e512>] ? check_events+0x12/0x20
[<ffffffff8020e512>] ? check_events+0x12/0x20
[<ffffffff8020e4ff>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff8023f86f>] ? vprintk+0x1df/0x3f0
[<ffffffff8020bb49>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[<ffffffff8020d928>] ? make_lowmem_page_readonly+0x28/0x40
[<ffffffff8075e9b0>] ? stack_segment+0x0/0x30
[<ffffffff8020a996>] ? cvt_gate_to_trap+0xe6/0xf0
[<ffffffff8020a9f9>] ? xen_convert_trap_info+0x59/0xa0
[<ffffffff8020b0cf>] ? xen_load_idt+0x3f/0x70
[<ffffffff809b3b3f>] ? cpu_init+0xf0/0x2da
[<ffffffff809b19a6>] ? cpu_bringup_and_idle+0x6/0x71
---[ end trace 4eaa2a86a8e2da36 ]---
Thanks,
Bryan Donlan
Bryan Donlan wrote:
> On Sun, Mar 29, 2009 at 11:01 PM, Jeremy Fitzhardinge <[email protected]> wrote:
>
>> Bryan Donlan wrote:
>>
>>> Using 2.6.29 or 2.6.28 as a 64-bit Xen domU, a number of ptrace()
>>> users seem to have issues with unexpected breakpoints. ltrace and gdb
>>> both seem to be affected, under both 64-bit and 32-bit userspace.
>>> 32-bit kernels do not seem to be affected. Typical symptoms look like:
>>>
>>>
>> It looks like this is because the kernel sets up int3 (breakpoint) and debug
>> (watchpoints, etc) to be on a separate debug stack in the tss. Xen doesn't
>> do this (and doesn't appear to have a mechanism to do so), so I guess the
>> on-stack format isn't what the kernel expects. Does the patch below work?
>>
>
> Hi,
>
> Sorry for the late reply; this message never made it to my inbox for
> some reason...
> I did try the patch, and saw similar results to Mark; ptrace works,
> but lots of warnings:
>
Thanks. Those warnings are harmless, but I have an updated version of
the fix to suppress them in xen-tip/next.
J