Hi all,
after converting a host from using a ubuntu patched 2.6.22 domU to using
2.6.23 xen via paravirt_ops (no configuration change except devicename change
in fstab, and another getty running) ldconfig no longer picks up the nosegneg
libc leading to segfaults for some programs.
/etc/ld.so.conf.d/xen.conf:
hwcap 0 nosegneg
On ubuntu's 2.6.22-xen:
ldd /usr/bin/perl
linux-gate.so.1 => (0xf57fe000)
libdl.so.2 => /lib/tls/i686/nosegneg/libdl.so.2 (0xb7f90000)
libm.so.6 => /lib/tls/i686/nosegneg/libm.so.6 (0xb7f6b000)
libpthread.so.0 => /lib/tls/i686/nosegneg/libpthread.so.0 (0xb7f53000)
libc.so.6 => /lib/tls/i686/nosegneg/libc.so.6 (0xb7e05000)
libcrypt.so.1 => /lib/tls/i686/nosegneg/libcrypt.so.1 (0xb7dd7000)
/lib/ld-linux.so.2 (0xb7f9d000)
On selfcompiled 2.6.23.9:
ldd /usr/bin/perl
linux-gate.so.1 => (0xf57fc000)
libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7f3b000)
libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f16000)
libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7efe000)
libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7db3000)
libcrypt.so.1 => /lib/tls/i686/cmov/libcrypt.so.1 (0xb7d85000)
/lib/ld-linux.so.2 (0xb7f46000)
I now discover, that after some playing around (during discovering what the
problem is) i must have changed something, because ldd still shows the use of
the i686/cmov libraries, but applications reliably causing crashes earlier,
dont do so anymore.
Perhaps Im all wrong, and paravirt_ops doesnt need the nosegneg versions and
just a ldconfig or so (which seems strange because the result before and
after the ldconfig were the same).
Greetings,
Andres Freund
PS: I did this conversion mainly, because I was getting quite random crashes
inside the domU. I guess, as this was a vendor kernel + xensource xen kernel,
this isnt appropriate for lkml? As well as the crashes of the Dom0 (ubuntu
again)?
Andres Freund wrote:
> after converting a host from using a ubuntu patched 2.6.22 domU to using
> 2.6.23 xen via paravirt_ops (no configuration change except devicename change
> in fstab, and another getty running) ldconfig no longer picks up the nosegneg
> libc leading to segfaults for some programs.
>
It shouldn't ever lead to segfaults. At worst, you should get poor
performance because the hypervisor emulates the -ve segment offset.
> /etc/ld.so.conf.d/xen.conf:
> hwcap 0 nosegneg
>
This looks OK. If this doesn't work, then it suggests there's something
wrong with the kernel vdso which is not exporting the nonegseg
(pseudo-)hardware capability. But we've gone over this several times
now, and I was fairly sure we'd finally got it right - works for me, at
least.
The alternative is that there's some bug on the userspace side in your
version of Ubuntu...
Roland?
> On ubuntu's 2.6.22-xen:
> ldd /usr/bin/perl
> linux-gate.so.1 => (0xf57fe000)
> libdl.so.2 => /lib/tls/i686/nosegneg/libdl.so.2 (0xb7f90000)
> libm.so.6 => /lib/tls/i686/nosegneg/libm.so.6 (0xb7f6b000)
> libpthread.so.0 => /lib/tls/i686/nosegneg/libpthread.so.0 (0xb7f53000)
> libc.so.6 => /lib/tls/i686/nosegneg/libc.so.6 (0xb7e05000)
> libcrypt.so.1 => /lib/tls/i686/nosegneg/libcrypt.so.1 (0xb7dd7000)
> /lib/ld-linux.so.2 (0xb7f9d000)
>
> On selfcompiled 2.6.23.9:
> ldd /usr/bin/perl
> linux-gate.so.1 => (0xf57fc000)
> libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7f3b000)
> libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f16000)
> libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7efe000)
> libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7db3000)
> libcrypt.so.1 => /lib/tls/i686/cmov/libcrypt.so.1 (0xb7d85000)
> /lib/ld-linux.so.2 (0xb7f46000)
>
>
> I now discover, that after some playing around (during discovering what the
> problem is) i must have changed something, because ldd still shows the use of
> the i686/cmov libraries, but applications reliably causing crashes earlier,
> dont do so anymore.
>
As I said, you shouldn't be seeing crashes either way. If you are, it
suggests a hypervisor bug. Does "xm dmesg" show anything?
> Perhaps Im all wrong, and paravirt_ops doesnt need the nosegneg versions and
> just a ldconfig or so (which seems strange because the result before and
> after the ldconfig were the same).
>
nosegneg is still necessary, and it should still be working as it did
before.
> PS: I did this conversion mainly, because I was getting quite random crashes
> inside the domU. I guess, as this was a vendor kernel + xensource xen kernel,
> this isnt appropriate for lkml? As well as the crashes of the Dom0 (ubuntu
> again)?
>
Yeah, anything which isn't 2.6.18-xen isn't officially "Xen project",
but its worth reporting the bugs to xen-devel (and Ubuntu, of course).
And to me, for that matter. I won't be able to help directly, but the
crashes may highlight things we need to be careful about in the pvops
Xen support.
J
Hi Jeremy,
On Wednesday 28 November 2007, Jeremy Fitzhardinge wrote in "Re: "hwcap 0
nosegneg" doesnt work with paravirt_ops xen as of 2.6.23.9":
> Andres Freund wrote:
> > after converting a host from using a ubuntu patched 2.6.22 domU to using
> > 2.6.23 xen via paravirt_ops (no configuration change except devicename
> > change in fstab, and another getty running) ldconfig no longer picks up
> > the nosegneg libc leading to segfaults for some programs.
> It shouldn't ever lead to segfaults. At worst, you should get poor
> performance because the hypervisor emulates the -ve segment offset.
strace -ff
perl /usr/bin/debsums --generate=nocheck -sp /var/cache/apt/archives
...
open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 4
fstat64(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
getdents64(4, /* 3 entries */, 4096) = 72
getdents64(4, /* 0 entries */, 4096) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 3238 detached
root@test:~# gdb perl5.8.8
GNU gdb 6.6-debian
...
Starting
program: /usr/bin/perl5.8.8 /usr/bin/debsums --generate=nocheck -sp /var/cache/apt/archives
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New Thread -1209940304 (LWP 3356)]
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1209940304 (LWP 3356)]
0xb7eda3d7 in readdir64_r () from /lib/tls/i686/cmov/libc.so.6
(gdb) bt
#0 0xb7eda3d7 in readdir64_r () from /lib/tls/i686/cmov/libc.so.6
#1 0x080fdca5 in Perl_pp_readdir ()
#2 0x080c0d29 in Perl_runops_standard ()
#3 0x0806727a in perl_run ()
#4 0x08063732 in main ()
This was before running an ldconfig. After running, it still selects the wrong
libraries, but doesnt crash anymore (hm?).
> > /etc/ld.so.conf.d/xen.conf:
> > hwcap 0 nosegneg
> This looks OK. If this doesn't work, then it suggests there's something
> wrong with the kernel vdso which is not exporting the nonegseg
> (pseudo-)hardware capability. But we've gone over this several times
> now, and I was fairly sure we'd finally got it right - works for me, at
> least.
> The alternative is that there's some bug on the userspace side in your
> version of Ubuntu...
The thing is, that the same image, aside from the neccessary config changes
(sda1->xvda1, xvc0->hvc0), recognizes the right libraries. So I dont see an
obvious problem there.
> > I now discover, that after some playing around (during discovering what
> > the problem is) i must have changed something, because ldd still shows
> > the use of the i686/cmov libraries, but applications reliably causing
> > crashes earlier, dont do so anymore.
> As I said, you shouldn't be seeing crashes either way. If you are, it
> suggests a hypervisor bug. Does "xm dmesg" show anything?
(XEN) traps.c:1698:d16 Domain attempted WRMSR 0000008b from 00000056:00000000
to 00000000:00000000.
(XEN) traps.c:1698:d17 Domain attempted WRMSR 00000404 from 00000000:00000001
to ffffffff:ffffffff.
And some more of it. But why does a ldconfig, which doesnt change the
libraries loaded, change that?
For all debug data of the crashes see above.
> > PS: I did this conversion mainly, because I was getting quite random
> > crashes inside the domU. I guess, as this was a vendor kernel + xensource
> > xen kernel, this isnt appropriate for lkml? As well as the crashes of the
> > Dom0 (ubuntu again)?
> Yeah, anything which isn't 2.6.18-xen isn't officially "Xen project",
> but its worth reporting the bugs to xen-devel (and Ubuntu, of course).
> And to me, for that matter. I won't be able to help directly, but the
> crashes may highlight things we need to be careful about in the pvops
> Xen support.
Ok, will do so in the next days and will CC you at that.
Btw, is there a roadmap how in-kernel Xen is planned to develop?
Greetings,
Andres
> > /etc/ld.so.conf.d/xen.conf:
> > hwcap 0 nosegneg
>
> This looks OK. [...]
No, it does not match this:
/* Bit used for the pseudo-hwcap for non-negative segments. We use
bit 1 to avoid bugs in some versions of glibc when bit 0 is
used; the choice is otherwise arbitrary. */
#define VDSO_NOTE_NONEGSEG_BIT 1
You really want "hwcap 1 nosegneg" given that.
Yes, we've been over this several times already.
Thanks,
Roland
Hi,
On Wednesday 28 November 2007, Roland McGrath wrote in "Re: "hwcap 0 nosegneg"
doesnt work with paravirt_ops xen as of 2.6.23.9":
> > > /etc/ld.so.conf.d/xen.conf:
> > > hwcap 0 nosegneg
> >
> > This looks OK. [...]
>
> No, it does not match this:
> /* Bit used for the pseudo-hwcap for non-negative segments. We use
> bit 1 to avoid bugs in some versions of glibc when bit 0 is
> used; the choice is otherwise arbitrary. */
> #define VDSO_NOTE_NONEGSEG_BIT 1
> You really want "hwcap 1 nosegneg" given that.
> Yes, we've been over this several times already.
Thanks for the information. I guess it would be nice to document this
somewhere (except in code)... At least I could not find any information about
it beside some patch (Not directed at you, Roland). Any idea where?
Besides, it has one Problem: It is not backwards compatible. Using it on
earlier Xen Versions yields the normal tls versions (xen 3.1). Isnt the
userbase of existing xen installations bigger than the one of some point in
FC5 test releases?
Ugh. Leaning a bit far out of the door with that one considering my knowledge
about it... Sorry about that.
Greetings,
Andres