I had some boot failures here with git-x86 with init and hotplug all
segfaulting early on userland with new glibc. Bisecting found
commit 6aea5bc37fa790eaf3a942f0785985914568e214
Author: Ingo Molnar <[email protected]>
Date: Sat Jan 5 13:27:08 2008 +0100
x86: move native_read_tsc() offline
move native_read_tsc() offline.
I think the problem is that the vsyscall/vdso code calls it through
vread and for that it has to be exported. There seems to be also
another bug with the old style vsyscalls not using the TSC vread
that masks it on older glibc
Stepping with gdb through old style vgettimeofday() confirms that RDTSC is
not used.
A long time ago we had a similar problem once and it was because of a
problem exporting the vsyscall variables in vmlinux.lds.S -- looks like that
has reappeared.
I think the new glibc shows it because it uses the vDSO not
the older vsyscall and the new vDSO probably still works. Anyways haven't
investigated why that is in detail yet, but that's a separate
regression.
Back to the boot failure:
Unfortunately simply adding __vsyscall_fn to native_read_tsc doesn't
work -- causes early kernel faults like
PANIC: early exception rip ffffffffff600105 error 10 cr2 ffffffffff600105
Pid: 0, comm: swapper Not tainted 2.6.24-rc6 #58
Call Trace:
[<ffffffff80211dec>] native_sched_clock+0x9/0x3f
[<ffffffff8022b758>] init_idle+0x33/0xd1
[<ffffffff80825449>] sched_init+0x26d/0x283
[<ffffffff80815899>] start_kernel+0x10b/0x2bd
[<ffffffff80815114>] _sinittext+0x114/0x11b
Not sure why that is -- in theory the vsyscall functions should be callable
from the main kernel. Might be a binutils problem or another code
regression.
Anyways it looks like the only good fix is to either revert that or
fork into two functions one for vread() and another for normal tsc ->read()
This is all in addition to the problem of it having incorrect barriers.
I note that my original patch didn't have any of these problems.
I'm using the appended revert patch here as a workaround for now.
-Andi
Revert rdtsc out of line change
Reverts
commit 6aea5bc37fa790eaf3a942f0785985914568e214
Author: Ingo Molnar <[email protected]>
Date: Sat Jan 5 13:27:08 2008 +0100
x86: move native_read_tsc() offline
move native_read_tsc() offline.
The function is called by vsyscalls in ring 3, so it can't be out of line this way.
Signed-off-by: Andi Kleen <[email protected]>
Index: linux/arch/x86/kernel/rtc.c
===================================================================
--- linux.orig/arch/x86/kernel/rtc.c
+++ linux/arch/x86/kernel/rtc.c
@@ -194,14 +194,3 @@ int update_persistent_clock(struct times
{
return set_rtc_mmss(now.tv_sec);
}
-
-unsigned long long native_read_tsc(void)
-{
- DECLARE_ARGS(val, low, high);
-
- asm volatile("rdtsc" : EAX_EDX_RET(val, low, high));
- rdtsc_barrier();
-
- return EAX_EDX_VAL(val, low, high);
-}
-
Index: linux/include/asm-x86/msr.h
===================================================================
--- linux.orig/include/asm-x86/msr.h
+++ linux/include/asm-x86/msr.h
@@ -91,7 +91,13 @@ static inline int native_write_msr_safe(
return err;
}
-extern unsigned long long native_read_tsc(void);
+static inline unsigned long long native_read_tsc(void)
+{
+ DECLARE_ARGS(val, low, high);
+
+ asm volatile("rdtsc" : EAX_EDX_RET(val, low, high));
+ return EAX_EDX_VAL(val, low, high);
+}
static inline unsigned long long native_read_pmc(int counter)
{
> I think the problem is that the vsyscall/vdso code calls it through
> vread and for that it has to be exported. There seems to be also
> another bug with the old style vsyscalls not using the TSC vread
> that masks it on older glibc
>
> Stepping with gdb through old style vgettimeofday() confirms that RDTSC is
> not used.
>
> A long time ago we had a similar problem once and it was because of a
> problem exporting the vsyscall variables in vmlinux.lds.S -- looks like that
> has reappeared.
>
> I think the new glibc shows it because it uses the vDSO not
> the older vsyscall and the new vDSO probably still works. Anyways haven't
> investigated why that is in detail yet, but that's a separate
> regression.
Actually that seems to be because the test system using the older
glibc didn't use the TSC because it was marked unstable due a
unsynchronized TSC. It should not have been -- this is a Core2
dual core single socket. Will investigate later what happened
there.
-Andi
* Andi Kleen <[email protected]> wrote:
> Unfortunately simply adding __vsyscall_fn to native_read_tsc doesn't
> work -- causes early kernel faults like
>
> PANIC: early exception rip ffffffffff600105 error 10 cr2 ffffffffff600105
> Pid: 0, comm: swapper Not tainted 2.6.24-rc6 #58
>
> Call Trace:
> [<ffffffff80211dec>] native_sched_clock+0x9/0x3f
> [<ffffffff8022b758>] init_idle+0x33/0xd1
> [<ffffffff80825449>] sched_init+0x26d/0x283
> [<ffffffff80815899>] start_kernel+0x10b/0x2bd
> [<ffffffff80815114>] _sinittext+0x114/0x11b
>
> Not sure why that is -- in theory the vsyscall functions should be
> callable from the main kernel. Might be a binutils problem or another
> code regression.
nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings
are installed via an __initcall, and that's too late for early use. The
combo patch below fixed the crash for me, does it work on your box too?
Ingo
Index: linux-x86.q/arch/x86/kernel/rtc.c
===================================================================
--- linux-x86.q.orig/arch/x86/kernel/rtc.c
+++ linux-x86.q/arch/x86/kernel/rtc.c
@@ -195,7 +195,7 @@ int update_persistent_clock(struct times
return set_rtc_mmss(now.tv_sec);
}
-unsigned long long native_read_tsc(void)
+unsigned long long __vsyscall_fn native_read_tsc(void)
{
DECLARE_ARGS(val, low, high);
Index: linux-x86.q/arch/x86/kernel/setup_64.c
===================================================================
--- linux-x86.q.orig/arch/x86/kernel/setup_64.c
+++ linux-x86.q/arch/x86/kernel/setup_64.c
@@ -46,6 +46,7 @@
#include <asm/mtrr.h>
#include <asm/uaccess.h>
#include <asm/system.h>
+#include <asm/vsyscall.h>
#include <asm/io.h>
#include <asm/smp.h>
#include <asm/msr.h>
@@ -464,6 +465,7 @@ void __init setup_arch(char **cmdline_p)
#endif
reserve_crashkernel();
paging_init();
+ map_vsyscall();
early_quirks();
Index: linux-x86.q/arch/x86/kernel/vsyscall_64.c
===================================================================
--- linux-x86.q.orig/arch/x86/kernel/vsyscall_64.c
+++ linux-x86.q/arch/x86/kernel/vsyscall_64.c
@@ -319,7 +319,7 @@ cpu_vsyscall_notifier(struct notifier_bl
return NOTIFY_DONE;
}
-static void __init map_vsyscall(void)
+void __init map_vsyscall(void)
{
extern char __vsyscall_0;
unsigned long physaddr_page0 = __pa_symbol(&__vsyscall_0);
@@ -335,7 +335,6 @@ static int __init vsyscall_init(void)
BUG_ON((unsigned long) &vtime != VSYSCALL_ADDR(__NR_vtime));
BUG_ON((VSYSCALL_ADDR(0) != __fix_to_virt(VSYSCALL_FIRST_PAGE)));
BUG_ON((unsigned long) &vgetcpu != VSYSCALL_ADDR(__NR_vgetcpu));
- map_vsyscall();
#ifdef CONFIG_SYSCTL
register_sysctl_table(kernel_root_table2);
#endif
> nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings
> are installed via an __initcall, and that's too late for early use. The
> combo patch below fixed the crash for me, does it work on your box too?
That gives
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati
on of function 'map_vsyscall'
But after I add a prototype to vsyscall.h it seems to work.
Just the barriers are still broken -- you have not replied to any
of my emails on that topic. What's up with that?
-Andi
* Andi Kleen <[email protected]> wrote:
> > nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings
> > are installed via an __initcall, and that's too late for early use. The
> > combo patch below fixed the crash for me, does it work on your box too?
>
> That gives
>
> /home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':
> /home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati
> on of function 'map_vsyscall'
i guess you have an old repository.
> But after I add a prototype to vsyscall.h it seems to work.
>
> Just the barriers are still broken [...]
since yesterday there's a full barrier around rdtsc.
Ingo
On Wed, Jan 09, 2008 at 04:22:08PM +0100, Ingo Molnar wrote:
>
> * Andi Kleen <[email protected]> wrote:
>
> > > nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings
> > > are installed via an __initcall, and that's too late for early use. The
> > > combo patch below fixed the crash for me, does it work on your box too?
> >
> > That gives
> >
> > /home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':
> > /home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati
> > on of function 'map_vsyscall'
>
> i guess you have an old repository.
I just applied the patch you sent out against yesterday's git-x86.
>
> > But after I add a prototype to vsyscall.h it seems to work.
> >
> > Just the barriers are still broken [...]
>
> since yesterday there's a full barrier around rdtsc.
Great.
-Andi
* Andi Kleen <[email protected]> wrote:
> On Wed, Jan 09, 2008 at 04:22:08PM +0100, Ingo Molnar wrote:
> >
> > * Andi Kleen <[email protected]> wrote:
> >
> > > > nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings
> > > > are installed via an __initcall, and that's too late for early use. The
> > > > combo patch below fixed the crash for me, does it work on your box too?
> > >
> > > That gives
> > >
> > > /home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':
> > > /home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati
> > > on of function 'map_vsyscall'
> >
> > i guess you have an old repository.
>
> I just applied the patch you sent out against yesterday's git-x86.
then you have a truly ancient x86.git repository ;-)
think of x86.git#mm as an open development tree. It's high-flux, based
against Linux-bleeding-edge, it's frequently updated (daily, sometimes
hourly), breakages are possible (and likely) and fixes and other
feedback is more than welcome. And please feel free to complain about
patches that are included. (like you did in the past) Also please try to
post your patches as early as possible instead of in big chunks - last
week's 75 patches patchbomb from you was (and still is) ... challenging
;-)
> > since yesterday there's a full barrier around rdtsc.
>
> Great.
i have measured the impact of the barriers and it was in the noise
level. Barriers are notoriously easy to get wrong (because almost
nothing tells the programmer that they are wrong), that's why i did this
barrier-safe rdtsc() [& friends]. We had so much trouble with RDTSC
during the past 10 years of its existence that being a bit more
conservative with it is the only really maintainable option.
Ingo
On Wed, Jan 09, 2008 at 05:30:18PM +0100, Ingo Molnar wrote:
>
> * Andi Kleen <[email protected]> wrote:
>
> > On Wed, Jan 09, 2008 at 04:22:08PM +0100, Ingo Molnar wrote:
> > >
> > > * Andi Kleen <[email protected]> wrote:
> > >
> > > > > nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings
> > > > > are installed via an __initcall, and that's too late for early use. The
> > > > > combo patch below fixed the crash for me, does it work on your box too?
> > > >
> > > > That gives
> > > >
> > > > /home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':
> > > > /home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati
> > > > on of function 'map_vsyscall'
> > >
> > > i guess you have an old repository.
> >
> > I just applied the patch you sent out against yesterday's git-x86.
>
> then you have a truly ancient x86.git repository ;-)
I update only infrequently because frankly git's remote branch tracking
is a mess. At least it doesn't really work for x86#mm here.
I usually have to blow away the repository and reclone
to get back to a sane state.
>
> think of x86.git#mm as an open development tree. It's high-flux, based
> against Linux-bleeding-edge, it's frequently updated (daily, sometimes
> hourly), breakages are possible (and likely) and fixes and other
> feedback is more than welcome. And please feel free to complain about
Right now Jan's cpa change you merged makes the cpa selftest to not pass
anymore on 32bit. While that was likely a latent bug it just
triggered it's a little inconvenient.
There's also the problem on NX handling still not being right.
> patches that are included. (like you did in the past) Also please try to
> post your patches as early as possible instead of in big chunks - last
> week's 75 patches patchbomb from you was (and still is) ... challenging
There are still quite a lot outstanding.
% wc -l patches/series
90 patches/series
Some of that is debug or tbd or obsolete, but most isn't.
> > > since yesterday there's a full barrier around rdtsc.
> >
> > Great.
>
> i have measured the impact of the barriers and it was in the noise
> level. Barriers are notoriously easy to get wrong (because almost
> nothing tells the programmer that they are wrong), that's why i did this
> barrier-safe rdtsc() [& friends]. We had so much trouble with RDTSC
> during the past 10 years of its existence that being a bit more
> conservative with it is the only really maintainable option.
I would still think that the explicit barriers would make this
all clearer, with long term giving cleaner semantics.
Admittedly I should have written some more comments in the
code.
I'm also a little unhappy on how many function calls the vgtod()
fast path contains now. Long ago it was a really lean function,
but it gets messed up more and more. Back when ->vread was introduced
the latency already suffered significantly (iirc multi digit
percent range), i suspect it now got worse again.
And after all that's still by far the most common system call
(not only for databases; i profiled this using systemtap in some
loads some time ago and it usually came up with >50%)
and quite important for many workloads.
We have a lot far uglier code for less gain in far less critical calls.
-Andi
On Wed, 9 Jan 2008 18:48:00 +0100
> And after all that's still by far the most common system call
> (not only for databases; i profiled this using systemtap in some
> loads some time ago and it usually came up with >50%)
> and quite important for many workloads.
>
btw be careful with this; the X server uses gettimeofday in it's equivalent of udelay()...
(and we all know how useful it is to make the delay loops faster ;-)
On Wed, Jan 09, 2008 at 12:25:59PM -0800, Arjan van de Ven wrote:
> On Wed, 9 Jan 2008 18:48:00 +0100
> > And after all that's still by far the most common system call
> > (not only for databases; i profiled this using systemtap in some
> > loads some time ago and it usually came up with >50%)
> > and quite important for many workloads.
> >
>
> btw be careful with this; the X server uses gettimeofday in it's equivalent of udelay()...
> (and we all know how useful it is to make the delay loops faster ;-)
People tend to make jokes about optimizing the idle loop too, but they're
actually wrong. Exit latency for the idle loop is important -- it
decides how quickly you can react to load changes on idle CPUs.
For short udelays I suspect shorter exit latency is also moderately useful.
But anyways there are plenty of gtod users outside the X server.
e.g. common not user space case is packet timestamps.
-Andi
Andi Kleen wrote:
>
> People tend to make jokes about optimizing the idle loop too, but they're
> actually wrong. Exit latency for the idle loop is important -- it
> decides how quickly you can react to load changes on idle CPUs.
>
> For short udelays I suspect shorter exit latency is also moderately useful.
>
> But anyways there are plenty of gtod users outside the X server.
>
> e.g. common not user space case is packet timestamps.
>
It's also very common to put gtod calls around select and poll, because
the standard interface doesn't tell you how long you have been away
(Linux at least used to have it, but glibc "fixed" that...)
-hpa
On 2008.01.09 18:48:00 +0100, Andi Kleen wrote:
> On Wed, Jan 09, 2008 at 05:30:18PM +0100, Ingo Molnar wrote:
> > then you have a truly ancient x86.git repository ;-)
>
> I update only infrequently because frankly git's remote branch tracking
> is a mess. At least it doesn't really work for x86#mm here.
>
> I usually have to blow away the repository and reclone
> to get back to a sane state.
Someone in #git had a similar problem today. Conclusion was that x86/mm
is not "stable" in the sense that commits are only added, instead
history gets rewritten. That breaks pull/merge/"basic rebase".
Basically, you'll want to "rebase --onto", taking your local commits
from the old branch history to the new branch history. One way to make
that bearable is to have two branches (or a branch and a tag). One
branch is used to keep your work, the other branch (or tag) is used to
"mark" where the old upstream ended and your work started.
Assuming that your remote is called "x86", this could look like this:
git branch myStuff x86/mm
git branch myStuff_start x86/mm
work work work commit commit commit
git fetch x86 mm
git rebase --onto x86/mm myStuff_start myStuff
git branch -f myStuff_start x86/mm
The rebase will take all of the commits that you added to myStuff, on
top of the old x86/mm (referred to by myStuff_start) and try to apply
them on top of the new x86/mm, updating myStuff to point to the rebased
commits afterwards.
Then myStuff_start is updated to point to the now current start of your
stuff, so that you can do the rebase again later.
That's not actually a problem of git's tracking branches, because
they're not made for that style of work at all. For such stuff, rebase
is the only useful option AFAIK.
HTH,
Bj?rn
On Wed, Jan 09, 2008 at 11:09:48PM +0100, Bj?rn Steinbrink wrote:
> On 2008.01.09 18:48:00 +0100, Andi Kleen wrote:
> > On Wed, Jan 09, 2008 at 05:30:18PM +0100, Ingo Molnar wrote:
> > > then you have a truly ancient x86.git repository ;-)
> >
> > I update only infrequently because frankly git's remote branch tracking
> > is a mess. At least it doesn't really work for x86#mm here.
> >
> > I usually have to blow away the repository and reclone
> > to get back to a sane state.
>
> Someone in #git had a similar problem today. Conclusion was that x86/mm
> is not "stable" in the sense that commits are only added, instead
> history gets rewritten. That breaks pull/merge/"basic rebase".
>
> Basically, you'll want to "rebase --onto", taking your local commits
I don't really have any local commits. I just want to use read-only
git to generate a patch that I can then import using quilt and
only look at log files and use git show. The fanciest thing
I use is git bisect occasionally.
> that bearable is to have two branches (or a branch and a tag). One
> branch is used to keep your work, the other branch (or tag) is used to
> "mark" where the old upstream ended and your work started.
>
> Assuming that your remote is called "x86", this could look like this:
>
> git branch myStuff x86/mm
> git branch myStuff_start x86/mm
>
> work work work commit commit commit
>
> git fetch x86 mm
>
> git rebase --onto x86/mm myStuff_start myStuff
> git branch -f myStuff_start x86/mm
Do you have a simple recipe to just update from the the remote branch,
assuming there are no local changes or local branches?
-Andi
On Wed, 2008-01-09 at 23:28 +0100, Andi Kleen wrote:
> Do you have a simple recipe to just update from the the remote branch,
> assuming there are no local changes or local branches?
>
> -Andi
For staying up to date I use the following:
# Add Linus's tree as a remote
git remote --add linus
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
# Add Ingo's tree as a remote
git remote --add x86
git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git
# With that setup, just run the following to get any changes you
# don't have. It will also notice any new branches Ingo/Linus
# add to their repo. Look in .git/config afterwards, the format
# to add new remotes is easy to figure out.
git remote update
Cheers,
Harvey
On Wed, Jan 09, 2008 at 02:35:55PM -0800, Harvey Harrison wrote:
> On Wed, 2008-01-09 at 23:28 +0100, Andi Kleen wrote:
>
> > Do you have a simple recipe to just update from the the remote branch,
> > assuming there are no local changes or local branches?
> >
> > -Andi
>
> For staying up to date I use the following:
>
> # Add Linus's tree as a remote
> git remote --add linus
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
>
> # Add Ingo's tree as a remote
> git remote --add x86
> git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git
>
> # With that setup, just run the following to get any changes you
> # don't have. It will also notice any new branches Ingo/Linus
> # add to their repo. Look in .git/config afterwards, the format
> # to add new remotes is easy to figure out.
> git remote update
I'm already cloning the branches; the problem is not getting conflicts etc.
when updating.
-Andi
On 2008.01.09 23:41:42 +0100, Andi Kleen wrote:
> On Wed, Jan 09, 2008 at 02:35:55PM -0800, Harvey Harrison wrote:
> > On Wed, 2008-01-09 at 23:28 +0100, Andi Kleen wrote:
> >
> > > Do you have a simple recipe to just update from the the remote branch,
> > > assuming there are no local changes or local branches?
> > >
> > > -Andi
> >
> > For staying up to date I use the following:
> >
> > # Add Linus's tree as a remote
> > git remote --add linus
> > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> >
> > # Add Ingo's tree as a remote
> > git remote --add x86
> > git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git
> >
> > # With that setup, just run the following to get any changes you
> > # don't have. It will also notice any new branches Ingo/Linus
> > # add to their repo. Look in .git/config afterwards, the format
> > # to add new remotes is easy to figure out.
> > git remote update
>
> I'm already cloning the branches; the problem is not getting conflicts etc.
> when updating.
I guess you're using "git pull" to update a local branch? That will try
to merge the new state of x86/mm into your local branch, and that
breaks. If you just want to have a local branch that gives you a "fixed"
state of x86/mm regardless of whether or not you already fetched newer
ones, you can do:
git branch myThing x86/mm # create the branch
work/test/whatever
To fetch a new "state" from the remote:
git fetch x86/mm # or git remote update, or whatever
To update your branch to point to the new state:
git branch - f myThing x86/mm
That basically replaces it with a new branch of the same name, but
pointing to the new x86/mm.
Or if you want to get your working tree to that state at the same time,
you can also do:
git checkout myThing
git reset --hard x86/mm
Bj?rn
On 1/10/08, Andi Kleen <[email protected]> wrote:
> I'm already cloning the branches; the problem is not getting conflicts etc.
> when updating.
Isn't git pull --force what you are looking for?
Ciao,
--
Paolo
http://paolo.ciarrocchi.googlepages.com/
On Jan 10, 2008 2:37 AM, Paolo Ciarrocchi <[email protected]> wrote:
> On 1/10/08, Andi Kleen <[email protected]> wrote:
>
> > I'm already cloning the branches; the problem is not getting conflicts etc.
> > when updating.
>
> Isn't git pull --force what you are looking for?
>
> Ciao,
> --
> Paolo
> http://paolo.ciarrocchi.googlepages.com/
>
Actually...no, it wouldn't help. Conflicts will coming up and only
the solution for now is to use remote tracking as Harvey wrote
in prev mail. (btw, git pull that would not conflict is when it
called with '-s ours' but that is not good)
- Cyrill -