Greetings all:
Sometime between 2.6.28.6 and 2.6.31.5 a regression (feature?) in the futex
system now causes futex test failures on glibc-2.9 which where not present
before. That is, recompiling the binaries of glibc-2.9 and rerunning its
test suite now produces futex errors that passed previously. The problem
appears now with glibc-2.9 compiled with either gcc-4.1.2 or gcc-4.4.2, and
with glibc-2.11 compiled with gcc-4.4.2, which is what I am currently
running on this machine, failures and all.
The system under discussion is a uniprocessor pentium3 with an AMI BIOS.
Full details available on request should that prove necessary.
I have tracked the test failures down to the fact that futex_cmpxchg_enabled
is not set because the test in futex_init now "fails" (actually succeeds).
This appears to be happening because the expected page fault intentionally
provoked by a null dereference appears to be working now in kernel mode.
This *may* (rank speculation) be associated with the AMI BIOS low-memory
corruption protection added sometime during this gap, and which is activated
on this machine.
Before I muck any further with this, especially involving the quite tricky
futex mess, I would appreciate some insight into the idea behind the test in
futex_init. I don't understand why you would bother to invoke a fault in
what is apparently a test to determine if the cmpxchg instruction works.
The fault is supposed to occur as a result of a null dereference that takes
place *before* the cmpxchg instruction is even executed. If you want to
test that cmpxchg works, why not just make a little test in futex_init that
uses it and fails (not succeeds) if it doesn't behave as expected, or if
there is a fault of some kind (like illegal instruction)? Or is the fact
that we don't get a fault the whole point here?
Regards,
Joseph
Please CC me directly as I am no longer subscribed to the list.
Joseph Parmelee wrote:
Hi Joseph,
Adding futex folks on CC.
> Greetings all:
>
> Sometime between 2.6.28.6 and 2.6.31.5 a regression (feature?) in the futex
> system now causes futex test failures on glibc-2.9 which where not present
> before. That is, recompiling the binaries of glibc-2.9 and rerunning its
> test suite now produces futex errors that passed previously. The problem
> appears now with glibc-2.9 compiled with either gcc-4.1.2 or gcc-4.4.2, and
> with glibc-2.11 compiled with gcc-4.4.2, which is what I am currently
> running on this machine, failures and all.
>
> The system under discussion is a uniprocessor pentium3 with an AMI BIOS.
> Full details available on request should that prove necessary.
>
> I have tracked the test failures down to the fact that
> futex_cmpxchg_enabled
> is not set because the test in futex_init now "fails" (actually
> succeeds). This appears to be happening because the expected page fault
> intentionally
> provoked by a null dereference appears to be working now in kernel mode.
> This *may* (rank speculation) be associated with the AMI BIOS low-memory
> corruption protection added sometime during this gap, and which is
> activated
> on this machine.
Hrm... interesting...
>
> Before I muck any further with this, especially involving the quite tricky
> futex mess, I would appreciate some insight into the idea behind the
> test in
> futex_init. I don't understand why you would bother to invoke a fault in
> what is apparently a test to determine if the cmpxchg instruction works.
I suspect this is because the test asks for a userspace address. So rather
than hack something up to use a real userspace address, we just send NULL.
EFAULT=success and ENOSYS=failure.
> The fault is supposed to occur as a result of a null dereference that takes
> place *before* the cmpxchg instruction is even executed. If you want to
> test that cmpxchg works, why not just make a little test in futex_init that
> uses it and fails (not succeeds) if it doesn't behave as expected, or if
> there is a fault of some kind (like illegal instruction)? Or is the fact
> that we don't get a fault the whole point here?
I believe the NULL pointer is just a convenience.
--
Darren Hart
IBM Linux Technology Center
Real-Time Linux Team
On Mon, 30 Nov 2009, Darren Hart wrote:
> > The system under discussion is a uniprocessor pentium3 with an AMI BIOS.
> > Full details available on request should that prove necessary.
> >
> > I have tracked the test failures down to the fact that
> > futex_cmpxchg_enabled
> > is not set because the test in futex_init now "fails" (actually
> > succeeds). This appears to be happening because the expected page fault
> > intentionally
> > provoked by a null dereference appears to be working now in kernel mode.
Can you please printk the return value of that cmpxchg() test and
provide a full bootlog (dmesg) of your machine ?
Could you also please do a quick check in which kernel version this
got introduced ?
> > This *may* (rank speculation) be associated with the AMI BIOS low-memory
> > corruption protection added sometime during this gap, and which is
> > activated on this machine.
That'd be a serious bug as it would let every NULL pointer dereference
in the kernel proceed.
> > Before I muck any further with this, especially involving the quite tricky
> > futex mess, I would appreciate some insight into the idea behind the
> > test in
> > futex_init. I don't understand why you would bother to invoke a fault in
> > what is apparently a test to determine if the cmpxchg instruction works.
The point is that we have to deal with architectures where we do not
know at compile time whether we actually have a working cmpxchg. So we
use cmpxchg with a NULL pointer which is supposed to fault and return
-EFAULT and if cmpxchg is not working on the machine the arch code is
supposed to return -ENOSYS.
See arch/x86/include/asm/futex.h:futex_atomic_cmpxchg_inatomic() for
an example.
> I suspect this is because the test asks for a userspace address. So rather
> than hack something up to use a real userspace address, we just send NULL.
> EFAULT=success and ENOSYS=failure.
We have no real user space address at this point.
> > The fault is supposed to occur as a result of a null dereference that takes
> > place *before* the cmpxchg instruction is even executed. If you want to
No, the fault happens _when_ cmpxchg is executed on the NULL pointer.
> > test that cmpxchg works, why not just make a little test in futex_init that
> > uses it and fails (not succeeds) if it doesn't behave as expected, or if
> > there is a fault of some kind (like illegal instruction)? Or is the fact
> > that we don't get a fault the whole point here?
Again, we expect it to fault.
IIRC, we tried to use a valid address for the test and let the
exception fixup code return -EFAULT when the instruction is not
available. That worked on x86 but did not work on other archs.
Thanks,
tglx
On Mon, 30 Nov 2009, Thomas Gleixner wrote:
> Can you please printk the return value of that cmpxchg() test and
> provide a full bootlog (dmesg) of your machine ?
Thanks for the responses, which verify that my understanding of the expected
behavior was correct. That means that we have a real bug.
Attached is the complete output of dmesg from the most recent boot run.
This line due to added printk in futex_init:
[ 0.147626] futex_init curval = F0006AA0
These lines due to added printk's in
arch/x86/include/asm/futex.h:futex_atomic_cmpxchg_inatomic().
[ 0.147384] cmpxchg: ax before=cf80e000, ax after=f0006aa0
[ 0.147444] cmpxchg: bx before=0, bx after=0
[ 0.147536] cmpxchg: cx before=0, cx after=0
The compiler generates cmpxchg %ecx,(%ebx), so I added extended asm to
dump the registers involved just before and after the cmpxchg into variables
for printk.
All is consistent with the fact that the fault is not occurring and the
cmpxchg is working "as expected" at address 0. Examining /proc/kcore with
gdb shows that address c0000000 contains f0006aa0. Direct access with gdb
to address 0 fails as expected.
To completely eliminate any possibility that the fault was getting lost in
the fixup code somehow, I removed all the fixup code from the cmpxchg
extended asm, and the results are exactly the same. In fact this run is
with the fixup code removed. The fault is not occurring.
> That'd be a serious bug as it would let every NULL pointer dereference
> in the kernel proceed.
Interestingly, a printk inserted in futex_init that attempts a null
dereference results in an oops as expected.
>
> Could you also please do a quick check in which kernel version this
> got introduced ?
This was known to be working in 2.6.28.6. Unfortunately, I didn't find it
until I updated glibc and ran its test suite on 2.6.31.5. However, I have
been noticing some nasty log messages about page allocation failures in pppd
with plenty of available memory starting from sometime in the 2.6.31 series.
One of them is also attached FWIW. But these didn't seem to be causing any
problems other than making me nervous.
I am located in the mountains of Costa Rica with only a very slow dialup, so
git bisect is not an option for me. But I do have old copies of vmlinuz
lying around that go back to the 2.6.29 series. Unfortunately, I don't have
the matching unstripped vmlinux which would allow debugging, but I might be
able to test other ways. I will post again as soon as I have something.
In the meantime, if you can think of any tests that you want to run, I will
be most happy to help.
Best regards,
Joseph
On Mon, 30 Nov 2009, Thomas Gleixner wrote:
> Could you also please do a quick check in which kernel version this
> got introduced ?
>
I tested by reinstalling the old glibc-2.9, rebooting test kernels, and
rerunning offending mutex tests from the glibc-2.9 test suite. I needed to
reinstall the old glibc because the new glibc-2.11 wants kernels in the
2.6.31 series or later.
The problem first appears in 2.6.31 and persists through the current
2.6.31.6.
The problem does not appear in 2.6.29 and 2.6.30 versions.
Yours,
Joseph
Joseph,
On Tue, 1 Dec 2009, Joseph Parmelee wrote:
> On Mon, 30 Nov 2009, Thomas Gleixner wrote:
> > Could you also please do a quick check in which kernel version this
> > got introduced ?
>
> The problem first appears in 2.6.31 and persists through the current
> 2.6.31.6.
>
> The problem does not appear in 2.6.29 and 2.6.30 versions.
thanks for testing. Can you please provide your 2.6.31.x .config file ?
Thanks,
tglx
On Tue, 1 Dec 2009, Thomas Gleixner wrote:
> thanks for testing. Can you please provide your 2.6.31.x .config file ?
Attached is the current 2.6.31.6 .config.
Joseph
On 12/01/2009 01:52 PM, Joseph Parmelee wrote:
>
> On Tue, 1 Dec 2009, Thomas Gleixner wrote:
>
>> thanks for testing. Can you please provide your 2.6.31.x .config file ?
>
> Attached is the current 2.6.31.6 .config.
>
> Joseph
Note: the best thing you could do -- although it is a bit time consuming
-- would be to do a "git bisect" between the last known working version
(2.6.28) and first broken version (2.6.31).
-hpa
On Tue, 1 Dec 2009, H. Peter Anvin wrote:
> Note: the best thing you could do -- although it is a bit time consuming
> -- would be to do a "git bisect" between the last known working version
> (2.6.28) and first broken version (2.6.31).
>
Just to clarify, the last known working version is 2.6.30.6; the first known
bad version is 2.6.31.
Joseph
On 12/01/2009 04:53 PM, Joseph Parmelee wrote:
>
>
>
> On Tue, 1 Dec 2009, H. Peter Anvin wrote:
>
>> Note: the best thing you could do -- although it is a bit time consuming
>> -- would be to do a "git bisect" between the last known working version
>> (2.6.28) and first broken version (2.6.31).
>
> Just to clarify, the last known working version is 2.6.30.6; the first known
> bad version is 2.6.31.
>
OK, you'd be bisecting 2.6.30..2.6.31 then.
-hpa
Hello:
Just installed 2.6.32 and the problem is still there. The value returned
for curval in futex_init is still the same 0xf0006aa0, and thus as before
futex_cmpxchg_enabled is not set, thus causing many kernel futex syscalls to
return -ENOSYS incorrectly, the proximate cause of the glibc test failures.
Regards,
Joseph
Hello again:
Did some more testing and developed a bit more data, possibly relevant:
1. Using gdb on /proc/kcore, 2.6.32 no longer allows access to 0xc0000000,
which could be accessed in 2.6.31.x, presumably showing the 16-bit real-mode
interrupt address table set up by the BIOS at physical address zero. But
.... (see 3).
2. Disabling the AMI BIOS corruption prevention in 2.6.32 had no effect on
the behavior described in 3 below.
3. In 2.6.31.x the following inserted in futex_atomic_cmpxchg_inatomic() in
futex.h caused an oops. It now works in 2.6.32.
printk(KERN_INFO "*uaddr=%x, oldval=%x, newval=%x",
*uaddr, oldval, newval);
Producing in dmesg:
[ 0.149408] *uaddr=f0006aa0, oldval=0, newval=0
The following also works in futex_init in futex.c:
unsigned int testval, *nullptr=NULL;
testval=*(unsigned int *)NULL;
printk(KERN_INFO "curval=%x testval=%x", curval, testval);
printk(KERN_INFO "An explicit null dereference=%x", *(unsigned int *)NULL);
printk(KERN_INFO "An implicit null dereference=%x", *nullptr);
Producing in dmesg:
[ 0.149466] curval=f0006aa0 testval=f0006aa0
[ 0.149557] An explicit null dereference=f0006aa0
[ 0.149648] An implicit null dereference=f0006aa0
Thus it is completely clear that the virtual zero page is mapped, probably
to physical zero, during this phase of kernel initialization.
I have used 2.6.32 for only one day, but so far see no ill effects from this
other than the lack of those futex functions which rely on a null pointer
dereference fault to set futex_cmpxchg_enabled. If zero-page mapping is now
"normal", the test in futex.c for cmpxchg functionality must be changed.
I have "fixed" this on our intel machines by simply commenting out the test
of curval in futex_init, thus causing futex_cmpxchg_enabled to be
unconditionally set. Not suitable for prime-time obviously, but "it works
for us" until somebody with more savvy comes up with a real fix.
Yours,
Joseph