2010-08-21 03:44:41

by Mark Stanovich

[permalink] [raw]
Subject: Linux 2.6.35.3

After updating to the 2.6.35.3 stable release, my machine was
rebooting automatically during the first few seconds of boot.

I bisected to the first bad commit of

commit 568132624386f53e87575195d868db
9afb2e9316
Author: H. Peter Anvin <[email protected]>
Date:?? Tue Jul 27 17:01:49 2010 -0700

??? x86: Add memory modify constraints to xchg() and cmpxchg()

??? commit 113fc5a6e8c2288619ff7e8187a6f556b7e0d372 upstream.

??? xchg() and cmpxchg() modify their memory operands, not merely read
??? them.? For some versions of gcc the "memory" clobber has apparently
??? dealt with the situation, but not for all.

I noticed that the latest version pulled from Linus's git tree did not
have the problem.? After looking at the changes to
arch/x86/include/asm/cmpxchg_32.h, I cherry-picked two patches from
Linus's tree that seemed to make a difference, namely:

69309a05907546fb686b251d4ab041c26afe1e1d
4532b305e8f0c238dd73048068ff8a6dd1380291

Applying these patches solved the rebooting problem on 2.6.35.3

I believe the same problem also happens for the latest stable version
of 2.6.34, but did not verify the fix worked for the 2.6.34 stable
releases.

I don't know if this is helpful, but I thought I would let you know in
case someone else is having the same problem.

If you need any further information, just let me know.

??? - Mark


2010-08-21 03:45:05

by H. Peter Anvin

[permalink] [raw]
Subject: Re: Linux 2.6.35.3

On 08/20/2010 08:40 PM, Mark Stanovich wrote:
> After updating to the 2.6.35.3 stable release, my machine was rebooting
> automatically during the first few seconds of boot.
>
> I bisected to the first bad commit of
>
> commit 568132624386f53e87575195d868db
> 9afb2e9316
> Author: H. Peter Anvin <[email protected] <mailto:[email protected]>>
> Date: Tue Jul 27 17:01:49 2010 -0700
>
> x86: Add memory modify constraints to xchg() and cmpxchg()
>
> commit 113fc5a6e8c2288619ff7e8187a6f556b7e0d372 upstream.
>
> xchg() and cmpxchg() modify their memory operands, not merely read
> them. For some versions of gcc the "memory" clobber has apparently
> dealt with the situation, but not for all.
>
> I noticed that the latest version pulled from Linus's git tree did not
> have the problem. After looking at the changes to
> arch/x86/include/asm/cmpxchg_32.h, I cherry-picked two patches from
> Linus's tree that seemed to make a difference, namely:
>
> 69309a05907546fb686b251d4ab041c26afe1e1d
> 4532b305e8f0c238dd73048068ff8a6dd1380291
>
> Applying these patches solved the rebooting problem on 2.6.35.3
>
> I believe the same problem also happens for the latest stable version of
> 2.6.34, but did not verify the fix worked for the 2.6.34 stable releases.
>
> I don't know if this is helpful, but I thought I would let you know in
> case someone else is having the same problem.
>
> If you need any further information, just let me know.
>

Yes, as a matter of fact, we have been trying to root-cause this
problem; please see:

https://bugzilla.kernel.org/show_bug.cgi?id=16612

If you could try the one-liner in there and see if it solves your
problem, it would be great.

-hpa

P.S. What version of gcc do you use?

--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.

2010-08-21 04:06:35

by Mark Stanovich

[permalink] [raw]
Subject: Re: Linux 2.6.35.3

On Fri, Aug 20, 2010 at 11:44 PM, H. Peter Anvin <[email protected]> wrote:
> On 08/20/2010 08:40 PM, Mark Stanovich wrote:
>> After updating to the 2.6.35.3 stable release, my machine was rebooting
>> automatically during the first few seconds of boot.
>>
>> I bisected to the first bad commit of
>>
>> commit 568132624386f53e87575195d868db
>> 9afb2e9316
>> Author: H. Peter Anvin <[email protected] <mailto:[email protected]>>
>> Date: ? Tue Jul 27 17:01:49 2010 -0700
>>
>> ? ? x86: Add memory modify constraints to xchg() and cmpxchg()
>>
>> ? ? commit 113fc5a6e8c2288619ff7e8187a6f556b7e0d372 upstream.
>>
>> ? ? xchg() and cmpxchg() modify their memory operands, not merely read
>> ? ? them. ?For some versions of gcc the "memory" clobber has apparently
>> ? ? dealt with the situation, but not for all.
>>
>> I noticed that the latest version pulled from Linus's git tree did not
>> have the problem. ?After looking at the changes to
>> arch/x86/include/asm/cmpxchg_32.h, I cherry-picked two patches from
>> Linus's tree that seemed to make a difference, namely:
>>
>> 69309a05907546fb686b251d4ab041c26afe1e1d
>> 4532b305e8f0c238dd73048068ff8a6dd1380291
>>
>> Applying these patches solved the rebooting problem on 2.6.35.3
>>
>> I believe the same problem also happens for the latest stable version of
>> 2.6.34, but did not verify the fix worked for the 2.6.34 stable releases.
>>
>> I don't know if this is helpful, but I thought I would let you know in
>> case someone else is having the same problem.
>>
>> If you need any further information, just let me know.
>>
>
> Yes, as a matter of fact, we have been trying to root-cause this
> problem; please see:
>
> https://bugzilla.kernel.org/show_bug.cgi?id=16612
>
> If you could try the one-liner in there and see if it solves your
> problem, it would be great.

The patch for __set_64bit on top of 2.6.35.3 solves the rebooting problem!

>
> ? ? ? ?-hpa
>
> P.S. What version of gcc do you use?

gcc (Gentoo 4.4.3-r2 p1.2) 4.4.3

Thanks,
Mark

2010-08-21 04:41:12

by H. Peter Anvin

[permalink] [raw]
Subject: Re: Linux 2.6.35.3

OK, I think we can consider this root-caused (it's unclear if it's our
bug or gcc's, but adding "volatile" to the pointer solves the problem.)

Greg, do you want to take the listed upstream patches (thus bringing the
code closer to mainline) or do you want to take the one-liner patch from
the bugzilla? [Attached for reference?]

-hpa



On 08/20/2010 09:06 PM, Mark Stanovich wrote:
> On Fri, Aug 20, 2010 at 11:44 PM, H. Peter Anvin <[email protected]> wrote:
>> On 08/20/2010 08:40 PM, Mark Stanovich wrote:
>>> After updating to the 2.6.35.3 stable release, my machine was rebooting
>>> automatically during the first few seconds of boot.
>>>
>>> I bisected to the first bad commit of
>>>
>>> commit 568132624386f53e87575195d868db
>>> 9afb2e9316
>>> Author: H. Peter Anvin <[email protected] <mailto:[email protected]>>
>>> Date: Tue Jul 27 17:01:49 2010 -0700
>>>
>>> x86: Add memory modify constraints to xchg() and cmpxchg()
>>>
>>> commit 113fc5a6e8c2288619ff7e8187a6f556b7e0d372 upstream.
>>>
>>> xchg() and cmpxchg() modify their memory operands, not merely read
>>> them. For some versions of gcc the "memory" clobber has apparently
>>> dealt with the situation, but not for all.
>>>
>>> I noticed that the latest version pulled from Linus's git tree did not
>>> have the problem. After looking at the changes to
>>> arch/x86/include/asm/cmpxchg_32.h, I cherry-picked two patches from
>>> Linus's tree that seemed to make a difference, namely:
>>>
>>> 69309a05907546fb686b251d4ab041c26afe1e1d
>>> 4532b305e8f0c238dd73048068ff8a6dd1380291
>>>
>>> Applying these patches solved the rebooting problem on 2.6.35.3
>>>
>>> I believe the same problem also happens for the latest stable version of
>>> 2.6.34, but did not verify the fix worked for the 2.6.34 stable releases.
>>>
>>> I don't know if this is helpful, but I thought I would let you know in
>>> case someone else is having the same problem.
>>>
>>> If you need any further information, just let me know.
>>>
>>
>> Yes, as a matter of fact, we have been trying to root-cause this
>> problem; please see:
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=16612
>>
>> If you could try the one-liner in there and see if it solves your
>> problem, it would be great.
>
> The patch for __set_64bit on top of 2.6.35.3 solves the rebooting problem!
>
>>
>> -hpa
>>
>> P.S. What version of gcc do you use?
>
> gcc (Gentoo 4.4.3-r2 p1.2) 4.4.3
>
> Thanks,
> Mark
>


--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.


Attachments:
volatilize.patch (548.00 B)

2010-08-21 04:50:15

by Greg KH

[permalink] [raw]
Subject: Re: Linux 2.6.35.3

On Fri, Aug 20, 2010 at 09:41:02PM -0700, H. Peter Anvin wrote:
> OK, I think we can consider this root-caused (it's unclear if it's our
> bug or gcc's, but adding "volatile" to the pointer solves the problem.)
>
> Greg, do you want to take the listed upstream patches (thus bringing the
> code closer to mainline) or do you want to take the one-liner patch from
> the bugzilla? [Attached for reference?]

Let me look at the 2 upstream patches tomorrow, and I'd prefer to take
them instead of something else, just to keep everything in sync.

thanks,

greg k-h

2010-08-23 01:31:49

by H. Peter Anvin

[permalink] [raw]
Subject: Re: Linux 2.6.35.3

On 08/20/2010 09:06 PM, Mark Stanovich wrote:
>>>
>>> I noticed that the latest version pulled from Linus's git tree did not
>>> have the problem. After looking at the changes to
>>> arch/x86/include/asm/cmpxchg_32.h, I cherry-picked two patches from
>>> Linus's tree that seemed to make a difference, namely:
>>>
>>> 69309a05907546fb686b251d4ab041c26afe1e1d
>>> 4532b305e8f0c238dd73048068ff8a6dd1380291
>>>
>>> Applying these patches solved the rebooting problem on 2.6.35.3
>>>
>>> I believe the same problem also happens for the latest stable version of
>>> 2.6.34, but did not verify the fix worked for the 2.6.34 stable releases.
>>>
>>> I don't know if this is helpful, but I thought I would let you know in
>>> case someone else is having the same problem.
>>>
>>> If you need any further information, just let me know.
>>>
>>
>> Yes, as a matter of fact, we have been trying to root-cause this
>> problem; please see:
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=16612
>>
>> If you could try the one-liner in there and see if it solves your
>> problem, it would be great.
>
> The patch for __set_64bit on top of 2.6.35.3 solves the rebooting problem!
>
>

By the way, I think only 69309a05907546fb686b251d4ab041c26afe1e1d should
solve the problem. Mark, could you try only this patch?

-hpa

--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.

2010-08-23 01:56:22

by Mark Stanovich

[permalink] [raw]
Subject: Re: Linux 2.6.35.3

On Sun, Aug 22, 2010 at 9:31 PM, H. Peter Anvin <[email protected]> wrote:
> On 08/20/2010 09:06 PM, Mark Stanovich wrote:
>>>>
>>>> I noticed that the latest version pulled from Linus's git tree did not
>>>> have the problem. ?After looking at the changes to
>>>> arch/x86/include/asm/cmpxchg_32.h, I cherry-picked two patches from
>>>> Linus's tree that seemed to make a difference, namely:
>>>>
>>>> 69309a05907546fb686b251d4ab041c26afe1e1d
>>>> 4532b305e8f0c238dd73048068ff8a6dd1380291
>>>>
>>>> Applying these patches solved the rebooting problem on 2.6.35.3
>>>>
>>>> I believe the same problem also happens for the latest stable version of
>>>> 2.6.34, but did not verify the fix worked for the 2.6.34 stable releases.
>>>>
>>>> I don't know if this is helpful, but I thought I would let you know in
>>>> case someone else is having the same problem.
>>>>
>>>> If you need any further information, just let me know.
>>>>
>>>
>>> Yes, as a matter of fact, we have been trying to root-cause this
>>> problem; please see:
>>>
>>> https://bugzilla.kernel.org/show_bug.cgi?id=16612
>>>
>>> If you could try the one-liner in there and see if it solves your
>>> problem, it would be great.
>>
>> The patch for __set_64bit on top of 2.6.35.3 solves the rebooting problem!
>>
>>
>
> By the way, I think only 69309a05907546fb686b251d4ab041c26afe1e1d should
> solve the problem. ?Mark, could you try only this patch?
>

Tested 69309a05907546fb686b251d4ab041c26afe1e1d on top of 2.6.35.3 and
this does solve the rebooting problem.

- Mark

2010-08-23 03:08:18

by Greg KH

[permalink] [raw]
Subject: Re: Linux 2.6.35.3

On Sun, Aug 22, 2010 at 09:56:19PM -0400, Mark Stanovich wrote:
> On Sun, Aug 22, 2010 at 9:31 PM, H. Peter Anvin <[email protected]> wrote:
> > On 08/20/2010 09:06 PM, Mark Stanovich wrote:
> >>>>
> >>>> I noticed that the latest version pulled from Linus's git tree did not
> >>>> have the problem. ?After looking at the changes to
> >>>> arch/x86/include/asm/cmpxchg_32.h, I cherry-picked two patches from
> >>>> Linus's tree that seemed to make a difference, namely:
> >>>>
> >>>> 69309a05907546fb686b251d4ab041c26afe1e1d
> >>>> 4532b305e8f0c238dd73048068ff8a6dd1380291
> >>>>
> >>>> Applying these patches solved the rebooting problem on 2.6.35.3
> >>>>
> >>>> I believe the same problem also happens for the latest stable version of
> >>>> 2.6.34, but did not verify the fix worked for the 2.6.34 stable releases.
> >>>>
> >>>> I don't know if this is helpful, but I thought I would let you know in
> >>>> case someone else is having the same problem.
> >>>>
> >>>> If you need any further information, just let me know.
> >>>>
> >>>
> >>> Yes, as a matter of fact, we have been trying to root-cause this
> >>> problem; please see:
> >>>
> >>> https://bugzilla.kernel.org/show_bug.cgi?id=16612
> >>>
> >>> If you could try the one-liner in there and see if it solves your
> >>> problem, it would be great.
> >>
> >> The patch for __set_64bit on top of 2.6.35.3 solves the rebooting problem!
> >>
> >>
> >
> > By the way, I think only 69309a05907546fb686b251d4ab041c26afe1e1d should
> > solve the problem. ?Mark, could you try only this patch?
> >
>
> Tested 69309a05907546fb686b251d4ab041c26afe1e1d on top of 2.6.35.3 and
> this does solve the rebooting problem.

Wonderful, thanks for testing, I'll just queue this one up then.

greg k-h

2010-08-23 13:47:27

by Mark Stanovich

[permalink] [raw]
Subject: Re: Linux 2.6.35.3

On Sun, Aug 22, 2010 at 11:03 PM, Greg KH <[email protected]> wrote:
> On Sun, Aug 22, 2010 at 09:56:19PM -0400, Mark Stanovich wrote:
>> On Sun, Aug 22, 2010 at 9:31 PM, H. Peter Anvin <[email protected]> wrote:
>> > On 08/20/2010 09:06 PM, Mark Stanovich wrote:
>> >>>>
>> >>>> I noticed that the latest version pulled from Linus's git tree did not
>> >>>> have the problem. ?After looking at the changes to
>> >>>> arch/x86/include/asm/cmpxchg_32.h, I cherry-picked two patches from
>> >>>> Linus's tree that seemed to make a difference, namely:
>> >>>>
>> >>>> 69309a05907546fb686b251d4ab041c26afe1e1d
>> >>>> 4532b305e8f0c238dd73048068ff8a6dd1380291
>> >>>>
>> >>>> Applying these patches solved the rebooting problem on 2.6.35.3
>> >>>>
>> >>>> I believe the same problem also happens for the latest stable version of
>> >>>> 2.6.34, but did not verify the fix worked for the 2.6.34 stable releases.
>> >>>>
>> >>>> I don't know if this is helpful, but I thought I would let you know in
>> >>>> case someone else is having the same problem.
>> >>>>
>> >>>> If you need any further information, just let me know.
>> >>>>
>> >>>
>> >>> Yes, as a matter of fact, we have been trying to root-cause this
>> >>> problem; please see:
>> >>>
>> >>> https://bugzilla.kernel.org/show_bug.cgi?id=16612
>> >>>
>> >>> If you could try the one-liner in there and see if it solves your
>> >>> problem, it would be great.
>> >>
>> >> The patch for __set_64bit on top of 2.6.35.3 solves the rebooting problem!
>> >>
>> >>
>> >
>> > By the way, I think only 69309a05907546fb686b251d4ab041c26afe1e1d should
>> > solve the problem. ?Mark, could you try only this patch?
>> >
>>
>> Tested 69309a05907546fb686b251d4ab041c26afe1e1d on top of 2.6.35.3 and
>> this does solve the rebooting problem.
>

69309a05907546fb686b251d4ab041c26afe1e1d is also needed on top of
2.6.34.5 in order for my machine to boot properly.

- Mark

2010-08-23 18:11:46

by Michael Tokarev

[permalink] [raw]
Subject: Re: Linux 2.6.35.3

21.08.2010 07:44, H. Peter Anvin wrote:
> On 08/20/2010 08:40 PM, Mark Stanovich wrote:
>> After updating to the 2.6.35.3 stable release, my machine was rebooting
>> automatically during the first few seconds of boot.
>>
>> I bisected to the first bad commit of
>>
>> commit 568132624386f53e87575195d868db
>> 9afb2e9316
>> Author: H. Peter Anvin <[email protected] <mailto:[email protected]>>
>> Date: Tue Jul 27 17:01:49 2010 -0700
>>
>> x86: Add memory modify constraints to xchg() and cmpxchg()

This commit also present in 2.6.32.stable (since 2.6.32.19).
Should it be fixed there as well?

Thanks!

/mjt

2010-08-23 18:45:26

by H. Peter Anvin

[permalink] [raw]
Subject: Re: Linux 2.6.35.3

On 08/23/2010 11:04 AM, Michael Tokarev wrote:
> 21.08.2010 07:44, H. Peter Anvin wrote:
>> On 08/20/2010 08:40 PM, Mark Stanovich wrote:
>>> After updating to the 2.6.35.3 stable release, my machine was rebooting
>>> automatically during the first few seconds of boot.
>>>
>>> I bisected to the first bad commit of
>>>
>>> commit 568132624386f53e87575195d868db
>>> 9afb2e9316
>>> Author: H. Peter Anvin <[email protected] <mailto:[email protected]>>
>>> Date: Tue Jul 27 17:01:49 2010 -0700
>>>
>>> x86: Add memory modify constraints to xchg() and cmpxchg()
>
> This commit also present in 2.6.32.stable (since 2.6.32.19).
> Should it be fixed there as well?
>

Yes, in particular 69309a05907546fb686b251d4ab041c26afe1e1d should be
applied to all instances of this commit.

As far as we can tell it's a gcc bug, but it hasn't been thoroughly
root-caused to that yet.

-hpa

2010-08-23 21:52:28

by Greg KH

[permalink] [raw]
Subject: Re: Linux 2.6.35.3

On Mon, Aug 23, 2010 at 09:47:23AM -0400, Mark Stanovich wrote:
> On Sun, Aug 22, 2010 at 11:03 PM, Greg KH <[email protected]> wrote:
> > On Sun, Aug 22, 2010 at 09:56:19PM -0400, Mark Stanovich wrote:
> >> On Sun, Aug 22, 2010 at 9:31 PM, H. Peter Anvin <[email protected]> wrote:
> >> > On 08/20/2010 09:06 PM, Mark Stanovich wrote:
> >> >>>>
> >> >>>> I noticed that the latest version pulled from Linus's git tree did not
> >> >>>> have the problem. ?After looking at the changes to
> >> >>>> arch/x86/include/asm/cmpxchg_32.h, I cherry-picked two patches from
> >> >>>> Linus's tree that seemed to make a difference, namely:
> >> >>>>
> >> >>>> 69309a05907546fb686b251d4ab041c26afe1e1d
> >> >>>> 4532b305e8f0c238dd73048068ff8a6dd1380291
> >> >>>>
> >> >>>> Applying these patches solved the rebooting problem on 2.6.35.3
> >> >>>>
> >> >>>> I believe the same problem also happens for the latest stable version of
> >> >>>> 2.6.34, but did not verify the fix worked for the 2.6.34 stable releases.
> >> >>>>
> >> >>>> I don't know if this is helpful, but I thought I would let you know in
> >> >>>> case someone else is having the same problem.
> >> >>>>
> >> >>>> If you need any further information, just let me know.
> >> >>>>
> >> >>>
> >> >>> Yes, as a matter of fact, we have been trying to root-cause this
> >> >>> problem; please see:
> >> >>>
> >> >>> https://bugzilla.kernel.org/show_bug.cgi?id=16612
> >> >>>
> >> >>> If you could try the one-liner in there and see if it solves your
> >> >>> problem, it would be great.
> >> >>
> >> >> The patch for __set_64bit on top of 2.6.35.3 solves the rebooting problem!
> >> >>
> >> >>
> >> >
> >> > By the way, I think only 69309a05907546fb686b251d4ab041c26afe1e1d should
> >> > solve the problem. ?Mark, could you try only this patch?
> >> >
> >>
> >> Tested 69309a05907546fb686b251d4ab041c26afe1e1d on top of 2.6.35.3 and
> >> this does solve the rebooting problem.
> >
>
> 69309a05907546fb686b251d4ab041c26afe1e1d is also needed on top of
> 2.6.34.5 in order for my machine to boot properly.

Ok, now queued up for both .34 and .35 stable trees.

thanks,

greg k-h