2009-12-14 19:52:25

by Tim Blechmann

[permalink] [raw]
Subject: 2.6.32 regression: unable to handle kernel paging request

hi all,

i cannot boot any 2.6.32 kernel (i tested back until -rc5) on my nehalem
machine (x86_64, 12gb ram).

there are bug messages in various (mostly module loader) functions,
stating "unable to handle kernel paging request". 2.6.31 on the same
machine works like charm. i haven't been able to save a boot log, but it
didn't look too different from this:

http://lkml.org/lkml/2009/12/8/207

thnx, tim

--
[email protected]
http://tim.klingt.org

I must say I find television very educational. The minute somebody
turns it on, I go to the library and read a good book.
Groucho Marx


Attachments:
signature.asc (197.00 B)
OpenPGP digital signature

2009-12-17 12:57:50

by Tim Blechmann

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

> i cannot boot any 2.6.32 kernel (i tested back until -rc5) on my nehalem
> machine (x86_64, 12gb ram).
>
> there are bug messages in various (mostly module loader) functions,
> stating "unable to handle kernel paging request". 2.6.31 on the same
> machine works like charm. i haven't been able to save a boot log, but it
> didn't look too different from this:
>
> http://lkml.org/lkml/2009/12/8/207

trying to bisect the kernel tree,
405d967dc70002991f8fc35c20e0d3cbc7614f63 seems to be the first bad commit.
during bisecting, i recorded several boot logs, which are attached.

i hope, this helps ...

tim

--
[email protected]
http://tim.klingt.org

The first question I ask myself when something doesn't seem to be
beautiful is why do I think it's not beautiful. And very shortly you
discover that there is no reason.
John Cage.


Attachments:
log (61.48 kB)
log2 (73.79 kB)
log3 (69.05 kB)
log4 (68.43 kB)
signature.asc (197.00 B)
OpenPGP digital signature
Download all attachments

2009-12-17 20:27:34

by Tim Blechmann

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

>> i cannot boot any 2.6.32 kernel (i tested back until -rc5) on my nehalem
>> machine (x86_64, 12gb ram).
>>
>> there are bug messages in various (mostly module loader) functions,
>> stating "unable to handle kernel paging request". 2.6.31 on the same
>> machine works like charm. i haven't been able to save a boot log, but it
>> didn't look too different from this:
>>
>> http://lkml.org/lkml/2009/12/8/207
>
> trying to bisect the kernel tree,
> 405d967dc70002991f8fc35c20e0d3cbc7614f63 seems to be the first bad commit.
> during bisecting, i recorded several boot logs, which are attached.

removing module-common.lds from LDFLAGS_MODULE, my machine boots fine
and without any error messages:

--- a/Makefile
+++ b/Makefile
@@ -326,7 +326,7 @@ CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \
MODFLAGS = -DMODULE
CFLAGS_MODULE = $(MODFLAGS)
AFLAGS_MODULE = $(MODFLAGS)
-LDFLAGS_MODULE = -T $(srctree)/scripts/module-common.lds
+LDFLAGS_MODULE =
CFLAGS_KERNEL =
AFLAGS_KERNEL =
CFLAGS_GCOV = -fprofile-arcs -ftest-coverage

i hope someone with more insights in these parts of the kernel can have
a closer look into this issue.

thanks, tim

--
[email protected]
http://tim.klingt.org

Wherever we are, what we hear is mostly noise. When we ignore it, it
disturbs us. When we listen to it, we find it fascinating.
John Cage


Attachments:
signature.asc (197.00 B)
OpenPGP digital signature

2009-12-18 03:46:51

by Tejun Heo

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

Hello,

On 12/18/2009 05:26 AM, Tim Blechmann wrote:
>> trying to bisect the kernel tree,
>> 405d967dc70002991f8fc35c20e0d3cbc7614f63 seems to be the first bad commit.
>> during bisecting, i recorded several boot logs, which are attached.
>
> removing module-common.lds from LDFLAGS_MODULE, my machine boots fine
> and without any error messages:

Looks like some necessary part of module image ends up in discarded
sections. Before the commit, the build process kept those sections so
it worked. The code has been there for some time now and you're the
first one to report it, so I suppose your .config is a bit different
from more usual ones. Can you please attach it?

Thanks.

--
tejun

2009-12-18 09:16:39

by Tim Blechmann

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

>>> trying to bisect the kernel tree,
>>> 405d967dc70002991f8fc35c20e0d3cbc7614f63 seems to be the first bad commit.
>>> during bisecting, i recorded several boot logs, which are attached.
>>
>> removing module-common.lds from LDFLAGS_MODULE, my machine boots fine
>> and without any error messages:
>
> Looks like some necessary part of module image ends up in discarded
> sections. Before the commit, the build process kept those sections so
> it worked. The code has been there for some time now and you're the
> first one to report it, so I suppose your .config is a bit different
> from more usual ones. Can you please attach it?

it is not too different ... it is based on the .config file of the
ubuntu kernel with a few tweaks ...
one odd thing is, that i once was able to boot 2.6.32-rcX kernels, at
one point, i kept getting these bug messages ... i didn't (knowingly)
change the configuration, though ...

tim

--
[email protected]
http://tim.klingt.org

Im übrigen ist es gescheiter, sich warm zuzudecken als sich zu
betrinken.
Werner Schwab


Attachments:
.config (89.02 kB)
signature.asc (197.00 B)
OpenPGP digital signature
Download all attachments

2009-12-21 07:14:12

by Tejun Heo

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

On 12/18/2009 06:15 PM, Tim Blechmann wrote:
>>>> trying to bisect the kernel tree,
>>>> 405d967dc70002991f8fc35c20e0d3cbc7614f63 seems to be the first bad commit.
>>>> during bisecting, i recorded several boot logs, which are attached.
>>>
>>> removing module-common.lds from LDFLAGS_MODULE, my machine boots fine
>>> and without any error messages:
>>
>> Looks like some necessary part of module image ends up in discarded
>> sections. Before the commit, the build process kept those sections so
>> it worked. The code has been there for some time now and you're the
>> first one to report it, so I suppose your .config is a bit different
>> from more usual ones. Can you please attach it?
>
> it is not too different ... it is based on the .config file of the
> ubuntu kernel with a few tweaks ...
> one odd thing is, that i once was able to boot 2.6.32-rcX kernels, at
> one point, i kept getting these bug messages ... i didn't (knowingly)
> change the configuration, though ...

Hmmm... weirdly I can't reproduce the problem here with the same
config. Can you please attach "readelf -a lp.ko" with and without the
module-common.lds struck out?

Thanks.

--
tejun

2009-12-21 09:33:50

by Tim Blechmann

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

>>>> removing module-common.lds from LDFLAGS_MODULE, my machine boots fine
>>>> and without any error messages:
>>>
>>> Looks like some necessary part of module image ends up in discarded
>>> sections. Before the commit, the build process kept those sections so
>>> it worked. The code has been there for some time now and you're the
>>> first one to report it, so I suppose your .config is a bit different
>>> from more usual ones. Can you please attach it?
>>
>> it is not too different ... it is based on the .config file of the
>> ubuntu kernel with a few tweaks ...
>> one odd thing is, that i once was able to boot 2.6.32-rcX kernels, at
>> one point, i kept getting these bug messages ... i didn't (knowingly)
>> change the configuration, though ...
>
> Hmmm... weirdly I can't reproduce the problem here with the same
> config. Can you please attach "readelf -a lp.ko" with and without the
> module-common.lds struck out?

as i said before, the configuration is not really uncommon ... the
readelf output is attached.

hth, tim

--
[email protected]
http://tim.klingt.org

Nothing exists until or unless it is observed. An artist is making
something exist by observing it. And his hope for other people is that
they will also make it exist by observing it. I call it 'creative
observation.' Creative viewing.
William S. Burroughs


Attachments:
readelf_bug (34.63 kB)
readelf_working (34.59 kB)
signature.asc (197.00 B)
OpenPGP digital signature
Download all attachments

2009-12-24 12:03:05

by Tejun Heo

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

Hello,

On 12/21/2009 06:32 PM, Tim Blechmann wrote:
>>>>> removing module-common.lds from LDFLAGS_MODULE, my machine boots fine
>>>>> and without any error messages:
>>>>
>>>> Looks like some necessary part of module image ends up in discarded
>>>> sections. Before the commit, the build process kept those sections so
>>>> it worked. The code has been there for some time now and you're the
>>>> first one to report it, so I suppose your .config is a bit different
>>>> from more usual ones. Can you please attach it?
>>>
>>> it is not too different ... it is based on the .config file of the
>>> ubuntu kernel with a few tweaks ...
>>> one odd thing is, that i once was able to boot 2.6.32-rcX kernels, at
>>> one point, i kept getting these bug messages ... i didn't (knowingly)
>>> change the configuration, though ...
>>
>> Hmmm... weirdly I can't reproduce the problem here with the same
>> config. Can you please attach "readelf -a lp.ko" with and without the
>> module-common.lds struck out?
>
> as i said before, the configuration is not really uncommon ... the
> readelf output is attached.

Hmmm.... I think it's more convoluted than I originally thought. I
was expecting a missing section but in the broken case the linker has
assigned addresses to sections even though it was a relocatable link.
That's just strange.

Can you please post the output of "ld --version"? Also, what distro
are you on?

Thanks.

--
tejun

2009-12-24 12:07:51

by Tim Blechmann

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

>>>>>> removing module-common.lds from LDFLAGS_MODULE, my machine boots fine
>>>>>> and without any error messages:
>>>>>
>>>>> Looks like some necessary part of module image ends up in discarded
>>>>> sections. Before the commit, the build process kept those sections so
>>>>> it worked. The code has been there for some time now and you're the
>>>>> first one to report it, so I suppose your .config is a bit different
>>>>> from more usual ones. Can you please attach it?
>>>>
>>>> it is not too different ... it is based on the .config file of the
>>>> ubuntu kernel with a few tweaks ...
>>>> one odd thing is, that i once was able to boot 2.6.32-rcX kernels, at
>>>> one point, i kept getting these bug messages ... i didn't (knowingly)
>>>> change the configuration, though ...
>>>
>>> Hmmm... weirdly I can't reproduce the problem here with the same
>>> config. Can you please attach "readelf -a lp.ko" with and without the
>>> module-common.lds struck out?
>>
>> as i said before, the configuration is not really uncommon ... the
>> readelf output is attached.
>
> Hmmm.... I think it's more convoluted than I originally thought. I
> was expecting a missing section but in the broken case the linker has
> assigned addresses to sections even though it was a relocatable link.
> That's just strange.
>
> Can you please post the output of "ld --version"? Also, what distro
> are you on?

i am not at that specific machine, but the distribution is the same as
on my laptop, ubuntu 9.10, x86-64. i don't have any issues, running
2.6.32.2 on my laptop, though ...

GNU ld (GNU Binutils for Ubuntu) 2.20

hth, tim

--
[email protected]
http://tim.klingt.org

Bill Gates left his university to start Micro$oft.
Steve Jobs sold his Wolkswagen bus to start Apple.
Linus Torvalds made a new thread in a forum and put a file on a ftp server.

2009-12-24 12:22:56

by Tejun Heo

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

Hello,

On 12/24/2009 09:07 PM, Tim Blechmann wrote:
>>> as i said before, the configuration is not really uncommon ... the
>>> readelf output is attached.
>>
>> Hmmm.... I think it's more convoluted than I originally thought. I
>> was expecting a missing section but in the broken case the linker has
>> assigned addresses to sections even though it was a relocatable link.
>> That's just strange.
>>
>> Can you please post the output of "ld --version"? Also, what distro
>> are you on?
>
> i am not at that specific machine, but the distribution is the same as
> on my laptop, ubuntu 9.10, x86-64. i don't have any issues, running
> 2.6.32.2 on my laptop, though ...
>
> GNU ld (GNU Binutils for Ubuntu) 2.20

Sections aren't supposed to have addresses assigned in relocatable
output and it of course breaks things. I'll try newer version of
binutils but at this point it looks like a linker bug. Can you please
report it to ubuntu too?

Thanks.

--
tejun

2009-12-24 12:44:45

by Tejun Heo

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

Hello, again.

On 12/24/2009 09:26 PM, Tejun Heo wrote:
>> GNU ld (GNU Binutils for Ubuntu) 2.20
>
> Sections aren't supposed to have addresses assigned in relocatable
> output and it of course breaks things. I'll try newer version of
> binutils but at this point it looks like a linker bug. Can you please
> report it to ubuntu too?

I just tried 2.20 on openSUSE.

$ ld --version
GNU ld (GNU Binutils; devel:gcc / openSUSE_11.2) 2.20.0.20091104-13.2
Copyright 2009 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

and it doesn't show such behavior. Can you please report this to
ubuntu and let me know the bug#?

Thanks.

--
tejun

2009-12-24 12:56:49

by Tim Blechmann

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

>> Sections aren't supposed to have addresses assigned in relocatable
>> output and it of course breaks things. I'll try newer version of
>> binutils but at this point it looks like a linker bug. Can you please
>> report it to ubuntu too?
>
> I just tried 2.20 on openSUSE.
>
> $ ld --version
> GNU ld (GNU Binutils; devel:gcc / openSUSE_11.2) 2.20.0.20091104-13.2
> Copyright 2009 Free Software Foundation, Inc.
> This program is free software; you may redistribute it under the terms of
> the GNU General Public License version 3 or (at your option) a later version.
> This program has absolutely no warranty.
>
> and it doesn't show such behavior. Can you please report this to
> ubuntu and let me know the bug#?

https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/500113

i will double-check on the other machine in a few days ... it may be,
that i am using a toolchain ppa, that installed some unstable/odd/buggy
binutils ...
but i somehow doubt, that this is the case :/

cheers, tim

--
[email protected]
http://tim.klingt.org

Just what the hell is the experimental tradition?
Morton Feldman

2009-12-27 09:51:53

by Tim Blechmann

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

> I just tried 2.20 on openSUSE.
>
> $ ld --version
> GNU ld (GNU Binutils; devel:gcc / openSUSE_11.2) 2.20.0.20091104-13.2
> Copyright 2009 Free Software Foundation, Inc.
> This program is free software; you may redistribute it under the terms of
> the GNU General Public License version 3 or (at your option) a later version.
> This program has absolutely no warranty.
>
> and it doesn't show such behavior. Can you please report this to
> ubuntu and let me know the bug#?

ah, back at that workstation, i found, that i was using a build of
binutils, installed via the ubuntu-toolchain-r ppa:

ld -v
GNU gold (GNU Binutils for Ubuntu 2.20) 1.9

i am pretty sure, it is related to this, will try to compile an
unpatched kernel now ...

thanks, tim

--
[email protected]
http://tim.klingt.org

The most wonderful opportunity which life offers is to be human.
Henry Miller


Attachments:
signature.asc (197.00 B)
OpenPGP digital signature

2010-01-02 02:22:32

by Tejun Heo

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

Hello,

On 01/02/2010 05:55 AM, Tim Blechmann wrote:
> trying to boot the current linus/master branch, i saw a similar issue as
> before, but with binutils 2.20

Given that the ldscript modification has been out in the wild for some
time and you are the only one reporting this type of problem, I don't
think it's likely to be a widespread problem. Can you please try it
from a clean installation just in case?

Thanks.

--
tejun

2010-01-01 20:56:43

by Tim Blechmann

[permalink] [raw]
Subject: Re: 2.6.32 regression: unable to handle kernel paging request

On 12/27/2009 10:50 AM, Tim Blechmann wrote:
>> I just tried 2.20 on openSUSE.
>>
>> $ ld --version
>> GNU ld (GNU Binutils; devel:gcc / openSUSE_11.2) 2.20.0.20091104-13.2
>> Copyright 2009 Free Software Foundation, Inc.
>> This program is free software; you may redistribute it under the terms of
>> the GNU General Public License version 3 or (at your option) a later version.
>> This program has absolutely no warranty.
>>
>> and it doesn't show such behavior. Can you please report this to
>> ubuntu and let me know the bug#?
>
> ah, back at that workstation, i found, that i was using a build of
> binutils, installed via the ubuntu-toolchain-r ppa:
>
> ld -v
> GNU gold (GNU Binutils for Ubuntu 2.20) 1.9

trying to boot the current linus/master branch, i saw a similar issue as
before, but with binutils 2.20

tim

--
[email protected]
http://tim.klingt.org

Our products just aren't engineered for security.
Brian Valentine, Microsoft's vice president of Windows development


Attachments:
signature.asc (197.00 B)
OpenPGP digital signature