2008-03-03 08:18:13

by Zhang, Rui

[permalink] [raw]
Subject: 2.6.25-rc regression: kernel panic on boot

Hi, all,

kernel panic on boot when I try to run 2.6.25-rc1 and later, but 2.6.24-rc8 boots okay.
The result of git-bisect shows that
edfaa7c36574f1bf09c65ad602412db9da5f96bf is first bad commit.

The git-bisect-result and the dmesg log when the kernel panic are attached.
Any help on this please? :)

thanks,
rui



Attachments:
git-bisect-result (2.18 kB)
dmesg (77.60 kB)
Download all attachments

2008-03-03 22:43:20

by Greg KH

[permalink] [raw]
Subject: Re: 2.6.25-rc regression: kernel panic on boot

On Mon, Mar 03, 2008 at 07:25:49AM +0800, Zhang, Rui wrote:
> Hi, all,
>
> kernel panic on boot when I try to run 2.6.25-rc1 and later, but 2.6.24-rc8 boots okay.
> The result of git-bisect shows that
> edfaa7c36574f1bf09c65ad602412db9da5f96bf is first bad commit.
>
> The git-bisect-result and the dmesg log when the kernel panic are attached.
> Any help on this please? :)

Are you using LVM? If so, please either upgrade your userspace/initrd
stuff, or enable CONFIG_SYSFS_DEPCRECATED

thanks,

greg k-h

2008-03-04 03:23:18

by Zhang, Rui

[permalink] [raw]
Subject: Re: 2.6.25-rc regression: kernel panic on boot


On Tue, 2008-03-04 at 06:42 +0800, Greg KH wrote:
> On Mon, Mar 03, 2008 at 07:25:49AM +0800, Zhang, Rui wrote:
> > Hi, all,
> >
> > kernel panic on boot when I try to run 2.6.25-rc1 and later, but
> 2.6.24-rc8 boots okay.
> > The result of git-bisect shows that
> > edfaa7c36574f1bf09c65ad602412db9da5f96bf is first bad commit.
> >
> > The git-bisect-result and the dmesg log when the kernel panic are
> attached.
> > Any help on this please? :)
>
> Are you using LVM? If so, please either upgrade your userspace/initrd
> stuff, or enable CONFIG_SYSFS_DEPCRECATED
>
Yes, enabling CONFIG_SYSFS_DEPRECATED works for me. :)

thanks,
rui

2008-03-04 13:55:50

by Ingo Molnar

[permalink] [raw]
Subject: Re: 2.6.25-rc regression: kernel panic on boot


* Greg KH <[email protected]> wrote:

> On Mon, Mar 03, 2008 at 07:25:49AM +0800, Zhang, Rui wrote:
> > Hi, all,
> >
> > kernel panic on boot when I try to run 2.6.25-rc1 and later, but 2.6.24-rc8 boots okay.
> > The result of git-bisect shows that
> > edfaa7c36574f1bf09c65ad602412db9da5f96bf is first bad commit.
> >
> > The git-bisect-result and the dmesg log when the kernel panic are
> > attached. Any help on this please? :)
>
> Are you using LVM? If so, please either upgrade your userspace/initrd
> stuff, or enable CONFIG_SYSFS_DEPCRECATED

we _really_ must handle this differently and this _is_ a regression that
multiple people have spent hours on bisecting already ...

So "enable CONFIG_SYSFS_DEPCRECATED" is _NOT_ the right answer, and this
has been pointed out to you in the past.

the problem is that SYSFS_DEPRECATED changed its meaning mid-course. Now
it means a different set of udev breakages. The correct way is to
_RENAME_ that darn option to SYSFS_DEPRECATED_V2 (via the patch below),
and to discontinue the user-configurability of the old SYSFS_DEPRECATED
option. Viola, things work just fine - and the fact of SYSFS_DEPRECATED
changing its meaning is documented as well.

this way any tester who comes over the 'make oldconfig' route is not hit
in the head with a nasty regression ...

tested patch below. (and this is one of those rare cases where 'select'
is the right thing to do in a Kconfig entry)

Ingo

---------------------->
Subject: sysfs: CONFIG_SYSFS_DEPRECATED fix
From: Ingo Molnar <[email protected]>
Date: Tue Mar 04 14:35:21 CET 2008

CONFIG_SYSFS_DEPRECATED=y changed its meaning recently and causes
regressions in working setups that had SYSFS_DEPRECATED disabled.

so rename it to SYSFS_DEPRECATED_V2 so that testers pick up the new
default via 'make oldconfig', even if their old .config's disabled
CONFIG_SYSFS_DEPRECATED ...

Signed-off-by: Ingo Molnar <[email protected]>
---
init/Kconfig | 4 ++++
1 file changed, 4 insertions(+)

Index: linux/init/Kconfig
===================================================================
--- linux.orig/init/Kconfig
+++ linux/init/Kconfig
@@ -367,9 +367,13 @@ config RESOURCE_COUNTERS
depends on CGROUPS

config SYSFS_DEPRECATED
+ bool
+
+config SYSFS_DEPRECATED_V2
bool "Create deprecated sysfs files"
depends on SYSFS
default y
+ select SYSFS_DEPRECATED
help
This option creates deprecated symlinks such as the
"device"-link, the <subsystem>:<name>-link, and the

2008-03-04 18:45:07

by Andrew Morton

[permalink] [raw]
Subject: Re: 2.6.25-rc regression: kernel panic on boot

On Tue, 4 Mar 2008 14:54:47 +0100 Ingo Molnar <[email protected]> wrote:

> * Greg KH <[email protected]> wrote:
>
> > On Mon, Mar 03, 2008 at 07:25:49AM +0800, Zhang, Rui wrote:
> > > Hi, all,
> > >
> > > kernel panic on boot when I try to run 2.6.25-rc1 and later, but 2.6.24-rc8 boots okay.
> > > The result of git-bisect shows that
> > > edfaa7c36574f1bf09c65ad602412db9da5f96bf is first bad commit.
> > >
> > > The git-bisect-result and the dmesg log when the kernel panic are
> > > attached. Any help on this please? :)
> >
> > Are you using LVM? If so, please either upgrade your userspace/initrd
> > stuff, or enable CONFIG_SYSFS_DEPCRECATED
>
> we _really_ must handle this differently and this _is_ a regression that
> multiple people have spent hours on bisecting already ...
>
> So "enable CONFIG_SYSFS_DEPCRECATED" is _NOT_ the right answer, and this
> has been pointed out to you in the past.

Isn't the device-mapper kernel code capable of detecting the older userspace
version and loudly warning about it? That at least would prevent us from
wasting lots of people's time like this.

2008-03-04 19:54:30

by Greg KH

[permalink] [raw]
Subject: Re: 2.6.25-rc regression: kernel panic on boot

On Tue, Mar 04, 2008 at 10:43:12AM -0800, Andrew Morton wrote:
> On Tue, 4 Mar 2008 14:54:47 +0100 Ingo Molnar <[email protected]> wrote:
>
> > * Greg KH <[email protected]> wrote:
> >
> > > On Mon, Mar 03, 2008 at 07:25:49AM +0800, Zhang, Rui wrote:
> > > > Hi, all,
> > > >
> > > > kernel panic on boot when I try to run 2.6.25-rc1 and later, but 2.6.24-rc8 boots okay.
> > > > The result of git-bisect shows that
> > > > edfaa7c36574f1bf09c65ad602412db9da5f96bf is first bad commit.
> > > >
> > > > The git-bisect-result and the dmesg log when the kernel panic are
> > > > attached. Any help on this please? :)
> > >
> > > Are you using LVM? If so, please either upgrade your userspace/initrd
> > > stuff, or enable CONFIG_SYSFS_DEPCRECATED
> >
> > we _really_ must handle this differently and this _is_ a regression that
> > multiple people have spent hours on bisecting already ...
> >
> > So "enable CONFIG_SYSFS_DEPCRECATED" is _NOT_ the right answer, and this
> > has been pointed out to you in the past.
>
> Isn't the device-mapper kernel code capable of detecting the older userspace
> version and loudly warning about it? That at least would prevent us from
> wasting lots of people's time like this.

Yes, this is an LVM issue with older userspace provided by some
distros, I think the dm code can notify the user about the problem, Kay
would know more.

The two distros I use here work just fine with this set to N :)

I have a patch queued up to change the wording of this config entry to
not say 2006, as that's not true anymore, and it should hopefully fix
some of the confusion here.

thanks,

greg k-h

2008-03-04 20:06:52

by Linus Torvalds

[permalink] [raw]
Subject: Re: 2.6.25-rc regression: kernel panic on boot



On Tue, 4 Mar 2008, Greg KH wrote:
>
> I have a patch queued up to change the wording of this config entry to
> not say 2006, as that's not true anymore, and it should hopefully fix
> some of the confusion here.

Well, I think some of the confusion is simply from people who already
turned it off *before* (without problems), and now just do "make
oldconfig" or otherwise just keep it off, and it causes problems for them.

Updating the help text won't do squat, because those people won't be
reading it - maybe they read it back when it was a new option (and it
worked for them), but not any more.

So I would almost suggest we simply force the option back on for now.
Then, in 2.6.26, we can make it a question again, and by that time, people
will have it as 'y' in their config files and will now have the option to
turn it off again.

In other words, how about just this diff?

Linus

---
init/Kconfig | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/init/Kconfig b/init/Kconfig
index f698a5a..a36e2e2 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -367,7 +367,7 @@ config RESOURCE_COUNTERS
depends on CGROUPS

config SYSFS_DEPRECATED
- bool "Create deprecated sysfs files"
+ bool "Create deprecated sysfs files" if EMBEDDED
depends on SYSFS
default y
help

2008-03-04 20:24:52

by Greg KH

[permalink] [raw]
Subject: Re: 2.6.25-rc regression: kernel panic on boot

On Tue, Mar 04, 2008 at 12:04:50PM -0800, Linus Torvalds wrote:
>
>
> On Tue, 4 Mar 2008, Greg KH wrote:
> >
> > I have a patch queued up to change the wording of this config entry to
> > not say 2006, as that's not true anymore, and it should hopefully fix
> > some of the confusion here.
>
> Well, I think some of the confusion is simply from people who already
> turned it off *before* (without problems), and now just do "make
> oldconfig" or otherwise just keep it off, and it causes problems for them.
>
> Updating the help text won't do squat, because those people won't be
> reading it - maybe they read it back when it was a new option (and it
> worked for them), but not any more.

Yeah, that makes sense :(

> So I would almost suggest we simply force the option back on for now.
> Then, in 2.6.26, we can make it a question again, and by that time, people
> will have it as 'y' in their config files and will now have the option to
> turn it off again.
>
> In other words, how about just this diff?

Well, this will force the distros that are going to ship based on 2.6.25
(Fedora 9, openSUSE 11.0, etc.) to either enable EMBEDDED, or patch this
chunk out, as they do want to run with that option enabled because they
do have up-to-date userspace code.

It's as if we should just change the config option name every kernel
release to make sure that users keep things up to date. So Ingo's
original proposal makes sense to me, more so than this patch.

thanks,

greg k-h

2008-03-05 01:22:42

by Alasdair G Kergon

[permalink] [raw]
Subject: Re: [dm-devel] Re: 2.6.25-rc regression: kernel panic on boot

On Tue, Mar 04, 2008 at 10:43:12AM -0800, Andrew Morton wrote:
> Isn't the device-mapper kernel code capable of detecting the older userspace
> version and loudly warning about it? That at least would prevent us from
> wasting lots of people's time like this.

It's not a device-mapper problem: It's between userspace LVM2 and kernel
sysfs.

LVM2 userspace expects a particular sysfs layout. That layout got
changed without offering sufficient lead time (>6-9 months) for the
required userspace changes to propagate into the main versions of the
distros people use.

LVM2 userspace version 2.02.29 (released 5th December 2007) can handle
old and new layouts. The patch was in the CVS head for a while before
that. Earlier versions of LVM2 require CONFIG_SYSFS_DEPRECATED. A
less-efficient workaround is to disable the use of sysfs in the LVM2
config: 'devices { sysfs_scan = 0 }'.

(I think the root cause of this mess was changing what
CONFIG_SYSFS_DEPRECATED did instead of adding a new setting that could
be enabled by default.)

Alasdair
--
[email protected]

2008-03-05 10:17:15

by Alexey Dobriyan

[permalink] [raw]
Subject: Re: 2.6.25-rc regression: kernel panic on boot

On Tue, Mar 04, 2008 at 02:54:47PM +0100, Ingo Molnar wrote:
> CONFIG_SYSFS_DEPRECATED=y changed its meaning recently and causes
> regressions in working setups that had SYSFS_DEPRECATED disabled.
>
> so rename it to SYSFS_DEPRECATED_V2 so that testers pick up the new
> default via 'make oldconfig', even if their old .config's disabled
> CONFIG_SYSFS_DEPRECATED ...

It has all the same help text, so people who disabled it in the past
will disable again!

> --- linux.orig/init/Kconfig
> +++ linux/init/Kconfig
> @@ -367,9 +367,13 @@ config RESOURCE_COUNTERS
> depends on CGROUPS
>
> config SYSFS_DEPRECATED
> + bool
> +
> +config SYSFS_DEPRECATED_V2
> bool "Create deprecated sysfs files"
> depends on SYSFS
> default y
> + select SYSFS_DEPRECATED
> help
> This option creates deprecated symlinks such as the
> "device"-link, the <subsystem>:<name>-link, and the

Is anyone aware of a case when turning SYSFS_DEPRECATED back on also
breaks something? I mean, option can be simply removed and sysfs people
can finally stop breaking boxes.

2008-03-05 13:26:37

by Ingo Molnar

[permalink] [raw]
Subject: Re: 2.6.25-rc regression: kernel panic on boot


* Greg KH <[email protected]> wrote:

> It's as if we should just change the config option name every kernel
> release to make sure that users keep things up to date. So Ingo's
> original proposal makes sense to me, more so than this patch.

this change is now upstream and included in -rc4:

------------------->
commit d47846c5866b7d98a1173c86a39d810a06647329
Author: Ingo Molnar <[email protected]>
Date: Tue Mar 4 14:54:47 2008 +0100

sysfs: CONFIG_SYSFS_DEPRECATED fix
<------------------

thanks Greg!

Ingo

2008-03-05 13:30:30

by Ingo Molnar

[permalink] [raw]
Subject: Re: 2.6.25-rc regression: kernel panic on boot


* Alexey Dobriyan <[email protected]> wrote:

> On Tue, Mar 04, 2008 at 02:54:47PM +0100, Ingo Molnar wrote:
> > CONFIG_SYSFS_DEPRECATED=y changed its meaning recently and causes
> > regressions in working setups that had SYSFS_DEPRECATED disabled.
> >
> > so rename it to SYSFS_DEPRECATED_V2 so that testers pick up the new
> > default via 'make oldconfig', even if their old .config's disabled
> > CONFIG_SYSFS_DEPRECATED ...
>
> It has all the same help text, so people who disabled it in the past
> will disable again!

please send a patch that changes the help text. For this to happen
people have to change the default consciously.

> > +config SYSFS_DEPRECATED_V2
> > bool "Create deprecated sysfs files"
> > depends on SYSFS
> > default y
> > + select SYSFS_DEPRECATED
> > help
> > This option creates deprecated symlinks such as the
> > "device"-link, the <subsystem>:<name>-link, and the
>
> Is anyone aware of a case when turning SYSFS_DEPRECATED back on also
> breaks something? I mean, option can be simply removed and sysfs
> people can finally stop breaking boxes.

i have no strong opinion either way, but i think distributions that ship
new userspace can legitimately disable this option.

That way once all new distributions have updated userspace we can flip
the default around and can mark it 'default n'. This at least gives us a
theoretical path towards getting rid of unwanted legacies. (I suspect
this was the plan all along, it's just that the clock got restarted now
with the different legacy/breakage property.)

longer term we could also put in a WARN_ON_ONCE() which kerneloops.org
would automatically track. That would give an objective metric about
when to change the default. (and when to get rid of the legacy
altogether)

Ingo

2008-03-05 15:37:50

by Greg KH

[permalink] [raw]
Subject: Re: 2.6.25-rc regression: kernel panic on boot

On Wed, Mar 05, 2008 at 01:16:17PM +0300, Alexey Dobriyan wrote:
> On Tue, Mar 04, 2008 at 02:54:47PM +0100, Ingo Molnar wrote:
> > CONFIG_SYSFS_DEPRECATED=y changed its meaning recently and causes
> > regressions in working setups that had SYSFS_DEPRECATED disabled.
> >
> > so rename it to SYSFS_DEPRECATED_V2 so that testers pick up the new
> > default via 'make oldconfig', even if their old .config's disabled
> > CONFIG_SYSFS_DEPRECATED ...
>
> It has all the same help text, so people who disabled it in the past
> will disable again!

I did change the help text. If you think the new wording is not
sufficient, please send me a patch against 2.6.25-rc4.

thanks,

greg k-h