LinuxLists.cc - What exactly are the issues with 2.6.0-test10 preempt?

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Mon, 24 Nov 2003, Bradley Chapman wrote:

> I saw in Linus' 2.6.0-test10 announcement that preempt is suffering from some
> problems and should not be used. However, I am currently running 2.6.0-test10
> with CONFIG_PREEMPT=y and nothing has appeared yet. To see if the problem appeared
> under stress, I started an A/V trailer playback with mplayer and then ran the
> find command on both my home directory and the 2.6.0-test10 kernel source directory,
> with the expected result - mplayer did not skip, neither find invocation broke,
> and there were no nasty errors in dmesg.
>
> So what exactly is the problem?

Well, FWIW, I'm getting 100% reproducible Oopses on __boot__ by enabling
preemption AND (almost) all kernel-hacking CONFIG_DEBUG_* options - see my
post of 21.11.2003 with subject "[OOPS] 2.6.0-test7 + preempt + hacking".
If required, could try to narrow it down to 1 CONFIG option. However, the
Oops itself happens somewhere in NFS code (see backtrace in above email
for details).

Guennadi
---
Guennadi Liakhovetski

2003-11-24 21:29:53

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

Mr. Liakhovetski,

--- Guennadi Liakhovetski <[email protected]> wrote:
> On Mon, 24 Nov 2003, Bradley Chapman wrote:
>
> > I saw in Linus' 2.6.0-test10 announcement that preempt is suffering from some
> > problems and should not be used. However, I am currently running 2.6.0-test10
> > with CONFIG_PREEMPT=y and nothing has appeared yet. To see if the problem
> appeared
> > under stress, I started an A/V trailer playback with mplayer and then ran the
> > find command on both my home directory and the 2.6.0-test10 kernel source
> directory,
> > with the expected result - mplayer did not skip, neither find invocation broke,
> > and there were no nasty errors in dmesg.
> >
> > So what exactly is the problem?
>
> Well, FWIW, I'm getting 100% reproducible Oopses on __boot__ by enabling
> preemption AND (almost) all kernel-hacking CONFIG_DEBUG_* options - see my
> post of 21.11.2003 with subject "[OOPS] 2.6.0-test7 + preempt + hacking".
> If required, could try to narrow it down to 1 CONFIG option. However, the
> Oops itself happens somewhere in NFS code (see backtrace in above email
> for details).

Hmmm. This is what I have enabled under "Kernel hacking":

#
# Kernel hacking
#
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_IOVIRT=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_SPINLOCK=y
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_DEBUG_INFO is not set
CONFIG_DEBUG_SPINLOCK_SLEEP=y
# CONFIG_FRAME_POINTER is not set
CONFIG_X86_EXTRA_IRQS=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y

So far, though, not a single problem -- I was playing Flash animations while
printing a huge document without a single hiccup (I have an HP DeskJet 880C with
very little memory, so there was a lot of in-kernel activity).

>
> Guennadi

Brad

=====

__________________________________
Do you Yahoo!?
Free Pop-Up Blocker - Get it now
http://companion.yahoo.com/

2003-11-24 21:49:35

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Mon, 24 Nov 2003, Bradley Chapman wrote:

> Hmmm. This is what I have enabled under "Kernel hacking":
>
> #
> # Kernel hacking
> #
> CONFIG_DEBUG_KERNEL=y
> CONFIG_DEBUG_STACKOVERFLOW=y
> CONFIG_DEBUG_SLAB=y
> CONFIG_DEBUG_IOVIRT=y
> CONFIG_MAGIC_SYSRQ=y
> CONFIG_DEBUG_SPINLOCK=y
> # CONFIG_DEBUG_PAGEALLOC is not set
^^^^^^^^^^^^^^^^^^^^^^

Here's a candidate. I did have it on. Yes, I know it causes a slowdown -
that was for testing. I could try to double check, if disabling that
single option fixes the problem - but for that I'd need to recompile and
re-install kernel and modules... I also didn't have
CONFIG_DEBUG_STACKOVERFLOW switched on - but I don't think that could be
the reason.

> # CONFIG_DEBUG_INFO is not set
> CONFIG_DEBUG_SPINLOCK_SLEEP=y
> # CONFIG_FRAME_POINTER is not set
> CONFIG_X86_EXTRA_IRQS=y
> CONFIG_X86_FIND_SMP_CONFIG=y
> CONFIG_X86_MPPARSE=y
>
> So far, though, not a single problem -- I was playing Flash animations while
> printing a huge document without a single hiccup (I have an HP DeskJet 880C with
> very little memory, so there was a lot of in-kernel activity).

Guennadi
---
Guennadi Liakhovetski

2003-11-24 21:55:25

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

Mr. Liakhovetski,

--- Guennadi Liakhovetski <[email protected]> wrote:
> On Mon, 24 Nov 2003, Bradley Chapman wrote:
>
> > Hmmm. This is what I have enabled under "Kernel hacking":
> >
> > #
> > # Kernel hacking
> > #
> > CONFIG_DEBUG_KERNEL=y
> > CONFIG_DEBUG_STACKOVERFLOW=y
> > CONFIG_DEBUG_SLAB=y
> > CONFIG_DEBUG_IOVIRT=y
> > CONFIG_MAGIC_SYSRQ=y
> > CONFIG_DEBUG_SPINLOCK=y
> > # CONFIG_DEBUG_PAGEALLOC is not set
> ^^^^^^^^^^^^^^^^^^^^^^
>
> Here's a candidate. I did have it on. Yes, I know it causes a slowdown -
> that was for testing. I could try to double check, if disabling that
> single option fixes the problem - but for that I'd need to recompile and
> re-install kernel and modules... I also didn't have
> CONFIG_DEBUG_STACKOVERFLOW switched on - but I don't think that could be
> the reason.

I've never enabled CONFIG_DEBUG_PAGEALLOC - the slowdown bit in the help text
never gave me a warm&fuzzy. If CONFIG_PREEMPT=y && CONFIG_DEBUG_PAGEALLOC=y,
perhaps the pagealloc debug code is exposing a messed-up preempt somewhere near
a memory allocation (IANAKC).

I found the thread regarding slab Oopsen where Linus mentioned CONFIG_PREEMPT
as a possiblity, but I'm not experiencing any issues with -test10 or -test9-bk22.
So I don't think the bug in that thread is affecting me at all, if it hasn't been
fixed.

>
> > # CONFIG_DEBUG_INFO is not set
> > CONFIG_DEBUG_SPINLOCK_SLEEP=y
> > # CONFIG_FRAME_POINTER is not set
> > CONFIG_X86_EXTRA_IRQS=y
> > CONFIG_X86_FIND_SMP_CONFIG=y
> > CONFIG_X86_MPPARSE=y
> >
> > So far, though, not a single problem -- I was playing Flash animations while
> > printing a huge document without a single hiccup (I have an HP DeskJet 880C with
> > very little memory, so there was a lot of in-kernel activity).
>
> Guennadi

Brad

=====

__________________________________
Do you Yahoo!?
Free Pop-Up Blocker - Get it now
http://companion.yahoo.com/

2003-11-24 21:59:31

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Mon, 24 Nov 2003, Guennadi Liakhovetski wrote:
>
> Well, FWIW, I'm getting 100% reproducible Oopses on __boot__ by enabling
> preemption AND (almost) all kernel-hacking CONFIG_DEBUG_* options - see my
> post of 21.11.2003 with subject "[OOPS] 2.6.0-test7 + preempt + hacking".
> If required, could try to narrow it down to 1 CONFIG option.

I'd love to have more info - I actually looked at your original report,
and it's one of those "impossible" things as far as I can tell. The low
bit of the work "pending" flag should acts as a lock on workqueues, and
serialize access to one workqueue totally - so having it show up with a
pending timer is "strange" to say the least. The only two ways to clear
the "pending" timer is by running the work-queue - either for the timer to
have gone off (for the delayed case) _or_ the timer not to have evern been
set in the first place (for the immediate case).

So more information would be wonderful.

Linus

2003-11-24 22:26:57

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

Mr. Torvalds,

--- Linus Torvalds <[email protected]> wrote:
>
>
> On Mon, 24 Nov 2003, Guennadi Liakhovetski wrote:
> >
> > Well, FWIW, I'm getting 100% reproducible Oopses on __boot__ by enabling
> > preemption AND (almost) all kernel-hacking CONFIG_DEBUG_* options - see my
> > post of 21.11.2003 with subject "[OOPS] 2.6.0-test7 + preempt + hacking".
> > If required, could try to narrow it down to 1 CONFIG option.
>
> I'd love to have more info - I actually looked at your original report,
> and it's one of those "impossible" things as far as I can tell. The low
> bit of the work "pending" flag should acts as a lock on workqueues, and
> serialize access to one workqueue totally - so having it show up with a
> pending timer is "strange" to say the least. The only two ways to clear
> the "pending" timer is by running the work-queue - either for the timer to
> have gone off (for the delayed case) _or_ the timer not to have evern been
> set in the first place (for the immediate case).
>
> So more information would be wonderful.

What sort of information would you like me to provide, sir? The bug you're
discussing here isn't affecting me; CONFIG_PREEMPT has been solid on 2.6.0-test10.
This is on a Gateway 600S laptop with a P4-M 2Ghz processor and an i845 Brookdale
chipset.

>
> Linus

Brad

=====

__________________________________
Do you Yahoo!?
Free Pop-Up Blocker - Get it now
http://companion.yahoo.com/

2003-11-24 22:32:34

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Mon, 24 Nov 2003, Bradley Chapman wrote:
>
> What sort of information would you like me to provide, sir? The bug you're
> discussing here isn't affecting me; CONFIG_PREEMPT has been solid on 2.6.0-test10.
> This is on a Gateway 600S laptop with a P4-M 2Ghz processor and an i845 Brookdale
> chipset.

Basically, there's something strange going on, which _seems_ to be memory
corruption, and seems to correlate reasonable well (but not 100%) with
CONFIG_PREEMPT.

It's actually unlikely to be preemption itself that is broken: it's much
more likely that some driver or other subsystem is broken, and preempt is
just better at triggering it by making some race conditions much easier to
see due to bigger windows for them to happen.

The problem is finding enough of a pattern to the reports to make sense of
what seems to be the common thread. A lot of people use preemption without
any trouble.

Linus

2003-11-24 22:45:22

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

Mr. Torvalds,

--- Linus Torvalds <[email protected]> wrote:
>
>
> On Mon, 24 Nov 2003, Bradley Chapman wrote:
> >
> > What sort of information would you like me to provide, sir? The bug you're
> > discussing here isn't affecting me; CONFIG_PREEMPT has been solid on
> 2.6.0-test10.
> > This is on a Gateway 600S laptop with a P4-M 2Ghz processor and an i845
> Brookdale
> > chipset.
>
> Basically, there's something strange going on, which _seems_ to be memory
> corruption, and seems to correlate reasonable well (but not 100%) with
> CONFIG_PREEMPT.

Ah, I see. I thought there was a definite issue with a certain subsystem that
just hadn't been fixed yet when CONFIG_PREEMPT=y.

>
> It's actually unlikely to be preemption itself that is broken: it's much
> more likely that some driver or other subsystem is broken, and preempt is
> just better at triggering it by making some race conditions much easier to
> see due to bigger windows for them to happen.
>
> The problem is finding enough of a pattern to the reports to make sense of
> what seems to be the common thread. A lot of people use preemption without
> any trouble.

Indeed. Do the same subsystems usually show the memory corruption issue with
preempt active, or does it just pop up all over the place, unpredictably?

>
> Linus

Brad

=====

__________________________________
Do you Yahoo!?
Free Pop-Up Blocker - Get it now
http://companion.yahoo.com/

2003-11-24 22:41:37

by Måns Rullgård

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

Bradley Chapman <[email protected]> writes:

> What sort of information would you like me to provide, sir? The bug
> you're discussing here isn't affecting me; CONFIG_PREEMPT has been
> solid on 2.6.0-test10. This is on a Gateway 600S laptop with a P4-M
> 2Ghz processor and an i845 Brookdale chipset.

I've been running 2.6.0-test10 for a few hours on my P4M Asus laptop.
Playing music, editing files, compiling some things, some web surfing,
no problems. Nothing unusual reported by the kernel.

--
M?ns Rullg?rd
[email protected]

2003-11-24 22:51:16

by Mike Fedyk

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Mon, Nov 24, 2003 at 11:41:33PM +0100, M?ns Rullg?rd wrote:
> I've been running 2.6.0-test10 for a few hours on my P4M Asus laptop.
> Playing music, editing files, compiling some things, some web surfing,
> no problems. Nothing unusual reported by the kernel.

Now take your .config and compare it with someone who is getting problems
with preempt, and others, and narrow it down.

2003-11-24 23:00:01

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Mon, 24 Nov 2003, Linus Torvalds wrote:

> On Mon, 24 Nov 2003, Guennadi Liakhovetski wrote:
> >
> > Well, FWIW, I'm getting 100% reproducible Oopses on __boot__ by enabling
> > preemption AND (almost) all kernel-hacking CONFIG_DEBUG_* options - see my
> > post of 21.11.2003 with subject "[OOPS] 2.6.0-test7 + preempt + hacking".
> > If required, could try to narrow it down to 1 CONFIG option.
>
> I'd love to have more info - I actually looked at your original report,
> and it's one of those "impossible" things as far as I can tell. The low
> bit of the work "pending" flag should acts as a lock on workqueues, and
> serialize access to one workqueue totally - so having it show up with a
> pending timer is "strange" to say the least. The only two ways to clear
> the "pending" timer is by running the work-queue - either for the timer to
> have gone off (for the delayed case) _or_ the timer not to have evern been
> set in the first place (for the immediate case).
>
> So more information would be wonderful.

I've got that Oops while testing my fix of the tmscsim driver for 2.6. It
was test7. The driver worked, except in that only configuration, where
I've got the reported Oops. I just tried a working configuration (with
preempt enabled) and enabled DEBUG_PAGEALLOC - it still worked... So,
unless it REALLY was an impossible bug, that somehow impossibly
disappeared again (the only thing I can say - I couldn't have written that
Oops by hand just out of my head...:-)) - it should really be a
combination of all those parameters. I am currently recompiling test10
with that configuration, will try test7 again, if this one doesn't
reproduce the bug, will report results tomorrow. If I get the Oops again -
will attach the complete .config.

Thanks
Guennadi
---
Guennadi Liakhovetski

2003-11-24 23:01:58

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Mon, 24 Nov 2003, Bradley Chapman wrote:
>
> Indeed. Do the same subsystems usually show the memory corruption issue with
> preempt active, or does it just pop up all over the place, unpredictably?

There are a few reports of "predictable" memory corruption, in the sense
that the same people tend to see the same kinds of oopses _without_ any
other signs of memory corruption (ie no random SIGSEGV's in user space
etc).

There's the magic slab corruption thing, there's a strange thread data
corruption (one person), and there's the sunrpc timer bug. All are
"impossible" bugs that would indicate a small amount of data corruption in
some core data structure.

They are hard to trigger, which makes me personally suspect some
user-after-free thing, where the bug happens only when somebody else
allocates (and uses) the entry immediately afterwards (so that the old
user overwrites stuff that just got initialized for the new user).

It's not likely to be a wild pointer: those tend to corrupt random memory,
and that in turn is a lot more likely to result in _user_ corruptions
(causing SIGSEGV's, corrupted files that magically become ok again when
re-read, etc), since 99% of all memory tends to be non-kernel data
structures.

The PAGEFREE debug option works well for page allocations, but the slab
cache is not very amenable to it. For slab debugging, it would be
wonderful if somebody made a _truly_ debugging slab allocator that didn't
use the slab cache at all, but used the page allocator (and screw the fact
that you use too much memory ;) instead.

(Sadly, some slab users actually use that stupid "initialize" crap. We
should rip it out: it's a disaster from a data cache standpoint too, since
it tends to do all the wrong things there, even though it's literally
meant to help).

Linus

2003-11-24 23:50:25

by Andi Kleen

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

Linus Torvalds <[email protected]> writes:
>
> The PAGEFREE debug option works well for page allocations, but the slab
> cache is not very amenable to it. For slab debugging, it would be
> wonderful if somebody made a _truly_ debugging slab allocator that didn't
> use the slab cache at all, but used the page allocator (and screw the fact
> that you use too much memory ;) instead.

One way to find slab corruptions qickly is to write a thrasher:
identify for which slab size the corruptions happens and then write a
small module that runs a thread that just allocates from that slab,
writes a pattern to it, sleeps a bit, and checks the pattern (allocate
multiple slabs to make it more effective). Repeat.

If you don't know the size cycle through the different cache sizes.

Me and Chris used that to track down some nasty corruptions on x86-64,
it is especially useful together with LTP which calls a lot of system
calls that could cause corruption.

-Andi

2003-11-24 23:45:55

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Mon, 24 Nov 2003, Linus Torvalds wrote:
>
> The PAGEFREE debug option works well for page allocations, but the slab
> cache is not very amenable to it. For slab debugging, it would be
> wonderful if somebody made a _truly_ debugging slab allocator that didn't
> use the slab cache at all, but used the page allocator (and screw the fact
> that you use too much memory ;) instead.

Ok, this is quite possibly the ugliest patch I've ever made, but it might
catch some of the "use-after-free" issues with slab.

IT WILL NOT WORK IN GENERAL! In particular, you'll need to have tons of
memory free for this patch to work, since it basically scrapes off all the
regular slab code, and replaces it with some really nasty crud. I was
lazy. Whatever. It means that the slab memory balancing won't work etc,
but maybe somebody else can integrate my quick hack better.

It boots for me, but I won't say anything beyond that (oh, do NOT enable
slab debugging if you want to test this out - I pretty much guarantee it
won't work with this hack. You should enable DEBUG_PAGEALLOC instead,
and see if you get anything interesting out of that..).

Linus

-----
--- 1.110/mm/slab.c Tue Oct 21 22:10:10 2003
+++ edited/mm/slab.c Mon Nov 24 15:34:23 2003
@@ -1906,6 +1906,21 @@

static inline void * __cache_alloc (kmem_cache_t *cachep, int flags)
{
+#if 1
+ void *ptr = (void*)__get_free_pages(flags, cachep->gfporder);
+ if (ptr) {
+ struct page *page = virt_to_page(ptr);
+ SET_PAGE_CACHE(page, cachep);
+ SET_PAGE_SLAB(page, 0x01020304);
+ if (cachep->ctor) {
+ unsigned long ctor_flags = SLAB_CTOR_CONSTRUCTOR;
+ if (!(flags & __GFP_WAIT))
+ ctor_flags |= SLAB_CTOR_ATOMIC;
+ cachep->ctor(ptr, cachep, ctor_flags);
+ }
+ }
+ return ptr;
+#else
unsigned long save_flags;
void* objp;
struct array_cache *ac;
@@ -1925,6 +1940,7 @@
local_irq_restore(save_flags);
objp = cache_alloc_debugcheck_after(cachep, flags, objp, __builtin_return_address(0));
return objp;
+#endif
}

/*
@@ -2043,10 +2059,20 @@
static inline void __cache_free (kmem_cache_t *cachep, void* objp)
{
struct array_cache *ac = ac_data(cachep);
+

check_irq_off();
objp = cache_free_debugcheck(cachep, objp, __builtin_return_address(0));

+#if 1
+ {
+ struct page *page = virt_to_page(objp);
+ int order = cachep->gfporder;
+ if (cachep->dtor)
+ cachep->dtor(objp, cachep, 0);
+ __free_pages(page, order);
+ }
+#else
if (likely(ac->avail < ac->limit)) {
STATS_INC_FREEHIT(cachep);
ac_entry(ac)[ac->avail++] = objp;
@@ -2056,6 +2082,7 @@
cache_flusharray(cachep, ac);
ac_entry(ac)[ac->avail++] = objp;
}
+#endif
}

/**

2003-11-25 00:01:48

by Mike Fedyk

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Tue, Nov 25, 2003 at 12:50:17AM +0100, Andi Kleen wrote:
> Linus Torvalds <[email protected]> writes:
> >
> > The PAGEFREE debug option works well for page allocations, but the slab
> > cache is not very amenable to it. For slab debugging, it would be
> > wonderful if somebody made a _truly_ debugging slab allocator that didn't
> > use the slab cache at all, but used the page allocator (and screw the fact
> > that you use too much memory ;) instead.
>
> One way to find slab corruptions qickly is to write a thrasher:
> identify for which slab size the corruptions happens and then write a
> small module that runs a thread that just allocates from that slab,
> writes a pattern to it, sleeps a bit, and checks the pattern (allocate
> multiple slabs to make it more effective). Repeat.
>
> If you don't know the size cycle through the different cache sizes.
>
> Me and Chris used that to track down some nasty corruptions on x86-64,
> it is especially useful together with LTP which calls a lot of system
> calls that could cause corruption.

Do you have your code posted anywhere, and when are you going to merge it
with LTP? ;)

2003-11-25 00:05:34

by Andi Kleen

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

> > Me and Chris used that to track down some nasty corruptions on x86-64,
> > it is especially useful together with LTP which calls a lot of system
> > calls that could cause corruption.
>
> Do you have your code posted anywhere, and when are you going to merge it
> with LTP? ;)

I wrote it always custom tailored to the problem (it is not very difficult, but
you often have to tune it a bit until it has the right frequency to find the
corruption) Don't have one here right now, sorry. Chris had a aimed to be
generic patch for 2.4 that may still be around. I don't think it would fit
into current LTP because it is an kernel module (LTP doesn't have a kernel build
infrastructure right now). But it would be an useful addition longer term to it
I agree, once they support kernel modules.

-Andi

2003-11-25 00:16:45

by Chris Mason

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Mon, 2003-11-24 at 19:05, Andi Kleen wrote:
> > > Me and Chris used that to track down some nasty corruptions on x86-64,
> > > it is especially useful together with LTP which calls a lot of system
> > > calls that could cause corruption.
> >
> > Do you have your code posted anywhere, and when are you going to merge it
> > with LTP? ;)
>
> I wrote it always custom tailored to the problem (it is not very difficult, but
> you often have to tune it a bit until it has the right frequency to find the
> corruption) Don't have one here right now, sorry. Chris had a aimed to be
> generic patch for 2.4 that may still be around. I don't think it would fit
> into current LTP because it is an kernel module (LTP doesn't have a kernel build
> infrastructure right now). But it would be an useful addition longer term to it
> I agree, once they support kernel modules.

Here's my 2.4 patch, which was actually against the suse kernel so you
might get fuzz/rejects for Config.in and the drivers/char/Makefile

It's fairly simple, just put the slab sizes you want into the sizes
array in crasher.c

2.6 port won't be hard, if people are interested, I'll do it after
Thanksgiving.

-chris

Attachments:

crasher-2.diff (4.20 kB)

2003-11-25 01:37:50

by Måns Rullgård

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

--
M?ns Rullg?rd
[email protected]

Attachments:

(No filename) (457.00 B)
.config (27.77 kB)
Download all attachments

2003-11-25 06:52:50

by Jos Hulzink

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Monday 24 Nov 2003 22:32, Linus Torvalds wrote:
> Basically, there's something strange going on, which _seems_ to be memory
> corruption, and seems to correlate reasonable well (but not 100%) with
> CONFIG_PREEMPT.
>
> It's actually unlikely to be preemption itself that is broken: it's much
> more likely that some driver or other subsystem is broken, and preempt is
> just better at triggering it by making some race conditions much easier to
> see due to bigger windows for them to happen.
>
> The problem is finding enough of a pattern to the reports to make sense of
> what seems to be the common thread. A lot of people use preemption without
> any trouble.

Maybe brute force is the best way to deal with this nasty one ? I'm thinking:
It must be rather easy to make a tool that takes a .config file and an
argument "This kernel seems in trouble Yes / No". If an option is enabled,
and this kernel config crashes, you increase the likelihood of that option
(i.e. you increment a counter). If an option is enabled, and this kernel
doesn't crash. you decrement a counter. In the end, you'll end with
statistics about which kernel option is likely to cause problems.

Jos

2003-11-25 07:19:33

[permalink] [raw]

Subject: Re: What exactly are the issues with 2.6.0-test10 preempt?

On Mon, 24 Nov 2003, Linus Torvalds wrote:

> On Mon, 24 Nov 2003, Guennadi Liakhovetski wrote:
> >
> > Well, FWIW, I'm getting 100% reproducible Oopses on __boot__ by enabling
> > preemption AND (almost) all kernel-hacking CONFIG_DEBUG_* options - see my
> > post of 21.11.2003 with subject "[OOPS] 2.6.0-test7 + preempt + hacking".
> > If required, could try to narrow it down to 1 CONFIG option.
>
> I'd love to have more info - I actually looked at your original report,
> and it's one of those "impossible" things as far as I can tell. The low
> bit of the work "pending" flag should acts as a lock on workqueues, and
> serialize access to one workqueue totally - so having it show up with a
> pending timer is "strange" to say the least. The only two ways to clear
> the "pending" timer is by running the work-queue - either for the timer to
> have gone off (for the delayed case) _or_ the timer not to have evern been
> set in the first place (for the immediate case).
>
> So more information would be wonderful.

SORRY. Please, inore this report. It IS 100% reproducable - if you load
wrong (compiled without debugging) modules... Maybe you should only accept
bug-reports either without modules or with CONFIG_MODVERSIONS...

Really sorry for taking your time.

Regards
Guennadi
---
Guennadi Liakhovetski

2003-11-25 16:20:13