2006-01-23 13:50:24

by Michal Piotrowski

[permalink] [raw]
Subject: 2.6.15-rt12 bugs

Hi,

I have noticed some bugs in latest 2.6.15-rt12, here are some of them:
http://www.stardust.webpages.pl/files/rt/2.6.15-rt12/rt-dmesg
Here is my config:
http://www.stardust.webpages.pl/files/rt/2.6.15-rt12/rt-config

Regards,
Michal Piotrowski


2006-01-24 00:25:07

by Steven Rostedt

[permalink] [raw]
Subject: Re: 2.6.15-rt12 bugs

On Mon, 2006-01-23 at 14:21 +0100, Michal Piotrowski wrote:
> Hi,
>
> I have noticed some bugs in latest 2.6.15-rt12, here are some of them:
> http://www.stardust.webpages.pl/files/rt/2.6.15-rt12/rt-dmesg

The only bug I see in this is:

BUG: kstopmachine:338 task might have lost a preemption check!
[<c0103b1b>] dump_stack+0x1b/0x1f (20)
[<c0118736>] preempt_enable_no_resched+0x4a/0x50 (20)
[<c014224d>] restart_machine+0x1a/0x1d (12)
[<c0142274>] do_stop+0x24/0x65 (20)
[<c012d6de>] kthread+0x7b/0xa9 (36)
[<c01010c5>] kernel_thread_helper+0x5/0xb (135438364)


Which I'll go and take a look at. I need to turn on
CONFIG_DEBUG_PREEMPT on my SMP machine.


> Here is my config:
> http://www.stardust.webpages.pl/files/rt/2.6.15-rt12/rt-config

You might want to turn off:

CONFIG_DEBUG_STACKOVERFLOW

This dumps out the biggest stack usage. Those dumps are caused by this,
and are not bugs. It just shows you what's using the most stack, that's
all. I find it quite annoying, and just turn it off.

-- Steve

2006-01-24 01:23:49

by Steven Rostedt

[permalink] [raw]
Subject: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

Here Ingo,

The kstop_machine has a legitimate preempt_enable_no_resched.

-- Steve

Signed-off-by: Steven Rostedt <[email protected]>

Index: linux-2.6.15-rt12/kernel/stop_machine.c
===================================================================
--- linux-2.6.15-rt12.orig/kernel/stop_machine.c 2006-01-23 09:15:37.000000000 -0500
+++ linux-2.6.15-rt12/kernel/stop_machine.c 2006-01-23 19:58:26.000000000 -0500
@@ -134,7 +134,7 @@
{
stopmachine_set_state(STOPMACHINE_EXIT);
raw_local_irq_enable();
- preempt_enable_no_resched();
+ __preempt_enable_no_resched();
}

struct stop_machine_data


2006-01-24 07:45:05

by Ingo Molnar

[permalink] [raw]
Subject: Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)


* Steven Rostedt <[email protected]> wrote:

> The kstop_machine has a legitimate preempt_enable_no_resched.

thanks, applied.

Ingo

2006-01-24 13:33:12

by Michal Piotrowski

[permalink] [raw]
Subject: Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

Hi,

On 24/01/06, Steven Rostedt <[email protected]> wrote:
> Here Ingo,
>
> The kstop_machine has a legitimate preempt_enable_no_resched.
>
> -- Steve
>
> Signed-off-by: Steven Rostedt <[email protected]>
>
> Index: linux-2.6.15-rt12/kernel/stop_machine.c
> ===================================================================
> --- linux-2.6.15-rt12.orig/kernel/stop_machine.c 2006-01-23 09:15:37.000000000 -0500
> +++ linux-2.6.15-rt12/kernel/stop_machine.c 2006-01-23 19:58:26.000000000 -0500
> @@ -134,7 +134,7 @@
> {
> stopmachine_set_state(STOPMACHINE_EXIT);
> raw_local_irq_enable();
> - preempt_enable_no_resched();
> + __preempt_enable_no_resched();
> }
>
> struct stop_machine_data
>
>
>

Great thanks!

Now 2.6.15-rt14 works fine (after disabling CONFIG_DEBUG_STACKOVERFLOW).

I have noticed only one warning
WARNING: softirq-tasklet/8 changed soft IRQ-flags.
[<c0103b1b>] dump_stack+0x1b/0x1f (20)
[<c0135218>] illegal_API_call+0x41/0x46 (20)
[<c0135267>] local_irq_disable+0x1d/0x1f (8)
[<f886dbd5>] skge_extirq+0x117/0x138 [skge] (32)
[<c01202d5>] tasklet_action+0x8a/0xf1 (24)
[<c01205c1>] ksoftirqd+0x10f/0x19e (32)
[<c012d7ca>] kthread+0x7b/0xa9 (36)
[<c01010c5>] kernel_thread_helper+0x5/0xb (1047875612)
---------------------------
| preempt count: 00000000 ]
| 0-level deep critical section nesting:
----------------------------------------

------------------------------
| showing all locks held by: | (softirq-tasklet/8 [c18a97d0, 98]):
------------------------------

And problems while loading ipv6 module
Running ntpdate to synchronize clockCould not allocate 4 bytes percpu data
modprobe: FATAL: Error inserting ipv6
(/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
memory

Could not allocate 4 bytes percpu data
modprobe: FATAL: Error inserting ipv6
(/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
memory

Here is my dmesg:
http://www.stardust.webpages.pl/files/rt/2.6.15-rt14/rt-dmesg
Here is my config:
http://www.stardust.webpages.pl/files/rt/2.6.15-rt14/rt-config

Regards,
Michal Piotrowski

2006-01-24 14:19:57

by Steven Rostedt

[permalink] [raw]
Subject: Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

On Tue, 2006-01-24 at 14:33 +0100, Michal Piotrowski wrote:

> Great thanks!
>
> Now 2.6.15-rt14 works fine (after disabling CONFIG_DEBUG_STACKOVERFLOW).
>
> I have noticed only one warning
> WARNING: softirq-tasklet/8 changed soft IRQ-flags.
> [<c0103b1b>] dump_stack+0x1b/0x1f (20)
> [<c0135218>] illegal_API_call+0x41/0x46 (20)
> [<c0135267>] local_irq_disable+0x1d/0x1f (8)
> [<f886dbd5>] skge_extirq+0x117/0x138 [skge] (32)
> [<c01202d5>] tasklet_action+0x8a/0xf1 (24)
> [<c01205c1>] ksoftirqd+0x10f/0x19e (32)
> [<c012d7ca>] kthread+0x7b/0xa9 (36)
> [<c01010c5>] kernel_thread_helper+0x5/0xb (1047875612)
> ---------------------------
> | preempt count: 00000000 ]
> | 0-level deep critical section nesting:
> ----------------------------------------
>
> ------------------------------
> | showing all locks held by: | (softirq-tasklet/8 [c18a97d0, 98]):
> ------------------------------

Yeah, I've seen this too. Since it's just a warning, it's been push to
the back of the TODO list.

>
> And problems while loading ipv6 module
> Running ntpdate to synchronize clockCould not allocate 4 bytes percpu data
> modprobe: FATAL: Error inserting ipv6
> (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> memory
>
> Could not allocate 4 bytes percpu data
> modprobe: FATAL: Error inserting ipv6
> (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> memory

Is this new with the -rt14? or has this happened before. If it has
happened before, then could you tell us when it started.

Thanks,

-- Steve


2006-01-24 15:37:49

by Michal Piotrowski

[permalink] [raw]
Subject: Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

Hi,

On 24/01/06, Steven Rostedt <[email protected]> wrote:
> On Tue, 2006-01-24 at 14:33 +0100, Michal Piotrowski wrote:
[snip]
> > And problems while loading ipv6 module
> > Running ntpdate to synchronize clockCould not allocate 4 bytes percpu data
> > modprobe: FATAL: Error inserting ipv6
> > (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> > memory
> >
> > Could not allocate 4 bytes percpu data
> > modprobe: FATAL: Error inserting ipv6
> > (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> > memory
>
> Is this new with the -rt14? or has this happened before. If it has
> happened before, then could you tell us when it started.

It's very hard to track down, because earlier versions of -rt ware too
buggy for me and most of them doesn't compile/boot.

Regards,
Michal Piotrowski

2006-01-24 18:11:13

by Steven Rostedt

[permalink] [raw]
Subject: Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

On Tue, 2006-01-24 at 16:37 +0100, Michal Piotrowski wrote:
> Hi,
>
> On 24/01/06, Steven Rostedt <[email protected]> wrote:
> > On Tue, 2006-01-24 at 14:33 +0100, Michal Piotrowski wrote:
> [snip]
> > > And problems while loading ipv6 module
> > > Running ntpdate to synchronize clockCould not allocate 4 bytes percpu data
> > > modprobe: FATAL: Error inserting ipv6
> > > (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> > > memory
> > >
> > > Could not allocate 4 bytes percpu data
> > > modprobe: FATAL: Error inserting ipv6
> > > (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> > > memory
> >
> > Is this new with the -rt14? or has this happened before. If it has
> > happened before, then could you tell us when it started.
>
> It's very hard to track down, because earlier versions of -rt ware too
> buggy for me and most of them doesn't compile/boot.

The 2.6.14-rtX series was pretty stable. How early did you go back?

-- Steve


2006-01-24 21:00:33

by Michal Piotrowski

[permalink] [raw]
Subject: Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

On 24/01/06, Steven Rostedt <[email protected]> wrote:
> On Tue, 2006-01-24 at 16:37 +0100, Michal Piotrowski wrote:
[snip]
> > It's very hard to track down, because earlier versions of -rt ware too
> > buggy for me and most of them doesn't compile/boot.
>
> The 2.6.14-rtX series was pretty stable. How early did you go back?

To 2.6.15-rt1 - it doesn't compile (ipv6 module). 2.6.15-rt2 gives me
wonderful series of oops/warnings/badness on boot ;).

I will try 2.6.14-rtX.

Regards,
Michal Piotrowski

2006-01-24 21:17:39

by Steven Rostedt

[permalink] [raw]
Subject: Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

On Tue, 2006-01-24 at 22:00 +0100, Michal Piotrowski wrote:
> On 24/01/06, Steven Rostedt <[email protected]> wrote:
> > On Tue, 2006-01-24 at 16:37 +0100, Michal Piotrowski wrote:
> [snip]
> > > It's very hard to track down, because earlier versions of -rt ware too
> > > buggy for me and most of them doesn't compile/boot.
> >
> > The 2.6.14-rtX series was pretty stable. How early did you go back?
>
> To 2.6.15-rt1 - it doesn't compile (ipv6 module). 2.6.15-rt2 gives me
> wonderful series of oops/warnings/badness on boot ;).

Ingo was busy at the time getting true mutexes into the kernel. So he
was pushing the -rt stuff out before testing. So I created a tree to
handle this:

These are patches against Ingo's -rt patches, that should help make some
of your problems work.

http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt1-sr2
http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt2-sr3
http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt3-sr1
http://home.stny.rr.com/rostedt/patches/patch-2.6.15-rt4-sr2

-- Steve


2006-01-26 17:47:20

by Michal Piotrowski

[permalink] [raw]
Subject: Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

Hi,

On 24/01/06, Steven Rostedt <[email protected]> wrote:
> On Tue, 2006-01-24 at 22:00 +0100, Michal Piotrowski wrote:
> > To 2.6.15-rt1 - it doesn't compile (ipv6 module). 2.6.15-rt2 gives me
> > wonderful series of oops/warnings/badness on boot ;).
>
> Ingo was busy at the time getting true mutexes into the kernel. So he
> was pushing the -rt stuff out before testing. So I created a tree to
> handle this:
>
> These are patches against Ingo's -rt patches, that should help make some
> of your problems work.
>
> http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt1-sr2
> http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt2-sr3
> http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt3-sr1
> http://home.stny.rr.com/rostedt/patches/patch-2.6.15-rt4-sr2
>
> -- Steve
>
>
>

Sorry for late answer, but I have exams.

Unfortunately these patches doesn't solve my problems. I have tried
many rt-patches, but everything below 2.6.15-rt8 doesn't work
(compilation or boot problems). Is there any other way to track down
this bug? (any magic ipv6 debugging option? :)

Regards,
Michal Piotrowski