2002-08-24 08:48:13

by Dag Nygren

[permalink] [raw]
Subject: Preempt note in the logs


Hi,

just installed the preempt stuff on a 2.4.19 here
running on a laptop and keep getting the
"dreaded" exited with preempt_count 1
messages.
there is modprobe,kdeinit,aplay,less,find,grep,automount
and others misbehaving.

As this is a laptop and I would like to save as much battery as
I can by keeping the disk not spinning, this behaviour will
flush the log to the disk all the time.

Would it be too much to ask if I requested that the creator of
this very useful patch would make it possible to switch this note
off through a sysctl ?

BRGDS


--
Dag Nygren email: [email protected]
Oy Espoon NewTech Ab phone: +358 9 8024910
Tr?sktorpet 3 fax: +358 9 8024916
02360 ESBO Mobile: +358 400 426312
FINLAND



2002-08-24 09:41:46

by Thunder from the hill

[permalink] [raw]
Subject: Re: Preempt note in the logs

Hi,

On Sat, 24 Aug 2002, Dag Nygren wrote:
> just installed the preempt stuff on a 2.4.19 here
> running on a laptop and keep getting the
> "dreaded" exited with preempt_count 1
> messages.
> there is modprobe,kdeinit,aplay,less,find,grep,automount
> and others misbehaving.

Not much. That's just unclean preempt.

Thunder
--
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-

2002-08-24 16:11:23

by Robert Love

[permalink] [raw]
Subject: Re: Preempt note in the logs

On Sat, 2002-08-24 at 05:45, Thunder from the hill wrote:

> Not much. That's just unclean preempt.

No, that is not just preempt being unclean.

Dan, you have a problem. Something in your kernel is doing stupid
things with locks and most of your tasks are not even preemptible.

Do you use XFS? If not, what fs?

Robert Love

2002-08-24 16:59:27

by Diego Calleja

[permalink] [raw]
Subject: Re: Preempt note in the logs

On 24 Aug 2002 12:15:32 -0400
Robert Love <[email protected]> escribi?:
> No, that is not just preempt being unclean.
>
> Dan, you have a problem. Something in your kernel is doing stupid
> things with locks and most of your tasks are not even preemptible.

I can see all those messages, too. A lot of tasks (if not all)
seems to exit with a "note: task[PID] exited with preempt_count 1"

The kernel I'm using is 2.4.19-ck2-rmap from Con Colivas
wich can be found at http://kernel.kolivas.net

It's a plain 2.4.29 +
sched-2.4.19-rc2-A4
preempt-kernel-rml-2.4.19-rc5-3.patch
2.4.19-low-latency.patch
2.4.19-rmap14a or 2.4.19-rc5aa1 05_vm_* (rmap in my case)

> Do you use XFS? If not, what fs?

Reiserfs

Diego Calleja

2002-08-24 17:08:34

by Thunder from the hill

[permalink] [raw]
Subject: Re: Preempt note in the logs

Hi,

On Sat, 24 Aug 2002, Arador wrote:
> I can see all those messages, too. A lot of tasks (if not all)
> seems to exit with a "note: task[PID] exited with preempt_count 1"

Symptoms confirmed, that is.

> > Do you use XFS? If not, what fs?
>
> Reiserfs

IMHO we can exclude reiserfs, at least the code must be clean. I'm on
2.4.19-rc5-aa1 w/reiserfs 3.6, which acts just the hell of a perfect
system. I'm serious. Of course it can be triggering.

We have to check
1. process kicking code
2. process killing code
3. memory allocation code
4. read/write code

Thunder
--
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-

2002-08-24 17:43:53

by Robert Love

[permalink] [raw]
Subject: Re: Preempt note in the logs

On Sat, 2002-08-24 at 13:12, Thunder from the hill wrote:

> We have to check
> 1. process kicking code
> 2. process killing code
> 3. memory allocation code
> 4. read/write code

The problem can be anywhere, actually. #4 (and other file I/O) is the
best bet though, I suspect.

It is caused by any mismatched locking - it is a good debugging check
regardless of preemption.

The only known place I or others see it now is on shutdown, since we
have kernel threads that do not drop the BKL and other locks when they
shutdown since the system is about to halt anyway.

There was some reports of XFS earlier... I thought they were fixed (I
talked to those guys). Maybe some other rogue module...

Robert Love

2002-08-24 17:57:10

by Thunder from the hill

[permalink] [raw]
Subject: Re: Preempt note in the logs

Hi,

On 24 Aug 2002, Robert Love wrote:
> It is caused by any mismatched locking - it is a good debugging check
> regardless of preemption.

Do you think it's useful to temporarily put a lock counter into struct
task (TEMPORARILY, Linus, temporarily!) and check that as well? Maybe that
will point us something.

Or we should extend that whole crap a bit so we could see exactly what
caused the preemption count to rise. I don't know if we can do that, but
we can try doing that.

Thunder
--
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-

2002-08-24 18:06:51

by Robert Love

[permalink] [raw]
Subject: Re: Preempt note in the logs

On Sat, 2002-08-24 at 14:01, Thunder from the hill wrote:

> Do you think it's useful to temporarily put a lock counter into struct
> task (TEMPORARILY, Linus, temporarily!) and check that as well? Maybe that
> will point us something.

There already is, it is called preempt_count.

> Or we should extend that whole crap a bit so we could see exactly what
> caused the preemption count to rise. I don't know if we can do that, but
> we can try doing that.

A lock trace would be helpful here. But since I know normal kernel code
is not suspect, all these users need to do is find which oddball module
or patch they are using... e.g. watch it be nvidia.

Robert Love

2002-08-24 18:14:13

by Thunder from the hill

[permalink] [raw]
Subject: Re: Preempt note in the logs

Hi,

On 24 Aug 2002, Robert Love wrote:
> On Sat, 2002-08-24 at 14:01, Thunder from the hill wrote:
>
> > Do you think it's useful to temporarily put a lock counter into struct
> > task (TEMPORARILY, Linus, temporarily!) and check that as well? Maybe that
> > will point us something.
>
> There already is, it is called preempt_count.

Yes, I know about this one. But you never know how many different-type
locks you'll find. In the worst case there's a kernelful of crap around,
you know.

> A lock trace would be helpful here. But since I know normal kernel code
> is not suspect, all these users need to do is find which oddball module
> or patch they are using... e.g. watch it be nvidia.

Hey, it's not all the world that is bad. I'm not just fixed on this bug
here. I'm suggesting this as a serious finder for broken things which
could be used all over the place.

(Of course my (and possibly even your) great dream is that we can find
these things before we boot the broken kernel. Good old genparsetree.pl
and a few AI might help here.)

Thunder
--
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-

2002-08-24 18:16:48

by Thunder from the hill

[permalink] [raw]
Subject: Re: Preempt note in the logs

Hi,

Hmmm... If we run a program that does neither read nor write, can the
library opens still trigger the preempt_count++ if it's really the r/w
code to blaim?

Oh no, it's raining in again!

Thunder
--
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-

2002-08-24 18:51:36

by Dag Nygren

[permalink] [raw]
Subject: Re: Preempt note in the logs



Whhops,

Lots of discussion,
but this is what I like with Linux:
lots of people that know what they are talking about
commenting/discussing things.

Anyway:
- I am not running xfs, but I do run resiserfs.
- I also have the ck2 patch that includes the low latency patch,
could that be it?

My patch got rid of the messages and my laptop behaves again,
but really I think that there is something fishy going on,
the messages shouldn't happen....

I am leaving it in your hands guys, as I don't have the experience
nor the time to go get this one.

BRGDS


--
Dag Nygren email: [email protected]
Oy Espoon NewTech Ab phone: +358 9 8024910
Tr?sktorpet 3 fax: +358 9 8024916
02360 ESBO Mobile: +358 400 426312
FINLAND


2002-08-24 21:11:39

by Thunder from the hill

[permalink] [raw]
Subject: Re: Preempt note in the logs

Hi,

On Sat, 24 Aug 2002, Dag Nygren wrote:
> Anyway:
> - I am not running xfs, but I do run resiserfs.
> - I also have the ck2 patch that includes the low latency patch,
> could that be it?

So let's be clear on that:

2.4.19+preempt + reiserfs = charm
2.4.19+preempt + ck2 = possibly no problem, dunno
2.4.19+preempt + ck2 + reiserfs = bad dude

ck2 seems to introduce a problem that reiserfs triggers. It's not a
solution to say anything against reiserfs here, never fix the symptoms,
but the root cause...

Have a look at ck2 then...

Thunder
--
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-

2002-08-24 22:59:06

by Con Kolivas

[permalink] [raw]
Subject: Re: Preempt note in the logs

Robert

Sorry you're getting involved in this. It all centres around me merging a few of
the more popular performance patches [O(1),preempt,low latency,rmap].

The preempt count in the log they're referring to only occurs frequently with my
patches so it's not due to your code; it happens with every single process using
my butchered patch. I based the merging on your latest patch for 2.4.19 and a
previous patch you had created for preempt on O(1) for 2.4.18

http://kernel.kolivas.net

I don't expect you to debug this, and I expect it's a fault in my version of
sched.c (most obvious choice).

Cheers,
Con Kolivas