Attachments:

(No filename) (602.00 B)
config-2.6.12-rc2-mm3 (5.77 kB)
(No filename) (76.00 B)
Download all attachments

2005-04-12 02:10:48

by Benjamin Herrenschmidt

[permalink] [raw]

Subject: Re: 2.6.12-rc2-mm3

On Tue, 2005-04-12 at 03:18 +0200, Juergen Kreileder wrote:
> Andrew Morton <[email protected]> writes:
>
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.12-rc2/2.6.12-rc2-mm3/
>
> I'm getting frequent lockups on my PowerMac G5 with rc2-mm3.
>
> 2.6.11-mm4 works fine but all 2.6.12 versions I've tried (all since
> -rc1-mm3) lock up randomly. The easiest way to reproduce the problem
> seems to be running Azareus. So it might be network related, but I'm
> not 100% sure about that, there was a least one deadlock with
> virtually no network usage.
>
> BTW, what's the SysRq key on recent Apple USB keyboards? Alt/Cmd-F13
> doesn't work for me.

No idea about sysrq, i don't use it. However, I haven't experienced any
such problem with the various G5s we have here (and no other G5 user
reported such a problem).

So it would be useful if you could provide a bit more informations here
though. For example, what exact G5 model is this, do you have any 3rd
party PCI card, what video card are you using, can you reproduce the
crash in console mode, that sort of thing ...

Also, did you run a memtest equivalent on the machine ?

Finally, it would be useful if you could point out which specific patch
or bk snapshot, or at least -mm rev. introduced the bug. As I said
previously, you are the only one to report that and none of the G5s here
is showing such a problem.

Ben.

2005-04-12 05:21:03

by Benjamin Herrenschmidt

[permalink] [raw]

Hello.

do_debug() returns void, do_int3() too when
!CONFIG_KPROBES.
This patch fixes the CONFIG_KPROBES variant
of do_int3() to return void too and adjusts
the entry.S accordingly.

Signed-off-by: Stas Sergeev <[email protected]>

Attachments:

kgdbfix1.diff (1.13 kB)

2005-04-12 20:41:53

by Stas Sergeev

[permalink] [raw]

Subject: [patch 0/3] Re: 2.6.12-rc2-mm3

Hello.

Andrew Morton wrote:
> OK, the `int $3' is part of the CONFIG_TRAP_BAD_SYSCALL_EXITS thing which I
> never use.
> I'm not sure what problem is actually being reported here, now you mention it.
The problem being reported here is that
CONFIG_TRAP_BAD_SYSCALL_EXITS does nothing
but locking up your machine. Actually the
bug was so obvious, that I had real troubles
finding it (the obvious things are difficult
to spot), so I found some more bugs in a
mean time.
What was the bug? GET_THREAD_INFO(%ebp) was
missing before TI_preempt_count(%ebp), hence
the accesses beyond the stack. I'll have
troubles beleiving this code worked without
a lock-ups for someone sometimes.
I fixed it differently though. The subsequent
patches are addressing the issue.

> Yup. But are you sure that the "+ 8" is correct, given these offsets are<>
> larger than that?
I don't think they are indeed larger. The %esp
points to "struct pt_regs", which is 60 bytes
in size, and without the xss/esp = 52. Adding
8 to this gives 60, so 56(+3) looks safe to me.

> Probably it decided that some syscall got a "bad exit". Disable
> CONFIG_TRAP_BAD_SYSCALL_EXITS.
Yes, that's the fix too.

>> > - p->thread.esp0 = (unsigned long) (childregs+1) - 8;
>> > + p->thread.esp0 = (unsigned long) (childregs+1) - 15;
>> 15 is somewhat nasty - it will make the
>> stack unaligned, should better be 16 I
>> think.
> ? It's still 4-byte-aligned.
I don't see your point. Why do you think that
I substract the stack pointer by 32 bytes, for
example? I literally substract it by 8 bytes,
you propose to substract it by 15 *bytes* (not
dwords), so why would it still be aligned?
But anyway, fortunately this bug is not about
the esp0 stuff at all.

> I'm suspecting this is all due to CONFIG_TRAP_BAD_SYSCALL_EXITS taking the
> debug trap..
Sure. And that looks silly. I removed "int $3".
Patches follow. Seems to work reliable now,
but I don't know how to test it since there
seem to be no such an offending syscalls here
to test.

2005-04-12 22:46:00

by Benjamin Herrenschmidt

[permalink] [raw]

Subject: Re: 2.6.12-rc2-mm3

> >>> Hrm... I just noticed you have CONFIG_PREEMPT enabled... Can you
> >>> test without it and let me know if it makes a difference ?
> >>
> >> IIRC I had disabled that for rc2-mm2 and it didn't make a
> >> difference. I'll disable it again when I try older versions.
> >>
> >> I just got another crash with rc2-mm3. The crash was a bit
> >> different this time, I still could move the mouse pointer and the
> >> logs contained some info:
> >
> > Ok, what about non-mm ? (just plain rc2)
>
> I've tried older kernels now. rc1-mm1 locks up (no logs); plain rc1
> seems to be OK (running fine for several hours now).

Interesting. Please try -rc2 too...

Ben.

2005-04-12 22:42:05

by Martin J. Bligh

[permalink] [raw]

Subject: Re: 2.6.12-rc2-mm3

>>
> See this patch from Steve French:
> http://cifs.bkbits.net:8080/linux-2.5cifs/gnupatch@4259f2138nVCJQt3SmaZowdXd8KB7A

Thanks, that fixed it.

M.

2005-04-13 01:34:15

Hello.

This patch moves the CONFIG_TRAP_BAD_SYSCALL_EXIT
from "Executable file formats" section to the
KGDB section. I had real problems finding that
option where it was.

Signed-off-by: Stas Sergeev <[email protected]>

Attachments:

kgdbconf.diff (1.01 kB)

2005-04-13 03:23:15

by Stas Sergeev

[permalink] [raw]

Subject: Re: [patch 2/3]: entry.S trap return fixes

Hello.

Andrew Morton wrote:
>> do_debug() returns void, do_int3() too when
> This patch is applicable to the mainline kernel, is it not?
I think so - with some offsets it applies
and looks valid.

2005-04-12 19:25:21

by Juergen Kreileder

[permalink] [raw]

Subject: Re: 2.6.12-rc2-mm3

Benjamin Herrenschmidt <[email protected]> writes:

> On Tue, 2005-04-12 at 06:42 +0200, Juergen Kreileder wrote:
>> Benjamin Herrenschmidt <[email protected]> writes:
>>
>>> On Tue, 2005-04-12 at 03:18 +0200, Juergen Kreileder wrote:
>>>> Andrew Morton <[email protected]> writes:
>>>>
>>>>> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.12-rc2/2.6.12-rc2-mm3/
>>>>
>>>> I'm getting frequent lockups on my PowerMac G5 with rc2-mm3.
>>>>
>>>> 2.6.11-mm4 works fine but all 2.6.12 versions I've tried (all
>>>> since -rc1-mm3) lock up randomly. The easiest way to reproduce
>>>> the problem seems to be running Azareus. So it might be network
>>>> related, but I'm not 100% sure about that, there was a least one
>>>> deadlock with virtually no network usage.
>>>
>>> Hrm... I just noticed you have CONFIG_PREEMPT enabled... Can you
>>> test without it and let me know if it makes a difference ?
>>
>> IIRC I had disabled that for rc2-mm2 and it didn't make a
>> difference. I'll disable it again when I try older versions.
>>
>> I just got another crash with rc2-mm3. The crash was a bit
>> different this time, I still could move the mouse pointer and the
>> logs contained some info:
>
> Ok, what about non-mm ? (just plain rc2)

I've tried older kernels now. rc1-mm1 locks up (no logs); plain rc1
seems to be OK (running fine for several hours now).

Juergen

--
Juergen Kreileder, Blackdown Java-Linux Team
http://blog.blackdown.de/

2005-04-13 04:37:24

by Steven Pratt

[permalink] [raw]

Subject: Re: 2.6.12-rc2-mm3

Andrew Morton wrote:

>Nick Piggin <[email protected]> wrote:
>
>
>>>> AS basically does its own TCQ strangulation, which IIRC involves things
>>>>
>>>>
>> > > like completing all reads before issuing new writes, and completing all
>> > > reads from one process before reads from another. As well as the
>> > > fundamental way that waiting for a 'dependant read' throttles TCQ.
>> >
>> > My (mpt-fusion-based) workstation is still really slow when there's a lot
>> > of writeout happening. Just from a quick test:
>> >
>> > > 2.6.12-rc2, as, tcq depth=2: 7.241 seconds
>> > > 2.6.12-rc2, as, tcq depth=64: 12.172 seconds
>> > > 2.6.12-rc2+patch,as, tcq depth=64: 7.199 seconds
>> > > 2.6.12-rc2, cfq2, tcq depth=64: much more than 5 minutes
>> > > 2.6.12-rc2, cfq3, tcq depth=64: much more than 5 minutes
>> >
>> > 2.6.11-rc4-mm1, as, mpt-f 39.349 seconds
>> >
>> > That was really really slow but had a sudden burst of read I/O at the end
>> > which made the thing look better than it really is. I wouldn't have a clue
>> > what tag depth it's using, and it's the only mpt-fusion based machine I
>> > have handy...
>> >
>>
>> Heh.
>>
>>
>
>Well with my current lineup on the mpt-fusion driver and no
>as-limit-queue-depth.patch that test takes 17 seconds. With
>as-limit-queue-depth.patch it's down to 10 seconds. Which is pretty darn
>good btw. I assume from this:
>
>scsi0 : ioc0: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=222, IRQ=25
>scsi1 : ioc1: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=222, IRQ=26
>
>that it's using a tag depth of 222.
>
> int req_depth; /* Number of request frames */
>
>I wonder if that's true...
>
>
>One thing which changed is that this kernel now has the fixed-up mpt-fusion
>chipset tuning. That doubles the IO bandwidth, which would pretty well
>account for that difference. I'll wait and see how irritating things get
>under writeout load.
>
>Yes, we'll need to decide if we want to retain as-limit-queue-depth.patch
>and toss out some of the older AS logic which was designed to address the
>TCQ problem.
>
>Steve, could you help to identify a not-too-hard-to-set-up workload at
>which AS was particularly poor? Thanks.
>
>

AS with XFS was pretty bad on a couple of workloads. random 4k reads
and "metadata" which was 40%create, 40%append, 20%delete multithreaded
workloads. I'll try to run a few tests with and without this patch on
my hardware setup over the next day or so and see how it does. I have
not really looked at AS performance since about 2.6.6/7. Our database
team recently re-checked IO Scheduler performance, and on the Ad Hoc
Decision Support Workload we still saw a 15-20% lower throughput on
RHEL4 with AS compared to other schedulers which were all within a
couple of %.

Steve

2005-04-13 18:36:37

Andrew Morton wrote:
>
> Oleg Nesterov <[email protected]> wrote:
> >
> > ...
> > Before this timer patch del_timer(pending_timer) worked as
> > a memory barrier for the caller, now it does not.
> >
>
> I wonder if we should still do this? It would seem to be a safer approach.
>
> (This barrier stuff continues to give me the creeps. Imagine being
> dependent upon accidental barriers in the add/del/mod_timer code. Ugh).

I can't judge. But if we added barriers now, it would be
very hard to remove them later. And I think it's not good
to have these barriers "just in case".

Oleg.