LinuxLists.cc - Linux 2.6.19-rc5

[permalink] [raw]

Subject: 2.6.19-rc5: known regressions with patches

This email lists some known regressions in 2.6.19-rc5 compared to 2.6.18
with patches available.

If you find your name in the Cc header, you are either submitter of one
of the bugs, maintainer of an affectected subsystem or driver, a patch
of you caused a breakage or I'm considering you in any other way
possibly involved with one or more of these issues.

Due to the huge amount of recipients, please trim the Cc when answering.

Subject : x86_64: Fix partial page check to ensure unusable memory
is not being marked usable
References : http://lkml.org/lkml/2006/11/9/239
Submitter : Aaron Durbin <[email protected]>
Caused-By : Mel Gorman <[email protected]>
commit 5cb248abf5ab65ab543b2d5fc16c738b28031fc0
Patch : http://lkml.org/lkml/2006/11/9/239
Status : patch available

Subject : libata must be initialized earlier
References : http://ozlabs.org/pipermail/linuxppc-dev/2006-November/027945.html
Submitter : Paul Mackerras <[email protected]>
Handled-By : Brian King <[email protected]>
Patch : http://marc.theaimsgroup.com/?l=linux-ide&m=116169938407596&w=2
Status : patch available

2006-11-13 22:56:59

by Brian King

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions with patches

Adrian Bunk wrote:
> Subject : libata must be initialized earlier
> References : http://ozlabs.org/pipermail/linuxppc-dev/2006-November/027945.html
> Submitter : Paul Mackerras <[email protected]>
> Handled-By : Brian King <[email protected]>
> Patch : http://marc.theaimsgroup.com/?l=linux-ide&m=116169938407596&w=2
> Status : patch available

I just resubmitted this patch a few minutes ago.

Brian

--
Brian King
eServer Storage I/O
IBM Linux Technology Center

2006-11-13 23:19:50

by Linus Torvalds

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions with patches

On Mon, 13 Nov 2006, Brian King wrote:

> Adrian Bunk wrote:
> > Subject : libata must be initialized earlier
> > References : http://ozlabs.org/pipermail/linuxppc-dev/2006-November/027945.html
> > Submitter : Paul Mackerras <[email protected]>
> > Handled-By : Brian King <[email protected]>
> > Patch : http://marc.theaimsgroup.com/?l=linux-ide&m=116169938407596&w=2
> > Status : patch available
>
> I just resubmitted this patch a few minutes ago.

I definitely want an ACK on this from Jeff - I'll take a few broken ppc64
machines any day over the worry that there might be problems elsewhere.

Jeff? Ack, Nack, or "I'll push it to you through my git tree", please..

Linus

2006-11-14 02:36:13

by Jeff Garzik

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions with patches

Linus Torvalds wrote:
>
> On Mon, 13 Nov 2006, Brian King wrote:
>
>> Adrian Bunk wrote:
>>> Subject : libata must be initialized earlier
>>> References : http://ozlabs.org/pipermail/linuxppc-dev/2006-November/027945.html
>>> Submitter : Paul Mackerras <[email protected]>
>>> Handled-By : Brian King <[email protected]>
>>> Patch : http://marc.theaimsgroup.com/?l=linux-ide&m=116169938407596&w=2
>>> Status : patch available
>> I just resubmitted this patch a few minutes ago.
>
> I definitely want an ACK on this from Jeff - I'll take a few broken ppc64
> machines any day over the worry that there might be problems elsewhere.
>
> Jeff? Ack, Nack, or "I'll push it to you through my git tree", please..

Reluctant ACK. But this whole subsys_init() mess is highly fragile, and
this is going to change again once a new dependency arises :/

Jeff

2006-11-15 10:21:18

[permalink] [raw]

Subject: 2.6.19-rc5: known regressions (v3)

This email lists some known regressions in 2.6.19-rc5 compared to 2.6.18
that are not yet fixed in Linus' tree.

If you find your name in the Cc header, you are either submitter of one
of the bugs, maintainer of an affectected subsystem or driver, a patch
of you caused a breakage or I'm considering you in any other way possibly
involved with one or more of these issues.

Due to the huge amount of recipients, please trim the Cc when answering.

Subject : PCI MSI setting corrupted during resume
References : http://bugzilla.kernel.org/show_bug.cgi?id=7479
Submitter : Stephen Hemminger <[email protected]>
Status : unknown

Subject : SMP kernel can not generate ISA irq properly
References : http://lkml.org/lkml/2006/10/22/15
http://lkml.org/lkml/2006/11/10/142
Submitter : Komuro <[email protected]>
Handled-By : "Eric W. Biederman" <[email protected]>
Ingo Molnar <[email protected]>
Status : problem is being debugged

Subject : ThinkPad R50p: boot fail with (lapic && on_battery)
References : http://lkml.org/lkml/2006/10/31/333
Submitter : Ernst Herzberg <[email protected]>
Handled-By : Len Brown <[email protected]>
Status : problem is being debugged

Subject : x86_64: Bad page state in process 'swapper'
References : http://lkml.org/lkml/2006/11/10/135
http://lkml.org/lkml/2006/11/10/208
Submitter : Andre Noll <[email protected]>
Handled-By : Andi Kleen <[email protected]>
Status : Andi is investigating

Subject : x86_64: oprofile doesn't work
References : http://lkml.org/lkml/2006/10/27/3
Submitter : Prakash Punnoor <[email protected]>
Status : unknown

Subject : unable to rip cd
References : http://lkml.org/lkml/2006/10/13/100
http://lkml.org/lkml/2006/11/8/42
Submitter : Alex Romosan <[email protected]>
Handled-By : Jens Axboe <[email protected]>
Status : Jens is investigating

Subject : can't disable OHCI wakeup via sysfs
References : http://lkml.org/lkml/2006/11/11/33
Submitter : Andrey Borzenkov <[email protected]>
Handled-By : Alan Stern <[email protected]>
Patch : http://lkml.org/lkml/2006/11/13/261
Status : patch available

2006-11-15 10:35:43

by Jens Axboe

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

On Wed, Nov 15 2006, Adrian Bunk wrote:
> Subject : unable to rip cd
> References : http://lkml.org/lkml/2006/10/13/100
> http://lkml.org/lkml/2006/11/8/42
> Submitter : Alex Romosan <[email protected]>
> Handled-By : Jens Axboe <[email protected]>
> Status : Jens is investigating

it's fixed and patched has been merged.

--
Jens Axboe

2006-11-15 10:36:05

by Eric Dumazet

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:

> Subject : x86_64: oprofile doesn't work
> References : http://lkml.org/lkml/2006/10/27/3
> Submitter : Prakash Punnoor <[email protected]>
> Status : unknown
>

I confirm a got this one too.
On a working kernel on an Opteron, we have normally 4 directories
in /dev/oprofile :

# ls -ld /dev/oprofile/?
drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/0
drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/1
drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/2
drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/3

With linux-2.6.19-rc5, the first one (0) is missing and we get 1,2,3

Maybe the 'bug' is in oprofile tools, that currently expect to find '0'

Eric

2006-11-15 10:50:43

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

> On a working kernel on an Opteron, we have normally 4 directories
> in /dev/oprofile :
>
> # ls -ld /dev/oprofile/?
> drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/0
> drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/1
> drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/2
> drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/3
>
> With linux-2.6.19-rc5, the first one (0) is missing and we get 1,2,3

That's because 0 was never available. It is used by the NMI watchdog.
The new kernel doesn't give it to oprofile anymore.

> Maybe the 'bug' is in oprofile tools, that currently expect to find '0'

Yes, it's likely a user space issue.

-Andi

2006-11-15 10:53:37

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

On Wed, Nov 15, 2006 at 11:35:05AM +0100, Jens Axboe wrote:
> On Wed, Nov 15 2006, Adrian Bunk wrote:
> > Subject : unable to rip cd
> > References : http://lkml.org/lkml/2006/10/13/100
> > http://lkml.org/lkml/2006/11/8/42
> > Submitter : Alex Romosan <[email protected]>
> > Handled-By : Jens Axboe <[email protected]>
> > Status : Jens is investigating
>
> it's fixed and patched has been merged.

Thanks for the information, I've removed it from my list.

> Jens Axboe

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

2006-11-15 11:06:25

by Brice Goglin

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

Adrian Bunk wrote:
> Subject : unable to rip cd
> References : http://lkml.org/lkml/2006/10/13/100
> http://lkml.org/lkml/2006/11/8/42
> Submitter : Alex Romosan <[email protected]>
> Handled-By : Jens Axboe <[email protected]>
> Status : Jens is investigating

I think this one is already fixed.

Brice

commit 616e8a091a035c0bd9b871695f4af191df123caa
author Jens Axboe <[email protected]> 1163437499 +0100
committer Linus Torvalds <[email protected]> 1163440020 -0800

[PATCH] Fix bad data direction in SG_IO

Contrary to what the name misleads you to believe, SG_DXFER_TO_FROM_DEV
is really just a normal read seen from the device side.

This patch fixes http://lkml.org/lkml/2006/10/13/100

2006-11-15 12:07:37

by Alan

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

> Subject : PCI MSI setting corrupted during resume
> References : http://bugzilla.kernel.org/show_bug.cgi?id=7479
> Submitter : Stephen Hemminger <[email protected]>
> Status : unknown

This is one of the minor resume problems as far as I can tell. I believe
the patches I posted for having a resume quirk run on each device if
appropriate should correctly resolve these. See the patch I sent to l/k.

There are a variety of other resume quirks we definitely require.

Alan

2006-11-15 16:08:34

by Stephen Hemminger

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

>
> Subject : PCI MSI setting corrupted during resume
> References : http://bugzilla.kernel.org/show_bug.cgi?id=7479
> Submitter : Stephen Hemminger <[email protected]>
> Status : unknown
>
Turns out this isn't a regression, it was always there. It has to do with ACPI
clearing state on resume. MSI wasn't being used the same in older kernels so
it didn't show up.

2006-11-15 16:38:13

by Eric W. Biederman

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

Stephen Hemminger <[email protected]> writes:

>>
>> Subject : PCI MSI setting corrupted during resume
>> References : http://bugzilla.kernel.org/show_bug.cgi?id=7479
>> Submitter : Stephen Hemminger <[email protected]>
>> Status : unknown
>>
> Turns out this isn't a regression, it was always there. It has to do with ACPI
> clearing state on resume. MSI wasn't being used the same in older kernels so
> it didn't show up.

Ok. Do we know enough to fix the MSI case?

Eric

2006-11-15 16:42:59

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

Andi Kleen wrote:
>>On a working kernel on an Opteron, we have normally 4 directories
>>in /dev/oprofile :
>>
>># ls -ld /dev/oprofile/?
>>drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/0
>>drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/1
>>drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/2
>>drwxr-xr-x 1 root root 0 15. Nov 12:38 /dev/oprofile/3
>>
>>With linux-2.6.19-rc5, the first one (0) is missing and we get 1,2,3
>
>
> That's because 0 was never available. It is used by the NMI watchdog.
> The new kernel doesn't give it to oprofile anymore.
>
>
>>Maybe the 'bug' is in oprofile tools, that currently expect to find '0'
>
>
> Yes, it's likely a user space issue.
>
> -Andi

OProfile has a simplistic view of the performance monitoring hardware. The
routines in libop/op_alloc_counter.c determine what set of performance registers
is available from the processor in use. There is no check to see what registers
are actually available in the /dev/oprofile directory.

opcontrol executes ophelp to determine which specific counters to count which
events. The function map_event_to_counter() in libop/op_alloc_counter.c does the
actual selection. It seems what is needed is for map_event_to_counter() to check
to see which counters are available and mark the others as unavailable.

-Will

2006-11-15 16:48:30

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

> OProfile has a simplistic view of the performance monitoring hardware. The
> routines in libop/op_alloc_counter.c determine what set of performance registers
> is available from the processor in use. There is no check to see what registers
> are actually available in the /dev/oprofile directory.
>
> opcontrol executes ophelp to determine which specific counters to count which
> events. The function map_event_to_counter() in libop/op_alloc_counter.c does the
> actual selection. It seems what is needed is for map_event_to_counter() to check
> to see which counters are available and mark the others as unavailable

Thanks for the explanation. Can you please fix it and release a new version?
Documentation/Changes could be adapted then.

-Andi

2006-11-15 18:44:16

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

On Wed, 15 Nov 2006 17:48:05 +0100
Andi Kleen <[email protected]> wrote:

>
> > OProfile has a simplistic view of the performance monitoring hardware. The
> > routines in libop/op_alloc_counter.c determine what set of performance registers
> > is available from the processor in use. There is no check to see what registers
> > are actually available in the /dev/oprofile directory.
> >
> > opcontrol executes ophelp to determine which specific counters to count which
> > events. The function map_event_to_counter() in libop/op_alloc_counter.c does the
> > actual selection. It seems what is needed is for map_event_to_counter() to check
> > to see which counters are available and mark the others as unavailable
>
> Thanks for the explanation. Can you please fix it and release a new version?
> Documentation/Changes could be adapted then.
>

Meanwhile we should restore the NMI counter to fix this bug.

2006-11-15 18:45:59

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

On Wednesday 15 November 2006 19:39, Andrew Morton wrote:
> On Wed, 15 Nov 2006 17:48:05 +0100
> Andi Kleen <[email protected]> wrote:
>
> >
> > > OProfile has a simplistic view of the performance monitoring hardware. The
> > > routines in libop/op_alloc_counter.c determine what set of performance registers
> > > is available from the processor in use. There is no check to see what registers
> > > are actually available in the /dev/oprofile directory.
> > >
> > > opcontrol executes ophelp to determine which specific counters to count which
> > > events. The function map_event_to_counter() in libop/op_alloc_counter.c does the
> > > actual selection. It seems what is needed is for map_event_to_counter() to check
> > > to see which counters are available and mark the others as unavailable
> >
> > Thanks for the explanation. Can you please fix it and release a new version?
> > Documentation/Changes could be adapted then.
> >
>
> Meanwhile we should restore the NMI counter to fix this bug.

No, it was always oprofile who was buggy here, silently taking
the nmi watchdog away.

-Andi

2006-11-15 19:12:28

by Linus Torvalds

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

On Wed, 15 Nov 2006, Andi Kleen wrote:
> >
> > Meanwhile we should restore the NMI counter to fix this bug.
>
> No, it was always oprofile who was buggy here, silently taking
> the nmi watchdog away.

Andi, your "blame game" doesn't matter.

The fact is, it used to work, and the kernel changed interfaces, so now it
doesn't.

In other words, a kernel interface to user land changed. THAT IS ALWAYS A
BUG. We don't change UI.

Yes, "oprofile" should be fixed to not depend on that, but the kernel
shouldn't change the interfaces, and we should add back the zero entry.

Linus

2006-11-15 19:24:03

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

> The fact is, it used to work, and the kernel changed interfaces, so now it
> doesn't.

No, it didn't work. oprofile may have done something, but it
just silently killed the NMI watchdog in the process.
That was never acceptable.

Now we do proper accounting of NMI sources and also proper allocation
of performance counters.

> Yes, "oprofile" should be fixed to not depend on that, but the kernel
> shouldn't change the interfaces, and we should add back the zero entry.

That would break the nmi watchdog again.

Anyways, there is a sysctl to disable the nmi watchdog if someone
is desperate.

But I think it is clearly oprofile who did wrong here and needs
to be fixed.

-Andi

2006-11-15 20:26:18

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

On Wed, 15 Nov 2006 20:23:53 +0100
Andi Kleen <[email protected]> wrote:

>
> > The fact is, it used to work, and the kernel changed interfaces, so now it
> > doesn't.
>
> No, it didn't work. oprofile may have done something, but it
> just silently killed the NMI watchdog in the process.
> That was never acceptable.

But people could get profiles out. I know, I've seen them!

> Now we do proper accounting of NMI sources and also proper allocation
> of performance counters.
>
>
> > Yes, "oprofile" should be fixed to not depend on that, but the kernel
> > shouldn't change the interfaces, and we should add back the zero entry.
>
> That would break the nmi watchdog again.
>
> Anyways, there is a sysctl to disable the nmi watchdog if someone
> is desperate.
>
> But I think it is clearly oprofile who did wrong here and needs
> to be fixed.
>

Is it correct to say that oprofile-on-2.6.18 works, and that
oprofile-on-2.6.19-rc5 does not?

Or is there some sort of workaround for this, or does 2.6.19-rc5 only fail
in some particular scenarios?

If it's really true that oprofile is simply busted then that's a serious
problem and we should find some way of unbusting it. If that means just
adding a dummy "0" entry which always returns zero or something like that,
then fine.

But we can't just go and bust it.

2006-11-15 21:20:31

by Eric W. Biederman

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

Andrew Morton <[email protected]> writes:

> Is it correct to say that oprofile-on-2.6.18 works, and that
> oprofile-on-2.6.19-rc5 does not?
>
> Or is there some sort of workaround for this, or does 2.6.19-rc5 only fail
> in some particular scenarios?
>
> If it's really true that oprofile is simply busted then that's a serious
> problem and we should find some way of unbusting it. If that means just
> adding a dummy "0" entry which always returns zero or something like that,
> then fine.
>
> But we can't just go and bust it.

The simple question. If we turn off the NMI watchdog on 2.6.19-rc5
does oprofile work? I believe that is what Andi said.

The description I read was a resource conflict. The resources oprofile
just expects it can used are already in use so we tell it no and
the user space oprofile doesn't cope.

Now I don't know the interface allows us to rename the interfaces
from 1 2 3 to 0 1 2. If we can then that looks like something we can
fix. Otherwise from the description I tend to agree with Andi.

The user space application assumed it own hardware that it did not.

Hmm. I bet if nothing else we could move the NMI watchdog from 0 to 3
and make things work that way...

Eric

2006-11-15 21:35:12

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

On Wed, 15 Nov 2006 14:18:24 -0700
[email protected] (Eric W. Biederman) wrote:

> Andrew Morton <[email protected]> writes:
>
> > Is it correct to say that oprofile-on-2.6.18 works, and that
> > oprofile-on-2.6.19-rc5 does not?
> >
> > Or is there some sort of workaround for this, or does 2.6.19-rc5 only fail
> > in some particular scenarios?
> >
> > If it's really true that oprofile is simply busted then that's a serious
> > problem and we should find some way of unbusting it. If that means just
> > adding a dummy "0" entry which always returns zero or something like that,
> > then fine.
> >
> > But we can't just go and bust it.
>
> The simple question. If we turn off the NMI watchdog on 2.6.19-rc5
> does oprofile work? I believe that is what Andi said.
>
> The description I read was a resource conflict. The resources oprofile
> just expects it can used are already in use so we tell it no and
> the user space oprofile doesn't cope.

That would have been a bug in earlier kernels.

> Now I don't know the interface allows us to rename the interfaces
> from 1 2 3 to 0 1 2. If we can then that looks like something we can
> fix. Otherwise from the description I tend to agree with Andi.
>
> The user space application assumed it own hardware that it did not.
>
> Hmm. I bet if nothing else we could move the NMI watchdog from 0 to 3
> and make things work that way...

Surely the appropriate behaviour is to allow oprofile to steal the NMI and
to then put the NMI back to doing the watchdog thing after oprofile has
finished with it.

If that's not a feasible thing to do for 2.6.19 then some short-term
hack which makes oprofile work again is needed.

2006-11-15 22:32:12

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

On Wed, Nov 15, 2006 at 12:06:22PM +0100, Brice Goglin wrote:
> Adrian Bunk wrote:
> > Subject : unable to rip cd
> > References : http://lkml.org/lkml/2006/10/13/100
> > http://lkml.org/lkml/2006/11/8/42
> > Submitter : Alex Romosan <[email protected]>
> > Handled-By : Jens Axboe <[email protected]>
> > Status : Jens is investigating
>
> I think this one is already fixed.

Thanks for this information (Jens already told me the same).

> Brice

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

2006-11-16 03:21:39

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

On Wed, Nov 15, 2006 at 12:21:18PM -0800, Andrew Morton wrote:
> Andi Kleen <[email protected]> wrote:
>
> >
> > > The fact is, it used to work, and the kernel changed interfaces, so now it
> > > doesn't.
> >
> > No, it didn't work. oprofile may have done something, but it
> > just silently killed the NMI watchdog in the process.
> > That was never acceptable.
>
> But people could get profiles out. I know, I've seen them!

Just the nmi watchdog was gone then.

>
> > Now we do proper accounting of NMI sources and also proper allocation
> > of performance counters.
> >
> >
> > > Yes, "oprofile" should be fixed to not depend on that, but the kernel
> > > shouldn't change the interfaces, and we should add back the zero entry.
> >
> > That would break the nmi watchdog again.
> >
> > Anyways, there is a sysctl to disable the nmi watchdog if someone
> > is desperate.
> >
> > But I think it is clearly oprofile who did wrong here and needs
> > to be fixed.
> >
>
> Is it correct to say that oprofile-on-2.6.18 works, and that
> oprofile-on-2.6.19-rc5 does not?
>
> Or is there some sort of workaround for this, or does 2.6.19-rc5 only fail

echo 0 > /proc/sys/kernel/nmi_watchdog before the oprofile module is loaded.
With builtin oprofile probably nmi_watchdog=0

> in some particular scenarios?

On x86-64 and on newer i386 machines (based on DMI year)

>
> If it's really true that oprofile is simply busted then that's a serious
> problem and we should find some way of unbusting it. If that means just
> adding a dummy "0" entry which always returns zero or something like that,
> then fine.

That could be probably done.

> But we can't just go and bust it.

It just did something unbelievable broken before. I would say it busted
itself.

-Andi

2006-11-16 05:08:54

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

On Thu, 16 Nov 2006 04:21:09 +0100
Andi Kleen <[email protected]> wrote:

> >
> > If it's really true that oprofile is simply busted then that's a serious
> > problem and we should find some way of unbusting it. If that means just
> > adding a dummy "0" entry which always returns zero or something like that,
> > then fine.
>
> That could be probably done.

I'm told that this is exactly what it was doing before it got changed.

> > But we can't just go and bust it.
>
> It just did something unbelievable broken before.

What did it do?

> I would say it busted
> itself.

It gave profiles, which was fairly handy.

2006-11-16 07:04:51

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

On Thursday 16 November 2006 06:05, Andrew Morton wrote:
> On Thu, 16 Nov 2006 04:21:09 +0100
> Andi Kleen <[email protected]> wrote:
>
> > >
> > > If it's really true that oprofile is simply busted then that's a serious
> > > problem and we should find some way of unbusting it. If that means just
> > > adding a dummy "0" entry which always returns zero or something like that,
> > > then fine.
> >
> > That could be probably done.
>
> I'm told that this is exactly what it was doing before it got changed.

Hmm, ok perhaps that can be arranged again.

The trouble is that I want to use this performance counter for
other purposes too, so we would run into trouble again
if oprofile keeps stealing it.

> > > But we can't just go and bust it.
> >
> > It just did something unbelievable broken before.
>
> What did it do?

Silently kill the nmi watchdog.

>
> > I would say it busted
> > itself.
>
> It gave profiles, which was fairly handy.

I'm sure it can be fixed there. Ok ok I keep sounding like a sysfs maintainer
now @)

-Andi

2006-11-16 11:05:20

by Mikael Pettersson

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

Andrew Morton writes:
> Surely the appropriate behaviour is to allow oprofile to steal the NMI and
> to then put the NMI back to doing the watchdog thing after oprofile has
> finished with it.

Which is _exactly_ what pre-2.6.19-rc1 kernels did. I implemented
the in-kernel API allowing real performance counter drivers like
oprofile (and perfctr) to claim the HW from the NMI watchdog,
do their work, and then release it which resumed the watchdog.

Note that oprofile (and perfctr) didn't do anything behind the
NMI watchdog's back. They went via the API. Nothing dodgy going on.

2006-11-16 15:36:36

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

Andi Kleen wrote:
> On Thursday 16 November 2006 06:05, Andrew Morton wrote:
>
>>On Thu, 16 Nov 2006 04:21:09 +0100
>>Andi Kleen <[email protected]> wrote:
>>
>>
>>>>If it's really true that oprofile is simply busted then that's a serious
>>>>problem and we should find some way of unbusting it. If that means just
>>>>adding a dummy "0" entry which always returns zero or something like that,
>>>>then fine.
>>>
>>>That could be probably done.
>>
>>I'm told that this is exactly what it was doing before it got changed.
>
>
> Hmm, ok perhaps that can be arranged again.
>
> The trouble is that I want to use this performance counter for
> other purposes too, so we would run into trouble again
> if oprofile keeps stealing it.

What other purposes do you see the performance counters useful for? To collect
information on process characteristics so they can be scheduled more efficiently?

Is this going to require sharing the nmi interrupt and knowing which perfcounter
register triggered the interrupt to get the correct action? Currently the
oprofile interrupt handler assumes any performance monitoring counter it sees
overflowing is something it should count.

-Will

2006-11-16 15:48:16

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

> What other purposes do you see the performance counters useful for?

Export one to user space as a cycle counter for benchmarking. RDTSC doesn't
do this job anymore.

> To collect information on process characteristics so they can be scheduled more efficiently?

That might happen at some point in the future, but i would expect
us to wait for CPUs with more performance counters first.

> Is this going to require sharing the nmi interrupt and knowing which perfcounter
> register triggered the interrupt to get the correct action? Currently the
> oprofile interrupt handler assumes any performance monitoring counter it sees
> overflowing is something it should count.

Yes. That needs to be fixed.

-Andi

2006-11-16 20:29:41

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

On Thu, 16 Nov 2006 11:55:46 +0100
Mikael Pettersson <[email protected]> wrote:

> Andrew Morton writes:
> > Surely the appropriate behaviour is to allow oprofile to steal the NMI and
> > to then put the NMI back to doing the watchdog thing after oprofile has
> > finished with it.
>
> Which is _exactly_ what pre-2.6.19-rc1 kernels did. I implemented
> the in-kernel API allowing real performance counter drivers like
> oprofile (and perfctr) to claim the HW from the NMI watchdog,
> do their work, and then release it which resumed the watchdog.

OK. But from Andi's comments it seems that the NMI watchdog was failing to
resume its operation.

> Note that oprofile (and perfctr) didn't do anything behind the
> NMI watchdog's back. They went via the API. Nothing dodgy going on.

2006-11-16 21:34:41

by Stephane Eranian

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

Hello,

On Thu, Nov 16, 2006 at 10:34:56AM -0500, William Cohen wrote:
>
> Is this going to require sharing the nmi interrupt and knowing which perfcounter
> register triggered the interrupt to get the correct action? Currently the
> oprofile interrupt handler assumes any performance monitoring counter it sees
> overflowing is something it should count.
>
Yes, you need to share the NMI interrupt. In my next perfmon patch you will
see that this can be made to work. You just need to add one check in the
NMI handler callback: is it for me or else try perfmon? Perfmon can auto-detect
if NMI is active and give up the right counter (there is an API to check
what is reserved). The interface propagates the list of available counters
to apps which then pass the information onto libpfm which tries to use
the remaining counters.

--
-Stephane

2006-11-17 10:08:14

by Mikael Pettersson

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

Andrew Morton writes:
> On Thu, 16 Nov 2006 11:55:46 +0100
> Mikael Pettersson <[email protected]> wrote:
>
> > Andrew Morton writes:
> > > Surely the appropriate behaviour is to allow oprofile to steal the NMI and
> > > to then put the NMI back to doing the watchdog thing after oprofile has
> > > finished with it.
> >
> > Which is _exactly_ what pre-2.6.19-rc1 kernels did. I implemented
> > the in-kernel API allowing real performance counter drivers like
> > oprofile (and perfctr) to claim the HW from the NMI watchdog,
> > do their work, and then release it which resumed the watchdog.
>
> OK. But from Andi's comments it seems that the NMI watchdog was failing to
> resume its operation.

It certainly worked when I originally implemented it. If it didn't work
that way before 2.6.19-rc1 butchered it then that would have been a bug
that should have been fixed.

2006-11-17 10:18:16

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

On Fri, 17 Nov 2006 10:59:07 +0100
Mikael Pettersson <[email protected]> wrote:

> Andrew Morton writes:
> > On Thu, 16 Nov 2006 11:55:46 +0100
> > Mikael Pettersson <[email protected]> wrote:
> >
> > > Andrew Morton writes:
> > > > Surely the appropriate behaviour is to allow oprofile to steal the NMI and
> > > > to then put the NMI back to doing the watchdog thing after oprofile has
> > > > finished with it.
> > >
> > > Which is _exactly_ what pre-2.6.19-rc1 kernels did. I implemented
> > > the in-kernel API allowing real performance counter drivers like
> > > oprofile (and perfctr) to claim the HW from the NMI watchdog,
> > > do their work, and then release it which resumed the watchdog.
> >
> > OK. But from Andi's comments it seems that the NMI watchdog was failing to
> > resume its operation.
>
> It certainly worked when I originally implemented it. If it didn't work
> that way before 2.6.19-rc1 butchered it then that would have been a bug
> that should have been fixed.

Oh. OK.

Meanwhile, 2.6.19-rc6 remains unfixed.

2006-11-17 10:30:05

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

On Friday 17 November 2006 10:59, Mikael Pettersson wrote:

> It certainly worked when I originally implemented it.

I don't think so. NMI watchdog never recovered no matter if oprofile
used the counter or not.

-Andi

2006-11-19 03:07:26

by Bill Davidsen

[permalink] [raw]

Subject: Re: [discuss] Re: 2.6.19-rc5: known regressions (v3)

Andrew Morton wrote:
> On Fri, 17 Nov 2006 10:59:07 +0100
> Mikael Pettersson <[email protected]> wrote:
>
>> Andrew Morton writes:
>> > On Thu, 16 Nov 2006 11:55:46 +0100
>> > Mikael Pettersson <[email protected]> wrote:
>> >
>> > > Andrew Morton writes:
>> > > > Surely the appropriate behaviour is to allow oprofile to steal the NMI and
>> > > > to then put the NMI back to doing the watchdog thing after oprofile has
>> > > > finished with it.
>> > >
>> > > Which is _exactly_ what pre-2.6.19-rc1 kernels did. I implemented
>> > > the in-kernel API allowing real performance counter drivers like
>> > > oprofile (and perfctr) to claim the HW from the NMI watchdog,
>> > > do their work, and then release it which resumed the watchdog.
>> >
>> > OK. But from Andi's comments it seems that the NMI watchdog was failing to
>> > resume its operation.
>>
>> It certainly worked when I originally implemented it. If it didn't work
>> that way before 2.6.19-rc1 butchered it then that would have been a bug
>> that should have been fixed.
>
> Oh. OK.
>
> Meanwhile, 2.6.19-rc6 remains unfixed.
>
Has anyone verified that nmi watchdog works at all in 2.6.19-rc6? I
haven't built a kernel since rc2, other things have been taking my time.

--
Bill Davidsen <[email protected]>
Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.

2006-11-22 10:27:51

by Eric Dumazet

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

On Wednesday 15 November 2006 11:35, Eric Dumazet wrote:
> On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:
> > Subject : x86_64: oprofile doesn't work
> > References : http://lkml.org/lkml/2006/10/27/3
> > Submitter : Prakash Punnoor <[email protected]>
> > Status : unknown
>

I hit the same problem on i386 architecture too, if CONFIG_ACPI is not set.

# opcontrol --setup --event=RESOURCE_STALLS:1000 --vmlinux=$VMFILE
# opcontrol --start
/usr/bin/opcontrol: line 911: /dev/oprofile/0/enabled: No such file or
directory
/usr/bin/opcontrol: line 911: /dev/oprofile/0/event: No such file or directory
/usr/bin/opcontrol: line 911: /dev/oprofile/0/count: No such file or directory
/usr/bin/opcontrol: line 911: /dev/oprofile/0/kernel: No such file or
directory
/usr/bin/opcontrol: line 911: /dev/oprofile/0/user: No such file or directory
/usr/bin/opcontrol: line 911: /dev/oprofile/0/unit_mask: No such file or
directory
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/oprofiled.log
Daemon started.
Profiler running.

# ls -l /dev/oprofile/
total 0
drwxr-xr-x 1 root root 0 Nov 22 11:18 1
-rw-r--r-- 1 root root 0 Nov 22 11:18 backtrace_depth
-rw-r--r-- 1 root root 0 Nov 22 11:18 buffer
-rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_size
-rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_watershed
-rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_buffer_size
-rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_type
-rw-rw-rw- 1 root root 0 Nov 22 11:18 dump
-rw-r--r-- 1 root root 0 Nov 22 11:18 enable
-rw-r--r-- 1 root root 0 Nov 22 11:18 pointer_size
drwxr-xr-x 1 root root 0 Nov 22 11:18 stats
# dmesg | grep oprofile
oprofile: using NMI interrupt.
# opcontrol --version
opcontrol: oprofile 0.9.2 compiled on Nov 22 2006 11:24:09

Eric

2006-11-22 10:42:30

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

On Wednesday 22 November 2006 11:28, Eric Dumazet wrote:
> On Wednesday 15 November 2006 11:35, Eric Dumazet wrote:
> > On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:
> > > Subject : x86_64: oprofile doesn't work
> > > References : http://lkml.org/lkml/2006/10/27/3
> > > Submitter : Prakash Punnoor <[email protected]>
> > > Status : unknown
> >
>
> I hit the same problem on i386 architecture too, if CONFIG_ACPI is not set.

oprofile is still broken because it cannot deal with the lack of perfctr 0.
You can disable the nmi watchdog as a workaround.

-Andi

2006-11-22 18:01:06

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

Eric Dumazet wrote:
> On Wednesday 15 November 2006 11:35, Eric Dumazet wrote:
>
>>On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:
>>
>>>Subject : x86_64: oprofile doesn't work
>>>References : http://lkml.org/lkml/2006/10/27/3
>>>Submitter : Prakash Punnoor <[email protected]>
>>>Status : unknown
>>
>
> I hit the same problem on i386 architecture too, if CONFIG_ACPI is not set.
>
> # opcontrol --setup --event=RESOURCE_STALLS:1000 --vmlinux=$VMFILE
> # opcontrol --start
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/enabled: No such file or
> directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/event: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/count: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/kernel: No such file or
> directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/user: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/unit_mask: No such file or
> directory
> Using 2.6+ OProfile kernel interface.
> Reading module info.
> Using log file /var/lib/oprofile/oprofiled.log
> Daemon started.
> Profiler running.
>
> # ls -l /dev/oprofile/
> total 0
> drwxr-xr-x 1 root root 0 Nov 22 11:18 1
> -rw-r--r-- 1 root root 0 Nov 22 11:18 backtrace_depth
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_size
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_watershed
> -rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_buffer_size
> -rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_type
> -rw-rw-rw- 1 root root 0 Nov 22 11:18 dump
> -rw-r--r-- 1 root root 0 Nov 22 11:18 enable
> -rw-r--r-- 1 root root 0 Nov 22 11:18 pointer_size
> drwxr-xr-x 1 root root 0 Nov 22 11:18 stats
> # dmesg | grep oprofile
> oprofile: using NMI interrupt.
> # opcontrol --version
> opcontrol: oprofile 0.9.2 compiled on Nov 22 2006 11:24:09
>
> Eric

Could you try the patch that I posted on the oprofile mailing list last week
November 17 2005 for op_allocate.c and see if that resolves the problem you are
having?

http://sourceforge.net/mailarchive/message.php?msg_id=37316102

-Will

2006-11-22 18:06:57

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

Eric Dumazet wrote:
> On Wednesday 15 November 2006 11:35, Eric Dumazet wrote:
>
>>On Wednesday 15 November 2006 11:21, Adrian Bunk wrote:
>>
>>>Subject : x86_64: oprofile doesn't work
>>>References : http://lkml.org/lkml/2006/10/27/3
>>>Submitter : Prakash Punnoor <[email protected]>
>>>Status : unknown
>>
>
> I hit the same problem on i386 architecture too, if CONFIG_ACPI is not set.
>
> # opcontrol --setup --event=RESOURCE_STALLS:1000 --vmlinux=$VMFILE
> # opcontrol --start
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/enabled: No such file or
> directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/event: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/count: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/kernel: No such file or
> directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/user: No such file or directory
> /usr/bin/opcontrol: line 911: /dev/oprofile/0/unit_mask: No such file or
> directory
> Using 2.6+ OProfile kernel interface.
> Reading module info.
> Using log file /var/lib/oprofile/oprofiled.log
> Daemon started.
> Profiler running.
>
> # ls -l /dev/oprofile/
> total 0
> drwxr-xr-x 1 root root 0 Nov 22 11:18 1
> -rw-r--r-- 1 root root 0 Nov 22 11:18 backtrace_depth
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_size
> -rw-r--r-- 1 root root 0 Nov 22 11:18 buffer_watershed
> -rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_buffer_size
> -rw-r--r-- 1 root root 0 Nov 22 11:18 cpu_type
> -rw-rw-rw- 1 root root 0 Nov 22 11:18 dump
> -rw-r--r-- 1 root root 0 Nov 22 11:18 enable
> -rw-r--r-- 1 root root 0 Nov 22 11:18 pointer_size
> drwxr-xr-x 1 root root 0 Nov 22 11:18 stats
> # dmesg | grep oprofile
> oprofile: using NMI interrupt.
> # opcontrol --version
> opcontrol: oprofile 0.9.2 compiled on Nov 22 2006 11:24:09
>
> Eric

You will also need another patch checked into the oprofile cvs last week mentioned:

http://sourceforge.net/mailarchive/message.php?msg_id=35422937

-Will

Attachments:

opalloc.diff (538.00 B)

2006-11-22 18:26:26

by Eric Dumazet

[permalink] [raw]

Subject: Re: 2.6.19-rc5: known regressions (v3)

On Wednesday 22 November 2006 19:05, William Cohen wrote:

> You will also need another patch checked into the oprofile cvs last week
> mentioned:
>
> http://sourceforge.net/mailarchive/message.php?msg_id=35422937
>
> -Will

Thank you William.

I confirm that CVS oprofile version + patches you gave here works with
linux-2.6.16-rc6 on i386, regardless of disabling nmi_watchdog (adding or
not nmi_watchdog=0 in boot params)

Eric

2006-11-22 18:47:59