2011-06-22 21:38:57

by Arun Sharma

[permalink] [raw]
Subject: Using a new perf tool against an older kernel


I ran into a few strange issues today using a new perf tool (3.0-rcX)
against a 2.6.38 based kernel. Basic things seem to work:

# perf record -aR -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.443 MB perf.data (~19334 samples) ]

But:

# perf record -agR -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.558 MB perf.data (~68077 samples) ]
Can't find id 9's machine
Found 1 unknown events!

Is this an older tool processing a perf.data file generated by a more
recent tool?

If that is not the case, consider reporting to [email protected].

I couldn't tell if this is kernel/user space incompatibility or tighter
error checking in the new perf tool. Before I start digging further,
thought I'd ask if this combination is supported. The error message
above doesn't ask me:

Are you using an old kernel?

-Arun


2011-06-23 14:22:46

by David Ahern

[permalink] [raw]
Subject: Re: Using a new perf tool against an older kernel

On 06/22/2011 03:39 PM, Arun Sharma wrote:
>
> I ran into a few strange issues today using a new perf tool (3.0-rcX)
> against a 2.6.38 based kernel. Basic things seem to work:
>
> # perf record -aR -- sleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.443 MB perf.data (~19334 samples) ]
>
> But:
>
> # perf record -agR -- sleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 1.558 MB perf.data (~68077 samples) ]
> Can't find id 9's machine
> Found 1 unknown events!
>
> Is this an older tool processing a perf.data file generated by a more
> recent tool?
>
> If that is not the case, consider reporting to
> [email protected].
>
> I couldn't tell if this is kernel/user space incompatibility or tighter
> error checking in the new perf tool. Before I start digging further,
> thought I'd ask if this combination is supported. The error message
> above doesn't ask me:
>
> Are you using an old kernel?

I have not seen issues like this using newer perf userspace against
older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
testing patches).

David

>
> -Arun
>
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-perf-users" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2011-06-23 19:39:07

by Arun Sharma

[permalink] [raw]
Subject: Re: Using a new perf tool against an older kernel

On 6/23/11 7:22 AM, David Ahern wrote:

> I have not seen issues like this using newer perf userspace against
> older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
> now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
> testing patches).

I narrowed it down to PERF_SAMPLE_RAW:

perf record -ag -- sleep 1

is fine, but:

perf record -agR -- sleep 1

fails for me most of the time. The reason I needed to use the -R in the
first place is that "perf script" fails on older kernels with:

Samples do not contain timestamps.

With the newer perf, I don't get errors, but the timestamp field is
invalid. So I need to use the -R flag to get valid timestamps +
stacktraces out of "perf script".

-Arun

2011-06-23 20:02:22

by David Ahern

[permalink] [raw]
Subject: Re: Using a new perf tool against an older kernel



On 06/23/2011 01:39 PM, Arun Sharma wrote:
> On 6/23/11 7:22 AM, David Ahern wrote:
>
>> I have not seen issues like this using newer perf userspace against
>> older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
>> now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
>> testing patches).
>
> I narrowed it down to PERF_SAMPLE_RAW:
>
> perf record -ag -- sleep 1
>
> is fine, but:
>
> perf record -agR -- sleep 1
>
> fails for me most of the time. The reason I needed to use the -R in the
> first place is that "perf script" fails on older kernels with:
>
> Samples do not contain timestamps.
>
> With the newer perf, I don't get errors, but the timestamp field is
> invalid. So I need to use the -R flag to get valid timestamps +
> stacktraces out of "perf script".

That should have been fixed.

And -T on record gets the timestamps.

David

>
> -Arun

2011-06-24 00:11:10

by Frederic Weisbecker

[permalink] [raw]
Subject: Re: Using a new perf tool against an older kernel

On Thu, Jun 23, 2011 at 12:39:54PM -0700, Arun Sharma wrote:
> On 6/23/11 7:22 AM, David Ahern wrote:
>
> >I have not seen issues like this using newer perf userspace against
> >older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
> >now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
> >testing patches).
>
> I narrowed it down to PERF_SAMPLE_RAW:
>
> perf record -ag -- sleep 1
>
> is fine, but:
>
> perf record -agR -- sleep 1
>
> fails for me most of the time. The reason I needed to use the -R in
> the first place is that "perf script" fails on older kernels with:
>
> Samples do not contain timestamps.
>
> With the newer perf, I don't get errors, but the timestamp field is
> invalid. So I need to use the -R flag to get valid timestamps +
> stacktraces out of "perf script".

I'm confused, you first said it happens with new tools on older kernel.

Can you tell us which combination of kernel/user raises the error? and
which error.

Thanks!

2011-06-24 00:11:54

by Frederic Weisbecker

[permalink] [raw]
Subject: Re: Using a new perf tool against an older kernel

On Thu, Jun 23, 2011 at 02:02:15PM -0600, David Ahern wrote:
>
>
> On 06/23/2011 01:39 PM, Arun Sharma wrote:
> > On 6/23/11 7:22 AM, David Ahern wrote:
> >
> >> I have not seen issues like this using newer perf userspace against
> >> older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
> >> now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
> >> testing patches).
> >
> > I narrowed it down to PERF_SAMPLE_RAW:
> >
> > perf record -ag -- sleep 1
> >
> > is fine, but:
> >
> > perf record -agR -- sleep 1
> >
> > fails for me most of the time. The reason I needed to use the -R in the
> > first place is that "perf script" fails on older kernels with:
> >
> > Samples do not contain timestamps.
> >
> > With the newer perf, I don't get errors, but the timestamp field is
> > invalid. So I need to use the -R flag to get valid timestamps +
> > stacktraces out of "perf script".
>
> That should have been fixed.
>
> And -T on record gets the timestamps.
>
> David

Right, it would be nice to suggest that from perf script when timestamps
are not recorded.

2011-06-24 00:42:38

by Arun Sharma

[permalink] [raw]
Subject: Re: Using a new perf tool against an older kernel

On 6/23/11 5:11 PM, Frederic Weisbecker wrote:

>
> I'm confused, you first said it happens with new tools on older kernel.
>
> Can you tell us which combination of kernel/user raises the error? and
> which error.
>

perf-3.0 + 2.6.38

perf record -agR gives:

Can't find id 9's machine
Found 1 unknown events!

perf-3.0 + 2.6.32

perf record -ag
perf script

samples have bogus timestamps

perf-3.0 + 2.6.32

perf record -agR
perf script

error: Samples do not contain timestamps

perf-3.0 + any kernel

perf record -agT
perf script

is happy everywhere. Thanks!

-Arun

2011-06-24 05:07:40

by David Ahern

[permalink] [raw]
Subject: Re: Using a new perf tool against an older kernel



On 06/23/2011 06:43 PM, Arun Sharma wrote:
> On 6/23/11 5:11 PM, Frederic Weisbecker wrote:
>
>>
>> I'm confused, you first said it happens with new tools on older kernel.
>>
>> Can you tell us which combination of kernel/user raises the error? and
>> which error.
>>

I tried to repeat your examples below using perf-core tip (3.0-rc3,
af07ce3e77).

>
> perf-3.0 + 2.6.38
>
> perf record -agR gives:
>
> Can't find id 9's machine
> Found 1 unknown events!

This does not reproduce on 2.6.38.8-32.fc15.x86_64 running bare metal or
2.6.38 in a VM. If you are running bare metal try -e cpu-clock and see
if it is cycles related. I doubt it, but that should be the only
difference in the tests.

That said, I did see that message while working on the gettimeofday
patches a few weeks ago. I believe the root cause was the early mixture
of cycles and tracepoints -- something I fixed before sending out the
patches (sample_type hack as well as how the time-of-day trace events
were added to the event list).

>
> perf-3.0 + 2.6.32
>
> perf record -ag
> perf script
>
> samples have bogus timestamps

I'm surprised by this one. I tried with an older Fedora 12 VM (2.6.32.21
kernel). I don't get timestamps with perf-script and perf-report -D
shows -1 which is what I expect given that SAMPLE_TIME attribute is not
set by default. One difference here is that VM's default to cpu-clock
for the event.

>
> perf-3.0 + 2.6.32
>
> perf record -agR
> perf script
>
> error: Samples do not contain timestamps

And this one did not work out so well with the F-12 VM. It caused a
kernel panic - top line on console is perf_swevent_hrtimer (lines
scrolled off the top).

>
> perf-3.0 + any kernel
>
> perf record -agT
> perf script
>
> is happy everywhere. Thanks!

Which is what I would expect. Glad to see my version of reality apply
elsewhere. ;-)

David

>
> -Arun
>

2011-06-24 05:14:24

by David Ahern

[permalink] [raw]
Subject: Re: Using a new perf tool against an older kernel

On 06/23/2011 06:11 PM, Frederic Weisbecker wrote:
> On Thu, Jun 23, 2011 at 02:02:15PM -0600, David Ahern wrote:
>>
>>
>> On 06/23/2011 01:39 PM, Arun Sharma wrote:
>>> On 6/23/11 7:22 AM, David Ahern wrote:
>>>
>>>> I have not seen issues like this using newer perf userspace against
>>>> older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
>>>> now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
>>>> testing patches).
>>>
>>> I narrowed it down to PERF_SAMPLE_RAW:
>>>
>>> perf record -ag -- sleep 1
>>>
>>> is fine, but:
>>>
>>> perf record -agR -- sleep 1
>>>
>>> fails for me most of the time. The reason I needed to use the -R in the
>>> first place is that "perf script" fails on older kernels with:
>>>
>>> Samples do not contain timestamps.
>>>
>>> With the newer perf, I don't get errors, but the timestamp field is
>>> invalid. So I need to use the -R flag to get valid timestamps +
>>> stacktraces out of "perf script".
>>
>> That should have been fixed.
>>
>> And -T on record gets the timestamps.
>>
>> David
>
> Right, it would be nice to suggest that from perf script when timestamps
> are not recorded.

Timestamps are enabled by default, but that output option is removed if
the samples do not have timestamps. That message is generated if the
user requests timestamps (-f time) in the perf-script output and the
samples do not have timestamps, but Arun did not request that. Arun's
mileage with perf-3.0 definitely varies from what I've seen.

David