Hi,
I found perf-record --tail-synthesize without --overwrite breaks symbols
for perf-script, perf-report, etc. For example:
[root@]# ~/perf record -ag --tail-synthesize -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.129 MB perf.data (3531 samples) ]
[root@]# ~/perf script | head
swapper 0 [000] 1250675.051971: 1 cycles:ppp:
ffffffff81009e15 [unknown] ([unknown])
ffffffff81196b19 [unknown] ([unknown])
ffffffff81196579 [unknown] ([unknown])
ffffffff81110ca7 [unknown] ([unknown])
ffffffff81a01f4a [unknown] ([unknown])
ffffffff81a017bf [unknown] ([unknown])
ffffffff8180e17a [unknown] ([unknown])
perf-record with --overwrite does NOT have this issue.
After digging into this, I found this issue is introduced by commit
a73e24d240bc136619d382b1268f34d75c9d25ce.
Reverting this commit does fix this issue. However, on a second thought,
I feel it is probably better just drop --tail-synthesize, as it doesn't
make much sense without --overwrite. All we need is to do tail_synthesize
when --overwrite is set.
Thoughts?
Thanks,
Song
On Wed, Nov 21, 2018 at 07:45:28AM +0000, Song Liu wrote:
> Hi,
>
> I found perf-record --tail-synthesize without --overwrite breaks symbols
> for perf-script, perf-report, etc. For example:
>
> [root@]# ~/perf record -ag --tail-synthesize -- sleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 1.129 MB perf.data (3531 samples) ]
> [root@]# ~/perf script | head
> swapper 0 [000] 1250675.051971: 1 cycles:ppp:
> ffffffff81009e15 [unknown] ([unknown])
> ffffffff81196b19 [unknown] ([unknown])
> ffffffff81196579 [unknown] ([unknown])
> ffffffff81110ca7 [unknown] ([unknown])
> ffffffff81a01f4a [unknown] ([unknown])
> ffffffff81a017bf [unknown] ([unknown])
> ffffffff8180e17a [unknown] ([unknown])
>
> perf-record with --overwrite does NOT have this issue.
actualy I think that now it works properly, if you turn on
task and mmap events, you'll see the samples get resolved
right after aux events are processed:
[root@krava perf]# ./perf script --show-info --show-task-events --show-mmap-events
...
swapper 0 [007] 620566.187355: 64821 cycles:ppp:
ffffffffab0b4d7c [unknown] ([unknown])
ffffffffaba01b1f [unknown] ([unknown])
ffffffffab75d567 [unknown] ([unknown])
ffffffffab0e1f56 [unknown] ([unknown])
ffffffffab0e216f [unknown] ([unknown])
ffffffffab050dc7 [unknown] ([unknown])
ffffffffab0000d5 [unknown] ([unknown])
swapper 0 [000] 0.000000: PERF_RECORD_MMAP -1/0: [0xffffffffab000000(0x152c8000) @ 0xffffffffab000000]: x [kernel.kallsyms]_text
swapper 0 [000] 0.000000: PERF_RECORD_MMAP -1/0: [0xffffffffc02c8000(0x4000) @ 0]: x /lib/modules/4.18.17-200.fc28.x86_64/kernel/drivers/thermal/int340x_thermal/int3403_thermal.ko.xz
swapper 0 [000] 0.000000: PERF_RECORD_MMAP -1/0: [0xffffffffc02cd000(0x4000) @ 0]: x /lib/modules/4.18.17-200.fc28.x86_64/kernel/drivers/thermal/intel_pch_thermal.ko.xz
...
ext4-rsv-conver 0 [000] 0.000000: PERF_RECORD_COMM: ext4-rsv-conver:989/989
perf 20063 [005] 620566.187438: 1056712 cycles:ppp:
ffffffffab1f605d filemap_map_pages+0x9d (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
ffffffffab23aaa8 __handle_mm_fault+0x1018 (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
ffffffffab23ae7a handle_mm_fault+0xda (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
ffffffffab06ac90 __do_page_fault+0x240 (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
ffffffffab06af62 do_page_fault+0x32 (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
ffffffffaba010ee page_fault+0x1e (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
...
if you record with --overwrite and --tail-synthesize the aux events
will go to the top of the perf.data, so all's good
>
> After digging into this, I found this issue is introduced by commit
> a73e24d240bc136619d382b1268f34d75c9d25ce.
right, we got rid of that, because it will store the 'current/live'
server mmaps for report/script, which is not what you want
> Reverting this commit does fix this issue. However, on a second thought,
> I feel it is probably better just drop --tail-synthesize, as it doesn't
> make much sense without --overwrite. All we need is to do tail_synthesize
> when --overwrite is set.
>
> Thoughts?
maybe we should make --tail-synthesize non optionable for --overwrite,
and get rid of that option completely.. Wang, any idea?
but definitelly not allow that for non overwrite
thanks,
jirka
Em Wed, Nov 21, 2018 at 07:45:28AM +0000, Song Liu escreveu:
> Hi,
>
> I found perf-record --tail-synthesize without --overwrite breaks symbols
> for perf-script, perf-report, etc. For example:
>
> [root@]# ~/perf record -ag --tail-synthesize -- sleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 1.129 MB perf.data (3531 samples) ]
> [root@]# ~/perf script | head
> swapper 0 [000] 1250675.051971: 1 cycles:ppp:
> ffffffff81009e15 [unknown] ([unknown])
> ffffffff81196b19 [unknown] ([unknown])
> ffffffff81196579 [unknown] ([unknown])
> ffffffff81110ca7 [unknown] ([unknown])
> ffffffff81a01f4a [unknown] ([unknown])
> ffffffff81a017bf [unknown] ([unknown])
> ffffffff8180e17a [unknown] ([unknown])
>
> perf-record with --overwrite does NOT have this issue.
>
> After digging into this, I found this issue is introduced by commit
> a73e24d240bc136619d382b1268f34d75c9d25ce.
>
> Reverting this commit does fix this issue. However, on a second thought,
> I feel it is probably better just drop --tail-synthesize, as it doesn't
> make much sense without --overwrite. All we need is to do tail_synthesize
> when --overwrite is set.
>
> Thoughts?
Wang, wdyt?
- Arnaldo
> On Nov 21, 2018, at 2:47 AM, Jiri Olsa <[email protected]> wrote:
>
> On Wed, Nov 21, 2018 at 07:45:28AM +0000, Song Liu wrote:
>> Hi,
>>
>> I found perf-record --tail-synthesize without --overwrite breaks symbols
>> for perf-script, perf-report, etc. For example:
>>
>> [root@]# ~/perf record -ag --tail-synthesize -- sleep 1
>> [ perf record: Woken up 1 times to write data ]
>> [ perf record: Captured and wrote 1.129 MB perf.data (3531 samples) ]
>> [root@]# ~/perf script | head
>> swapper 0 [000] 1250675.051971: 1 cycles:ppp:
>> ffffffff81009e15 [unknown] ([unknown])
>> ffffffff81196b19 [unknown] ([unknown])
>> ffffffff81196579 [unknown] ([unknown])
>> ffffffff81110ca7 [unknown] ([unknown])
>> ffffffff81a01f4a [unknown] ([unknown])
>> ffffffff81a017bf [unknown] ([unknown])
>> ffffffff8180e17a [unknown] ([unknown])
>>
>> perf-record with --overwrite does NOT have this issue.
>
> actualy I think that now it works properly, if you turn on
> task and mmap events, you'll see the samples get resolved
> right after aux events are processed:
>
> [root@krava perf]# ./perf script --show-info --show-task-events --show-mmap-events
> ...
> swapper 0 [007] 620566.187355: 64821 cycles:ppp:
> ffffffffab0b4d7c [unknown] ([unknown])
> ffffffffaba01b1f [unknown] ([unknown])
> ffffffffab75d567 [unknown] ([unknown])
> ffffffffab0e1f56 [unknown] ([unknown])
> ffffffffab0e216f [unknown] ([unknown])
> ffffffffab050dc7 [unknown] ([unknown])
> ffffffffab0000d5 [unknown] ([unknown])
>
> swapper 0 [000] 0.000000: PERF_RECORD_MMAP -1/0: [0xffffffffab000000(0x152c8000) @ 0xffffffffab000000]: x [kernel.kallsyms]_text
> swapper 0 [000] 0.000000: PERF_RECORD_MMAP -1/0: [0xffffffffc02c8000(0x4000) @ 0]: x /lib/modules/4.18.17-200.fc28.x86_64/kernel/drivers/thermal/int340x_thermal/int3403_thermal.ko.xz
> swapper 0 [000] 0.000000: PERF_RECORD_MMAP -1/0: [0xffffffffc02cd000(0x4000) @ 0]: x /lib/modules/4.18.17-200.fc28.x86_64/kernel/drivers/thermal/intel_pch_thermal.ko.xz
> ...
> ext4-rsv-conver 0 [000] 0.000000: PERF_RECORD_COMM: ext4-rsv-conver:989/989
> perf 20063 [005] 620566.187438: 1056712 cycles:ppp:
> ffffffffab1f605d filemap_map_pages+0x9d (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
> ffffffffab23aaa8 __handle_mm_fault+0x1018 (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
> ffffffffab23ae7a handle_mm_fault+0xda (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
> ffffffffab06ac90 __do_page_fault+0x240 (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
> ffffffffab06af62 do_page_fault+0x32 (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
> ffffffffaba010ee page_fault+0x1e (/usr/lib/debug/lib/modules/4.18.17-200.fc28.x86_64/vmlinux)
> ...
>
> if you record with --overwrite and --tail-synthesize the aux events
> will go to the top of the perf.data, so all's good
>
>>
>> After digging into this, I found this issue is introduced by commit
>> a73e24d240bc136619d382b1268f34d75c9d25ce.
>
> right, we got rid of that, because it will store the 'current/live'
> server mmaps for report/script, which is not what you want
Agreed.
>> Reverting this commit does fix this issue. However, on a second thought,
>> I feel it is probably better just drop --tail-synthesize, as it doesn't
>> make much sense without --overwrite. All we need is to do tail_synthesize
>> when --overwrite is set.
>>
>> Thoughts?
>
> maybe we should make --tail-synthesize non optionable for --overwrite,
> and get rid of that option completely.. Wang, any idea?
>
> but definitelly not allow that for non overwrite
>
--overwrite forces tail-synthesize already:
if (record.opts.overwrite)
record.opts.tail_synthesize = true;
So I guess all we need is to remove --tail-synthesize. If we all agree
with this direction, I can draft patch for it.
Thanks,
Song
On 2018/11/21 21:11, Arnaldo Carvalho de Melo wrote:
> Em Wed, Nov 21, 2018 at 07:45:28AM +0000, Song Liu escreveu:
>> Hi,
>>
>> I found perf-record --tail-synthesize without --overwrite breaks symbols
>> for perf-script, perf-report, etc. For example:
>>
>> [root@]# ~/perf record -ag --tail-synthesize -- sleep 1
>> [ perf record: Woken up 1 times to write data ]
>> [ perf record: Captured and wrote 1.129 MB perf.data (3531 samples) ]
>> [root@]# ~/perf script | head
>> swapper 0 [000] 1250675.051971: 1 cycles:ppp:
>> ffffffff81009e15 [unknown] ([unknown])
>> ffffffff81196b19 [unknown] ([unknown])
>> ffffffff81196579 [unknown] ([unknown])
>> ffffffff81110ca7 [unknown] ([unknown])
>> ffffffff81a01f4a [unknown] ([unknown])
>> ffffffff81a017bf [unknown] ([unknown])
>> ffffffff8180e17a [unknown] ([unknown])
>>
>> perf-record with --overwrite does NOT have this issue.
>>
>> After digging into this, I found this issue is introduced by commit
>> a73e24d240bc136619d382b1268f34d75c9d25ce.
>>
>> Reverting this commit does fix this issue. However, on a second thought,
>> I feel it is probably better just drop --tail-synthesize, as it doesn't
>> make much sense without --overwrite. All we need is to do tail_synthesize
>> when --overwrite is set.
>>
Some cases we use --overwrite without --tail-synthesize. How about setting
--tail-synthesize when selecting --overwrite by default, throw a warning
when --overwrite is not set and leave a --no-tail-synthesize option?
Thank you.
>> Thoughts?
>
> Wang, wdyt?
>
> - Arnaldo
>
On Thu, Nov 22, 2018 at 09:50:15AM +0800, Wangnan (F) wrote:
>
>
> On 2018/11/21 21:11, Arnaldo Carvalho de Melo wrote:
> > Em Wed, Nov 21, 2018 at 07:45:28AM +0000, Song Liu escreveu:
> >> Hi,
> >>
> >> I found perf-record --tail-synthesize without --overwrite breaks symbols
> >> for perf-script, perf-report, etc. For example:
> >>
> >> [root@]# ~/perf record -ag --tail-synthesize -- sleep 1
> >> [ perf record: Woken up 1 times to write data ]
> >> [ perf record: Captured and wrote 1.129 MB perf.data (3531 samples) ]
> >> [root@]# ~/perf script | head
> >> swapper 0 [000] 1250675.051971: 1 cycles:ppp:
> >> ffffffff81009e15 [unknown] ([unknown])
> >> ffffffff81196b19 [unknown] ([unknown])
> >> ffffffff81196579 [unknown] ([unknown])
> >> ffffffff81110ca7 [unknown] ([unknown])
> >> ffffffff81a01f4a [unknown] ([unknown])
> >> ffffffff81a017bf [unknown] ([unknown])
> >> ffffffff8180e17a [unknown] ([unknown])
> >>
> >> perf-record with --overwrite does NOT have this issue.
> >>
> >> After digging into this, I found this issue is introduced by commit
> >> a73e24d240bc136619d382b1268f34d75c9d25ce.
> >>
> >> Reverting this commit does fix this issue. However, on a second thought,
> >> I feel it is probably better just drop --tail-synthesize, as it doesn't
> >> make much sense without --overwrite. All we need is to do tail_synthesize
> >> when --overwrite is set.
> >>
>
> Some cases we use --overwrite without --tail-synthesize. How about setting
> --tail-synthesize when selecting --overwrite by default, throw a warning
> when --overwrite is not set and leave a --no-tail-synthesize option?
sounds good to me
thanks,
jirka
> On Nov 21, 2018, at 5:50 PM, Wangnan (F) <[email protected]> wrote:
>
>
>
> On 2018/11/21 21:11, Arnaldo Carvalho de Melo wrote:
>> Em Wed, Nov 21, 2018 at 07:45:28AM +0000, Song Liu escreveu:
>>> Hi,
>>>
>>> I found perf-record --tail-synthesize without --overwrite breaks symbols
>>> for perf-script, perf-report, etc. For example:
>>>
>>> [root@]# ~/perf record -ag --tail-synthesize -- sleep 1
>>> [ perf record: Woken up 1 times to write data ]
>>> [ perf record: Captured and wrote 1.129 MB perf.data (3531 samples) ]
>>> [root@]# ~/perf script | head
>>> swapper 0 [000] 1250675.051971: 1 cycles:ppp:
>>> ffffffff81009e15 [unknown] ([unknown])
>>> ffffffff81196b19 [unknown] ([unknown])
>>> ffffffff81196579 [unknown] ([unknown])
>>> ffffffff81110ca7 [unknown] ([unknown])
>>> ffffffff81a01f4a [unknown] ([unknown])
>>> ffffffff81a017bf [unknown] ([unknown])
>>> ffffffff8180e17a [unknown] ([unknown])
>>>
>>> perf-record with --overwrite does NOT have this issue.
>>>
>>> After digging into this, I found this issue is introduced by commit
>>> a73e24d240bc136619d382b1268f34d75c9d25ce.
>>>
>>> Reverting this commit does fix this issue. However, on a second thought,
>>> I feel it is probably better just drop --tail-synthesize, as it doesn't
>>> make much sense without --overwrite. All we need is to do tail_synthesize
>>> when --overwrite is set.
>>>
>
> Some cases we use --overwrite without --tail-synthesize. How about setting
> --tail-synthesize when selecting --overwrite by default, throw a warning
> when --overwrite is not set and leave a --no-tail-synthesize option?
>
--overwrite implies --tail-synthesize. So with --overwrite, it is always
tail-synthesize. Do we really need --overwrite and --no-tail-synthesize?
If not, we can probably just drop --tail-synthesize (or mark it as
deprecated)?
Thanks,
Song