Date:   Sat, 18 Feb 2023 13:30:09 -0800 (PST)
Subject:     Re: [PATCH -next V7 6/7] samples: ftrace: Add riscv support for SAMPLE_FTRACE_DIRECT[_MULTI]
In-Reply-To: <CAJF2gTR5QFGseLOJqBbi9mMAVE5jXyva7o3--mV2j=OkGtp5GQ@mail.gmail.com>
CC:     e.shatokhin@yadro.com, suagrfillet@gmail.com,
        linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org,
        anup@brainfault.org, Paul Walmsley <paul.walmsley@sifive.com>,
        Conor Dooley <conor.dooley@microchip.com>, heiko@sntech.de,
        rostedt@goodmis.org, mhiramat@kernel.org, jolsa@redhat.com,
        bp@suse.de, jpoimboe@kernel.org, andy.chiu@sifive.com,
        linux@yadro.com
From:   Palmer Dabbelt <palmer@dabbelt.com>
To:     guoren@kernel.org
Message-ID: <mhng-b3cb21cc-7fc1-41d2-8396-b9605412a293@palmer-ri-x1c9a>
Mime-Version: 1.0 (MHng)
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

On Wed, 18 Jan 2023 22:05:04 PST (-0800), guoren@kernel.org wrote:
> Thx Evgenii & Song,
>
> I got it; it would be put into v8.

Sorry if I missed it, but I don't see a v8 on patchwork.  I queued up 
the first three patches onto for-next as they seem like pretty 
independent fixes/cleanups, I'll hold off on the others until v8.

Thanks!

>
> On Wed, Jan 18, 2023 at 11:19 PM Evgenii Shatokhin
> <e.shatokhin@yadro.com> wrote:
>>
>> On 18.01.2023 05:37, Song Shuai wrote:
>> > Evgenii Shatokhin <e.shatokhin@yadro.com> 于2023年1月17日周二 16:22写道：
>> >>
>> >> On 17.01.2023 16:16, Evgenii Shatokhin wrote:
>> >>> Hi, Song,
>> >>>
>> >>> On 17.01.2023 12:32, Song Shuai wrote:
>> >>>>
>> >>>> Hi, Evgenii:
>> >>>>
>> >>>> Evgenii Shatokhin <e.shatokhin@yadro.com> 于2023年1月16日周一 14:30写道：
>> >>>>
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> On 12.01.2023 12:06, guoren@kernel.org wrote:
>> >>>>>> From: Song Shuai <suagrfillet@gmail.com>
>> >>>>>>
>> >>>>>> select HAVE_SAMPLE_FTRACE_DIRECT and HAVE_SAMPLE_FTRACE_DIRECT_MULTI
>> >>>>>> for ARCH_RV64I in arch/riscv/Kconfig. And add riscv asm code for
>> >>>>>> the ftrace-direct*.c files in samples/ftrace/.
>> >>>>>>
>> >>>>>> Signed-off-by: Song Shuai <suagrfillet@gmail.com>
>> >>>>>> Tested-by: Guo Ren <guoren@kernel.org>
>> >>>>>> Signed-off-by: Guo Ren <guoren@kernel.org>
>> >>>>>> ---
>> >>>>>>     arch/riscv/Kconfig                          |  2 ++
>> >>>>>>     samples/ftrace/ftrace-direct-modify.c       | 33 ++++++++++++++++++
>> >>>>>>     samples/ftrace/ftrace-direct-multi-modify.c | 37
>> >>>>>> +++++++++++++++++++++
>> >>>>>>     samples/ftrace/ftrace-direct-multi.c        | 22 ++++++++++++
>> >>>>>>     samples/ftrace/ftrace-direct-too.c          | 26 +++++++++++++++
>> >>>>>>     samples/ftrace/ftrace-direct.c              | 22 ++++++++++++
>> >>>>>>     6 files changed, 142 insertions(+)
>> >>>>>
>> >>>>> The samples were built OK now, but ftrace-direct-multi and
>> >>>>> ftrace-direct-multi-modify report incorrect values of ip/pc it seems.
>> >>>>>
>> >>>>> I ran 'insmod ftrace-direct-multi.ko', waited a little and then checked
>> >>>>> the messages in the trace:
>> >>>>>
>> >>>>> #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
>> >>>>> #              | |         |   |||||     |         |
>> >>>>>         migration/1-19      [001] .....  3858.532131: my_direct_func1: my
>> >>>>> direct func1 ip 0
>> >>>>>         migration/0-15      [000] d.s2.  3858.532136: my_direct_func1: my
>> >>>>> direct func1 ip ff60000001ba9600
>> >>>>>         migration/0-15      [000] d..2.  3858.532204: my_direct_func1: my
>> >>>>> direct func1 ip ff60000003334d00
>> >>>>>         migration/0-15      [000] .....  3858.532232: my_direct_func1: my
>> >>>>> direct func1 ip 0
>> >>>>>           rcu_sched-14      [001] .....  3858.532257: my_direct_func1: my
>> >>>>> direct func1 ip 0
>> >>>>>              insmod-415     [000] .....  3858.532270: my_direct_func1: my
>> >>>>> direct func1 ip 7fffffffffffffff
>> >>>>>              <idle>-0       [001] ..s1.  3858.539051: my_direct_func1: my
>> >>>>> direct func1 ip ff60000001ba9600
>> >>>>>              <idle>-0       [001] dns2.  3858.539124: my_direct_func1: my
>> >>>>> direct func1 ip ff60000001ba9600
>> >>>>>           rcu_sched-14      [001] .....  3858.539208: my_direct_func1: my
>> >>>>> direct func1 ip 0
>> >>>>> [...]
>> >>>>>
>> >>>>> If I understand it right, my_direct_func1() should print the address of
>> >>>>> some location in the code, probably - at the beginning of the traced
>> >>>>> functions.
>> >>>>>
>> >>>>> The printed values (0x0, 0x7fffffffffffffff, ...) are not valid code
>> >>>>> addresses.
>> >>>>>
>> >>>> The invalid code address is only printed by accessing the schedule()
>> >>>> function's first argument whose address stores in a0 register.
>> >>>> While schedule() actually has no parameter declared, so my_direct_func
>> >>>> just prints the a0 in the context of the schedule()'s caller and
>> >>>> the address maybe varies depending on the caller.
>> >>>>
>> >>>> I can't really understand why tracing the first argument of the
>> >>>> schedule() function, but it seems nonsense at this point.
>> >>>
>> >>> The question is, what should be passed as the argument(s) of
>> >>> my_direct_func() in this particular sample module. The kernel docs and
>> >>> commit logs seem to contain no info on that.
>> >>>
>> >>> With direct functions, I suppose, the trampoline can pass anything it
>> >>> wants to my_direct_func(), not just the arguments of the traced function.
>> >>>
>> >>> I'd check what these sample modules do on x86 and would try to match
>> >>> that behaviour on RISC-V.
>> >>
>> >> I have checked ftrace-direct-multi.ko and ftrace-direct-multi-modify.ko
>> >> on 6.2-rc4 built for x86-64 - yes, ip/pc in the traced function should
>> >> be passed to my_direct_func().
>> >>
>> >> ftrace-direct-multi.ko:
>> >> #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
>> >> #              | |         |   |||||     |         |
>> >>             insmod-10829   [000] d.h1.  1719.518535: my_direct_func: ip
>> >> ffffffff87332f45 // wake_up_process+0x5
>> >>    rcu_tasks_kthre-11      [000] .....  1719.518696: my_direct_func: ip
>> >> ffffffff8828d935 // schedule+0x5
>> >>             insmod-10829   [000] .....  1719.518708: my_direct_func: ip
>> >> ffffffff8828d935
>> >>    systemd-journal-293     [001] .....  1719.518823: my_direct_func: ip
>> >> ffffffff8828d935
>> >>            systemd-1       [000] .....  1719.519141: my_direct_func: ip
>> >> ffffffff8828d935
>> >>             <idle>-0       [001] ..s1.  1719.521889: my_direct_func: ip
>> >> ffffffff87332f45
>> >>             <idle>-0       [000] d.s2.  1719.521901: my_direct_func: ip
>> >> ffffffff87332f45
>> >>        rcu_preempt-15      [001] .....  1719.521917: my_direct_func: ip
>> >> ffffffff8828d935
>> >> [...]
>> >>
>> >> The ip values are wake_up_process+0x5 and schedule+0x5, the locations
>> >> where the execution of the traced functions resumes after the Ftrace
>> >> trampoline has finished.
>> >>
>> >> The results with ftrace-direct-multi-modify.ko are similar to that.
>> >>
>> >> The samples look like a demonstration, that one can pass anything
>> >> necessary to the handler in case of "direct" functions.
>> >>
>> >> I suppose, the RISC-V-specific asm code in these two sample modules
>> >> could be updated to pass the saved pc value to my_direct_func() in a0.
>> >
>> > Yes, you're right.
>> >
>> > I added 'mv a0,t0' in front of `call my_direct_func` to pass the address of
>> > traced function with mcount offset.
>> >
>> > Here is the updated patch for your reference.
>> > https://github.com/sugarfillet/linux/commit/95b174fb104dd970b982ee6fa19879393e229318
>>
>> Thank you for the quick fix. This one looks good to me.
>>
>> ftrace-direct-multi*.ko now report the ip values corresponding to
>> schedule+0x8 and wake_up_process+0x8, which is what was expected here.
>>
>> One more thing: please change my "Co-developed-by:" into "Tested-by:" in
>> your patch, becase this is what I actually did: tested it and reported
>> the results. I cannot take your credit for development of this patch ;-)
>>
>> Looking forward for v8 of the series.
>> >
>> >
>> >>
>> >>>
>> >>>>
>> >>>> As for this patch, it just impls a simple mcount (direct_caller) to
>> >>>> trace kernel functions, and basically saves the necessary ABI,
>> >>>> call the tracing function, and restores the ABI, just like other
>> >>>> arches do.
>> >>>> so It shouldn't be blamed.
>> >>>>
>> >>>> I started an independent patch to replace schedule with kick_process
>> >>>> to make these samples more reasonable. And It has no conflict with the
>> >>>> current patch, so we can go on.
>> >>>>
>> >>>> Link:
>> >>>> https://lore.kernel.org/linux-kernel/20230117091101.3669996-1-suagrfillet@gmail.com/T/#u
>> >>>>
>> >>>>> The same issue is with ftrace-direct-multi-modify.ko.
>> >>>>>
>> >>>>> Is anything missing here?
>> >>>>>
>> >>>>>>
>> >>>>>> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
>> >>>>>> index 307a9f413edd..e944af44f681 100644
>> >>>>>> --- a/arch/riscv/Kconfig
>> >>>>>> +++ b/arch/riscv/Kconfig
>> >>>>>> @@ -112,6 +112,8 @@ config RISCV
>> >>>>>>            select HAVE_POSIX_CPU_TIMERS_TASK_WORK
>> >>>>>>            select HAVE_REGS_AND_STACK_ACCESS_API
>> >>>>>>            select HAVE_FUNCTION_ARG_ACCESS_API
>> >>>>>> +       select HAVE_SAMPLE_FTRACE_DIRECT
>> >>>>>> +       select HAVE_SAMPLE_FTRACE_DIRECT_MULTI
>> >>>>>>            select HAVE_STACKPROTECTOR
>> >>>>>>            select HAVE_SYSCALL_TRACEPOINTS
>> >>>>>>            select HAVE_RSEQ
>> >>>>>> diff --git a/samples/ftrace/ftrace-direct-modify.c
>> >>>>>> b/samples/ftrace/ftrace-direct-modify.c
>> >>>>>> index de5a0f67f320..be7bf472c3c7 100644
>> >>>>>> --- a/samples/ftrace/ftrace-direct-modify.c
>> >>>>>> +++ b/samples/ftrace/ftrace-direct-modify.c
>> >>>>>> @@ -23,6 +23,39 @@ extern void my_tramp2(void *);
>> >>>>>>
>> >>>>>>     static unsigned long my_ip = (unsigned long)schedule;
>> >>>>>>
>> >>>>>> +#ifdef CONFIG_RISCV
>> >>>>>> +
>> >>>>>> +asm (" .pushsection    .text, \"ax\", @progbits\n"
>> >>>>>> +"      .type           my_tramp1, @function\n"
>> >>>>>> +"      .globl          my_tramp1\n"
>> >>>>>> +"   my_tramp1:\n"
>> >>>>>> +"      addi sp,sp,-16\n"
>> >>>>>> +"      sd   t0,0(sp)\n"
>> >>>>>> +"      sd   ra,8(sp)\n"
>> >>>>>> +"      call my_direct_func1\n"
>> >>>>>> +"      ld   t0,0(sp)\n"
>> >>>>>> +"      ld   ra,8(sp)\n"
>> >>>>>> +"      addi sp,sp,16\n"
>> >>>>>> +"      jr t0\n"
>> >>>>>> +"      .size           my_tramp1, .-my_tramp1\n"
>> >>>>>> +
>> >>>>>> +"      .type           my_tramp2, @function\n"
>> >>>>>> +"      .globl          my_tramp2\n"
>> >>>>>> +"   my_tramp2:\n"
>> >>>>>> +"      addi sp,sp,-16\n"
>> >>>>>> +"      sd   t0,0(sp)\n"
>> >>>>>> +"      sd   ra,8(sp)\n"
>> >>>>>> +"      call my_direct_func2\n"
>> >>>>>> +"      ld   t0,0(sp)\n"
>> >>>>>> +"      ld   ra,8(sp)\n"
>> >>>>>> +"      addi sp,sp,16\n"
>> >>>>>> +"      jr t0\n"
>> >>>>>> +"      .size           my_tramp2, .-my_tramp2\n"
>> >>>>>> +"      .popsection\n"
>> >>>>>> +);
>> >>>>>> +
>> >>>>>> +#endif /* CONFIG_RISCV */
>> >>>>>> +
>> >>>>>>     #ifdef CONFIG_X86_64
>> >>>>>>
>> >>>>>>     #include <asm/ibt.h>
>> >>>>>> diff --git a/samples/ftrace/ftrace-direct-multi-modify.c
>> >>>>>> b/samples/ftrace/ftrace-direct-multi-modify.c
>> >>>>>> index d52370cad0b6..10884bf418f7 100644
>> >>>>>> --- a/samples/ftrace/ftrace-direct-multi-modify.c
>> >>>>>> +++ b/samples/ftrace/ftrace-direct-multi-modify.c
>> >>>>>> @@ -21,6 +21,43 @@ void my_direct_func2(unsigned long ip)
>> >>>>>>     extern void my_tramp1(void *);
>> >>>>>>     extern void my_tramp2(void *);
>> >>>>>>
>> >>>>>> +#ifdef CONFIG_RISCV
>> >>>>>> +
>> >>>>>> +asm (" .pushsection    .text, \"ax\", @progbits\n"
>> >>>>>> +"      .type           my_tramp1, @function\n"
>> >>>>>> +"      .globl          my_tramp1\n"
>> >>>>>> +"   my_tramp1:\n"
>> >>>>>> +"       addi sp,sp,-24\n"
>> >>>>>> +"       sd   a0,0(sp)\n"
>> >>>>>> +"       sd   t0,8(sp)\n"
>> >>>>>> +"       sd   ra,16(sp)\n"
>> >>>>>> +"       call my_direct_func1\n"
>> >>>>>> +"       ld   a0,0(sp)\n"
>> >>>>>> +"       ld   t0,8(sp)\n"
>> >>>>>> +"       ld   ra,16(sp)\n"
>> >>>>>> +"       addi sp,sp,24\n"
>> >>>>>> +"      jr t0\n"
>> >>>>>> +"      .size           my_tramp1, .-my_tramp1\n"
>> >>>>>> +
>> >>>>>> +"      .type           my_tramp2, @function\n"
>> >>>>>> +"      .globl          my_tramp2\n"
>> >>>>>> +"   my_tramp2:\n"
>> >>>>>> +"       addi sp,sp,-24\n"
>> >>>>>> +"       sd   a0,0(sp)\n"
>> >>>>>> +"       sd   t0,8(sp)\n"
>> >>>>>> +"       sd   ra,16(sp)\n"
>> >>>>>> +"       call my_direct_func2\n"
>> >>>>>> +"       ld   a0,0(sp)\n"
>> >>>>>> +"       ld   t0,8(sp)\n"
>> >>>>>> +"       ld   ra,16(sp)\n"
>> >>>>>> +"       addi sp,sp,24\n"
>> >>>>>> +"      jr t0\n"
>> >>>>>> +"      .size           my_tramp2, .-my_tramp2\n"
>> >>>>>> +"      .popsection\n"
>> >>>>>> +);
>> >>>>>> +
>> >>>>>> +#endif /* CONFIG_RISCV */
>> >>>>>> +
>> >>>>>>     #ifdef CONFIG_X86_64
>> >>>>>>
>> >>>>>>     #include <asm/ibt.h>
>> >>>>>> diff --git a/samples/ftrace/ftrace-direct-multi.c
>> >>>>>> b/samples/ftrace/ftrace-direct-multi.c
>> >>>>>> index ec1088922517..a35bf43bf6d7 100644
>> >>>>>> --- a/samples/ftrace/ftrace-direct-multi.c
>> >>>>>> +++ b/samples/ftrace/ftrace-direct-multi.c
>> >>>>>> @@ -16,6 +16,28 @@ void my_direct_func(unsigned long ip)
>> >>>>>>
>> >>>>>>     extern void my_tramp(void *);
>> >>>>>>
>> >>>>>> +#ifdef CONFIG_RISCV
>> >>>>>> +
>> >>>>>> +asm ("       .pushsection    .text, \"ax\", @progbits\n"
>> >>>>>> +"       .type           my_tramp, @function\n"
>> >>>>>> +"       .globl          my_tramp\n"
>> >>>>>> +"   my_tramp:\n"
>> >>>>>> +"       addi sp,sp,-24\n"
>> >>>>>> +"       sd   a0,0(sp)\n"
>> >>>>>> +"       sd   t0,8(sp)\n"
>> >>>>>> +"       sd   ra,16(sp)\n"
>> >>>>>> +"       call my_direct_func\n"
>> >>>>>> +"       ld   a0,0(sp)\n"
>> >>>>>> +"       ld   t0,8(sp)\n"
>> >>>>>> +"       ld   ra,16(sp)\n"
>> >>>>>> +"       addi sp,sp,24\n"
>> >>>>>> +"       jr t0\n"
>> >>>>>> +"       .size           my_tramp, .-my_tramp\n"
>> >>>>>> +"       .popsection\n"
>> >>>>>> +);
>> >>>>>> +
>> >>>>>> +#endif /* CONFIG_RISCV */
>> >>>>>> +
>> >>>>>>     #ifdef CONFIG_X86_64
>> >>>>>>
>> >>>>>>     #include <asm/ibt.h>
>> >>>>>> diff --git a/samples/ftrace/ftrace-direct-too.c
>> >>>>>> b/samples/ftrace/ftrace-direct-too.c
>> >>>>>> index e13fb59a2b47..3b62e33c2e6d 100644
>> >>>>>> --- a/samples/ftrace/ftrace-direct-too.c
>> >>>>>> +++ b/samples/ftrace/ftrace-direct-too.c
>> >>>>>> @@ -18,6 +18,32 @@ void my_direct_func(struct vm_area_struct *vma,
>> >>>>>>
>> >>>>>>     extern void my_tramp(void *);
>> >>>>>>
>> >>>>>> +#ifdef CONFIG_RISCV
>> >>>>>> +
>> >>>>>> +asm ("       .pushsection    .text, \"ax\", @progbits\n"
>> >>>>>> +"       .type           my_tramp, @function\n"
>> >>>>>> +"       .globl          my_tramp\n"
>> >>>>>> +"   my_tramp:\n"
>> >>>>>> +"       addi sp,sp,-40\n"
>> >>>>>> +"       sd   a0,0(sp)\n"
>> >>>>>> +"       sd   a1,8(sp)\n"
>> >>>>>> +"       sd   a2,16(sp)\n"
>> >>>>>> +"       sd   t0,24(sp)\n"
>> >>>>>> +"       sd   ra,32(sp)\n"
>> >>>>>> +"       call my_direct_func\n"
>> >>>>>> +"       ld   a0,0(sp)\n"
>> >>>>>> +"       ld   a1,8(sp)\n"
>> >>>>>> +"       ld   a2,16(sp)\n"
>> >>>>>> +"       ld   t0,24(sp)\n"
>> >>>>>> +"       ld   ra,32(sp)\n"
>> >>>>>> +"       addi sp,sp,40\n"
>> >>>>>> +"       jr t0\n"
>> >>>>>> +"       .size           my_tramp, .-my_tramp\n"
>> >>>>>> +"       .popsection\n"
>> >>>>>> +);
>> >>>>>> +
>> >>>>>> +#endif /* CONFIG_RISCV */
>> >>>>>> +
>> >>>>>>     #ifdef CONFIG_X86_64
>> >>>>>>
>> >>>>>>     #include <asm/ibt.h>
>> >>>>>> diff --git a/samples/ftrace/ftrace-direct.c
>> >>>>>> b/samples/ftrace/ftrace-direct.c
>> >>>>>> index 1f769d0db20f..2cfe5a7d2d70 100644
>> >>>>>> --- a/samples/ftrace/ftrace-direct.c
>> >>>>>> +++ b/samples/ftrace/ftrace-direct.c
>> >>>>>> @@ -15,6 +15,28 @@ void my_direct_func(struct task_struct *p)
>> >>>>>>
>> >>>>>>     extern void my_tramp(void *);
>> >>>>>>
>> >>>>>> +#ifdef CONFIG_RISCV
>> >>>>>> +
>> >>>>>> +asm ("       .pushsection    .text, \"ax\", @progbits\n"
>> >>>>>> +"       .type           my_tramp, @function\n"
>> >>>>>> +"       .globl          my_tramp\n"
>> >>>>>> +"   my_tramp:\n"
>> >>>>>> +"       addi sp,sp,-24\n"
>> >>>>>> +"       sd   a0,0(sp)\n"
>> >>>>>> +"       sd   t0,8(sp)\n"
>> >>>>>> +"       sd   ra,16(sp)\n"
>> >>>>>> +"       call my_direct_func\n"
>> >>>>>> +"       ld   a0,0(sp)\n"
>> >>>>>> +"       ld   t0,8(sp)\n"
>> >>>>>> +"       ld   ra,16(sp)\n"
>> >>>>>> +"       addi sp,sp,24\n"
>> >>>>>> +"       jr t0\n"
>> >>>>>> +"       .size           my_tramp, .-my_tramp\n"
>> >>>>>> +"       .popsection\n"
>> >>>>>> +);
>> >>>>>> +
>> >>>>>> +#endif /* CONFIG_RISCV */
>> >>>>>> +
>> >>>>>>     #ifdef CONFIG_X86_64
>> >>>>>>
>> >>>>>>     #include <asm/ibt.h>
>> >>>>>> --
>> >>>>>> 2.36.1
>> >>>>
>> >>>> --
>> >>>> Thanks,
>> >>>> Song
>> >>>>
>> >>>
>> >>> Regards,
>> >>> Evgenii
>> >>
>> >>
>> >
>> >
>> > --
>> > Thanks,
>> > Song
>> >
>>
>>
>
>
> -- 
> Best Regards
>  Guo Ren