Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <38957384-2129-4440-5e82-37f8afcf23c3@meta.com>
Date:   Thu, 8 Jun 2023 21:29:17 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.11.2
Subject: Re: [PATCH bpf-next v3 1/3] bpf, x86: allow function arguments up to
 12 for TRACING
Content-Language: en-US
To:     Menglong Dong <menglong8.dong@gmail.com>
Cc:     alexei.starovoitov@gmail.com, davem@davemloft.net,
        dsahern@kernel.org, ast@kernel.org, daniel@iogearbox.net,
        andrii@kernel.org, martin.lau@linux.dev, song@kernel.org,
        yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org,
        sdf@google.com, x86@kernel.org, imagedong@tencent.com,
        benbjiang@tencent.com, netdev@vger.kernel.org, bpf@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org
References: <20230607125911.145345-1-imagedong@tencent.com>
 <20230607125911.145345-2-imagedong@tencent.com>
 <4ca27e23-b027-0e39-495b-2ba3376342cc@meta.com>
 <CADxym3a=_FF3NUG3-210GQN0JSvbcsGdYRiVwBEQzGTtqN3kVQ@mail.gmail.com>
From:   Yonghong Song <yhs@meta.com>
In-Reply-To: <CADxym3a=_FF3NUG3-210GQN0JSvbcsGdYRiVwBEQzGTtqN3kVQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RHVnV0QxQWxOZzZSVjcxdlR2K3BZM282VUljTjVib2dLZWI2UC9MZm1nQnV5?=
 =?utf-8?B?bE9UZmFROVdaRDlhUTJuci9ybC9wVmpuSk16QzMrTTg4aUpSUHVBZ2t4ZFEw?=
 =?utf-8?B?TTdqYk1MS0x3U1dEQjFxRWxYakxPYTE3V2tJdjM4dnV6NEFHZXJ5OTVQVFIw?=
 =?utf-8?B?aDZOa1Fqd09HbkE3N0E1QWFNS2c5eG5rUTBieG5sNjdSSUpYT1pVeFBEOVFm?=
 =?utf-8?B?MUY0M0VBVW1oaUNMVnM0ci82akN6bllOeklsUUx4L0ZLUmVJRTVGdFk2TlV3?=
 =?utf-8?B?eHFyc2lFVXU0QVVkQmtCMW1RdmRDMkE0Qmtabkl5aDBjcXhVeTBPS2VVRTRu?=
 =?utf-8?B?TXN6ZjNsZllNRXUxZEdDSklndEU0UnZ1RDdyZ2VMcVBWdDZRRmNlQW9Yc25F?=
 =?utf-8?B?VUdua2hpVDd5V2JXMmZGeWwyRk14Snc1akJtd2VwbVBiZXJPenBDVW1tYVNi?=
 =?utf-8?B?ajIrUFV1T3RoWUpoU3BoWis5OWVIQ1BBZm8reDJtcEVqeXY4NkU2QmtTNHgy?=
 =?utf-8?B?b3VTWFZ0VXA2ZGUxd0FjWm5UZFFEV2NsRlc2VDFXTVVmTTF2dm10WHFuakZy?=
 =?utf-8?B?UTI1bzJrTm5YU2FJS1V0OWxKQWhOYk8zbHI4aHZvYnFzbFg3V3FRVUNsVjRj?=
 =?utf-8?B?ajliS1dTRjNuc0ZZdFlzQzQ5ZnRENVZFZUtSRVdnK3U1cjFjNktxUTR5b05D?=
 =?utf-8?B?Mms3Tlp4dXVyb2YvRW1ka01WT1JqMWFPaGZIMk1uRS9uMENsODJjaWpPbmNr?=
 =?utf-8?B?L2orVldVSGZoZld1eXFKTHViMDZpdjZBVnlhRlJ3WFdENlFDUjZ0VHZnckN5?=
 =?utf-8?B?WkUrK1ZJQjlOTUpLOFJ1N2tPZEpEeG1EQ1A3REwvaFk1RU9KRG5mYU9xbXRB?=
 =?utf-8?B?VEFWYW1ialNUTkxNd1c1b1J6bGJlU3dPOW9nS1EzeUh1cVp6V1lGZ1pzWnRi?=
 =?utf-8?B?OFF0K1M5MkNlOEF2cmROOStZMDhCd09NaG9mNWNjYnB0YThFNzgzazlnRFRp?=
 =?utf-8?B?Skc2UTJDU1hEL3JvZ0J2aE9vQlBZbC95M2MrenQvaEVBTUxTQnJuYVpJdnRE?=
 =?utf-8?B?Ym54c3ZnS042U2MvRDRIQVI5NHFSWFBtZVlrNUdJRWx4SXZYMllYR3RVd0RU?=
 =?utf-8?B?R2lta3ZCazdncG9KQnRqNFFBVmN0VTBSN1NCN0EyVlU3VW9yZWNpbGcrVFZn?=
 =?utf-8?B?UmdtS2Fqdm01aWFNY05QZjFvblFwMnBkYXdUK2dWYVllVHFyTmdDZjBzR3BO?=
 =?utf-8?B?K0VmdkwvZXB0ZXh4UXF3YUlYVk5mK0ZSUWNuT05sVWo5Q0RML3k1MEw4V2tj?=
 =?utf-8?B?K1BmQlcxS1lOMWxGTDZkZzNWYjIxUjhPbTkvSHhYY2xnSFArRnAvaUR5VEYx?=
 =?utf-8?B?Zng0a3dtTmhLN204TXJ0dU9wdU52eTlXOHFacUVBbVRTZ2hwSGVmS1NoNXJJ?=
 =?utf-8?B?OGNLdEVRNFMzcjdQaUdsT0FiS01TSk54OFRhQzhPaFM1T05uaDJhc2dHR0N1?=
 =?utf-8?B?ZHJtczdXL3M1MnR3K0xKelBmbjZ3cmtNa2JkLytZbGtVWXFyRVlGYjFYdklZ?=
 =?utf-8?B?cWxWVkN2NDBoUW03TGpwNXluMXQvWnFTRmRtb0FTc2d4QUo4Z2t3OHEzaDRp?=
 =?utf-8?B?L0tOV2tQNnAxY0g2eGNFSHlDb0pwUWxacnRNNG1oL0V5dTUwZUJ5bVFQcllB?=
 =?utf-8?B?SjZIaHFJVG1UWlVrRnk0ZWh1VlpXcDNocnRSZVB6b0hSbzd4a3lNaEFJaFh3?=
 =?utf-8?B?WUlMVmt0SDJ0aE9tTk9oVngzVkRlZEROQ0NmQXkwK1lUalpuaGwvbFM0OERC?=
 =?utf-8?B?M3FhbzFmV28wUlcrbU9vR1l0cHF2VkVGaWxDUHZNdCtYMC9ucnJwYXZiUGwr?=
 =?utf-8?B?eHRVRHV4dlJRMnBZSXZHQjExNEU0dkVIVHBqT3AzcE4rM3E3ZXltVktTR3dF?=
 =?utf-8?B?R2VieWdTL2E3T2RreVB2WXNiM2hlZnp0cHROTTBnUnpxekpqQlBMQ0N6WGdl?=
 =?utf-8?B?K0R4SjNtN1BYUU1SYVViRVA2bHFnTXF4VHhRV1Nud2N0NUZLY3h3aE1xTXVs?=
 =?utf-8?B?d0lVbXIzM2RJL08wK0U1NlhRN05zSGhENUNxYWZUbkJDL05oZHgwVVNvbVJB?=
 =?utf-8?B?dzdVNC9rMDFsOHVRRWErMWFNbUpkMEl6OVZ0R1NiS291RjFvSlhWZHNyZ2R3?=
 =?utf-8?B?akE9PQ==?=
X-OriginatorOrg: meta.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a5ca161a-1eac-4fae-e81d-08db68a218cb
X-MS-Exchange-CrossTenant-AuthSource: SN6PR1501MB2064.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Jun 2023 04:29:21.4592
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: KKHMukAUjBgexhoKfUvooKCOijLQJEv37T/SnoX96CkZI+AaUAH9bxjHGP79jK4S
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR15MB5604
X-Proofpoint-GUID: puWXXHbHXaAZpCd6P7ICHNK7vl3bjClX
X-Proofpoint-ORIG-GUID: puWXXHbHXaAZpCd6P7ICHNK7vl3bjClX
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26
 definitions=2023-06-09_02,2023-06-08_01,2023-05-22_02
X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_PASS,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 6/8/23 7:12 PM, Menglong Dong wrote:
> On Fri, Jun 9, 2023 at 5:07 AM Yonghong Song <yhs@meta.com> wrote:
>>
>>
>>
>> On 6/7/23 5:59 AM, menglong8.dong@gmail.com wrote:
>>> From: Menglong Dong <imagedong@tencent.com>
>>>
>>> For now, the BPF program of type BPF_PROG_TYPE_TRACING can only be used
>>> on the kernel functions whose arguments count less than 6. This is not
>>> friendly at all, as too many functions have arguments count more than 6.
>>
>> Since you already have some statistics, maybe listed in the commit message.
>>
>>>
>>> Therefore, let's enhance it by increasing the function arguments count
>>> allowed in arch_prepare_bpf_trampoline(), for now, only x86_64.
>>>
>>> For the case that we don't need to call origin function, which means
>>> without BPF_TRAMP_F_CALL_ORIG, we need only copy the function arguments
>>> that stored in the frame of the caller to current frame. The arguments
>>> of arg6-argN are stored in "$rbp + 0x18", we need copy them to
>>> "$rbp - regs_off + (6 * 8)".
>>
>> Maybe I missed something, could you explain why it is '$rbp + 0x18'?
>>
>> In the current upstream code, we have
>>
>>           /* Generated trampoline stack layout:
>>            *
>>            * RBP + 8         [ return address  ]
>>            * RBP + 0         [ RBP             ]
>>            *
>>            * RBP - 8         [ return value    ]  BPF_TRAMP_F_CALL_ORIG or
>>            *
>> BPF_TRAMP_F_RET_FENTRY_RET flags
>>            *
>>            *                 [ reg_argN        ]  always
>>            *                 [ ...             ]
>>            * RBP - regs_off  [ reg_arg1        ]  program's ctx pointer
>>            *
>>            * RBP - nregs_off [ regs count      ]  always
>>            *
>>            * RBP - ip_off    [ traced function ]  BPF_TRAMP_F_IP_ARG flag
>>            *
>>            * RBP - run_ctx_off [ bpf_tramp_run_ctx ]
>>            */
>>
>> Next on-stack argument will be RBP + 16, right?
>>
> 
> Sorry for the confusing, it seems there should be
> some comments here.
> 
> It's not the next on-stack argument, but the next next on-stack
> argument. The call chain is:
> 
> caller -> origin call -> trampoline
> 
> So, we have to skip the "RIP" in the stack frame of "origin call",
> which means RBP + 16 + 8. To be clear, there are only 8-byte
> in the stack frame of "origin call".

Thanks. It does make sense now. So we have
   caller -> origin call -> (5 nops changed to a call) -> trampoline
          8 bytes                                    8 bytes
and inside trampoline we have 8 bytes in stack with 'push rbp'.
Yes, it would be great there is an explanation in the code.

> 
> Thanks!
> Menglong Dong
> 
> 
>>>
>>> For the case with BPF_TRAMP_F_CALL_ORIG, we need prepare the arguments
>>> in stack before call origin function, which means we need alloc extra
>>> "8 * (arg_count - 6)" memory in the top of the stack. Note, there should
>>> not be any data be pushed to the stack before call the origin function.
>>> Then, we have to store rbx with 'mov' instead of 'push'.
>>>
>>> We use EMIT3_off32() or EMIT4() for "lea" and "sub". The range of the
>>> imm in "lea" and "sub" is [-128, 127] if EMIT4() is used. Therefore,
>>> we use EMIT3_off32() instead if the imm out of the range.
>>>
>>> It works well for the FENTRY and FEXIT, I'm not sure if there are other
>>> complicated cases.
>>
>> MODIFY_RETURN is also impacted by this patch.
>>
>>>
>>> Reviewed-by: Jiang Biao <benbjiang@tencent.com>
>>> Signed-off-by: Menglong Dong <imagedong@tencent.com>
>> [...]