Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <26c90595-f45e-a813-d538-0892c3ef2424@meta.com>
Date:   Wed, 24 May 2023 22:26:32 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.11.0
Subject: Re: [PATCH bpf-next] bpf: Export rx queue info for reuseport ebpf
 prog
Content-Language: en-US
To:     Joe Damato <jdamato@fastly.com>, bpf@vger.kernel.org
Cc:     netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
        ast@kernel.org, edumazet@google.com, martin.lau@linux.dev,
        song@kernel.org, john.fastabend@gmail.com, kpsingh@kernel.org,
        sdf@google.com, jolsa@kernel.org, haoluo@google.com
References: <20230525033757.47483-1-jdamato@fastly.com>
From:   Yonghong Song <yhs@meta.com>
In-Reply-To: <20230525033757.47483-1-jdamato@fastly.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dkNVSm9sWFdwcHBaYkRTdmVKRENvbnNBcW1URDRmU2hwNXFDRTFwVEdxU1Y3?=
 =?utf-8?B?Z3VRdDA1cWR4VXp2M3pDcC9PdVlPdlg4Z0NHN1U4TGpiMGh6NE1EdkwwSEpp?=
 =?utf-8?B?VWxvTy9PS2gzOC91S0ZFOG9oaDhlbmY4Nmp6cTI5cUdkYkMycGRTVWFIb3Jo?=
 =?utf-8?B?d2FKMUJNZFRNYm9uekcyUTRmS2NVRmpWb0hETWpBQm9HK3ZPVDRSRWN0WDFr?=
 =?utf-8?B?QlVHbzNjVjRBUlQ3SnNpcXpEK29JWUliQ1NFbDJpb2E5eGtscDlTWXl4Ni9I?=
 =?utf-8?B?TnlvQkxRK3Bna2RjdDdRb1FqVVFmOURMVUszYkphazYvNXdTcjZSN2dlVk1p?=
 =?utf-8?B?eUtwNDZhcVZYa2h3aGVUODhUa1BoZGsvOG5JV1drYjNHQitSdzZaSFRnVWtI?=
 =?utf-8?B?M2twYVVKWDYwMWo5OVYzbkdTem80U1h5TjJSZmVYaysvR0doYUFpeTdYRE9H?=
 =?utf-8?B?aDVrMi9xckJlQUxzSEU2aXV5SExMOXJHbTVEeWVXSmRPeUpDei9RcmwwQnhn?=
 =?utf-8?B?eDdTTTN5ZHRLd0svNVRYcFJmeHJBbUl5bElNOXBDOGVDNzhFbDlvaUFINXdi?=
 =?utf-8?B?ZjJzc0QxbzhvOGdQNm1nb0w3dWFKOFJZZXJabHNCNGRBdDJpUHNYc1lJZDA5?=
 =?utf-8?B?SzFacW9ONTlWb1FWdmE2V2xSRjdIWi9XK21nMEtDRHV4dlVsR0tSK1BBcHI4?=
 =?utf-8?B?bUdiaG9JejhOM3cvWVUyRXRIYmhZVjBiTVlwUzRMQ3dlOVV5NnhuM0RkbHM4?=
 =?utf-8?B?NEt2OGlpWGtQLzQ1V3d5S2hKK2RhTGtGTXVoN1FQZVRkdHE1T1YzQVd0UHZT?=
 =?utf-8?B?VXljRnZkT1hXb3F6bWgxTkhUVUYzbEpCNjRjL1JXUkE3RDI5ZnpyUUtpUzBI?=
 =?utf-8?B?blEzTjNPTGpwNmVENXM1bXB0NXR2K3lKWG82VzQ2S1Rmb3NsWWwvQ1NUSjJC?=
 =?utf-8?B?ZHVZc2pWU0dhTjdMblRzMmtWU1pMUnpoNkN6aWZoYjJQcjV4blRkZVQ4aVdF?=
 =?utf-8?B?TlFQb01ONUhlUllEZHNPdlE3NHJFTldYZHBDQkI4eGxraGpUTWp5QzduN3Ni?=
 =?utf-8?B?bk4rMmFwZnZYcGZ1N2hUTWtIeHEwTjBDSnVqUFJVanREYmhKMHFtYTArdy9p?=
 =?utf-8?B?Tkhiam0wKzlDYnBUM2VCcFBLL3hXSFhuS2p6UjlETCtSczdla1Z4ZnpuQjN4?=
 =?utf-8?B?Qkd5TSsxai96ckxzL3krOGc0Z3hOS1Vza0ZpRi9XcWhkRVZPT0c4SjNxeFhG?=
 =?utf-8?B?ZTVLVVRzUTV5K2JydGZTWERFL0hFUW55MnNxUGVmQmszVDh6NmxaL1h6aDkr?=
 =?utf-8?B?a2RnZzFjdVJ3dkxrSzdWL3ZJV0Zxc2ZBbXFWR2sxc2dIWmovVXNvRlVFamNh?=
 =?utf-8?B?cU1KOWtaSUtlRHJtTXJZSkVwQm5SSGtoMXBXNVV2dFJjQ29tUDRXUjNtUWhS?=
 =?utf-8?B?Q2FGamlTdVVXV3h4cnp4NjVQUVVkTzJweUMyN1FQU3ppWm1Pb2dYZG1UNFE0?=
 =?utf-8?B?MEJwNEw0YTF5UW4vVDBjaDFWeWxzL0VDaUdTRUxadldRY2dYOTZDZ1luQzB2?=
 =?utf-8?B?ZDFhRVR4Y0FNV0I0clNHSzd5NXBCMHBzdGFoa2NpcDVKUlI5dmVVQXFBSEIz?=
 =?utf-8?B?QXdJU2cwVWRueGYwMThQeFRoV0l4TmVPUFViSFF5V25GZmRHN1NCNjZtZWUy?=
 =?utf-8?B?OXJQVytLQWt2Q3JlQjJPVkFaL2lUQkhuU1RScVgvRVRtVWtYNXVnTEF2VEpR?=
 =?utf-8?B?Q0RRQ3Y2Y2kwRi81QVJrNlRiNnNWUmJubU9DV2Uvbnp6TVU1bUVhMFpPRlRs?=
 =?utf-8?B?bUkwNk0vWGZ0T3Z3ckd6dUVoOVozaWpPbjdkOU81aGdxMGFCenplVzVuKzhW?=
 =?utf-8?B?NE5oT2xiYkh6RjhMVnQyanV1Q25KeHdvcEYrajNabE1TcE5OTmR2WE5BT0FD?=
 =?utf-8?B?VzF2dkdwcFdrc2dkY0JlUlZyZnpCMmFFaGlScHhqYitoWGdjcEtHQnRkZUJR?=
 =?utf-8?B?R3NrbjM0MXB4MDVBNXlmb21vVzl6MnFYZ3ZheG11QXlURlBzVGhpSWVEYy9h?=
 =?utf-8?B?ZkhBUWR3ZTBBRzNEL0VRMVArbWhFSnErQTg4NUVkVkhEMjJiY1pxRGxHVkRs?=
 =?utf-8?B?eTRZQTlBSFRiNEpGZHRBZVBBRnpOMHRPY3ZhcWhoV21RQUNCNWF5cmJXMi94?=
 =?utf-8?B?R0E9PQ==?=
X-OriginatorOrg: meta.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 339db506-e3c5-4c16-ef29-08db5ce09b60
X-MS-Exchange-CrossTenant-AuthSource: SN6PR1501MB2064.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 May 2023 05:26:35.3893
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 20ixFqtgCwynoN4a2VbznZc4nzrijoM0yaBNQ8xM29x29vW5bQ+BmwiUk6hWKetF
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR15MB4030
X-Proofpoint-GUID: 75UubAPUV-fP-WX3uUPGDXUQ8s6jC3s5
X-Proofpoint-ORIG-GUID: 75UubAPUV-fP-WX3uUPGDXUQ8s6jC3s5
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26
 definitions=2023-05-25_02,2023-05-24_01,2023-05-22_02
X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_PASS,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 5/24/23 8:37 PM, Joe Damato wrote:
> BPF_PROG_TYPE_SK_REUSEPORT / sk_reuseport ebpf programs do not have
> access to the queue_mapping or napi_id of the incoming skb. Having
> this information can help ebpf progs determine which listen socket to
> select.
> 
> This patch exposes both queue_mapping and napi_id so that
> sk_reuseport ebpf programs can use this information to direct incoming
> connections to the correct listen socket in the SOCKMAP.
> 
> For example:
> 
> A multi-threaded userland program with several threads accepting client
> connections via a reuseport listen socket group might want to direct
> incoming connections from specific receive queues (or NAPI IDs) to specific
> listen sockets to maximize locality or for use with epoll busy-poll.
> 
> Signed-off-by: Joe Damato <jdamato@fastly.com>
> ---
>   include/uapi/linux/bpf.h |  2 ++
>   net/core/filter.c        | 10 ++++++++++
>   2 files changed, 12 insertions(+)
> 
> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> index 9273c654743c..31560b506535 100644
> --- a/include/uapi/linux/bpf.h
> +++ b/include/uapi/linux/bpf.h
> @@ -6286,6 +6286,8 @@ struct sk_reuseport_md {
>   	 */
>   	__u32 eth_protocol;
>   	__u32 ip_protocol;	/* IP protocol. e.g. IPPROTO_TCP, IPPROTO_UDP */
> +	__u32 rx_queue_mapping; /* Rx queue associated with the skb */
> +	__u32 napi_id;          /* napi id associated with the skb */
>   	__u32 bind_inany;	/* Is sock bound to an INANY address? */
>   	__u32 hash;		/* A hash of the packet 4 tuples */

This won't work. You will need to append to the end of data structure
to keep it backward compatibility.

Also, recent kernel has a kfunc bpf_cast_to_kern_ctx() which converts
a ctx to a kernel ctx and you can then use tracing-coding-style to
access those fields. In this particular case, you can do

    struct sk_reuseport_kern *kctx = bpf_cast_to_kern_ctx(ctx);

We have

struct sk_reuseport_kern {
         struct sk_buff *skb;
         struct sock *sk;
         struct sock *selected_sk;
         struct sock *migrating_sk;
         void *data_end;
         u32 hash;
         u32 reuseport_id;
         bool bind_inany;
};

through sk and skb pointer, you should be access the fields presented in
this patch. You can access more fields too.

So using bpf_cast_to_kern_ctx(), there is no need for more uapi changes.
Please give a try.

>   	/* When reuse->migrating_sk is NULL, it is selecting a sk for the
> diff --git a/net/core/filter.c b/net/core/filter.c
> index 968139f4a1ac..71826e1ef7dc 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -11134,6 +11134,8 @@ sk_reuseport_is_valid_access(int off, int size,
>   	case bpf_ctx_range(struct sk_reuseport_md, ip_protocol):
>   	case bpf_ctx_range(struct sk_reuseport_md, bind_inany):
>   	case bpf_ctx_range(struct sk_reuseport_md, len):
> +	case bpf_ctx_range(struct sk_reuseport_md, rx_queue_mapping):
> +	case bpf_ctx_range(struct sk_reuseport_md, napi_id):
>   		bpf_ctx_record_field_size(info, size_default);
>   		return bpf_ctx_narrow_access_ok(off, size, size_default);
>   
> @@ -11183,6 +11185,14 @@ static u32 sk_reuseport_convert_ctx_access(enum bpf_access_type type,
>   		SK_REUSEPORT_LOAD_SKB_FIELD(protocol);
>   		break;
>   
> +	case offsetof(struct sk_reuseport_md, rx_queue_mapping):
> +		SK_REUSEPORT_LOAD_SKB_FIELD(queue_mapping);
> +		break;
> +
> +	case offsetof(struct sk_reuseport_md, napi_id):
> +		SK_REUSEPORT_LOAD_SKB_FIELD(napi_id);
> +		break;
> +
>   	case offsetof(struct sk_reuseport_md, ip_protocol):
>   		SK_REUSEPORT_LOAD_SK_FIELD(sk_protocol);
>   		break;