Message-ID: <ae69c05b-7f7d-ccd2-fb52-f89b74eae2d4@intel.com>
Date:   Fri, 10 Mar 2023 16:52:53 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.8.0
Subject: Re: [PATCH bpf-next v2 0/3] xdp: recycle Page Pool backed skbs built
 from XDP frames
To:     Jesper Dangaard Brouer <jbrouer@redhat.com>
CC:     Alexei Starovoitov <ast@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Andrii Nakryiko <andrii@kernel.org>,
        Martin KaFai Lau <martin.lau@linux.dev>, <brouer@redhat.com>,
        Maciej Fijalkowski <maciej.fijalkowski@intel.com>,
        Larysa Zaremba <larysa.zaremba@intel.com>,
        =?UTF-8?Q?Toke_H=c3=b8iland-J=c3=b8rgensen?= <toke@redhat.com>,
        Song Liu <song@kernel.org>,
        Jesper Dangaard Brouer <hawk@kernel.org>,
        Menglong Dong <imagedong@tencent.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Eric Dumazet <edumazet@google.com>, <bpf@vger.kernel.org>,
        <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>
References: <20230303133232.2546004-1-aleksander.lobakin@intel.com>
 <73b5076c-335b-c746-b227-0edd40435ef5@redhat.com>
Content-Language: en-US
From:   Alexander Lobakin <aleksander.lobakin@intel.com>
In-Reply-To: <73b5076c-335b-c746-b227-0edd40435ef5@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SFJTbloyV1lEUDlRVno3RUl5TmptZVl2TVErR3FUY0JFNFk5MVFiaVY5NUZQ?=
 =?utf-8?B?YTBhT3NrZHNSTHBmaVdUTG5CNGFMTTgyWlhKdG1qc0Q5NlBVVEo5NkxoWGlu?=
 =?utf-8?B?Qks1UmZIY1RhREhhM3JaT3NxU2VwS2s5cktvb0ZGTzg0Q0laMlkvRW5sQng2?=
 =?utf-8?B?ejZyOHNhaGNRNzJKekRGZ0NZRFY5K2tqa2hvTkY2VDI1TVVoR3lFS1VVbDds?=
 =?utf-8?B?dXA3bGM3ZjRKT1ZDMDBEQ2VQMjNHS3E1WXB6b3ZxMzhZejVNb0grWStoTmsr?=
 =?utf-8?B?RkJ3RXJCTnBRK3VPaUxoaEp1dW9Oc25LRnZjVDE4NlRVeUlVQjJWcmhyVlV0?=
 =?utf-8?B?Y3Eyay9SUi95Y3NSWjI0SHBlZlBDdVFlM1YyLzVCcjBBYjRTaUNDUDZURkxX?=
 =?utf-8?B?VnFuUGh6R1ZuOXRRcFpDQzBuUktIWk91TndLYU41dFVzRElkZS9VNkxxRENE?=
 =?utf-8?B?TUdRSzFJTk5nT1RRWDY2bWlRWlAwWHlTcyt2WjRPcXFxRDRNSzNzN3gvQ1hY?=
 =?utf-8?B?TERzalFsdTErN2tMR1JsbUdQK2pycXNsb3RqU1l1SmxtQWZHVmh3a2FlbW5Q?=
 =?utf-8?B?bWtWTElxVW5iUUVRQ3hHdUduMXRJR3RXN3lnUXdtdzVWc1FOS3lneGx0NGZl?=
 =?utf-8?B?ekhuQzVrUllQOXNrdXNzZVVwWlFKNEp5dHlveFpTTk1QZzUwVDcwbldYSHlj?=
 =?utf-8?B?RFZJYzd3Y1V1K2pPZWpNZHBOcEJ6eUNoQlBGUmp0K2pYbmpnbjA1NWhmZWhU?=
 =?utf-8?B?ckpFM0x3NzZKalZBcW9YYjB1b3BvTjNjZmY5eU8vRVpFLytGZElLb3RQZmNi?=
 =?utf-8?B?YWRQZjFxbnc1eDVacVgrRC9zL0NpSDZMZUZVcEJXWVluSWZxNDJIaSt4Qk9Q?=
 =?utf-8?B?NmRJR3RkcTJ5d0w1cTBqMnJyVlNvMlJmdk1EblZKVGkvVEoxRTBUa0ZUWUhF?=
 =?utf-8?B?RGZCT0FLaVJSbVJhSi84R1Y2eERwT0hKS2tUN3U2Yk5pZlE3R09QNDFjaERx?=
 =?utf-8?B?UithMTNiMzVzUk1uWEhmcDE4eHlaOUtDSVBSdU5WU2lnd0pvc0RTTldCNm5G?=
 =?utf-8?B?QzZrVENlVW9SeUl5THM2NjlFZ2IwZGttMTVsOTZSSFBXYi80OVpGSmVHbElt?=
 =?utf-8?B?RGIvZ3NqTTRxaVhGM09reEk5NUJLRjlhK1lONnp2MFlEQjhwb2ZKUStpVG5B?=
 =?utf-8?B?bnFMZWFIeTV0ZlpMdW9tQzBWUjFKYW5kK1MrZEgzTWZ3WGxiVVE0Y0JFYUgw?=
 =?utf-8?B?Q21wOE52OEttL2xUNDNxeUxEbkFFWnMyb28xYTFZZllGa0Y1NyticXZFMXR2?=
 =?utf-8?B?U2N0Y1JndytXbDFxN3dMZjB4Z3RLYjVRL1UwL2xiU0xVTmhKUDAzemVkS0VF?=
 =?utf-8?B?ZWZyNWc1MlRUaDkvNU1rWnBuNnA4aU9iZ3gzUDgzT0RLNU1JTDZsaWtHaUN5?=
 =?utf-8?B?SkhQSFdwL1lvZlRFN05aZWpmZmVVTnZGdUNjcTFkSnJiRkxXM3B2dkZocENL?=
 =?utf-8?B?YmNlK05hUDZTNW1iUEg0d1hicTFQUDZZbzQvN1IrZmFtK2hHa1FiakRHOUlQ?=
 =?utf-8?B?ZzNFMUVsU1B1MmxBTjVybHdoWDNRV0MxQzdMb3plTnBWd0RqMjJiSHpVWnNz?=
 =?utf-8?B?bENaelVncjNDMEhsV2IyZXllRmgvTGVuSWZqRkFTaUgzMXl5ZWcxWHF1anRW?=
 =?utf-8?B?aENCR3loMWFHRi8zQkhBK0NmeUJxNXJ1Q2hwZDgrRnl0NHRzd3VpejVJTHhw?=
 =?utf-8?B?NWFZL3hscHoyNG9Ib3Q3ZGxPWDAralVtZDJGMlRiK0ZkYlJvVmpmVmpINGVh?=
 =?utf-8?B?TzVaNEJlRTZnakt6dk5waVZML0N2WDRaeU1PTmNkNmU0ZExRL0VOQlNzZnJ3?=
 =?utf-8?B?RmRiU1ZrRnc4WHV1d0l5aFRGcFIxTkZRbFJGZGVGNXBhWlk3MDdiTmNhdWIy?=
 =?utf-8?B?SS9YaTZXTTZvcTlLUWh3cCt1QmF2bkxKZ0Y4QTg0S1Nid2VncmQrNDRNdFBn?=
 =?utf-8?B?MDZVLzcyWGt3elgzZUI0SHJ3ekRwdmVKbEJub01acDRBTHFqQ1R5WDBsYmhR?=
 =?utf-8?B?UFVEM0RYSGJKcnlteHZHRDRGRHFydWdTMGVyL2dvR0xHMzhYZWhsZUJMcERQ?=
 =?utf-8?B?OHYxRHRrS1pKbGtPTDN6ZHFkdFJUYm5SWW50VDljYkpVdmV2VHBuTjE4clYr?=
 =?utf-8?Q?4IYH5ZgJYb5r4SX+hLeHBnI=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 98cac9c1-728a-40be-2b7d-08db217fa2a1
X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB3625.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Mar 2023 15:53:47.8279
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EKn6g+ZiL6vNBDjAE2HoGJ7Q9Ca4h03Z3jKBX3diiQWPMTRCz1E+rdKfGwcjg/zMNMxbwn2nkouCsuKCbeXT+GNYtAkgD9XdVtp+ssmhLSE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB5067
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

From: Jesper Dangaard Brouer <jbrouer@redhat.com>
Date: Thu, 9 Mar 2023 17:43:51 +0100

[...]

>> Some numbers on 1 Xeon Platinum core bombed with 27 Mpps of 64-byte
>> IPv6 UDP, iavf w/XDP[0] (CONFIG_PAGE_POOL_STATS is enabled):
>>
>> Plain %XDP_PASS on baseline, Page Pool driver:
>>
>> src cpu Rx     drops  dst cpu Rx
>>    2.1 Mpps       N/A    2.1 Mpps
>>
>> cpumap redirect (w/o leaving its node) on baseline:
> 
> What does it mean "without leaving its node" ?
> I interpret this means BPF program CPU redirect to "same" CPU ?
> Or does the "node" reference a NUMA node?

Yes, NUMA node. It's a two-socket system. I redirect to a different
physical core, but within one NUMA node. When crossing nodes, results
usually are likely worse.

> 
>>
>>    6.8 Mpps  5.0 Mpps    1.8 Mpps
>>
>> cpumap redirect with skb PP recycling:
> 
> Does this test use two CPUs?

Yes, one serves interrupt / NAPI polling function and then redirects all
packets to a different core, which passes them up the stack.

These drops come from that the "source" CPU handles the queue much
faster than the "dest" one is able to process (no GRO on cpumap yet* +
software checksum computation + ...). Still faster than XDP_PASS when
one CPU does everything.

* well, there is cpumap GRO implementation in my repo, but without
hardware checksum status it's pretty useless and currently there's no
hints support in cpumap. So I didn't send it standalone (for now).

> 
>>
>>    7.9 Mpps  5.7 Mpps    2.2 Mpps
>>                         +22% (from cpumap redir on baseline)
>> [0] https://github.com/alobakin/linux/commits/iavf-xdp

[...]

Thanks,
Olek