Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Penigalapati, Sandeep" <sandeep.penigalapati@intel.com>
To:     "kerneljasonxing@gmail.com" <kerneljasonxing@gmail.com>,
        "Brandeburg, Jesse" <jesse.brandeburg@intel.com>,
        "Nguyen, Anthony L" <anthony.l.nguyen@intel.com>,
        "davem@davemloft.net" <davem@davemloft.net>,
        "kuba@kernel.org" <kuba@kernel.org>,
        "ast@kernel.org" <ast@kernel.org>,
        "daniel@iogearbox.net" <daniel@iogearbox.net>,
        "hawk@kernel.org" <hawk@kernel.org>,
        "john.fastabend@gmail.com" <john.fastabend@gmail.com>,
        "andrii@kernel.org" <andrii@kernel.org>,
        "kafai@fb.com" <kafai@fb.com>,
        "songliubraving@fb.com" <songliubraving@fb.com>,
        "yhs@fb.com" <yhs@fb.com>,
        "kpsingh@kernel.org" <kpsingh@kernel.org>
CC:     Shujin Li <lishujin@kuaishou.com>,
        Jason Xing <xingwanli@kuaishou.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "intel-wired-lan@lists.osuosl.org" <intel-wired-lan@lists.osuosl.org>,
        "bpf@vger.kernel.org" <bpf@vger.kernel.org>
Subject: RE: [Intel-wired-lan] [PATCH v7] ixgbe: let the xdpdrv work with more
 than 64 cpus
Thread-Topic: [Intel-wired-lan] [PATCH v7] ixgbe: let the xdpdrv work with
 more than 64 cpus
Thread-Index: AQHXnxodNuAOE0/4vU+mtMk0gqjMO6u4/REw
Date:   Tue, 28 Sep 2021 04:03:02 +0000
Message-ID: <MW3PR11MB455445ABC2EE8E063D72AEF89CA89@MW3PR11MB4554.namprd11.prod.outlook.com>
References: <20210901101206.50274-1-kerneljasonxing@gmail.com>
In-Reply-To: <20210901101206.50274-1-kerneljasonxing@gmail.com>
Accept-Language: en-US
Content-Language: en-US
dlp-reaction: no-action
dlp-version: 11.6.200.16
dlp-product: dlpe-windows
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?eGGwxR2zG/FMkmzxU1s9P2YuaNdNhUb4qd5DGMdkNlF+ZldmGuhcLZSXck89?=
 =?us-ascii?Q?Odxj2IxEf0GQojekBha/m2t7yJ2ARd4jXotr5rAb5oSvBa4jqnxc7K2qv5SQ?=
 =?us-ascii?Q?UK0mEZ1z4z1iWG4GsHlBqv1JpJfd1wOp4JlNdZi5e8apCbQ8MqNrdX7eLVqk?=
 =?us-ascii?Q?AoIypF5j2/J42et/5m/AHi1JwTA96VKDjy8AJQC7RCDzszz1ME68j+3e849S?=
 =?us-ascii?Q?yiahwa+hANmip/AXSRleSaE4JGscVWFZVHYH/E+x/Z0HSvNuWcL71mrFpmFn?=
 =?us-ascii?Q?dqloC0exJWGLf9SJCJkTVdhJdqnoh2qmpcc8ua7pbPiVcRC+vSE9EOAAiKW+?=
 =?us-ascii?Q?YYUp75zWS8olXWSD98HlqEK/4NS5Z2jco1UXerTExTmHhqOIJRIC2WGvkYBG?=
 =?us-ascii?Q?BD/kltQMJFxoL7Uk5jjGPFJhV4wjayN9X7XrkPYPMDsYurz929348CAZ0ByH?=
 =?us-ascii?Q?hfM/8MGtTLgDCGpIp59UzHnCTrXqSz+c3wmo9LvI3Gdsf5WHIpvG/NMNXIoc?=
 =?us-ascii?Q?g49dSqMk82FznnQoUgOqN8YQmKv3k/eLOQHh3EsNJum9SjkGzYOGtvIWSgfW?=
 =?us-ascii?Q?JriN5Gm6eNn4ZgD/6mQm2UqsQG1xFIjf0uRQUhTqeoyVobH7lybf3BNATf0F?=
 =?us-ascii?Q?igwLIw4t7cLDZgQ90LsKaNrMpRZvbHPKV9coiREtEthRuGHvlCLZ14TVYr4b?=
 =?us-ascii?Q?1jUYM46oAW5pOdYMcKJjhuR4mCu17nD6bjW2/0IXsErB2j1Pi1tBWYKwus+W?=
 =?us-ascii?Q?fWSk6fzZXNGfxIUl14ZBWI7Q7+1ac0Z8m6EYh02pg5da56xz/QpnejGJHvyP?=
 =?us-ascii?Q?J27INWcU5aKbKyVtlP9HQZCneMznkYOwOhW+ECrQJ1jtsdxHoB9uRSdkJ2RK?=
 =?us-ascii?Q?2GXbKwmW7KoPYpSNU+kJl5Jv60K2tlrd30OTYLE3Ph48Lg3vhY12ngVSPnxr?=
 =?us-ascii?Q?A5JBq/3JLIXreYdDUcsEKOUTune/3moNNTGUModQJ9+6VrnBZsceHdDbwDMW?=
 =?us-ascii?Q?/a4ZKXUcGJD8hqWIfT8g9bd9tDGRYFsL8ZFcsfWCdNj8/ImZQ8mTtukC+vji?=
 =?us-ascii?Q?9eM9gNPzKQErzuOw0fC8naGVnsdmIJRuCHK9dV8Vx3jqH/Mb8NxmZzYBlMn8?=
 =?us-ascii?Q?GYL3m5xbv7B0vZm5eYCxXM1JMGzwqpHLyudvkczc6wdnBaqT1XvpSGoFp5/t?=
 =?us-ascii?Q?NTb+2KSXPa1KPcafJnqQUBYXPVYzfWX9hk0fTijoUMbDBFn3uj4kCMjgO7Pu?=
 =?us-ascii?Q?WgiI1QckMZ6TM7o+PWQ3kn1SFGDhQdIxVb/hnzAU18qt8Hbj7dTYRVArQoLY?=
 =?us-ascii?Q?GXwE7wTrNAzp3gJCj23EJdER?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MW3PR11MB4554.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a4375506-5f6b-44ff-8fc8-08d98234de3e
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Sep 2021 04:03:02.7361
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: PIhUVX2rcM44qjV00WquzvsuHTxvEIlAsAYwsCracjGBTzdaUw57F9K6hdEI4marU77M2AC7KPxsIihPkYXH5v488AWsBbGYWNI3ppGqa+M=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR11MB1375
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

>-----Original Message-----
>From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of
>kerneljasonxing@gmail.com
>Sent: Wednesday, September 1, 2021 3:42 PM
>To: Brandeburg, Jesse <jesse.brandeburg@intel.com>; Nguyen, Anthony L
><anthony.l.nguyen@intel.com>; davem@davemloft.net; kuba@kernel.org;
>ast@kernel.org; daniel@iogearbox.net; hawk@kernel.org;
>john.fastabend@gmail.com; andrii@kernel.org; kafai@fb.com;
>songliubraving@fb.com; yhs@fb.com; kpsingh@kernel.org
>Cc: Shujin Li <lishujin@kuaishou.com>; Jason Xing
><xingwanli@kuaishou.com>; kerneljasonxing@gmail.com;
>netdev@vger.kernel.org; linux-kernel@vger.kernel.org; intel-wired-
>lan@lists.osuosl.org; bpf@vger.kernel.org
>Subject: [Intel-wired-lan] [PATCH v7] ixgbe: let the xdpdrv work with more
>than 64 cpus
>
>From: Jason Xing <xingwanli@kuaishou.com>
>
>Originally, ixgbe driver doesn't allow the mounting of xdpdrv if the serve=
r is
>equipped with more than 64 cpus online. So it turns out that the loading o=
f
>xdpdrv causes the "NOMEM" failure.
>
>Actually, we can adjust the algorithm and then make it work through mappin=
g
>the current cpu to some xdp ring with the protect of @tx_lock.
>
>Here're some numbers before/after applying this patch with xdp-example
>loaded on the eth0X:
>
>As client (tx path):
>                     Before    After
>TCP_STREAM send-64   734.14    714.20
>TCP_STREAM send-128  1401.91   1395.05
>TCP_STREAM send-512  5311.67   5292.84
>TCP_STREAM send-1k   9277.40   9356.22 (not stable)
>TCP_RR     send-1    22559.75  21844.22
>TCP_RR     send-128  23169.54  22725.13
>TCP_RR     send-512  21670.91  21412.56
>
>As server (rx path):
>                     Before    After
>TCP_STREAM send-64   1416.49   1383.12
>TCP_STREAM send-128  3141.49   3055.50
>TCP_STREAM send-512  9488.73   9487.44
>TCP_STREAM send-1k   9491.17   9356.22 (not stable)
>TCP_RR     send-1    23617.74  23601.60
>...
>
>Notice: the TCP_RR mode is unstable as the official document explaines.
>
>I tested many times with different parameters combined through netperf.
>Though the result is not that accurate, I cannot see much influence on thi=
s
>patch. The static key is places on the hot path, but it actually shouldn't=
 cause a
>huge regression theoretically.
>
>Fixes: 33fdc82f08 ("ixgbe: add support for XDP_TX action")
>Reported-by: kernel test robot <lkp@intel.com>
>Co-developed-by: Shujin Li <lishujin@kuaishou.com>
>Signed-off-by: Shujin Li <lishujin@kuaishou.com>
>Signed-off-by: Jason Xing <xingwanli@kuaishou.com>
>---
>v7:
>- Factorized to a single spin_lock/unlock in ixgbe_xdp_xmit() (Eric)
>- Handle other parts of lock/unlock in ixgbe_run_xdp()/_zc() (Jason)
>
>v6:
>- Move the declaration of static-key to the proper position (Test Robot)
>- Add reported-by tag (Jason)
>- Add more detailed performance test results (Jason)
>
>v5:
>- Change back to nr_cpu_ids (Eric)
>
>v4:
>- Update the wrong commit messages. (Jason)
>
>v3:
>- Change nr_cpu_ids to num_online_cpus() (Maciej)
>- Rename MAX_XDP_QUEUES to IXGBE_MAX_XDP_QS (Maciej)
>- Rename ixgbe_determine_xdp_cpu() to ixgbe_determine_xdp_q_idx()
>(Maciej)
>- Wrap ixgbe_xdp_ring_update_tail() with lock into one function (Maciej)
>
>v2:
>- Adjust cpu id in ixgbe_xdp_xmit(). (Jesper)
>- Add a fallback path. (Maciej)
>- Adjust other parts related to xdp ring.
>---
> drivers/net/ethernet/intel/ixgbe/ixgbe.h           | 23 +++++++++-
> drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c       |  9 +++-
> drivers/net/ethernet/intel/ixgbe/ixgbe_main.c      | 50 ++++++++++++++++-=
----
>-
> .../net/ethernet/intel/ixgbe/ixgbe_txrx_common.h   |  3 +-
> drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c       | 16 ++++---
> 5 files changed, 77 insertions(+), 24 deletions(-)
>
Tested-by: Sandeep Penigalapati <sandeep.penigalapati@intel.com>