Date:   Thu, 26 Jan 2023 13:17:20 +0100
From:   Maciej Fijalkowski <maciej.fijalkowski@intel.com>
To:     Jason Xing <kerneljasonxing@gmail.com>
CC:     <jesse.brandeburg@intel.com>, <anthony.l.nguyen@intel.com>,
        <davem@davemloft.net>, <edumazet@google.com>, <kuba@kernel.org>,
        <pabeni@redhat.com>, <richardcochran@gmail.com>, <ast@kernel.org>,
        <daniel@iogearbox.net>, <hawk@kernel.org>,
        <john.fastabend@gmail.co>, <intel-wired-lan@lists.osuosl.org>,
        <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <bpf@vger.kernel.org>, Jason Xing <kernelxing@tencent.com>,
        <magnus.karlsson@intel.com>, <tirthendu.sarkar@intel.com>,
        <alexandr.lobakin@intel.com>
Subject: Re: [PATCH net] ixgbe: allow to increase MTU to some extent with XDP
 enalbed
Message-ID: <Y9JvUKBgBifiosOa@boxer>
References: <20230121085521.9566-1-kerneljasonxing@gmail.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230121085521.9566-1-kerneljasonxing@gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?fc8ln7dpovWR7Aej1VJqx2dKvtsLKKcXeTJANoNbPixcGPya1OdejXYZpD7u?=
 =?us-ascii?Q?l8l3F1oejGzP1tED+xX+UGN9X4mwA4NMaAV1n/8Ho1Um4J4lwaiD/QqVzIfs?=
 =?us-ascii?Q?JfPb/Zl42nwAePz35trKkjd3Q2A8G+dM6WD9AhIJxxe39m3lSoPXwZHipuaC?=
 =?us-ascii?Q?Wcn/BoIhFoEOSP4SEb48ZBzx2uprc92mWzjIAJKQKt/Npp4GSZz4CGQyplsI?=
 =?us-ascii?Q?+OcwnSZgXqzwPpXE0OU8UH5XLbiznkTgG701eXY1SZsYIoStTaum/GBHJTIv?=
 =?us-ascii?Q?Fqrx8NJklejqXwsmqYbCSfdZppZbNrKbncRzix/WfVcFWS5Gao6Ke5Bq8xqW?=
 =?us-ascii?Q?VD5Y/CYHQ1LxxuE0JL8fuJRNNmBWCCSRn1qhFGB0fT5duiw0WfwrOziHgKYg?=
 =?us-ascii?Q?PU4yHcxP4oY+hdXo/4bUREKGrrehwZLznzc/4i1CWCW6PJ913xVYuhfGa3Bi?=
 =?us-ascii?Q?fZiZIlpsKAJbCzpJwkw5x72PkfIqC13AE2j9zq6fJqvnK8oDaPSUIyeGyVzz?=
 =?us-ascii?Q?s2+3BSkViiJwAsi8Te9qtMUQICqZlwYwkDu/Y8gN1lKRkfCqKseMvV2gIqA3?=
 =?us-ascii?Q?TSfJvWbQ+/gX5lgZBvVhDRoDLcmJWDwFpnUTlH1tNDKqerMs/4lw0JriG60D?=
 =?us-ascii?Q?edeYjkWlw6nWsLA3af8rwroqewCbGVTRbhSUNBceQxBHRKpV647L4OIwsW46?=
 =?us-ascii?Q?h8+Lv/opvKKFBePyzj/P/z5zRjaPaP1S+j9k9W11+WVyRyzwl/7G3wwjgLYd?=
 =?us-ascii?Q?bKgQCVblk8O39Oe+06xMHIGkOtUmJPTYmVbIEKhQqA+8kPRmLpIDKqDKB59V?=
 =?us-ascii?Q?9Os1NMj8S291HGLa30qyoELjFQIo3rJVBZ99HwC6Rt/HGUYiMt/0KkyP93+F?=
 =?us-ascii?Q?F9msy6/nJmTt+XTL2iGksZZkNvXk6di4uWOBRbiwwbQskeu07YoUUdXPcFci?=
 =?us-ascii?Q?CMvx7IwbjO19b13Rrf2etyf9gDTKH4DRcBuLrDned557jPyMz4YcRhk3GfUT?=
 =?us-ascii?Q?F+jlgwUKpIEQDURe+3jikL8vLisPnuvMynEvTZPObS2wbWPj4tY9ybb3vjAn?=
 =?us-ascii?Q?SkiPHUCSfm4klxzC3c3KinxxwXJD3Cx9CpFNuOIgxwjLiF7xq0rDCcRsbRg4?=
 =?us-ascii?Q?RnMoh+8wTx6E/ckRF/CCHbKjxPaOEl4AlZIdc72mYaYtd+M6VKpQKZOUgByU?=
 =?us-ascii?Q?BLA42TY4s3tMN88trv1qbSE2P804C7LtG7ifx6+jI9TJCZIdwBvFhcqiSiN/?=
 =?us-ascii?Q?ltL/vigIJNdNNOblKASb9n383CmRLG75JLtXTY9mL5rV9hyHAaa4YYNKkTPD?=
 =?us-ascii?Q?7UbcLkWNFLKv9fHutg8dnQx3dylYm/gFTFEoNE7VO2oFi4eefdNi8b+z8lw7?=
 =?us-ascii?Q?BWWJucol62pD4uqL4ZQWps0TUPolZ8lsKIwFJMWJSBszNvM2OAWun2hksZe1?=
 =?us-ascii?Q?BkzRxYNzeRZ9kevFx0dVbSQZrpszcNiyjpiKNo3yUu5nFJ/MwqLsdPjR3d+n?=
 =?us-ascii?Q?CRYPvb3p1OE17xUcsIjPctFvrNvLm3WYP5NIgCPqfP6aLMZPVrre3n7khAIJ?=
 =?us-ascii?Q?vr90lR3rkwRJQKLsQfuSaVy1KHaw2PGyvXi0YMZbOrrNYf7whS/3vbYvycFe?=
 =?us-ascii?Q?S3BXHBr14puzOkMdDKaRjak=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b71a773a-6dc8-48f4-0b2a-08daff974ec1
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6117.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 12:17:35.4327
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4W/nCHeIXM9J46uQ1vYYocivowhF84CURCOwWH2K3/rngIwCkduup68ZcPvOCHJfWjH1kqyOlQPbNi6Myzd4Yk1jL+n4+seN/T1/PCuoCO4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7629
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Sat, Jan 21, 2023 at 04:55:21PM +0800, Jason Xing wrote:
> From: Jason Xing <kernelxing@tencent.com>
> 
> I encountered one case where I cannot increase the MTU size with XDP
> enabled if the server is equipped with IXGBE card, which happened on
> thousands of servers. I noticed it was prohibited from 2017[1] and
> added size checks[2] if allowed soon after the previous patch.
> 
> Interesting part goes like this:
> 1) Changing MTU directly from 1500 (default value) to 2000 doesn't
> work because the driver finds out that 'new_frame_size >
> ixgbe_rx_bufsz(ring)' in ixgbe_change_mtu() function.
> 2) However, if we change MTU to 1501 then change from 1501 to 2000, it
> does work, because the driver sets __IXGBE_RX_3K_BUFFER when MTU size
> is converted to 1501, which later size check policy allows.
> 
> The default MTU value for most servers is 1500 which cannot be adjusted
> directly to the value larger than IXGBE_MAX_2K_FRAME_BUILD_SKB (1534 or
> 1536) if it loads XDP.
> 
> After I do a quick study on the manner of i40E driver allowing two kinds
> of buffer size (one is 2048 while another is 3072) to support XDP mode in
> i40e_max_xdp_frame_size(), I believe the default MTU size is possibly not
> satisfied in XDP mode when IXGBE driver is in use, we sometimes need to
> insert a new header, say, vxlan header. So setting the 3K-buffer flag
> could solve the issue.
> 
> [1] commit 38b7e7f8ae82 ("ixgbe: Do not allow LRO or MTU change with XDP")
> [2] commit fabf1bce103a ("ixgbe: Prevent unsupported configurations with
> XDP")
> 
> Signed-off-by: Jason Xing <kernelxing@tencent.com>
> ---
>  drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
> index ab8370c413f3..dc016582f91e 100644
> --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
> +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
> @@ -4313,6 +4313,9 @@ static void ixgbe_set_rx_buffer_len(struct ixgbe_adapter *adapter)
>  		if (IXGBE_2K_TOO_SMALL_WITH_PADDING ||
>  		    (max_frame > (ETH_FRAME_LEN + ETH_FCS_LEN)))
>  			set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state);
> +
> +		if (ixgbe_enabled_xdp_adapter(adapter))
> +			set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state);

This will result with unnecessary overhead for 1500 MTU because you will
be working on order-1 pages. Instead I would focus on fixing
ixgbe_change_mtu() and stop relying on ixgbe_rx_bufsz() in there. You can
check what we do on ice/i40e sides.

I'm not looking actively into ixgbe internals but I don't think that there
is anything that stops us from using 3k buffers with XDP.

>  #endif
>  	}
>  }
> -- 
> 2.37.3
>