Message-ID: <48639eb0-27d9-5754-0687-286e909ceff0@intel.com>
Date:   Thu, 26 Jan 2023 16:56:28 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.0
Subject: Re: [PATCH net] ixgbe: allow to increase MTU to some extent with XDP
 enalbed
Content-Language: en-US
To:     Jason Xing <kerneljasonxing@gmail.com>
CC:     Maciej Fijalkowski <maciej.fijalkowski@intel.com>,
        <jesse.brandeburg@intel.com>, <anthony.l.nguyen@intel.com>,
        <davem@davemloft.net>, <edumazet@google.com>, <kuba@kernel.org>,
        <pabeni@redhat.com>, <richardcochran@gmail.com>, <ast@kernel.org>,
        <daniel@iogearbox.net>, <hawk@kernel.org>,
        <john.fastabend@gmail.co>, <intel-wired-lan@lists.osuosl.org>,
        <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <bpf@vger.kernel.org>, Jason Xing <kernelxing@tencent.com>,
        <magnus.karlsson@intel.com>, <tirthendu.sarkar@intel.com>
References: <20230121085521.9566-1-kerneljasonxing@gmail.com>
 <Y9JvUKBgBifiosOa@boxer>
From:   Alexander Lobakin <alexandr.lobakin@intel.com>
In-Reply-To: <Y9JvUKBgBifiosOa@boxer>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UDgwbkFITmcxSThHZ2Q1MUNIeWVKdDlTOVM5MCtZckgrbGhLZythcWUyaDBW?=
 =?utf-8?B?VUxpSzFjakNOTjRPL2U1REN4Z290THJqbWVrL3R3OTR6TEp3UUxKRklwd1RB?=
 =?utf-8?B?MDF6ZUlia3dDaHFlWGVSSDJqUjcyNG9KNWRyR1Bub01vUVoyTFJFbExSVEpY?=
 =?utf-8?B?dnRIS1RRZW9sVWV1L29nZ25GODMxR1RodFMyWE13VFhCYTFMajZreDVCL251?=
 =?utf-8?B?aVluazllb3lmYjNzSHR1d0d6bWM5SHFKM3VGV0dRN0w5VjRkMWJyVFdtR1RT?=
 =?utf-8?B?bnM4OEt6QnhzbmFnTlpVYm4rdE9wMHg1MGRSRW9ldnN5ZjFFOGJabDBYOHMv?=
 =?utf-8?B?clFjNFBLVkRlUzJJUnBJOEg3alZwVDNnREJwRlRlNGlVUGh5Y1FJbVp5RFNC?=
 =?utf-8?B?RXpHV3d0ajcwTE9iSU9hN3A0aVQyc0lGR1NBSzY0SnYwbjJRNXhrYXpZWjNr?=
 =?utf-8?B?T1NmYVhTNUZjWVNtWjdNOStIUXU4NVBqeC9DdGxLVzdVd1ZrRUZiaU5nWTFi?=
 =?utf-8?B?Q3NReEtXRyt3OXZlWWZxVmpXTHBZYXo5ampOcHprckZIK3d6bE1OWjFCYjNE?=
 =?utf-8?B?TEpvcjgrWXJWQlpMOTlaVUtJRFZyOENpdmorSmZ1ZnUzRU15Y01nVXlYWkFJ?=
 =?utf-8?B?YkRMbEx4QmZtMm9DZlozSkMvWC83MXFWZ3Y5VGZ0OU14S3dvWTFXeUdwWWV1?=
 =?utf-8?B?Wm9LTzNrc29CeksrZUpaT0JCeisxRW52dklGdlkxQ3Vzc016L1ptWGdxd3Yr?=
 =?utf-8?B?UVhOZlFObEtQR1BGcjVzWDQ2QVV4NXpCL1dwYVRGMi9tV3pmTndFNHB0RmFs?=
 =?utf-8?B?VHNsd0ZoTWlGbUhnRTFDWmdlR0RXQ3BpT3J4MDNaWndPM1BtZDIvYUJXN3A5?=
 =?utf-8?B?SmpVY2kvVHZNVGJCM0NkMjFEcUVTK2pNRXdBVE02RkVzVkt6RUtZeTVyRzZY?=
 =?utf-8?B?bm9yTHJYMk1hM3o3U3F6ZUJMQWZzUVgxSzlYVzRmTG1GNjVTR1hCcXFLdlFM?=
 =?utf-8?B?SWZOYW1yVEZaTE01bkxsRUV5NUlWK1hTVVRYSHQrc2dBOTZTdXRpdndyeThL?=
 =?utf-8?B?R29mYk5SanZTTHhmckJTU1pQQUpJaHZlajNZbHNQUkt0NzZUb2pZZHdTYyth?=
 =?utf-8?B?TVhMRFE1SFFRTU84YVJEaURab29zQWduQzVyUkFlNTFOUGVNdVlub29rVTMw?=
 =?utf-8?B?WXNXRW03bC9yZ3NRVEtWcEh4OEdFL0Y0NGRZajZ0TjlGR1lTVGNKRHJyOEJZ?=
 =?utf-8?B?MlY5ZGwxU2lZSmI5OGRMWjBUY2dmbHM4ajNlY1NRZWJPVDZmZ1JwRE85YjBQ?=
 =?utf-8?B?eWJTOG1Mcno1OWtQMklHUW4wWXdJcWpCWWF2S29JaUR3WnVkck8rN01QWEhI?=
 =?utf-8?B?ZlFMdGNqaEZQbHRpeVo3enZRNXV2OHZXVjkvcnhaWU50SW9RaElEQzh5RGtn?=
 =?utf-8?B?SnhMOVlGaWFJa2l2aE9XVGMxRUVubG5CRFZlcW44UUttL0Z0TUQ0aCszZjM5?=
 =?utf-8?B?cVlmbFdKZFNqVzJVMHhxR3J3Znh3ditKRFpyTnkrak1IdGo1RGdzUGRsdnhD?=
 =?utf-8?B?czJGNDRYT2FqYkdraUZkanM2YUR6ZXk0aU0wSHhLcVZqcjdabDhvbFBCMzJq?=
 =?utf-8?B?L0VjZmRrRnlQN0djbXY1MzVsbEhFbVNlaGlFUGlwNWU0T09nS3RYTGRIcFRk?=
 =?utf-8?B?MzQzeGtCVXMwaEEwbG9uenNyWndrYWJMd0hxandYV3FBSUpVa2drWExFam1p?=
 =?utf-8?B?L3N5L0YyTUQvSWx3MlRUaVY4eExsaEdWTk56b2lJSUJHVThINXNCUXlvSmMr?=
 =?utf-8?B?bGpjTVZQYjJwNVZVNlF4RnRXdEN0TWFubC9CV0l0TldNV0IxZzJyMlRwaFBK?=
 =?utf-8?B?L0pEUkVLWGhTVWVtTEh5SWE2VjVJc1p5Wk83a2I3YnhsY05LeHlsK2pJUGpK?=
 =?utf-8?B?bkhKdUhYc0p0QkduV3BmQ1R0bUVkM1hqdTlXeHhPRXhKTFFjc1NTbldPRHIy?=
 =?utf-8?B?SkQvU3NWMS9DdTR1ajJOV1pRVkdMYm5DQmk2dGNwWEEwZ1d3aHlpeDBDTWI1?=
 =?utf-8?B?enlic0hqQ3BrMFNJaHg1R0lLZXFSTHRvLzZWMVRSOG05STM4djF0SlE5dGRK?=
 =?utf-8?B?a0xZWUR4a1dkeHo4SFVaT3ZWb25rbVNjZ3g0U2E4dGhvQnhmeVJiT2R3QnMz?=
 =?utf-8?B?cHc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f1468c1f-7e4d-499b-2c78-08daffb5e6c6
X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB3625.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 15:56:35.5225
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: KjGQFxDUHyBqCM55n/cAi0fJNJ517fviTNxVSU8uvYIZZhFju6NfjZNXB8YvCt+VCOINrbt+YnZQZ4VMUD+P2+UR9GdNoQ9p/NX8l3mOGg0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6749
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

From: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Date: Thu, 26 Jan 2023 13:17:20 +0100

> On Sat, Jan 21, 2023 at 04:55:21PM +0800, Jason Xing wrote:
>> From: Jason Xing <kernelxing@tencent.com>
>>
>> I encountered one case where I cannot increase the MTU size with XDP
>> enabled if the server is equipped with IXGBE card, which happened on
>> thousands of servers. I noticed it was prohibited from 2017[1] and
>> added size checks[2] if allowed soon after the previous patch.
>>
>> Interesting part goes like this:
>> 1) Changing MTU directly from 1500 (default value) to 2000 doesn't
>> work because the driver finds out that 'new_frame_size >
>> ixgbe_rx_bufsz(ring)' in ixgbe_change_mtu() function.
>> 2) However, if we change MTU to 1501 then change from 1501 to 2000, it
>> does work, because the driver sets __IXGBE_RX_3K_BUFFER when MTU size
>> is converted to 1501, which later size check policy allows.
>>
>> The default MTU value for most servers is 1500 which cannot be adjusted
>> directly to the value larger than IXGBE_MAX_2K_FRAME_BUILD_SKB (1534 or
>> 1536) if it loads XDP.
>>
>> After I do a quick study on the manner of i40E driver allowing two kinds
>> of buffer size (one is 2048 while another is 3072) to support XDP mode in
>> i40e_max_xdp_frame_size(), I believe the default MTU size is possibly not
>> satisfied in XDP mode when IXGBE driver is in use, we sometimes need to
>> insert a new header, say, vxlan header. So setting the 3K-buffer flag
>> could solve the issue.
>>
>> [1] commit 38b7e7f8ae82 ("ixgbe: Do not allow LRO or MTU change with XDP")
>> [2] commit fabf1bce103a ("ixgbe: Prevent unsupported configurations with
>> XDP")
>>
>> Signed-off-by: Jason Xing <kernelxing@tencent.com>
>> ---
>>  drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
>> index ab8370c413f3..dc016582f91e 100644
>> --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
>> +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
>> @@ -4313,6 +4313,9 @@ static void ixgbe_set_rx_buffer_len(struct ixgbe_adapter *adapter)
>>  		if (IXGBE_2K_TOO_SMALL_WITH_PADDING ||
>>  		    (max_frame > (ETH_FRAME_LEN + ETH_FCS_LEN)))
>>  			set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state);
>> +
>> +		if (ixgbe_enabled_xdp_adapter(adapter))
>> +			set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state);
> 
> This will result with unnecessary overhead for 1500 MTU because you will
> be working on order-1 pages. Instead I would focus on fixing
> ixgbe_change_mtu() and stop relying on ixgbe_rx_bufsz() in there. You can
> check what we do on ice/i40e sides.
> 
> I'm not looking actively into ixgbe internals but I don't think that there
> is anything that stops us from using 3k buffers with XDP.

I think it uses the same logics as the rest of drivers: splits a 4k page
into two 2k buffers when MTU is <= 1536, otherwise uses order-1 pages
and uses 3k buffers.

OTOH ixgbe is not fully correct in terms how it calculates Rx headroom,
but the main problem is how it calculates the maximum MTU available when
XDP is on. Our usual MTU supported when XDP is on is 3046 bytes.
For MTU <= 1536, 2k buffers are used even for XDP, so the fix is not
correct. Maciej is right that i40e and ice do that way better and don't
have such issue.

> 
>>  #endif
>>  	}
>>  }
>> -- 
>> 2.37.3
>>

Thanks,
Olek