Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <5aac6822-6fe5-e182-935e-7aa86f1e820d@intel.com>
Date:   Tue, 6 Jun 2023 14:47:56 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.0
Subject: Re: [Intel-wired-lan] [PATCH net-next v2 03/12] iavf: optimize Rx
 buffer allocation a bunch
Content-Language: en-US
To:     Alexander Duyck <alexander.duyck@gmail.com>
CC:     Paul Menzel <pmenzel@molgen.mpg.de>,
        Jesper Dangaard Brouer <hawk@kernel.org>,
        Larysa Zaremba <larysa.zaremba@intel.com>,
        <netdev@vger.kernel.org>,
        Ilias Apalodimas <ilias.apalodimas@linaro.org>,
        <linux-kernel@vger.kernel.org>,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Michal Kubiak <michal.kubiak@intel.com>,
        <intel-wired-lan@lists.osuosl.org>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Christoph Hellwig <hch@lst.de>,
        Magnus Karlsson <magnus.karlsson@intel.com>
References: <20230525125746.553874-1-aleksander.lobakin@intel.com>
 <20230525125746.553874-4-aleksander.lobakin@intel.com>
 <8828262f1c238ab28be9ec87a7701acd791af926.camel@gmail.com>
 <cb7d3479-63a5-31b4-355d-b12a7e1b2878@intel.com>
 <CAKgT0Ud204CiJeB-5zcTKdrv7ODrfP09t73CqRhps7g3qhWU5w@mail.gmail.com>
 <d375fef9-43c4-9f2a-41c9-5247fcb3aa1e@intel.com>
 <CAKgT0Uc4UQ=PpVtjUAP=hjTDrWWkc79PeSwp39T6MSpo1ZyOag@mail.gmail.com>
 <cd88ac7e-fe82-fdc0-3410-0decf57d3c43@intel.com>
 <CAKgT0UeEz2Gqb62sn0pP3_yBMc-LpR0Twmv5_HTREvHBLpCsNw@mail.gmail.com>
From:   Alexander Lobakin <aleksander.lobakin@intel.com>
In-Reply-To: <CAKgT0UeEz2Gqb62sn0pP3_yBMc-LpR0Twmv5_HTREvHBLpCsNw@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aGFwUEdLbzhjRVcvM2trbHRSQVA4cEFIMmF3UlZYdUY5Nzc0NU5OZk9qVVBM?=
 =?utf-8?B?d2FuYmRkS2Zuamc1c01ic0FCOGlkbXp3OWhwOU45UVZhaXNJVjUrTlZBRFEr?=
 =?utf-8?B?R24zZlE4a2ZkQmRmTmREMnE5TUpWejI2bTFYcDM0eEpEM09LRUJkTjFnd0Vi?=
 =?utf-8?B?SWN2VXBHcjBVRGR6UmFQWllFMmhIbHJDWERkck10MmQ0RzZKMWtSRjJsUGRx?=
 =?utf-8?B?V1B4QW1QbW9ERno3c0dQV1lCZVc3UndqWXRBanBZdFUvc0tLU2wrdkJDd1My?=
 =?utf-8?B?TFp1dEpsdE1Zei9ZdmxIQ2pMS1Z2NmtGRmJwWkU1anZoS056TFEvcWk3Qi8r?=
 =?utf-8?B?ZVVvRUtpVURpdUtac2VDanRTcmpiSkloMmRkV0FpbFNyS29md2pQbUV4Nysv?=
 =?utf-8?B?amdTY09rWW5iVCtKUWxtZUZUNWNLczR0emR3ZFlEQlNJVzZpNEQ4MklBUHow?=
 =?utf-8?B?ODBVYzV6VUMyTFkxeEI1dTB3OTl3ZGZnS0JFWjJZYTNiM1ZaQ3lWZDRBS0sz?=
 =?utf-8?B?a2R4MlY1ZHhYOUJlTWJoRG1YeWNqaW1BR1lIaGdYSnlUdHZ5Ky9MZy82NDYy?=
 =?utf-8?B?ekdlYmVveGNOOXUrQUx6OElsTUM2aWRFamxlajQ3SU44Uld0a0U5d0lHNnJ2?=
 =?utf-8?B?VGFvZkQzM0MyRXp0Zk9XK1oxb1FqSzdGdjR6NVdwdnVzUWEyL3dycVl2S3la?=
 =?utf-8?B?enJGaFhmYUhES3N1dDZkQTdpRDJlZXkrWDhPdWxQbXRBdFpqUlpZNUJVMXlH?=
 =?utf-8?B?UWpmbmVMZXVBbTVDYXFjTUJyb1U5RUM3dmd2QTFlV084K3kwZ2ZMakJHbThT?=
 =?utf-8?B?Uk5kdWJNck1RMmFCNW9wZUZhaERmNmFacWMwUXBGNURtZzFhcnpVS3krQkRv?=
 =?utf-8?B?TVdGdnZxNEVyVGhZL215clB4eHhONU1EZDllendBUFkzbVZSdVYxMXV1RHdz?=
 =?utf-8?B?KzcwYUQwdDhUNUNrcVgwZUJBdjFmYXdnSTBOT2h3SFZmbGMwZXg3dkE4Mm9s?=
 =?utf-8?B?RDRWdUExdG1IajJ6RGRLa3NvaFZZVURudmNZSVJpN3JUUFFGMUptLy91ZkNO?=
 =?utf-8?B?blEwaGRLR0syVXRNdjVUSzQwT284RHlrcm9qSFM0MGQ4TlZJWUhnMTE1L2Fl?=
 =?utf-8?B?M2djYnREZENzSG5sTkwvMWRtM2czSFFWUWI2QlV6ZUNxYTVMTWpuK1BnMElw?=
 =?utf-8?B?YU1aelFjaGR4Qjlnb2Z0YkxFM3RBWHgzUVUwTWxKeTN3d0c3RlVMZTJpblUv?=
 =?utf-8?B?Zk9NWTVBbG55U0hxYTF3dmNENFVOQjUrSXN4aldKWmo5eHBFWGVFTVpuTmlk?=
 =?utf-8?B?Um10Nkwra3Q0R1dURTdyZk5HaEo3WFlaZjZDOEtJdE50MXE1dWljZGxrZmla?=
 =?utf-8?B?bFlqUjdSUktucFhlSDVvQ3pjMVoxZTlhWDErOW5NQzAzYUJkRC9zUHhEbjlj?=
 =?utf-8?B?TGxMVWFRVTdscFNWMy8zbktraVRPZWcvYkFqSTcxemRURldqNm1qS1NUOE5U?=
 =?utf-8?B?ODRXRGgrK1JJWmpkYnJIOXdTeVBocnRac1BRRG5sbVFrcDlhT2Z2RDl1ZUd1?=
 =?utf-8?B?MGhDYTloaUNCY0s4eGRpTndoRkZLTjdDa3FhckpvaVdaQlRySjZ0WGRna2RN?=
 =?utf-8?B?K2hMejZoNjRVQTEzaWQ3NzBqTEcxZ2dIMmpocWtJakEyb3kxeENDVDVlcnlL?=
 =?utf-8?B?VjVSYTdiRVdTWG1EeUxvWlRQYXdjUTU2SG92QVQxRHBrN2Zwa0hxWW1HYkhE?=
 =?utf-8?B?OW9aKzVlb1dxVlFxNFNXN2dIMG44ZmNrUGJMYTZlTWxSVDVySUJXSTQ4RTZV?=
 =?utf-8?B?MzQ0UTVBRkpPcCs2ZlhLUWdlcHpLWnRIekJ5NTYzUEFPUHdRSFl2OXU5ZXRY?=
 =?utf-8?B?emduSWhPTUVKL2VqQ09aQTZZU1dMNVlaUWM1aWhVOHVSWGN4SzA2QjE2OWdn?=
 =?utf-8?B?LzRLUWxraEVyd3M4OEhPeVpISm9qVWdqSlRpbjVNWG5IVU10L1hONEVFb0RK?=
 =?utf-8?B?djAvS3BEQUFuN2g4NVB4SmI5eVBuSEUwNmt5cWNmRFV5cmhhT3FUZ3ZQVnR5?=
 =?utf-8?B?RUJhTWwwUzM5VVMxZk53eVNzUmR1SmErWWhRMTJsNzdNM2F0WlN1WDFjL1pZ?=
 =?utf-8?B?SWVyZ3VxZTdBZm5XUzA4VnM1ZUJWcWdHNFdHaFo1YkpXck9HQWVxRlduemRZ?=
 =?utf-8?B?c2c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 20a86b44-ac24-4419-5fea-08db668c8216
X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB3625.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2023 12:49:47.1130
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mnYUYLElIwXKq5GcUoZja7svRqMIb8Fbln37qvq4G+F01ydEU6opLpBk8HuLmM6YUklTzI7s4uxFxyNCrXCH/11hgh5Yf5l5SAYQdsbKXbc=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6678
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

From: Alexander Duyck <alexander.duyck@gmail.com>
Date: Fri, 2 Jun 2023 10:50:02 -0700

Sorry for the silence, had sorta long weekend :p

> On Fri, Jun 2, 2023 at 9:16 AM Alexander Lobakin
> <aleksander.lobakin@intel.com> wrote:

[...]

>> Ok, maybe I phrased it badly.
>> If we don't stop the loop until skb is passed up the stack, how we can
>> go out of the loop with an unfinished skb? Previously, I thought lots of
>> drivers do that, as you may exhaust your budget prior to reaching the
>> last fragment, so you'll get back to the skb on the next poll.
>> But if we count 1 skb as budget unit, not descriptor, how we can end up
>> breaking the loop prior to finishing the skb? I can imagine only one
>> situation: HW gave us some buffers, but still processes the EOP buffer,
>> so we don't have any more descriptors to process, but the skb is still
>> unfinished. But sounds weird TBH, I thought HW processes frames
>> "atomically", i.e. it doesn't give you buffers until they hold the whole
>> frame :D
> 
> The problem is the frames aren't necessarily written back atomically.
> One big issue is descriptor write back. The hardware will try to cache
> line optimize things in order to improve performance. It is possible
> for a single frame to straddle either side of a cache line. As a
> result the first half may be written back, the driver then processes
> that cache line, and finds the next one isn't populated while the
> hardware is collecting enough descriptors to write back the next one.

Ah okay, that's was I was suspecting. So it's not atomic and
skb/xdp_buff is stored on the ring to handle such cases, not budget
exhausting.
Thanks for the detailed explanation. I feel 1 skb = 1 unit more logical
optimal to me now :D

> 
> It is also one of the reasons why I went to so much effort to prevent
> us from writing to the descriptor ring in the cleanup paths. You never
> know when you might be processing an earlier frame and accidently
> wander into a section that is in the process of being written. I think
> that is addressed now mostly through the use of completion queues
> instead of the single ring that used to process both work and
> completions.

Completion rings are neat, you totally avoid writing anything to HW on
Rx polling and vice versa, no descriptor read on refilling. My
preference is to not refill anything on NAPI and do a separate workqueue
for that, esp. given that most NICs nowadays have "refill me please"
interrupt.
Please don't look at the idpf code, IIRC from what I've been told they
do it the "old" way and touch both receive and refill queues on Rx
polling :s :D

>> ice has xdp_buff on the ring for XDP multi-buffer. It's more lightweight
>> than skb, but also carries the frags, since frags is a part of shinfo,
>> not skb.
>> It's totally fine and we'll end up doing the same here, my question was
>> as I explained below.
> 
> Okay. I haven't looked at ice that closely so I wasn't aware of that.

No prob, just FYI. This moves us one step closer to passing something
more lightweight than skb up the stack in non-extreme cases, so that the
stack will take care of it when GROing :)

>>> Yep, now the question is how many drivers can be pulled into using
>>> this library. The issue is going to be all the extra features and
>>> workarounds outside of your basic Tx/Rx will complicate the code since
>>> all the drivers implement them a bit differently. One of the reasons
>>> for not consolidating them was to allow for performance optimizing for
>>> each driver. By combining them you are going to likely need to add a
>>> number of new conditional paths to the fast path.
>>
>> When I was counting the number of spots in the Rx polling function that
>> need to have switch-cases/ifs in order to be able to merge the code
>> (e.g. parsing the descriptors), it was something around 4-5 (per
>> packet). So it can only be figured out during the testing whether adding
>> new branches actually hurts there.
> 
> The other thing is you may want to double check CPU(s) you are
> expected to support as last I knew switch statements were still
> expensive due to all the old spectre/meltdown workarounds.
Wait, are switch-cases also affected? I wasn't aware of that. For sure I
didn't even consider using ops/indirect calls, but switch-cases... I saw
lots o'times people replacing indirections with switch-cases, what's the
point otherwise :D

Thanks,
Olek