Received-SPF: pass (google.com: domain of linux-kernel+bounces-88595-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <e803766b-6a3d-4eab-8335-d415518392e0@gmail.com>
Date: Fri, 1 Mar 2024 16:02:26 +0100
Precedence: bulk
MIME-Version: 1.0
Subject: Re: [PATCH net-next 1/3] net: gro: set {inner_,}network_header in
 receive phase
To: Eric Dumazet <edumazet@google.com>
Cc: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com,
 dsahern@kernel.org, shuah@kernel.org, liujian56@huawei.com,
 horms@kernel.org, aleksander.lobakin@intel.com, linyunsheng@huawei.com,
 therbert@google.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
 linux-kselftest@vger.kernel.org
References: <e1d22505-c5f8-4c02-a997-64248480338b@gmail.com>
 <446695cb-50b8-4187-bf11-63aedb6e9aed@gmail.com>
 <CANn89iJxCT0Bcmqjdd1kp4VCf5i3Me7yATsZXO7SkYTPQ9BOAA@mail.gmail.com>
 <a10b6d19-232c-4b6d-bd71-eb3451675f64@gmail.com>
 <CANn89iK8rDOc14HMGMq=sw0zECcg88Mb=67044r_OA=mSh=xtQ@mail.gmail.com>
From: Richard Gobert <richardbgobert@gmail.com>
In-Reply-To: <CANn89iK8rDOc14HMGMq=sw0zECcg88Mb=67044r_OA=mSh=xtQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit


Eric Dumazet wrote:
> On Thu, Feb 29, 2024 at 2:22 PM Richard Gobert <richardbgobert@gmail.com> wrote:
>>
>>
>>
>> Eric Dumazet wrote:
>>>
>>> My intuition is that this patch has a high cost for normal GRO processing.
>>> SW-GRO is already a bottleneck on ARM cores in smart NICS.
>>>
>>> I would suggest instead using parameters to give both the nhoff and thoff values
>>> this would avoid many conditionals in the fast path.
>>>
>>> ->
>>>
>>> INDIRECT_CALLABLE_SCOPE int udp6_gro_complete(struct sk_buff *skb, int
>>> nhoff, int thoff)
>>> {
>>>  const struct ipv6hdr *ipv6h = (const struct ipv6hdr *)(skb->data + nhoff);
>>>  struct udphdr *uh = (struct udphdr *)(skb->data + thoff);
>>> ...
>>> }
>>>
>>> INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int
>>> nhoff, int thoff)
>>> {
>>>        const struct ipv6hdr *iph =  (const struct ipv6hdr *)(skb->data + nhoff);
>>>        struct tcphdr *th = (struct tcphdr *)(skb->data + thoff);
>>>
>>> Why storing in skb fields things that really could be propagated more
>>> efficiently as function parameters ?
>>
>> Hi Eric,
>> Thanks for the review!
>>
>> I agree, the conditionals could be a problem and are actually not needed.
>> The third commit in this patch series introduces an optimisation for
>> ipv6/ipv4 using the correct {inner_}network_header. We can remove the
>> conditionals; I thought about multiple ways to do so. First, remove the
>> conditional in skb_gro_network_offset:
>>
>>     static inline int skb_gro_network_offset(const struct sk_buff *skb)
>>     {
>>         const u32 mask = NAPI_GRO_CB(skb)->encap_mark - 1;
>>         return (skb_network_offset(skb) & mask) | (skb_inner_network_offset(skb) & ~mask);
>>     }
> 
> I was trying to say that we do not need all these helpers, storing
> state in NAPI_GRO_CB(skb),
> dirtying cache lines...
> 
> Ideally, the skb network/transport/... headers could be set at the
> last stage, in gro_complete(big_gro_skb),
> instead of doing this for each segment.
> 
> All the gro_receive() could be much faster by using additional
> parameters (nhoff, thoff)
> 
> skb_gro_offset() could be replaced by the current offset (nhoff or
> other name), passed as a parameter.
> 
> Here is a WIP for gro_complete() step, this looks large but this is
> only adding a 2nd 'offset' parameter
> 
> Prior offset (typically network offset), called p_off
> Old argument nhoff, (renamed thoff if that makes sense), pointing to
> the current offset.
> 

You're right, it seemed to me like a broad change but it is mainly
cosmetic. I'll finish your version and submit it to fix the bug.

I still believe that setting inner_network_header is a valuable change.
For example, although skb_gro_network_offset is used - setting it in
encapsulation protocol functions (such as ipip_gro_receive) allow us to
remove conditionals from {ipv6,inet}_gro_receive gro_list loop and remove
flush_id from napi_gro_cb as written in the 3rd commit.
What are your thoughts about it as a separate patch?