Received: by 2002:a05:6a10:17d3:0:0:0:0 with SMTP id hz19csp2455446pxb; Tue, 13 Apr 2021 02:14:56 -0700 (PDT) X-Google-Smtp-Source: ABdhPJytO4rQfGQccpIr+CbvH0ozY2rXOKn0n7bD2g2AHzMfGEGgXkx6REt7CQgwdE99VOV4ulxG X-Received: by 2002:a17:90a:e50d:: with SMTP id t13mr3816947pjy.160.1618305296263; Tue, 13 Apr 2021 02:14:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1618305296; cv=none; d=google.com; s=arc-20160816; b=zuggtSPY0TfpEh9hnglrUf//GFWdJDNGd6ZUeMUBBYnAAfTmDOmBwwpJilHkwIsXdi dR+kjqBa/MgmgSzhXNpsrg0finvB4xyGYYFKy4NNjvFEJbOzRcatrUtIe6jtcJ8hamgv A2FTGI4vNK56SSqdXyoofv2VkL1fCRya1d+Kwr96LMYzGxJz3mb2iyRK0oTD8b7wmzdk Wuy08mz81e4HJBLf1xFS5Q2SuZJVELoshJKdOZL+I+/W4HZDcpqehGdjj/G2YZxYUxjP PO3LS9lkt8PGFBs2m2WDCclcAl2sBHkAT4WJN8tj6rLWDaG52mETBFmtJFCV/dpMSRFo eoxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:in-reply-to :references:mime-version:dkim-signature; bh=JXWyaO6uozRpzUejkv6WQM7k14W2GQD/gxKY/Zgj0os=; b=m3zt3cpXme3XBqnMv9mPI0XvXh6eoklz2rqJEGX2/G8yrJCWZY9wZVLoxg0MS+2y9m KOVXuV9QzVrFbBOlwDnx6nHXA4zomOr6+kSAuG3NmSTvcif3xBxZ//LR65oH8mXM+97f Vz5O4snEjj6kr8MF7djkDQDidWrwKbVfVthXSUGocHhRgivgSb2EEauc3Fu3SYV27/+k IAOom0nmN6UNRXTaIrpIWA4i4Sj6z69hXUkAuuv+eFXLV89e+FOWgha60gxujouT9vrl iO024i0I4NnLsCUjFJGK8g+V4JoEuhnsACt7Cy9me8QQxPJbLmh7cdRa4LVccV6tVOjL CRkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=IgthMJcs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id m15si16889481plx.61.2021.04.13.02.14.42; Tue, 13 Apr 2021 02:14:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=IgthMJcs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237408AbhDMHOe (ORCPT + 99 others); Tue, 13 Apr 2021 03:14:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229802AbhDMHOd (ORCPT ); Tue, 13 Apr 2021 03:14:33 -0400 Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27195C061574; Tue, 13 Apr 2021 00:14:14 -0700 (PDT) Received: by mail-pf1-x42c.google.com with SMTP id a85so10401494pfa.0; Tue, 13 Apr 2021 00:14:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=JXWyaO6uozRpzUejkv6WQM7k14W2GQD/gxKY/Zgj0os=; b=IgthMJcsWYYBaUVsCfL/HbE0ZaJtQFIY5p4YI9kFhAsE+Hjq0kxXZ+QXPpaa3VzZ5D Uw7RK/1Qo9fsGZY03B6ckRYtpYFHbHVtQFCPYovaQwORWXldZsvgRevJX/CudvS2oRGy SI9TgvFcU5BwPNYQWQMQbDXVr/UzkOX9luFk/oXJLr99hZVwVP+910tsp6lRpMLmXImn znGmsQvTGsLsc8LELvOFJKJsf/9LWtFWEK/Vg6kfYjTSO7+oGZU1nHEMQIoSMDXXM/n6 mTBBeGyPwVECAF7LMtvXdBNOWZhYtAGuwcRItQwRSMCVVoLBCkqiao+AIRnuU554OTi3 BR9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=JXWyaO6uozRpzUejkv6WQM7k14W2GQD/gxKY/Zgj0os=; b=Miu+1r3AmrCjO1VRtFVEB35KYiBE8MYouy5LJbj+XokikoHqdjarrwn+DUmml/z//N pJKlQFkR6L2FLS/I175cuVO4cn9KdCQHkyj8r77ZthnnZuIMvqI7LObdiCocmAydT+yO FAqRKgwmJ/Zrxj/JHVY7DStit2s/hxsWUh4CswRpmW3NPhvD9ehRHLVcLkNrxjgdRMHP 4DuyakQSQOARfuG36lrfnG4jh1GcfV/2jdhQNd/RUmxXXQS/ssFFspt1nUrvjcg7zlft oauVP8EFWcUGJ99RWjKKgbs/JytiBjDW5pealCSvMHqmtLgpquDBXdy7rz9Y82HAgDKl PkMg== X-Gm-Message-State: AOAM531dMEYuuGcyjQD33gnmPRAQfpFAoNtC17wxYGvb0+473dQ3b77u uRtfUZejj8Sxg/JVsX6LlWDRBhLxDaWFPjpJOF8= X-Received: by 2002:a63:cf55:: with SMTP id b21mr32288770pgj.126.1618298053549; Tue, 13 Apr 2021 00:14:13 -0700 (PDT) MIME-Version: 1.0 References: <1618278328.0085247-1-xuanzhuo@linux.alibaba.com> In-Reply-To: <1618278328.0085247-1-xuanzhuo@linux.alibaba.com> From: Magnus Karlsson Date: Tue, 13 Apr 2021 09:14:02 +0200 Message-ID: Subject: Re: [PATCH v2 bpf-next 0/2] xsk: introduce generic almost-zerocopy xmit To: Xuan Zhuo Cc: Alexei Starovoitov , Daniel Borkmann , =?UTF-8?B?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Jonathan Lemon , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh , Network Development , bpf , open list , Alexander Lobakin Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Apr 13, 2021 at 3:49 AM Xuan Zhuo wrote: > > On Mon, 12 Apr 2021 16:13:12 +0200, Magnus Karlsson wrote: > > On Wed, Mar 31, 2021 at 2:27 PM Alexander Lobakin wrote: > > > > > > This series is based on the exceptional generic zerocopy xmit logics > > > initially introduced by Xuan Zhuo. It extends it the way that it > > > could cover all the sane drivers, not only the ones that are capable > > > of xmitting skbs with no linear space. > > > > > > The first patch is a random while-we-are-here improvement over > > > full-copy path, and the second is the main course. See the individual > > > commit messages for the details. > > > > > > The original (full-zerocopy) path is still here and still generally > > > faster, but for now it seems like virtio_net will remain the only > > > user of it, at least for a considerable period of time. > > > > > > From v1 [0]: > > > - don't add a whole SMP_CACHE_BYTES because of only two bytes > > > (NET_IP_ALIGN); > > > - switch to zerocopy if the frame is 129 bytes or longer, not 128. > > > 128 still fit to kmalloc-512, while a zerocopy skb is always > > > kmalloc-1024 -> can potentially be slower on this frame size. > > > > > > [0] https://lore.kernel.org/netdev/20210330231528.546284-1-alobakin@pm.me > > > > > > Alexander Lobakin (2): > > > xsk: speed-up generic full-copy xmit > > > > I took both your patches for a spin on my machine and for the first > > one I do see a small but consistent drop in performance. I thought it > > would go the other way, but it does not so let us put this one on the > > shelf for now. > > > > > xsk: introduce generic almost-zerocopy xmit > > > > This one wreaked havoc on my machine ;-). The performance dropped with > > 75% for packets larger than 128 bytes when the new scheme kicks in. > > Checking with perf top, it seems that we spend much more time > > executing the sendmsg syscall. Analyzing some more: > > > > $ sudo bpftrace -e 'kprobe:__sys_sendto { @calls = @calls + 1; } > > interval:s:1 {printf("calls/sec: %d\n", @calls); @calls = 0;}' > > Attaching 2 probes... > > calls/sec: 1539509 with your patch compared to > > > > calls/sec: 105796 without your patch > > > > The application spends a lot of more time trying to get the kernel to > > send new packets, but the kernel replies with "have not completed the > > outstanding ones, so come back later" = EAGAIN. Seems like the > > transmission takes longer when the skbs have fragments, but I have not > > examined this any further. Did you get a speed-up? > > Regarding this solution, I actually tested it on my mlx5 network card, but the > performance was severely degraded, so I did not continue this solution later. I > guess it might have something to do with the physical network card. We can try > other network cards. I tried it on a third card and got a 40% degradation, so let us scrap this idea. It should stay optional as it is today as the (software) drivers that benefit from this can turn it on explicitly. > links: https://www.spinics.net/lists/netdev/msg710918.html > > Thanks. > > > > > > net/xdp/xsk.c | 32 ++++++++++++++++++++++---------- > > > 1 file changed, 22 insertions(+), 10 deletions(-) > > > > > > -- > > > Well, this is untested. I currently don't have an access to my setup > > > and is bound by moving to another country, but as I don't know for > > > sure at the moment when I'll get back to work on the kernel next time, > > > I found it worthy to publish this now -- if any further changes will > > > be required when I already will be out-of-sight, maybe someone could > > > carry on to make a another revision and so on (I'm still here for any > > > questions, comments, reviews and improvements till the end of this > > > week). > > > But this *should* work with all the sane drivers. If a particular > > > one won't handle this, it's likely ill. Any tests are highly > > > appreciated. Thanks! > > > -- > > > 2.31.1 > > > > > >