Received: by 2002:a25:c205:0:0:0:0:0 with SMTP id s5csp536501ybf; Fri, 28 Feb 2020 02:56:23 -0800 (PST) X-Google-Smtp-Source: APXvYqzco85WUInWbKZsiVWsQJHZXEvvFYXdzzE53wck07oZ8fNMiLB2OeF2uIZVV3kw4gix3uD8 X-Received: by 2002:a05:6830:1353:: with SMTP id r19mr2812792otq.288.1582887382865; Fri, 28 Feb 2020 02:56:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1582887382; cv=none; d=google.com; s=arc-20160816; b=QzfbLsotwCfXRImPBm+n5f876M93OaY8xAG03TGQmhGHCAlDMPVerB8xtiXO1SvUm6 ixnKva6bb5J17Ut2q3T36WdP4m+5aC7Z1WMOCLSH5MZpe0Mjz/FonBOyOU03rHy+nLXO QsAUCV4hsQnwobBvVx4h3bL/kF3BfQuRJgibTRyheg/pYpwAV/cPD7RcyfGDnRjgXMEE 6CU9gLAWVaqcg+Ftvc7/0uFAvoYuAlmu6H/q/MDrD991gtagyugJQ6VzHiObDcJzsZp9 Nu0R20JaA17IkbfqznRxv4ow4IRAcChq0ZvzKgt05CJSlHo4sgbRQ6MyEQcWDWzHEj1g GmKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:from:subject:mime-version :message-id:date:dkim-signature; bh=tdQUdS7x2rCx1v8FEB8YDbTZb5Bh+jYuKbWwAYyw4WQ=; b=L6j5zhfmGFD8qbKb/PAVPbyUk0WqPAgz+ec61aCNiLC1QZ89zTkS3m6ZYv3mhAj413 pUTT6lCbWc1pwuIKmWuLxyHjcdQMbDkojHP/g9WYZ+cXbJCTQ4fiD14NHQvNIbzUgkWs fkZpmjH0WlqbcfIwUtGdjeJ9JPYfXJHk1g+Xhfa0P85MqtjYo94qRZT/V95f4a5cVWgc Lh/zxbUns0nQtgZU+HJzm3xWlvTtFK1hpHGdoiRJqLhGGFZ4/OMFiD2LZ3oZWXU11n8d bhY8SfLS4JWqVb7GOEXcjnY3pSpN51CPXFKbZJQIF5EnVxDc5XVwamx5fxek9gTk8wpi LRGA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=uyA+4Lzo; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x10si1344416oto.305.2020.02.28.02.56.09; Fri, 28 Feb 2020 02:56:22 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=uyA+4Lzo; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726838AbgB1Kyn (ORCPT + 99 others); Fri, 28 Feb 2020 05:54:43 -0500 Received: from mail-pj1-f74.google.com ([209.85.216.74]:57952 "EHLO mail-pj1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726063AbgB1Kym (ORCPT ); Fri, 28 Feb 2020 05:54:42 -0500 Received: by mail-pj1-f74.google.com with SMTP id ca1so1539194pjb.7 for ; Fri, 28 Feb 2020 02:54:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=tdQUdS7x2rCx1v8FEB8YDbTZb5Bh+jYuKbWwAYyw4WQ=; b=uyA+4LzoUf+htbphfAJGGjZNowcHtlIYcCb1h9czIGbE2qb6Tz6eXbY73v3BbogaXY iYeidB88Z06KBrgMfWUaYHfQZOTcPtBf+ZeZHjum0w5n4QjCr3Kx17RgYuXuCOTVu77O swmEPuElqezEngAnVpoi7/Z0oMH7/3lDie0G6dhpH49HSmqXKfBkcD1ao5Y8VJlAp20C QTbtwrgO1479T4zWxhC7T1MBPBihwcdRZhRbhD4s2wogdH/LFfH8oTeeVQicecE5zPtF VeoqiJvwhOPB4H3M2m0RR2MNL09Zorvbfc6lXvJkh2/tuZ7aouOwUCSTJST7qMPsbjl+ y/4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=tdQUdS7x2rCx1v8FEB8YDbTZb5Bh+jYuKbWwAYyw4WQ=; b=j5mQUnhab6xB66+VCT2Iftr27azD8EVwsqQbo1FWt7iXNmD8EYuaxZLcHDJS6NYaFn 7x3mI0wkxXMaaYF/d/jQ0B6guNgRp8jdvwh0NDVSTqUOXXH9ngBAyvKfE9SG8lkehZRj cuuZ9kSWqRs9WdxFJw6Xk9H/13ypbb0lVgP+2DVF8Oli9vEjAaKDZ0j7hdnpUAQqo7Rx PrBFkBYWiiRWwOKqYrlRh95LkH5XKDpw6e211aGZpVd9yF/jYX+eoZeHLLs2yoKSx9vo Ua7hKvN7QLtGTjAdu+FqpIyd1FPxWyZY8Xwg3DyipzAhMTSq6qot+YfOb+uGiIgdlMbK KaLw== X-Gm-Message-State: APjAAAWeWFI7wDcairQQcn3gMzktwijkFGgzSMUlDevrRDE8WMusf7BG 01X0D2BIIJH5PyJbU05PoYPkBF5nagY= X-Received: by 2002:a63:fb04:: with SMTP id o4mr4081186pgh.423.1582887279503; Fri, 28 Feb 2020 02:54:39 -0800 (PST) Date: Fri, 28 Feb 2020 02:54:35 -0800 Message-Id: <20200228105435.75298-1-lrizzo@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog Subject: [PATCH v4] netdev attribute to control xdpgeneric skb linearization From: Luigi Rizzo To: netdev@vger.kernel.org, toke@redhat.com, davem@davemloft.net, hawk@kernel.org, sameehj@amazon.com Cc: linux-kernel@vger.kernel.org, Luigi Rizzo Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add a netdevice flag to control skb linearization in generic xdp mode. The attribute can be modified through /sys/class/net//xdpgeneric_linearize The default is 1 (on) Motivation: xdp expects linear skbs with some minimum headroom, and generic xdp calls skb_linearize() if needed. The linearization is expensive, and may be unnecessary e.g. when the xdp program does not need access to the whole payload. This sysfs entry allows users to opt out of linearization on a per-device basis (linearization is still performed on cloned skbs). On a kernel instrumented to grab timestamps around the linearization code in netif_receive_generic_xdp, and heavy netperf traffic with 1500b mtu, I see the following times (nanoseconds/pkt) The receiver generally sees larger packets so the difference is more significant. ns/pkt RECEIVER SENDER p50 p90 p99 p50 p90 p99 LINEARIZATION: 600ns 1090ns 4900ns 149ns 249ns 460ns NO LINEARIZATION: 40ns 59ns 90ns 40ns 50ns 100ns v1 --> v2 : added Documentation v2 --> v3 : adjusted for skb_cloned v3 --> v4 : renamed to xdpgeneric_linearize, documentation Signed-off-by: Luigi Rizzo --- Documentation/ABI/testing/sysfs-class-net | 10 ++++++++++ include/linux/netdevice.h | 3 ++- net/core/dev.c | 8 ++++++-- net/core/net-sysfs.c | 16 ++++++++++++++++ 4 files changed, 34 insertions(+), 3 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-class-net b/Documentation/ABI/testing/sysfs-class-net index 664a8f6a634f..d5531bf223d7 100644 --- a/Documentation/ABI/testing/sysfs-class-net +++ b/Documentation/ABI/testing/sysfs-class-net @@ -301,3 +301,13 @@ Contact: netdev@vger.kernel.org Description: 32-bit unsigned integer counting the number of times the link has been down + +What: /sys/class/net//xdpgeneric_linearize +Date: Feb 2020 +KernelVersion: 5.6 +Contact: netdev@vger.kernel.org +Description: + boolean controlling whether skbs should be linearized in + generic XDP. Defaults to true. Turning this off can increase + the performance of generic XDP at the cost of making the XDP + program unable to access packet fragments after the first one. diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 6c3f7032e8d9..f06294b2e8bb 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1985,7 +1985,8 @@ struct net_device { struct netdev_rx_queue *_rx; unsigned int num_rx_queues; - unsigned int real_num_rx_queues; + unsigned int real_num_rx_queues:31; + unsigned int xdpgeneric_linearize : 1; struct bpf_prog __rcu *xdp_prog; unsigned long gro_flush_timeout; diff --git a/net/core/dev.c b/net/core/dev.c index dbbfff123196..c539489d3166 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4520,9 +4520,12 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb, /* XDP packets must be linear and must have sufficient headroom * of XDP_PACKET_HEADROOM bytes. This is the guarantee that also * native XDP provides, thus we need to do it here as well. + * For non shared skbs, xdpgeneric_linearize controls linearization. */ - if (skb_cloned(skb) || skb_is_nonlinear(skb) || - skb_headroom(skb) < XDP_PACKET_HEADROOM) { + if (skb_cloned(skb) || + (skb->dev->xdpgeneric_linearize && + (skb_is_nonlinear(skb) || + skb_headroom(skb) < XDP_PACKET_HEADROOM))) { int hroom = XDP_PACKET_HEADROOM - skb_headroom(skb); int troom = skb->tail + skb->data_len - skb->end; @@ -9806,6 +9809,7 @@ struct net_device *alloc_netdev_mqs(int sizeof_priv, const char *name, dev->gso_max_segs = GSO_MAX_SEGS; dev->upper_level = 1; dev->lower_level = 1; + dev->xdpgeneric_linearize = 1; INIT_LIST_HEAD(&dev->napi_list); INIT_LIST_HEAD(&dev->unreg_list); diff --git a/net/core/net-sysfs.c b/net/core/net-sysfs.c index cf0215734ceb..eab06a427d90 100644 --- a/net/core/net-sysfs.c +++ b/net/core/net-sysfs.c @@ -442,6 +442,21 @@ static ssize_t proto_down_store(struct device *dev, } NETDEVICE_SHOW_RW(proto_down, fmt_dec); +static int change_xdpgeneric_linearize(struct net_device *dev, + unsigned long val) +{ + dev->xdpgeneric_linearize = !!val; + return 0; +} + +static ssize_t xdpgeneric_linearize_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t len) +{ + return netdev_store(dev, attr, buf, len, change_xdpgeneric_linearize); +} +NETDEVICE_SHOW_RW(xdpgeneric_linearize, fmt_dec); + static ssize_t phys_port_id_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -536,6 +551,7 @@ static struct attribute *net_class_attrs[] __ro_after_init = { &dev_attr_phys_port_name.attr, &dev_attr_phys_switch_id.attr, &dev_attr_proto_down.attr, + &dev_attr_xdpgeneric_linearize.attr, &dev_attr_carrier_up_count.attr, &dev_attr_carrier_down_count.attr, NULL, -- 2.25.1.481.gfbce0eb801-goog