Received: by 2002:a05:6358:3188:b0:123:57c1:9b43 with SMTP id q8csp2797424rwd; Mon, 29 May 2023 00:35:58 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7Q9/Bek2+vMnZE73AUYUS9VfLiJO+8+0lSdqxH/VvYVI0orxlL68Dk8PcUBk83s5FQPxpI X-Received: by 2002:a17:90b:109:b0:256:9b90:c742 with SMTP id p9-20020a17090b010900b002569b90c742mr187099pjz.8.1685345758281; Mon, 29 May 2023 00:35:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685345758; cv=none; d=google.com; s=arc-20160816; b=QU6N3G/6Ti160hBYebvrf5vWMgwljFMhjNg32tk+d5hEBLYpP1xECjHecScYX10Fm0 G7YZCqjrZbxPpWNmNAyq1syh7RqKn7dYBuZMa5WgSXf7W5VLvdsGfpVY/n1I0VQm3Ow5 u5LsMrB23aXOAhUdX50sV1nCpD0fseo0VzitYkoNC5WG5oH7nQCFhcsmFGjvjPmjpdni 4hut0EsbJFynIzhGrfGuPPjQSvPLu9YYfBKiE1LDPBGt0MooVTvY0Cvc5FwpImCiC382 HpjVZ0moKIGxIwMbnVGdYlE83PMp98YUtptOgQt2ysnJBDsvS7TYv+DpGunDWmufCsRt dktg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:cc:to:subject :message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=TuyYcu8t+w4tBaiuC6qtUy6PrdNxgiYYt2fH8+OWVDY=; b=jv4IEd0GYaPYMdvYSxfEplw/5xfC9fyv9or++KgNVCQ7at7S2zzwd4fktfmdBOAv47 mtuYf1bqS+ihu1Y3n/OXDkWGf3Dbb4aV6o3B9JCq17SB/i6DF7hjbRk2EtgNN1DKdwG+ 7PLiSF0fmIJOQ3UGKMOSF5oYvHdvJWHj0njSntCfuyMciC8CTGAs3rL1iPSx2D1CAjhK C5cZJ2w57Dzlrzhka1rVsw6zgA/nBW04pMGhSNuaYrIPkA0RlzhiK9Ioh8G+ulNisI4H +awMkpj8GfcjSceN3wtCjWWyc91n9CY5DMHZzpMpCo481llIA8xCE9Es7z6a2Xao9AWn Tqkg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=QyZQ45wM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k130-20020a633d88000000b00528d0d8b7bbsi8509324pga.595.2023.05.29.00.35.34; Mon, 29 May 2023 00:35:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=QyZQ45wM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229597AbjE2H35 (ORCPT + 99 others); Mon, 29 May 2023 03:29:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229807AbjE2H3p (ORCPT ); Mon, 29 May 2023 03:29:45 -0400 Received: from mail-lj1-x236.google.com (mail-lj1-x236.google.com [IPv6:2a00:1450:4864:20::236]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F446B2; Mon, 29 May 2023 00:29:41 -0700 (PDT) Received: by mail-lj1-x236.google.com with SMTP id 38308e7fff4ca-2af20198f20so30311431fa.0; Mon, 29 May 2023 00:29:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685345379; x=1687937379; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=TuyYcu8t+w4tBaiuC6qtUy6PrdNxgiYYt2fH8+OWVDY=; b=QyZQ45wMo1H/zbmv5hl67s6/CQ/iTljYSFFQrpTwhPoSKo3Y8hcx0rtgxQr81Jdzwh XENE8WRNIT/SfwkQb/k7hwarAmqWGmTMniberK9F5KXA8oxRAqqY52+zIrNAUJHEU2rk 5Z2UP1FSVWZM5ra4pGDgXOcEy941BjAASlxf71N2LdhyyrYZt5dos6SRHCvONPUivPuo mdY/0+6M4ukzeuf7AAZ0Mi1SOqy/bym8ZkWPENgI0yjNtNa6eloLhaC1JgyrljkexlvH 1x4XE33mrfpX2BjGwqEG8S4kiEE2J+Su2S3BAA0fcZqbuBLVPJxD+QntLFy26wexSwBr vJIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685345379; x=1687937379; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TuyYcu8t+w4tBaiuC6qtUy6PrdNxgiYYt2fH8+OWVDY=; b=ZyLSuzOLbJmzKY4Wu17MZZcYM5hS7Y39GrNdXfJ5jB4ax5KkIgPJzoilbBxKDvMbUc 5YbHGmh3pGtb2z0yqVAI1nzbri3W0TkegHW0A5dw3Mgt69j2T8C/OlnXe4waaB+OGF1N fBjraW9oWsEmLU/FobTGeDx2C1H0n5bFgFxGP/jRc0qSzACniNsqXRZPB+znOJI4Cac+ 2IlaTOX5onhfOVliV0eOuIaAHhAqxvMEmITfztg4EyzHaRej411vyCBpD2MxIqAKOq2i 2vk5YpEBjB210hJNWvTVpaRkHgcwFyiKaEvkoEJ6JxCDpvPfoUzzoT1LWK/61uCP3lE2 uu9g== X-Gm-Message-State: AC+VfDyU4+bF86yvvaT20PSc3/SbSBFI/SMenLv4yYLoFsphEE35bE/A DiMgHD5BCsS4cWiKgWiI0BsCrhy4+tpexWfZ6qk= X-Received: by 2002:a2e:94c3:0:b0:2ad:9a11:b131 with SMTP id r3-20020a2e94c3000000b002ad9a11b131mr3432418ljh.32.1685345379171; Mon, 29 May 2023 00:29:39 -0700 (PDT) MIME-Version: 1.0 References: <20230526054621.18371-1-liangchen.linux@gmail.com> <20230526054621.18371-3-liangchen.linux@gmail.com> <20230528022057-mutt-send-email-mst@kernel.org> In-Reply-To: <20230528022057-mutt-send-email-mst@kernel.org> From: Liang Chen Date: Mon, 29 May 2023 15:29:26 +0800 Message-ID: Subject: Re: [PATCH net-next 3/5] virtio_net: Add page pool fragmentation support To: "Michael S. Tsirkin" Cc: jasowang@redhat.com, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, xuanzhuo@linux.alibaba.com, kuba@kernel.org, edumazet@google.com, davem@davemloft.net, pabeni@redhat.com, alexander.duyck@gmail.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, May 28, 2023 at 2:25=E2=80=AFPM Michael S. Tsirkin = wrote: > > On Fri, May 26, 2023 at 01:46:19PM +0800, Liang Chen wrote: > > To further enhance performance, implement page pool fragmentation > > support and introduce a module parameter to enable or disable it. > > > > In single-core vm testing environments, there is an additional performa= nce > > gain observed in the normal path compared to the one packet per page > > approach. > > Upstream codebase: 47.5 Gbits/sec > > Upstream codebase with page pool: 50.2 Gbits/sec > > Upstream codebase with page pool fragmentation support: 52.3 Gbits/se= c > > > > There is also some performance gain for XDP cpumap. > > Upstream codebase: 1.38 Gbits/sec > > Upstream codebase with page pool: 9.74 Gbits/sec > > Upstream codebase with page pool fragmentation: 10.3 Gbits/sec > > > > Signed-off-by: Liang Chen > > I think it's called fragmenting not fragmentation. > > Sure, thanks! > > --- > > drivers/net/virtio_net.c | 72 ++++++++++++++++++++++++++++++---------- > > 1 file changed, 55 insertions(+), 17 deletions(-) > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > index 99c0ca0c1781..ac40b8c66c59 100644 > > --- a/drivers/net/virtio_net.c > > +++ b/drivers/net/virtio_net.c > > @@ -32,7 +32,9 @@ module_param(gso, bool, 0444); > > module_param(napi_tx, bool, 0644); > > > > static bool page_pool_enabled; > > +static bool page_pool_frag; > > module_param(page_pool_enabled, bool, 0400); > > +module_param(page_pool_frag, bool, 0400); > > > > /* FIXME: MTU in config. */ > > #define GOOD_PACKET_LEN (ETH_HLEN + VLAN_HLEN + ETH_DATA_LEN) > > So here again same questions. > > -when is this a net perf gain when does it have no effect? > -can be on by default > - can we get rid of the extra modes? > > Yeah, now I believe it makes sense to enable it by default to avoid the extra modes. Thanks. > > @@ -909,23 +911,32 @@ static struct page *xdp_linearize_page(struct rec= eive_queue *rq, > > struct page *p, > > int offset, > > int page_off, > > - unsigned int *len) > > + unsigned int *len, > > + unsigned int *pp_frag_offset) > > { > > int tailroom =3D SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); > > struct page *page; > > + unsigned int pp_frag_offset_val; > > > > if (page_off + *len + tailroom > PAGE_SIZE) > > return NULL; > > > > if (rq->page_pool) > > - page =3D page_pool_dev_alloc_pages(rq->page_pool); > > + if (rq->page_pool->p.flags & PP_FLAG_PAGE_FRAG) > > + page =3D page_pool_dev_alloc_frag(rq->page_pool, = pp_frag_offset, > > + PAGE_SIZE); > > + else > > + page =3D page_pool_dev_alloc_pages(rq->page_pool)= ; > > else > > page =3D alloc_page(GFP_ATOMIC); > > > > if (!page) > > return NULL; > > > > - memcpy(page_address(page) + page_off, page_address(p) + offset, *= len); > > + pp_frag_offset_val =3D pp_frag_offset ? *pp_frag_offset : 0; > > + > > + memcpy(page_address(page) + page_off + pp_frag_offset_val, > > + page_address(p) + offset, *len); > > page_off +=3D *len; > > > > while (--*num_buf) { > > @@ -948,7 +959,7 @@ static struct page *xdp_linearize_page(struct recei= ve_queue *rq, > > goto err_buf; > > } > > > > - memcpy(page_address(page) + page_off, > > + memcpy(page_address(page) + page_off + pp_frag_offset_val= , > > page_address(p) + off, buflen); > > page_off +=3D buflen; > > virtnet_put_page(rq, p); > > @@ -1029,7 +1040,7 @@ static struct sk_buff *receive_small_xdp(struct n= et_device *dev, > > SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); > > xdp_page =3D xdp_linearize_page(rq, &num_buf, page, > > offset, header_offset, > > - &tlen); > > + &tlen, NULL); > > if (!xdp_page) > > goto err_xdp; > > > > @@ -1323,6 +1334,7 @@ static void *mergeable_xdp_get_buf(struct virtnet= _info *vi, > > unsigned int headroom =3D mergeable_ctx_to_headroom(ctx); > > struct page *xdp_page; > > unsigned int xdp_room; > > + unsigned int page_frag_offset =3D 0; > > > > /* Transient failure which in theory could occur if > > * in-flight packets from before XDP was enabled reach > > @@ -1356,7 +1368,8 @@ static void *mergeable_xdp_get_buf(struct virtnet= _info *vi, > > xdp_page =3D xdp_linearize_page(rq, num_buf, > > *page, offset, > > VIRTIO_XDP_HEADROOM, > > - len); > > + len, > > + &page_frag_offset); > > if (!xdp_page) > > return NULL; > > } else { > > @@ -1366,14 +1379,19 @@ static void *mergeable_xdp_get_buf(struct virtn= et_info *vi, > > return NULL; > > > > if (rq->page_pool) > > - xdp_page =3D page_pool_dev_alloc_pages(rq->page_p= ool); > > + if (rq->page_pool->p.flags & PP_FLAG_PAGE_FRAG) > > + xdp_page =3D page_pool_dev_alloc_frag(rq-= >page_pool, > > + &page= _frag_offset, PAGE_SIZE); > > + else > > + xdp_page =3D page_pool_dev_alloc_pages(rq= ->page_pool); > > else > > xdp_page =3D alloc_page(GFP_ATOMIC); > > + > > if (!xdp_page) > > return NULL; > > > > - memcpy(page_address(xdp_page) + VIRTIO_XDP_HEADROOM, > > - page_address(*page) + offset, *len); > > + memcpy(page_address(xdp_page) + VIRTIO_XDP_HEADROOM + > > + page_frag_offset, page_address(*page) + o= ffset, *len); > > } > > > > *frame_sz =3D PAGE_SIZE; > > @@ -1382,7 +1400,7 @@ static void *mergeable_xdp_get_buf(struct virtnet= _info *vi, > > > > *page =3D xdp_page; > > > > - return page_address(*page) + VIRTIO_XDP_HEADROOM; > > + return page_address(*page) + VIRTIO_XDP_HEADROOM + page_frag_offs= et; > > } > > > > static struct sk_buff *receive_mergeable_xdp(struct net_device *dev, > > @@ -1762,6 +1780,7 @@ static int add_recvbuf_mergeable(struct virtnet_i= nfo *vi, > > void *ctx; > > int err; > > unsigned int len, hole; > > + unsigned int pp_frag_offset; > > > > /* Extra tailroom is needed to satisfy XDP's assumption. This > > * means rx frags coalescing won't work, but consider we've > > @@ -1769,13 +1788,29 @@ static int add_recvbuf_mergeable(struct virtnet= _info *vi, > > */ > > len =3D get_mergeable_buf_len(rq, &rq->mrg_avg_pkt_len, room); > > if (rq->page_pool) { > > - struct page *page; > > + if (rq->page_pool->p.flags & PP_FLAG_PAGE_FRAG) { > > + if (unlikely(!page_pool_dev_alloc_frag(rq->page_p= ool, > > + &pp_frag_o= ffset, len + room))) > > + return -ENOMEM; > > + buf =3D (char *)page_address(rq->page_pool->frag_= page) + > > + pp_frag_offset; > > + buf +=3D headroom; /* advance address leaving hol= e at front of pkt */ > > + hole =3D (PAGE_SIZE << rq->page_pool->p.order) > > + - rq->page_pool->frag_offset; > > + if (hole < len + room) { > > + if (!headroom) > > + len +=3D hole; > > + rq->page_pool->frag_offset +=3D hole; > > + } > > + } else { > > + struct page *page; > > > > - page =3D page_pool_dev_alloc_pages(rq->page_pool); > > - if (unlikely(!page)) > > - return -ENOMEM; > > - buf =3D (char *)page_address(page); > > - buf +=3D headroom; /* advance address leaving hole at fro= nt of pkt */ > > + page =3D page_pool_dev_alloc_pages(rq->page_pool)= ; > > + if (unlikely(!page)) > > + return -ENOMEM; > > + buf =3D (char *)page_address(page); > > + buf +=3D headroom; /* advance address leaving hol= e at front of pkt */ > > + } > > } else { > > if (unlikely(!skb_page_frag_refill(len + room, alloc_frag= , gfp))) > > return -ENOMEM; > > @@ -3800,13 +3835,16 @@ static void virtnet_alloc_page_pool(struct rece= ive_queue *rq) > > struct virtio_device *vdev =3D rq->vq->vdev; > > > > struct page_pool_params pp_params =3D { > > - .order =3D 0, > > + .order =3D page_pool_frag ? SKB_FRAG_PAGE_ORDER : 0, > > .pool_size =3D rq->vq->num_max, > > .nid =3D dev_to_node(vdev->dev.parent), > > .dev =3D vdev->dev.parent, > > .offset =3D 0, > > }; > > > > + if (page_pool_frag) > > + pp_params.flags |=3D PP_FLAG_PAGE_FRAG; > > + > > rq->page_pool =3D page_pool_create(&pp_params); > > if (IS_ERR(rq->page_pool)) { > > dev_warn(&vdev->dev, "page pool creation failed: %ld\n", > > -- > > 2.31.1 >