Received: by 2002:a05:6a10:2785:0:0:0:0 with SMTP id ia5csp404287pxb; Wed, 13 Jan 2021 06:39:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJzdGOuQkdjUefshsoDmSF0xrF7vlI5fzLbQ10xhO/j/gnLnDQntuxoYMklvW/tcEous1QpH X-Received: by 2002:a05:6402:1692:: with SMTP id a18mr1944154edv.321.1610548785616; Wed, 13 Jan 2021 06:39:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1610548785; cv=none; d=google.com; s=arc-20160816; b=rMReNLzjuHhr2LEq782ShOhQQqSQRjAztLxy/DfvJLZ2/Xq2qaotGu2IfUXJRT0J0A HC/fhT2KmxpbyA7bg1vpYHqNChGL50rMsQSIP//stLJTH/2wYYDh74H+iSPtFileMFkE F1Rp2UZKkHtc7nkqESHPKnDwrEIW1swnQCspbkx8N4JRlAbXjCOdOSDZK80ABxPQfIgL /ffx/IY68XI6jrcmICvy2aLw7352w7J1O3iuoWpYd3HDjuh7xxz/oNypnFpVczcOBh8O kVENUGAGYt80+H1DwGG8cMxc9s9nLylt6QIzrNexjlRYAftu2gMd15O4LiWu9hd7Z/Sg z5Qw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:in-reply-to :references:mime-version:dkim-signature; bh=0W94dDXCvx9LJOldtOq3X07sn95kbaYg17AudZ1qMf0=; b=cHBqBEYB0evc/+7vBU8p5xBzgTqG9pKJkSd0BU6bXzzQ+VsNTg2Hu3TBoeo4JZxHaG W+zCm3fFa3Y5iWbh3ZmY+Va+xY2EiftgO5z47zDS/BE4V9AKf1DXrLDYpGR38pneIBjN oByA/cuLmK0rnXP9PkN0kSV+2dvluViuw/5CB+idm61YRueV4Qshj6PoHHkG2++caWis gw4sv8KwuSp69lGBGF69NE3I432becLA7IBWRj6v0/ME3I/12URKsmMGyWtcGIGH6Gpk sQ+Yl2MVQlam1BYsb9aBs/DNg12Yk/R9eNA7FGEJ7qw0dCKEgGra5p9L/5mG0TVzY0aj 2reg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=togBzejO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id u6si1106984edy.204.2021.01.13.06.39.22; Wed, 13 Jan 2021 06:39:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=togBzejO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727179AbhAMOg6 (ORCPT + 99 others); Wed, 13 Jan 2021 09:36:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38526 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726701AbhAMOg5 (ORCPT ); Wed, 13 Jan 2021 09:36:57 -0500 Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70575C061575 for ; Wed, 13 Jan 2021 06:36:17 -0800 (PST) Received: by mail-io1-xd2a.google.com with SMTP id w18so4561273iot.0 for ; Wed, 13 Jan 2021 06:36:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0W94dDXCvx9LJOldtOq3X07sn95kbaYg17AudZ1qMf0=; b=togBzejOMR3dc3xZWuGAJVxYAeCFTfaSFuj/KL/Oq+4GwlQ04I9bVHZyhKQNziOo+V dHPJQOAhMSzu4NkpjfpU+RN0K/2iNJiN6sjfvSBroEt3ajtjTMPsrGjsLWB4g7XLfQ5/ csNPkoLN1TW1BDtH6AwnX9WpwoNrhyRE1QWxftBIJu2HSULKXb4fbDLZRePEeMg26M76 34dCjlq+E6NJa+QqFIej6JnK9XaQZO9C+pSXeTQKHH/8VTmUjnnGE2UfHWn5XyQhiwuv m8rADzVBz1XiRomCKiVlQGuome1ifeaBRBSjY1PMJNsn6c9g+vwNadl/yPaH2HXtQw8c 9DCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0W94dDXCvx9LJOldtOq3X07sn95kbaYg17AudZ1qMf0=; b=DK5f8J277eD0P2aZTaZYoO70vW2ZAimSj8AMMKrsYurY+RQHDj6CxtLjWmdfYcVfli 3CVRT+Q9naK64YWLggfE+6WqP1FNrZvTuJ9J1UO6Brkoh84xg9LZIOoCi+o0S/0IePKB 7rGedH72utRTmbafH+t7u6uxnOgK3S5CDmgGioRKa93gI39C/kzN7GP6E1bUgUrKOiPn ncXCuC5+RwGJqMERX0D8XTqf80kJ06hg5iO+cWCbE9ZDEv8z+AzVdNprPYZ/JZisXMpd x0p9Tzl4AYW1nSQpfXOlMn2VXdB4XMf/TKlU9F2NIPg2gN4C+6Aj8KmL/+4PocgxVAUn yoDg== X-Gm-Message-State: AOAM530KwdA4hT5VrVZz7p443VyThJVZMqGuNQtQBT65SKq6iKlDAKp6 FU+kDwyWvMxik0T25hWRjWvA/i9ABKP2Po1sQEFZHA== X-Received: by 2002:a92:9f59:: with SMTP id u86mr2600070ili.205.1610548576533; Wed, 13 Jan 2021 06:36:16 -0800 (PST) MIME-Version: 1.0 References: <20210113133523.39205-1-alobakin@pm.me> <20210113133635.39402-1-alobakin@pm.me> <20210113133635.39402-2-alobakin@pm.me> In-Reply-To: <20210113133635.39402-2-alobakin@pm.me> From: Eric Dumazet Date: Wed, 13 Jan 2021 15:36:05 +0100 Message-ID: Subject: Re: [PATCH v2 net-next 2/3] skbuff: (re)use NAPI skb cache on allocation path To: Alexander Lobakin Cc: "David S. Miller" , Jakub Kicinski , Edward Cree , Jonathan Lemon , Willem de Bruijn , Miaohe Lin , Steffen Klassert , Guillaume Nault , Yadu Kishore , Al Viro , netdev , LKML , Dmitry Vyukov Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jan 13, 2021 at 2:37 PM Alexander Lobakin wrote: > > Instead of calling kmem_cache_alloc() every time when building a NAPI > skb, (re)use skbuff_heads from napi_alloc_cache.skb_cache. Previously > this cache was only used for bulk-freeing skbuff_heads consumed via > napi_consume_skb() or __kfree_skb_defer(). > > Typical path is: > - skb is queued for freeing from driver or stack, its skbuff_head > goes into the cache instead of immediate freeing; > - driver or stack requests NAPI skb allocation, an skbuff_head is > taken from the cache instead of allocation. > > Corner cases: > - if it's empty on skb allocation, bulk-allocate the first half; > - if it's full on skb consuming, bulk-wipe the second half. > > Also try to balance its size after completing network softirqs > (__kfree_skb_flush()). I do not see the point of doing this rebalance (especially if we do not change its name describing its purpose more accurately). For moderate load, we will have a reduced bulk size (typically one or two). Number of skbs in the cache is in [0, 64[ , there is really no risk of letting skbs there for a long period of time. (32 * sizeof(sk_buff) = 8192) I would personally get rid of this function completely. Also it seems you missed my KASAN support request ? I guess this is a matter of using kasan_unpoison_range(), we can ask for help. > > prefetchw() on CONFIG_SLUB is dropped since it makes no sense anymore. > > Suggested-by: Edward Cree > Signed-off-by: Alexander Lobakin > --- > net/core/skbuff.c | 54 ++++++++++++++++++++++++++++++----------------- > 1 file changed, 35 insertions(+), 19 deletions(-) > > diff --git a/net/core/skbuff.c b/net/core/skbuff.c > index dc3300dc2ac4..f42a3a04b918 100644 > --- a/net/core/skbuff.c > +++ b/net/core/skbuff.c > @@ -364,6 +364,7 @@ struct sk_buff *build_skb_around(struct sk_buff *skb, > EXPORT_SYMBOL(build_skb_around); > > #define NAPI_SKB_CACHE_SIZE 64 > +#define NAPI_SKB_CACHE_HALF (NAPI_SKB_CACHE_SIZE / 2) > > struct napi_alloc_cache { > struct page_frag_cache page; > @@ -487,7 +488,15 @@ EXPORT_SYMBOL(__netdev_alloc_skb); > > static struct sk_buff *napi_skb_cache_get(struct napi_alloc_cache *nc) > { > - return kmem_cache_alloc(skbuff_head_cache, GFP_ATOMIC); > + if (unlikely(!nc->skb_count)) > + nc->skb_count = kmem_cache_alloc_bulk(skbuff_head_cache, > + GFP_ATOMIC, > + NAPI_SKB_CACHE_HALF, > + nc->skb_cache); > + if (unlikely(!nc->skb_count)) > + return NULL; > + > + return nc->skb_cache[--nc->skb_count]; > } > > /** > @@ -867,40 +876,47 @@ void __consume_stateless_skb(struct sk_buff *skb) > void __kfree_skb_flush(void) > { > struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache); > + size_t count; > + void **ptr; > + > + if (unlikely(nc->skb_count == NAPI_SKB_CACHE_HALF)) > + return; > + > + if (nc->skb_count > NAPI_SKB_CACHE_HALF) { > + count = nc->skb_count - NAPI_SKB_CACHE_HALF; > + ptr = nc->skb_cache + NAPI_SKB_CACHE_HALF; > > - /* flush skb_cache if containing objects */ > - if (nc->skb_count) { > - kmem_cache_free_bulk(skbuff_head_cache, nc->skb_count, > - nc->skb_cache); > - nc->skb_count = 0; > + kmem_cache_free_bulk(skbuff_head_cache, count, ptr); > + nc->skb_count = NAPI_SKB_CACHE_HALF; > + } else { > + count = NAPI_SKB_CACHE_HALF - nc->skb_count; > + ptr = nc->skb_cache + nc->skb_count; > + > + nc->skb_count += kmem_cache_alloc_bulk(skbuff_head_cache, > + GFP_ATOMIC, count, > + ptr); > } > } >