Received: by 2002:a05:6a10:8c0a:0:0:0:0 with SMTP id go10csp3367651pxb; Mon, 1 Mar 2021 08:17:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJwSzkBVJUNyI4yfHWedF9/XXiaMiAtkJldgzAGqrhmA0NlkwQ/7B1zQTLpKckqejw/dmBFx X-Received: by 2002:a05:6402:11c6:: with SMTP id j6mr17738209edw.138.1614615464873; Mon, 01 Mar 2021 08:17:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1614615464; cv=none; d=google.com; s=arc-20160816; b=Mg5mZs+oTFtOknMULc7ogg54/jYNIfeO53We5MXw4JOuxrZGtR8u8u3Igbz7Bwp7zE ko9I/+BXpAMCHeafIT+++07JVwAYODOM5eNkgmeBIBEeGrUUyRsX8mhv7QZBjqamOU0+ uWi57jkL2eE8h8VythOe4DdxRZk1/pFABYAszZKWCehjNWJLnEQMe24lGA42mgEnvbra Ra+MJeuPVmTq1mdXc1IvBrRsowSEMwmV3LuOS06v4YnytT3KV5cj7TTT/E43KPvYxA0c 3U9ntlsjFSsfAQv8sstZrLwhnK47O8z28HbrKgitjOESgfKciwrUJLh1ytjZ39hQeLYk xdtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=YHWJdnFdXBbs3QFFu0oLknjxkEF6nQ2WE7eZN+AJvrQ=; b=oMi9a/+8zJnDcyR5AaEBIRouwSblDMKTAKWXQDIp/Pr15fZuwp+t7RO0sxabV9e2JI nq2E/8NwJ65s7+4e8SAfQ3+sEszAw3xkZe5DsmpDwc+E+UPmYbVtzX+ZmxFcepSQEb8+ 4kW5Uy09yZMZWxicX/3M+++CZ8ygcUd3Mfzee7W+XcadigkSltro45M9/Zdh9MOd92Im 87Q3GfyjxkJOTbfhb1/2Bv/8zTAiigO6/oD7mxDcsLbJ7YYNRmf5JdIu4U0qhYeUOMzA uVZ9Ijg5LR4deoRLZ14571dRnpPX/02k445zN9envRFhJqrmkl/r5izKTuyq/27rKj/i QcGg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id jl16si6301706ejc.330.2021.03.01.08.17.19; Mon, 01 Mar 2021 08:17:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237661AbhCAQPU (ORCPT + 99 others); Mon, 1 Mar 2021 11:15:20 -0500 Received: from outbound-smtp31.blacknight.com ([81.17.249.62]:39168 "EHLO outbound-smtp31.blacknight.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237674AbhCAQN0 (ORCPT ); Mon, 1 Mar 2021 11:13:26 -0500 Received: from mail.blacknight.com (pemlinmail02.blacknight.ie [81.17.254.11]) by outbound-smtp31.blacknight.com (Postfix) with ESMTPS id 2CCA2C0B06 for ; Mon, 1 Mar 2021 16:12:02 +0000 (GMT) Received: (qmail 30566 invoked from network); 1 Mar 2021 16:12:01 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 1 Mar 2021 16:12:01 -0000 From: Mel Gorman To: Andrew Morton Cc: Chuck Lever , Jesper Dangaard Brouer , LKML , Linux-Net , Linux-MM , Linux-NFS , Mel Gorman Subject: [PATCH 5/5] net: page_pool: use alloc_pages_bulk in refill code path Date: Mon, 1 Mar 2021 16:12:00 +0000 Message-Id: <20210301161200.18852-6-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210301161200.18852-1-mgorman@techsingularity.net> References: <20210301161200.18852-1-mgorman@techsingularity.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Jesper Dangaard Brouer There are cases where the page_pool need to refill with pages from the page allocator. Some workloads cause the page_pool to release pages instead of recycling these pages. For these workload it can improve performance to bulk alloc pages from the page-allocator to refill the alloc cache. For XDP-redirect workload with 100G mlx5 driver (that use page_pool) redirecting xdp_frame packets into a veth, that does XDP_PASS to create an SKB from the xdp_frame, which then cannot return the page to the page_pool. In this case, we saw[1] an improvement of 18.8% from using the alloc_pages_bulk API (3,677,958 pps -> 4,368,926 pps). [1] https://github.com/xdp-project/xdp-project/blob/master/areas/mem/page_pool06_alloc_pages_bulk.org Signed-off-by: Jesper Dangaard Brouer Signed-off-by: Mel Gorman --- net/core/page_pool.c | 63 ++++++++++++++++++++++++++++---------------- 1 file changed, 40 insertions(+), 23 deletions(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index a26f2ceb6a87..567680bd91c4 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -208,44 +208,61 @@ noinline static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, gfp_t _gfp) { + const int bulk = PP_ALLOC_CACHE_REFILL; + struct page *page, *next, *first_page; unsigned int pp_flags = pool->p.flags; - struct page *page; + unsigned int pp_order = pool->p.order; + int pp_nid = pool->p.nid; + LIST_HEAD(page_list); gfp_t gfp = _gfp; - /* We could always set __GFP_COMP, and avoid this branch, as - * prep_new_page() can handle order-0 with __GFP_COMP. - */ - if (pool->p.order) + /* Don't support bulk alloc for high-order pages */ + if (unlikely(pp_order)) { gfp |= __GFP_COMP; + first_page = alloc_pages_node(pp_nid, gfp, pp_order); + if (unlikely(!first_page)) + return NULL; + goto out; + } - /* FUTURE development: - * - * Current slow-path essentially falls back to single page - * allocations, which doesn't improve performance. This code - * need bulk allocation support from the page allocator code. - */ - - /* Cache was empty, do real allocation */ -#ifdef CONFIG_NUMA - page = alloc_pages_node(pool->p.nid, gfp, pool->p.order); -#else - page = alloc_pages(gfp, pool->p.order); -#endif - if (!page) + if (unlikely(!__alloc_pages_bulk_nodemask(gfp, pp_nid, NULL, + bulk, &page_list))) return NULL; + /* First page is extracted and returned to caller */ + first_page = list_first_entry(&page_list, struct page, lru); + list_del(&first_page->lru); + + /* Remaining pages store in alloc.cache */ + list_for_each_entry_safe(page, next, &page_list, lru) { + list_del(&page->lru); + if (pp_flags & PP_FLAG_DMA_MAP && + unlikely(!page_pool_dma_map(pool, page))) { + put_page(page); + continue; + } + if (likely(pool->alloc.count < PP_ALLOC_CACHE_SIZE)) { + pool->alloc.cache[pool->alloc.count++] = page; + pool->pages_state_hold_cnt++; + trace_page_pool_state_hold(pool, page, + pool->pages_state_hold_cnt); + } else { + put_page(page); + } + } +out: if (pp_flags & PP_FLAG_DMA_MAP && - unlikely(!page_pool_dma_map(pool, page))) { - put_page(page); + unlikely(!page_pool_dma_map(pool, first_page))) { + put_page(first_page); return NULL; } /* Track how many pages are held 'in-flight' */ pool->pages_state_hold_cnt++; - trace_page_pool_state_hold(pool, page, pool->pages_state_hold_cnt); + trace_page_pool_state_hold(pool, first_page, pool->pages_state_hold_cnt); /* When page just alloc'ed is should/must have refcnt 1. */ - return page; + return first_page; } /* For using page_pool replace: alloc_pages() API calls, but provide -- 2.26.2