Received: by 2002:a25:683:0:0:0:0:0 with SMTP id 125csp54058ybg; Mon, 8 Jun 2020 16:14:35 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzS4GHaEo62dn+Ikn4IVKuGIJB1zgx7hlwOAv/A5g1PxoWD17mqU2/8/ygP3kRtnUc8BBLN X-Received: by 2002:aa7:cb4a:: with SMTP id w10mr23448415edt.46.1591658075124; Mon, 08 Jun 2020 16:14:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1591658075; cv=none; d=google.com; s=arc-20160816; b=sg4S0Hm+Eaj9yND3eaIpv1uJesrk9bnZP2WQX79ntjpeRdDwI27sJAzitw7INAieNn fYcEDDGYBsjnxk0JcreBR5tfgh4bguVi/Q5fu1Xl1g8jOmNAg36H6OsZuJUk//9EAs8j EPXjy9Dis93Lg+TSyt1f7fj/XZdtnXpEYe1a8U8XSxrNGKc0AoFuxxOrZCRUC6/CaVxk t/ES3t/2SCS3mn0KuVQUob26iRnTF7DO+B/hhySe0+wVWfXsLcXIspdfxw55GvhgL+hm Do1lzq9pNhR+vsxjpOe5Y6NZpLIWXZ+Jx2zsFJPs4gTj7mP4GuXmNknmXgRy/3dRqCd8 CvzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xv3D1ZzTPXQpLOGZBHb8beXJg7rPZ8NI/XSYbM9Me3g=; b=n+vUQxkaD9gSy4Jqm+4iACWdBfakbXRkSiO3yF63p2G8d8jhBTlT+nlkB3EViUlQ/6 qoew/dCtQaJbQcKbkgxk079U5T1+UF2Xi80tKs7Avv7R6xA9xjcgG3hDnKGxg/lNJ5Ft K6z5lrFmvOOYfbJDZWqENYI1r71gs78nO6hiv/Qt6SguDue2RiD8NnGv7er9YhHGrkSA hic0pr+XPk9e4BPPcvU/GT2q/w9GHlZITQiTiVT40Uw+ooaYDoq+pOZxq09U+wwB6gSA C6v3rEk0Lkc07F+w7DjN9bxe63KbhiHBIMlBBMzea3yI3a289QKywfXVAbJvAgEtueLz U+1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=Wv23lxQ0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id f24si9972583edr.137.2020.06.08.16.14.10; Mon, 08 Jun 2020 16:14:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=Wv23lxQ0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728984AbgFHXLu (ORCPT + 99 others); Mon, 8 Jun 2020 19:11:50 -0400 Received: from mail.kernel.org ([198.145.29.99]:56278 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728580AbgFHXKM (ORCPT ); Mon, 8 Jun 2020 19:10:12 -0400 Received: from sasha-vm.mshome.net (c-73-47-72-35.hsd1.nh.comcast.net [73.47.72.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3E832214D8; Mon, 8 Jun 2020 23:10:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1591657812; bh=TG/+imYiM+9gXTHKmURbQBfHS3wFWY7Af9lIh/lBkt4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Wv23lxQ09SzC3bEvez3Zl44mlzASkI82VDtAObO9gB1zqGmbgAn3G3+g+p5+8hX2p oS/x/C8hAclWTPrXxP5X2XDvfTF+0n0aoAwU2zuDwzGQkAyNWCyqE11QHiGu28SkVU 0mEBeq+5Z+MotmW2kvLJZdzmsedWveGgGbERPDMo= From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Bhupesh Sharma , kexec@lists.infradead.org, Ariel Elior , GR-everest-linux-l2@marvell.com, Manish Chopra , "David S . Miller" , Sasha Levin , netdev@vger.kernel.org Subject: [PATCH AUTOSEL 5.7 187/274] net: qed*: Reduce RX and TX default ring count when running inside kdump kernel Date: Mon, 8 Jun 2020 19:04:40 -0400 Message-Id: <20200608230607.3361041-187-sashal@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200608230607.3361041-1-sashal@kernel.org> References: <20200608230607.3361041-1-sashal@kernel.org> MIME-Version: 1.0 X-stable: review X-Patchwork-Hint: Ignore Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Bhupesh Sharma [ Upstream commit 73e030977f7884dbe1be0018bab517e8d02760f8 ] Normally kdump kernel(s) run under severe memory constraint with the basic idea being to save the crashdump vmcore reliably when the primary kernel panics/hangs. Currently the qed* ethernet driver ends up consuming a lot of memory in the kdump kernel, leading to kdump kernel panic when one tries to save the vmcore via ssh/nfs (thus utilizing the services of the underlying qed* network interfaces). An example OOM message log seen in the kdump kernel can be seen here [1], with crashkernel size reservation of 512M. Using tools like memstrack (see [2]), we can track the modules taking up the bulk of memory in the kdump kernel and organize the memory usage output as per 'highest allocator first'. An example log for the OOM case indicates that the qed* modules end up allocating approximately 216M memory, which is a large part of the total crashkernel size: dracut-pre-pivot[676]: ======== Report format module_summary: ======== dracut-pre-pivot[676]: Module qed using 149.6MB (2394 pages), peak allocation 149.6MB (2394 pages) dracut-pre-pivot[676]: Module qede using 65.3MB (1045 pages), peak allocation 65.3MB (1045 pages) This patch reduces the default RX and TX ring count from 1024 to 64 when running inside kdump kernel, which leads to a significant memory saving. An example log with the patch applied shows the reduced memory allocation in the kdump kernel: dracut-pre-pivot[674]: ======== Report format module_summary: ======== dracut-pre-pivot[674]: Module qed using 141.8MB (2268 pages), peak allocation 141.8MB (2268 pages) <..snip..> [dracut-pre-pivot[674]: Module qede using 4.8MB (76 pages), peak allocation 4.9MB (78 pages) Tested crashdump vmcore save via ssh/nfs protocol using underlying qed* network interface after applying this patch. [1] OOM log: ------------ kworker/0:6: page allocation failure: order:6, mode:0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null) kworker/0:6 cpuset=/ mems_allowed=0 CPU: 0 PID: 145 Comm: kworker/0:6 Not tainted 4.18.0-109.el8.aarch64 #1 Hardware name: To be filled by O.E.M. Saber/Saber, BIOS 0ACKL025 01/18/2019 Workqueue: events work_for_cpu_fn Call trace: dump_backtrace+0x0/0x188 show_stack+0x24/0x30 dump_stack+0x90/0xb4 warn_alloc+0xf4/0x178 __alloc_pages_nodemask+0xcac/0xd58 alloc_pages_current+0x8c/0xf8 kmalloc_order_trace+0x38/0x108 qed_iov_alloc+0x40/0x248 [qed] qed_resc_alloc+0x224/0x518 [qed] qed_slowpath_start+0x254/0x928 [qed] __qede_probe+0xf8/0x5e0 [qede] qede_probe+0x68/0xd8 [qede] local_pci_probe+0x44/0xa8 work_for_cpu_fn+0x20/0x30 process_one_work+0x1ac/0x3e8 worker_thread+0x44/0x448 kthread+0x130/0x138 ret_from_fork+0x10/0x18 Cannot start slowpath qede: probe of 0000:05:00.1 failed with error -12 [2]. Memstrack tool: https://github.com/ryncsn/memstrack Cc: kexec@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: Ariel Elior Cc: GR-everest-linux-l2@marvell.com Cc: Manish Chopra Cc: David S. Miller Signed-off-by: Bhupesh Sharma Signed-off-by: David S. Miller Signed-off-by: Sasha Levin --- drivers/net/ethernet/qlogic/qede/qede.h | 2 ++ drivers/net/ethernet/qlogic/qede/qede_main.c | 11 +++++++++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/qlogic/qede/qede.h b/drivers/net/ethernet/qlogic/qede/qede.h index 234c6f30effb..234c7e35ee1e 100644 --- a/drivers/net/ethernet/qlogic/qede/qede.h +++ b/drivers/net/ethernet/qlogic/qede/qede.h @@ -574,12 +574,14 @@ int qede_add_tc_flower_fltr(struct qede_dev *edev, __be16 proto, #define RX_RING_SIZE ((u16)BIT(RX_RING_SIZE_POW)) #define NUM_RX_BDS_MAX (RX_RING_SIZE - 1) #define NUM_RX_BDS_MIN 128 +#define NUM_RX_BDS_KDUMP_MIN 63 #define NUM_RX_BDS_DEF ((u16)BIT(10) - 1) #define TX_RING_SIZE_POW 13 #define TX_RING_SIZE ((u16)BIT(TX_RING_SIZE_POW)) #define NUM_TX_BDS_MAX (TX_RING_SIZE - 1) #define NUM_TX_BDS_MIN 128 +#define NUM_TX_BDS_KDUMP_MIN 63 #define NUM_TX_BDS_DEF NUM_TX_BDS_MAX #define QEDE_MIN_PKT_LEN 64 diff --git a/drivers/net/ethernet/qlogic/qede/qede_main.c b/drivers/net/ethernet/qlogic/qede/qede_main.c index 34fa3917eb33..1a83d1fd8ccd 100644 --- a/drivers/net/ethernet/qlogic/qede/qede_main.c +++ b/drivers/net/ethernet/qlogic/qede/qede_main.c @@ -29,6 +29,7 @@ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE * SOFTWARE. */ +#include #include #include #include @@ -707,8 +708,14 @@ static struct qede_dev *qede_alloc_etherdev(struct qed_dev *cdev, edev->dp_module = dp_module; edev->dp_level = dp_level; edev->ops = qed_ops; - edev->q_num_rx_buffers = NUM_RX_BDS_DEF; - edev->q_num_tx_buffers = NUM_TX_BDS_DEF; + + if (is_kdump_kernel()) { + edev->q_num_rx_buffers = NUM_RX_BDS_KDUMP_MIN; + edev->q_num_tx_buffers = NUM_TX_BDS_KDUMP_MIN; + } else { + edev->q_num_rx_buffers = NUM_RX_BDS_DEF; + edev->q_num_tx_buffers = NUM_TX_BDS_DEF; + } DP_INFO(edev, "Allocated netdev with %d tx queues and %d rx queues\n", info->num_queues, info->num_queues); -- 2.25.1