Received: by 2002:ab2:710b:0:b0:1ef:a325:1205 with SMTP id z11csp1684784lql; Wed, 13 Mar 2024 05:39:27 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCWHZx+YRVFVaNEB32a4dUBPK3foD2PWi83ooW+eIPOCPOjsvzENX4o+f5UpPh2j/p59WLwWU3lfj06shTa33ZEzc3lMp/NoaSvpsbS3Uw== X-Google-Smtp-Source: AGHT+IFgDj98rhullT+puposmA+HX2kqMEtPT2WaV/tMTTiSXtoHqlv/t+UtjLH94TGNPp9g1rzL X-Received: by 2002:a17:902:f68b:b0:1dc:6152:7c24 with SMTP id l11-20020a170902f68b00b001dc61527c24mr3563763plg.43.1710333567646; Wed, 13 Mar 2024 05:39:27 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710333567; cv=pass; d=google.com; s=arc-20160816; b=ded8a06tDNlD2rQ+nYwf8ERj6likofnh4YfOxNJBwQJOInyv09bh/ubiJCIXVZb23b YooDRKZ59TSukR7M5ny9+EVMvdYhSqkJ81ccUUablJkMDXJQ937TaQMHYy8Un9Lp6KmM xzf8wpTkCm/RkFZ01r7G1054aJRRG1B1Y5LDxZNy4h51xq0lfspjO3iVAEp81RiUFyEl 0F9PDYJlFdEvbaLUyBMK8iaJYCJgAVYJefBAAm9Ha9FBv7OkuXAwmaMLV3FgQ/HQunf6 g1vbedwBe1ZRzratUMuTYpzFJFEqOCh1MRGuHn6QO8HWPrVmBs0g6hKIlNt/Fkm9gxbS Ntpg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=p793P1BTk7k277weTIdJuZGu4hnig42k4ieb9wGqdXY=; fh=YW67rZyHld6fcbChJpMg6ZIY6jcKWHTmONtMWzRyN3I=; b=0Ihb1QiZvNc6Gle/nJ+Z66SQxGtuNcXjPRGFBP77sS4ZjhwioMbWc6NxzilEjK3ZIY Amc3F67/Qf1iMyL27ZQAb5/pETM4m/lSBVVYnm8KTBa+YHKnHUoUqff9Nv/kR+eRnW+8 QTnVDaL9+O5MaOVQnECfAB9ozdIhgXRUX+XFnzyzBIslCrI8VCt8lAjDGsTU52USHf4F swYyiQcOQB1Jwy1DWVLUhQ9isy837puSKconwjbFV+kBB80LMwPe5ByddobkNWgJEF97 upJ1/GokeBTfBW+wiW3EtLVFJdlqmk4F4+8ceZTeqnymcsWeKlRJFzTpX2TsbRoTW8SE FeUQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@smartx-com.20230601.gappssmtp.com header.s=20230601 header.b=XyMHnB6V; arc=pass (i=1 dkim=pass dkdomain=smartx-com.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-101482-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-101482-linux.lists.archive=gmail.com@vger.kernel.org" Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id i10-20020a17090332ca00b001dd997eca4fsi6493674plr.19.2024.03.13.05.39.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Mar 2024 05:39:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-101482-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@smartx-com.20230601.gappssmtp.com header.s=20230601 header.b=XyMHnB6V; arc=pass (i=1 dkim=pass dkdomain=smartx-com.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-101482-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-101482-linux.lists.archive=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 3567C284FDC for ; Wed, 13 Mar 2024 12:39:27 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8994A3FE46; Wed, 13 Mar 2024 12:39:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=smartx-com.20230601.gappssmtp.com header.i=@smartx-com.20230601.gappssmtp.com header.b="XyMHnB6V" Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C5AA28F3 for ; Wed, 13 Mar 2024 12:39:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710333549; cv=none; b=IiAu6P3u87P0MeXuxa3ab3pMz4TRK3CUZKjFNtT3CDA8rXuEPuSaKLZTNC8e0UrH2BRJRlMiQC+pfRNJ7d0U2LVWi6evCwES/g5AXRno8OPOKtnYi/ghAjCM/J4Ku4TmhLS5TSAQTNlpzyITdioaAbExoAhwasDM7MXadLM8xnM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710333549; c=relaxed/simple; bh=VxnYqFpMNrkrvkCd/Uk+38u8KTxd2R1sXfsxtDeMJDA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LjfJeNOTAG0kMkd3HdR8E0R1P4AR3M3kswIUE9zZ6NiNExasiBaMbOsZkEg/E0HOipT9SgbodltPxAYmhQdLemFgTh3gvLnLzwOMGJtMAeI2+OyWSXm29TMFg5EQIcUTQktrq/Inebs0CZf0eOBKeDeaYY/jCzRt8qN/THTysJg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=smartx.com; spf=none smtp.mailfrom=smartx.com; dkim=pass (2048-bit key) header.d=smartx-com.20230601.gappssmtp.com header.i=@smartx-com.20230601.gappssmtp.com header.b=XyMHnB6V; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=smartx.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=smartx.com Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1dd10a37d68so8913685ad.2 for ; Wed, 13 Mar 2024 05:39:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1710333546; x=1710938346; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p793P1BTk7k277weTIdJuZGu4hnig42k4ieb9wGqdXY=; b=XyMHnB6Vrl6qGN6n7LSzkQfSJYz0SaaLQI8oIbNdGuHUKaYirpCyc6nZCUjjNN3Zvg yqh6vTQaH5qKTfwe0RAfWF7IGf/ga4rPyg+KFeP9XReVNHV99RHFymJ5aAGw+Qw5PRiB G/LZTOJGJxx57Tkhre669B2181N78ENk+4im8uD5JF3ch+sKZ9zSbSlOmI+v0/3tv+Xr Zq7M/PB1VjdpsYhhoUL9vhk2fq4FL+jPdN3wFmocREsul/4ZBSvFOnlM1yeYVy2m1fin bkZ4Mr9Ebmgud2Ku8uDD3OHUQKHVmKOmvHw9Y88L1mmXf7zs11z+vkW0oEUKC2qOnhhF r7Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710333546; x=1710938346; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p793P1BTk7k277weTIdJuZGu4hnig42k4ieb9wGqdXY=; b=SPy4EQhknz6zs6mcY65PCkk1WbUdQFaJPmehOTenqn2GlQ+In0UGr8ZbSTJjJFiuCQ bSXJah30FzbgoajSBTIVd5XgFLonHDj6ia7XWjibI1dFVJvtiP49tlGaUA9ZOJciV+dA 6DxwRGJT2A7/3KyXuXdawZQtDZTQlVseJVLHhkemVpIyJYduNTbsHTor98TwbKGENXEe GcmB9jupW36RnWpafzDMAFhumby9uJxiVCQT08CHkRfsKJyHV8TUIyex2eriqSAGuXT9 e55M4fpJiY4y++fSq/mpgGvDlFEfciqRFc5wPiaUeAdrIYMFhezvyEtT7U231+mqc04m JG/w== X-Forwarded-Encrypted: i=1; AJvYcCUTMNvDQjemZ0no/Rr5V/8qt7cHP09vuG87u34EWhcbQTsWVlMGdg1T2t/JWkDvcV99fms39CpOUJBuq0JxidJYez81M8YptKvinCtd X-Gm-Message-State: AOJu0Yy6/4d8/J6Yyzjb55I8jSZjmRklpoaDPVoaY/Fl65seVdQxMksA MuxaXolIOv/mLCfm5fI3K7w9502ZrcZLqJyhPRQmYz0r6opbf0dlz3RMjagGb6A= X-Received: by 2002:a17:903:22c8:b0:1dd:72cd:9442 with SMTP id y8-20020a17090322c800b001dd72cd9442mr3612081plg.30.1710333546263; Wed, 13 Mar 2024 05:39:06 -0700 (PDT) Received: from localhost.localdomain.cc ([2407:cdc0:ab48:2284:ee43:5c89:b26a:f291]) by smtp.gmail.com with ESMTPSA id w17-20020a170902e89100b001dd95b5dd0fsm6440696plg.69.2024.03.13.05.39.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Mar 2024 05:39:05 -0700 (PDT) From: Li Feng To: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , linux-nvme@lists.infradead.org (open list:NVM EXPRESS DRIVER), linux-kernel@vger.kernel.org (open list) Cc: Anton.Gavriliuk@hpe.ua, Li Feng Subject: [PATCH v2 2/2] nvme/tcp: Add wq_unbound modparam for nvme_tcp_wq Date: Wed, 13 Mar 2024 20:38:10 +0800 Message-ID: <20240313123816.625115-2-fengli@smartx.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240313123816.625115-1-fengli@smartx.com> References: <20240313123816.625115-1-fengli@smartx.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit The default nvme_tcp_wq will use all CPUs to process tasks. Sometimes it is necessary to set CPU affinity to improve performance. A new module parameter wq_unbound is added here. If set to true, users can configure cpu affinity through /sys/devices/virtual/workqueue/nvme_tcp_wq/cpumask. Signed-off-by: Li Feng --- drivers/nvme/host/tcp.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 2ec1186db0a3..34a882b2ec53 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -36,6 +36,14 @@ static int so_priority; module_param(so_priority, int, 0644); MODULE_PARM_DESC(so_priority, "nvme tcp socket optimize priority"); +/* + * Use the unbound workqueue for nvme_tcp_wq, then we can set the cpu affinity + * from sysfs. + */ +static bool wq_unbound; +module_param(wq_unbound, bool, 0644); +MODULE_PARM_DESC(wq_unbound, "Use unbound workqueue for nvme-tcp IO context (default false)"); + /* * TLS handshake timeout */ @@ -1551,7 +1559,10 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) else if (nvme_tcp_poll_queue(queue)) n = qid - ctrl->io_queues[HCTX_TYPE_DEFAULT] - ctrl->io_queues[HCTX_TYPE_READ] - 1; - queue->io_cpu = cpumask_next_wrap(n - 1, cpu_online_mask, -1, false); + if (wq_unbound) + queue->io_cpu = WORK_CPU_UNBOUND; + else + queue->io_cpu = cpumask_next_wrap(n - 1, cpu_online_mask, -1, false); } static void nvme_tcp_tls_done(void *data, int status, key_serial_t pskid) @@ -2790,6 +2801,8 @@ static struct nvmf_transport_ops nvme_tcp_transport = { static int __init nvme_tcp_init_module(void) { + unsigned int wq_flags = WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_SYSFS; + BUILD_BUG_ON(sizeof(struct nvme_tcp_hdr) != 8); BUILD_BUG_ON(sizeof(struct nvme_tcp_cmd_pdu) != 72); BUILD_BUG_ON(sizeof(struct nvme_tcp_data_pdu) != 24); @@ -2799,8 +2812,10 @@ static int __init nvme_tcp_init_module(void) BUILD_BUG_ON(sizeof(struct nvme_tcp_icresp_pdu) != 128); BUILD_BUG_ON(sizeof(struct nvme_tcp_term_pdu) != 24); - nvme_tcp_wq = alloc_workqueue("nvme_tcp_wq", - WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_SYSFS, 0); + if (wq_unbound) + wq_flags |= WQ_UNBOUND; + + nvme_tcp_wq = alloc_workqueue("nvme_tcp_wq", wq_flags, 0); if (!nvme_tcp_wq) return -ENOMEM; -- 2.44.0