Received-SPF: pass (google.com: domain of linux-kernel+bounces-80406-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: Adam Li <adamli@os.amperecomputing.com>
To: corbet@lwn.net,
	davem@davemloft.net,
	edumazet@google.com,
	kuba@kernel.org,
	pabeni@redhat.com
Cc: willemb@google.com,
	yangtiezhu@loongson.cn,
	atenart@kernel.org,
	kuniyu@amazon.com,
	wuyun.abel@bytedance.com,
	leitao@debian.org,
	alexander@mihalicyn.com,
	dhowells@redhat.com,
	paulmck@kernel.org,
	joel.granados@gmail.com,
	urezki@gmail.com,
	joel@joelfernandes.org,
	linux-doc@vger.kernel.org,
	linux-kernel@vger.kernel.org,
	netdev@vger.kernel.org,
	patches@amperecomputing.com,
	cl@os.amperecomputing.com,
	shijie@os.amperecomputing.com,
	Adam Li <adamli@os.amperecomputing.com>
Subject: [PATCH] net: make SK_MEMORY_PCPU_RESERV tunable
Date: Mon, 26 Feb 2024 02:24:52 +0000
Message-Id: <20240226022452.20558-1-adamli@os.amperecomputing.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?V3g/BVq78QcwczZVvQV6Naqqlag+HYSx4DBRlosHvEQaSekvZpR2dNYEIYvW?=
 =?us-ascii?Q?TJuJL0emMc4E7ZyjEaYmzCgtv9JQertKw8B5JQviG8l/0/z6C4viwoUtVj/x?=
 =?us-ascii?Q?Cc5NKubvi6Cgo+JDpupnIJNwxWa6DOwyXcEZ7xfNvz7IsjItiuTa5jAvPytf?=
 =?us-ascii?Q?ySiTqZ80AZD87vAQikxSQnNrz6sISHST4PjvV3EnlI+hn8DGcmVXu5hrr8SW?=
 =?us-ascii?Q?0hbwH1oYP5VHtGiQuxAE/hSN4SE4GtLgN4PJD+mptlP6ymB/Sud/e684mDOl?=
 =?us-ascii?Q?ZLPRnfCGPg1iQpaq+ZTxkIbJZSeDTfh5DFeTnBdZqGpx7Yy2TXcJ4s57G+IX?=
 =?us-ascii?Q?tthCrxZImLrM5gjzOu53uMWqt2F4EZzyWpKsZmuwDFdZO+WCHUgpArXbLxlb?=
 =?us-ascii?Q?e44i4KJcDRaZWC+YYmhhO9qmsItBslpRZ4huD+LkI4T8rffj7Z0u//Z7xEIT?=
 =?us-ascii?Q?gDTohE3SNdlFnozw7FlO/a/I5jZCOukqZh1eZ+qIiSekfhsKoI5pHWXq7X3H?=
 =?us-ascii?Q?rUskpbbRE5zMqp0bjHM2/wLvM9GpRMn8cL9QImJMRXAucuToi1Rhr+M8Gc74?=
 =?us-ascii?Q?u4YLkuOW8IzY9TZE8IUc4yB5xAFyR7m4zcH7jlMBUG/OnwtvLhVC3GcJknHt?=
 =?us-ascii?Q?8FEOJ9kfBXv7XQtfXC7II8zUCgLBIzvAY6SjAP59U7eQ6O6Srv4/DAHGU3u1?=
 =?us-ascii?Q?XVYnFZvdQh33ab7fNmbdfjAsw+VhlbRXHo7xwZrywoAHsUWzTDTjixYAyo4z?=
 =?us-ascii?Q?DuzLtQYgKe2h5JSRzT0DdXohmmzyKydKOZc2dyZsVoaDokkFxjcuwQNmtBVF?=
 =?us-ascii?Q?2zcdJmXG8VDGRZb7lA1iTcBYr7I+xzir6uCno7V5a30NrzeIYzsbeuP8OYfe?=
 =?us-ascii?Q?/6QCfg0OqK5BvUqoVt+rWu49WSONZCpQqNAXmINcNClz8F2K5RIYxdeoAIku?=
 =?us-ascii?Q?klE4VAPhZzsZws1eNcL5eVfLS6AFZB80YbqyTOQIhbrbUKv+YSwil9z3q/rL?=
 =?us-ascii?Q?GOiYQJgxsdhwBiqXzuKB/hX5gdPgrcCesPT6uQNRyOURcigYoq7w5F6ESiCk?=
 =?us-ascii?Q?K8CNeVj2/y2fWz26GowKZzR1aMqe68rHqCbWep6lHcqhDbr5ILUhXqLiMJKs?=
 =?us-ascii?Q?EUY8Mn4Rp8Ub767pGNzfsUIkvxETgAS5YjaR3UKHs+NIZ4WaiydHSz+BSPZ/?=
 =?us-ascii?Q?1FnagLmwXOQoopMlSe6DLsqv+FeQwcJXf9XPcebd1MGt5igJ61SCidQw6apm?=
 =?us-ascii?Q?2wghXlvanvyIkLuxVRaPVlMSIcXbc6j/EFfZ9phDSHr82npfmD/4Boc8SqSu?=
 =?us-ascii?Q?NLRCi4EIc/uU7vrmkGPW+/bhe7VccsKLYIotSiDUzV9bAd03VFXQUVJkIH5x?=
 =?us-ascii?Q?V1oSL/Fm5lMPEQCv/KwcI7KFHx6O+QayhGNaH8TRY1dVwsM1x4UXUsXWfzB3?=
 =?us-ascii?Q?OjIQdYeg2H3RvhNZci6GK9e7LEmn59GsOLIUYIF5Mm3QI7MYm9lWeqpLgw5R?=
 =?us-ascii?Q?a5K9QUKMfQvwdEeMH/eysP5/0TAxE4CFhXTXrTDYG5IN12Ly4OigRF/GidlS?=
 =?us-ascii?Q?Kjd47xqXGZrZExRiM+drLQ9ECFN/dhnMzIK1BTrLdG1hrjBHfDica1GpqCVL?=
 =?us-ascii?Q?Ww=3D=3D?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ccf45a97-a546-43fd-cf1e-08dc3672343a
X-MS-Exchange-CrossTenant-AuthSource: MN0PR01MB7876.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2024 02:25:31.3320
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4kYFnIkqd8OHIMwtllNxxiGDjShoi/akfBHD29A/PFrO+f0TDswRxZkUir5cv01nlW/DBiA6TVG4JtfyY+SgqGTAqpXLMvR8KfbA8es0JhatmafcXffXv7Qi9E3QtvFL
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY3PR01MB6740

This patch adds /proc/sys/net/core/mem_pcpu_rsv sysctl file,
to make SK_MEMORY_PCPU_RESERV tunable.

Commit 3cd3399dd7a8 ("net: implement per-cpu reserves for
memory_allocated") introduced per-cpu forward alloc cache:

"Implement a per-cpu cache of +1/-1 MB, to reduce number
of changes to sk->sk_prot->memory_allocated, which
would otherwise be cause of false sharing."

sk_prot->memory_allocated points to global atomic variable:
atomic_long_t tcp_memory_allocated ____cacheline_aligned_in_smp;

If increasing the per-cpu cache size from 1MB to e.g. 16MB,
changes to sk->sk_prot->memory_allocated can be further reduced.
Performance may be improved on system with many cores.

Signed-off-by: Adam Li <adamli@os.amperecomputing.com>
---
 Documentation/admin-guide/sysctl/net.rst | 5 +++++
 include/net/sock.h                       | 5 +++--
 net/core/sock.c                          | 1 +
 net/core/sysctl_net_core.c               | 9 +++++++++
 4 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst
index 396091651955..7250c0542828 100644
--- a/Documentation/admin-guide/sysctl/net.rst
+++ b/Documentation/admin-guide/sysctl/net.rst
@@ -206,6 +206,11 @@ Will increase power usage.
 
 Default: 0 (off)
 
+mem_pcpu_rsv
+------------
+
+Per-cpu reserved forward alloc cache size in page units. Default 1MB per CPU.
+
 rmem_default
 ------------
 
diff --git a/include/net/sock.h b/include/net/sock.h
index 796a902cf4c1..09a0cde8bf52 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1443,6 +1443,7 @@ sk_memory_allocated(const struct sock *sk)
 
 /* 1 MB per cpu, in page units */
 #define SK_MEMORY_PCPU_RESERVE (1 << (20 - PAGE_SHIFT))
+extern int sysctl_mem_pcpu_rsv;
 
 static inline void
 sk_memory_allocated_add(struct sock *sk, int amt)
@@ -1451,7 +1452,7 @@ sk_memory_allocated_add(struct sock *sk, int amt)
 
 	preempt_disable();
 	local_reserve = __this_cpu_add_return(*sk->sk_prot->per_cpu_fw_alloc, amt);
-	if (local_reserve >= SK_MEMORY_PCPU_RESERVE) {
+	if (local_reserve >= READ_ONCE(sysctl_mem_pcpu_rsv)) {
 		__this_cpu_sub(*sk->sk_prot->per_cpu_fw_alloc, local_reserve);
 		atomic_long_add(local_reserve, sk->sk_prot->memory_allocated);
 	}
@@ -1465,7 +1466,7 @@ sk_memory_allocated_sub(struct sock *sk, int amt)
 
 	preempt_disable();
 	local_reserve = __this_cpu_sub_return(*sk->sk_prot->per_cpu_fw_alloc, amt);
-	if (local_reserve <= -SK_MEMORY_PCPU_RESERVE) {
+	if (local_reserve <= -READ_ONCE(sysctl_mem_pcpu_rsv)) {
 		__this_cpu_sub(*sk->sk_prot->per_cpu_fw_alloc, local_reserve);
 		atomic_long_add(local_reserve, sk->sk_prot->memory_allocated);
 	}
diff --git a/net/core/sock.c b/net/core/sock.c
index 8d86886e39fa..df2ac54a8f74 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -283,6 +283,7 @@ __u32 sysctl_rmem_max __read_mostly = SK_RMEM_MAX;
 EXPORT_SYMBOL(sysctl_rmem_max);
 __u32 sysctl_wmem_default __read_mostly = SK_WMEM_MAX;
 __u32 sysctl_rmem_default __read_mostly = SK_RMEM_MAX;
+int sysctl_mem_pcpu_rsv __read_mostly = SK_MEMORY_PCPU_RESERVE;
 
 int sysctl_tstamp_allow_data __read_mostly = 1;
 
diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c
index 0f0cb1465e08..986f15e5d6c4 100644
--- a/net/core/sysctl_net_core.c
+++ b/net/core/sysctl_net_core.c
@@ -30,6 +30,7 @@ static int int_3600 = 3600;
 static int min_sndbuf = SOCK_MIN_SNDBUF;
 static int min_rcvbuf = SOCK_MIN_RCVBUF;
 static int max_skb_frags = MAX_SKB_FRAGS;
+static int min_mem_pcpu_rsv = SK_MEMORY_PCPU_RESERVE;
 
 static int net_msg_warn;	/* Unused, but still a sysctl */
 
@@ -407,6 +408,14 @@ static struct ctl_table net_core_table[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= &min_rcvbuf,
 	},
+	{
+		.procname	= "mem_pcpu_rsv",
+		.data		= &sysctl_mem_pcpu_rsv,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= &min_mem_pcpu_rsv,
+	},
 	{
 		.procname	= "dev_weight",
 		.data		= &weight_p,
-- 
2.25.1