Received-SPF: pass (google.com: domain of linux-kernel+bounces-84172-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Tue, 27 Feb 2024 15:08:18 -0800 (PST)
From: "Lameter, Christopher" <cl@os.amperecomputing.com>
To: Eric Dumazet <edumazet@google.com>
cc: Adam Li <adamli@os.amperecomputing.com>, corbet@lwn.net, 
    davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, 
    willemb@google.com, yangtiezhu@loongson.cn, atenart@kernel.org, 
    kuniyu@amazon.com, wuyun.abel@bytedance.com, leitao@debian.org, 
    alexander@mihalicyn.com, dhowells@redhat.com, paulmck@kernel.org, 
    joel.granados@gmail.com, urezki@gmail.com, joel@joelfernandes.org, 
    linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, 
    netdev@vger.kernel.org, patches@amperecomputing.com, 
    shijie@os.amperecomputing.com
Subject: Re: [PATCH] net: make SK_MEMORY_PCPU_RESERV tunable
In-Reply-To: <CANn89iLbA4_YdQrF+9Rmv2uVSb1HLhu0qXqCm923FCut1E78FA@mail.gmail.com>
Message-ID: <bc168824-25dd-7541-1a34-38b1a3c00489@os.amperecomputing.com>
References: <20240226022452.20558-1-adamli@os.amperecomputing.com> <CANn89iLbA4_YdQrF+9Rmv2uVSb1HLhu0qXqCm923FCut1E78FA@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?q3i2Dm7L643C1KQI8fyXWQa5M4slZMgVPVQVbpQFKmpSYAhgiJYBNCqpYR14?=
 =?us-ascii?Q?sBvPjIPV2kPBU/xmG6z+RgRmP+eXlVb/q1Lm7UMvQplwPDysOsJmip//RgMA?=
 =?us-ascii?Q?ZiIjJoo7XUdw2S/MjUS717XBsRLQYfqEyCaJdxD5H9ic+dGPxGB4q34rXIzI?=
 =?us-ascii?Q?4OsgYccfpwrvXTrK1z8G/bfgeIrKfOJIMN1QRUP8mLXzxZ93emj4ifR/Tgyx?=
 =?us-ascii?Q?519L7a3Saxy6bHReWcl9OpzavR/KbOC1aDgl8XJ2xalVs8z7nKhiWlHl+RJP?=
 =?us-ascii?Q?ulzkcLeL8Xbk8mR6hHlyDQucleXlPT7nRR8E1w4cueh8CnfG6sSN/P4bFONq?=
 =?us-ascii?Q?Z0KPcKMznJiUnwkVo22CbkNjm48L5PePVebPRaugKdcHyX0GfkyQEvwneAat?=
 =?us-ascii?Q?m2iQPHvX/p5dI+dxmwFeYIswMjYt/9QAN4GAHKlkOpnTdvUQ7BUsuEJxBhwU?=
 =?us-ascii?Q?bcPBqJXflP900ZPfT+VIXjwOrGNsqln7F1CiU5//hPyjZ+Zq2siQidiibzoQ?=
 =?us-ascii?Q?VLPUdIhPUX8U0jf8ly4fpXtyu6LlHmk/Y4dxosYbTlJN6Qusi275+v/b0wF7?=
 =?us-ascii?Q?i16BMdwDIkXnLAMki2QMK5k2lFTyRBl5vFV8q0g+DG4HQMVwhJi0QdcuS/0P?=
 =?us-ascii?Q?RfZbUN34OpsPw5TEY1x/86pR0RJgYLEZtfoN3dFcbTtMeDvu90Yau+2/hQEn?=
 =?us-ascii?Q?k4s4578AlqDgjGbXgnLT3jk5hTaA1zFkMcrBswzPL5mOLSxdt7WdhbAoR4Jh?=
 =?us-ascii?Q?3T5F1+/rwA1ClDUz11uj+i3LX1reUu+1c7wwuWteF0BZhUX9UTkOJQHpUf0/?=
 =?us-ascii?Q?ELNZi9fCL+XBNJ8H8LEdZTWFDMZxv0T+QlxoVA6j5S/gGnvQm+BNedTFCV+o?=
 =?us-ascii?Q?wxcXDrO5tiSaEwU/FGaE5K7POTg/MdZqCewUGjgfZ7tywY0ninJjJS2Hncsq?=
 =?us-ascii?Q?IDxwyGrrUoibIw/upAMNptQX6jIHfTY5IcrRZolMdsOo2c1caU3VthbSGg2H?=
 =?us-ascii?Q?27BF+ElksdguGznJNiyJ95l5eE2QgTRGJs0raVTJtMQo4m6jSzSPDHFvC3Hq?=
 =?us-ascii?Q?VSJuDFaVfB5Snp5q2z3XUlwzNmaaycO5CclPuQ75XlWfDsozLriTi2pnf/1a?=
 =?us-ascii?Q?HDBhn3+LYV4eI/tJoefdDf9Y22YE2len3y5DyUtmCSk4fpkI3Z7tOZE6Vf6e?=
 =?us-ascii?Q?TcHt3iRB8QcFHS4C23u4YrId98KSzDtzSD5V32UmAV25EttjRPTTLI1BvpZF?=
 =?us-ascii?Q?acyBnEPR46BHwKm8LEg0EP+NM+orn86Q4iA+OF3iYjyGOGjjvrG5/ezCOaCe?=
 =?us-ascii?Q?Ddakbua8b7F/oujMKStah9gdfdbxCwGjHPALuT3z6lkiHMs/alkppDu5iR1p?=
 =?us-ascii?Q?cIKaIDpz1B23izCdKGJ22EG9x0sg2pj02CiAD2pSCpRqUxWXd8nW29AG4lzz?=
 =?us-ascii?Q?lLUpWQqGpY2lMgikIj2tdyLstEbTPsH3kFvlK9R78Ym+bjjSEbvLPGQddYG9?=
 =?us-ascii?Q?FEwOMv9gewtyGW0TgvGasPldRQ4xEuRrlKXfUe1GgjUdUZUefTV0iB1KngAM?=
 =?us-ascii?Q?wuvepnmEoqYffd/Ka+lLnLRR0Gu4ePsFw73yJnzPB80ne2tURpgI8u+alpo9?=
 =?us-ascii?Q?H39eAcQhTfjDhYK6WX2F+q4=3D?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4c7db4bb-a809-4f09-bb39-08dc37e8fee6
X-MS-Exchange-CrossTenant-AuthSource: DM6PR01MB5259.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2024 23:08:23.0886
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hFpIYIB689enl5yb5bWF9Vi9hRkjIo67eIyJalhXa/VjZSQ7oR/HyUvZfKrfUi4s47lhZt3v0lKg4QXmgiXhMzTAf7WaJV2II1LBH112GMY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR01MB7055

On Tue, 27 Feb 2024, Eric Dumazet wrote:

>> sk_prot->memory_allocated points to global atomic variable:
>> atomic_long_t tcp_memory_allocated ____cacheline_aligned_in_smp;
>>
>> If increasing the per-cpu cache size from 1MB to e.g. 16MB,
>> changes to sk->sk_prot->memory_allocated can be further reduced.
>> Performance may be improved on system with many cores.
>
> This looks good, do you have any performance numbers to share ?
>
> On a host with 384 threads, 384*16 ->  6 GB of memory.

Those things also come with corresponding memories of a couple of TB...

> With this kind of use, we might need a shrinker...

Yes. No point of keeping the buffers around if the core stops doing 
networking. But to be done at times when there is no contention please. 
Isnt there something like a timeout for skbs in the network stack already?