Received-SPF: pass (google.com: domain of linux-kernel+bounces-85081-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <a8de785f-8cc3-4075-a5f2-259e20222dcb@os.amperecomputing.com>
Date: Wed, 28 Feb 2024 21:23:23 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] net: make SK_MEMORY_PCPU_RESERV tunable
Content-Language: en-US
To: Eric Dumazet <edumazet@google.com>
Cc: corbet@lwn.net, davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com,
 willemb@google.com, yangtiezhu@loongson.cn, atenart@kernel.org,
 kuniyu@amazon.com, wuyun.abel@bytedance.com, leitao@debian.org,
 alexander@mihalicyn.com, dhowells@redhat.com, paulmck@kernel.org,
 joel.granados@gmail.com, urezki@gmail.com, joel@joelfernandes.org,
 linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
 netdev@vger.kernel.org, patches@amperecomputing.com,
 cl@os.amperecomputing.com, shijie@os.amperecomputing.com
References: <20240226022452.20558-1-adamli@os.amperecomputing.com>
 <CANn89iLbA4_YdQrF+9Rmv2uVSb1HLhu0qXqCm923FCut1E78FA@mail.gmail.com>
From: Adam Li <adamli@os.amperecomputing.com>
In-Reply-To: <CANn89iLbA4_YdQrF+9Rmv2uVSb1HLhu0qXqCm923FCut1E78FA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?VmVqUThYUG03aFJyalU1SlZId0Zid1k5YXJ1U3psdTJ1MFFwQjVYYkZQck5p?=
 =?utf-8?B?Y0NsTXI0bEttU0JJWEl0YVhvTkJwbDBENzVGTHlvRHR1b0pLRUl0TVkxYUxy?=
 =?utf-8?B?VVcrR0FQejhuRmNZeTIwSWp5dWVNd0NTOEVGMFNMN3VaZElCV28xRTliQThS?=
 =?utf-8?B?U2NNV2V4SjJYdkJMdjdQOGhvSmg5VTRjVDlxWXdXRGxxbGVSdkNHWDRjcHhp?=
 =?utf-8?B?MHl0b1ZkQkxrYTlRVVRQMnNtSjhGMVRzR0Q1NTFlWFkyTEVyQmlVUS9WcVk0?=
 =?utf-8?B?dHRMRkYya3E2RVhQL21EN2NRbzRFVkRsZ3JVaUlrbnprWi9iRHdVYlBKQ1VQ?=
 =?utf-8?B?QjlEeHNHOU5DdHVIa21LMENnWWJpRkFyT1ZCRDRvY2Mzejc3aG42d3gwK0FQ?=
 =?utf-8?B?TmRzeDhCbkRIOXBLVEVPWWpFRSt4cWZqWEczaGZxQWEvZDlGdmw5WUQ0am9n?=
 =?utf-8?B?MzZ5ZUZOcVlQL05kbks4SzRJUUpMSVhiYklJWEJFOFJGSVlHUHRTWHV2Y3pj?=
 =?utf-8?B?eHYwRGVqa1RrSWtMY05tbDNNUnh3SVFLd21ZMHRaMmhkR0Y0T3k0UVMremVN?=
 =?utf-8?B?bFExemtpa3FBNE5DOEplTC9aSVk4c2s5WVJMczRrNlJJRkZZNEgyaXVDVHl5?=
 =?utf-8?B?RWhRa0NTOTVoaStIVDlYVEVqZHhmN3duVHZQZk1MTWJ6b2Yva0EzRzk0UTdE?=
 =?utf-8?B?Sld2dTVla3VzcERQeWZIQ0N1S1kreERWVTA1T2xkcUxWdktwVmtoSngreWk1?=
 =?utf-8?B?YmgyOXVQL0xabU4rdVd0VWhhTmFiLzYvYWx5WlFrSHgybE5RVDRSejJVWkpq?=
 =?utf-8?B?Slp0NVlvaVpHYzgwVW1tb21lTGpscEp5bld4YW1xbHBVWTdqK3J5N1d4L3ZM?=
 =?utf-8?B?SVZ0VWlraUZVSDdOM0NtT205MW9aUE9HbU9pcUF5WGdtVFlTQ2pGU0JUUklt?=
 =?utf-8?B?b2xKaWJpb01LaWdSVExwOUk0WGk0YWJTNytobUNNLysrUVhMODVtR2VkUk9R?=
 =?utf-8?B?cXV5cGhPVlVNUHBTd3BnT3ZIK3RkQkxmOGdYc09kRytyRmtiY2pYUkxoTWVk?=
 =?utf-8?B?blYwNEFvZ2ZrbTV1Z3FFN0E3VllYajMvWjdXSVIzYng2aUsvVnNXbUJqSUV5?=
 =?utf-8?B?TW1rM3RhVG1mQ1loYzdBTVJXN0ljQ0JJckxDM0NIb2tNT3VpZFhHWWxndjRN?=
 =?utf-8?B?SUxMcTZNTHdXbm81WWpNcjllQVR0NjlyaENQMzZlV1h5eVJYVDN5WXJKSloz?=
 =?utf-8?B?aFNLUGlhdmpLR1AxeDkyYkZKTzRnTUc4NlhpVTNZNWI2cEhFT0tyWGpKcWxk?=
 =?utf-8?B?NmozT1BDekVWOWlGckxXcnlIWnBjNm03S2xRemhvOWRUQ1ZTTnBxbWV1YUJI?=
 =?utf-8?B?MHFoVUcrVllqV3hsdGhMc1FWU09lSlJRY2Z4TTZNVGNtQXFPb3IxOXg3SnRK?=
 =?utf-8?B?SWFiUWZteUpUa2lGSEFJMURuajJCeEdxanU1RjB6dTJrMDdpaFNLbzBqQVly?=
 =?utf-8?B?VmkrOGpvTEhXQlZySjNUaE5IN016bExuSXovSEF4NmJUNmtia3JmbE95Q2tr?=
 =?utf-8?B?TWNCNy9kYk56NTdGK3FEUnRzbWUzTWkrQUlNMzVablBNTklNOUdUNzlqOXhx?=
 =?utf-8?B?RElZSjhJNy95KzZ3aXhQUksxejhFTmZXeXJscFpvNm1DM1hzWW9jc1JWRDBl?=
 =?utf-8?B?azdQdkk0TTBITVphYUhxbTBIOVBVdi9FT0UwNjZoKy82eVMzREdTMHRjVHNv?=
 =?utf-8?B?NnJ4bmUrZ3k2cFdTeWpmNkFRaFZZZWhaa2FZNXFTcFcwMGc4SzJuVXNSa3BW?=
 =?utf-8?B?eXR6d0pWTlkzMkhRcE55Y3A3UmJlWnFHNFZyZTBvL29DM2xKdXU2bUx4dGdu?=
 =?utf-8?B?SUFPS0hwelVWZWZZQXlZWGN2dWJxRWJTa1gvS0JIQnlKZ016YW1DRzdmbE1m?=
 =?utf-8?B?Wm9rYjYyc2NreFJCNDgvNUZ5RzVTckJSaGt0dVF3dWtoU0xBWHNhYUU2MnBh?=
 =?utf-8?B?NDd6Z2t0WUJaQmpVTTN4TjdQdkZnRTlsZDRHVTBTY29PcjZlYlhCSy8wSnhO?=
 =?utf-8?B?NGhDWFhnbXpGUmloQ0tsM1ZPOHI3TXZrL2VKU2g0WHo0Qk5yVFRYdEpwNHJM?=
 =?utf-8?B?TVljY1ZQcE5yRm9TY28rUEYyaXJyVWMzVWhIeXJjUFlESTF0SWowTHcvd2hT?=
 =?utf-8?Q?l2yBKTiUSB+0UFp2bLwQioA=3D?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f2746218-69cf-490e-403a-08dc3860376f
X-MS-Exchange-CrossTenant-AuthSource: BN7PR01MB3730.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Feb 2024 13:21:48.6170
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: vGykPKUBvbl5ksZMQ0CGjgQV+f1NMh/SQVJPBuMrwlaIxOYH0D2UfZtarKcqLnHVCzvIXmVMfCZtIX9akhMmIMKDkLe15bIcAygeeq59cbtaCxtBkqcmyMEtxFW8xpgO
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR01MB6457

On 2/28/2024 4:38 AM, Eric Dumazet wrote:

>>
>> sk_prot->memory_allocated points to global atomic variable:
>> atomic_long_t tcp_memory_allocated ____cacheline_aligned_in_smp;
>>
>> If increasing the per-cpu cache size from 1MB to e.g. 16MB,
>> changes to sk->sk_prot->memory_allocated can be further reduced.
>> Performance may be improved on system with many cores.
> 
> This looks good, do you have any performance numbers to share ?

I ran localhost memcached test on system with 320 CPU threads,
perf shows 4% cycles spent in __sk_mem_raise_allocated() -->sk_memory_allocated().
If increasing SK_MEMORY_PCPU_RESERV to 16MB, perf cycles spent in
__sk_mem_raise_allocated() drops to 0.4%.

Thanks,
-adam