Received-SPF: pass (google.com: domain of linux-kernel+bounces-72576-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <c2bd73b6-b21f-4ad8-a176-eec677bc6cf3@amperemail.onmicrosoft.com>
Date: Tue, 20 Feb 2024 16:37:18 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] net: skbuff: allocate the fclone in the current NUMA node
To: Eric Dumazet <edumazet@google.com>
Cc: Huang Shijie <shijie@os.amperecomputing.com>, kuba@kernel.org,
 patches@amperecomputing.com, davem@davemloft.net, horms@kernel.org,
 ast@kernel.org, dhowells@redhat.com, linyunsheng@huawei.com,
 aleksander.lobakin@intel.com, linux-kernel@vger.kernel.org,
 netdev@vger.kernel.org, cl@os.amperecomputing.com
References: <20240220021804.9541-1-shijie@os.amperecomputing.com>
 <CANn89iJoHDzfYfhcwVvR4m7DiVG-UfFNqm+D1WD-2wjOttk6ew@mail.gmail.com>
 <bea860f8-a196-4dff-a655-4da920e2ebfa@amperemail.onmicrosoft.com>
 <CANn89i+1uMAL_025rNc3C1Ut-E5S8Nat6KhKEzcFeC1xxcFWaA@mail.gmail.com>
From: Shijie Huang <shijie@amperemail.onmicrosoft.com>
In-Reply-To: <CANn89i+1uMAL_025rNc3C1Ut-E5S8Nat6KhKEzcFeC1xxcFWaA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?ajJVOHoxTld4SXNkaGpsTm03a1lGS0dDVkxPdG5MZ0Qwdm9UWERJekpPNUhK?=
 =?utf-8?B?ODYxYUxDWDhtSDJXSVo5alFMdkpZc2prK25LVzFUMFN3Yi9Qd2tiakhUQ1kr?=
 =?utf-8?B?WkJJN0kwSkNUR0ZJMlFIS0ZPcFY0R05vbTFBRG5uNHI3S1lxVkF0eis2SGJV?=
 =?utf-8?B?UjlNRjFvMkUvUDI1UTVrRlFsOVJEK0oyN0VxL1JQSlJycGg0MGJOMWJGalo1?=
 =?utf-8?B?cUJaTFZwd2hzNVpvdXN2WTZLL2xYMU9RTldGaU1FSW1BcEFhNFV5ckhndHhk?=
 =?utf-8?B?OEdKdGNIWFB1WThmSmlZMnQ1R1hnaGJvOVFTVWtOYWtvSDY0NGRZOEVWVUg0?=
 =?utf-8?B?OEJONHowbGFiMzFVYzJrMkpNVUxxUFpHTEVnVUFCbU5ScmZSZmVXRkxHWXdI?=
 =?utf-8?B?R2JwSEkwcTZQcFFEalgzYVlIVXFmQWV4YnpSTjVXbmN1dXhKc1h5ZGRiSkoz?=
 =?utf-8?B?NG5zcURuM3p4aXo5elBiYWJ0M3JhNjNmbE5zcUxWeXBDY0lNR3VpN0hSd1Ux?=
 =?utf-8?B?TUMwOElGVWFERHorNXl6OW5ubk9SV3k4RkJlSjRHNW9FMUhUaHBENXZCeksw?=
 =?utf-8?B?MGc4bnNQRFdGc1V4Tko0a2hpK3MzcnJlM2dva2hNY0gyb3hyek8rUlVab3Uy?=
 =?utf-8?B?eWRCNUlqWjhsZG5ML0tLVDhPYk5aVHV5N0tzSzF1S20ybHdCRENqc2wrYzE5?=
 =?utf-8?B?bU9pWjRwNmNDalN2QytpNlBlUUZCRGdqRnE1UW52d1o5NFRmN0laMDFTU2Fs?=
 =?utf-8?B?QSszdy8yM3dkRFk3WHp3VnBIc1NVQU1ySit5YWM0RHdWOXUwdThiVzZIVnlz?=
 =?utf-8?B?U0VUQWk5MWxIV24vMkJ3SkNoT0NDc1dPTGtDTlpTWEdWczg1Uk91clF2Mmdi?=
 =?utf-8?B?LzBGc3NsS2lCbHRHSGdBWTBUR0F2Sjk0THZ0dVJoV3Exbm1QS1RuY2wyY2Iy?=
 =?utf-8?B?N1FFbWtNbDlGN0t5UkM2K1pzU0tiMUtGY2tINU1IUUU1bE12bUhTczFxbll3?=
 =?utf-8?B?QjdzY0NBQlQra1YxQTRiSU5DdkZGZnh3emVubXVxZXcyUmlINjZaMTJwVWxC?=
 =?utf-8?B?cjM0R2I2c1RIWjRNdVJRdmphUGFRSGNqNXRIKzUwYjFxVm01SmFvQWZTQlQz?=
 =?utf-8?B?eXdYMjkyZlRHUEJQcnBPK2Z3M3BYL3NWdmdiOUNsdk1KVXltTjNvV1p0bnZz?=
 =?utf-8?B?aXIyYVlBa3ErbDRpWkZ0L3hHVjEzak5BY0psYmY1MXlQR3R4bmtyTmgwVW1o?=
 =?utf-8?B?UVNtczNKM3FLanhYU00xR3RNdVJHNDVQc3dXSVpBUG9TWGNtajRpRm5ZdTl0?=
 =?utf-8?B?R0tOVnVNcUJnMFd3cEU0SXA3cVRCQUdUNTEzckRVK2hWRndySnQ3VzV1ZStz?=
 =?utf-8?B?OXRlVGh0RW54c1NsblRtSHNqSzB4enB0UzN6eGlVdHJlQURmK3lldG90aS80?=
 =?utf-8?B?TmJZY1poMU0zd3d2QWttWU0xaEw1cmU5VFRpQnQ4TnFnOG5KQWp0am00MGdo?=
 =?utf-8?B?NnIrcEJSeHdLSk1LWVBaajUybDRFcVNjWUhubk51UG9SMm90S21YQTVaWWpr?=
 =?utf-8?B?eXMxWUF6RkhRSFYzYmpuVnJ1MEVLWFVWbHdlbEMxeFl3WGxGMUxoaXU1TGQ0?=
 =?utf-8?B?QkdmNWNHYlpOOTlpV1ptdHNlT3ZiZ0VrT2tuSGZ1cC9yNHNnWHJRczF0Z2Jv?=
 =?utf-8?B?SGR0ZUZlVnhBT2FsQ3I2UTFGenU1NDhVcTBvU1d1Z1dsd2pLY0M1djRYZ0xE?=
 =?utf-8?B?THFuRHE5S210N1RrV3BNTE4yeDE0ZW4yb2xUNlhCT1U4VEowSlZDU0p1N043?=
 =?utf-8?B?T1JGTVFRQitoMVBBaEVqT3lTU1loUmIzb0MxNHhYUWNaLy9rZVhWSnczd1JH?=
 =?utf-8?B?K2duL2VuTFNWbGphanp4RVQ1RW52c05SdTBoM2ZvTkt5TmxIMGF3WmdsWDJS?=
 =?utf-8?B?dmtiMzJLZDgxTjY2eHppYlR0V3pROC92bzg0bnlsQXM0RUNBOEtNb2Z1QUNV?=
 =?utf-8?B?Y1hWWGFHUElQYnRjcEcxb2xUb2xGTzRUNUhlbTI4dGh4VHhVb1pwYXJ2UjVU?=
 =?utf-8?B?UUJ1Mmw2c3hIYTdzbG9iQlBHNUVxNk5TRkFzY0RUeSt4VkFTd0t6VUR5cVZ1?=
 =?utf-8?B?alJSeEdhZ0pQcFRjWHFLVGp0RXlvRERCR04zRWIyYllDS1dCUHRaTHNTejMr?=
 =?utf-8?B?NlE9PQ==?=
X-OriginatorOrg: amperemail.onmicrosoft.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 47e25453-0f5a-4658-af1f-08dc31ef2be9
X-MS-Exchange-CrossTenant-AuthSource: PH0PR01MB7975.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2024 08:37:28.5897
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 6YwmD1l5N40Mer2iMLmpZ/lFsPWqaBpTNj5QMnn3QqdXsbTwF9wbZtKO3NuDFGYAbsGgjjXIpr1Nu6pk1sYcmYVpyjkQOxuCS+bGjE+mJVS0WAy8vp0FUc8vw7Dq0tv6
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR01MB6905


在 2024/2/20 16:17, Eric Dumazet 写道:
> On Tue, Feb 20, 2024 at 7:26 AM Shijie Huang
> <shijie@amperemail.onmicrosoft.com> wrote:
>>
>> 在 2024/2/20 13:32, Eric Dumazet 写道:
>>> On Tue, Feb 20, 2024 at 3:18 AM Huang Shijie
>>> <shijie@os.amperecomputing.com> wrote:
>>>> The current code passes NUMA_NO_NODE to __alloc_skb(), we found
>>>> it may creates fclone SKB in remote NUMA node.
>>> This is intended (WAI)
>> Okay. thanks a lot.
>>
>> It seems I should fix the issue in other code, not the networking.
>>
>>> What about the NUMA policies of the current thread ?
>> We use "numactl -m 0" for memcached, the NUMA policy should allocate
>> fclone in
>>
>> node 0, but we can see many fclones were allocated in node 1.
>>
>> We have enough memory to allocate these fclones in node 0.
>>
>>> Has NUMA_NO_NODE behavior changed recently?
>> I guess not.
>>> What means : "it may creates" ? Please be more specific.
>> When we use the memcached for testing in NUMA, there are maybe 20% ~ 30%
>> fclones were allocated in
>>
>> remote NUMA node.
> Interesting, how was it measured exactly ?

I created a private patch to record the status for each fclone allocation.


> Are you using SLUB or SLAB ?

I think I use SLUB. (CONFIG_SLUB=y, 
CONFIG_SLAB_MERGE_DEFAULT=y,CONFIG_SLUB_CPU_PARTIAL=y)


Thanks

Huang Shijie