Received-SPF: pass (google.com: domain of linux-kernel+bounces-29731-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <1cfdf574-90cf-4143-b735-7b8354098e6d@linux.dev>
Date: Thu, 18 Jan 2024 14:15:24 +0800
Precedence: bulk
MIME-Version: 1.0
Subject: Re: [PATCH v3 3/7] padata: dispatch works on different nodes
Content-Language: en-US
To: Tim Chen <tim.c.chen@linux.intel.com>
Cc: linux-mm@kvack.org, Andrew Morton <akpm@linux-foundation.org>,
 Mike Kravetz <mike.kravetz@oracle.com>, David Rientjes
 <rientjes@google.com>, linux-kernel@vger.kernel.org,
 ligang.bdlg@bytedance.com, David Hildenbrand <david@redhat.com>,
 Muchun Song <muchun.song@linux.dev>, Gang Li <gang.li@linux.dev>
References: <20240102131249.76622-1-gang.li@linux.dev>
 <20240102131249.76622-4-gang.li@linux.dev>
 <1d9074955618ea0b4b155701f7c1b8b18a43fa8d.camel@linux.intel.com>
 <feaf7851-f924-48f4-b16a-2fa5efdb28cf@linux.dev>
 <1bd6ee64a600daad58866ce684b591d39879c470.camel@linux.intel.com>
 <ea4a5417-1fce-4b36-be4d-215086fd7e96@linux.dev>
 <cc135e4ba87bf64b384a529ccbd4c644bb135266.camel@linux.intel.com>
From: Gang Li <gang.li@linux.dev>
In-Reply-To: <cc135e4ba87bf64b384a529ccbd4c644bb135266.camel@linux.intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi Tim,

On 2024/1/18 06:14, Tim Chen wrote:
> On Mon, 2024-01-15 at 16:57 +0800, Gang Li wrote:
>> How about:
>> ```
>> nid = global_nid;
>> list_for_each_entry(pw, &works, pw_list)
>> 	if (job->numa_aware) {
>> 		int old_node = nid;
>> 		queue_work_node(nid, system_unbound_wq, &pw->pw_work);
>> 		nid = next_node(nid, node_states[N_CPU]);
>> 		cmpxchg(&global_nid, old_node, nid);


>> 	} else
>> 		queue_work(system_unbound_wq, &pw->pw_work);
>>
>> ```
>>

My original idea was to have all tasks from a single
padata_do_multithreaded distributed continuously across NUMA nodes.

In that case, the task distribution would be predictable for a single
padata_do_multithreaded call.

> 
> I am thinking something like
> 
> static volatile atomic_t last_used_nid;
> 
> list_for_each_entry(pw, &works, pw_list)
>   	if (job->numa_aware) {
> 		int old_node = atomic_read(&last_used_nid);
> 		
> 		do {
> 			nid = next_node_in(old_node, node_states[N_CPU]);
> 		} while (!atomic_try_cmpxchg(&last_used_nid, &old_node, nid));


However, having the tasks from all parallel padata_do_multithreaded
globally distributed across NUMA nodes is also fine by me.

I don't have a strong preference.

>   		queue_work_node(nid, system_unbound_wq, &pw->pw_work);		
>   	} else {
>   		queue_work(system_unbound_wq, &pw->pw_work);
> 	}
> 
> Note that we need to use next_node_in so we'll wrap around the node mask.
>