Message-ID: <ccba1a65-fe4f-89d5-a32b-2efba30a1350@amd.com>
Date:   Tue, 7 Feb 2023 12:11:47 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.13.1
Subject: Re: [PATCH V2 2/3] sched/numa: Enhance vma scanning logic
Content-Language: en-US
From:   Raghavendra K T <raghavendra.kt@amd.com>
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        Ingo Molnar <mingo@redhat.com>, Mel Gorman <mgorman@suse.de>,
        Andrew Morton <akpm@linux-foundation.org>,
        David Hildenbrand <david@redhat.com>, rppt@kernel.org,
        Bharata B Rao <bharata@amd.com>,
        Disha Talreja <dishaa.talreja@amd.com>
References: <cover.1675159422.git.raghavendra.kt@amd.com>
 <5f0872657ddb164aa047a2231f8dc1086fe6adf6.1675159422.git.raghavendra.kt@amd.com>
 <Y9zs5A/T1WQpJTuM@hirez.programming.kicks-ass.net>
 <048964e8-179f-de7b-1190-831779d9911f@amd.com>
In-Reply-To: <048964e8-179f-de7b-1190-831779d9911f@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bHd6RHd0VUZWN1FLeEhRZERpVURGVndWc001bnBUQ1kzNGxzMUxyZUpWaXRE?=
 =?utf-8?B?VFgwVDh0dGpMeTV0eEE0UnZ0Q3BBQkZqRG5xR09FMFVtL3UwaGFoZkNnZ2Q4?=
 =?utf-8?B?bUhUOEsrdyt2WW5oa1J2S0s1Y09qU0w4QWhRR2FDZXBuNnl5cHRSYmM4VWxz?=
 =?utf-8?B?dWd4alg0VGpXL0V5KzFoV3M0WDJEWFNxd1RrR2dsbEFJN09zVVduS3VtVnRI?=
 =?utf-8?B?UStmY0tsQkd0Q3k3aDUzcWlIWlRtZVNDM2NxU3RwREMzdmVuQTVkZnNQd1JN?=
 =?utf-8?B?WWxodk9zVU52QjZrYjN2aFh2L1NZaW5EOXVxR2kxdHdhWXRxYUE5aDNLKzE2?=
 =?utf-8?B?eXpBWnJMOURocTUxNlRuY3ZRVlNuR2JneTg4SHQ0WnlKUzhsaGpYZ2N4RGFM?=
 =?utf-8?B?UzBTOHF2TE5NdzR5a0lmYm9xQ0xIajBDVHVtd25Bdzk4SlVMcW85Y2hmRVhF?=
 =?utf-8?B?QWxXMjBveHlIYWVJN3lJbFV3cmVubGsvS3NKVHBHaGxoTG1neWJMQk16MVpZ?=
 =?utf-8?B?S1E3SCtGZmxVQkp1MGlELzd2VzdoeGhqZUFndnZDZ2kxN2thSEp6LzVnZFA0?=
 =?utf-8?B?T1Vxb1RKbklNZG84OEpLM20rTDRPb0sxSXQ4V1JTYUROZllGMHROUE1JUmNp?=
 =?utf-8?B?TGN3QW8yZjFneEUxVVJ5YUFxLzVCM1hBQ3RFMTFlNnFwcElwK3g5dFljYnZu?=
 =?utf-8?B?cHVyL3RtendwRExXSnhyaDVXU3dDclhsdGNCOUx6V1VrR1pJUGJQcklqQUkx?=
 =?utf-8?B?UXRKM05ZVnJSbFA2MEtUYWNvWGpxcDJHY29iK0VpQzJuekhkVUZiRGo2dVdR?=
 =?utf-8?B?VVc0d2NZR0F3aVdDaHRTSWMzOXNmWWZDOXBqN0VrdU9jQTBUSmdFUDROYk03?=
 =?utf-8?B?d05FRllFVHZhYTRkMXZZcG40ZUd1elhVUGY3Z3p2dzVYUEZRc2xIVk5IMmhx?=
 =?utf-8?B?Y3NiVUwxWmFqcXFJRGdqbk1Takdrc2hDY0RwVzAxQ040ek4xOS8xazJRV2xl?=
 =?utf-8?B?ZEl2NXhQUFlxSUR3UURqczhnTFJMMkREVmt1S2R5YkdSazk3NUswYklLVnBO?=
 =?utf-8?B?MGIrWm95WGJ4NjV0WmV0bWlza1JHWThsL0o3ZjRQTnE2MVF0VWpTRlJpUnkr?=
 =?utf-8?B?cWd3L05XaCtHczNZM2I4d0psZVNiK2xXRUdHM1Nwd1cyMWJ3bm11NERLTDQy?=
 =?utf-8?B?bEpSeFo1Vk0wODM2Wk0vbE9makY5TUVqU1ZicDBjSzVPUnRCTFJRMG1aVWs2?=
 =?utf-8?B?cVJibzV4UE1ldmduS1dQYTJYLzMzZjh3YkdROVA1SjY3d0VKeDFFWnpWMEVp?=
 =?utf-8?B?SDF0MkNqVlVjNWxna1c3YlBPYjdsZXhUM1F1V1g1WnI1eHQvRXBaV09mcXNS?=
 =?utf-8?B?MUh4KzZKbUo3ZHVudWQ5QlNSeWNzaEprbVdjTlgwZzdxMkowczRXeThNV1pX?=
 =?utf-8?B?VG1jb2tQZmV3SXU3NW5mQlJVK2JtbXFqZmZ4U1BxeWtmdFI3ZjBBUXBwTVhT?=
 =?utf-8?B?VmFJWkZzbHd4VVVGYlFNc29xNDE1ZzZqVFZhY0hRZzdsM3g2b2dNVDluQTNa?=
 =?utf-8?B?WjY1U3Y2MDdUcVZjMUIxbWpxTXI2cDhmbW9BQTVZNm9JYnY0TUtJZ3c3VURP?=
 =?utf-8?B?SWgzeXN1VzNhOXR0Z1p2TUpEbUUxVXRRemNrWEdPUlM4Uis4TTBCMUluT29J?=
 =?utf-8?B?QUk0WUhqK2ZxS25QaE4wY1JIb05mQURvVVhQTjNNejQ5YkxjTTRhSnQyZE45?=
 =?utf-8?B?RS9hY3lIcjlZQkkveEZEOGJycmxiQko0aDVoRzU3R25sSlR0STdZNXpmSHlG?=
 =?utf-8?B?WTEzMEtyOFR5czhtYjVMNDZxQlhQSVY3b2IzTXNQSXl1YnpIa3lBeE1rOTBH?=
 =?utf-8?B?cis3SU5VbkhSUndjZWxxWlBsV2Y3bk1tcENBMENOdlVza0tmVkUxclBrZWRl?=
 =?utf-8?B?WkgwRm1aMjNqb1kxdU1uOSt5MnF2bm5pTlZMMm9QTEU1N1d3Q0ZkRUhvVk4r?=
 =?utf-8?B?Nm9SdmhFVUMrZXVYYTNYbVVudkdUcTR3dmc5MnFINkZwVHU4TjRsWGgzTHdJ?=
 =?utf-8?B?MlUvcHBBTFVOQW50WE5UNjlnOHZwWmpkSlBFWStNVVUwTFRhcWk2KzJSL0VB?=
 =?utf-8?Q?37a3TtA8u8pg9us+FlJiyQnag?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c9d4188c-2e63-40fa-b8a0-08db08d668bf
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB3008.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2023 06:41:58.0381
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: W7BpkYiKcdrgG2X4ESBWxw9vxAV7EpI5U8k2xhSTlgURgbB6IAaFNGV66SynNdB9T6QccKypF6N2klXiC+Xjaw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB8430
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2/4/2023 11:44 PM, Raghavendra K T wrote:
> On 2/3/2023 4:45 PM, Peter Zijlstra wrote:
>> On Wed, Feb 01, 2023 at 01:32:21PM +0530, Raghavendra K T wrote:
[...]
> 
>>> +        if (!vma_is_accessed(vma))
>>> +            continue;
>>> +
>>>           do {
>>>               start = max(start, vma->vm_start);
>>>               end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
>>
>>
>> This feels wrong, specifically we track numa_scan_offset per mm, now, if
>> we divide the threads into two dis-joint groups each only using their
>> own set of vmas (in fact quite common for workloads with proper data
>> partitioning) it is possible to consistently sample one set of threads
>> and thus not scan the other set of vmas.
>>
>> It seems somewhat unlikely, but not impossible to create significant
>> unfairness.
>>
> 
> Agree, But that is the reason why we want to allow first few
> unconditional scans Or am I missing something?
> 

Thinking further, may be we can summarize the different aspects of 
thread/ two disjoint set case itself into:

1) Unfairness because of way in which threads gets opportunity
to scan.

2) Disjoint set of vmas in the partition set could be of different sizes

3) Disjoint set of vmas could be associated with different number of
threads

Each of above can potentially help or make some thread do heavy lifting

but (2), and (3). is what I think we are trying to be Okay with by
making sure tasks mostly do not scan others' vmas

(1) could be a real issue (though I know that there are many places we
  have corrected the issue by introducing offset in p->numa_next_scan)
but how the distribution looks now practically, I take it as a TODO and
post.