Message-ID: <3f6624cc-2f7e-f830-eff5-173548d529e0@amd.com>
Date:   Mon, 20 Feb 2023 18:38:08 -0600
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.2.1
Subject: Re: [RFC PATCH V1 1/1] sched/numa: Enhance vma scanning logic
Content-Language: en-US
To:     Bharata B Rao <bharata@amd.com>, Mel Gorman <mgorman@suse.de>,
        Raghavendra K T <raghavendra.kt@amd.com>, mizhang@google.com
Cc:     linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Matthew Wilcox <willy@infradead.org>,
        Vlastimil Babka <vbabka@suse.cz>,
        "Liam R . Howlett" <Liam.Howlett@Oracle.com>,
        Peter Xu <peterx@redhat.com>,
        David Hildenbrand <david@redhat.com>,
        xu xin <cgel.zte@gmail.com>, Yu Zhao <yuzhao@google.com>,
        Colin Cross <ccross@google.com>, Arnd Bergmann <arnd@arndb.de>,
        Hugh Dickins <hughd@google.com>,
        Disha Talreja <dishaa.talreja@amd.com>,
        Sean Christopherson <seanjc@google.com>, jhubbard@nvidia.com,
        ligang.bdlg@bytedance.com
References: <cover.1673610485.git.raghavendra.kt@amd.com>
 <67bf778d592c39d02444825c416c2ed11d2ef4b2.1673610485.git.raghavendra.kt@amd.com>
 <20230117145951.s2jmva4v54lfrhds@suse.de>
 <933d1843-be6c-cbd4-ffb2-b0adcbeeccd5@amd.com>
From:   "Kalra, Ashish" <ashish.kalra@amd.com>
In-Reply-To: <933d1843-be6c-cbd4-ffb2-b0adcbeeccd5@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cnZ2eTlNQ0dIZ09IendnekdrQ2RBd25BZVNCdjk5VFpKb2R6MngxZlhyNUZG?=
 =?utf-8?B?RllwM1dpaDE2NWVLTW4rdzZ1ZDgvWXZGeTRVbWc3TklXL3BOaDFWcDZwUjcw?=
 =?utf-8?B?V3p3OTY4T2Q0alpJRzlZQ1k4Z0ZOZE45RXJLd0FxZ3FtTVFtQ09Fc3QzcmFR?=
 =?utf-8?B?Sjg3NFh5VkMxckMvWmJJL3RsRms1QnFCcjJEYllTNmpoMjhKbEREekxoZ2VF?=
 =?utf-8?B?dWFoNi9ZMnlLbkpJNEZ2WWpBK1VWYU5NVWNlSkZMMDZPak1IY1VmNlJlREd2?=
 =?utf-8?B?Ym9kOG93OUhYM0kxWHUvb3BIS1A2dGpZM0x3WWl0a3dPMUhXR2ZOc2xxS1Nr?=
 =?utf-8?B?L1loWFJaQU1CcXRUYlNLelJ1ZEZUUHdLSFhTZHpQSFFPSGdKVzN6SmhTWlAv?=
 =?utf-8?B?Z3ZBMjRiemp5QTlnbVNXU29hZmZjck9VZ3BJeTlMcmRiQ3h6SXVQWUViMmpx?=
 =?utf-8?B?dHFNeVNRL0swRHFZaFFhNlF5YllkdmVwcjJra0ZGZWZRMDl0VHF5NGxxanFa?=
 =?utf-8?B?c0o1ODNwaHMrb2paa0tPRTNPaEt0UDA0Vis1czZ0NUJLZHY1MmZ5QlErWXBq?=
 =?utf-8?B?bXVKMWcwMWQxYi9nRWh6N1pEc1N2ZGZ0cVJhaVZLNG9yTWpCcktzaGdteEpG?=
 =?utf-8?B?U01pQnFrbHRtZkdxTGpOV2U3ejlMaUhhVDMzbmRvdDRJcDhUY2h0OTh3K3cw?=
 =?utf-8?B?dTdVaTd3b2Z5TUNaRXhkSVVFWGRGb3gzMDZ4ZDhJN2Y3TEpRUDBSYlN6WGI1?=
 =?utf-8?B?OWZLN2xsaGFIMkZodkF4MnovcTVVdGNDYzFLOVdmNEVZeFpidU1qNHR6N2Rw?=
 =?utf-8?B?RXJDZ3RMY1RZS0M3cnNiZ2tGOXowVlJSdTNUajdMMGVvSjJoeER0c05IV1JD?=
 =?utf-8?B?TSswbGxzQVpxYzNUWjdiYktmeWlobUVUaWVFSHdqdnV2Ry9HbkhHM0UvczJE?=
 =?utf-8?B?a29JSDRRTWRSeWtjQmpSRUxXT09pQ2RuVWZnVTY1ZlJJS1BvVGNDdGZVOXdS?=
 =?utf-8?B?UzBRQWdueGtJZFZjbFJ1VDkxcXF3bC9EeFlIQy8zZ0ZaTVRYTWIrejd6RzBD?=
 =?utf-8?B?RkdVeWJ2MzBsaHZXWVVXa0liNDRHdDVLM1FqOWVzTUVucm14RXlzTlJZQ1Rj?=
 =?utf-8?B?L1lFUlk5SlNBMExhSUxXUjRXcUJrVEZ1L1JTaXdrdGsvWSthUE9rWjhUMjlP?=
 =?utf-8?B?VUVaQTBHRXBZREZpMnNHcGRXNUZBOWIzMVlHQXlkZzRSMFZmSmdKbFpraysv?=
 =?utf-8?B?Y2xqcHZSL2lYSmxyUlJsOG43VTJPUlpWMTlFMGFkNlFsUFZQYVoySi9HYmpx?=
 =?utf-8?B?VGRUZDdWYUYycktDYzJHdkhSay84ZUk4cDFHMVI2NG1HRjNObnlmcm9Bek1K?=
 =?utf-8?B?elJzZ3p2dzlrZlFPYU1vVTFhbE9BSFlLSDRkdDV1bUxGam1wSmZwb0c3QTZl?=
 =?utf-8?B?NGdoY3lrNTZWTEU2dTluOGh1TFFuVit0N2tiTVhCekNHMFFYWlhJdmdKeEpJ?=
 =?utf-8?B?UnJIVFcwcFRJTUpub0RFSXlQRlJsWTRrZk9iUFN1Q2pnNjdKOG5vTm9yTkY0?=
 =?utf-8?B?YXptRS83Tk5FWVhtTGNKaTRQc2VvR1g5cW1qcU9FU3BaNktyT1R5eHBpek14?=
 =?utf-8?B?YUVEbFB1cFEvR1NnRFYwNkpTenJyQ3dZUVUwSE8xcG1aTVFrWHlCUDl1RFNF?=
 =?utf-8?B?Uno0QVZSVlJlU2hrQm9DenBONFMxbDhQUjNIOE1tWnJzK0Z4bENvM0c2bnBD?=
 =?utf-8?B?TW1zeDlxVDdPeXAvYVcyQStKWTcrbkFDQkcvdy9EeDJEcG80OUMyTWtTZGtQ?=
 =?utf-8?B?K3YxR2dVcHQvWEVIYk5rU3dVUGZjYkh6ZTQ3ckFURmVEU3NGYnR4M01BdW1T?=
 =?utf-8?B?WTVIQ1QzRzBkUVdnSmwrS2tnbUVZTzVMS1FLeC9vcjJKWlJkeTRyWjc5WnNt?=
 =?utf-8?B?VzVhZTdhZUxGa3VNTkRZL0FTYmxveXdWUzJuR2QwaVl3QkZHcHRFQ3FJT2F3?=
 =?utf-8?B?YS94T2h0eXNFYU56cmpkUTNSbkovem1qSUVXdHRkeFFUOEt0MnlwYTJ6SmNy?=
 =?utf-8?B?bFNzd0FhRTE0dUJHdm5LL0JnbWk3WE5sTS9lZXJpVkVvQVZxU1B3S21MVWN3?=
 =?utf-8?Q?jABFTdPm3GAMI7KIE7Ho0+Mw2?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e3faaeac-aeb3-4b2a-35b7-08db13a3e9ce
X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB2767.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Feb 2023 00:38:12.8603
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: O3hO930GNgunKge5OP6RbzATg9nIw0fayvJiucKpKTkTncesbRzDbjlIv1Qv1jTb19vki65/CyAutduWcGF7Cg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6622
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hello Mingwei, Sean,

Looking forward to your thoughts/feedback on the MMU invalidation 
notifier issues with SEV guests as mentioned below ?

Thanks,
Ashish

On 1/17/2023 10:43 PM, Bharata B Rao wrote:
> On 1/17/2023 8:29 PM, Mel Gorman wrote:
>> Note that the cc list is excessive for the topic.
> 
> (Wasn't sure about pruning the CC list mid-thread, hence continuing with it)
> 
> <snip>
> 
>>
>> This is a build-tested only prototype to illustrate how VMA could track
>> NUMA balancing state. It starts with applying the scan delay to every VMA
>> instead of every task to avoid scanning new or very short-lived VMAs. I
>> went back to my old notes on how I hoped to reduce excessive scanning in
>> NUMA balancing and it happened to be second on my list and straight-forward
>> to prototype in a few minutes.
> 
> While on the topic of improving NUMA balancer scanning relevancy, the following
> additional points may be worth noting:
> 
> Recently there have been reports about NUMA balancing induced scanning and
> subsequent MMU notifier invalidations causing problems in different scenarios.
> 
> 1. Currently NUMA balancing won't check at scan time, if a page (or a VMA )is
> not migratable since the page (or the address range) is pinned. It will go ahead
> with MMU invalidation notifications and changes the PTE protection to PAGE_NONE
> only to realize later that the pinned pages can't be migrated before reinstalling
> the original PTE.
> 
> This was found to cause issues to SEV guests whose pages are completely pinned.
> This was discussed here - https://lore.kernel.org/all/20220927000729.498292-1-Ashish.Kalra@amd.com/
> 
> We could probably use page_maybe_dma_pinned() to determine if the page is long
> term pinned and avoid MMU invalidation and protection change for such a page.
> However then we would have to do per-page invalidations (as against one time
> PMD range invalidation that is done currently) which is probably not desirable.
> 
> Also MMU invalidations are expected to be issued under sleepable context (mostly
> except in the OOM notification which uses nonblock verion, AFAICS). This makes it
> difficult to check the pinned state of the page prior to MMU invalidation. Some of
> this is discussed here: https://lore.kernel.org/linux-arm-kernel/YuEMkKY2RU%2F2KiZW@monolith.localdoman/
> 
> This current patchset where we attempt to restrict scanning to relevant VMAs may
> help the above case partially, but any ideas on addressing this issue
> comprehensively? It would have been ideal if we could identify such non-migratable
> pages (long term pinned) clearly and avoid them entirely from scanning and protection
> change.
> 
> 2. Applications that run on GPUs may like to avoid the NUMA balancing activity
> completely and they benefit from per-process enabling/disabling of NUMA balancing.
> The patchset (which has a different use case for per-process control) that helps
> this is here - https://lore.kernel.org/all/49ed07b1-e167-7f94-9986-8e86fb60bb09@nvidia.com/
> 
> Improvements to increase the relevant scanning can help this case to an extent
> but per-process NUMA balancing control should be a useful control to have.
> 
> Regards,
> Bharata.
>