Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 14 Aug 2023 17:09:18 +0800
From:   Yan Zhao <yan.y.zhao@intel.com>
To:     John Hubbard <jhubbard@nvidia.com>
CC:     David Hildenbrand <david@redhat.com>, <linux-mm@kvack.org>,
        <linux-kernel@vger.kernel.org>, <kvm@vger.kernel.org>,
        <pbonzini@redhat.com>, <seanjc@google.com>,
        <mike.kravetz@oracle.com>, <apopple@nvidia.com>, <jgg@nvidia.com>,
        <rppt@kernel.org>, <akpm@linux-foundation.org>,
        <kevin.tian@intel.com>, Mel Gorman <mgorman@techsingularity.net>
Subject: Re: [RFC PATCH v2 0/5] Reduce NUMA balance caused TLB-shootdowns in
 a VM
Message-ID: <ZNnvPuRUVsUl5umM@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20230810085636.25914-1-yan.y.zhao@intel.com>
 <41a893e1-f2e7-23f4-cad2-d5c353a336a3@redhat.com>
 <ZNSyzgyTxubo0g/D@yzhao56-desk.sh.intel.com>
 <6b48a161-257b-a02b-c483-87c04b655635@redhat.com>
 <1ad2c33d-95e1-49ec-acd2-ac02b506974e@nvidia.com>
 <846e9117-1f79-a5e0-1b14-3dba91ab8033@redhat.com>
 <d0ad2642-6d72-489e-91af-a7cb15e75a8a@nvidia.com>
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <d0ad2642-6d72-489e-91af-a7cb15e75a8a@nvidia.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?wLF1m8oNMO2zjcwwuPubg+NiWJmx2zL1Td7HB9nnx2pGx8fZ4wRyxBQMTj?=
 =?iso-8859-1?Q?HtsNKNfIQfLKOAy4bmLNWukm3bNwzqnIXxbJsNykF/LpOIxfGJJu4GJf/z?=
 =?iso-8859-1?Q?mr+ldMBCRml3y2406wMcfW33mvwS9LwcUNE0/W/S4CEyrv9hz8RyNJ+PL6?=
 =?iso-8859-1?Q?uSCoUldq0aTtJYTUDyyf9DA70/HF7pbiO4QmHFyVqv91/zRz9N0OGUGeFo?=
 =?iso-8859-1?Q?hiTokF98j6TdySFrsUtctjlauOda6/mLb5P5c0/zmOdbpiqIKUY5y9GwWA?=
 =?iso-8859-1?Q?sPljoZ2s7GbiDbISlFZNeK03wpAdKkpszyTXLUCY9CUNJDUJnsAcN42svH?=
 =?iso-8859-1?Q?V2mSKVMGaCwUWxG0zC5yE8E8QoYvP83HNjn5hiDcsLMLmvwFwAeACPKwAS?=
 =?iso-8859-1?Q?CzrjH5bNj3JX73vdF7VbO+llWHRI0mZIa1Z55An4dtGDDIJZII3TEUpj0O?=
 =?iso-8859-1?Q?gjLP+W0WoNR44X5b/3WJD0HUkQzluyBF0xR+o2Mefg29+w4lZEjF5/GNTe?=
 =?iso-8859-1?Q?neREMKhAqy+kGechwpPUvjLigXDcdjeX+z+sEOoRsyTQqlftAkoUCh1Z6r?=
 =?iso-8859-1?Q?sc6NpbAau2p+YuJJ7TyxXwef9yNg3uNFeMEz1f6lf6PlifvFr8isvXacqp?=
 =?iso-8859-1?Q?EwDfZbK4fEiALI30NwHVK9kDTQ4sDd+DJzQSYc2wlm6ZdRvn/oIT5nC6zO?=
 =?iso-8859-1?Q?Lx6OM0Z4GGBfxfkGyrzP42C2/FeL8tge6cspDhiv8hNcStzu2Zq6GxVLt5?=
 =?iso-8859-1?Q?1lnv9PE+vFiIEBXXUeC8tOhF9Jeb22Sli2WQ8oTMiPKvZD/2zpxizUSAQT?=
 =?iso-8859-1?Q?hWl2UdFwUo/V6WHzECdnmYDMqCK0uUPZ5quiw73v5bQk5ZbJ1GaeprCtqc?=
 =?iso-8859-1?Q?GCIaG4FgIvRvhDeWq2jVjq1x+htSRvukQgeKSzxZ14W4xc5PWQycB5YZ0I?=
 =?iso-8859-1?Q?kRNfsRQ21WfYfDjJNRACKc2ahuOTLiGUNiIPIyzYDd5gjiTYynGjuABtE0?=
 =?iso-8859-1?Q?gvCcDet4PavJO9rYXfHELKJt1jC8N+fYs2qtyS2SUP7gWg7oX9k+qzimaa?=
 =?iso-8859-1?Q?JOIjT97qg8PyyzC/iTaMAl1hDD/Br8Mp1rVV+aUYSL4sGE3dumiHBGmdfN?=
 =?iso-8859-1?Q?R9gL4HRwzkSnhNOeCKNX5Qmne9TVqJ+HOBdvsgswnP4BGKiBTRcOUgml9U?=
 =?iso-8859-1?Q?as5VEXq7tYYRWNSlPfSq41BwN7TN2EXPw5C0kyp1fTe33K/RfdyMd/BB7C?=
 =?iso-8859-1?Q?Bv6d20E4/MfvVKORWC0T0AizsHmPicCor4D3mVeEGPfaPCCTOhVMhMVBkk?=
 =?iso-8859-1?Q?7ObGFRmbkqi11CYncBa64jebOkIU0ivCzJgCs5ZlzlO6WABuZX4J+DR5/t?=
 =?iso-8859-1?Q?aPAzST4wV8gES0QZajEww1D/DxceX9HXtCkEavMJqxxdTqijDt9w/KmnlW?=
 =?iso-8859-1?Q?42tBRbIqwHpY7HAVbGY+ls0QXGkSUPKkTAFDzElmQhq2sXiaFSpOZhZGpd?=
 =?iso-8859-1?Q?YPABhMsJ+2NQxFhtlqu5eYAbVP0Jlhq23JYD+dgQoPtsHgwslnl8Jm+/cy?=
 =?iso-8859-1?Q?fabuanycQSna7hdY8cnsLcV0Lt3xadk97dOSMfe0YwNgnqH0L7ONrEcVEo?=
 =?iso-8859-1?Q?n9vnlA/7ovkdUN6ijaVkHw4iRTicbu0nGs?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ccc647d1-f269-4f3c-101d-08db9ca9e9d3
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2023 09:36:19.0984
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 25UvOCujL595U+JvvYxoBvAamS4YQYNMUfay+62/GVRkqhXXffxyyGHKv46inWUXyn8yP9JtepHz9YU9ad5ygg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6514
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Aug 11, 2023 at 12:35:27PM -0700, John Hubbard wrote:
> On 8/11/23 11:39, David Hildenbrand wrote:
> ...
> > > > Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are
> > > similar issues with GPU memory:? NUMA hinting is actually counter-productive and they end up disabling it.
> > > > 
> > > 
> > > Yes, NUMA balancing is incredibly harmful to performance, for GPU and
> > > accelerators that map memory...and VMs as well, it seems. Basically,
> > > anything that has its own processors and page tables needs to be left
> > > strictly alone by NUMA balancing. Because the kernel is (still, even
> > > today) unaware of what those processors are doing, and so it has no way
> > > to do productive NUMA balancing.
> > 
> > Is there any existing way we could handle that better on a per-VMA level, or on the process level? Any magic toggles?
> > 
> > MMF_HAS_PINNED might be too restrictive. MMF_HAS_PINNED_LONGTERM might be better, but with things like iouring still too restrictive eventually.
> > 
> > I recall that setting a mempolicy could prevent auto-numa from getting active, but that might be undesired.
> > 
> > CCing Mel.
> > 
> 
> Let's discern between page pinning situations, and HMM-style situations.
> Page pinning of CPU memory is unnecessary when setting up for using that
> memory by modern GPUs or accelerators, because the latter can handle
> replayable page faults. So for such cases, the pages are in use by a GPU
> or accelerator, but unpinned.
> 
> The performance problem occurs because for those pages, the NUMA
> balancing causes unmapping, which generates callbacks to the device
> driver, which dutifully unmaps the pages from the GPU or accelerator,
> even if the GPU might be busy using those pages. The device promptly
> causes a device page fault, and the driver then re-establishes the
> device page table mapping, which is good until the next round of
> unmapping from the NUMA balancer.
> 
> hmm_range_fault()-based memory management in particular might benefit
> from having NUMA balancing disabled entirely for the memremap_pages()
> region, come to think of it. That seems relatively easy and clean at
> first glance anyway.
> 
> For other regions (allocated by the device driver), a per-VMA flag
> seems about right: VM_NO_NUMA_BALANCING ?
> 
Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard in mmap() in QEMU because a VMA may not be used for DMA until
after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?