Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 10 Aug 2023 21:58:43 +0800
From:   Chao Gao <chao.gao@intel.com>
To:     Yan Zhao <yan.y.zhao@intel.com>
CC:     <linux-mm@kvack.org>, <linux-kernel@vger.kernel.org>,
        <kvm@vger.kernel.org>, <pbonzini@redhat.com>, <seanjc@google.com>,
        <mike.kravetz@oracle.com>, <apopple@nvidia.com>, <jgg@nvidia.com>,
        <rppt@kernel.org>, <akpm@linux-foundation.org>,
        <kevin.tian@intel.com>, <david@redhat.com>
Subject: Re: [RFC PATCH v2 0/5] Reduce NUMA balance caused TLB-shootdowns in
 a VM
Message-ID: <ZNTtExiPZx4b180d@chao-email>
References: <20230810085636.25914-1-yan.y.zhao@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230810085636.25914-1-yan.y.zhao@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?pCJf86msKYrlTsEzeSXh8QXQ1Dv8ZNbjj2lAyyx58d4ILZ9Z915VaJ5rvkyn?=
 =?us-ascii?Q?IuEObGF7OTwT4F8PA89Q8Alu3QZpjsjYKcGfkUcGo/6AM5oDb80bTjB5XYDZ?=
 =?us-ascii?Q?pYwn7nGqlNT9SYx2spC2qYcCZDh/3HC0OYpl9fiA740ZS0Zcqn5n5NVwUT19?=
 =?us-ascii?Q?/GjemKXwjfTYkzK/j8jP7iSIaow6V0kXHdJbC6Mi9fhyvIS3mz4xMeTH8zI7?=
 =?us-ascii?Q?Z5R2Z2cple78O9dgYgF46JrniYdNT4RK1NPDesHt4rdAsNx294bYKCjk+69b?=
 =?us-ascii?Q?+E9lrAts2PeDOrS4dyD/XjA5XYhBSqM2mYagTrA41k3y++WA+blwvubrCXoH?=
 =?us-ascii?Q?s7eiN59atlQZ7XH4Yer+lkWBz+3+A8FNqmqNaE90wdCxfzOWh8di/iPY2lA3?=
 =?us-ascii?Q?Vpw2ieVIjsiY9mqkegaC4O4Y5PXHg6caK/rFjr6LmssWfabDkqXmm28nwAHm?=
 =?us-ascii?Q?/uwe55L+nPvPi7l0SKEmdfXxFgbHf6EcQ3TO3cfjzvWNFiN2XC8y042ntRUr?=
 =?us-ascii?Q?1zdOPHI2pkNde2Ym8/TWf3pRUpfRR1A6o6ABSWARkMG/PtBP1izkXZaxlg42?=
 =?us-ascii?Q?BCtHpzKdo6aEBtutf54eQ8qck/jqIb+/qUgFsZO4q5v6Nbta0dW7jfxioBwm?=
 =?us-ascii?Q?fXEfq2mhSj22IgTcj7JrItmd5/gM8Rn6Mcdnkm7Y3TLFXDr6C8KCf8dTNl5q?=
 =?us-ascii?Q?36OTqojrgGUJfPauPwo1MP+i/dpMsi20V57CvibfWj3KIUEza7BWTK9dueEf?=
 =?us-ascii?Q?EXf74i41SjWLW5yha4EAMcZxzhPYNwGN7e1KyRd3bZ/y1XTFRbRYbOCqtFM5?=
 =?us-ascii?Q?waBHE+r0JgD09Gy0dV79N+pqli7Mko8FaSx0WuWoIS9/xpCaaNaaBQ6nGLyl?=
 =?us-ascii?Q?fX5hZCfA0doTFdoMNgMS8O/hvk5BeG3FiSVG8l1lti3wxBbL0emTXHxwdmZo?=
 =?us-ascii?Q?VCQoW0kmQQn30AoOckpZ1dRCGVWGt82Af09hGgOZIQBrIn8LOWpTRZ/ZBMLa?=
 =?us-ascii?Q?NYFdMYbbK6CYaAca8AzEt/hNYXg9NwDI2xg0MEDhSJ3gQ2E94YSfAkzM/GEf?=
 =?us-ascii?Q?8f5xgnhK6b8bseN7uRVEXugEUsB08RETx7KfXp4Co05EWAYf3XlmFwEtB/4o?=
 =?us-ascii?Q?P0z+bYxc6buWAaXw3M6iieaC4MxqUav1CJyJKMVysH0IsSA8+MzTILATN3z2?=
 =?us-ascii?Q?Nsfj9gCz+Isbg/BdGzc5+4I10vHIMBfMvyba6fr+RYCa2e/PACLQNauFN9mx?=
 =?us-ascii?Q?Nq54GvDftp6N/ZsGAlL9AguMF3TS4jsPf0Y6DlCT9SV0bjAOaHKgg2HPJAI1?=
 =?us-ascii?Q?4eGXZvU5yUqFs//XQOl588AEJq8If09/HpRRq6aqjL6Bg8HLXYZPDLL6H6zr?=
 =?us-ascii?Q?7pbjkiUDadxnM5xDT32wbyWKkUl0I9p+2JdcqWJsRRHOiS+nIUC7K4whtCGw?=
 =?us-ascii?Q?qSysY0dyr1FRnZm84w/9rViCvyrdFN3AoTlrCQOjMxGWLnnZu5Lezv6ZPf7B?=
 =?us-ascii?Q?cCQM0NpUv64MNxo1pAuD0j5VCygJXxvOBkcBua3puTxNpr0CtVGA0dZssb+b?=
 =?us-ascii?Q?jL6RtiV1KzWqu7Exb05FH2z7qlIUmT844b1uswD3?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5d0b7d74-922d-48cc-e011-08db99a9f049
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB6780.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Aug 2023 13:58:56.4430
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: A2Jzobd7vFJQ9mZmsuCDou4trXiCBJ2lV5OIOLGz77R9DILXVDRE9PjMLnQ74iqCxkZZpWOgpCyxkvh+hppSHg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB6822
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
        RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Aug 10, 2023 at 04:56:36PM +0800, Yan Zhao wrote:
>This is an RFC series trying to fix the issue of unnecessary NUMA
>protection and TLB-shootdowns found in VMs with assigned devices or VFIO
>mediated devices during NUMA balance.
>
>For VMs with assigned devices or VFIO mediated devices, all or part of
>guest memory are pinned for long-term.
>
>Auto NUMA balancing will periodically selects VMAs of a process and change
>protections to PROT_NONE even though some or all pages in the selected
>ranges are long-term pinned for DMAs, which is true for VMs with assigned
>devices or VFIO mediated devices.
>
>Though this will not cause real problem because NUMA migration will
>ultimately reject migration of those kind of pages and restore those
>PROT_NONE PTEs, it causes KVM's secondary MMU to be zapped periodically
>with equal SPTEs finally faulted back, wasting CPU cycles and generating
>unnecessary TLB-shootdowns.

In my understanding, NUMA balancing also moves tasks closer to the memory
they are accessing. Can this still work with this series applied?

>
>This series first introduces a new flag MMU_NOTIFIER_RANGE_NUMA in patch 1
>to work with mmu notifier event type MMU_NOTIFY_PROTECTION_VMA, so that
>the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation
>event is sent for NUMA migration purpose in specific.
>
>Patch 2 skips setting PROT_NONE to long-term pinned pages in the primary
>MMU to avoid NUMA protection introduced page faults and restoration of old
>huge PMDs/PTEs in primary MMU.
>
>Patch 3 introduces a new mmu notifier callback .numa_protect(), which
>will be called in patch 4 when a page is ensured to be PROT_NONE protected.
>
>Then in patch 5, KVM can recognize a .invalidate_range_start() notification
>is for NUMA balancing specific and do not do the page unmap in secondary
>MMU until .numa_protect() comes.
>
>
>Changelog:
>RFC v1 --> v2:
>1. added patch 3-4 to introduce a new callback .numa_protect()
>2. Rather than have KVM duplicate logic to check if a page is pinned for
>long-term, let KVM depend on the new callback .numa_protect() to do the
>page unmap in secondary MMU for NUMA migration purpose.
>
>RFC v1:
>https://lore.kernel.org/all/20230808071329.19995-1-yan.y.zhao@intel.com/ 
>
>Yan Zhao (5):
>  mm/mmu_notifier: introduce a new mmu notifier flag
>    MMU_NOTIFIER_RANGE_NUMA
>  mm: don't set PROT_NONE to maybe-dma-pinned pages for NUMA-migrate
>    purpose
>  mm/mmu_notifier: introduce a new callback .numa_protect
>  mm/autonuma: call .numa_protect() when page is protected for NUMA
>    migrate
>  KVM: Unmap pages only when it's indeed protected for NUMA migration
>
> include/linux/mmu_notifier.h | 16 ++++++++++++++++
> mm/huge_memory.c             |  6 ++++++
> mm/mmu_notifier.c            | 18 ++++++++++++++++++
> mm/mprotect.c                | 10 +++++++++-
> virt/kvm/kvm_main.c          | 25 ++++++++++++++++++++++---
> 5 files changed, 71 insertions(+), 4 deletions(-)
>
>-- 
>2.17.1
>