Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 11 Aug 2023 11:45:29 +0800
From:   Yan Zhao <yan.y.zhao@intel.com>
To:     bibo mao <maobibo@loongson.cn>
CC:     <linux-mm@kvack.org>, <linux-kernel@vger.kernel.org>,
        <kvm@vger.kernel.org>, <pbonzini@redhat.com>, <seanjc@google.com>,
        <mike.kravetz@oracle.com>, <apopple@nvidia.com>, <jgg@nvidia.com>,
        <rppt@kernel.org>, <akpm@linux-foundation.org>,
        <kevin.tian@intel.com>, <david@redhat.com>
Subject: Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed
 protected for NUMA migration
Message-ID: <ZNWu2YCxy2FQBl4z@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20230810085636.25914-1-yan.y.zhao@intel.com>
 <20230810090218.26244-1-yan.y.zhao@intel.com>
 <277ee023-dc94-6c23-20b2-7deba641f1b1@loongson.cn>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <277ee023-dc94-6c23-20b2-7deba641f1b1@loongson.cn>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ZlP2YB4pCnPAEXiTD+7x1REHtGq4AOB2AWCRtUFSiZ+2ivfomW/EkNHzre/s?=
 =?us-ascii?Q?rVy2/8pZ0+txf8sM/TvZKZZBEyS5ti91y3IuruHqWVdUWmFtvACjfJP0Djke?=
 =?us-ascii?Q?JEuWsIf096WRwv31M+jvKtasohnowu5oT2MCkRaWzbyqXLyOfvdojZot7JzV?=
 =?us-ascii?Q?LsK8+VH3XOPibxmVdwVtwE7FRKSOjbbUVwS2vrcdOJi99iAI8JHGZbBTWkh8?=
 =?us-ascii?Q?LjH9hBajHvTywoogcAFoFjmqFole42VA93XHFOdbTqcoPc2uUOOmeWd04uC/?=
 =?us-ascii?Q?2QHLXiJ5GLm6yqRePRvZ5H1NhONIiU4Bgz1mUGqeBYsblkKMfoIGvmhNu9MZ?=
 =?us-ascii?Q?fv2zeSyaIKA/hAhJTqvrh/uXT4phVczm5kxn6wzc4AICA/+ucZAblY8/JEEw?=
 =?us-ascii?Q?l00CJ4TtajxBNkUxY94GRtOX2DXl9rmH7dxvDh2R0LrDG4yBwaKF+/LKk4Ug?=
 =?us-ascii?Q?ZezjXsrtdwLpOynbili6sETvU146If5pKV7BuEoLovdeUjGLL8xLrYoMcCIa?=
 =?us-ascii?Q?xmA1d+7xMQsu8ms7f7sHS1HYkhPDPIki/WGRI2MLpEV2/mxcuUsJVlY9RQRy?=
 =?us-ascii?Q?5kIFdaP+eVfDrchE71vSU54flkVlTebqIcqA+exPLLjOGks/c9nb4GpfPXuW?=
 =?us-ascii?Q?XOHliNb3TWF8ljb3zppwYuDYqBVwxxmDq4TTR8Zif+4jNBfsVaUnR/jSD1Ej?=
 =?us-ascii?Q?+PQxvi/8gpIM1NGahDuULoPBoSJDy5zMrg1URC/j4wLmZn4JlLHACCbSsV+q?=
 =?us-ascii?Q?xYKQxpmS5kProeXYj1LAFJgrh1awMlYXS8Zdx1cngPFrxH17fWzEF7W+yTwa?=
 =?us-ascii?Q?LNxEFdQjbsv/l26jGQxBbUUsRfuc1EtVhaEiLx01trIduADMVQ8dAURNg2d7?=
 =?us-ascii?Q?1zqZfpKCIdkZxGuaK1UxwR43NwBquwrnu4USNITG4TRBpT3NPjiRHTF6fVGU?=
 =?us-ascii?Q?HTcrvzp29tccr3710ESiXZ1AmVDnHtgxRWUdI/cbjy21NeFDSoX333z4/mib?=
 =?us-ascii?Q?hM+F2aADMbcCyXfZHw4USilt99HepqIEan7xecl0cpmzbaHch7X7Hc1wqzN+?=
 =?us-ascii?Q?8C9unBc5YJEy3fL+au7Fm+YQEIaMbYNXZNdBlESEhinpvYrBQt2uIhmn8yAb?=
 =?us-ascii?Q?rpbtmUZ65ebmqM15feXZdGk0Zeg2RAAHzgHbuwi8eubw4lEyPMEEWcyFIfuT?=
 =?us-ascii?Q?nYK98nhNkuUjKf0OeSrWNX42AMbCufkQcyO6S73NdnM1O4+OH1o/w+b6YWPk?=
 =?us-ascii?Q?lujLm55UBqj5jpE2KPONbPwARyLWrMR7KRpy9hhAzgcW2JasYyK4Xq7yCGKo?=
 =?us-ascii?Q?awfduixAhabpGhRYNl2iCKRPKVCewx1YMNT+ZypMt+dyUVJWXG2xYXJhHPRE?=
 =?us-ascii?Q?0U7PpGbC24uAVvM9ncGsA0ATpHPdFCoSC0B9LZbyBU8aKeGcHUk4JrQO2gsU?=
 =?us-ascii?Q?zCmx6IAzgzkJcM4Si7/suvVlcf8BVAutEdJuShBKHSkXD0DtlTSKtgODm5/q?=
 =?us-ascii?Q?lTN+5ONezKa81NhM9VmekumhiF+xCBQ2boKVo1FXr/8ciwZqVG/JGl/oOWu9?=
 =?us-ascii?Q?1DwCJIZb7bczxpPiA/niiiq/aBizy1dtpA/RPea0?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 4eb1a123-f96b-48e8-20a1-08db9a212d48
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Aug 2023 04:12:28.9225
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Am7wUuUb+ZWscWRylPo6BzE0jhoIEy7xDYhdisDMpuNbV1Smf2LbT4bOn9T6ToSJdagfEKBcTJZ+jo7U6v8b1g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR11MB7653
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> > +static void kvm_mmu_notifier_numa_protect(struct mmu_notifier *mn,
> > +					  struct mm_struct *mm,
> > +					  unsigned long start,
> > +					  unsigned long end)
> > +{
> > +	struct kvm *kvm = mmu_notifier_to_kvm(mn);
> > +
> > +	WARN_ON_ONCE(!READ_ONCE(kvm->mn_active_invalidate_count));
> > +	if (!READ_ONCE(kvm->mmu_invalidate_in_progress))
> > +		return;
> > +
> > +	kvm_handle_hva_range(mn, start, end, __pte(0), kvm_unmap_gfn_range);
> > +}
> numa balance will scan wide memory range, and there will be one time
Though scanning memory range is wide, .invalidate_range_start() is sent
for each 2M range.

> ipi notification with kvm_flush_remote_tlbs. With page level notification,
> it may bring out lots of flush remote tlb ipi notification.

Hmm, for VMs with assigned devices, apparently, the flush remote tlb IPIs
will be reduced to 0 with this series.

For VMs without assigned devices or mdev devices, I was previously also
worried about that there might be more IPIs.
But with current test data, there's no more remote tlb IPIs on average.

The reason is below:

Before this series, kvm_unmap_gfn_range() is called for once for a 2M
range.
After this series, kvm_unmap_gfn_range() is called for once if the 2M is
mapped to a huge page in primary MMU, and called for at most 512 times
if mapped to 4K pages in primary MMU.


Though kvm_unmap_gfn_range() is only called once before this series,
as the range is blockable, when there're contentions, remote tlb IPIs
can be sent page by page in 4K granularity (in tdp_mmu_iter_cond_resched())
if the pages are mapped in 4K in secondary MMU.

With this series, on the other hand, .numa_protect() sets range to be
unblockable. So there could be less remote tlb IPIs when a 2M range is
mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.

Below is my testing data on a VM without assigned devices:
The data is an average of 10 times guest boot-up.
                   
    data           | numa balancing caused  | numa balancing caused    
  on average       | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() 
-------------------|------------------------|--------------------------
before this series |         35             |     8625                 
after  this series |      10037             |     4610                 

For a single guest bootup,
                   | numa balancing caused  | numa balancing caused    
    best  data     | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() 
-------------------|------------------------|--------------------------
before this series |         28             |       13                  
after  this series |        406             |      195                  

                   | numa balancing caused  | numa balancing caused    
   worst  data     | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() 
-------------------|------------------------|--------------------------
before this series |         44             |    43920               
after  this series |      17352             |     8668                 


> 
> however numa balance notification, pmd table of vm maybe needs not be freed
> in kvm_unmap_gfn_range.
>