Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Date:   Thu, 17 Aug 2023 13:05:07 +0800
From:   Yan Zhao <yan.y.zhao@intel.com>
To:     John Hubbard <jhubbard@nvidia.com>
CC:     David Hildenbrand <david@redhat.com>, <linux-mm@kvack.org>,
        <linux-kernel@vger.kernel.org>, <kvm@vger.kernel.org>,
        <pbonzini@redhat.com>, <seanjc@google.com>,
        <mike.kravetz@oracle.com>, <apopple@nvidia.com>, <jgg@nvidia.com>,
        <rppt@kernel.org>, <akpm@linux-foundation.org>,
        <kevin.tian@intel.com>, Mel Gorman <mgorman@techsingularity.net>,
        <alex.williamson@redhat.com>
Subject: Re: [RFC PATCH v2 0/5] Reduce NUMA balance caused TLB-shootdowns in
 a VM
Message-ID: <ZN2qg4cPC2hEgtmY@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <6b48a161-257b-a02b-c483-87c04b655635@redhat.com>
 <1ad2c33d-95e1-49ec-acd2-ac02b506974e@nvidia.com>
 <846e9117-1f79-a5e0-1b14-3dba91ab8033@redhat.com>
 <d0ad2642-6d72-489e-91af-a7cb15e75a8a@nvidia.com>
 <ZNnvPuRUVsUl5umM@yzhao56-desk.sh.intel.com>
 <4271b91c-90b7-4b48-b761-b4535b2ae9b7@nvidia.com>
 <f523af84-59de-5b57-a3f3-f181107de197@redhat.com>
 <ZNyRnU+KynjCzwRm@yzhao56-desk.sh.intel.com>
 <ded3c4dc-2df9-2ef2-add0-c17f0cdfaf32@redhat.com>
 <37325c27-223d-400d-bd86-34bdbfb92a5f@nvidia.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <37325c27-223d-400d-bd86-34bdbfb92a5f@nvidia.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?lMa3TUzW4rv02GRL8FPdOGvFy/3F/XDWQMrApOsV6KvmInwBGodf256q/ssP?=
 =?us-ascii?Q?IqnX3NfRXWICqo9c/75cYYqMBpHSjCcDQ/R6i9Sbz5IohO/brCL3Hg/gcMV4?=
 =?us-ascii?Q?E8zgYmSvQ2cX1XhRrogacLrkC1C4MIpaftnQrM8vHhnY8AjoHiSMdu1k8YUK?=
 =?us-ascii?Q?20SWbezvEsmlCHwjsGplzVNizdrrucOunyknr0zqIpCUMhxm3aia4dvDPsSh?=
 =?us-ascii?Q?L+yRmWrNyzQCg1eFIQrP4ol8k1PnausWC/7em/L6h4UtiVoU91/ceyj8sPGo?=
 =?us-ascii?Q?GaFz3SlTJD3Cr/bZFEHVD9kBnxy2vxqmVN8MhicJNmS/vDuq211I/Hsr7VHI?=
 =?us-ascii?Q?gNsyAB9tKJuELVJPM+vL0tnHzVwO4XFZCrzL2TpccRL6GOwnE4ZO3qsGPHBN?=
 =?us-ascii?Q?nfwjw+jA4EfqIRlW8sd1vKx6JUUM+H3kJRl6xmYYt5NPGhLC1d/ImiF5TWq9?=
 =?us-ascii?Q?pc/r3CE83Pj9sLaM+BkHmwZgt0441ZWaimUlgfzo7UefAiCGtgFdOta6fQQd?=
 =?us-ascii?Q?gf/f5mh8u09CdOyfuPq7G77rmbR9SvzfYwvZvarXqRJJbZ0P3RTD0eP1DdZP?=
 =?us-ascii?Q?T4XQSnzqJQYbB7q+sOQVcKMQR8AZyoRDvG1W/VGMqig32z5EyLNdDzGSXJOw?=
 =?us-ascii?Q?H7+psr4n6UAufZMx/KlKDeqzVNIf1LS3rXPSbI/lkK3sYoA0tWUT3Fp1a9T1?=
 =?us-ascii?Q?XAIBpnt4yVrVlJWJS2ai2rt5uFmQeKe1lEdiDkCtE03YSZHQoUGKORpqXkBd?=
 =?us-ascii?Q?SJ1z9pNPTZ0iPGUIcKZxfp7haFwF4aj+SyOIoxVeVtkkez/NPLdtHgzlygrc?=
 =?us-ascii?Q?r2j04VcQ+4wF/Xwe7dtsrnN/G34f9D/qiEK0uv4YwKNabv2cyGYwT6ZNkecT?=
 =?us-ascii?Q?ZvQOPbLobjv7EfIMP3LLik50XjFHN9XvgLLwW7aa3KXCEImtqfPI8B+RrvS4?=
 =?us-ascii?Q?e+fYABYxS0pZm65hmQYQ0PUW1ib6CNTCN8uzGlK3S9VWYXaw0S/V2ZADs4Ie?=
 =?us-ascii?Q?iNs91zw2JBqcSCnHIBYPHhz8AHb3C+VmNgpzsTH2uvHTBFxu5DT6pGG0kWfe?=
 =?us-ascii?Q?mvgaaPAnDZXacZns5EtAkYsY4PI9Hcopj2qjJ/YKiBhOnQNOlbVTAxHGxTfy?=
 =?us-ascii?Q?UGig3L6UpgqIecPUyk+TFgaD2ar/uvQz0zYstvgKYKvzZCfR75Dzjg40HqAs?=
 =?us-ascii?Q?T0l6NKp7lxE+bpYZSJEXddERAXqGF0O103Qj5fOHEdqG9ZkWSake0Hkz5/TY?=
 =?us-ascii?Q?xwMESZhfU3SbM42QkCFBaWWu/F9Yr3I4FWBOzZ+bj2FsVK3h6eu9FMUL4TIY?=
 =?us-ascii?Q?SlSvKxT8Jm7YwLPX1NWqLDwxoZexegMuQlgPu3e/x4xPP9ntOVPkO1FhpK6H?=
 =?us-ascii?Q?5yHid66gW1GzR0xFvZ4+36RYLWD4J0BMUN5R5O+Ub43fKx0203QJk90fx9F7?=
 =?us-ascii?Q?SPLNeGKJTUS7N/pn5E+MgQSKewleCcxlJcMFSxIfDsKuI5lFmR0o3ECGmp9c?=
 =?us-ascii?Q?sM5kYTwxkPcOi63fUguAvu9XCCkNptjw5GrVh++puZ6DDsVRu+Jkvd8ze9c5?=
 =?us-ascii?Q?do9uql2GTKH+19xTJ1LliVKP/C8yQUePUxCgM3R5?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1b3b1896-4e6c-48ef-92fc-08db9ee34d45
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Aug 2023 05:32:09.6345
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ma5CCtaRDwuGUAOL0m+lTfXA48GYj93BRH9RgNUb6iRepuFOZlj2oj6WteM/3vwLMQ+y5rlw57G/M8CaoKOTOA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5181
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
        RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Aug 16, 2023 at 11:00:36AM -0700, John Hubbard wrote:
> On 8/16/23 02:49, David Hildenbrand wrote:
> > But do 32bit architectures even care about NUMA hinting? If not, just
> > ignore them ...
> 
> Probably not!
> 
> ...
> > > So, do you mean that let kernel provide a per-VMA allow/disallow
> > > mechanism, and
> > > it's up to the user space to choose between per-VMA and complex way or
> > > global and simpler way?
> > 
> > QEMU could do either way. The question would be if a per-vma settings
> > makes sense for NUMA hinting.
> 
> From our experience with compute on GPUs, a per-mm setting would suffice.
> No need to go all the way to VMA granularity.
> 
After an offline internal discussion, we think a per-mm setting is also
enough for device passthrough in VMs.

BTW, if we want a per-VMA flag, compared to VM_NO_NUMA_BALANCING, do you
think it's of any value to providing a flag like VM_MAYDMA?
Auto NUMA balancing or other components can decide how to use it by
themselves.