Received-SPF: pass (google.com: domain of linux-kernel+bounces-59717-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Fri, 9 Feb 2024 13:19:03 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: Ankit Agrawal <ankita@nvidia.com>, "Tian, Kevin" <kevin.tian@intel.com>,
	Yishai Hadas <yishaih@nvidia.com>,
	"mst@redhat.com" <mst@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>,
	"clg@redhat.com" <clg@redhat.com>,
	"oleksandr@natalenko.name" <oleksandr@natalenko.name>,
	"K V P, Satyanarayana" <satyanarayana.k.v.p@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"brett.creeley@amd.com" <brett.creeley@amd.com>,
	"horms@kernel.org" <horms@kernel.org>,
	Rahul Rameshbabu <rrameshbabu@nvidia.com>,
	Aniket Agashe <aniketa@nvidia.com>, Neo Jia <cjia@nvidia.com>,
	Kirti Wankhede <kwankhede@nvidia.com>,
	"Tarun Gupta (SW-GPU)" <targupta@nvidia.com>,
	Vikram Sethi <vsethi@nvidia.com>, Andy Currid <acurrid@nvidia.com>,
	Alistair Popple <apopple@nvidia.com>,
	John Hubbard <jhubbard@nvidia.com>, Dan Williams <danw@nvidia.com>,
	"Anuj Aggarwal (SW-GPU)" <anuaggarwal@nvidia.com>,
	Matt Ochs <mochs@nvidia.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"virtualization@lists.linux-foundation.org" <virtualization@lists.linux-foundation.org>
Subject: Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module
 for grace hopper
Message-ID: <20240209171903.GQ10476@nvidia.com>
References: <20240205230123.18981-1-ankita@nvidia.com>
 <20240205230123.18981-4-ankita@nvidia.com>
 <BN9PR11MB527666B48A975B7F4304837C8C442@BN9PR11MB5276.namprd11.prod.outlook.com>
 <SA1PR12MB71996EBCA4142458E8BEE367B04B2@SA1PR12MB7199.namprd12.prod.outlook.com>
 <20240209085531.73f25a98.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240209085531.73f25a98.alex.williamson@redhat.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?nzzoFfUJf5OuSkYef1L/foYzZ8QCRRzDn53mDnhrOg9AFg+3mZvTZCYE1W6O?=
 =?us-ascii?Q?s4Zt38IYILoSlG0vOy/HRxjbmOParNS6rUnz/V4D1vRB5OVhM+jsNKmfWZ2p?=
 =?us-ascii?Q?Kt2Z0LOH72cf7Wo4puinUzLIdx0OrxYoNCWagCVqT5PXhTlYEBvx2w4hQPH3?=
 =?us-ascii?Q?sO9CicWASSDRDWjZwolGyBKgfaLkbGPFhj/2q4+lWVhCjApaxYJMmh12e5e/?=
 =?us-ascii?Q?Iyi4f0mZA9cv5BBaib3yY8TGbLjKWETZURgBHn1dVOTcvoQ7YX1dXD+FF8QL?=
 =?us-ascii?Q?jOkpyLdMoeZ6dfsWzHX0hrso+fW+JitBWpVuxPyUIKibikMaO0vMGyXfY7Rg?=
 =?us-ascii?Q?5nebI1v3MXZBtOGWBH58+4fbHpxdFwtNgry4YGaNkSTtMd85wZCsOYoTqkDa?=
 =?us-ascii?Q?vzfzwIxYlhXQ4zQOOYrnA4yoUiOPJWkgN46AuxKx1lSz4kboU0Xl4rI7QZKU?=
 =?us-ascii?Q?bKBi9SUWJnIxEp1nI7HU3BcKaUXzKiHmmDRLXWX7HJJgrGI3xPf7Pfdtw+tV?=
 =?us-ascii?Q?tC5dHTT+ozjj/HJS+u75lpv364mopLNWdoZnLkZxHJ328JGz2if2bClOEjff?=
 =?us-ascii?Q?z25Q5IOPEb28ge6n+TN3kGqzMpfqS4ha1/6EalY5qlkP1KruoRmo3a8A8DyR?=
 =?us-ascii?Q?al4kWFxZeLyrsrr7P3VdgHcKBOf07nu4KmfJOq+suLwiju1qlMCYc4V93bQP?=
 =?us-ascii?Q?O64RUaru86yDmrUKdLUlOGyxqp4Zpl4JAZXAkh38/wOzAdSh/wZGmz3pR2xj?=
 =?us-ascii?Q?oTITeCIn9lv+/TbTO+wCN498rG8Badu2B9AIhnvly9CeAR9G/cuSQSE/lvSf?=
 =?us-ascii?Q?MujNPxpmEX6wBUSguWY6cEBPYRYgFugZ6aBl0xXNWNCI4oFAyavqVGYYqg/n?=
 =?us-ascii?Q?v7tTAeC8yAyPU2bSNAIFlLlXEupX5QGRe2OzCVWj1IZwBH0o4QcgaRgnZMxS?=
 =?us-ascii?Q?SQigSE0pMyR6FwUNJUrWdOg57QZSo19MnhZihZAiPLBVl0WNOcyUvJPEzrou?=
 =?us-ascii?Q?ut3acVPTA8AalXdKlQWkfuxIflnriPEkF28rwRkxtyTPTEbkpkr97GxX0Cbk?=
 =?us-ascii?Q?LBg9t6MJ9pSGIjXN8X4Ln6xRjgnJlS08f4QNWrWLS7zZak4bcNoQB91e3Qvw?=
 =?us-ascii?Q?S4EYqdktecEBv6RuzZGDl74Y566i8qViQY5UURwjMpTXj8feGq0qwEpMqGmJ?=
 =?us-ascii?Q?MHfDdn4C4mATkevFnzyiQZxPZ80tj0vLldKsr3ZdkGMbkYVZ6aHKFRpvXncX?=
 =?us-ascii?Q?P2JXDm8/wU8jeGKhKc7tlShwyNEEL0M4UG06W/+XBjBnve1pRIxkaXJRRYlm?=
 =?us-ascii?Q?f1jHyJTcXydc6+NxKwAcWrh49/+4Qrhd0ojTvHx5AeSA4eVn4WnEAfCEPQ8K?=
 =?us-ascii?Q?wZxLblhntC7m+7WyMjii1NStSZcYt7U46vYmROgQaHGl3d39xFKuFK+fYAqS?=
 =?us-ascii?Q?EKOjhx4ymRYO9n5x3aXOgF7t/ALs7kZjAV8+gXFftMmZ32rcl/Zl9vHZc+KM?=
 =?us-ascii?Q?0wTjgXrlkusKKVCH6zgeJYiGwS2vbUcMAtJ8GZsIXeDrEwrnGnTHe11z63fz?=
 =?us-ascii?Q?pZ3K7WN6GY57+itFzv92FHR/If1F7H6G7EBUaPak?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 07436d66-d670-4953-d83f-08dc299337a5
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2024 17:19:05.1551
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 0jklR9EDdA6rQoZvlvwEiWZ67QTeUEXzN3V1NVLNXx1sa2b0xAWeiheyT+3Z/oHD
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5331

On Fri, Feb 09, 2024 at 08:55:31AM -0700, Alex Williamson wrote:
> I think Kevin's point is also relative to this latter scenario, in the
> L1 instance of the nvgrace-gpu driver the mmap of the usemem BAR is
> cachable, but in the L2 instance of the driver where we only use the
> vfio-pci-core ops nothing maintains that cachable mapping.  Is that a
> problem?  An uncached mapping on top of a cachable mapping is often
> prone to problems.  

On these CPUs the ARM architecture won't permit it, the L0 level
blocks uncachable using FWB and page table attributes. The VM, no
matter what it does, cannot make the cachable memory uncachable.

Jason