Received-SPF: pass (google.com: domain of linux-kernel+bounces-70128-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: "ankita@nvidia.com" <ankita@nvidia.com>, "jgg@nvidia.com"
	<jgg@nvidia.com>, "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"yishaih@nvidia.com" <yishaih@nvidia.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "mst@redhat.com" <mst@redhat.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "jgg@ziepe.ca"
	<jgg@ziepe.ca>, "oleksandr@natalenko.name" <oleksandr@natalenko.name>,
	"clg@redhat.com" <clg@redhat.com>, "K V P, Satyanarayana"
	<satyanarayana.k.v.p@intel.com>, "brett.creeley@amd.com"
	<brett.creeley@amd.com>, "horms@kernel.org" <horms@kernel.org>,
	"shannon.nelson@amd.com" <shannon.nelson@amd.com>
CC: "aniketa@nvidia.com" <aniketa@nvidia.com>, "cjia@nvidia.com"
	<cjia@nvidia.com>, "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
	"targupta@nvidia.com" <targupta@nvidia.com>, "vsethi@nvidia.com"
	<vsethi@nvidia.com>, "Currid, Andy" <acurrid@nvidia.com>,
	"apopple@nvidia.com" <apopple@nvidia.com>, "jhubbard@nvidia.com"
	<jhubbard@nvidia.com>, "danw@nvidia.com" <danw@nvidia.com>,
	"anuaggarwal@nvidia.com" <anuaggarwal@nvidia.com>, "mochs@nvidia.com"
	<mochs@nvidia.com>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"virtualization@lists.linux-foundation.org"
	<virtualization@lists.linux-foundation.org>
Subject: RE: [PATCH v18 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for
 grace hopper
Thread-Topic: [PATCH v18 3/3] vfio/nvgrace-gpu: Add vfio pci variant module
 for grace hopper
Thread-Index: AQHaYISYdCMnkFQa2kyZb3lTo7TENbEPSnAQ
Date: Sun, 18 Feb 2024 01:00:23 +0000
Message-ID: <BN9PR11MB5276CF18ADF0B687429118918C522@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20240216030128.29154-1-ankita@nvidia.com>
 <20240216030128.29154-4-ankita@nvidia.com>
In-Reply-To: <20240216030128.29154-4-ankita@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?37YT8FVPjEHewXhJ8fnlfenLvCVQHtyZgFodFeKoBjrxdHNvjZWpwGL9eDp/?=
 =?us-ascii?Q?x0DvBE7iztzMMo5Ilr6bG2oAQatyalJ/gr+9ZkU/WRDGsFUOQuBnnbXynGUh?=
 =?us-ascii?Q?BbgJYEvd5T4PWZ6ASYm4Hc2liDCaTfxOn1We7ZKOpQuHV064r3Cxomaxv5dF?=
 =?us-ascii?Q?Dt1xCs86PQ4X2Xzkv6I09KxUMJKYEa84kdr8xq/de5Csd52ezIEJXJAnhG/c?=
 =?us-ascii?Q?Q8MuESHRpt6+gBxItVsRPFY3NYP/VbqGRGZ5zo+dGMCaYyHvBlEscvsPQkdv?=
 =?us-ascii?Q?aVOX4FUuocguXhYacetHjk+sFdSG9z0QTdCAJe+70fmQhZfYigCZzpWov5tf?=
 =?us-ascii?Q?ZoYvC0Sqta/dhURRwCtLSgY/PAxclv/7257+4/bV/CKQmQbCxB4UuYJh8mcx?=
 =?us-ascii?Q?H3jcuGyXbpe/v6mwt0lCLE454sidmmtPyPg3wpDECrfaBR6UO/jxwyCUwf/X?=
 =?us-ascii?Q?23ar/2Clqq2BfFbE07yvfGn7RZWz9GfWPeP/4iWSWVI7shh0l43MNNUDx843?=
 =?us-ascii?Q?Ejb3C3Yv7aCu2FRm5oGDh2fI2da/RB6ItlzfZl4WISQ9llxwmmlqR4uyM3B1?=
 =?us-ascii?Q?1/kihEdo/6VvrfqzDLStgl6TehwtOS8KVR1VlyE2QFxnkNltobwHKHoVYdnB?=
 =?us-ascii?Q?hDhVocZ4hKf/+b2Nwp/hGOgS6VIEV20fNFVzVHZnxtgEyRAaIxvcS9brkw10?=
 =?us-ascii?Q?J9hwejb0OJ1uKzyfXs75CtyJSshWmzCCV0AxYNmedzVwXVty1iArWJxHDkpl?=
 =?us-ascii?Q?mnikh/lgEH9FaW1VgmYO8GUOnteYbZdhX2m9s9t1+lVku8rIk4vDIYlpuiV5?=
 =?us-ascii?Q?yDGvQOmwV7UYjo4Rq36O2YiQJBS8qJOvRdIeF20KosVH33Wkr33d549yuC/o?=
 =?us-ascii?Q?yRKSixTWmah6g418NaSs2F+EjGmVXmMbF9FVlAcvGKhmm1JIawFe97NkNjLl?=
 =?us-ascii?Q?hLvth50cI6lWKvGKoX4E4v76FDS6itH1IPPgxkHHMlMuE9513IvYwPKtkO6W?=
 =?us-ascii?Q?lAwuLcQD2TEYQi8RF8D2vE0aB3k/1gjWOKnvf+J4AAyAibmrgHBTbN/9hVUp?=
 =?us-ascii?Q?L2k/jfcnBJclhkymithtBHeouXre1AQRp4NojYi6Dn6tEHYD0pWCNaqWDQ6Z?=
 =?us-ascii?Q?u/iIqIOSI9VevMlXwYS/qbvoXImWVkgbm51TUZrP4O/LN5Pn5rEfepwiAtI2?=
 =?us-ascii?Q?NYRUItNXpCCZhFiYi7Dm24xIoeUJuFyvf44cid+Dz3NiATbApjakbZFNBrlZ?=
 =?us-ascii?Q?Gk0fXn2ICn7VQi61nYTdUkj/wFULNhYxHxoiPPsBlsZLFEanKXv1QPFQbZY2?=
 =?us-ascii?Q?LNOtzXHxzyUJ199WDI23HZd/rdWpNujw35bFPSv4hE8viJJHS8r6aeH2DkmJ?=
 =?us-ascii?Q?OK6A7iXaM0yXgON7nkhOq+LC5jH7X1RT4wzOwzuSWQj7qR+kJdtJPMIzVf1c?=
 =?us-ascii?Q?ODjQFrpRNQk3II3uTt3kfhXaUM53jPpHbsxOABFgjZagIZoTjoIklzqz+rAd?=
 =?us-ascii?Q?skE1rBTNIWFC8hqqkl6r/TYqRT81ZZGkMRdh0/LTs7oZjVwrCflarpsaN+/V?=
 =?us-ascii?Q?4bRa6FEXuQAh9u+WJ/+AJ03pDlwYpsC/OzXZDnn7?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: af3834b1-590c-4bf5-0f58-08dc301cfca8
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Feb 2024 01:00:23.5742
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: IoCUCYzttpyPFYgFK8drNC7il30D/H8NebV40uH7UZXgK4Vs2k5J0CPDqC1mSRqDkYMOHF8C5EKjN2WD63kLAA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB7375
X-OriginatorOrg: intel.com

> From: ankita@nvidia.com <ankita@nvidia.com>
> Sent: Friday, February 16, 2024 11:01 AM
>=20
> From: Ankit Agrawal <ankita@nvidia.com>
>=20
> NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device
> for the on-chip GPU that is the logical OS representation of the
> internal proprietary chip-to-chip cache coherent interconnect.
>=20
> The device is peculiar compared to a real PCI device in that whilst
> there is a real 64b PCI BAR1 (comprising region 2 & region 3) on the
> device, it is not used to access device memory once the faster
> chip-to-chip interconnect is initialized (occurs at the time of host
> system boot). The device memory is accessed instead using the chip-to-chi=
p
> interconnect that is exposed as a contiguous physically addressable
> region on the host. This device memory aperture can be obtained from host
> ACPI table using device_property_read_u64(), according to the FW
> specification. Since the device memory is cache coherent with the CPU,
> it can be mmap into the user VMA with a cacheable mapping using
> remap_pfn_range() and used like a regular RAM. The device memory
> is not added to the host kernel, but mapped directly as this reduces
> memory wastage due to struct pages.
>=20
> There is also a requirement of a minimum reserved 1G uncached region
> (termed as resmem) to support the Multi-Instance GPU (MIG) feature [1].
> This is to work around a HW defect. Based on [2], the requisite propertie=
s
> (uncached, unaligned access) can be achieved through a VM mapping (S1)
> of NORMAL_NC and host (S2) mapping with MemAttr[2:0]=3D0b101. To provide
> a different non-cached property to the reserved 1G region, it needs to
> be carved out from the device memory and mapped as a separate region
> in Qemu VMA with pgprot_writecombine(). pgprot_writecombine() sets the
> Qemu VMA page properties (pgprot) as NORMAL_NC.
>=20
> Provide a VFIO PCI variant driver that adapts the unique device memory
> representation into a more standard PCI representation facing userspace.
>=20
> The variant driver exposes these two regions - the non-cached reserved
> (resmem) and the cached rest of the device memory (termed as usemem) as
> separate VFIO 64b BAR regions. This is divergent from the baremetal
> approach, where the device memory is exposed as a device memory region.
> The decision for a different approach was taken in view of the fact that
> it would necessiate additional code in Qemu to discover and insert those
> regions in the VM IPA, along with the additional VM ACPI DSDT changes to
> communicate the device memory region IPA to the VM workloads. Moreover,
> this behavior would have to be added to a variety of emulators (beyond
> top of tree Qemu) out there desiring grace hopper support.
>=20
> Since the device implements 64-bit BAR0, the VFIO PCI variant driver
> maps the uncached carved out region to the next available PCI BAR (i.e.
> comprising of region 2 and 3). The cached device memory aperture is
> assigned BAR region 4 and 5. Qemu will then naturally generate a PCI
> device in the VM with the uncached aperture reported as BAR2 region,
> the cacheable as BAR4. The variant driver provides emulation for these
> fake BARs' PCI config space offset registers.
>=20
> The hardware ensures that the system does not crash when the memory
> is accessed with the memory enable turned off. It synthesis ~0 reads
> and dropped writes on such access. So there is no need to support the
> disablement/enablement of BAR through PCI_COMMAND config space
> register.
>=20
> The memory layout on the host looks like the following:
>                devmem (memlength)
> |--------------------------------------------------|
> |-------------cached------------------------|--NC--|
> |                                           |
> usemem.memphys                              resmem.memphys
>=20
> PCI BARs need to be aligned to the power-of-2, but the actual memory on t=
he
> device may not. A read or write access to the physical address from the
> last device PFN up to the next power-of-2 aligned physical address
> results in reading ~0 and dropped writes. Note that the GPU device
> driver [6] is capable of knowing the exact device memory size through
> separate means. The device memory size is primarily kept in the system
> ACPI tables for use by the VFIO PCI variant module.
>=20
> Note that the usemem memory is added by the VM Nvidia device driver [5]
> to the VM kernel as memblocks. Hence make the usable memory size
> memblock
> (MEMBLK_SIZE) aligned. This is a hardwired ABI value between the GPU FW
> and
> VFIO driver. The VM device driver make use of the same value for its
> calculation to determine USEMEM size.
>=20
> Currently there is no provision in KVM for a S2 mapping with
> MemAttr[2:0]=3D0b101, but there is an ongoing effort to provide the same =
[3].
> As previously mentioned, resmem is mapped pgprot_writecombine(), that
> sets the Qemu VMA page properties (pgprot) as NORMAL_NC. Using the
> proposed changes in [3] and [4], KVM marks the region with
> MemAttr[2:0]=3D0b101 in S2.
>=20
> If the device memory properties are not present, the driver registers the
> vfio-pci-core function pointers. Since there are no ACPI memory propertie=
s
> generated for the VM, the variant driver inside the VM will only use
> the vfio-pci-core ops and hence try to map the BARs as non cached. This
> is not a problem as the CPUs have FWB enabled which blocks the VM
> mapping's ability to override the cacheability set by the host mapping.
>=20
> This goes along with a qemu series [6] to provides the necessary
> implementation of the Grace Hopper Superchip firmware specification so
> that the guest operating system can see the correct ACPI modeling for
> the coherent GPU device. Verified with the CUDA workload in the VM.
>=20
> [1] https://www.nvidia.com/en-in/technologies/multi-instance-gpu/
> [2] section D8.5.5 of
> https://developer.arm.com/documentation/ddi0487/latest/
> [3] https://lore.kernel.org/all/20240211174705.31992-1-ankita@nvidia.com/
> [4] https://lore.kernel.org/all/20230907181459.18145-2-ankita@nvidia.com/
> [5] https://github.com/NVIDIA/open-gpu-kernel-modules
> [6] https://lore.kernel.org/all/20231203060245.31593-1-ankita@nvidia.com/
>=20
> Signed-off-by: Aniket Agashe <aniketa@nvidia.com>
> Signed-off-by: Ankit Agrawal <ankita@nvidia.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>