Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     "ankita@nvidia.com" <ankita@nvidia.com>,
        "jgg@nvidia.com" <jgg@nvidia.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>
CC:     "aniketa@nvidia.com" <aniketa@nvidia.com>,
        "cjia@nvidia.com" <cjia@nvidia.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "targupta@nvidia.com" <targupta@nvidia.com>,
        "vsethi@nvidia.com" <vsethi@nvidia.com>,
        "Currid, Andy" <acurrid@nvidia.com>,
        "apopple@nvidia.com" <apopple@nvidia.com>,
        "jhubbard@nvidia.com" <jhubbard@nvidia.com>,
        "danw@nvidia.com" <danw@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v2 1/1] vfio/nvgpu: Add vfio pci variant module for grace
 hopper
Thread-Topic: [PATCH v2 1/1] vfio/nvgpu: Add vfio pci variant module for grace
 hopper
Thread-Index: AQHZgiwdnOavLCCjbEOsVWRum+0Dua9eG+Lg
Date:   Wed, 17 May 2023 07:29:09 +0000
Message-ID: <BN9PR11MB52763B1BC1C88EB8C24E2AD88C7E9@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20230509040734.24392-1-ankita@nvidia.com>
In-Reply-To: <20230509040734.24392-1-ankita@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?cH7b9tYo1Bkf7RbQs8gWx9RIuyqipbAhmGqGnoSajwPW52I2tTm9FSakFyK+?=
 =?us-ascii?Q?cQP/HJsKY29JvCsWQ9lWUYHqJpZCAV5Sa5f1Ruq3cmU3nBv0PXx7nFONiO5o?=
 =?us-ascii?Q?nHFrLQSBRuwnf5Mk7Ja9aL0m3IriOF5E4YSWAWIIeo5wbgzJyrM7j9Ts2IJG?=
 =?us-ascii?Q?ipX/qfCtJNU8r+XSTC83M3snb2oS6027D0IKyL2X/3kVKQtTnJ9+9pUNDzXC?=
 =?us-ascii?Q?hWVMRqic/wg1BaTjyzftuL78B40Ra1ntnuEURidl9zMILzdqgSI3214R+E0v?=
 =?us-ascii?Q?d+aLL9S0RDRqQQfWrwpXWhuAkU2IcxTljv3z+4tacTmLtR2lFy8iiPaOHiDJ?=
 =?us-ascii?Q?e2z3bZbxRqlc8eAMfI4S8IqB17bm2X9duaHcn1AfJprlElBuC/xl6RGFWssB?=
 =?us-ascii?Q?wB0+PbZF5MKYOW+8zLt8oV6xFsFub5eSgL5CRXQ2/MHasEQ8tBZ3Z4VOpqxi?=
 =?us-ascii?Q?ZG4vJnikFWc37rtsEru+r3jdwxDDg1Zc69fiHhADTOu+sXU8jtb6di1uGumJ?=
 =?us-ascii?Q?wWwuFBIa2O9dGO0RIt/dlkhqlGiw2s2o+NpULs504w8YSM5AwSWgBfMIOMXg?=
 =?us-ascii?Q?EDsowFcBQEg7ME8++PEdDFQGAG1N/FYzLPEOgryQwLGkleWch6wR+UrsBMrU?=
 =?us-ascii?Q?Lxftf4mL30kbJDjzQqB28xtFuedCEhiBwvAooT/EgYfKTAyorj9v4ywBUz/J?=
 =?us-ascii?Q?ul1NDHCXZno9HN4Zi9rJCl4Q5Q2O4Wq8fNHdxcRS74nmIxPfExrGG6H/Ms0N?=
 =?us-ascii?Q?ja9UhKasdFvrE3RZkQkZ2WJVCFE1UyE7Y+wQBT0UscmwM95N/MsELXuwUxgh?=
 =?us-ascii?Q?gJnrIjwf7pgYxOWY+Zp3sYqsemhDgrcNxjTUqt7x8kHMSAP36Vf8CGbeyGmo?=
 =?us-ascii?Q?BRb8lmoOL8Fd4zCX17GmhKMEd+zYpd5YlZu8rFAdo9BvdR/9Ps1qa67KsK+z?=
 =?us-ascii?Q?94OJqYsA33AqqeBZpgevsiV801/m+NM5XC+jIM/0ptBv4xrWT06XF/lrprNS?=
 =?us-ascii?Q?fVOFzBdfcHxp/sakjF4C+5DXjP/i2dhHQoqjAnhxUNcCvWTqtrsRUoBYL/2U?=
 =?us-ascii?Q?zjy9nCI6cVEbDnGHkMQDfKWrr9wJ8+TjHwIucXhoo78CulzURA8sCOpiGqsP?=
 =?us-ascii?Q?GKjXWHUTP8nrtMfOpmWBCgw+b/Iie+pHg/GB71fDIFIFFucCs10qfchNTxuL?=
 =?us-ascii?Q?+bYl8hi02zSaE1ELSWeyzmpM4oHE3BzbTTswf5GCnEeSO81G2ZHqkhMMS171?=
 =?us-ascii?Q?CO2W/AScvDzyUbh7k0r67BlIj3PF/LGlijUAlststXNdv95fc76g4TIiyfwH?=
 =?us-ascii?Q?NmutlmmIel4X/armwh4j94U/7PyEKux0t2Vwcf84Nk+l7bwRDj7iNZCa7jYw?=
 =?us-ascii?Q?Kd+kJPZrlCcSU4Mv6DOf9IgaL/h1M/Vik4vjxKlOSRJ92ozzLM9RZ4QjJ1CF?=
 =?us-ascii?Q?mQAC2992dWq9xW1Ss8swSk0ByxrvRHI/nA+Rd5byQUqI/MHZqOdjK3xqGzxH?=
 =?us-ascii?Q?sFpAiHy77UOMPuaXoTov3S4ZCAIsDnSu9yWuF6OvvVgToGDktHgNLPdDb/n9?=
 =?us-ascii?Q?/vSQ15VI/n2OCR5YNmKAI8Pvy73kckHH8kg9j4lI?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5a3b99e1-891d-49fc-3833-08db56a867cd
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 May 2023 07:29:09.8995
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: fuunWkcjPkWoxBjBIMZBK8XxWwlaTPGPSlgImgEHI/KESxdkfBUuDZWcXWRFvT9CbTWjN11N0hDSYxPm+4Zdbg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5095
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: ankita@nvidia.com <ankita@nvidia.com>
> Sent: Tuesday, May 9, 2023 12:08 PM
>=20
> From: Ankit Agrawal <ankita@nvidia.com>
>=20
> NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device
> for the on-chip GPU that is the logical OS representation of the
> internal propritary cache coherent interconnect.
>=20
> This representation has a number of limitations compared to a real PCI
> device, in particular, it does not model the coherent GPU memory
> aperture as a PCI config space BAR, and PCI doesn't know anything
> about cacheable memory types.
>=20
> Provide a VFIO PCI variant driver that adapts the unique PCI
> representation into a more standard PCI representation facing
> userspace. The GPU memory aperture is obtained from ACPI using
> device_property_read_u64(), according to the FW specification,
> and exported to userspace as the VFIO_REGION that covers the first
> PCI BAR. qemu will naturally generate a PCI device in the VM where the
> cacheable aperture is reported in BAR1.

BAR2.

and it's more informative by describing how many BARs this device
already implements then BAR2 is selected because it's free.

> +
> +static int nvgpu_vfio_pci_open_device(struct vfio_device *core_vdev)
> +{
> +	struct vfio_pci_core_device *vdev =3D
> +		container_of(core_vdev, struct vfio_pci_core_device, vdev);
> +	int ret;
> +
> +	ret =3D vfio_pci_core_enable(vdev);
> +	if (ret)
> +		return ret;
> +
> +	vfio_pci_core_finish_enable(vdev);
> +
> +	return ret;
> +}

NIT. "return 0" as other variant drivers do.


> +
> +MODULE_LICENSE("GPL v2");
> +MODULE_AUTHOR("Ankit Agrawal <ankita@nvidia.com>");
> +MODULE_AUTHOR("Aniket Agashe <aniketa@nvidia.com>");
> +MODULE_DESCRIPTION(
> +	"VFIO NVGPU PF - User Level driver for NVIDIA devices with CPU
> coherently accessible device memory");

what does 'PF' mean? Physical function?

Probably needs a more specific name for the coherent part... nvgpu-vfio-pci
sounds covering all NV GPUs.