Received-SPF: pass (google.com: domain of linux-kernel+bounces-15917-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Wed, 3 Jan 2024 15:33:56 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: Ankit Agrawal <ankita@nvidia.com>, Yishai Hadas <yishaih@nvidia.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>,
	"kevin.tian@intel.com" <kevin.tian@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"brett.creeley@amd.com" <brett.creeley@amd.com>,
	"horms@kernel.org" <horms@kernel.org>,
	Aniket Agashe <aniketa@nvidia.com>, Neo Jia <cjia@nvidia.com>,
	Kirti Wankhede <kwankhede@nvidia.com>,
	"Tarun Gupta (SW-GPU)" <targupta@nvidia.com>,
	Vikram Sethi <vsethi@nvidia.com>, Andy Currid <acurrid@nvidia.com>,
	Alistair Popple <apopple@nvidia.com>,
	John Hubbard <jhubbard@nvidia.com>, Dan Williams <danw@nvidia.com>,
	"Anuj Aggarwal (SW-GPU)" <anuaggarwal@nvidia.com>,
	Matt Ochs <mochs@nvidia.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v15 1/1] vfio/nvgrace-gpu: Add vfio pci variant module
 for grace hopper
Message-ID: <20240103193356.GU50406@nvidia.com>
References: <20231217191031.19476-1-ankita@nvidia.com>
 <20231218151717.169f80aa.alex.williamson@redhat.com>
 <BY5PR12MB3763179CAC0406420AB0C9BAB095A@BY5PR12MB3763.namprd12.prod.outlook.com>
 <20240102091001.5fcc8736.alex.williamson@redhat.com>
 <20240103165727.GQ50406@nvidia.com>
 <20240103110016.5067b42e.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240103110016.5067b42e.alex.williamson@redhat.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?ZS5ynI+2pQTjA4Kk7JMP+Jfc8bNw+/H0sewcAMLOdQIZ0K9axNDnaQHfaOuB?=
 =?us-ascii?Q?XKWGLXSIXNvOag0RWn2kr/dp0gAtxlBcmAP8vQ4ckvfxd61lTEkQ4bm0Ssni?=
 =?us-ascii?Q?m4vVvMoos3D7xRDiBkcfFAb0e3++FfHcvumLa8K0Uelo4w8AoPx36UIPY14x?=
 =?us-ascii?Q?B62ixZQd1NeexSrAJAp0F6pGuGJ9cJ06qWStcKb4Qshw2INbAgWVsOIZ0jbe?=
 =?us-ascii?Q?oo2FP/tyTovZExWzgUtnabIQelubuyOZptk54piys+fGggNEmUbnT1pLKJWL?=
 =?us-ascii?Q?kJYEjmQYRhly/uPkH8QalrmOmVoOz1m/JG36KxcHUWV92429KrPyb/gobBSN?=
 =?us-ascii?Q?KKKHivpd82tp49McCWX28K1YqmkMsIGClojaxH3G6iu6CiYYuBDJ7g1KgFCU?=
 =?us-ascii?Q?k/jKabnzbkG6B8LN9/X0EjY+G20xARJvA/egztssFS/DFV57CiAMYs6vVswv?=
 =?us-ascii?Q?5r+OHz8qq23+qNghWvJ0sH4EV5/GHPtslGYRU7QAtS+wmklA+ZBPbO4nPC4K?=
 =?us-ascii?Q?lfuTzbRBh/ElfQzKTq38CYayEyDJexDWZA/huXBupJ89stnZZVPbUbnct+Ev?=
 =?us-ascii?Q?m1pKJNWGF7fuKLVNWPYg4hnuZoye4G/phf61PYWSJh8se/fI7oyxKlEDdFCQ?=
 =?us-ascii?Q?kbkOJYi3oZfqKfB7e9/+pzKRKO7czqxCzwJonf2JEDjnzunzsEHtpWvmgPgG?=
 =?us-ascii?Q?cvN5ecFhJ/rj3oVqWG4bbLl+k0R8E+7HlBoejfrVwYKM8K5Tue0llyitDcI7?=
 =?us-ascii?Q?vGeroHuWi4IaXEKA51HVaI7BD7uXy/cjYofsGV2YnpMHJdtDg0m2fF54VEUq?=
 =?us-ascii?Q?cIFTlMruxF6Tzkn39ucBl9x7q1YsPinEDce9JOs7ParFIlv45I3OXauMOkLk?=
 =?us-ascii?Q?qAM47RdvBiW0e0LzYKQy+Nuh+zqMZQEeNnfWE9HVi6IBg19c8MEfre7VZXKT?=
 =?us-ascii?Q?5ATFncnFGhAZ45Tj4YfMjI2eeFWh0qfcUsqSzAAsTE2DPWI62QKBB6/g2w9D?=
 =?us-ascii?Q?IF4GAmLbR+DNvqiPJGqjirTjo7loMe15BiGtpzxfCcl/J9rZVn4SngM2LAqu?=
 =?us-ascii?Q?yDNJhNsMc5p/QYyETPMeH3/id4UPkqq1P6cD1W+EBhf5Gry/0767sQ6BHPN6?=
 =?us-ascii?Q?jsi9wmcF4e3RhdctDAUwO+5TPlzyy9NDysiIMjV0R3KLyNYJpDKPtCunU+8n?=
 =?us-ascii?Q?G+u6Cmyh4BWDLdpYaibRoOwefq4ComA7ybu5MUPvD6HU4vClOAion0a0o2oZ?=
 =?us-ascii?Q?Sw834Kw9KDhZaplH0FHREywOwuST3ifnQ59Crp48VUlDWbdl9KcN3xMj3yYh?=
 =?us-ascii?Q?yUhRPK+4KpE0qTnFGkz0jKw1eLz3VPi6c+rETCDJHSm9cseg3D2GMHOkh+he?=
 =?us-ascii?Q?lTZPum1OLqVV+jnq57LwZQIRxP5QlQNtAUixdujNYlAGEVIVFnOSHdprbR/+?=
 =?us-ascii?Q?biWLHpIG0tiwy0NajuO1D/KbnJFCdQXbV3w7m6K/sLI+YGCoaCdr4NitCnsY?=
 =?us-ascii?Q?IUPNa2ZteZfjzrBY+fXQvxKFLSkFrmybS4yQYVZc4qWyr6rBfKW8b4Vj8avb?=
 =?us-ascii?Q?LbugdQOYNprzfw3MzZrOAnPoQ/voPbhAXN7JE8Mf?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 249f5ead-432e-497e-16b8-08dc0c92edb9
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2024 19:33:57.4887
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: CESyYUJ/P6pguHZhoC+MR61G/2rX6Zy39c3UYDWH8QV8APQjDEGGAwBwncp65Zua
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5925

On Wed, Jan 03, 2024 at 11:00:16AM -0700, Alex Williamson wrote:
> On Wed, 3 Jan 2024 12:57:27 -0400
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Tue, Jan 02, 2024 at 09:10:01AM -0700, Alex Williamson wrote:
> > 
> > > Yes, it's possible to add support that these ranges honor the memory
> > > enable bit, but it's not trivial and unfortunately even vfio-pci isn't
> > > a great example of this.  
> > 
> > We talked about this already, the HW architects here confirm there is
> > no issue with reset and memory enable. You will get all 1's on read
> > and NOP on write. It doesn't need to implement VMA zap.
> 
> We talked about reset, I don't recall that we discussed that coherent
> and uncached memory ranges masquerading as PCI BARs here honor the
> memory enable bit in the command register.

Why do it need to do anything special? If the VM read/writes from
memory that the master bit is disabled on it gets undefined
behavior. The system doesn't crash and it does something reasonable.

> > > around device reset or relative to the PCI command register.  The
> > > variant driver becomes a trivial implementation that masks BARs 2 & 4
> > > and exposes the ACPI range as a device specific region with only mmap
> > > support.  QEMU can then map the device specific region into VM memory
> > > and create an equivalent ACPI table for the guest.  
> > 
> > Well, no, probably not. There is an NVIDIA specification for how the
> > vPCI function should be setup within the VM and it uses the BAR
> > method, not the ACPI.
> 
> Is this specification available?  It's a shame we've gotten this far
> without a reference to it.

No, at this point it is internal short form only.

> > There are a lot of VMMs and OSs this needs to support so it must all
> > be consistent. For better or worse the decision was taken for the vPCI
> > spec to use BAR not ACPI, in part due to feedback from the broader VMM
> > ecosystem, and informed by future product plans.
> > 
> > So, if vfio does special regions then qemu and everyone else has to
> > fix it to meet the spec.
> 
> Great, this is the sort of justification and transparency that had not
> been previously provided.  It is curious that only within the past
> couple months the device ABI changed by adding the uncached BAR, so
> this hasn't felt like a firm design.

That is to work around some unfortunate HW defect, and is connected to
another complex discussion about how ARM memory types need to
work. Originally people hoped this could simply work transparently but
it eventually became clear it was not possible for the VM to degrade
from cachable without VMM help. Thus a mess was born..

> Also I believe it's been stated that the driver supports both the
> bare metal representation of the device and this model where the
> coherent memory is mapped as a BAR, so I'm not sure what obstacles
> remain or how we're positioned for future products if take the bare
> metal approach.

It could work, but it is not really the direction that was decided on
for the vPCI functions.

> > I thought all the meaningful differences are fixed now?
> > 
> > The main remaining issue seems to be around the config space
> > emulation?
> 
> In the development of the virtio-vfio-pci variant driver we noted that
> r/w access to the IO BAR didn't honor the IO bit in the command
> register, which was quickly remedied and now returns -EIO if accessed
> while disabled.  We were already adding r/w support to the coherent BAR
> at the time as vfio doesn't have a means to express a region as only
> having mmap support and precedent exists that BAR regions must support
> these accesses.  So it was suggested that r/w accesses should also
> honor the command register memory enable bit, but of course memory BARs
> also support mmap, which snowballs into a much more complicated problem
> than we have in the case of the virtio IO BAR.

I think that has just become too pedantic, accessing the regions with
the enable bits turned off is broadly undefined behavior. So long as
the platform doesn't crash, I think it is fine to behave in a simple
way.

There is no use case for providing stronger emulation of this.

> So where do we go?  Do we continue down the path of emulating full PCI
> semantics relative to these emulated BARs?  Does hardware take into
> account the memory enable bit of the command register?  Do we
> re-evaluate the BAR model for a device specific region?

It has to come out as a BAR in the VM side so all of this can't be
avoided. The simple answer is we don't need to care about enables
because there is no reason to care. VMs don't write to memory with the
enable turned off because on some platforms you will crash the system
if you do that.

> I'd suggest we take a look at whether we need to continue to pursue
> honoring the memory enable bit for these BARs and make a conscious and
> documented decision if we choose to ignore it.

Let's document it.

> Ideally we could also make this shared spec that we're implementing
> to available to the community to justify the design decisions here.
> In the case of GPUDirect Cliques we had permission to post the spec
> to the list so it could be archived to provide a stable link for
> future reference.  Thanks,

Ideally. I'll see that people consider it at least.

Jason