Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
Date:   Mon, 23 Oct 2023 08:43:12 -0600
From:   Alex Williamson <alex.williamson@redhat.com>
To:     Ankit Agrawal <ankita@nvidia.com>
Cc:     Jason Gunthorpe <jgg@nvidia.com>,
        Yishai Hadas <yishaih@nvidia.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "kevin.tian@intel.com" <kevin.tian@intel.com>,
        Aniket Agashe <aniketa@nvidia.com>, Neo Jia <cjia@nvidia.com>,
        Kirti Wankhede <kwankhede@nvidia.com>,
        "Tarun Gupta (SW-GPU)" <targupta@nvidia.com>,
        Vikram Sethi <vsethi@nvidia.com>,
        Andy Currid <acurrid@nvidia.com>,
        Alistair Popple <apopple@nvidia.com>,
        John Hubbard <jhubbard@nvidia.com>,
        Dan Williams <danw@nvidia.com>,
        "Anuj Aggarwal (SW-GPU)" <anuaggarwal@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for
 grace hopper
Message-ID: <20231023084312.15b8e37e.alex.williamson@redhat.com>
In-Reply-To: <BY5PR12MB3763356FC8CD2A7B307BD9AAB0D8A@BY5PR12MB3763.namprd12.prod.outlook.com>
References: <20231015163047.20391-1-ankita@nvidia.com>
        <20231017165437.69a84f0c.alex.williamson@redhat.com>
        <BY5PR12MB3763356FC8CD2A7B307BD9AAB0D8A@BY5PR12MB3763.namprd12.prod.outlook.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Mon, 23 Oct 2023 12:48:22 +0000
Ankit Agrawal <ankita@nvidia.com> wrote:

> > After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated
> > as an IO Port BAR, it occurs to me that there's no config space
> > emulation of BAR2 (or BAR3) here.=C2=A0 Doesn't this mean that QEMU reg=
isters
> > the BAR as 32-bit, non-prefetchable?=C2=A0 ie. VFIOBAR.type & .mem64 are
> > wrong? =20
>=20
> Maybe I didn't understand the question, but the PCI config space read/wri=
te
> would still be handled by vfio_pci_core_read/write() which returns the
> appropriate flags. I have checked that the device BARs are 64b and
> prefetchable in the VM.

vfio_pci_core_read/write() accesses the physical device, which doesn't
implement BAR2.  Why would an unimplemented BAR2 on the physical device
report 64-bit, prefetchable?

QEMU records VFIOBAR.type and .mem64 from reading the BAR register in
vfio_bar_prepare() and passes this type to pci_register_bar() in
vfio_bar_register().  Without an implementation of a config space read
op in the variant driver and with no physical implementation of BAR2 on
the device, I don't see how we get correct values in these fields.

> > We also need to decide how strictly variant drivers need to emulate
> > vfio_pci_config_rw with respect to BAR sizing, where the core code
> > provides emulation of sizing and Yishai's virtio driver only emulates
> > the IO port indicator bit. =20
>=20
> Sorry, it isn't clear to me how would resizable BAR is applicable in this
> variant driver as the BAR represents the device memory. Should we be
> exposing such feature as unsupported for this variant driver?

Bar SIZING, not resizing.  This is the standard in-band mechanism for
determining the BAR size as described in PCIe 6.0.1, 7.5.1.2.1.  QEMU
makes use of the region size but does rely on the BAR flags when
registering the BAR into QEMU as described above.  Additionally,
vfio-pci-core supports this in-band sizing mechanism for physical BARs.
A variant driver which does not implement config space BAR sizing for a
virtual BAR is arguably not presenting a PCI compatible config space
where a non-QEMU userspace may depend on standard PCI behavior here.
Thanks,

Alex