Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Date:   Wed, 25 Oct 2023 08:29:47 -0600
From:   Alex Williamson <alex.williamson@redhat.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     Ankit Agrawal <ankita@nvidia.com>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Yishai Hadas <yishaih@nvidia.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        Aniket Agashe <aniketa@nvidia.com>, Neo Jia <cjia@nvidia.com>,
        Kirti Wankhede <kwankhede@nvidia.com>,
        "Tarun Gupta (SW-GPU)" <targupta@nvidia.com>,
        Vikram Sethi <vsethi@nvidia.com>,
        "Currid, Andy" <acurrid@nvidia.com>,
        Alistair Popple <apopple@nvidia.com>,
        John Hubbard <jhubbard@nvidia.com>,
        Dan Williams <danw@nvidia.com>,
        "Anuj Aggarwal (SW-GPU)" <anuaggarwal@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for
 grace hopper
Message-ID: <20231025082947.6094361d.alex.williamson@redhat.com>
In-Reply-To: <BN9PR11MB5276A59033E514C051E9E4618CDEA@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20231015163047.20391-1-ankita@nvidia.com>
        <20231017165437.69a84f0c.alex.williamson@redhat.com>
        <BY5PR12MB3763356FC8CD2A7B307BD9AAB0D8A@BY5PR12MB3763.namprd12.prod.outlook.com>
        <20231023084312.15b8e37e.alex.williamson@redhat.com>
        <BY5PR12MB37636C06DED20856CF604A86B0DFA@BY5PR12MB3763.namprd12.prod.outlook.com>
        <20231024082854.0b767d74.alex.williamson@redhat.com>
        <BN9PR11MB5276A59033E514C051E9E4618CDEA@BN9PR11MB5276.namprd11.prod.outlook.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Wed, 25 Oct 2023 08:28:44 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:

> > From: Alex Williamson <alex.williamson@redhat.com>
> > Sent: Tuesday, October 24, 2023 10:29 PM
> >=20
> > On Tue, 24 Oct 2023 14:03:25 +0000
> > Ankit Agrawal <ankita@nvidia.com> wrote:
> >  =20
> > > >> > After looking at Yishai's virtio-vfio-pci driver where BAR0 is e=
mulated
> > > >> > as an IO Port BAR, it occurs to me that there's no config space
> > > >> > emulation of BAR2 (or BAR3) here.=C2=A0 Doesn't this mean that Q=
EMU =20
> > registers =20
> > > >> > the BAR as 32-bit, non-prefetchable?=C2=A0 ie. VFIOBAR.type & .m=
em64 are
> > > >> > wrong? =20
> > > >>
> > > >> Maybe I didn't understand the question, but the PCI config space =
=20
> > read/write =20
> > > >> would still be handled by vfio_pci_core_read/write() which returns=
 the
> > > >> appropriate flags. I have checked that the device BARs are 64b and
> > > >> prefetchable in the VM. =20
> > > >
> > > > vfio_pci_core_read/write() accesses the physical device, which does=
n't
> > > > implement BAR2.=C2=A0 Why would an unimplemented BAR2 on the physic=
al =20
> > device =20
> > > > report 64-bit, prefetchable?
> > > >
> > > > QEMU records VFIOBAR.type and .mem64 from reading the BAR register =
=20
> > in =20
> > > > vfio_bar_prepare() and passes this type to pci_register_bar() in
> > > > vfio_bar_register().=C2=A0 Without an implementation of a config sp=
ace read
> > > > op in the variant driver and with no physical implementation of BAR=
2 on
> > > > the device, I don't see how we get correct values in these fields. =
=20
> > >
> > > I think I see the cause of confusion. There are real PCIe compliant B=
ARs
> > > present on the device, just that it isn't being used once the C2C
> > > interconnect is active. The BARs are 64b prefetchable. Here it the ls=
pci
> > > snippet of the device on the host.
> > > # lspci -v -s 9:1:0.0
> > > 0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1)
> > >         Subsystem: NVIDIA Corporation Device 16eb
> > >         Physical Slot: 0-5
> > >         Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node=
 0, =20
> > IOMMU group 19 =20
> > >         Memory at 661002000000 (64-bit, prefetchable) [size=3D16M]
> > >         Memory at 662000000000 (64-bit, prefetchable) [size=3D128G]
> > >         Memory at 661000000000 (64-bit, prefetchable) [size=3D32M]
> > >
> > > I suppose this answers the BAR sizing question as well? =20
> >=20
> > Does this BAR2 size match the size we're reporting for the region?  Now
> > I'm confused why we need to intercept the BAR2 region info if there's
> > physically a real BAR behind it.  Thanks,
> >  =20
>=20
> same confusion.
>=20
> probably vfio-pci-core can include a helper for cfg space emulation
> on emulated BARs to be used by all variant drivers in that category?
>=20
> btw intel vgpu also includes an emulation of BAR sizing. same for
> future SIOV devices. so there sounds like a general requirement but
> of course sharing it between vfio-pci and mdev/siov would be more
> difficult.

Yes, I see a need for this in the virtio-vfio-pci driver as well.  It
would simplify config emulation a lot if the variant driver could
manipulate the perm_bits.virt and .write bit arrays and simply update
vconfig for things like device-id and revision.  Thanks,

Alex