Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
Date:   Tue, 24 Oct 2023 08:28:54 -0600
From:   Alex Williamson <alex.williamson@redhat.com>
To:     Ankit Agrawal <ankita@nvidia.com>
Cc:     Jason Gunthorpe <jgg@nvidia.com>,
        Yishai Hadas <yishaih@nvidia.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "kevin.tian@intel.com" <kevin.tian@intel.com>,
        Aniket Agashe <aniketa@nvidia.com>, Neo Jia <cjia@nvidia.com>,
        Kirti Wankhede <kwankhede@nvidia.com>,
        "Tarun Gupta (SW-GPU)" <targupta@nvidia.com>,
        Vikram Sethi <vsethi@nvidia.com>,
        Andy Currid <acurrid@nvidia.com>,
        Alistair Popple <apopple@nvidia.com>,
        John Hubbard <jhubbard@nvidia.com>,
        Dan Williams <danw@nvidia.com>,
        "Anuj Aggarwal (SW-GPU)" <anuaggarwal@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for
 grace hopper
Message-ID: <20231024082854.0b767d74.alex.williamson@redhat.com>
In-Reply-To: <BY5PR12MB37636C06DED20856CF604A86B0DFA@BY5PR12MB3763.namprd12.prod.outlook.com>
References: <20231015163047.20391-1-ankita@nvidia.com>
        <20231017165437.69a84f0c.alex.williamson@redhat.com>
        <BY5PR12MB3763356FC8CD2A7B307BD9AAB0D8A@BY5PR12MB3763.namprd12.prod.outlook.com>
        <20231023084312.15b8e37e.alex.williamson@redhat.com>
        <BY5PR12MB37636C06DED20856CF604A86B0DFA@BY5PR12MB3763.namprd12.prod.outlook.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, 24 Oct 2023 14:03:25 +0000
Ankit Agrawal <ankita@nvidia.com> wrote:

> >> > After looking at Yishai's virtio-vfio-pci driver where BAR0 is emula=
ted
> >> > as an IO Port BAR, it occurs to me that there's no config space
> >> > emulation of BAR2 (or BAR3) here.=C2=A0 Doesn't this mean that QEMU =
registers
> >> > the BAR as 32-bit, non-prefetchable?=C2=A0 ie. VFIOBAR.type & .mem64=
 are
> >> > wrong? =20
> >>
> >> Maybe I didn't understand the question, but the PCI config space read/=
write
> >> would still be handled by vfio_pci_core_read/write() which returns the
> >> appropriate flags. I have checked that the device BARs are 64b and
> >> prefetchable in the VM. =20
> >
> > vfio_pci_core_read/write() accesses the physical device, which doesn't
> > implement BAR2.=C2=A0 Why would an unimplemented BAR2 on the physical d=
evice
> > report 64-bit, prefetchable?
> >
> > QEMU records VFIOBAR.type and .mem64 from reading the BAR register in
> > vfio_bar_prepare() and passes this type to pci_register_bar() in
> > vfio_bar_register().=C2=A0 Without an implementation of a config space =
read
> > op in the variant driver and with no physical implementation of BAR2 on
> > the device, I don't see how we get correct values in these fields. =20
>=20
> I think I see the cause of confusion. There are real PCIe compliant BARs
> present on the device, just that it isn't being used once the C2C
> interconnect is active. The BARs are 64b prefetchable. Here it the lspci
> snippet of the device on the host.
> # lspci -v -s 9:1:0.0
> 0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1)
>         Subsystem: NVIDIA Corporation Device 16eb
>         Physical Slot: 0-5
>         Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node 0, =
IOMMU group 19
>         Memory at 661002000000 (64-bit, prefetchable) [size=3D16M]
>         Memory at 662000000000 (64-bit, prefetchable) [size=3D128G]
>         Memory at 661000000000 (64-bit, prefetchable) [size=3D32M]
>=20
> I suppose this answers the BAR sizing question as well?

Does this BAR2 size match the size we're reporting for the region?  Now
I'm confused why we need to intercept the BAR2 region info if there's
physically a real BAR behind it.  Thanks,

Alex