Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <48FF6F8E-95E2-4A29-A059-12EF614B381C@oracle.com>
 <20211001115455.GJ3544071@ziepe.ca> <4EAE3BC9-26B6-41E3-B040-2ADAB77D96CE@oracle.com>
 <20211001120153.GL3544071@ziepe.ca>
In-Reply-To: <20211001120153.GL3544071@ziepe.ca>
From:   Si-Wei Liu <siwliu.kernel@gmail.com>
Date:   Tue, 5 Oct 2021 16:09:54 -0700
Message-ID: <CAPWQSg0wODmw7evfzdtP4gW-toVgoVfigP5t0CVosOAkarNTTg@mail.gmail.com>
Subject: Re: Enabling RO on a VF
To:     Jason Gunthorpe <jgg@ziepe.ca>
Cc:     Haakon Bugge <haakon.bugge@oracle.com>,
        Leon Romanovsky <leon@kernel.org>,
        Doug Ledford <dledford@redhat.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        OFED mailing list <linux-rdma@vger.kernel.org>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Fri, Oct 1, 2021 at 6:02 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Fri, Oct 01, 2021 at 11:59:15AM +0000, Haakon Bugge wrote:
> >
> >
> > > On 1 Oct 2021, at 13:54, Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > >
> > > On Fri, Oct 01, 2021 at 11:05:15AM +0000, Haakon Bugge wrote:
> > >> Hey,
> > >>
> > >>
> > >> Commit 1477d44ce47d ("RDMA/mlx5: Enable Relaxed Ordering by default
> > >> for kernel ULPs") uses pcie_relaxed_ordering_enabled() to check if
> > >> RO can be enabled. This function checks if the Enable Relaxed
> > >> Ordering bit in the Device Control register is set. However, on a
> > >> VF, this bit is RsvdP (Reserved for future RW
> > >> implementations. Register bits are read-only and must return zero
> > >> when read. Software must preserve the value read for writes to
> > >> bits.).
> > >>
> > >> Hence, AFAICT, RO will not be enabled when using a VF.
> > >>
> > >> How can that be fixed?
> > >
> > > When qemu takes a VF and turns it into a PF in a VM it must emulate
> > > the RO bit and return one
> >
> > I have a pass-through VF:
> >
> > # lspci -s ff:00.0 -vvv
> > ff:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]
> > []
> >               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> >                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
>
> Like I said, it is a problem in the qemu area..
>
> Jason
Can you clarify why this is a problem in the QEMU area?

Even though Mellanox device might well support it (on VF), there's no
way for QEMU to really know if an arbitrary passthrough device may
support RO. It either has to resort to the host kernel to detect all
PCIe device functions up to the root port throughout the PCIe fabric,
or it may follow PF's enabling status if it is at all capable. I don't
see what QEMU can do by just forcefully emulating the bit?

Not to mention the current implementation only takes care of broken
root port but not the intermediate switches.
https://lore.kernel.org/linux-arm-kernel/MWHPR12MB1600255ACFCD3FB3C80EB8B6C88B0@MWHPR12MB1600.namprd12.prod.outlook.com/

-Siwei