Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 5 Oct 2021 20:28:34 -0300
From:   Jason Gunthorpe <jgg@ziepe.ca>
To:     Si-Wei Liu <siwliu.kernel@gmail.com>
Cc:     Haakon Bugge <haakon.bugge@oracle.com>,
        Leon Romanovsky <leon@kernel.org>,
        Doug Ledford <dledford@redhat.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        OFED mailing list <linux-rdma@vger.kernel.org>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: Enabling RO on a VF
Message-ID: <20211005232834.GB2688930@ziepe.ca>
References: <48FF6F8E-95E2-4A29-A059-12EF614B381C@oracle.com>
 <20211001115455.GJ3544071@ziepe.ca>
 <4EAE3BC9-26B6-41E3-B040-2ADAB77D96CE@oracle.com>
 <20211001120153.GL3544071@ziepe.ca>
 <CAPWQSg0wODmw7evfzdtP4gW-toVgoVfigP5t0CVosOAkarNTTg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAPWQSg0wODmw7evfzdtP4gW-toVgoVfigP5t0CVosOAkarNTTg@mail.gmail.com>
Precedence: bulk

On Tue, Oct 05, 2021 at 04:09:54PM -0700, Si-Wei Liu wrote:
> On Fri, Oct 1, 2021 at 6:02 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Fri, Oct 01, 2021 at 11:59:15AM +0000, Haakon Bugge wrote:
> > >
> > >
> > > > On 1 Oct 2021, at 13:54, Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > > >
> > > > On Fri, Oct 01, 2021 at 11:05:15AM +0000, Haakon Bugge wrote:
> > > >> Hey,
> > > >>
> > > >>
> > > >> Commit 1477d44ce47d ("RDMA/mlx5: Enable Relaxed Ordering by default
> > > >> for kernel ULPs") uses pcie_relaxed_ordering_enabled() to check if
> > > >> RO can be enabled. This function checks if the Enable Relaxed
> > > >> Ordering bit in the Device Control register is set. However, on a
> > > >> VF, this bit is RsvdP (Reserved for future RW
> > > >> implementations. Register bits are read-only and must return zero
> > > >> when read. Software must preserve the value read for writes to
> > > >> bits.).
> > > >>
> > > >> Hence, AFAICT, RO will not be enabled when using a VF.
> > > >>
> > > >> How can that be fixed?
> > > >
> > > > When qemu takes a VF and turns it into a PF in a VM it must emulate
> > > > the RO bit and return one
> > >
> > > I have a pass-through VF:
> > >
> > > # lspci -s ff:00.0 -vvv
> > > ff:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]
> > > []
> > >               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> > >                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
> >
> > Like I said, it is a problem in the qemu area..
> >
> > Jason
> Can you clarify why this is a problem in the QEMU area?
> 
> Even though Mellanox device might well support it (on VF), there's no
> way for QEMU to really know if an arbitrary passthrough device may
> support RO. 

That isn't what the cap bit means

The cap bit on the PF completely disables generation of RO at the
device at all.

If the PF's cap bit is disabled then no VF can generate RO, and qemu
should expose a wired to zero RO bit in the emulated PF.

If the cap bit is enabled then the VFs could generate RO, depending on
their drivers, and qemu should generate defaulted to 1 bit in the
emulated PF.

> PCIe device functions up to the root port throughout the PCIe fabric,
> or it may follow PF's enabling status if it is at all capable. I don't
> see what QEMU can do by just forcefully emulating the bit?

IMHO Kernel/BIOS should be responsible to clear the RO bit at the PF
if RO is not supportable in the environment. It is proper to prevent
the device from using RO completely if it is broken.

> Not to mention the current implementation only takes care of broken
> root port but not the intermediate switches.
> https://lore.kernel.org/linux-arm-kernel/MWHPR12MB1600255ACFCD3FB3C80EB8B6C88B0@MWHPR12MB1600.namprd12.prod.outlook.com/

Which is what this message suggests doing

Jason