Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 3 May 2021 08:44:32 -0600
From:   Alex Williamson <alex.williamson@redhat.com>
To:     Vikram Sethi <vsethi@nvidia.com>
Cc:     Mark Kettenis <mark.kettenis@xs4all.nl>,
        Marc Zyngier <maz@kernel.org>,
        Shanker Donthineni <sdonthineni@nvidia.com>,
        "will@kernel.org" <will@kernel.org>,
        "catalin.marinas@arm.com" <catalin.marinas@arm.com>,
        "christoffer.dall@arm.com" <christoffer.dall@arm.com>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "kvmarm@lists.cs.columbia.edu" <kvmarm@lists.cs.columbia.edu>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        Jason Sequeira <jsequeira@nvidia.com>
Subject: Re: [RFC 1/2] vfio/pci: keep the prefetchable attribute of a BAR
 region in VMA
Message-ID: <20210503084432.75e0126d@x1.home.shazbot.org>
In-Reply-To: <BL0PR12MB253296086906C4A850EC68E6BD5B9@BL0PR12MB2532.namprd12.prod.outlook.com>
References: <20210429162906.32742-1-sdonthineni@nvidia.com>
        <20210429162906.32742-2-sdonthineni@nvidia.com>
        <20210429122840.4f98f78e@redhat.com>
        <470360a7-0242-9ae5-816f-13608f957bf6@nvidia.com>
        <20210429134659.321a5c3c@redhat.com>
        <e3d7fda8-5263-211c-3686-f699765ab715@nvidia.com>
        <87czucngdc.wl-maz@kernel.org>
        <1edb2c4e-23f0-5730-245b-fc6d289951e1@nvidia.com>
        <878s4zokll.wl-maz@kernel.org>
        <BL0PR12MB2532CC436EBF626966B15994BD5E9@BL0PR12MB2532.namprd12.prod.outlook.com>
        <87eeeqvm1d.wl-maz@kernel.org>
        <BL0PR12MB25329EF5DFA7BBAA732064A7BD5C9@BL0PR12MB2532.namprd12.prod.outlook.com>
        <87bl9sunnw.wl-maz@kernel.org>
        <c1bd514a531988c9@bloch.sibelius.xs4all.nl>
        <BL0PR12MB253296086906C4A850EC68E6BD5B9@BL0PR12MB2532.namprd12.prod.outlook.com>
Organization: Red Hat
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Mon, 3 May 2021 13:59:43 +0000
Vikram Sethi <vsethi@nvidia.com> wrote:

> > From: Mark Kettenis <mark.kettenis@xs4all.nl> =20
> > > From: Marc Zyngier <maz@kernel.org> =20
>=20
> snip
> > > If, by enumerating the properties of Prefetchable, you can show that
> > > they are a strict superset of Normal_NC, I'm on board. I haven't seen
> > > such an enumeration so far.
> > > =20
> snip
> > > Right, so we have made a small step in the direction of mapping
> > > "prefetchable" onto "Normal_NC", thanks for that. What about all the
> > > other properties (unaligned accesses, ordering, gathering)? =20
> >  =20
> Regarding gathering/write combining, that is also allowed to prefetchable=
 per PCI spec

As others have stated, gather/write combining itself is not well
defined.

> From 1.3.2.2 of 5/0 base spec:
> A PCI Express Endpoint requesting memory resources through a BAR must set=
 the BAR's Prefetchable bit unless
> the range contains locations with read side-effects or locations in which=
 the Function does not tolerate write
> merging.

"write merging"  This is a very specific thing, per PCI 3.0, 3.2.6:

  Byte Merging =E2=80=93 occurs when a sequence of individual memory writes
  (bytes or words) are merged into a single DWORD.

The semantics suggest quadword support in addition to dword, but don't
require it.  Writes to bytes within a dword can be merged, but
duplicate writes cannot.

It seems like an extremely liberal application to suggest that this one
write semantic encompasses full write combining semantics, which itself
is not clearly defined.

> Further 7.5.1.2.1 says " A Function is permitted
> to mark a range as prefetchable if there are no side effects on reads, th=
e Function returns all bytes on reads regardless of
> the byte enables, and host bridges can merge processor writes into this r=
ange139 without causing errors"
>=20
> The "regardless of byte enables" suggests to me that unaligned is OK, as =
only=20
> certain byte enables may be set, what do you think?
>=20
> So to me prefetchable in PCIe spec allows for write combining, read witho=
ut

Ironically here, the above PCI spec section defining write merging has
separate sections for "combining", "merging", and "collapsing".  Only
merging is indicated as a requirement for prefetchable resources.

> sideeffect (prefetch/speculative as long as uncached), and unaligned. Reg=
arding
> ordering I didn't find a statement one way or other in PCIe prefetchable =
definition, but
> I think that goes beyond what PCIe says or doesn't say anyway since reord=
ering can=20
> also happen in the CPU, and since driver must be aware of correctness iss=
ues in its=20
> producer/consumer models it will need the right barriers where they are r=
equired=20
> for correctness anyway (required for the driver/userspace to work on host=
 w/ ioremap_wc).

A lot of hand waving here, imo.  Thanks,

Alex