Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Vikram Sethi <vsethi@nvidia.com>
To:     Mark Kettenis <mark.kettenis@xs4all.nl>,
        Marc Zyngier <maz@kernel.org>
CC:     Shanker Donthineni <sdonthineni@nvidia.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "will@kernel.org" <will@kernel.org>,
        "catalin.marinas@arm.com" <catalin.marinas@arm.com>,
        "christoffer.dall@arm.com" <christoffer.dall@arm.com>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "kvmarm@lists.cs.columbia.edu" <kvmarm@lists.cs.columbia.edu>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        Jason Sequeira <jsequeira@nvidia.com>
Subject: RE: [RFC 1/2] vfio/pci: keep the prefetchable attribute of a BAR
 region in VMA
Thread-Topic: [RFC 1/2] vfio/pci: keep the prefetchable attribute of a BAR
 region in VMA
Thread-Index: AQHXPRTigOHff2apwEeawQns3nBPOqrL0MoAgAAM5gCAAAj7gIABBh4AgAAGT4CAADU7AIAACSoAgAABt+CAASvZAIACHFDwgAEVfYCAADdZj4AAAavQ
Date:   Mon, 3 May 2021 13:59:43 +0000
Message-ID: <BL0PR12MB253296086906C4A850EC68E6BD5B9@BL0PR12MB2532.namprd12.prod.outlook.com>
References: <20210429162906.32742-1-sdonthineni@nvidia.com>
 <20210429162906.32742-2-sdonthineni@nvidia.com>
 <20210429122840.4f98f78e@redhat.com>
 <470360a7-0242-9ae5-816f-13608f957bf6@nvidia.com>
 <20210429134659.321a5c3c@redhat.com>
 <e3d7fda8-5263-211c-3686-f699765ab715@nvidia.com>
 <87czucngdc.wl-maz@kernel.org>
 <1edb2c4e-23f0-5730-245b-fc6d289951e1@nvidia.com>
 <878s4zokll.wl-maz@kernel.org>
 <BL0PR12MB2532CC436EBF626966B15994BD5E9@BL0PR12MB2532.namprd12.prod.outlook.com>
 <87eeeqvm1d.wl-maz@kernel.org>
 <BL0PR12MB25329EF5DFA7BBAA732064A7BD5C9@BL0PR12MB2532.namprd12.prod.outlook.com>
 <87bl9sunnw.wl-maz@kernel.org> <c1bd514a531988c9@bloch.sibelius.xs4all.nl>
In-Reply-To: <c1bd514a531988c9@bloch.sibelius.xs4all.nl>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 4c19da9f-8698-4e06-ebb8-08d90e3bb404
X-MS-Exchange-CrossTenant-originalarrivaltime: 03 May 2021 13:59:43.3667
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: lCO7HUdTBV/WRyvNvWJRiVSjeWjygIkM4Nz0oIIiLptXif1SCsq0nXnv+D8DbINj00x+qLqyf9JnVgtGDdeiQA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3856
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> From: Mark Kettenis <mark.kettenis@xs4all.nl>
> > From: Marc Zyngier <maz@kernel.org>

snip
> > If, by enumerating the properties of Prefetchable, you can show that
> > they are a strict superset of Normal_NC, I'm on board. I haven't seen
> > such an enumeration so far.
> >
snip
> > Right, so we have made a small step in the direction of mapping
> > "prefetchable" onto "Normal_NC", thanks for that. What about all the
> > other properties (unaligned accesses, ordering, gathering)?
>=20
Regarding gathering/write combining, that is also allowed to prefetchable p=
er PCI spec
From 1.3.2.2 of 5/0 base spec:
A PCI Express Endpoint requesting memory resources through a BAR must set t=
he BAR's Prefetchable bit unless
the range contains locations with read side-effects or locations in which t=
he Function does not tolerate write
merging.
Further 7.5.1.2.1 says " A Function is permitted
to mark a range as prefetchable if there are no side effects on reads, the =
Function returns all bytes on reads regardless of
the byte enables, and host bridges can merge processor writes into this ran=
ge139 without causing errors"

The "regardless of byte enables" suggests to me that unaligned is OK, as on=
ly=20
certain byte enables may be set, what do you think?

So to me prefetchable in PCIe spec allows for write combining, read without
sideeffect (prefetch/speculative as long as uncached), and unaligned. Regar=
ding
ordering I didn't find a statement one way or other in PCIe prefetchable de=
finition, but
I think that goes beyond what PCIe says or doesn't say anyway since reorder=
ing can=20
also happen in the CPU, and since driver must be aware of correctness issue=
s in its=20
producer/consumer models it will need the right barriers where they are req=
uired=20
for correctness anyway (required for the driver/userspace to work on host w=
/ ioremap_wc).

But perhaps the bigger question is since WC doesn't exist as a Memory type
on armv8, yet we are trying to fit something onto ioremap_wc which came fro=
m
x86 world, shouldn't the arm64 MT we use for WC match the semantics of=20
whatever drivers and userspace expected from ioremap_wc as defined on=20
x86, which as Mark notes below includes unaligned? If we agree to that,=20
we can codify it in the documentation of ioremap_wc and allow for
Normal NC on arm64 for ioremap_wc in host or guest.=20
Beyond that, if we don't want to do it automatically based on prefetchable
but from explicit call from userspace is fine too.=20

> On x86 WC:
>=20
> 1. Is not cached (but stores are buffered).
>=20
> 2. Allows unaligned access just like normal memory.
>=20
> 3. Allows speculative reads.
>=20
> 4. Has weaker ordering than normal memory; [lsm]fence instructions are
>    needed to guarantee a particular ordering of writes with respect to
>    other writes and reads.
>=20
> 5. Stores are buffered.  This buffer isn't snooped so it has to be
>    flushed before changes are globally visible.  The [sm]fence
>    instructions flush the store buffer.
>=20
> 6. The store buffer may combine multiple writes into a single write.
>=20
> Now whether the fact the unaligned access is allowed is really part of th=
e
> semantics of WC mappings is debatable as x86 always allows unaligned
> access, even for areas mapped with ioremap().
>=20
> However, this is where userland comes in.  The userland graphics stack do=
es
> assume that graphics memory mapped throug a prefetchable PCIe BAR
> allows unaligned access if the architecture allows unaligned access for
> cacheable memory.  On arm64 this means that such memory needs to be
> "Normal NC".  And since kernel drivers tend to map such memory using
> ioremap_wc() that pretty much implies ioremap_wc() shoul use "Normal NC"
> as well isn't it?
>=20
> > > > How do we translate this into something consistent? I'd like to
> > > > see an actual description of what we *really* expect from WC on
> > > > prefetchable PCI regions, turn that into a documented definition
> > > > agreed across architectures, and then we can look at implementing
> > > > it with one memory type or another on arm64.
> > > >
> > > > Because once we expose that memory type at S2 for KVM guests, it
> > > > becomes ABI and there is no turning back. So I want to get it
> > > > right once and for all.
> > > >
> > > I agree that we need a precise definition for the Linux ioremap_wc
> > > API wrt what drivers (kernel and userspace) can expect and whether
> > > memset/memcpy is expected to work or not and whether aligned
> > > accesses are a requirement.
> > > To the extent ABI is set, I would think that the ABI is also already
> > > set in the host kernel for arm64 WC =3D Normal NC, so why should that
> > > not also be the ABI for same driver in VMs.
> >
> > KVM is an implementation of the ARM architecture, and doesn't really
> > care about what WC is. If we come to the conclusion that Normal_NC is
> > the natural match for Prefetchable attributes, than we're good and we
> > can have Normal_NC being set by userspace, or even VFIO. But I don't
> > want to set it only because "it works when bare-metal Linux uses it".
> > Remember KVM doesn't only run Linux as guests.
> >
> >       M.
> >
> > --
> > Without deviation from the norm, progress is not possible.
> >
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> >