Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
Date:   Fri, 13 Oct 2023 10:45:41 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Catalin Marinas <catalin.marinas@arm.com>
Cc:     Will Deacon <will@kernel.org>,
        Lorenzo Pieralisi <lpieralisi@kernel.org>, ankita@nvidia.com,
        maz@kernel.org, oliver.upton@linux.dev, aniketa@nvidia.com,
        cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com,
        vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com,
        jhubbard@nvidia.com, danw@nvidia.com,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v1 2/2] KVM: arm64: allow the VM to select DEVICE_* and
 NORMAL_NC for IO memory
Message-ID: <20231013134541.GP3952@nvidia.com>
References: <ZQIFfqgR5zcidRR3@nvidia.com>
 <ZRKW6uDR/+eXYMzl@lpieralisi>
 <ZRLiDf204zCpO6Mv@arm.com>
 <ZR6IZwcFNw55asW0@lpieralisi>
 <20231012123541.GB11824@willie-the-truck>
 <ZSf6Ue09IO6QMBs1@arm.com>
 <20231012144807.GA12374@willie-the-truck>
 <ZSgsKSCv-zWgtWkm@arm.com>
 <20231013092934.GA13524@willie-the-truck>
 <ZSlBOiebenPKXBY4@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZSlBOiebenPKXBY4@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?qDEdSY7+scZs+f6xHW90ntRg/00uqpMc4Drxz0NtSF0mW1picpIhEWR5Mbdd?=
 =?us-ascii?Q?uIpwMwoIoG9gYCFBcahIrI7qhzTZqx58ALaOuTUNf48FGI2ufBKvWG5nbqBo?=
 =?us-ascii?Q?xWuW0Is/wXw312ykg2i49kGAXyp2Q30IH2vdsQVJy8M/ZgEcWNm7ur4cUFO0?=
 =?us-ascii?Q?FMBS2pDc7w/Ttt16cgLapJtxoodLZcgWdCaLT3hyrJM+JSdnOFnMD/eOcips?=
 =?us-ascii?Q?l1bbynqiEsbGU5fU6O0D7ZeJINLFAxgZA8imc9z1yjXvqE74G6IdymhHuNxk?=
 =?us-ascii?Q?pv75CpUBz2AL8I3qlgsVFluQSlQ0mPYkDA805XuYIkAciglyhc8hg5dt3yA3?=
 =?us-ascii?Q?qDQgDLYOY7yMZ4WTqE7bNHMC1gJnH3qthKDpTrJKWeko1bV1FToLQsrjiqpr?=
 =?us-ascii?Q?lmSpdxTHO2BOKk8ixRs3Wv0LrKw5x1SSVtVG0E3rEYr/PvuWCiZn/ycvhyzO?=
 =?us-ascii?Q?Wg5p6uZLYGZeFb2cVjyci+sSA93maD0+73GItHFCbiZ8s7XMTobMDqmK7lFm?=
 =?us-ascii?Q?NxSaSPyV4lyb90wH6dK0pKF4/v2fGgDwTbc1Q2nGPGRRCrfwYsecYZjLa3ZA?=
 =?us-ascii?Q?OnPPoV7xBcsmMkG7dv1F4yHQvYHgRCg6L1xbl3sN+5nnt+UerOq5ji1fQ6+L?=
 =?us-ascii?Q?AIJAuCHUDSH9a1QcHtw/kYo1t+4n6TF5VDasBHSINhiVVbT02p831nFDKm0C?=
 =?us-ascii?Q?EKhlIQRE2VD2vA4Mmy2AkvEssNRZWQtQm31z7byjLYQHM/idyUhHUpf33jT/?=
 =?us-ascii?Q?L1L/lXVymVxp+NDdyUZZ1PsKn1Vsuew6XxbP+Q923DGERkkkOjrmglwHEIVW?=
 =?us-ascii?Q?vy69QqQThXSCjPMKodxxgKovjvn0JDoe3nai8M4Swj/Y5CSpfXnNfpw7yFuF?=
 =?us-ascii?Q?sHhr+u9hu8kjpjF0cNXs/Mj6ILCQMsxjBQAP0ZMGSCCVCvn57rVJeAx8HQ/p?=
 =?us-ascii?Q?bwxycaeSOQwVVDUbFaVyuScQps6wYiHIbfLqYABTNmvDhuSqlYgq4+Zh60mM?=
 =?us-ascii?Q?I8u59AI7EXBnk79xejoittRqPcFDbRXTbPqVrIRx45tVI8atknjDcdGTTu3A?=
 =?us-ascii?Q?+Ee+W556HqdzinvrPO2+kYqCLIZJRhCVCypOrzYQpC5pmbpAoj77wMHIMjH+?=
 =?us-ascii?Q?drgbQ5bCTfLKgRc/MSVl4OdVBNshs2eyPgxcqzJp4yXvcWLuML3C1fV9NXiQ?=
 =?us-ascii?Q?FArWnYpF4czwb4uzf6DLDF3K12NS8cyU9PaCfkPlialYDjfvRMsW4EwjDiV3?=
 =?us-ascii?Q?cTWwcdLMC4nYZUx84det0C3XUcGPTfnnf5rXPLCny7lfewZVWOEraBB6g69s?=
 =?us-ascii?Q?87Dzr0UHD4EcTPYm9sSHsdcSFzaizpjmh5cWJb42RPbkiqK9EeCKSSfA5p/L?=
 =?us-ascii?Q?bTPTM6+hwTPHQ+f76HU6fH6ST+vzRXaBXh0fYt53tvibAWQ6oVhahtPPZloO?=
 =?us-ascii?Q?3y+hyjT1DVMctkw08MP0kU4d0aV/Lo3GKJJHrshLAzrg724cgQHvDLrDvM9T?=
 =?us-ascii?Q?OiTmjKLPKgicdTtCPgP6nH4kWGS3fYN6f9VezRYEthDGiGQQx6cGJJWjHaT3?=
 =?us-ascii?Q?P17r6rH6+lMGBy79sjA=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d725e508-9ce7-43df-3774-08dbcbf2b247
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Oct 2023 13:45:43.8223
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: wUvYBeYgtxLvcYgniw6CstdGUMv0/hOZlo5NRXz3FtK4y5PggQuK4zBrYkIQi4cM
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7039
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 13 Oct 2023 06:46:01 -0700 (PDT)

On Fri, Oct 13, 2023 at 02:08:10PM +0100, Catalin Marinas wrote:
> On Fri, Oct 13, 2023 at 10:29:35AM +0100, Will Deacon wrote:
> > On Thu, Oct 12, 2023 at 06:26:01PM +0100, Catalin Marinas wrote:
> > > On Thu, Oct 12, 2023 at 03:48:08PM +0100, Will Deacon wrote:
> > > > Claiming back the device also seems strange if the guest has been using
> > > > non-cacheable accesses since I think you could get write merging and
> > > > reordering with subsequent device accesses trying to reset the device.
> > > 
> > > True. Not sure we have a good story here (maybe reinvent the DWB barrier ;)).
> > 
> > We do have a good story for this part: use Device-nGnRE!
> 
> Don't we actually need Device-nGnRnE for this, coupled with a DSB for
> endpoint completion?
> 
> Device-nGnRE may be sufficient as a read from that device would ensure
> that the previous write is observable (potentially with a DMB if
> accessing separate device regions) but I don't think we do this now
> either. Even this, isn't it device-specific? I don't know enough about
> PCIe, posted writes, reordering, maybe others can shed some light.
> 
> For Normal NC, if the access doesn't have side-effects (or rather the
> endpoint is memory-like), I think we are fine. The Stage 2 unmapping +
> TLBI + DSB (DVM + DVMSync) should ensure that a pending write by the CPU
> was pushed sufficiently far as not to affect subsequent writes by other
> CPUs.
> 
> For I/O accesses that change some state of the device, I'm not sure the
> TLBI+DSB is sufficient. But I don't think Device nGnRE is either, only
> nE + DSB as long as the PCIe device plays along nicely.

Can someone explain this concern a little more simply please?

Let's try something simpler. I have no KVM. My kernel driver 
creates a VMA with pgprot_writecombine (NormalNC). Userpsace does a
write to the NormalNC and immediately unmaps the VMA

What is the issue?

And then how does making KVM the thing that creates the NormalNC
change this?

Not knowing the whole details, here is my story about how it should work:

Unmapping the VMA's must already have some NormalNC friendly ordering
barrier across all CPUs or we have a bigger problem. This barrier
definately must close write combining.

VFIO issues a config space write to reset the PCI function.  Config
space writes MUST NOT write combine with anything. This is already
impossible for PCIe since they are different TLP types at the PCIe
level.

By the PCIe rules, config space write must order strictly after all
other CPU's accesses. Once the reset non-posted write returns back to
VFIO we know that:

 1) There is no reference in any CPU page table to the MMIO PFN
 2) No CPU has pending data in any write buffer
 3) The interconnect and PCIe fabric have no inflight operations
 4) The device is in a clean post-reset state

?

> knows all the details. The safest is for the VMM to keep it as Device (I
> think vfio-pci goes for the strongest nGnRnE).

We are probably going to allow VFIO to let userspace pick if it should
be pgprot_device or pgprot_writecombine.

The alias issue could be resolved by teaching KVM how to insert a
physical PFN based on some VFIO FD/dmabuf rather than a VMA so that
the PFNs are never mapped in the hypervisor side.

Jsaon