Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8;
Date:   Fri, 20 Oct 2023 11:28:58 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Lorenzo Pieralisi <lpieralisi@kernel.org>
Cc:     Catalin Marinas <catalin.marinas@arm.com>,
        Will Deacon <will@kernel.org>, ankita@nvidia.com,
        maz@kernel.org, oliver.upton@linux.dev, aniketa@nvidia.com,
        cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com,
        vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com,
        jhubbard@nvidia.com, danw@nvidia.com,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v1 2/2] KVM: arm64: allow the VM to select DEVICE_* and
 NORMAL_NC for IO memory
Message-ID: <20231020142858.GH3952@nvidia.com>
References: <20231012144807.GA12374@willie-the-truck>
 <ZSgsKSCv-zWgtWkm@arm.com>
 <20231013092934.GA13524@willie-the-truck>
 <ZSlBOiebenPKXBY4@arm.com>
 <20231013134541.GP3952@nvidia.com>
 <ZTEN_oe97VRWbnHb@arm.com>
 <20231019115142.GQ3952@nvidia.com>
 <ZTJizmxV4Gtb4g3S@arm.com>
 <20231020114719.GE3952@nvidia.com>
 <ZTKIzZk3lGATLkGz@lpieralisi>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZTKIzZk3lGATLkGz@lpieralisi>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?cGTHAByW6su7k3MekRbhWlttP8szHOVbtAt0wTIpCdhOVM9FhQ2hwD+M0yyb?=
 =?us-ascii?Q?f89KGHxdqD+l1LChI3Wzz0sNAxt1KhcmjngaVzSw1njjDOWGHS998b154XwU?=
 =?us-ascii?Q?l5ZnpyRt1lnZWRiO3Csx/c20gpU+dX7yvcDwuWcu7ath6MvW/rRPhoG0ymWQ?=
 =?us-ascii?Q?08NYFuUF+M23RkIkYLXaIQEauVzJ7jJcvvsP6l+pdG2C0fL8r4mMN1/QPeoD?=
 =?us-ascii?Q?NBocdzragffQEqLTpV10WYDmXjdwhJrs+5ALoW2BM5fXcdZxPt/ntYyUTy/n?=
 =?us-ascii?Q?x84lxCRwSrS/pfY4TXPemyivlh5CbG4cKsMpATNfpuO8fWVPEBatgNBS9BgP?=
 =?us-ascii?Q?wPh4X/t1kPMWUEAPM/Jx608aA6EAoFAGR7a/LLvKrItvi0AWslVSWsJ8V7ht?=
 =?us-ascii?Q?Oug6iUelW2DXcnWiwfNhlB+di0UuV/tw9/so4BEPz5R6pS4vfMbtSfi+gt96?=
 =?us-ascii?Q?kbK+AYeVxWqnJ7j29pkqlnZlbU0JK/j63coQQp8P/WfslWkHuYXcQcEiTZOL?=
 =?us-ascii?Q?rodGIdGtQkXRvJJSzvkK2i+MedXs79+vrTHz3DUbGyIbAzBcQ12FzxmGYx1k?=
 =?us-ascii?Q?OBPCPE3HDBVvBgBJxagl28sqxY3aKACmQ2xROnHguQc2dxd9GY12UNJ6e6l+?=
 =?us-ascii?Q?xqG7pFMxEm9hI4wU1zZxRpHlS9QezyqyHbybAjwbCbQgHXt+9j+CIWnrFI5a?=
 =?us-ascii?Q?qc+CjpsAwi8oIjaR0PnbflYrAs3V1Y0fzZ94+o3yG6zW8qOHIOE80XqfMAzH?=
 =?us-ascii?Q?B3dgMk5Z9q8fti1zndYZu700jI8iohv7is73KWFgFobVx0TgvYRZrScCMw0N?=
 =?us-ascii?Q?6pXM70OU7AJ8qq2U/VesQ8/LOTgtn4zgwirPMVxzTfs723mWZbiReV5wi0tp?=
 =?us-ascii?Q?FJrv9BygO3h0Wm68nwiy5oVavps1853wuXP9cfUEmw3XhHece1SHhfzMnrTP?=
 =?us-ascii?Q?AjNUPyF5OWf7oR/5sAEYna380LhKIKUBJDsF83OUEA9KoCTkc2ozxGX1eY/a?=
 =?us-ascii?Q?gmodiIB0avUltLCOvOGfZ4uy0uCRKy94Kky1BtIZ11IVtyv8HRyjlhep2+uT?=
 =?us-ascii?Q?TmE13xxIuw8b8Zm9YbjqSR2PrkmYf2pp92RwiGwf5VWgL89iGL2fC7e2yRF/?=
 =?us-ascii?Q?MtDd4xuLu2A/vDHdfZKbmmwbDk3uNglZ06q3M/h1gshBFoaJBKMa4pYJyPGi?=
 =?us-ascii?Q?levDvWIc/jzu25KG06VTaZkMv1H6XcNWCEjlreBa9/x7uri+IkkW76E2M2MU?=
 =?us-ascii?Q?r8P7U06LlrVcBnWbTSsmp0H/XD+efBxdIRfLringJwxh4IOJHo6tXBMAhxo7?=
 =?us-ascii?Q?5txcJag+FJ2/zFMu7h9JgA08M/Ns3IiPdVm1vVQfT2+EbnKrmvJY2DQQjXK1?=
 =?us-ascii?Q?uGB57ozuHyTiIugpCkhSrmpUJoCBnEzVwapeZIjT9lRp9lH1sMP+u13BZxVz?=
 =?us-ascii?Q?72De8AG+DeMaQCOkOIEq7ssBrUGhD60q84WoG9JTIalPCNr3W50uFfc06XNu?=
 =?us-ascii?Q?nshLXKoAy9SJdUbINTrgAzeVLMoi+PWikKJvPcLkIJPHckRdxJS6Crh4Ov9X?=
 =?us-ascii?Q?ePTKgROF6/E+a/qHmUo4+NV9mEXBDQ+GuGCejitp?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1e5dfbf9-3986-4971-8140-08dbd178e6e9
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2023 14:29:00.5505
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: jCf3L0/lqfLAsMBojMyLdKRWktbNRmnCsCF/Z8QYfGDIUFsYREvGBZAPcPTgPD/F
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7860
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 20 Oct 2023 07:29:21 -0700 (PDT)

On Fri, Oct 20, 2023 at 04:03:57PM +0200, Lorenzo Pieralisi wrote:
> > My point is if this becomes a real world concern we have a solid
> > answer on how to resolve it - fix the VFIO driver to have a stronger
> > barrier before reset.
> 
> Just to make sure I am parsing this correctly: this case above is
> related to a non-PCI VFIO device passthrough where a guest would want to
> map the device MMIO at stage-1 with normal-NC memory type (well, let's
> say with a memory attribute != device-nGnRE - that combined with the new
> stage-2 default might cause transactions ordering/grouping trouble with
> eg device resets), correct ? 

This is what I have understood was Will's concern, yes.

> IIRC, all requests related to honouring "write-combine" style
> stage-1 mappings were for PCI(e) devices but that's as far as what
> *I* was made aware of goes.

Yes, this is what I am aware of as well.

Though I do not object to the idea from the VFIO side that platform
devices would also have to support NormalNC too.

The theoretical missing peice is that someone would say they have a
SOC issue XYZ and thus their VFIO platform devices must fully block
NormalNC. I suggest if someone comes with this and really, really
wants VFIO, then we could use a VMA flag to indicate that KVM must not
upgrade it. Currently I have no knowledge of such a thing existing.

With PCI we've made the argument that if NormalNC is broken unsafe for
KVM in the SOC then probably so is Device-*. I think the same basic
argument holds for platform devices too. Thus I'm skeptical that
someone can come and say they have SOC issue XYZ and NormalNC is
broken but Device-* is perfectly safe.

> We will do that, I will start adding the recent discussions to the
> new documentation file. Side note: for those who attend LPC it would be
> useful to review the resulting documentation together there, it should
> happen around v6.7-rc1.

I will be there, let me know

Thanks,
Jason