Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
Date:   Fri, 20 Oct 2023 08:47:19 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Catalin Marinas <catalin.marinas@arm.com>
Cc:     Will Deacon <will@kernel.org>,
        Lorenzo Pieralisi <lpieralisi@kernel.org>, ankita@nvidia.com,
        maz@kernel.org, oliver.upton@linux.dev, aniketa@nvidia.com,
        cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com,
        vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com,
        jhubbard@nvidia.com, danw@nvidia.com,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v1 2/2] KVM: arm64: allow the VM to select DEVICE_* and
 NORMAL_NC for IO memory
Message-ID: <20231020114719.GE3952@nvidia.com>
References: <20231012123541.GB11824@willie-the-truck>
 <ZSf6Ue09IO6QMBs1@arm.com>
 <20231012144807.GA12374@willie-the-truck>
 <ZSgsKSCv-zWgtWkm@arm.com>
 <20231013092934.GA13524@willie-the-truck>
 <ZSlBOiebenPKXBY4@arm.com>
 <20231013134541.GP3952@nvidia.com>
 <ZTEN_oe97VRWbnHb@arm.com>
 <20231019115142.GQ3952@nvidia.com>
 <ZTJizmxV4Gtb4g3S@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZTJizmxV4Gtb4g3S@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?508zRRgYQPNjJm7IjaIJOmYXEpD9dbYPbTbffLg1kL9bl2a7d1/ENKi+MI3K?=
 =?us-ascii?Q?fdjJI4DqUhEzamNMkUpwch1m49S3DbDe7VhNZ1mRy2YC3KtBdCAuV1HrDzJ6?=
 =?us-ascii?Q?FzVbPuqmIJeZzvAkwibTOLOH6UeT72ecz34fMv4qBfJahbM+WZbCw8tBzcOT?=
 =?us-ascii?Q?U+R33Rz3f7ZMIXyHzveD5gCbvU5CNGMDLeVxpwX+wUje811tT6xliVqQrOVn?=
 =?us-ascii?Q?JAcrBSEiuccQy5IcCuW1ManPWevYvgYRYbLnXvmkBam6SP7FiwHJ8YOiTegE?=
 =?us-ascii?Q?Z1ZkRo5bPTht9IZkBfVDwMmFLtSfwlvLCB4lP3weaPU3cfuLO5AXQmv59HPm?=
 =?us-ascii?Q?rnTHKRrxGiobVlWISZ5JghQhUi1Bf7LdeJL/qempwGYP/nGUZkeESJWtYpoH?=
 =?us-ascii?Q?2Q+raGXNv4RQiY1cIhKoW0K+SAMWeFQBR5i4XvMeB/ey70vbpYiJnRj269XM?=
 =?us-ascii?Q?cOa5fWHLmH/a4mLG6rOdCulwpJYH3TGvI4MuNxvEtKTrEJCGEeKdLFegx3a+?=
 =?us-ascii?Q?0kWh+PQOARD4v9WvYY4rVBydhUjC/X2pxqvXVf346SUXyBYxpOz1Xatg2NG8?=
 =?us-ascii?Q?WfuSTorbxGZXvh9nHifTPOMscg4g+AgY3NLQDC5l8QYKhMdnyXewHww20PN7?=
 =?us-ascii?Q?Ffk/M4cHszzgq6rZRDQWAyrfYihRVfwVwnQB1QQzxO4GbfIiZoMEbp9s9hEj?=
 =?us-ascii?Q?cRW8D2/lChuUfSTKO3yoagbLO2fQNvPZVGtgUXn9nYTYejCNlRCIAwcVGupw?=
 =?us-ascii?Q?QeUEeLPxoZ2rDVw0DWzGzLLmbN2pL07o0X6FWpE3y5LjstkbKNNlSPJl3rQP?=
 =?us-ascii?Q?WAuBauPY+Wk2Z6vFuCvBibA1aO9cmJ8Fm1WQvriFOahItWeF2AlTwWOvS8Dc?=
 =?us-ascii?Q?ZnJzHglJK+orYsIfHn7I2tSFskfe2R3xbZWC8i9BmTQIjzDiywnn+aYsJ9ig?=
 =?us-ascii?Q?sxGDRR/S+lsSQNomd3X/WOsJjGuw3+uVxi34XvHbpbSqdrHwHvSdez/H3RWx?=
 =?us-ascii?Q?Tcy70NOHL9979mZZbQoG1i8tcI1wLXS87aXKtvLwZ0vyeRb5G4QdvNQEIcvL?=
 =?us-ascii?Q?kVKeRL3Vq4n+QRaX1Zs5jUg24IgplGXQHux3UCtBcXCxhqDQjjGTSw+wz58C?=
 =?us-ascii?Q?/Hb+nRwPvUyMkIEqOYspdnhQAGjb2ge81gDlZvnpKOMEuHXRCIk8tOHkL7Zh?=
 =?us-ascii?Q?qYlA8TRBFO4dKw807p0olXnnEtQNMxYUAbbajGhA3+swN4YDtOpkDYQu7e6k?=
 =?us-ascii?Q?FireH4U1mW8A9lgTGNprvpXzGL/C2iWnNX8Vk+sy9O48cHeDcp3GEYteDX/G?=
 =?us-ascii?Q?iVEdzAiElwIVGh9cBj8kxC1Ze3w7gDQ2dFyDT75ZQEbtdaULWbKd5vb0as7C?=
 =?us-ascii?Q?+ATEO1ei2IgzAvG1cJ9x3IJcAjLTkR4gsssrxCSGKaAKH7FgXeTrewurukW0?=
 =?us-ascii?Q?ecoNIAbqhO4E14Of/9/fRkbasXrmC3M66fUHkU0tlq76c3qVYNtDOKlxXaxJ?=
 =?us-ascii?Q?AcoSTYSxwIdm5Y3/XgZ7IzHBvzOCrFHK3tvMN7OCeJVZqvMzY97gzPYOL7cN?=
 =?us-ascii?Q?Sg+5OJVYvX+0BWXZDWYdQ2q7WF3Co49l/NRyf9xN?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 075b1314-11b0-48c7-e415-08dbd16251b2
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2023 11:47:21.2077
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ld1mK+N1B4Azr5J1SQtLnMaeOeFwIu2g/DkDBaHcvzvLnobmW00p2HgrNgDklFRn
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8956
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 20 Oct 2023 04:47:34 -0700 (PDT)

On Fri, Oct 20, 2023 at 12:21:50PM +0100, Catalin Marinas wrote:
> On Thu, Oct 19, 2023 at 08:51:42AM -0300, Jason Gunthorpe wrote:
> > On Thu, Oct 19, 2023 at 12:07:42PM +0100, Catalin Marinas wrote:
> > > Talking to Will earlier, I think we can deem the PCIe scenario
> > > (somewhat) safe but not as a generic mechanism for other non-PCIe
> > > devices (e.g. platform). With this concern, can we make this Stage 2
> > > relaxation in KVM only for vfio-pci mappings? I don't have an example of
> > > non-PCIe device assignment to figure out how this should work though.
> > 
> > It is not a KVM problem. As I implied above it is VFIO's
> > responsibility to reliably reset the device, not KVMs. If for some
> > reason VFIO cannot do that on some SOC then VFIO devices should not
> > exist.
> > 
> > It is not KVM's job to double guess VFIO's own security properties.
> 
> I'd argue that since KVM is the one relaxing the memory attributes
> beyond what the VFIO driver allows the VMM to use, it is KVM's job to
> consider the security implications. This is fine for vfio-pci and
> Normal_NC but I'm not sure we can generalise.

I can see that, but I belive we should take this responsibility into
VFIO as a requirement. As I said in the other email we do want to
extend VFIO to support NormalNC VMAs for DPDK, so we need to take this
anyhow.

> > Specifically about platform the generic VFIO platform driver is the
> > ACPI based one. If the FW provides an ACPI method for device reset
> > that is not properly serializing, that is a FW bug. We can quirk it in
> > VFIO and block using those devices if they actually exist.
> > 
> > I expect the non-generic VFIO platform drivers to take care of this
> > issue internally with, barriers, read from devices, whatver is
> > required to make their SOCs order properly. Just as I would expect a
> > normal Linux platform driver to directly manage whatever
> > implementation specific ordering quirks the SOC may have.
> 
> This would be a new requirement if an existing VFIO platform driver
> relied on all mappings being Device. But maybe that's just theoretical
> at the moment, are there any concrete examples outside vfio-pci? If not,
> we can document it as per Lorenzo's suggestion to summarise this
> discussion under Documentation/.

My point is if this becomes a real world concern we have a solid
answer on how to resolve it - fix the VFIO driver to have a stronger
barrier before reset.

I'm confident it is not a problem for PCI and IIRC the remaining ARM
platform drivers were made primarily for DPDK, not KVM.

So I agree with documenting and perhaps a comment someplace in VFIO is
also warranted.

Jason