Received-SPF: pass (google.com: domain of linux-kernel+bounces-185198-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Tue, 21 May 2024 13:07:14 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: Alex Williamson <alex.williamson@redhat.com>,
	"Vetter, Daniel" <daniel.vetter@intel.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"pbonzini@redhat.com" <pbonzini@redhat.com>,
	"seanjc@google.com" <seanjc@google.com>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"luto@kernel.org" <luto@kernel.org>,
	"peterz@infradead.org" <peterz@infradead.org>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"bp@alien8.de" <bp@alien8.de>, "hpa@zytor.com" <hpa@zytor.com>,
	"corbet@lwn.net" <corbet@lwn.net>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
	"Liu, Yi L" <yi.l.liu@intel.com>
Subject: Re: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <20240521160714.GJ20229@nvidia.com>
References: <20240507061802.20184-1-yan.y.zhao@intel.com>
 <20240507062138.20465-1-yan.y.zhao@intel.com>
 <20240509121049.58238a6f.alex.williamson@redhat.com>
 <Zj33cUe7HYOIfj5N@yzhao56-desk.sh.intel.com>
 <20240510105728.76d97bbb.alex.williamson@redhat.com>
 <ZkG9IEQwi7HG3YBk@yzhao56-desk.sh.intel.com>
 <BN9PR11MB52766D78684F6206121590B98CED2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240516143159.0416d6c7.alex.williamson@redhat.com>
 <20240517171117.GB20229@nvidia.com>
 <BN9PR11MB5276250B2CF376D15D16FF928CE92@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5276250B2CF376D15D16FF928CE92@BN9PR11MB5276.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?xWmnCLUj1nFGw6BewyVgPyufuwP19wyl9oM+pe5Alm12P7ZXCWd8cZn+w64w?=
 =?us-ascii?Q?dzVimAXFN767CQcQRDMGW98tlQ+vlHpL3VXg9mcVqu1r32xkXlnDS5qY3yEA?=
 =?us-ascii?Q?rmSAK7MOuVvm7qQ9qyJO51GJRfyaRgwLc2OyM2VEcJx1dkPNC7FSn5cxjBnK?=
 =?us-ascii?Q?j2WCaM8Ptzt0B9oMxsefQ3GKi4/s4dAF/al5IIF82p/eMqqLg4wJCpCgK5zj?=
 =?us-ascii?Q?pblMCvGtqAbrKwytCYMrcQl3xT8Q8W3D6DwpEUD6csdA9m2cK+bQq+zPmYAg?=
 =?us-ascii?Q?bVZloWJ1vCQUevnnTW/+To0X47XtTrxOl5ot7OvIWy7TpO4jvzEfQsOudcDs?=
 =?us-ascii?Q?oYZqKOwfHd6UrpZBdupk0z8pocs4FBvTw1rf6BiKFfSWMTFv7DjUp989jJtE?=
 =?us-ascii?Q?kXIH7R0o5c847+L/MVReDmmFPZ1OjkGFtdToJWq5ncyAJcR8ncL7WUPtbsQR?=
 =?us-ascii?Q?ySnGVJtqsHKa63/URPacEbpKLfWVZnfBwBTGup9ooUfl1u/9+TFQDfZMjVvO?=
 =?us-ascii?Q?TTKZr8DaZfn5FlW7UsPz+ASB/Af4ktqBC6+pX347aQoTVglDG/ub8l1kiIHi?=
 =?us-ascii?Q?BvPpDJa/63fFowKeKKxr30Fyyd7ekYeaUdVvS1fkS7NdgnVWAVOkam30wiSP?=
 =?us-ascii?Q?xFfZaRfnoCKm1iF+3bhQEgwJrQ7eGY7/rz21bHDjgdgRO3w+/ZaA/qf/KjBc?=
 =?us-ascii?Q?vLhY7xcnTPmmJF1Ik3qFKtA0ZL97VlkP/YZ4k6Rx9DT/n1IGK2ZfV/2xcKhK?=
 =?us-ascii?Q?Paofk6NBJzFqIcBXp6SSqkLhk9u4FGgtg8xBrgnDyxVqVlU3k+39uFraIABC?=
 =?us-ascii?Q?Gkjj5aizTvlx8CVtlgebIPTDaHqvEfS5GcxMr1BURrTCC322ed85Gi8Nepr6?=
 =?us-ascii?Q?u8OHYAAyQ86+xdllpycucRhLOMbqZkP16mzCXcxJZu8cCOKa82vJWii66il5?=
 =?us-ascii?Q?MHGGeZTaNlsr45OvuJhximfSxpTN4beX2jIZT6A7istj3Tln6DY/0YKtI+ph?=
 =?us-ascii?Q?MoIRLR6qYr5SMErCw7foAQUbwS3Igzwafx/ElOTiYKwCIzUfHhMX26tWZeZm?=
 =?us-ascii?Q?BJE2dHiQH0L0jzjwjbTadONqsvOfmojB1zZvfFOzD9AZL66SQIxjSmkBOlC9?=
 =?us-ascii?Q?W9yn8w6OVy/ScLNg/3t+c3kOAw9LqDnFYIlDFyYBQVnZLyttQwFQKixSKSYJ?=
 =?us-ascii?Q?hJZoCZQHLWB0mGZh8JRDWb2tJRUfKp2GYVNEt3gl4qBFM4wzdgqPYvtZIDcj?=
 =?us-ascii?Q?8NVE9wdX5dBS439gw0y7o+JVBID2jzA/p/ZJHoLHD3GVrCmvZlSbLSqjJbOl?=
 =?us-ascii?Q?2rDVt5heZ7RtZ9jgcLD0e3ETRWW+a14aUOwJQuWvrzVtzuWrfknZSXTu4IzI?=
 =?us-ascii?Q?zrRkrcIaJSlqpIB2SvBa00kMHAFMky7lugdnYdFZAvHXU9rPlfztYq5t2ue4?=
 =?us-ascii?Q?9RkkK0yTk3ZJM+tNv9Gun7hQb37ttEw+Sya24JvS3bqv5g+C1S0AJ/Z+G5dI?=
 =?us-ascii?Q?L07PogWv9P7l/zWxbL+xchz1fqrx0j/4qES3kUBqomSFwfkWg+fdPMeN7iUP?=
 =?us-ascii?Q?tjkgy3viyNcqAe38bHqPl2gcTf8inVjdoNi3h+mr?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 03ada3f0-2e19-4a0d-65f6-08dc79b01533
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 May 2024 16:07:15.7914
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LofwvGrYrhqrOoecWxv3fVCIY/c8WWolRcK5J2SkibcxaEq2Z4VZDL1ee82JcqAJ
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4318

On Mon, May 20, 2024 at 02:52:43AM +0000, Tian, Kevin wrote:
> +Daniel
> 
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Saturday, May 18, 2024 1:11 AM
> > 
> > On Thu, May 16, 2024 at 02:31:59PM -0600, Alex Williamson wrote:
> > 
> > > Yes, exactly.  Zero'ing the page would obviously reestablish the
> > > coherency, but the page could be reallocated without being zero'd and as
> > > you describe the owner of that page could then get inconsistent
> > > results.
> > 
> > I think if we care about the performance of this stuff enough to try
> > and remove flushes we'd be better off figuring out how to disable no
> > snoop in PCI config space and trust the device not to use it and avoid
> > these flushes.
> > 
> > iommu enforcement is nice, but at least ARM has been assuming that the
> > PCI config space bit is sufficient.
> > 
> > Intel/AMD are probably fine here as they will only flush for weird GPU
> > cases, but I expect ARM is going to be unhappy.
> > 
> 
> My impression was that Intel GPU is not usable w/o non-coherent DMA,
> but I don't remember whether it's unusable being a functional breakage
> or a user experience breakage. e.g. I vaguely recalled that the display
> engine cannot afford high resolution/high refresh rate using the snoop
> way so the IOMMU dedicated for the GPU doesn't implement the force
> snoop capability.
> 
> Daniel, can you help explain the behavior of Intel GPU in case nosnoop
> is disabled in the PCI config space?
> 
> Overall it sounds that we are talking about different requirements. For
> Intel GPU nosnoop is a must but it is not currently done securely so we
> need add proper flush to fix it, while for ARM looks you don't have a
> case which relies on nosnoop so finding a way to disable it is more
> straightforward?

Intel GPU weirdness should not leak into making other devices
insecure/slow. If necessary Intel GPU only should get some variant
override to keep no snoop working.

It would make alot of good sense if VFIO made the default to disable
no-snoop via the config space.

Jason