Received-SPF: pass (google.com: domain of linux-kernel+bounces-186224-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Wed, 22 May 2024 09:29:39 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: Alex Williamson <alex.williamson@redhat.com>,
	"Vetter, Daniel" <daniel.vetter@intel.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"pbonzini@redhat.com" <pbonzini@redhat.com>,
	"seanjc@google.com" <seanjc@google.com>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"luto@kernel.org" <luto@kernel.org>,
	"peterz@infradead.org" <peterz@infradead.org>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"bp@alien8.de" <bp@alien8.de>, "hpa@zytor.com" <hpa@zytor.com>,
	"corbet@lwn.net" <corbet@lwn.net>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
	"Liu, Yi L" <yi.l.liu@intel.com>
Subject: Re: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <20240522122939.GT20229@nvidia.com>
References: <BN9PR11MB52766D78684F6206121590B98CED2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240516143159.0416d6c7.alex.williamson@redhat.com>
 <20240517171117.GB20229@nvidia.com>
 <BN9PR11MB5276250B2CF376D15D16FF928CE92@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240521160714.GJ20229@nvidia.com>
 <20240521102123.7baaf85a.alex.williamson@redhat.com>
 <20240521163400.GK20229@nvidia.com>
 <20240521121945.7f144230.alex.williamson@redhat.com>
 <20240521183745.GP20229@nvidia.com>
 <BN9PR11MB52769E209C5B978C7094A5C08CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <BN9PR11MB52769E209C5B978C7094A5C08CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?YXRJcXl4c1h1dkdyVVNpTW9lSGFDejdLLzRzVnFOZWZaelgyNHVPMFVDWkFL?=
 =?utf-8?B?bXVJdzlJbFpkKzVtTHBnb3c1RTI0ZzI2eWprOEh4S1ZXRDJkOFlZdDMrREpQ?=
 =?utf-8?B?QjhmVHdiaytvK3kzdmEzZjZDVjgvZjExV3hHSGREL3puYTg5dFB3R1RkMW05?=
 =?utf-8?B?T1BLbWsxaXlGZjZiMnZFU3FGNjBlay9KV2NDQkkzRVRFdGVCMUwyVG9pV0Ur?=
 =?utf-8?B?TFFsa1phVmxZSzlZZ1hzNTB3L0hGM1lidVBvVDBjYVdieWdJMWprdWZsc3Zv?=
 =?utf-8?B?Z2dTS3dIRE5xUGZ4MTUwT2RTR21xSXJTSnVmSHg4TG9mai9jMUpDc2xZMHg0?=
 =?utf-8?B?MnpoakVmVUNjLzl3Zm1rUEp4SWJuNlpOTUY1NURHV3ZyWUtlYThQS0J2Zldx?=
 =?utf-8?B?SmpMNjgrWjRUM216dVQ2OHhoR1pCRWhyWlhKRkh6Z0VuQi9qdVNNU1pzYXlh?=
 =?utf-8?B?ZkZvRGJiL0J0OVBWZE1raHVIQlJWc3JNd2NsTGJFendnTmRBdFpXcGN4cGlH?=
 =?utf-8?B?bHhCL1FDcXlMQnZoWjBVYXdMdllXZVQrKzNvSnMyWUt6dEszZ3pDTStsRGxw?=
 =?utf-8?B?aXFoY0syQ2RVWG8xYVVRdVdiVUlnaWVOWFl5ck9Lb1BkclN3TE1YRTFGWmw1?=
 =?utf-8?B?MHNnaXM1NWt6YXZUVjgxM3ZyNm13ams5cko3UUo2QUs2bFVxeFhOdlRVL3J4?=
 =?utf-8?B?c0MrcEh5Qm40eXkwYzBleEorTmJOYUhDWnhOK0F1Z2ZVNkplV0JCSis4eVFJ?=
 =?utf-8?B?Sm5XcFk5bFdKSVRrTE1tWW9ZQ1VBQm9oc1I1cmdDNkZnTzV0UDJiSHFRTjFP?=
 =?utf-8?B?MGpocFU2SmVKWHBHWm04YUlwMjVYZ0daU0h0elo5QTRCaWlPeGtpbmlEQWsx?=
 =?utf-8?B?bUhObzc2dFZZL1hkU2NsM2ZuZ1Bza3JFZDJkOUNVL1ZvQWx5RGxpTjRqR2ha?=
 =?utf-8?B?YkE2S3JSZXREQWt6WlcwdXk5UzhuVEFVT0NHd0lhUXVRUzFvRHNPZ2YxQ3pX?=
 =?utf-8?B?V3pRTmhRODQ3aXJzajY4UTljRDdvdDQwU25tcDN1R28xU1JHU3Jtd1dtdytF?=
 =?utf-8?B?V0lPZjc4WlI2eVhqdlJETUpsVHJZS2tsUkQ2Z2J5c1gvMHVCcExLYVBpclo0?=
 =?utf-8?B?ZmZhK0V6TFVrUUdJNVBlNjFoOUFZSnB0YmlMaVA0cTNROXpKZHFUbXJldU55?=
 =?utf-8?B?dkM1T0owaWZuUlJyZS9NVkdXTU5md0NSSE5PMWdHMk42VmdlazAzSVJUMmhJ?=
 =?utf-8?B?TnF3WVd4U2xpREQrcFhRVkVqTkJ6VHFJUDVUazE4MEJ3QlRVZWlyWG91M1RK?=
 =?utf-8?B?dnR2dVFkVzhoQU5jNHZvckpQeWxuYS9wM2M4S0hyanptU3ovaEh1N2h2RFF0?=
 =?utf-8?B?RnRKbzdlTHh4OWlZL2RoMUlWWlNQV3M2Yk9HSEYzMlRQOWNsZEN2c2NtNmx0?=
 =?utf-8?B?TUY4cU9xV2J5dVNuYU1mWUtJR2kycGRvOGVSVXY0SEV3MTYvOEluQnljOGxa?=
 =?utf-8?B?cGhXT1FQa3VKampKY3dKaFNSZFBHelpyVEFCbHJaOXVqRmxjL0liY1RkUWQ2?=
 =?utf-8?B?WUdiTUwzU0dJUUVWdmV4MWE2R2RnWXNnM0pYeThkMUF3d1d4OWpwWFVXMlha?=
 =?utf-8?B?MHA4WmN2ckhtRVhuOFJVcThvNGtoeXVjckdLRlNqdHpRRDM1UGJCa1lveVZZ?=
 =?utf-8?B?UDdmc1ZHcU1YbXlQOWZCTTd2aHlIa3RjdVo3UjlvUWY2NXI5Z1ZOMW9NZVhE?=
 =?utf-8?B?Yk5KSEFNQW9zUnhCWDdPNm0zdXNDQjlBMTREelQyZFNQZzc5V3hQVk1PYUpD?=
 =?utf-8?B?dWNTamtPby9xMDlpMkV1VFZBbXB0REVXZm16SDhzaUNDOGxRZGZuNnZjQ3lK?=
 =?utf-8?B?RmVNSFdab2xuNVVWM2FCNk9nZ0UrZ2ZrNzN0eGlQalR6T3BtTko0UWIwZUNU?=
 =?utf-8?B?Nk1KeG92OFhEb2U2aXhjK3p4cmM2a05qYmRlTzRtQkkybjV5SW9aUDhqdWMz?=
 =?utf-8?B?TzBQbUxBSm95ajhUcUphQm1lcnBMV1gvdnQyZUpoTEMzK050ZXFmTVhyOTlH?=
 =?utf-8?B?ekRNbkw0ZzRxbzJtc3hwUDRrWTRhd09tRzIwNTc4WWlyZmtKV2FpZHBuVmFK?=
 =?utf-8?Q?isJ0=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4167d802-ee59-44ac-7fef-08dc7a5adad3
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 May 2024 12:29:41.7713
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: x4JZP/t1c5Vrk4lSq5X3VqWE0L0BV8DVsZq6Epl9TTiGb2ixsz2V1C4/Mi1RaLvP
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7126

On Wed, May 22, 2024 at 06:24:14AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, May 22, 2024 2:38 AM
> > 
> > On Tue, May 21, 2024 at 12:19:45PM -0600, Alex Williamson wrote:
> > > > I'm OK with this. If devices are insecure then they need quirks in
> > > > vfio to disclose their problems, we shouldn't punish everyone who
> > > > followed the spec because of some bad actors.
> > > >
> > > > But more broadly in a security engineered environment we can trust the
> > > > no-snoop bit to work properly.
> > >
> > >  The spec has an interesting requirement on devices sending no-snoop
> > >  transactions anyway (regarding PCI_EXP_DEVCTL_NOSNOOP_EN):
> > >
> > >  "Even when this bit is Set, a Function is only permitted to Set the No
> > >   Snoop attribute on a transaction when it can guarantee that the
> > >   address of the transaction is not stored in any cache in the system."
> > >
> > > I wouldn't think the function itself has such visibility and it would
> > > leave the problem of reestablishing coherency to the driver, but am I
> > > overlooking something that implicitly makes this safe?
> > 
> > I think it is just bad spec language! People are clearly using
> > no-snoop on cachable memory today. The authors must have had some
> > other usage in mind than what the industry actually did.
> 
> sure no-snoop can be used on cacheable memory but then the driver
> needs to flush the cache before triggering the no-snoop DMA so it
> still meets the spec "the address of the transaction is not stored
> in any cache in the system".

Flush does not mean evict.. The way I read the above it is trying to
say the driver must map all the memory non-cachable to ensure it never
gets pulled into a cache in the first place.

> > Maybe not entire, but as an additional step to reduce the cost of
> > this. ARM would like this for instance.
> 
> I searched PCI_EXP_DEVCTL_NOSNOOP_EN but surprisingly it's not
> touched by i915 driver. sort of suggesting that Intel GPU doesn't follow
> the spec to honor that bit...

Or the BIOS turns it on and the OS just leaves it..

> I'm fine to do a special check in the attach path to enable the flush
> only for Intel GPU.

We already effectively do this already by checking the domain
capabilities. Only the Intel GPU will have a non-coherent domain.

> or alternatively could ARM SMMU driver implement
> @enforce_cache_coherency by disabling PCI nosnoop cap when
> the SMMU itself cannot force snoop? Then VFIO/IOMMUFD could
> still check enforce_cache_coherency generally to apply the cache
> flush trick... ????

I like this a lot less than having vfio understand it..

Jason