Received-SPF: pass (google.com: domain of linux-kernel+bounces-185333-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Tue, 21 May 2024 15:37:45 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
	"Vetter, Daniel" <daniel.vetter@intel.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"pbonzini@redhat.com" <pbonzini@redhat.com>,
	"seanjc@google.com" <seanjc@google.com>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"luto@kernel.org" <luto@kernel.org>,
	"peterz@infradead.org" <peterz@infradead.org>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"bp@alien8.de" <bp@alien8.de>, "hpa@zytor.com" <hpa@zytor.com>,
	"corbet@lwn.net" <corbet@lwn.net>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
	"Liu, Yi L" <yi.l.liu@intel.com>
Subject: Re: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <20240521183745.GP20229@nvidia.com>
References: <20240510105728.76d97bbb.alex.williamson@redhat.com>
 <ZkG9IEQwi7HG3YBk@yzhao56-desk.sh.intel.com>
 <BN9PR11MB52766D78684F6206121590B98CED2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240516143159.0416d6c7.alex.williamson@redhat.com>
 <20240517171117.GB20229@nvidia.com>
 <BN9PR11MB5276250B2CF376D15D16FF928CE92@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240521160714.GJ20229@nvidia.com>
 <20240521102123.7baaf85a.alex.williamson@redhat.com>
 <20240521163400.GK20229@nvidia.com>
 <20240521121945.7f144230.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240521121945.7f144230.alex.williamson@redhat.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?roIl7ffFd8cs9Tu/lQn51Rb4tFwg0atu6wvpnGD0CdhdUhUIAzho4LsupAzd?=
 =?us-ascii?Q?3UHng216FlOIPkgkS4nl5I554E90FLIZzZ85cB1hyZ6JSK1VqIqhzpbHd0v0?=
 =?us-ascii?Q?KGY5aLcb9zuA7++wc1GtzSI5aJS0akPtAFL9ModIjr2oTHA22U4N0uennnpl?=
 =?us-ascii?Q?dFbkZHtsUMQ2Q0qY7uzrysZdhSa79FBQlDZnriEYm3sAg15LPmQdeaWy6fVW?=
 =?us-ascii?Q?GpeAQqu2CzPYz781C9Hg2+UQc+0m+sld2j0CVOllQLb9JL7C5wqNfoW2iWPX?=
 =?us-ascii?Q?gowLCbYrg6rzr5EPletpX00DqistEPtUsKlfNY9DYHwhCXi76Erm/R/0GTYS?=
 =?us-ascii?Q?vcMjAg0kQlEDdWKLxtW6z+Yx9LWUKVwqjFDLP4QOjkyZRXTavfRcW1uopgT+?=
 =?us-ascii?Q?iE+BHRnAt02NTlnw5W9CORFBTM33p3yrB8o+joGpYjmLNEC6tSlpoL1iQNMz?=
 =?us-ascii?Q?Ib3B2x4e2wxfxuaTUBAm01CLpOCmDeSW/fHbbRfDWK+vZzCZZiYkNKBgEcPz?=
 =?us-ascii?Q?NZ3kVtaGh0er6HjwbVrZjylMPfzBuCGhg7FxNx2HMMpqlHbkrDQCv8pFV41e?=
 =?us-ascii?Q?uwPxh5f2MKdgnXPKKqxA/cSknyFOa+4PIiMo+dMEofotqHTF3LnJ0358G1ty?=
 =?us-ascii?Q?rPe/3/R1bzFJFqgqQ2wmXhZgPPO9xAOzyAslE7WRQOSXz4YNFmyragggCykt?=
 =?us-ascii?Q?QesS3mry97Q8jEHNZ7ChIw9f4AqGbRBMCEIWxQNJMTUxlv/n6J6HD7zkic4k?=
 =?us-ascii?Q?ZC5CAX8TVb3x9jK7VDI0Yu1gBr8LHGmKeJSu48EdAWy0IdPFuxYd9UGbFOT4?=
 =?us-ascii?Q?4nrYBn2qgpI0yP36zWzZDFLF6D6NFOOE3gBZQyFi+ug9q7plqaTQ5yaBflrp?=
 =?us-ascii?Q?05iGViq0XSOO8MjnUwcwED2nta8k3TnIg4eoKX13SPqD52DOSzl8Nse28Ky6?=
 =?us-ascii?Q?cMb9mntgGqvBWmUjKsEqBaIToGTevuPSF2+M+reoAZ2Q+IOBBGs2rjdGwe9u?=
 =?us-ascii?Q?AzU0xmduvF8WWnh8QsqCxXehPrnU+p02Ah2i7hUH0uXabcintcRdneHXfxwp?=
 =?us-ascii?Q?Oio4kCVHsnR72D8heXkw0FNJ31iQw6xzPbWN1PjxAxmHwk3dDZtT7qP3oFdm?=
 =?us-ascii?Q?4r5zFRnkaxNAI6kdTWzFpEOIyjerccE0PAP+u4+qXSqZdc4XV2amyphzETPy?=
 =?us-ascii?Q?ay8kQjEAehMve75650r8RqzrFnm4Zj9BFqAAXHajyiKhEgiU0uejNXz+neM8?=
 =?us-ascii?Q?BcCrzv6XP+vBvMubAulchRcUrCVdK7R05woA6K421wD5tEBw6IDB8v3lWoNT?=
 =?us-ascii?Q?+UJlYd/vCJmXQoUVDdAiz85fQU0/7Xl/AWAAEeJyZHAtdAufWUbDY1e1en78?=
 =?us-ascii?Q?h/M6qifcfuXH3SkJCbUsMvinpwXtpfKqePXQlR9VcLOWk875znC1m5Vn5Xhi?=
 =?us-ascii?Q?zllQckKNqVSO+szn1y2b49OLOmdTLBKJfnJGYdEM1zYolRX1UHgQdhJsssXM?=
 =?us-ascii?Q?qZ/N1zjOwcdyYMcQHS29BInJS6/1V8lnM7x6KFNZSaxpo5vwqxrqjki2q27K?=
 =?us-ascii?Q?D0Xme9PUNrQ3UFqaEHU=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 110841c2-36f4-48e2-55f5-08dc79c51d1c
X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3843.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 May 2024 18:37:48.6565
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: flrFkPeC6Mf2nUHqefp2z9wXBcfdh+Q/ahv9IlOPGOyblvK+MZ7Q4RodOBT1Ks2i
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB9071

On Tue, May 21, 2024 at 12:19:45PM -0600, Alex Williamson wrote:
> > I'm OK with this. If devices are insecure then they need quirks in
> > vfio to disclose their problems, we shouldn't punish everyone who
> > followed the spec because of some bad actors.
> > 
> > But more broadly in a security engineered environment we can trust the
> > no-snoop bit to work properly.
> 
>  The spec has an interesting requirement on devices sending no-snoop
>  transactions anyway (regarding PCI_EXP_DEVCTL_NOSNOOP_EN):
> 
>  "Even when this bit is Set, a Function is only permitted to Set the No
>   Snoop attribute on a transaction when it can guarantee that the
>   address of the transaction is not stored in any cache in the system."
> 
> I wouldn't think the function itself has such visibility and it would
> leave the problem of reestablishing coherency to the driver, but am I
> overlooking something that implicitly makes this safe?  

I think it is just bad spec language! People are clearly using
no-snoop on cachable memory today. The authors must have had some
other usage in mind than what the industry actually did.

> But there's no capability bit that allows us to report whether the
> device supports no-snoop, we're just hoping that a driver writing to
> the bit doesn't generate a fault if the bit doesn't stick.  For example
> the no-snoop bit in the TLP itself may only be a bandwidth issue, but
> if the driver thinks no-snoop support is enabled it may request the
> device use the attribute for a specific transaction and the device
> could fault if it cannot comply.

It could, but that is another wierdo quirk IMHO. We already see things
in config space under hypervisor control because the VF don't have the
bits :\

> > > The more secure approach might be that we need to do these cache
> > > flushes for any IOMMU that doesn't maintain coherency, even for
> > > no-snoop transactions.  Thanks,  
> > 
> > Did you mean 'even for snoop transactions'?
> 
> I was referring to IOMMUs that maintain coherency regardless of
> no-snoop transactions, ie domain->enforce_cache_coherency (ex. snoop
> control/SNP on Intel), so I meant as typed, the IOMMU maintaining
> coherency even for no-snoop transactions.
> 
> That's essentially the case we expect and we don't need to virtualize
> no-snoop enable on the device.

It is the most robust case to be sure, and then we don't need
flushing.

My point was we could extend the cases where we don't need to flush if
we pay attention to, or virtualize, the PCI_EXP_DEVCTL_NOSNOOP_EN.

> > That is where this series is, it assumes a no-snoop transaction took
> > place even if that is impossible, because of config space, and then
> > does pessimistic flushes.
> 
> So are you proposing that we can trust devices to honor the
> PCI_EXP_DEVCTL_NOSNOOP_EN bit and virtualize it to be hardwired to zero
> on IOMMUs that do not enforce coherency as the entire solution?

Maybe not entire, but as an additional step to reduce the cost of
this. ARM would like this for instance.
 
> Or maybe we trap on setting the bit to make the flushing less
> pessimistic?

Also a good idea. The VMM could then decide on policy.

Jason