Received-SPF: pass (google.com: domain of linux-kernel+bounces-188750-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Fri, 24 May 2024 10:50:00 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
	"Vetter, Daniel" <daniel.vetter@intel.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"pbonzini@redhat.com" <pbonzini@redhat.com>,
	"seanjc@google.com" <seanjc@google.com>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"luto@kernel.org" <luto@kernel.org>,
	"peterz@infradead.org" <peterz@infradead.org>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"bp@alien8.de" <bp@alien8.de>, "hpa@zytor.com" <hpa@zytor.com>,
	"corbet@lwn.net" <corbet@lwn.net>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
	"Liu, Yi L" <yi.l.liu@intel.com>
Subject: Re: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <20240524135000.GY20229@nvidia.com>
References: <20240521163400.GK20229@nvidia.com>
 <20240521121945.7f144230.alex.williamson@redhat.com>
 <20240521183745.GP20229@nvidia.com>
 <BN9PR11MB52769E209C5B978C7094A5C08CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240522122939.GT20229@nvidia.com>
 <BN9PR11MB527604CDF2E7FA49176200028CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240522233213.GI20229@nvidia.com>
 <BN9PR11MB5276C2DD3F924ED2EB6AD3988CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240523145848.GN20229@nvidia.com>
 <20240523164753.32e714d5.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240523164753.32e714d5.alex.williamson@redhat.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?qyi20CXaTPund6bEb/XpGHrRnKjIizQDQhvknlmBZyctoEA8dZ0BGTT4CJVm?=
 =?us-ascii?Q?nM+RfU7ZDdzQxm6v1BR3aV6gZN/78k3pLmJttpYZrcAT1Px/lqnI3DcE88Q5?=
 =?us-ascii?Q?GWLgk2cWOJlVeiqEhkkrszDq80zZyWmJHfHgEczmlrOIRWd18RtZs242Qeic?=
 =?us-ascii?Q?jX9WUv4gNxUwcjjLz1zMJ/637pzL719sH9xKJrDJYpHYiNu6T8VknBjmomry?=
 =?us-ascii?Q?L9RRu0kVRNMxa4g9cHHw/ag5Ll9OeT524s3pNiFp1nk8l5KYRd1YM+NXVUUw?=
 =?us-ascii?Q?fC99kUKFLgqeTD8YYiQfCjOu2hhrrgp+vdB+56CmnC493xYQ85KUHhEQgSDV?=
 =?us-ascii?Q?NJVd8Isvm2B71jzUzGyZbf7igxb0WBjD0TmKG0Cv3fyrkeG/gIY/EsrSlFyI?=
 =?us-ascii?Q?/+ILKYM/aUfYgp16xZ6mEwXH4yEkXCwZRO4yfkJVyUglQBnr+wP7HAG8OpBz?=
 =?us-ascii?Q?UDoWC9d1ArxLqrt5s4kHhU8NsvDpWaVGHh5aL+VU7jbq02UVr9ApDRRTtbxz?=
 =?us-ascii?Q?CH7YMGDU8D/dxSrQp+hYvmwiZChrLFAbcN4QXWuS9Se0L5z9AZg01Pc3kHdU?=
 =?us-ascii?Q?bwEh+jySaDpQX13urnFt5lnleghW/ea3XPIxNHchTBC/xY2P9pWvy2uAj8DX?=
 =?us-ascii?Q?7vB2ajiZuxzxmCVCmGaGeRm6EpLiC00ElujiZxIbufZOFoaaVYB3KXTKgci2?=
 =?us-ascii?Q?MrWfbjFrCzy2k25qaJkBpph5tzBA2qLXgRVmnYSd/zbwaumM0nJeyq6UTlFm?=
 =?us-ascii?Q?2cWiAwOyc4eR+fAy/7VZM9G9b7BkXtRwug9/KLciNxS+5EidSl23WQmuGFdF?=
 =?us-ascii?Q?f0bNud/oJ2wxGdlMq9VcXbuQ1ta+LoOHTjfQo970R2/SulE/akZXGN8wQJvk?=
 =?us-ascii?Q?pzylNm+j/eVGOuM3MPPg/FDPSJ1+hnXpBkJokg4q0N5A6403i5nNa2O1UxBb?=
 =?us-ascii?Q?c4Btq50Lru4psANdjKxrtYr7eEyRiRMApMRhICEXdgJk66p0cf71KAsd3rXy?=
 =?us-ascii?Q?a1KpsxavPNumR5GaS0F9QOBawhM4AWmijv0uMc0T/Ymvc67n9HXwq79yr9TV?=
 =?us-ascii?Q?dSG9mY0OnRpny+AZ/cNLZkwLuanJ8zX2aVAdJJiPgpeN1blCzXkINJYr53mP?=
 =?us-ascii?Q?m1rlTBMuV9XWddqfLrrdfC/UeoLCJzUn9mMUrPAn0UDl/YZsVZC0W0Myjk40?=
 =?us-ascii?Q?iFeS7qT9AJd+JMWOwqYQCqu/mvRhF+mVGBkaEyhvidVnaN5wuDPQSwp6Hiq3?=
 =?us-ascii?Q?V7hSL7+Spq1wHIW21JFhV9sXwo/Jd9JyA050xDCKKcwpa2TR5LbBZOxgNOyO?=
 =?us-ascii?Q?YCG3hhZQBTGbDzux/Sa/LgK6KQom/mpYheTj87ZrlLJldo3nzDU3V2AbgRXV?=
 =?us-ascii?Q?NY+im8Iq4meTNwd4ASpOQ+HJuNEIEBVdXxNsLHmT7kef87Xi3mtK/GuioHM2?=
 =?us-ascii?Q?45YsBQmzTDU/vQ/bbVqR44PF37DE5JAtVeFuk2F0cQuUZaqOLYfnXdwzctgI?=
 =?us-ascii?Q?zKbl3dcP3SgVj01yGvtW2Dc9hcjBUgXta0kuyYE+ttZrgV1bItq4h0Yddq8/?=
 =?us-ascii?Q?Q15cf/Jx07elv9AddDqsUVqX42ivoD/jziJoe7Hi?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1199e791-5d96-4817-cc3f-08dc7bf86866
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 May 2024 13:50:01.4578
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: NYamzHYGlQdkhhfS717TDub7F4llImwOhhVMp6ENay7w6zJX9NcDl7p4hjKW/ij4
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8327

On Thu, May 23, 2024 at 04:47:53PM -0600, Alex Williamson wrote:

> > > > I am suggesting to do both checks:
> > > >  - If the iommu domain indicates it has force coherency then leave PCI
> > > >    no-snoop alone and no flush
> > > >  - If the PCI NOSNOOP bit is or can be 0 then no flush
> > > >  - Otherwise flush  
> > > 
> > > How to judge whether PCI NOSNOOP can be 0? If following PCI spec
> > > it can always be set to 0 but then we break the requirement for Intel
> > > GPU. If we explicitly exempt Intel GPU in 2nd check  then what'd be
> > > the value of doing that generic check?  
> > 
> > Non-PCI environments still have this problem, and the first check does
> > help them since we don't have PCI config space there.
> > 
> > PCI can supply more information (no snoop impossible) and variant
> > drivers can add in too (want no snoop)
> 
> I'm not sure I follow either.  Since i915 doesn't set or test no-snoop
> enable, I think we need to assume drivers expect the reset value, so a
> device that supports no-snoop expects to use it, ie. we can't trap on
> no-snoop enable being set, the device is more likely to just operate
> with reduced performance if we surreptitiously clear the bit.

I'm not sure I understand this paragraph?

> The current proposal is to enable flushing based only on the domain
> enforcement of coherency.  I think the augmentation is therefore that
> if the device is PCI and the no-snoop enable bit is zero after reset
> (indicating hardwired to zero), we also don't need to flush.

Yes, that is a good additional starting point.
 
> I'm not sure the polarity of the variant drive statement above is
> correct.  If the no-snoop enable bit is set after reset, we'd assume
> no-snoop is possible, so the variant driver would only need a way to
> indicate the device doesn't actually use no-snoop.  For that it might
> just virtualize the no-snoop enable setting to vfio-pci-core.  Thanks,

I wrote that with the idea that VFIO would always force no-snoop to
0. The variant driver could opt out of this. We could also do the
reverse and leave no-snoop alone and have something force it off.

The other issue to keep in mind is that if no-snoop is disabled when
we attach the domains we shouldn't allow the VMM to turn it on later.

Jason