Received-SPF: pass (google.com: domain of linux-kernel+bounces-188181-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Alex Williamson <alex.williamson@redhat.com>, Jason Gunthorpe
	<jgg@nvidia.com>
CC: "Vetter, Daniel" <daniel.vetter@intel.com>, "Zhao, Yan Y"
	<yan.y.zhao@intel.com>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>, "iommu@lists.linux.dev"
	<iommu@lists.linux.dev>, "pbonzini@redhat.com" <pbonzini@redhat.com>,
	"seanjc@google.com" <seanjc@google.com>, "dave.hansen@linux.intel.com"
	<dave.hansen@linux.intel.com>, "luto@kernel.org" <luto@kernel.org>,
	"peterz@infradead.org" <peterz@infradead.org>, "tglx@linutronix.de"
	<tglx@linutronix.de>, "mingo@redhat.com" <mingo@redhat.com>, "bp@alien8.de"
	<bp@alien8.de>, "hpa@zytor.com" <hpa@zytor.com>, "corbet@lwn.net"
	<corbet@lwn.net>, "joro@8bytes.org" <joro@8bytes.org>, "will@kernel.org"
	<will@kernel.org>, "robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>, "Liu, Yi L"
	<yi.l.liu@intel.com>
Subject: RE: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in
 non-coherent domains
Thread-Topic: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in
 non-coherent domains
Thread-Index: AQHaoEb1yTw+2+R5T02uRFeJZuQMNbGPN+OAgAER64CAAGvrAIAEE0QAgATCszCAANP1gIABWkKAgAPDRsCAAnQoAIAAA/SAgAADhwCAAB2LgIAABQiAgAC9HuCAAG5egIAAtd9QgAADQICAAADyYIABAfAAgACDD4CAABpGIA==
Date: Fri, 24 May 2024 00:30:20 +0000
Message-ID: <BN9PR11MB52768B71B41ADD916ED51D348CF52@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20240521160714.GJ20229@nvidia.com>
	<20240521102123.7baaf85a.alex.williamson@redhat.com>
	<20240521163400.GK20229@nvidia.com>
	<20240521121945.7f144230.alex.williamson@redhat.com>
	<20240521183745.GP20229@nvidia.com>
	<BN9PR11MB52769E209C5B978C7094A5C08CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
	<20240522122939.GT20229@nvidia.com>
	<BN9PR11MB527604CDF2E7FA49176200028CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
	<20240522233213.GI20229@nvidia.com>
	<BN9PR11MB5276C2DD3F924ED2EB6AD3988CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
	<20240523145848.GN20229@nvidia.com>
 <20240523164753.32e714d5.alex.williamson@redhat.com>
In-Reply-To: <20240523164753.32e714d5.alex.williamson@redhat.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?lDxDEo5Sy3mP/dGgYq1YT/TxqjqiABoLNB4z0zjjVY0XDJm1iRW27J0gF5k/?=
 =?us-ascii?Q?nKMywwSfZbRHD/cls490IjxCNpXo7MhsLCYQyiixRs32Q8a1oywspk0ysDL/?=
 =?us-ascii?Q?D7sX/7xVEf9pZsWcIQzvUgHKIGI2VVOuv/pdGwK21hjr4H9VK+V0+kAkT1Fl?=
 =?us-ascii?Q?DLwBfOMA1GKG2a+IYRaxl1lhH4Sdurw8EvzRCkkPCdYTBN50X0hyZtkTPHpX?=
 =?us-ascii?Q?KUAOBYCMOcYiV0yCbGYbbS92EBTuU9woc83h9qwyXKRZVZ7FcIX348kOA76j?=
 =?us-ascii?Q?KWvkT1UjDb6BF14Wv/h1va5EDGw7EM9VXXbimQrkQYijNSgeUF18VxkdDuSa?=
 =?us-ascii?Q?bDCGfN3Fq4BFgWj3np+sH5XxSdCIDYDWp/6wEqpFzhCkXJnSytvXXRtM2xuB?=
 =?us-ascii?Q?m/+m5ZF+Ky4lGT2S9D3EEojPm0SM2MTV1ye3ffjmN54r/ys5fpsmmDY+UEw2?=
 =?us-ascii?Q?/zwdVzUUnzyr31bg9BPy0g6m48O22fSfkFIomcyBel12Ll1I8y4uDBrL9vl9?=
 =?us-ascii?Q?oONw/+dPuqKlg/e8RM/MvdfbpEDRy3vhAKjw/31jKuEvesgmxTJXre0PvgFH?=
 =?us-ascii?Q?T/P+RWTRuAQxnudpQIEUrd1m+6B3UbA/4mmJImvGyDJqAtyBHFbFT82wvnks?=
 =?us-ascii?Q?/E+o6OspvqPbdeH1a07cNI/cd1Hmjex0UuHSYQqOVO4sFjma9R7lTdXhgwoS?=
 =?us-ascii?Q?nG1u6bbeyZdW+9LXBMejstWoKz/mr9gpYb69P2e/bVUq6WEV9io0CARWGVNZ?=
 =?us-ascii?Q?sXVlxMV5ATV786bmqRZ6MiVH5ROihQWkCPFbFf+eed7fy/RmCHPyjmZLSLof?=
 =?us-ascii?Q?ZElutU5FPAc+Iz1E8APpaav4eNLePkPHUH2kaT+TDrEn3w5wR67JSt8obHNu?=
 =?us-ascii?Q?s2w9x+oX7ePe5tVUe6nckTU5cN+oNDIyctEPwGqiO28N9/P12ZXtkQSITIlw?=
 =?us-ascii?Q?nJzMZ7ETLlhrQPZP/u+472Ur7+ijbLDjPcAV+yqhsXiI/7+5k5cBXLofxzC3?=
 =?us-ascii?Q?+FpXD9mBTxJJK5rFBQ4V8A7wiGAlMLlaRrdDgh9HlljXrT9SHipDEYpO6AzN?=
 =?us-ascii?Q?/8Wu0eHxSmoNneeh380APLeKWHPTOMoyk+H8KaDzZ9N01HiLGa9iB2GMjQbr?=
 =?us-ascii?Q?5i00AhJN8iU3I/A2nI5n8EMX6XFGwz6W1Rw2Zu2e4Y+pghiUHjq9hm3TnobE?=
 =?us-ascii?Q?i2hitWZTPl2ysI9MRbhvxzwT67aueVcMHREV7CDpHr7v7opa7VYu5cLbSdMV?=
 =?us-ascii?Q?E/+TY2+4VQBiUiC3bSroVE9GojNBtY6xndYCsFh4MzEQ5VBQOvYUbiDP/bLK?=
 =?us-ascii?Q?ha25BPDkdvF1//PqR1CYXiDpqwvQ3y5lthHY/R41gUEv5aPYFziDQqjrAd1+?=
 =?us-ascii?Q?7BZf2mrSmZdyx7Eg3uyxcp2Md0f7Ljr8TNA9D6vocqGvf2r6F79yylWqhymS?=
 =?us-ascii?Q?pWohcrPwZXVmI6BVBls6JPqLAtH+3eXNHqylw6cptlu17wKiJGoRI7LCaBas?=
 =?us-ascii?Q?AwR4EOZ0cdW+p1oHRjTYn/mkmKfLCAjlWRh6TRH7h7JIkCneB5sM1QiV0PDy?=
 =?us-ascii?Q?vXImAca4KPPe70jJZkrFTuCoc/mNcz0b//GMNcu7?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7dd25baf-f937-49ad-9e90-08dc7b88b176
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 May 2024 00:30:20.2556
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: BWORcorudxIWrQx7JYNn4GZcTlus6NaZHhs193SY9YJAEg+gKUtO8iv5HTkzLdT8gxeSKzLT9LjX09hfW1/dmw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB8229
X-OriginatorOrg: intel.com

> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Friday, May 24, 2024 6:48 AM
>=20
> On Thu, 23 May 2024 11:58:48 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
>=20
> > On Wed, May 22, 2024 at 11:40:58PM +0000, Tian, Kevin wrote:
> > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > Sent: Thursday, May 23, 2024 7:32 AM
> > > >
> > > > On Wed, May 22, 2024 at 11:26:21PM +0000, Tian, Kevin wrote:
> > > > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > > > Sent: Wednesday, May 22, 2024 8:30 PM
> > > > > >
> > > > > > On Wed, May 22, 2024 at 06:24:14AM +0000, Tian, Kevin wrote:
> > > > > > > I'm fine to do a special check in the attach path to enable t=
he flush
> > > > > > > only for Intel GPU.
> > > > > >
> > > > > > We already effectively do this already by checking the domain
> > > > > > capabilities. Only the Intel GPU will have a non-coherent domai=
n.
> > > > > >
> > > > >
> > > > > I'm confused. In earlier discussions you wanted to find a way to =
not
> > > > > publish others due to the check of non-coherent domain, e.g. some
> > > > > ARM SMMU cannot force snoop.
> > > > >
> > > > > Then you and Alex discussed the possibility of reducing pessimist=
ic
> > > > > flushes by virtualizing the PCI NOSNOOP bit.
> > > > >
> > > > > With that in mind I was thinking whether we explicitly enable thi=
s
> > > > > flush only for Intel GPU instead of checking non-coherent domain
> > > > > in the attach path, since it's the only device with such requirem=
ent.
> > > >
> > > > I am suggesting to do both checks:
> > > >  - If the iommu domain indicates it has force coherency then leave =
PCI
> > > >    no-snoop alone and no flush
> > > >  - If the PCI NOSNOOP bit is or can be 0 then no flush
> > > >  - Otherwise flush
> > >
> > > How to judge whether PCI NOSNOOP can be 0? If following PCI spec
> > > it can always be set to 0 but then we break the requirement for Intel
> > > GPU. If we explicitly exempt Intel GPU in 2nd check  then what'd be
> > > the value of doing that generic check?
> >
> > Non-PCI environments still have this problem, and the first check does
> > help them since we don't have PCI config space there.
> >
> > PCI can supply more information (no snoop impossible) and variant
> > drivers can add in too (want no snoop)
>=20
> I'm not sure I follow either.  Since i915 doesn't set or test no-snoop
> enable, I think we need to assume drivers expect the reset value, so a
> device that supports no-snoop expects to use it, ie. we can't trap on
> no-snoop enable being set, the device is more likely to just operate
> with reduced performance if we surreptitiously clear the bit.
>=20
> The current proposal is to enable flushing based only on the domain
> enforcement of coherency.  I think the augmentation is therefore that
> if the device is PCI and the no-snoop enable bit is zero after reset
> (indicating hardwired to zero), we also don't need to flush.
>=20
> I'm not sure the polarity of the variant drive statement above is
> correct.  If the no-snoop enable bit is set after reset, we'd assume
> no-snoop is possible, so the variant driver would only need a way to
> indicate the device doesn't actually use no-snoop.  For that it might
> just virtualize the no-snoop enable setting to vfio-pci-core.  Thanks,
>=20

Yeah. I re-checked the use of PCI_EXP_DEVCTL_NOSNOOP_EN and
actually all references are about clearing the bit, echo'ing the point
that if a driver wants to use nosnoop it expects the reset value w/o
doing an explicit set and the virtualization of the no-snoop enable
bit is more reasonable to catch the intention of 'clear'.