Received-SPF: pass (google.com: domain of linux-kernel+bounces-187678-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Thu, 23 May 2024 11:58:48 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: Alex Williamson <alex.williamson@redhat.com>,
	"Vetter, Daniel" <daniel.vetter@intel.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"pbonzini@redhat.com" <pbonzini@redhat.com>,
	"seanjc@google.com" <seanjc@google.com>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"luto@kernel.org" <luto@kernel.org>,
	"peterz@infradead.org" <peterz@infradead.org>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"bp@alien8.de" <bp@alien8.de>, "hpa@zytor.com" <hpa@zytor.com>,
	"corbet@lwn.net" <corbet@lwn.net>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
	"Liu, Yi L" <yi.l.liu@intel.com>
Subject: Re: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <20240523145848.GN20229@nvidia.com>
References: <20240521160714.GJ20229@nvidia.com>
 <20240521102123.7baaf85a.alex.williamson@redhat.com>
 <20240521163400.GK20229@nvidia.com>
 <20240521121945.7f144230.alex.williamson@redhat.com>
 <20240521183745.GP20229@nvidia.com>
 <BN9PR11MB52769E209C5B978C7094A5C08CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240522122939.GT20229@nvidia.com>
 <BN9PR11MB527604CDF2E7FA49176200028CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240522233213.GI20229@nvidia.com>
 <BN9PR11MB5276C2DD3F924ED2EB6AD3988CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5276C2DD3F924ED2EB6AD3988CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?fOmzRvcmRcddxcmxr3Gqh5P0PIXCWVZzFnS7xHS3jwePW99YOqn68v+qQcTJ?=
 =?us-ascii?Q?gvqzsjn64mkGAq1a2LU+8GYLvu7i20mi5Nf5yifRfvpSXMpGFCraWE5Pb8Yn?=
 =?us-ascii?Q?yDsL8hmUMzglr5JJ82dwaJWXlakTE0IGHziRKAJ1hkjfes/XWTO75++kzfr3?=
 =?us-ascii?Q?HNSMr/hPsA7h98Q9PSXA820T897YgTUM9n5uRzP+MzbahvoV46GWNChfpnYn?=
 =?us-ascii?Q?Lb7YO+nksSfU0JTuhjlkGslxKcCnFYyecX6MZtNKdor6gdo4535+WCo7fvBS?=
 =?us-ascii?Q?J0jBh78igdZlpcabehxct4aIlRlGkPsneFtpVD1O+EOp+liccd+ShRH1TbuC?=
 =?us-ascii?Q?6p4tIM3gTryenkCOZvtknQNhC1MVymnXKRytkKeS0clnVgRU9K1NsYqGoaRo?=
 =?us-ascii?Q?GmCo2zgolNCX6pnW4KTfa2GE0eSK2YQ5z9hQVYa+TehmzLUm/A1Bn9LPxiDf?=
 =?us-ascii?Q?YR2Ed0c9OVPPLlbG5JyZ3qVqfDsxlgCHl5GmEzXNMO6UPF5CiRu3mYRarELL?=
 =?us-ascii?Q?PE3xwi2oO5khz7C3uDsvC+hjanm+X6HxTagsQTOQEf4zUk0TXMkCfoKq6DiJ?=
 =?us-ascii?Q?aDVmL8p5hdFiHjXqGKBiq70YJ75BXzrYCQgp+eYEnaXa+qCJ70AQN6ktBLPh?=
 =?us-ascii?Q?K9VHD4YFu9OZKrnp/PMMCE0240geD6T8AlsuirHc5N016ZCdmVdryt8B9Hzw?=
 =?us-ascii?Q?Bx/TEzVHWLpAeAVz5EGkqEy06qBeXBZKs+eurq0FH22Khuke1nHuDfhpAKbx?=
 =?us-ascii?Q?iAm4cAf7MzqdMiF+kWPbdf6ehtIMnWo+5ug62b1GC0FRJKsBCeSAPih4FZBU?=
 =?us-ascii?Q?U1onVRwNSFtnjqnIRMfTiXRWZ+GH9KrJYwyTjd9gismKCLJAWsgD1k+xeLoE?=
 =?us-ascii?Q?qKMTWpiGCmwZi45DSYi1G/9DKvnAIUONA+lNjVVU6SwvM8rI5Ck1XwElLUiu?=
 =?us-ascii?Q?mHAUWVqsH0mqwzjbke1WIBNBNqCVxP9JSlLRrdAuiRlCn0muVLUTnZKypuJL?=
 =?us-ascii?Q?UL8UlOvnBl4yVL4VuNke0tbaaQdcaHqKJr/yqexqw7Eaxsu6ZxJ/gegMZZgz?=
 =?us-ascii?Q?vllhG3FxKNC0HN/QofBwdGjZ1OIYbtP2NX6/FXKeyZw+A3vC80z50facoccc?=
 =?us-ascii?Q?bOGwpuOrbVpHwibPSGvT1Gd7kiA3GRh4OkDsVF3Sm8HQ/eRA2NTlAzU0/xC7?=
 =?us-ascii?Q?yDSuEVIG3XOuGgpMxxL+9OFZx/+ivUcgOGFh5POzAw9H5X49UT1SgqVLyz8U?=
 =?us-ascii?Q?0OrkA9nj5rwLqYmJ0KWVZTpv1/BkctyKNyQrJsO0fdQ35HTRv/faqJup4iEM?=
 =?us-ascii?Q?MejJoHAejeMNlLhhaXsUkbgFBKb33k6ZVlTxkH+4FBGlBUl6L6guAqPOb6N8?=
 =?us-ascii?Q?JdDIbm5xNz8KHys0V7KlqIsx2IscX1jzdDj/htveQsdKWRVPPQHOb5icape+?=
 =?us-ascii?Q?x8EhDjFCAdlvb6fmRk4zp/p5EqklVSZL0VHDbnZOiUkZdLoFnGNPzBeFodcI?=
 =?us-ascii?Q?pj7MaEbELHuqvtGBIrtVuQQnBkD/xABchpM9g3elQHt/XWKawtfMmZGxjQn2?=
 =?us-ascii?Q?BtnA+alYtByrd6PgISU=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bd185a79-42d0-44f8-31e8-08dc7b38db11
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 May 2024 14:58:50.4825
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: naiJYk7qZ0wOff2k2ot138dQuHyCDf1Pfj/8R06hpXQlvDZSZD+c1bRIvoxnSMcT
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8073

On Wed, May 22, 2024 at 11:40:58PM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Thursday, May 23, 2024 7:32 AM
> > 
> > On Wed, May 22, 2024 at 11:26:21PM +0000, Tian, Kevin wrote:
> > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > Sent: Wednesday, May 22, 2024 8:30 PM
> > > >
> > > > On Wed, May 22, 2024 at 06:24:14AM +0000, Tian, Kevin wrote:
> > > > > I'm fine to do a special check in the attach path to enable the flush
> > > > > only for Intel GPU.
> > > >
> > > > We already effectively do this already by checking the domain
> > > > capabilities. Only the Intel GPU will have a non-coherent domain.
> > > >
> > >
> > > I'm confused. In earlier discussions you wanted to find a way to not
> > > publish others due to the check of non-coherent domain, e.g. some
> > > ARM SMMU cannot force snoop.
> > >
> > > Then you and Alex discussed the possibility of reducing pessimistic
> > > flushes by virtualizing the PCI NOSNOOP bit.
> > >
> > > With that in mind I was thinking whether we explicitly enable this
> > > flush only for Intel GPU instead of checking non-coherent domain
> > > in the attach path, since it's the only device with such requirement.
> > 
> > I am suggesting to do both checks:
> >  - If the iommu domain indicates it has force coherency then leave PCI
> >    no-snoop alone and no flush
> >  - If the PCI NOSNOOP bit is or can be 0 then no flush
> >  - Otherwise flush
> 
> How to judge whether PCI NOSNOOP can be 0? If following PCI spec
> it can always be set to 0 but then we break the requirement for Intel
> GPU. If we explicitly exempt Intel GPU in 2nd check  then what'd be
> the value of doing that generic check?

Non-PCI environments still have this problem, and the first check does
help them since we don't have PCI config space there.

PCI can supply more information (no snoop impossible) and variant
drivers can add in too (want no snoop)

Jason