Received-SPF: pass (google.com: domain of linux-kernel+bounces-186825-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Wed, 22 May 2024 20:32:13 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: Alex Williamson <alex.williamson@redhat.com>,
	"Vetter, Daniel" <daniel.vetter@intel.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"pbonzini@redhat.com" <pbonzini@redhat.com>,
	"seanjc@google.com" <seanjc@google.com>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"luto@kernel.org" <luto@kernel.org>,
	"peterz@infradead.org" <peterz@infradead.org>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"bp@alien8.de" <bp@alien8.de>, "hpa@zytor.com" <hpa@zytor.com>,
	"corbet@lwn.net" <corbet@lwn.net>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
	"Liu, Yi L" <yi.l.liu@intel.com>
Subject: Re: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <20240522233213.GI20229@nvidia.com>
References: <20240517171117.GB20229@nvidia.com>
 <BN9PR11MB5276250B2CF376D15D16FF928CE92@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240521160714.GJ20229@nvidia.com>
 <20240521102123.7baaf85a.alex.williamson@redhat.com>
 <20240521163400.GK20229@nvidia.com>
 <20240521121945.7f144230.alex.williamson@redhat.com>
 <20240521183745.GP20229@nvidia.com>
 <BN9PR11MB52769E209C5B978C7094A5C08CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240522122939.GT20229@nvidia.com>
 <BN9PR11MB527604CDF2E7FA49176200028CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB527604CDF2E7FA49176200028CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?9NLLO0auv1IdxIgPKR04BiKHsaeTweFHL98baYz4xptOEifXBaur+lXj4FKy?=
 =?us-ascii?Q?OpPrhNPpKYfjAYJUhWGD8X0K/3iDb/lcl5yLWELTXGQNL/Wc//kJguo6fdIt?=
 =?us-ascii?Q?q+vNqgCRIKpkBDVFUSwvfkE4xAT5RcMIrtOIowNi8VA1v481jgWl8LgEXi9P?=
 =?us-ascii?Q?7zDxeQBuh6u47Et1S/jID6Nuw1Vld20yhDod6sVLwVJvdionmDkpAhm3Ivyw?=
 =?us-ascii?Q?qbORjbirTj4dJ7+XI3Fe1Kw8v05geRjK4GxRNPtl7CK9wSf76QZ1WwWdVMwb?=
 =?us-ascii?Q?HjdZlBfoTsazwmyfy+90CHjjXOFKfayOu+UFhxEdbIfIcC9dUd3j69ht/tgn?=
 =?us-ascii?Q?vfV7tuEb7GgCs1NXNZ/+tOuZbqBlrd1gQxjkZRChiyzF4fzZeVSMM3mU4Cxf?=
 =?us-ascii?Q?ZnVQQ+KX57fDYQBe43xpyZ1AGtEee+DZwR0t5k9/x1jDSsgARLHQIMYhWFno?=
 =?us-ascii?Q?2gzNyg1v+QJCER0vNnhaiG7wrLvM6V1l79HMtqyBpSbmhzl1Ao8e7y4F/EjO?=
 =?us-ascii?Q?nBLsZv6JzKbl42PxsVISfb23Ol5L/8zyTAoeKIkCbup20CHrnsRk5o8JlxpL?=
 =?us-ascii?Q?pxpz7dzp1Qw40vmzO9XVV66tk01ZG63OsbKIH/S/4LwiUpMW+egvGvOm5JPK?=
 =?us-ascii?Q?4LugW8+ew1kU19pczq9baci/KrUQw6A1v9WORREuF58JYnw/fEiQRf7KPyPx?=
 =?us-ascii?Q?ivjSrb4c1SIw5BbzSWfQu4zDF6cuR7KI4IKAzLjKwDpT0Rkv39VS/O7qr/CD?=
 =?us-ascii?Q?VYFTfHNt+Z+fFJuDC26hJPzQDatAc18++9uWmFOm6G+T+LCKpzz1XjWZ5dco?=
 =?us-ascii?Q?vGM9GxL5gHVXUxHWKlAsAuukPzkbYIPZcgCS4cm+uq9OdJyS98v9El/ZJznC?=
 =?us-ascii?Q?6B28ZA2sm3QmSdcafx9IWq85a6KdlkF+luhcfePivU0K0R2Cynra1ecTPU+E?=
 =?us-ascii?Q?XAh0k01KfmtgfDI5MBKoZkuEfNkOk0C+/m7CYJ3r4PIpQedAT33mX81OXug1?=
 =?us-ascii?Q?UW/SM2ClnqwudzJH24SqUKAkjwlFalPzWOeXGT6+Fn/HiVUeLIhsppHuL6TJ?=
 =?us-ascii?Q?cxrYDwDlvIvuxJABeJfzcXp+/lEwQFhzzeOQGx7mgRZ2PMePcGI4uuwpnFfX?=
 =?us-ascii?Q?IC1Ri1m2ut1uHFfN3YWKrhBkhcNqV75tq78r+DjLE7E6vl8hEF5Iru91BlYu?=
 =?us-ascii?Q?oRivV/bZopi4wJpmByF9bIQPQYt4eWAlZ0JCyeV7x8VJhjJwnhwnK0PL0Cer?=
 =?us-ascii?Q?6o8w9s0vCRc/gKCtDsEwRk/tDVkbzLgaR2GP/NArFbGgi1vA241gsz1Vhfx7?=
 =?us-ascii?Q?O/fkD6ChoO5Jukt4xxNHwA/NGJHmUHKfI9lwUTb/6VwbCSNku9byBHlV+lzH?=
 =?us-ascii?Q?qcOGNJnzUnTrAalFgvUr4LuifPwjsJKYGwMdnZeGrNJczdrllvM61E1krTMC?=
 =?us-ascii?Q?LlMUhZmXFNWN9bQfCRJss5qzrdAUQEmxKi+4oXfDzE370k22EHri0KSoGtXn?=
 =?us-ascii?Q?eYm1/kHeUc9mg8KtCGkzCo8xANOa/dX4D/Dhg4u/Mm3fkO9nwguOtE8EqC8b?=
 =?us-ascii?Q?ohbheHw+uj5/mOTNnBU=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 89fe2bb1-c77c-4a51-daae-08dc7ab76a23
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 May 2024 23:32:15.9180
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hFVAwKfKNZ4EbV4lw3Y82noROQm+whAl5DxX+X32Edyvtd3rcChTBLa3qZUN8C2r
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5593

On Wed, May 22, 2024 at 11:26:21PM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, May 22, 2024 8:30 PM
> > 
> > On Wed, May 22, 2024 at 06:24:14AM +0000, Tian, Kevin wrote:
> > > I'm fine to do a special check in the attach path to enable the flush
> > > only for Intel GPU.
> > 
> > We already effectively do this already by checking the domain
> > capabilities. Only the Intel GPU will have a non-coherent domain.
> > 
> 
> I'm confused. In earlier discussions you wanted to find a way to not
> publish others due to the check of non-coherent domain, e.g. some
> ARM SMMU cannot force snoop.
> 
> Then you and Alex discussed the possibility of reducing pessimistic
> flushes by virtualizing the PCI NOSNOOP bit.
> 
> With that in mind I was thinking whether we explicitly enable this
> flush only for Intel GPU instead of checking non-coherent domain
> in the attach path, since it's the only device with such requirement.

I am suggesting to do both checks:
 - If the iommu domain indicates it has force coherency then leave PCI
   no-snoop alone and no flush
 - If the PCI NOSNOOP bit is or can be 0 then no flush
 - Otherwise flush

I'm not sure there is a good reason to ignore the data we get from the
iommu domain that it enforces coherency?

Jason