Received-SPF: pass (google.com: domain of linux-kernel+bounces-175822-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Fri, 10 May 2024 10:29:28 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Yan Zhao <yan.y.zhao@intel.com>
Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org,
	alex.williamson@redhat.com, kevin.tian@intel.com,
	iommu@lists.linux.dev, pbonzini@redhat.com, seanjc@google.com,
	dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org,
	tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com,
	corbet@lwn.net, joro@8bytes.org, will@kernel.org,
	robin.murphy@arm.com, baolu.lu@linux.intel.com, yi.l.liu@intel.com
Subject: Re: [PATCH 5/5] iommufd: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <20240510132928.GS4650@nvidia.com>
References: <20240507061802.20184-1-yan.y.zhao@intel.com>
 <20240507062212.20535-1-yan.y.zhao@intel.com>
 <20240509141332.GP4650@nvidia.com>
 <Zj3UuHQe4XgdDmDs@yzhao56-desk.sh.intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Zj3UuHQe4XgdDmDs@yzhao56-desk.sh.intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?aonQCCqaBEMSadhJzwW5Anz3552NQq5JCOKwbvWAu3Iq3FG6T8cF9E4XmrQW?=
 =?us-ascii?Q?UYCKyz4imeENrWee1nfO4PyS+BFm5nKVblgPOF6fAoBAGOWKkFUF1DoNPvia?=
 =?us-ascii?Q?YpRm85rV4YeVcqNrHZBvwg9qTDDKmaDZdjtuO+PGEB8gUg7jJGh4JWxK5gg2?=
 =?us-ascii?Q?YRk/tmy+raojtZ8mWCYUL/OpXlVN4ydMwwZJsp/2XkZn9k1o2duZTA2LgUkN?=
 =?us-ascii?Q?rEEwe2i1c3TCjkGyBGW75CGAejC1ebXSfhPhJIx0taYDwFQod8YisSdNKrno?=
 =?us-ascii?Q?T7EX4lHnGU8M1CZwtAfbheCuPafiR0Zw0Ep4umDpmd68h+uy42ILVKd6tZ6k?=
 =?us-ascii?Q?Say/6O2dCc/EB2aCq0Fd0GXFqg9tTmRZRYFKXZEwDC4roEDC0zwWEToEYoAp?=
 =?us-ascii?Q?LqkTe1O9BwruIGMuxH8CZvjwQY+UOxVUxkaZ3cCEn1FkvvXxh7DEcdQKotzD?=
 =?us-ascii?Q?VVmz0nUONxWCTenwAcmVBzjUmvPhgLdCLGfYAJ7Wb/IVSwpbZ6+NwEjBeMfl?=
 =?us-ascii?Q?cjwpULS+UUUyEhxkGr7/DUApFNCSfaAeqmYANnkPnmZ68YOv8z4vOp01uPE2?=
 =?us-ascii?Q?QazyU1fSs7KDFn8R1VAxiYW/YqJN54w1We59LXQCF+E3IdAWRM081fPiqGcn?=
 =?us-ascii?Q?AdvFObHLPRjro7DFZHwgUk8Px+7ttljWApZH4SZLGqX3hCFNfmZwCTKbgtQZ?=
 =?us-ascii?Q?JMUHdARXFFbRgHT+ZNAMjtEBDJ+MVmeZbJIZ5rUeYKFts90+uvDxpuNatwyr?=
 =?us-ascii?Q?ZDX8COn9LCaUmr+IAuTxK5OWncxjW2HQKSuCq4492BY1hFbgzS0yEZT9dpti?=
 =?us-ascii?Q?wHXz4Ve0mDowjpSzMYme18OrfO3u3N314FijNlbIiLenpgG+biM7d7qkDGkC?=
 =?us-ascii?Q?GPx3C0Om3edO9Jt9dHPMd2UsI4yUbnLnILS1VPoaIkU8X9QQasASF9Re1DNU?=
 =?us-ascii?Q?VG2+VfGBlF5djRAdA/iRIrEvH3kaJqRkA49UK3/hfM61xRqCN9acZurBGpxN?=
 =?us-ascii?Q?vMhjA/bBphLdoS1M5B6Rtr4vaxj7GsOA/JXtZ7iwnRB2+MLpisJ8w/hOCk63?=
 =?us-ascii?Q?0jxG9e8L3jemYNseaLjDb1kuT7mbUgp3FpmYtPyayr43+3g3GOjE3nEa1Tzv?=
 =?us-ascii?Q?oX5o9sVM4MAicOjtCvVtUpM05YTdqPgur1xGdOqnn9f07t/PoGY9su3sMVV+?=
 =?us-ascii?Q?24HZYZpqBYh/u47X3/ytPpENOlKgGWPVVnondGgbmBD/hFXn3P7su8mVuptL?=
 =?us-ascii?Q?Uo+y3JMjzIooge9VVw7cNrqbZ3tLydnVUVgofCNOAw+Xnd49a0q9hVfCfc0F?=
 =?us-ascii?Q?d8eGIY6+Cz5dtYjBVcf1eR7DnsoMaSBEDO/+uXFm5CgMs6E11Hr7yLRlDMNO?=
 =?us-ascii?Q?VHbz5yQS67eyS89OGHt4+elUxoQZzvCrfHEnAgJHPqtbHyt2GHUPxpJyeCIR?=
 =?us-ascii?Q?0MCIswIUKCJx1IZBxnlmAO9v+0eRQpm+ygxDQoz6Mbwd8vipVdiZGP6qtvXl?=
 =?us-ascii?Q?pU0h+Dn23aqX37/4wCSQ8fMjjzmUc9VO6SfvpFWKcW04xDDsLGP3HyYfzKIA?=
 =?us-ascii?Q?PCzaBfBT7AgzFqj1+5guhk13Eks026+lgB8EGfnU?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6e4d62f3-7fce-41fb-9837-08dc70f53853
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2024 13:29:29.5476
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HJjTQuoVMEBFCEcNqhcQJ7CsKY5H722z1IMlbsKOTLAeqsUFXJAPXBdQhD/eYN/h
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8287

On Fri, May 10, 2024 at 04:03:04PM +0800, Yan Zhao wrote:
> > > @@ -1358,10 +1377,17 @@ int iopt_area_fill_domain(struct iopt_area *area, struct iommu_domain *domain)
> > >  {
> > >  	unsigned long done_end_index;
> > >  	struct pfn_reader pfns;
> > > +	bool cache_flush_required;
> > >  	int rc;
> > >  
> > >  	lockdep_assert_held(&area->pages->mutex);
> > >  
> > > +	cache_flush_required = area->iopt->noncoherent_domain_cnt &&
> > > +			       !area->pages->cache_flush_required;
> > > +
> > > +	if (cache_flush_required)
> > > +		area->pages->cache_flush_required = true;
> > > +
> > >  	rc = pfn_reader_first(&pfns, area->pages, iopt_area_index(area),
> > >  			      iopt_area_last_index(area));
> > >  	if (rc)
> > > @@ -1369,6 +1395,9 @@ int iopt_area_fill_domain(struct iopt_area *area, struct iommu_domain *domain)
> > >  
> > >  	while (!pfn_reader_done(&pfns)) {
> > >  		done_end_index = pfns.batch_start_index;
> > > +		if (cache_flush_required)
> > > +			iopt_cache_flush_pfn_batch(&pfns.batch);
> > > +
> > 
> > This is a bit unfortunate, it means we are going to flush for every
> > domain, even though it is not required. I don't see any easy way out
> > of that :(
> Yes. Do you think it's possible to add an op get_cache_coherency_enforced
> to iommu_domain_ops?

Do we need that? The hwpt already keeps track of that? the enforced could be
copied into the area along side storage_domain

Then I guess you could avoid flushing in the case the page came from
the storage_domain...

You'd want the storage_domain to preferentially point to any
non-enforced domain.

Is it worth it? How slow is this stuff?

Jason