Received-SPF: pass (google.com: domain of linux-kernel+bounces-186530-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Wed, 22 May 2024 14:01:50 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Yan Zhao <yan.y.zhao@intel.com>
Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org,
	alex.williamson@redhat.com, kevin.tian@intel.com,
	iommu@lists.linux.dev, pbonzini@redhat.com, seanjc@google.com,
	dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org,
	tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com,
	corbet@lwn.net, joro@8bytes.org, will@kernel.org,
	robin.murphy@arm.com, baolu.lu@linux.intel.com, yi.l.liu@intel.com
Subject: Re: [PATCH 5/5] iommufd: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <20240522170150.GE20229@nvidia.com>
References: <ZkHEsfaGAXuOFMkq@yzhao56-desk.sh.intel.com>
 <ZkN/F3dGKfGSdf/6@nvidia.com>
 <ZkRe/HeAIgscsYZw@yzhao56-desk.sh.intel.com>
 <ZkUeWAjHuvIhLcFH@nvidia.com>
 <ZkVwS8n7ARzKAbyW@yzhao56-desk.sh.intel.com>
 <20240517170418.GA20229@nvidia.com>
 <Zkq5ZL+saJbEkfBQ@yzhao56-desk.sh.intel.com>
 <20240521160442.GI20229@nvidia.com>
 <Zk1jrI8bOR5vYKlc@yzhao56-desk.sh.intel.com>
 <Zk2Qv4pnSKZBsLYv@yzhao56-desk.sh.intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Zk2Qv4pnSKZBsLYv@yzhao56-desk.sh.intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?qkR0YwCDpwDhW90Di47FOceHGoU/5X7jwlOaXDIG8RYReXt+80VPNr5f7z36?=
 =?us-ascii?Q?oKWqZOghactjfIHh4jfsCHXvph7ac3v8Cq4Ax/dC7EcnpDdtRkFUfU29LK9T?=
 =?us-ascii?Q?fQCrwHqXV4ueMqVGVJ5hwzGbZpoWBDIEc5T40pI0Q15HgjUZBohKVXSvS7e7?=
 =?us-ascii?Q?z78AfZWWPHB8Xnj9MlgQ44QuA8vYyX12Apun+rXa1Pqtdssq2eVT78bv3Th2?=
 =?us-ascii?Q?NrZh+1qgjdec3OQHj9Kl10SQOzLv0LaNK3U+qY8U0Isl3IKnteylVgef7oHY?=
 =?us-ascii?Q?tDgqUolswZNVixo9F50tXyEABt/sWvPdhw4CtW9pUP67xDgxVPgYqyg5iE8l?=
 =?us-ascii?Q?6zo10L0qB3UlxaimKfBeW/fHQX70pb55vJtD/dw8JxChHprMqwetZpoteG0m?=
 =?us-ascii?Q?BLo8S9PKTmQ02q2hOAL1fjF/0keu5emyTA0ThIiwvWkyonb66ORFPEhNRcw5?=
 =?us-ascii?Q?MuhxqDE/YPU3PMSrGqCgw/eXTbqXUHlOpQMc1QhpAckFacCAXG7YbQnWEN5y?=
 =?us-ascii?Q?LSVbPKNbi1U7YoGheDKotwEset7mkWcvht8uMQSJzTBV8TDv3vwXGR/WHJhS?=
 =?us-ascii?Q?G8MsZ+01Dx0gpBna7rjrIilDf6+DdNQfLPWzCyHr+BbILQK1Gozb3NuygULf?=
 =?us-ascii?Q?mhMiCRPYnWSkcZvdObN31BIgIwWW+rulLFFCHLKpJmibT37wzGdmOu5Ecqwn?=
 =?us-ascii?Q?+FuQBzcf37kZ5gAckLgUPt0a0gLtEcS6DKyVmoMGdsKbt+g0ujZ9guBt++ir?=
 =?us-ascii?Q?l4Wsuay3bO22urlf8A7kv/Iv9PP39loFr59CxPz2XXSKekw2fQfFpACoKM4+?=
 =?us-ascii?Q?stlUQnGGtV70MD0JrWolk/WgSjujeddd6wpCToHai+Pj7yATjK7LqkGutUWb?=
 =?us-ascii?Q?WX0ouJHNd+1rPJO8396EeiuK6b847lU6R+OnhDBtw1p543vJ1vkWQo+DU/KD?=
 =?us-ascii?Q?GszZmCFnPoxN6Vu2FxFUz/PsPd7nxU3iRF2z89rN22kLr8w6/k6Z+XTwp8zl?=
 =?us-ascii?Q?mvY2nIwdR5eSi4JNC3S4Pg2UY6xllvfXNkHNPUR/eeHoRGwC7gPpveiSWu3P?=
 =?us-ascii?Q?ZlN6YQEFsUX1edKL4XhlBtIQzTNLr/ZVsTnAWl+8e0mvhYQevRg21YU3IBvc?=
 =?us-ascii?Q?M9rB1OilWRZISstDupbk6hT251mnLuW8Th00ynv4gGLp+W30cygBKcvUJz+T?=
 =?us-ascii?Q?C+SwECww+cy+ITeG21vLDzbwrPEAZd/g0rDXD1QhsnsSAEDGyqaGNMd8YKdI?=
 =?us-ascii?Q?eDOC77BJiUuJ48+jGtNC5rzC9VYnS1dGlU1ZJeMSczYYNEJvouSg5XJX8TxE?=
 =?us-ascii?Q?0QmX+SHjvCaKNvqU+tbg21bujgeDUJnKDYZNJljvcVdXDY3eeeHiGWFUAV1b?=
 =?us-ascii?Q?vqwF4pYCu3G4bhl8l2dft2iAdW/yOaRpsl4NCj7dcRkgwPQ6Yp0Wa0Uod3aY?=
 =?us-ascii?Q?P5aNs+htYkZ94n7yN+bL2mfcYmwnHqcE14+yiW03Tgbx+NReJIysNe75jan5?=
 =?us-ascii?Q?HubwFgHLYZCbgV+nOK6anfysj/Sh+wCCbmO2bk100p4ExL7iQqR2vjwHv1Ve?=
 =?us-ascii?Q?IcEtcFDSX5kB4maG9z8=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 04fa9997-fa94-42fa-a24c-08dc7a80e11c
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 May 2024 17:01:53.2688
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JUd3t17vvS3/Z+ltJuF32nWRorIo24NHdhQ3IutS7/YlUHZd2p0y+MTkIDCXVYSL
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8131

On Wed, May 22, 2024 at 02:29:19PM +0800, Yan Zhao wrote:
> > > If you want to minimize flushes then you can't store flush
> > > minimization information in the pages because it isn't global to the
> > > pages and will not be accurate enough.
> > > 
> > > > > If pfn_reader_fill_span() does batch_from_domain() and
> > > > > the source domain's storage_domain is non-coherent then you can skip
> > > > > the flush. This is not pedantically perfect in skipping all flushes, but
> > > > > in practice it is probably good enough.
> > > 
> > > > We don't know whether the source storage_domain is non-coherent since
> > > > area->storage_domain is of "struct iommu_domain".
> > >  
> > > > Do you want to add a flag in "area", e.g. area->storage_domain_is_noncoherent,
> > > > and set this flag along side setting storage_domain?
> > > 
> > > Sure, that could work.
> > When the storage_domain is set in iopt_area_fill_domains(),
> >     "area->storage_domain = xa_load(&area->iopt->domains, 0);"
> > is there a convenient way to know the storage_domain is non-coherent?
> Also asking for when storage_domain is switching to an arbitrary remaining domain
> in iopt_unfill_domain().
> 
> And in iopt_area_unfill_domains(), after iopt_area_unmap_domain_range()
> of a non-coherent domain which is not the storage domain, how can we know that
> the domain is non-coherent?

Yes, it would have to keep track of hwpts in more case unfortunately
:(

Jason