Received-SPF: pass (google.com: domain of linux-kernel+bounces-190125-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Mon, 27 May 2024 15:15:39 +0800
From: Yan Zhao <yan.y.zhao@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
CC: <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <x86@kernel.org>,
	<alex.williamson@redhat.com>, <kevin.tian@intel.com>,
	<iommu@lists.linux.dev>, <pbonzini@redhat.com>, <seanjc@google.com>,
	<dave.hansen@linux.intel.com>, <luto@kernel.org>, <peterz@infradead.org>,
	<tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>, <hpa@zytor.com>,
	<corbet@lwn.net>, <joro@8bytes.org>, <will@kernel.org>,
	<robin.murphy@arm.com>, <baolu.lu@linux.intel.com>, <yi.l.liu@intel.com>
Subject: Re: [PATCH 5/5] iommufd: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <ZlQzGzU1/CLY0eOg@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <ZkN/F3dGKfGSdf/6@nvidia.com>
 <ZkRe/HeAIgscsYZw@yzhao56-desk.sh.intel.com>
 <ZkUeWAjHuvIhLcFH@nvidia.com>
 <ZkVwS8n7ARzKAbyW@yzhao56-desk.sh.intel.com>
 <20240517170418.GA20229@nvidia.com>
 <Zkq5ZL+saJbEkfBQ@yzhao56-desk.sh.intel.com>
 <20240521160442.GI20229@nvidia.com>
 <Zk1jrI8bOR5vYKlc@yzhao56-desk.sh.intel.com>
 <Zk2Qv4pnSKZBsLYv@yzhao56-desk.sh.intel.com>
 <20240522170150.GE20229@nvidia.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240522170150.GE20229@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?x8CNmw/KDni0fYkKYrSf92OlSBuUKyhsQHEz5pol7McPCeZKuIyzDK/TISFx?=
 =?us-ascii?Q?F5+bt9AId8c5vzu0sNgtnfODKyAS0Dtvi67vEzgiDqtWGKCaWdYa50mWuXE9?=
 =?us-ascii?Q?Etih4vdp2pFS+pC0vI3ZREbOlM21HQ9zD0kJUoSAwpwPuku3slycxI0Tn/Mp?=
 =?us-ascii?Q?0hTR13T0u20qY6WRQJnxpFcHesTQqHUkgGQ/GvXTUzGM0dIJ2lDo40XMrp7w?=
 =?us-ascii?Q?wJKkIaI8nA6z0hCnjGsbT0wEAHh/55pNW1jRFJyipZC8FptQRgbcTBlG8nQP?=
 =?us-ascii?Q?JPf97iRJbjkTq/03TXcTZTNRZhuqDQ97GYXYfZj/fmfKbpb/Ghshf0d16RI7?=
 =?us-ascii?Q?AGOTHa/GmO0FoPGi5RpPZTxFVNps6VlVInOtyeXPYE68aYuxvrjSmqF8HsXV?=
 =?us-ascii?Q?Qt/LdkiNfnXW1gx+H8zrZs44HwsnJeEG4fBWJdAyo4M6fIxmiQghyoL+nrj9?=
 =?us-ascii?Q?uz6qgqna8eU1AWaJemm0nMW6/9llDWagei/fXKlPoTiyh3YtHTkIAkbwFf2R?=
 =?us-ascii?Q?hBNpyzBbSuWWFK/UKr+Jpin6+qaZxoXBtAk448JBXYzVrVNSDmbAmd2KUcps?=
 =?us-ascii?Q?yuUtel7JJL74LRG90rOG6TzeheMM8WUv4KtEgzNPiSlqWrV0KiqE5gYdSnrS?=
 =?us-ascii?Q?G2wnUUmMGWWa8w01VSZgxUiJVs+vaKucpYO1FyOErTV8ikoKPjt9itxY4m93?=
 =?us-ascii?Q?pDOHDr67UlcZCMbFnFPwGA897gHbmdOnAR41tnbjTnsLH2qltsMNt7st2t3g?=
 =?us-ascii?Q?oYESO2HLyLcS1iVOTkWbPZcZBloMQLF+CvA+uFoGuhMhpxCiqPTolFKM5ziy?=
 =?us-ascii?Q?ACXP4fiwi8uJYdYPiF26vDR5we6CZgJnMcZrT+oybRZYEfhZ3zEcVd3P/q4A?=
 =?us-ascii?Q?1wWW/pyYlE3cAQRVBboLeJM9qDRS9l5eo4PW9vt3Ys7A5jzgxAIEfs8oHIT0?=
 =?us-ascii?Q?4ALb6UyiB66+8+jwa3xdqL4t8ab+T8WlRbmC4gm48KBuHAYC6HJXkl3wbXpw?=
 =?us-ascii?Q?+p04I+mZnDoQsSwQ+o2QFgAkSa59fENefhCH7UODfYWdEk6eAV5+I4HAUM62?=
 =?us-ascii?Q?5VuUjYKMMSH1I0uzqwZgbOiWOz6rjlxYimtsKWxiR1NMvBaPsWi4fcwJ8bdc?=
 =?us-ascii?Q?T55XCtHXSkl8GmPqMqwJ+gvm9JFZ6aCmJDYhmLIT9v7xaPRRmJ7fHJ6BX4Qb?=
 =?us-ascii?Q?yp1fu25Ydo6q8rdKZ/ircB5sbZQElA7iwr0ZMUJAQXl/xNkebv1bj6GO9ro/?=
 =?us-ascii?Q?xUysNbKmRutui8ePIlgCnNNyePN3mwZD2RyK8LTtIY7FwbCBtWXqNP1CWNpH?=
 =?us-ascii?Q?6h52K/bYFO92ZUq/cnycqyyE3zqdsBJiDQpb1G0xqnEo79i6QEemZtdvkw9/?=
 =?us-ascii?Q?zKl6zDevtZlzfHdQeReTKFq0Qp+NiGJfWTRYCOnEqkzDR6zJEGO2P/4bFuAZ?=
 =?us-ascii?Q?BCrQrBdsHpYip+vAp538/iJGGr5Uljkr8ICQvQb0KDUUU1tNSVwPp7+yN663?=
 =?us-ascii?Q?324znsIv4ROyjC8IjvgyRFVJiZqXSDBVlsMQzs2GOpWMrtEDhLK909wRFSx4?=
 =?us-ascii?Q?P2e8W3cqPvVqCK1btMGtJqJeGfS0yaEacP6B2BPy?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 978b7675-accc-4f9c-14a4-08dc7e1cf156
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 May 2024 07:16:35.4483
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: M0UFAqdSA1oEuZ68mrx6brGMAFbvt5vLwlhoNF1TIFmnqwW4tQleNfAL43aPjcoAzXmFZoEqsYi0d1FseixBVw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB7423
X-OriginatorOrg: intel.com

On Wed, May 22, 2024 at 02:01:50PM -0300, Jason Gunthorpe wrote:
> On Wed, May 22, 2024 at 02:29:19PM +0800, Yan Zhao wrote:
> > > > If you want to minimize flushes then you can't store flush
> > > > minimization information in the pages because it isn't global to the
> > > > pages and will not be accurate enough.
> > > > 
> > > > > > If pfn_reader_fill_span() does batch_from_domain() and
> > > > > > the source domain's storage_domain is non-coherent then you can skip
> > > > > > the flush. This is not pedantically perfect in skipping all flushes, but
> > > > > > in practice it is probably good enough.
> > > > 
> > > > > We don't know whether the source storage_domain is non-coherent since
> > > > > area->storage_domain is of "struct iommu_domain".
> > > >  
> > > > > Do you want to add a flag in "area", e.g. area->storage_domain_is_noncoherent,
> > > > > and set this flag along side setting storage_domain?
> > > > 
> > > > Sure, that could work.
> > > When the storage_domain is set in iopt_area_fill_domains(),
> > >     "area->storage_domain = xa_load(&area->iopt->domains, 0);"
> > > is there a convenient way to know the storage_domain is non-coherent?
> > Also asking for when storage_domain is switching to an arbitrary remaining domain
> > in iopt_unfill_domain().
> > 
> > And in iopt_area_unfill_domains(), after iopt_area_unmap_domain_range()
> > of a non-coherent domain which is not the storage domain, how can we know that
> > the domain is non-coherent?
> 
> Yes, it would have to keep track of hwpts in more case unfortunately
> :(
Current iopt keeps an xarray of domains. Could I modify the xarray to store
hwpt instead?