Received-SPF: pass (google.com: domain of linux-kernel+bounces-197954-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Sat, 1 Jun 2024 13:48:28 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Yan Zhao <yan.y.zhao@intel.com>
Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org,
	alex.williamson@redhat.com, kevin.tian@intel.com,
	iommu@lists.linux.dev, pbonzini@redhat.com, seanjc@google.com,
	dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org,
	tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com,
	corbet@lwn.net, joro@8bytes.org, will@kernel.org,
	robin.murphy@arm.com, baolu.lu@linux.intel.com, yi.l.liu@intel.com
Subject: Re: [PATCH 5/5] iommufd: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <ZltQ3PyHKiQmN9SU@nvidia.com>
References: <ZkRe/HeAIgscsYZw@yzhao56-desk.sh.intel.com>
 <ZkUeWAjHuvIhLcFH@nvidia.com>
 <ZkVwS8n7ARzKAbyW@yzhao56-desk.sh.intel.com>
 <20240517170418.GA20229@nvidia.com>
 <Zkq5ZL+saJbEkfBQ@yzhao56-desk.sh.intel.com>
 <20240521160442.GI20229@nvidia.com>
 <Zk1jrI8bOR5vYKlc@yzhao56-desk.sh.intel.com>
 <Zk2Qv4pnSKZBsLYv@yzhao56-desk.sh.intel.com>
 <20240522170150.GE20229@nvidia.com>
 <ZlQzGzU1/CLY0eOg@yzhao56-desk.sh.intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZlQzGzU1/CLY0eOg@yzhao56-desk.sh.intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?hu7Ka6iGyv5zMPGBrR4H05LouMdVpZYxahABYhHLr9nYR+nfuGeE1HxOOS53?=
 =?us-ascii?Q?/3qTikQGctDoAjEvE5bZCW1oRxtqMILP+crn2H76Vt0Pq8o/6ajQnHKsnqMf?=
 =?us-ascii?Q?/Ma9+j/DZlwZM/uzKwrPjZ8jNNqOQJnH4cyLQJQflGUa9pxG3kYai/D2xlEi?=
 =?us-ascii?Q?CtSW0fzazacmj0Kjk6pZI4EzkCh7fJ30Jsa2fF39cV84tLXiUdTQ/JYmEedj?=
 =?us-ascii?Q?WVJyfgWlAJHW30IAoMylXReP29WCtyD2pNz4NGXFGYVs8hmmG7mtwqvwZ+CP?=
 =?us-ascii?Q?O3Gjjrs1AbbPjtXfqQ9gPD22Sfwr1UfCWf9iqvRZPeguG6a1lHfwYZxR6l46?=
 =?us-ascii?Q?eW05c/QTojg4Lld2PQp5JgxWFWizVfOFRfiknN2Oa2i8RTpuYzbiuFJY98Op?=
 =?us-ascii?Q?KHt/sl8dThe5ddY5tg7lg3JdfFL7x8+D0KexsjPsE8ctko2e4fheNel6UsSI?=
 =?us-ascii?Q?lGY2xBK0Ogp50kH9+TaFVTGa1WvNdEmEn7ILTvFityPUPKofIYhJn1Lr1Mf/?=
 =?us-ascii?Q?up5JB8uW1vKeLq9Li3+tg1KvO+NWWikBb+qRYQ/6qyCNZ59blxq9CsyaXeoy?=
 =?us-ascii?Q?3QYEbar06O2SBXN1H1TzNEZjrhcHr2A/YaB/ICIGsV3vHTDqOnyFhEEvIRBq?=
 =?us-ascii?Q?IJIEuVSVfv10K11/L6d1vHQ9QjWVZ/zKqF0YlA022lhjxiNsgLdq3tOE/2Ob?=
 =?us-ascii?Q?Zq1T8P1s+QGRmkxrefDB0b7IUCYUNe9rWrDmdg7ZZH1oqQilBaxHUVQpFxre?=
 =?us-ascii?Q?Lcs/FcJKodXYuidt/4avzhND62XwQXNa6HAWUqIRXakGVwZtfMegT2ap/Ie8?=
 =?us-ascii?Q?gqwxEMH+Wneb3T6xjgZDIT7FNqtAr/PwpZPiv6V2Dll1R4yFyoG/P69d9KK0?=
 =?us-ascii?Q?5rZa9WMT90D3BGjS37nBoe2cF52Qp20iBqZ1DEyatvjuSa1jE4afzpsiPdHZ?=
 =?us-ascii?Q?FXMuyr09nsXrQIqZ+Qe/hZMR0VxEnF17b95dJp22Zo/2wV8vZt4pUfo6V6d/?=
 =?us-ascii?Q?ryn2wha7gxM/796BMoW1hCOOyqIQwStr9DmLz0Ipacbpbz4Is1lWx3CUDEnN?=
 =?us-ascii?Q?fvZnP/5Ws/kQnbUMethdrJaMQXba/CjuYtzS0St3d7AnJO/YgsWk1QJbRTU+?=
 =?us-ascii?Q?wVbrM2L+mTkt3//xunCFtrIyKZabTRwZ4eYnY3k66Adji93rzP55uHrethAM?=
 =?us-ascii?Q?g0AKAAS6CbOxxfUvNIRtQZ9HZMRUJDkY2GLtscv9CS9JTNJRnUoKUtyAbaib?=
 =?us-ascii?Q?/eWayiYGhv/NRRlulJM+3yWGT5HGP7g3I6IO6ou8fl6PVlhKOZ5db70dTulj?=
 =?us-ascii?Q?5I4n093Hj6oFIhKIexQcdvT1Cq/NVSowAglaS2hSGIp2QJs5qVvS6nDtQWuw?=
 =?us-ascii?Q?skXeJllUOsyqMY3H/dGqyGpLqz6VHkwbeyZBEtDrIgS9W59qutk3aOI4SZCO?=
 =?us-ascii?Q?fbRNPwPv5LlRfWRb+MJYn7ghyBUp6/MsQIN2H7YD6uKtIHbVo9LWhv8FkRcJ?=
 =?us-ascii?Q?HQi+HcOeodSyeaGYREUu/ORXvWi8wMW6r0B83fKrp3zy9fHnQTf4L8cyMFC6?=
 =?us-ascii?Q?/5Uc8boZEQOFEbPHZKJqFNbsWE+CuJiMQRtjJDwZ?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: dd21882a-90a8-4308-6827-08dc825aaa5c
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jun 2024 16:48:29.7914
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: oRPlreXWCyRoUqKtSwK4vYnuHcUmU431cuH8oE2iXuZXtNony8dJBmAu7wJpls1u
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7644

On Mon, May 27, 2024 at 03:15:39PM +0800, Yan Zhao wrote:
> On Wed, May 22, 2024 at 02:01:50PM -0300, Jason Gunthorpe wrote:
> > On Wed, May 22, 2024 at 02:29:19PM +0800, Yan Zhao wrote:
> > > > > If you want to minimize flushes then you can't store flush
> > > > > minimization information in the pages because it isn't global to the
> > > > > pages and will not be accurate enough.
> > > > > 
> > > > > > > If pfn_reader_fill_span() does batch_from_domain() and
> > > > > > > the source domain's storage_domain is non-coherent then you can skip
> > > > > > > the flush. This is not pedantically perfect in skipping all flushes, but
> > > > > > > in practice it is probably good enough.
> > > > > 
> > > > > > We don't know whether the source storage_domain is non-coherent since
> > > > > > area->storage_domain is of "struct iommu_domain".
> > > > >  
> > > > > > Do you want to add a flag in "area", e.g. area->storage_domain_is_noncoherent,
> > > > > > and set this flag along side setting storage_domain?
> > > > > 
> > > > > Sure, that could work.
> > > > When the storage_domain is set in iopt_area_fill_domains(),
> > > >     "area->storage_domain = xa_load(&area->iopt->domains, 0);"
> > > > is there a convenient way to know the storage_domain is non-coherent?
> > > Also asking for when storage_domain is switching to an arbitrary remaining domain
> > > in iopt_unfill_domain().
> > > 
> > > And in iopt_area_unfill_domains(), after iopt_area_unmap_domain_range()
> > > of a non-coherent domain which is not the storage domain, how can we know that
> > > the domain is non-coherent?
> > 
> > Yes, it would have to keep track of hwpts in more case unfortunately
> > :(
> Current iopt keeps an xarray of domains. Could I modify the xarray to store
> hwpt instead? 

Probably yes

Jason