Received-SPF: pass (google.com: domain of linux-kernel+bounces-181740-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Fri, 17 May 2024 13:00:38 +0800
From: Yan Zhao <yan.y.zhao@intel.com>
To: Alex Williamson <alex.williamson@redhat.com>
CC: <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <x86@kernel.org>,
	<jgg@nvidia.com>, <kevin.tian@intel.com>, <iommu@lists.linux.dev>,
	<pbonzini@redhat.com>, <seanjc@google.com>, <dave.hansen@linux.intel.com>,
	<luto@kernel.org>, <peterz@infradead.org>, <tglx@linutronix.de>,
	<mingo@redhat.com>, <bp@alien8.de>, <hpa@zytor.com>, <corbet@lwn.net>,
	<joro@8bytes.org>, <will@kernel.org>, <robin.murphy@arm.com>,
	<baolu.lu@linux.intel.com>, <yi.l.liu@intel.com>
Subject: Re: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <Zkbkdi+hipp3/YF1@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20240507061802.20184-1-yan.y.zhao@intel.com>
 <20240507062138.20465-1-yan.y.zhao@intel.com>
 <20240509121049.58238a6f.alex.williamson@redhat.com>
 <Zj33cUe7HYOIfj5N@yzhao56-desk.sh.intel.com>
 <20240510105728.76d97bbb.alex.williamson@redhat.com>
 <ZkG9IEQwi7HG3YBk@yzhao56-desk.sh.intel.com>
 <20240516145009.3bcd3d0c.alex.williamson@redhat.com>
 <ZkbK9CzmcxgqhSuR@yzhao56-desk.sh.intel.com>
 <20240516224442.56df5c23.alex.williamson@redhat.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240516224442.56df5c23.alex.williamson@redhat.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?BLl7v91oTNNK+uSBMsRJTKtNmcD5GhAPwTFGTibL/N6FqCx6A3CFpqSbVFX9?=
 =?us-ascii?Q?bZnhjCurSbzPdYcxNxZOw0QVYhD3JJfcsZqPbWh+ZNnqfwAIWeRewId9MMDm?=
 =?us-ascii?Q?w9zNZoeVTIbMVwgsTvfLl23ay0IdeBYA9++Z3u8RHWI9NDqG65nugxRFC4pr?=
 =?us-ascii?Q?u5OkxjJavALn8ePuQ13MqXCF76imFdRFOaL7utcOSAo5OfSW2p1JnMdwJ94O?=
 =?us-ascii?Q?W/WwJ3dZnJU/tMNzncI0x6Zh365dbG7OItoLwMmRxWT/6hALvKVyikJ2mp/h?=
 =?us-ascii?Q?lP4wM0khRpjk2VUzmpVszh4tTp4pmKobchHwl+fsCW5g/0XCH5S9N/r4xpdd?=
 =?us-ascii?Q?2PCol+/FCRErySumIZWV0KXOk/qYpSiIik5kDRKPucacZo3+HyV1iKMRMiJ7?=
 =?us-ascii?Q?o35yMaPlWK0710UIbACOso41TthCx25pVbstHGqkeMAyAcYjojHXABXH6v9I?=
 =?us-ascii?Q?bWoOe3TyPfBYmAHa+zGBdbyT3Ngh5hhbJ0C4mHYXUcsVS3EoZfuNnB4Q5y5z?=
 =?us-ascii?Q?Akd6j1RPWTvSuwhw0PiESY0n8/mAmxCWjDrsUt01EicwjopB9ceh8HQZY72q?=
 =?us-ascii?Q?1VQ9jIF5zqyEcc3ib2wPKpxh4Y4iy83G8E7RcrRevbQuTDB2JfWgXhGobnPF?=
 =?us-ascii?Q?NpBGK5dhoopvNm5/kecy5u4/SWysosV8/V9aSE2h6DhUiC7dNMpwB/MUyUoi?=
 =?us-ascii?Q?Q1dAjZ+vmcjgkTzxetIzJCChjviF0imddM5QyF/W0wW06va5WQf0IW4TfikA?=
 =?us-ascii?Q?STAGv7FAg/F1435ZKQAtVip1FouP3nBx1H948jfYxNtvZ+DLLHiUgZCORAnI?=
 =?us-ascii?Q?eL9pw0GB0ud5N/Bk2Avdv7z3OuHMHxIjYMXIq4JOiHhZy4LbB479MUhOi8u9?=
 =?us-ascii?Q?LEI9zZHQ61ENOmXRSlv38N4S5Yzn5uUsE5B96pPViMxLwaCzKGs8FurHEY/N?=
 =?us-ascii?Q?kaXK/mCBxvLxTufHyBNA9jBn52o4Yh/P3Ag7iZXWGc8TZHYum37xSCyGA+JE?=
 =?us-ascii?Q?EeIYEIWL8fvf+7COrjve2cM1SxV5CMaO6nRkyPPfhb5ux1W2vy50mayNyiP2?=
 =?us-ascii?Q?XcH5E/Cv6Ha0TspWI6O+dBMvAqRmkcTcnfV36ihG4UzvCubigjBT/W3V2Pki?=
 =?us-ascii?Q?tS7Q0hdYGfm7mj8sMK8CQhI+N3K+9Hd/r417+7sKSuzY+BDoefTn5nW952iE?=
 =?us-ascii?Q?zksDk/Fmex/yKGpVcyutS5W3pJ0+atTktEvDGFA+Blq3pANtha8hjpY9CZpO?=
 =?us-ascii?Q?kr4sFAHssfhrfkXooBLcBKaDWBBlm2b+6pF84KjSwO6LzTkg4b2nkx4lkCZd?=
 =?us-ascii?Q?PYbc/C42BWN0aP0ok+dq2hhCaK+/USL2ENsVRvyGhFXA2PjwjFwgGMpwipo7?=
 =?us-ascii?Q?joPXyswJ8ygIhYeEI2+GLvaDFhm/a0dODk/ESjd953MxDsBxZeOn0aKUVPq3?=
 =?us-ascii?Q?14KPMPDyTajySOpS8EGqJ0YJ8oIUMsxnqa5HCEGYAqxMrrBZ73FeWO8gOUNx?=
 =?us-ascii?Q?Nlj70sIi5tTFalexnXkl069cuPhoQ6cG4RvD6r4WphLE58ukfuSA0Izn3ahX?=
 =?us-ascii?Q?Qi4Keb9wsftsqXPl3hLKFz/zYt7th/BVEsKQjqtk?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c821d591-6ec1-4e6f-e07c-08dc762e693b
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 May 2024 05:01:28.7298
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: OelTdtRmV/bozm5dREa4FiQoX698zE2t30uoaryI8hmZ8qvv1moWv8UcNRVpUpztQihyZTZuxQtZzgWTdcGoPw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6655
X-OriginatorOrg: intel.com

On Thu, May 16, 2024 at 10:44:42PM -0600, Alex Williamson wrote:
> On Fri, 17 May 2024 11:11:48 +0800
> Yan Zhao <yan.y.zhao@intel.com> wrote:
> 
> > On Thu, May 16, 2024 at 02:50:09PM -0600, Alex Williamson wrote:
> > > On Mon, 13 May 2024 15:11:28 +0800
> > > Yan Zhao <yan.y.zhao@intel.com> wrote:
> > >   
> > > > On Fri, May 10, 2024 at 10:57:28AM -0600, Alex Williamson wrote:  
> > > > > On Fri, 10 May 2024 18:31:13 +0800
> > > > > Yan Zhao <yan.y.zhao@intel.com> wrote:
> > > > >     
> > > > > > On Thu, May 09, 2024 at 12:10:49PM -0600, Alex Williamson wrote:    
> > > > > > > On Tue,  7 May 2024 14:21:38 +0800
> > > > > > > Yan Zhao <yan.y.zhao@intel.com> wrote:      
> > ...   
> > > > > > > > @@ -1683,9 +1715,14 @@ static int vfio_iommu_replay(struct vfio_iommu *iommu,
> > > > > > > >  	for (; n; n = rb_next(n)) {
> > > > > > > >  		struct vfio_dma *dma;
> > > > > > > >  		dma_addr_t iova;
> > > > > > > > +		bool cache_flush_required;
> > > > > > > >  
> > > > > > > >  		dma = rb_entry(n, struct vfio_dma, node);
> > > > > > > >  		iova = dma->iova;
> > > > > > > > +		cache_flush_required = !domain->enforce_cache_coherency &&
> > > > > > > > +				       !dma->cache_flush_required;
> > > > > > > > +		if (cache_flush_required)
> > > > > > > > +			dma->cache_flush_required = true;      
> > > > > > > 
> > > > > > > The variable name here isn't accurate and the logic is confusing.  If
> > > > > > > the domain does not enforce coherency and the mapping is not tagged as
> > > > > > > requiring a cache flush, then we need to mark the mapping as requiring
> > > > > > > a cache flush.  So the variable state is something more akin to
> > > > > > > set_cache_flush_required.  But all we're saving with this is a
> > > > > > > redundant set if the mapping is already tagged as requiring a cache
> > > > > > > flush, so it could really be simplified to:
> > > > > > > 
> > > > > > > 		dma->cache_flush_required = !domain->enforce_cache_coherency;      
> > > > > > Sorry about the confusion.
> > > > > > 
> > > > > > If dma->cache_flush_required is set to true by a domain not enforcing cache
> > > > > > coherency, we hope it will not be reset to false by a later attaching to domain 
> > > > > > enforcing cache coherency due to the lazily flushing design.    
> > > > > 
> > > > > Right, ok, the vfio_dma objects are shared between domains so we never
> > > > > want to set 'dma->cache_flush_required = false' due to the addition of a
> > > > > 'domain->enforce_cache_coherent == true'.  So this could be:
> > > > > 
> > > > > 	if (!dma->cache_flush_required)
> > > > > 		dma->cache_flush_required = !domain->enforce_cache_coherency;    
> > > > 
> > > > Though this code is easier for understanding, it leads to unnecessary setting of
> > > > dma->cache_flush_required to false, given domain->enforce_cache_coherency is
> > > > true at the most time.  
> > > 
> > > I don't really see that as an issue, but the variable name originally
> > > chosen above, cache_flush_required, also doesn't convey that it's only
> > > attempting to set the value if it wasn't previously set and is now
> > > required by a noncoherent domain.  
> > Agreed, the old name is too vague.
> > What about update_to_noncoherent_required?
> 
> set_noncoherent?  Thanks,
> 
Concise!

> 
> > Then in vfio_iommu_replay(), it's like
> > 
> > update_to_noncoherent_required = !domain->enforce_cache_coherency && !dma->is_noncoherent;
> > if (update_to_noncoherent_required)
> >          dma->is_noncoherent = true;
> > 
> > ...
> > if (update_to_noncoherent_required)
> > 	arch_flush_cache_phys((phys, size);
> > >   
> > > > > > > It might add more clarity to just name the mapping flag
> > > > > > > dma->mapped_noncoherent.      
> > > > > > 
> > > > > > The dma->cache_flush_required is to mark whether pages in a vfio_dma requires
> > > > > > cache flush in the subsequence mapping into the first non-coherent domain
> > > > > > and page unpinning.    
> > > > > 
> > > > > How do we arrive at a sequence where we have dma->cache_flush_required
> > > > > that isn't the result of being mapped into a domain with
> > > > > !domain->enforce_cache_coherency?    
> > > > Hmm, dma->cache_flush_required IS the result of being mapped into a domain with
> > > > !domain->enforce_cache_coherency.
> > > > My concern only arrives from the actual code sequence, i.e.
> > > > dma->cache_flush_required is set to true before the actual mapping.
> > > > 
> > > > If we rename it to dma->mapped_noncoherent and only set it to true after the
> > > > actual successful mapping, it would lead to more code to handle flushing for the
> > > > unwind case.
> > > > Currently, flush for unwind is handled centrally in vfio_unpin_pages_remote()
> > > > by checking dma->cache_flush_required, which is true even before a full
> > > > successful mapping, so we won't miss flush on any pages that are mapped into a
> > > > non-coherent domain in a short window.  
> > > 
> > > I don't think we need to be so literal that "mapped_noncoherent" can
> > > only be set after the vfio_dma is fully mapped to a noncoherent domain,
> > > but also we can come up with other names for the flag.  Perhaps
> > > "is_noncoherent".  My suggestion was more from the perspective of what
> > > does the flag represent rather than what we intend to do as a result of
> > > the flag being set.  Thanks,   
> > Makes sense!
> > I like the name "is_noncoherent" :)
> > 
>