Received-SPF: pass (google.com: domain of linux-kernel+bounces-205745-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Fri, 7 Jun 2024 17:39:21 +0800
From: Yan Zhao <yan.y.zhao@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
CC: Christoph Hellwig <hch@infradead.org>, <kvm@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <x86@kernel.org>,
	<alex.williamson@redhat.com>, <kevin.tian@intel.com>,
	<iommu@lists.linux.dev>, <pbonzini@redhat.com>, <seanjc@google.com>,
	<dave.hansen@linux.intel.com>, <luto@kernel.org>, <peterz@infradead.org>,
	<tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>, <hpa@zytor.com>,
	<corbet@lwn.net>, <joro@8bytes.org>, <will@kernel.org>,
	<robin.murphy@arm.com>, <baolu.lu@linux.intel.com>, <yi.l.liu@intel.com>,
	Russell King <linux@armlinux.org.uk>
Subject: Re: [PATCH 3/5] x86/mm: Introduce and export interface
 arch_clean_nonsnoop_dma()
Message-ID: <ZmLVSU39rJBzQFX0@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20240507061802.20184-1-yan.y.zhao@intel.com>
 <20240507062044.20399-1-yan.y.zhao@intel.com>
 <ZktZDmcNnsHhp4Tm@infradead.org>
 <20240521154939.GH20229@nvidia.com>
 <20240521160016.GA2513156@nvidia.com>
 <ZlV7rlmWdU7dJZKo@infradead.org>
 <Zlt6huNJeW8ekJlE@nvidia.com>
 <ZmEjavnYePBLDbrS@yzhao56-desk.sh.intel.com>
 <20240606115503.GZ19897@nvidia.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240606115503.GZ19897@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?xulCcF7NURQOFDydLYx3SIG9/frlbGl3hpCkWQapzZaNVeBvECvwHvtm4JBV?=
 =?us-ascii?Q?D48LgEuyRLM/kdN6EiJXS8E0olUnw0UP+SiULtgIJlBbXaSoBNU9ONxAxf1A?=
 =?us-ascii?Q?1dm0gITnn6B7u2k8vqSgEocKeQDNvlST51BQDwwzG5lxWZLV3+FRsUHA6yTG?=
 =?us-ascii?Q?dD63qME4WAGISXuADXz85LVQcOMWGDV7AZR5UTKjkrLmOO8lBxbbgL/7zdsC?=
 =?us-ascii?Q?+7PbBVgGU60qWf8IODd/WxVE31sqPTC+uaSgfpVO51EusRrHYffr8tOXRLK4?=
 =?us-ascii?Q?jVR9f/Zok6tnKbIjqOM1GjrVMokCuzKd9n6tu7BRT8S3+Mhe5QPC2+XCDADD?=
 =?us-ascii?Q?vNyp6zsX8WCAQ34Qcusz89mhGFGcYDK+pv85AjXsNDBRVyIVMoNWtiszbTjx?=
 =?us-ascii?Q?GibyToKdSutYIueSRc4bHMiNZXscmKAHViLgBKWpvOrdHYXe9ksw+viIYfkC?=
 =?us-ascii?Q?Q16/BxhUE/jNpQjdUmqXQ1G/WMTuNx24VeFOkwNVl/mG+DXlqiNwc5y6gexy?=
 =?us-ascii?Q?64MJCYKovMwcJ6AurhRruPPCCjS/ji/BxC3lKMnHdvbpR/fuDZZsJcC1ETax?=
 =?us-ascii?Q?tXBHB3TZSWDucAoxrTptRo71Q2BZ/QGXqm7KWeZyzrDOap78BM2cbZa1S60C?=
 =?us-ascii?Q?ONEy9/PsGbOBcwSmLRPbitCA1GMPe48wHSqltfGo2vzwvFAVjZjPv4wDxYgp?=
 =?us-ascii?Q?hi1b1/6D2foerzZLmdh3O/5iWbkh47Lgv7N/4gnPsNkd1xXMjWP1OIWdgcj/?=
 =?us-ascii?Q?ci0N6F6NpXMhx1r6PsNcWsWazfriClnZCh8XGcobdeOcm3IL9KBE2SzJSwtw?=
 =?us-ascii?Q?DP3WTHYK95UrZ7VWJNq8xxNiY2wyZI5A1BFf4sc1m2q7szuNxZkUJTr8uohq?=
 =?us-ascii?Q?r4G7CYo1rjwIE+YDIWEwYkzYmUewO2wpN0+7eLdD2Bog9fAX77L7bTntvxfx?=
 =?us-ascii?Q?6UTdlmqYwNUjkEjzLRJ8Z29rrIEMN/KcZNDcv3+NsSxE+dQSSFCC1e4idP9C?=
 =?us-ascii?Q?kgAsXHBRzE4ifTXtfceJHzuch0apWEIkX2qw354sy6m63mYePjIohmAF9GR+?=
 =?us-ascii?Q?OfdSla0rsM3/VRJtHVZ7E5OZZ1RHGtvEm4+ov63FXdPGQKkGIk4bHSi62Iom?=
 =?us-ascii?Q?sS2PAYRNCez6ihkha8sHhG/3R6NE4OI+vJnE8q401eiu01sr6L10cl/oeL5g?=
 =?us-ascii?Q?Rq9nI44fWvrCpSRQZVQL9/cfFmjZoexvC+FfriFhRSkVWKsNtDmKPYVMRiU/?=
 =?us-ascii?Q?XRFy7ub17bdoSHd8n3KBSvOtxewjhiefnUddG6xyn3TuDUq2NdIjn7pXsZft?=
 =?us-ascii?Q?E4KS5vhKgXp8hhLaVuZwAFfPfBU3FNBfVVk6UaYAGgqgr34dd3E7XDjlnKxJ?=
 =?us-ascii?Q?NhnGVVu5dFblYyqrUZANNxsFUe8/kpRM7srAAZceOD/YwHqJq6L8BrBvOWAi?=
 =?us-ascii?Q?tzMFnZiCHK9XC3l0qdwMo+aBarLF/qWfl15rUpZtQmalrpZKiUdVXwKafF9W?=
 =?us-ascii?Q?yUrrcHIaSwgbb7OQErwdSDMemEyFzC+0oa7zYYzITAopzHkHhg3TZMPgxdrO?=
 =?us-ascii?Q?w0yKvO17UHlFC9Mi5uW3Zn+giIEmyMpgXptcV49h?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 053ee7a2-a4d0-4779-a6d9-08dc86d5dace
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jun 2024 09:40:23.7960
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MA+LJcEauHP0s1/SqixBUXS0msM6uguI+Cr1ppqRNlMrlHfxifx8i0QchY9n3X7VlRDYQackb6DwUUUpdBk1pg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6798
X-OriginatorOrg: intel.com

On Thu, Jun 06, 2024 at 08:55:03AM -0300, Jason Gunthorpe wrote:
> On Thu, Jun 06, 2024 at 10:48:10AM +0800, Yan Zhao wrote:
> > On Sat, Jun 01, 2024 at 04:46:14PM -0300, Jason Gunthorpe wrote:
> > > On Mon, May 27, 2024 at 11:37:34PM -0700, Christoph Hellwig wrote:
> > > > On Tue, May 21, 2024 at 01:00:16PM -0300, Jason Gunthorpe wrote:
> > > > > > > Err, no.  There should really be no exported cache manipulation macros,
> > > > > > > as drivers are almost guaranteed to get this wrong.  I've added
> > > > > > > Russell to the Cc list who has been extremtly vocal about this at least
> > > > > > > for arm.
> > > > > > 
> > > > > > We could possibly move this under some IOMMU core API (ie flush and
> > > > > > map, unmap and flush), the iommu APIs are non-modular so this could
> > > > > > avoid the exported symbol.
> > > > > 
> > > > > Though this would be pretty difficult for unmap as we don't have the
> > > > > pfns in the core code to flush. I don't think we have alot of good
> > > > > options but to make iommufd & VFIO handle this directly as they have
> > > > > the list of pages to flush on the unmap side. Use a namespace?
> > > > 
> > > > Just have a unmap version that also takes a list of PFNs that you'd
> > > > need for non-coherent mappings?
> > > 
> > > VFIO has never supported that so nothing like that exists yet.. This
> > > is sort of the first steps to some very basic support for a
> > > non-coherent cache flush in a limited case of a VM that can do its own
> > > cache flushing through kvm.
> > > 
> > > The pfn list is needed for unpin_user_pages() and it has an ugly
> > > design where vfio/iommufd read back the pfns seperately from unmap,
> > > and they both do it differently without a common range list
> > > datastructure here.
> > > 
> > > So, we'd need to build some new unmap function that returns a pfn list
> > > that it internally fetches via the read ops. Then it can do the read,
> > > unmap, flush iotlb, flush cache in core code.
> > Would the core code flush CPU caches by providing page physical address?
> 
> Physical address is all we will have in the core code..
> 
> > If yes, do you think it's still necessary to export arch_flush_cache_phys()
> > (as what's implemented in this patch)?
> 
> Christoph is asking not to export it, that would mean relying on the
> iommu core to be non-modulare and putting the arch calls there with a
> more restricted exported API - ie based on unmap.

Got it. Thanks for explanation!
> 
> > > I've been working towards this very slowly as I want to push this
> > > stuff down into the io page table walk and remove the significant
> > > inefficiency, so it is not throw away work, but it is certainly some
> > > notable amount of work to do.
> > Will VFIO also be switched to this new unmap interface? Do we need to care
> > about backporting?
> 
> I don't know :)
>  
> > And is it possible for VFIO alone to implement in the current proposed way
> > in this series as the first step for easier backport?
> 
> I think this series is the best option we have right now, but make the
> EXPORT a NS export to try to discourage abuse of it while we continue
> working
Will do. Thanks!