Received-SPF: pass (google.com: domain of linux-kernel+bounces-203548-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Thu, 6 Jun 2024 10:48:10 +0800
From: Yan Zhao <yan.y.zhao@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
CC: Christoph Hellwig <hch@infradead.org>, <kvm@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <x86@kernel.org>,
	<alex.williamson@redhat.com>, <kevin.tian@intel.com>,
	<iommu@lists.linux.dev>, <pbonzini@redhat.com>, <seanjc@google.com>,
	<dave.hansen@linux.intel.com>, <luto@kernel.org>, <peterz@infradead.org>,
	<tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>, <hpa@zytor.com>,
	<corbet@lwn.net>, <joro@8bytes.org>, <will@kernel.org>,
	<robin.murphy@arm.com>, <baolu.lu@linux.intel.com>, <yi.l.liu@intel.com>,
	Russell King <linux@armlinux.org.uk>
Subject: Re: [PATCH 3/5] x86/mm: Introduce and export interface
 arch_clean_nonsnoop_dma()
Message-ID: <ZmEjavnYePBLDbrS@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20240507061802.20184-1-yan.y.zhao@intel.com>
 <20240507062044.20399-1-yan.y.zhao@intel.com>
 <ZktZDmcNnsHhp4Tm@infradead.org>
 <20240521154939.GH20229@nvidia.com>
 <20240521160016.GA2513156@nvidia.com>
 <ZlV7rlmWdU7dJZKo@infradead.org>
 <Zlt6huNJeW8ekJlE@nvidia.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Zlt6huNJeW8ekJlE@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ON5og7HHBy8IN3KXQKPHFzf0JCIO5DPjH55sjeA8WLjqV0s72TVDDO+7sJfl?=
 =?us-ascii?Q?bwfpiE+3C92uz6dMJ9iwFSbLSRowyADEFfVX3dgrrGX5Dsxi+PDhBwq73GAQ?=
 =?us-ascii?Q?7r3WykIzIoXOX/YNqKm8cahQ1yiAPVPaG8jh0u0gMCXfu2xRYaYN/IV7h1cr?=
 =?us-ascii?Q?1J157YGokMTizeANa/GDaw6CuXRWjF6W3S96pEMJ+vqoEg3ORsLkLC899783?=
 =?us-ascii?Q?5FdxQBSCV3LsxnfPd3hHlw7m/KDZswLKb/BjDC82rPfVwTQNtR7ekhU4Sjmj?=
 =?us-ascii?Q?MuPar/MnLeCMKDNZ2TefQ879pE3HqGoDtO4w/xueUDGpQyiOfCe/go97XdI+?=
 =?us-ascii?Q?hELZ0+HGJVr5YWuFT1oXYb+vUSmcamlK5C9F/A7g8y/c+wpDhdrNguKw+qnl?=
 =?us-ascii?Q?q3ouZKIkTKOTeigRJWuZvJNw1GMcmAiLDmHjqpqOWlXOjLmBcPEXhsYbxVjy?=
 =?us-ascii?Q?8an0fpJAtMrUB5x6I6HdyZqBjUflNqYaJI0EqmXsXfm7CB2ad8FU2IR9ABGM?=
 =?us-ascii?Q?qdBecv3T/NBLIoy/yRUGpY3XZVGgEjuzypYeUi0xyUKyZny1JBslkwKLAJeP?=
 =?us-ascii?Q?J90Q9IUxp1yjZXmiSlwR1Kf4TYlKkyJEfaWfDIrLG5LqJ7R6BxGbBd7MMZaG?=
 =?us-ascii?Q?6x3TJ6zLM+fHbz6j/khepctGTDYXpzOF+/EIN6WD7ds6GTt9CR9EQDO8cu07?=
 =?us-ascii?Q?n4MpZL8RpH4VP4Dydz+nQHSh+cHy1AClTB2X+70cGkTxqcoD/ccAGMlxr9hH?=
 =?us-ascii?Q?VpAZRTKuB49F+lSxM4h/YqWHiwxv3j5X8BTNedYEs1YJDgVEmT64VJNe0xGn?=
 =?us-ascii?Q?JTqwDzLbMpDnHTNmdl1EshgXUjlJGGeNpWx75P9pkaC5fjErGTw/boHmmxWP?=
 =?us-ascii?Q?EHmuEP7eFYbwgLkURhpe7gC35/znTGiV5LVF1m848pmknxWa4k0lt5TtBjev?=
 =?us-ascii?Q?l6XhkITGkeTFCNCi0TF8bGRjXAZYBOnjhRjoLmQSX/qvIkazmdg2gGrzerEd?=
 =?us-ascii?Q?bDnyJ0UFQ6NCcNK58TkV/daVTYKHscR18e6f3iwWXBUEmHH3f153syw3KSDl?=
 =?us-ascii?Q?EkgESmbpZTm+Iql7LOSuan0ZIe4YaTLJZcsqw+8sKzqOvdwNCU/GIkj9sTnN?=
 =?us-ascii?Q?Ap7VEcy7xhpkEb5tXXLuuFRbjSDJ7EC0TgkxJf6FgCDxqBIZB2SoVw2jc0S4?=
 =?us-ascii?Q?qKFg+Jg/jJoNsJ0d/JMrCvAHTAPYyDjCoJzFnvN6pcgeAmsR6sIAN2zTd8yH?=
 =?us-ascii?Q?oO1ZcMlvt1xARjAcGzXzat6ih3sX9HAFKomnucaWxFuUIC6JW2j9tZ6tcEFP?=
 =?us-ascii?Q?Dskm1gypRTb21fj3Ur7kok7qG6L0bzxEZwrHdCfZzUG1D2Y8Cahf0XW2M8FP?=
 =?us-ascii?Q?VxCkAhN1HeHxkNQxgDbBqAvypNAFSMkxqT6Kz6PtLPYo0NaZiLsgcBl3EBQx?=
 =?us-ascii?Q?4u66KC1g+9LlAibxdgqZVj6WTEv7bE8a2qkGlolDOj15wKbuRZWRwbnNWlnA?=
 =?us-ascii?Q?96yJmtUWvyXZyYa1Ves2BcnLuuP1qyYRJQu/NMxH9Ip1R0YU3UQezFsICVUD?=
 =?us-ascii?Q?RL5CloTerL/c+w5YV9uPdAfwpxUmBPPGJR09Jhs3?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 24f367d5-d876-4e75-9c4f-08dc85d34004
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2024 02:49:14.0045
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: KyaPunXM0qNRzdEubVSH+HXi1JpB5xVl0XIYH57gwWPFe2xeN4cO5vXzTc5B6XwZ2yOkU+y793NqePT4x0UIaw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB4598
X-OriginatorOrg: intel.com

On Sat, Jun 01, 2024 at 04:46:14PM -0300, Jason Gunthorpe wrote:
> On Mon, May 27, 2024 at 11:37:34PM -0700, Christoph Hellwig wrote:
> > On Tue, May 21, 2024 at 01:00:16PM -0300, Jason Gunthorpe wrote:
> > > > > Err, no.  There should really be no exported cache manipulation macros,
> > > > > as drivers are almost guaranteed to get this wrong.  I've added
> > > > > Russell to the Cc list who has been extremtly vocal about this at least
> > > > > for arm.
> > > > 
> > > > We could possibly move this under some IOMMU core API (ie flush and
> > > > map, unmap and flush), the iommu APIs are non-modular so this could
> > > > avoid the exported symbol.
> > > 
> > > Though this would be pretty difficult for unmap as we don't have the
> > > pfns in the core code to flush. I don't think we have alot of good
> > > options but to make iommufd & VFIO handle this directly as they have
> > > the list of pages to flush on the unmap side. Use a namespace?
> > 
> > Just have a unmap version that also takes a list of PFNs that you'd
> > need for non-coherent mappings?
> 
> VFIO has never supported that so nothing like that exists yet.. This
> is sort of the first steps to some very basic support for a
> non-coherent cache flush in a limited case of a VM that can do its own
> cache flushing through kvm.
> 
> The pfn list is needed for unpin_user_pages() and it has an ugly
> design where vfio/iommufd read back the pfns seperately from unmap,
> and they both do it differently without a common range list
> datastructure here.
> 
> So, we'd need to build some new unmap function that returns a pfn list
> that it internally fetches via the read ops. Then it can do the read,
> unmap, flush iotlb, flush cache in core code.
Would the core code flush CPU caches by providing page physical address?
If yes, do you think it's still necessary to export arch_flush_cache_phys()
(as what's implemented in this patch)?

> 
> I've been working towards this very slowly as I want to push this
> stuff down into the io page table walk and remove the significant
> inefficiency, so it is not throw away work, but it is certainly some
> notable amount of work to do.
Will VFIO also be switched to this new unmap interface? Do we need to care
about backporting?

And is it possible for VFIO alone to implement in the current proposed way
in this series as the first step for easier backport?