Received-SPF: pass (google.com: domain of linux-kernel+bounces-197994-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Sat, 1 Jun 2024 16:46:14 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Christoph Hellwig <hch@infradead.org>
Cc: Yan Zhao <yan.y.zhao@intel.com>, kvm@vger.kernel.org,
	linux-kernel@vger.kernel.org, x86@kernel.org,
	alex.williamson@redhat.com, kevin.tian@intel.com,
	iommu@lists.linux.dev, pbonzini@redhat.com, seanjc@google.com,
	dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org,
	tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com,
	corbet@lwn.net, joro@8bytes.org, will@kernel.org,
	robin.murphy@arm.com, baolu.lu@linux.intel.com, yi.l.liu@intel.com,
	Russell King <linux@armlinux.org.uk>
Subject: Re: [PATCH 3/5] x86/mm: Introduce and export interface
 arch_clean_nonsnoop_dma()
Message-ID: <Zlt6huNJeW8ekJlE@nvidia.com>
References: <20240507061802.20184-1-yan.y.zhao@intel.com>
 <20240507062044.20399-1-yan.y.zhao@intel.com>
 <ZktZDmcNnsHhp4Tm@infradead.org>
 <20240521154939.GH20229@nvidia.com>
 <20240521160016.GA2513156@nvidia.com>
 <ZlV7rlmWdU7dJZKo@infradead.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZlV7rlmWdU7dJZKo@infradead.org>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?P/8kWTuwQoFriPEkjQniHEJvlQK4LkaKSrvr0dXh7qUflSsTges7Gtu4xHad?=
 =?us-ascii?Q?rcldTNVwvsJfG2FAfEldhV7AKUdmymkP+5WXRfKAb8JwfJSSxnqkBs5G4+fg?=
 =?us-ascii?Q?Cwgd0jMCNOcW1TAxU0xr3L9TtDH0ZjqhE+q6KiGZbVZ4p3rDm9GGEVR/2hXy?=
 =?us-ascii?Q?8LYsujT90my5IxMqhx+jI0krk3ZXabBW5goiX0z+JrcJbOqDST4EzwK8EjNo?=
 =?us-ascii?Q?Y6AAUCVccMJjvGN75ZonHjf8hEzKbfMJq07LD/ZnNvkUN2h1M530RCP2WEIx?=
 =?us-ascii?Q?qYP6U3GIsl3UecpNiCyW/YPb4N2LYwSgnRA7dsAuX0TM6GJtjHyUc9EClklh?=
 =?us-ascii?Q?e9t4+Js3W6Z0qndNuD2gZ2ggh+a/HkKbPKlOwMSl7voCXgKfxa8jjv4uEoN1?=
 =?us-ascii?Q?Z250v/DlyqRVZr7uKXuSWlPxh84MlTsB7jQyRk+8EOl1eJBXW6jxuMjp8/i8?=
 =?us-ascii?Q?mj7i+0cG06ZDAsUMTNNC8gnRjqIjE88IxcMMcnsqNYh+N3TvJoG6E1UVRJ4A?=
 =?us-ascii?Q?2SmBMblWUQWNFkh6xzCj9LA17vcRiWDWoC5F/fNmBa8P3MDrzqw7Ie/iN0LD?=
 =?us-ascii?Q?Znf7295ficAcRmR9x9EHjIo9HsaJ18cIu7DzsCrVfAhKhmIQkDx+1Udf5nW5?=
 =?us-ascii?Q?/eng2pWRcK/QyLXLE1qprMYAJJmIIQ725aOQ6PGhHyeBdV4dmvNjR2g74kqB?=
 =?us-ascii?Q?xJ5kjSEP7gWuvDzdp36RusPwn+rSisYrmro3CgjLrk0TqKmkdr5tovL1PsPj?=
 =?us-ascii?Q?0oRIjLbPDPTNa45kqhOH0ZHUYZ0ipUoCaDxtFWC7ATXe3U5GamgRHrffjYJs?=
 =?us-ascii?Q?pJ26jyNKrekd/h6PBCTjUdKqovXbH7gEzQ6AcI49lScMigus1fbwF3XagNh+?=
 =?us-ascii?Q?XE69wxjvpCLsVtdIekH6s+ZER3+YOq+e4f4bEctBFaqq2k+nwOaL1im5BgvA?=
 =?us-ascii?Q?pEMWmbPtsmvtA0Yo+GZTaJVvmh6SUkKNmSXMi7F904jf+UVHAXXRslEBL86Y?=
 =?us-ascii?Q?HLL5rlYrLoZwN93IEMAvn9+JyILkUCgPi7jkO7QOdHCxXfv5sMQ+9PFmDhDE?=
 =?us-ascii?Q?8TpkFVlCbPUeFXTEvE3FhhEdAMshwi0UNNwnUeGiTxVnRCYzPqijhCj1jiT+?=
 =?us-ascii?Q?Xi5GaHRoHvGVlgUNF4udpT4HDl6CPZq3+LY+FMCzMAs6pO82dg2NvbIlF0V5?=
 =?us-ascii?Q?+QE2X7Q7Thl7TxrV7RNlqZfUpgfhRmq87JoynUE/IUgGIO7S1HDfMaXh+V2L?=
 =?us-ascii?Q?CGIATwmAqsNVTmNWnpbI+00o0I0AUO0xMBbMQEUvbEfMWFWZ1PH1tL00UgZx?=
 =?us-ascii?Q?nhJGEDAUoZSJbA/IqKHeADHmRV8+1b8z/3WnNOTbs+RmJzZ0reexcik6tGPp?=
 =?us-ascii?Q?VmFsI6zKNvGXVDRebKIzvLTUVE4LHQ0SBBlSsJWEzQQBnIEzYmVX5R7pR3U+?=
 =?us-ascii?Q?+ZlcHI2SWNDAQsQNq9mpFMiDX5CZ9fZ+E4DTmL4jKtWHoo7wbBuiz70oRfCl?=
 =?us-ascii?Q?PteBQK4TnCM0GbbSqIw4if4ixsVP+eDh5VHDAavkM2O7NQWkhQHnvoJQB4Ui?=
 =?us-ascii?Q?/ZULHwxonSKl1qNPgygrAAz2RVUJwOtkN4I2YUoz?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 76ba1885-f7f3-481e-d511-08dc82753833
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jun 2024 19:58:34.6771
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: qjdotqbYU/h1WmL8nfzHLxwp8opmcEYxUqfBTKrJbvGeFYFNxKhZZ1Mly3xgoMZ+
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR12MB6451

On Mon, May 27, 2024 at 11:37:34PM -0700, Christoph Hellwig wrote:
> On Tue, May 21, 2024 at 01:00:16PM -0300, Jason Gunthorpe wrote:
> > > > Err, no.  There should really be no exported cache manipulation macros,
> > > > as drivers are almost guaranteed to get this wrong.  I've added
> > > > Russell to the Cc list who has been extremtly vocal about this at least
> > > > for arm.
> > > 
> > > We could possibly move this under some IOMMU core API (ie flush and
> > > map, unmap and flush), the iommu APIs are non-modular so this could
> > > avoid the exported symbol.
> > 
> > Though this would be pretty difficult for unmap as we don't have the
> > pfns in the core code to flush. I don't think we have alot of good
> > options but to make iommufd & VFIO handle this directly as they have
> > the list of pages to flush on the unmap side. Use a namespace?
> 
> Just have a unmap version that also takes a list of PFNs that you'd
> need for non-coherent mappings?

VFIO has never supported that so nothing like that exists yet.. This
is sort of the first steps to some very basic support for a
non-coherent cache flush in a limited case of a VM that can do its own
cache flushing through kvm.

The pfn list is needed for unpin_user_pages() and it has an ugly
design where vfio/iommufd read back the pfns seperately from unmap,
and they both do it differently without a common range list
datastructure here.

So, we'd need to build some new unmap function that returns a pfn list
that it internally fetches via the read ops. Then it can do the read,
unmap, flush iotlb, flush cache in core code.

I've been working towards this very slowly as I want to push this
stuff down into the io page table walk and remove the significant
inefficiency, so it is not throw away work, but it is certainly some
notable amount of work to do.

Jason