Received-SPF: pass (google.com: domain of linux-kernel+bounces-183365-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Mon, 20 May 2024 10:45:56 +0800
From: Yan Zhao <yan.y.zhao@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
CC: <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <x86@kernel.org>,
	<alex.williamson@redhat.com>, <kevin.tian@intel.com>,
	<iommu@lists.linux.dev>, <pbonzini@redhat.com>, <seanjc@google.com>,
	<dave.hansen@linux.intel.com>, <luto@kernel.org>, <peterz@infradead.org>,
	<tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>, <hpa@zytor.com>,
	<corbet@lwn.net>, <joro@8bytes.org>, <will@kernel.org>,
	<robin.murphy@arm.com>, <baolu.lu@linux.intel.com>, <yi.l.liu@intel.com>
Subject: Re: [PATCH 5/5] iommufd: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <Zkq5ZL+saJbEkfBQ@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20240507062212.20535-1-yan.y.zhao@intel.com>
 <20240509141332.GP4650@nvidia.com>
 <Zj3UuHQe4XgdDmDs@yzhao56-desk.sh.intel.com>
 <20240510132928.GS4650@nvidia.com>
 <ZkHEsfaGAXuOFMkq@yzhao56-desk.sh.intel.com>
 <ZkN/F3dGKfGSdf/6@nvidia.com>
 <ZkRe/HeAIgscsYZw@yzhao56-desk.sh.intel.com>
 <ZkUeWAjHuvIhLcFH@nvidia.com>
 <ZkVwS8n7ARzKAbyW@yzhao56-desk.sh.intel.com>
 <20240517170418.GA20229@nvidia.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240517170418.GA20229@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?VWvVRxiNqO9o0ZJ1CkYi8vPUOSZNsPKVSIjdMsKuN18y1GIYqouJdJDNiZ1b?=
 =?us-ascii?Q?XUJWeZO55jqExMXWmxkSQ+puDz6Li+1IOWCO8s8MCH5atd8SY+BKTZPmtl7W?=
 =?us-ascii?Q?AfAuyEBK7ombhRGX40qBayoZtf1BpftBFKsS1wH719zGPReP8m8uCx0Uj+Ke?=
 =?us-ascii?Q?rFQa9IkgNE9Eh5VfZ0vZmHEEMYFr5w5ZjlZpHiQTD6ohQ2ICcEY0wYg4iqXO?=
 =?us-ascii?Q?sa9auU3RMwW50YkL2ErvarOiOsEe6z3bMIj8y8jY91iFW7y9vp29K/WYwvMj?=
 =?us-ascii?Q?sjg8ctCbBdexjfwq+YbBC0+naBWnj8eHjBr5+v0LGRgvA4NkcbClvnsVHxPB?=
 =?us-ascii?Q?DcnAgur3xn/MaedVVszmdWnaVfAtOkgaQQzfRVEq8CXqgbBpQ5Y7cHhVmyXw?=
 =?us-ascii?Q?3bVeBxmJk4/1+Ki52fp1RPct1D7QmHdfiBhkqjhCPgkx5v1sK7VH7lH0Qbla?=
 =?us-ascii?Q?yl5u2/lHPjVzjRpC8eQSoMznl8gUHPAn89okqm7F+1R9aPtaLNZZRuBDArh7?=
 =?us-ascii?Q?jKKbkLphnA+ng5kOPSJqe30yE73ra1PUxQg33gWCtkHZjedAzZ9UOpu8AyZp?=
 =?us-ascii?Q?pAgdk4Fsxz0Escofc3f1o0+/BS+zl4CU8tp5HvYzDerwghIFYdKX8L5DpFWt?=
 =?us-ascii?Q?rrMP1JdIyAxEN0kgFKdxlrJiyTb3ajkZlc+Irk3YBpyasVF3T/KwXOTDm69v?=
 =?us-ascii?Q?yh/+q3t0obI9O6YgMeVpqtpnW0eFZjJlOinF5O3sgi/F9HuX9zDgdkz9BAQT?=
 =?us-ascii?Q?TeDLtWlr2epCCWqb2ZXOhgsbVCwaP2uKkSlwdtxUQd9EtsGyp5SbZflvkY/z?=
 =?us-ascii?Q?Rgcsuh/3yRNLHGWgT1EHt1NwnBPH8Q6/G2xmz+C6hMVJDMqAQhTIUvHFA2EJ?=
 =?us-ascii?Q?WoWSAbpuqPyoR7wDIOliXAVEp5EgMRV9OtK8/rCGDI2FjB0ItDo3Nb5WNZOn?=
 =?us-ascii?Q?piH2vKO44IxB0B3R1rpMXDlMFG7Pj1vS26JEVZ3UIFiv+06bdQS/dW2l6UtJ?=
 =?us-ascii?Q?l0QO8Oam8D9Aro3gFGs0jTSP0XtIdI4lmHk8VwMCkPwZ5Q1TTMvEPIHH4j0q?=
 =?us-ascii?Q?XC1kO66VkndOBo0lig8m/Mc07BWxR7UCktTG4HRCi+TZrOPlUVfDpnz9gg8I?=
 =?us-ascii?Q?bQlkh5/qdrmorxdvkneRnamzywVRf62zInzewXQrvWhsCqG0i249wjuZB5JQ?=
 =?us-ascii?Q?keCRjQrc6LAAWJjkzVvXJWhY5t4M1G24HgL000uqArx20SO8Ko2ODjd9el4i?=
 =?us-ascii?Q?6jutnniKSV6S32Ez57t5iPYOhWFiTQ7337/AtL7L6EoVAHyRg5Num+Rgq0J0?=
 =?us-ascii?Q?TPEYb21+VKVfTGeIXGFJCz2dq1SnZGpP9I1P+DKZStBiHHNLFZIMiLzxJeXL?=
 =?us-ascii?Q?1nEsrl6AJ8zjn+rw8A0xnXx/NWSR6KYOlcMCyME0j0ZvufMiemlEtuSYqZ4I?=
 =?us-ascii?Q?eIHgF+3YJFgB2P1ZWRyaSwt7ZP4ZZYiXqmbJz0PH+4tc4Ljox/Xq6a+S1w9t?=
 =?us-ascii?Q?Q+S45T5Q8i27AQ+TcM9ey7IX7C/EnxqOd4PkdedQKgqcCXGSxVwYhQ/ooM27?=
 =?us-ascii?Q?/D6UjhEObYOXKbwv/RHBvG1UsZsWOYWS6WIwXS/S?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7b663876-1d9a-491a-fa9e-08dc78771870
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 May 2024 02:46:48.8236
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 9658j8Lxw/887nXu7YuSExchcTYNj8Hk0T/FjWRv3ng5J9tnCVNdPKvdfcu7uES867tWzA2yIQbAmcxdayhXJQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7197
X-OriginatorOrg: intel.com

On Fri, May 17, 2024 at 02:04:18PM -0300, Jason Gunthorpe wrote:
> On Thu, May 16, 2024 at 10:32:43AM +0800, Yan Zhao wrote:
> > On Wed, May 15, 2024 at 05:43:04PM -0300, Jason Gunthorpe wrote:
> > > On Wed, May 15, 2024 at 03:06:36PM +0800, Yan Zhao wrote:
> > > 
> > > > > So it has to be calculated on closer to a page by page basis (really a
> > > > > span by span basis) if flushing of that span is needed based on where
> > > > > the pages came from. Only pages that came from a hwpt that is
> > > > > non-coherent can skip the flushing.
> > > > Is area by area basis also good?
> > > > Isn't an area either not mapped to any domain or mapped into all domains?
> > > 
> > > Yes, this is what the span iterator turns into in the background, it
> > > goes area by area to cover things.
> > > 
> > > > But, yes, considering the limited number of non-coherent domains, it appears
> > > > more robust and clean to always flush for non-coherent domain in
> > > > iopt_area_fill_domain().
> > > > It eliminates the need to decide whether to retain the area flag during a split.
> > > 
> > > And flush for pin user pages, so you basically always flush because
> > > you can't tell where the pages came from.
> > As a summary, do you think it's good to flush in below way?
> > 
> > 1. in iopt_area_fill_domains(), flush before mapping a page into domains when
> >    iopt->noncoherent_domain_cnt > 0, no matter where the page is from.
> >    Record cache_flush_required in pages for unpin.
> > 2. in iopt_area_fill_domain(), pass in hwpt to check domain non-coherency.
> >    flush before mapping a page into a non-coherent domain, no matter where the
> >    page is from.
> >    Record cache_flush_required in pages for unpin.
> > 3. in batch_unpin(), flush if pages->cache_flush_required before
> >    unpin_user_pages.
> 
> It does not quite sound right, there should be no tracking in the
> pages of this stuff.
What's the downside of having tracking in the pages?

Lazily flush pages right before unpin pages is not only to save flush count
for performance, but also for some real problem we encountered. see below.

> 
> If pfn_reader_fill_span() does batch_from_domain() and
> the source domain's storage_domain is non-coherent then you can skip
> the flush. This is not pedantically perfect in skipping all flushes, but
> in practice it is probably good enough.
We don't know whether the source storage_domain is non-coherent since
area->storage_domain is of "struct iommu_domain".

Do you want to add a flag in "area", e.g. area->storage_domain_is_noncoherent,
and set this flag along side setting storage_domain?
(But looks this is not easy in iopt_area_fill_domains() as we don't have hwpt
there.)

> __iopt_area_unfill_domain() (and children) must flush after
> iopt_area_unmap_domain_range() if the area's domain is
> non-coherent. This is also not perfect, but probably good enough.
Do you mean flush after each iopt_area_unmap_domain_range() if the domain is
non-coherent?
The problem is that iopt_area_unmap_domain_range() knows only IOVA, the
IOVA->PFN relationship is not available without iommu_iova_to_phys() and
iommu_domain contains no coherency info.
Besides, when the non-coherent domain is a storage domain, we still need to do
the flush in batch_unpin(), right?
Then, with a more complex case, if the non-coherent domain is a storage domain,
and if some pages are still held in pages->access_itree when unfilling the
domain, should we get PFNs from pages->pinned_pfns and do the flush in
__iopt_area_unfill_domain()?
> 
> Doing better in both cases would require inspecting the areas under
> the used span to see what is there. This is not so easy.
My feeling is that checking non-coherency of target domain and save non-coherency
in pages might be the easiest way with least code change.