Received-SPF: pass (google.com: domain of linux-kernel+bounces-180865-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Thu, 16 May 2024 17:48:20 +0800
From: Yan Zhao <yan.y.zhao@intel.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
CC: Jason Gunthorpe <jgg@nvidia.com>, "kvm@vger.kernel.org"
	<kvm@vger.kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "x86@kernel.org" <x86@kernel.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>, "pbonzini@redhat.com"
	<pbonzini@redhat.com>, "seanjc@google.com" <seanjc@google.com>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"luto@kernel.org" <luto@kernel.org>, "peterz@infradead.org"
	<peterz@infradead.org>, "tglx@linutronix.de" <tglx@linutronix.de>,
	"mingo@redhat.com" <mingo@redhat.com>, "bp@alien8.de" <bp@alien8.de>,
	"hpa@zytor.com" <hpa@zytor.com>, "corbet@lwn.net" <corbet@lwn.net>,
	"joro@8bytes.org" <joro@8bytes.org>, "will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>, "baolu.lu@linux.intel.com"
	<baolu.lu@linux.intel.com>, "Liu, Yi L" <yi.l.liu@intel.com>
Subject: Re: [PATCH 5/5] iommufd: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <ZkXWZKKgBLB44fkQ@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20240507062212.20535-1-yan.y.zhao@intel.com>
 <20240509141332.GP4650@nvidia.com>
 <Zj3UuHQe4XgdDmDs@yzhao56-desk.sh.intel.com>
 <20240510132928.GS4650@nvidia.com>
 <ZkHEsfaGAXuOFMkq@yzhao56-desk.sh.intel.com>
 <ZkN/F3dGKfGSdf/6@nvidia.com>
 <ZkRe/HeAIgscsYZw@yzhao56-desk.sh.intel.com>
 <ZkUeWAjHuvIhLcFH@nvidia.com>
 <ZkVwS8n7ARzKAbyW@yzhao56-desk.sh.intel.com>
 <BN9PR11MB5276C9F1281F689B32440B558CED2@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5276C9F1281F689B32440B558CED2@BN9PR11MB5276.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?6rN4LFCd91DHFERM9w8Tk0K+/D390wxSFc3EUvFrUb4BaC4Yazu2601H8lRL?=
 =?us-ascii?Q?YPn+MDh5RGe8EjpvSl7KdAhrB5gr9+cl9bZFggi4OAL85C4TZSvtOolEgos1?=
 =?us-ascii?Q?Zve50CLv8Nt+WzfKgsksddIJErlZr5dJE1jVfv+rz10ua6BkNokFA/lliIaZ?=
 =?us-ascii?Q?JqVrsq99NWR2QY9XIQZG6Tipaga6YEQp9qoLKjWotf2tofUVKremWLT/TEUg?=
 =?us-ascii?Q?iwPiQGmfBLs/1n/e400SskJf779QTR+aGsk9ldsnIPT+CTJvok9nYs5NJB07?=
 =?us-ascii?Q?m5kt4TjIMzcawIyMK9JZtEAp+aSe1gWEn+55bmutL+RW7xTo+t72nF7siFQw?=
 =?us-ascii?Q?L9S5uDyQLo75FZE0TtgKDatdOfrDxfzN+DkVQKXjoJRu6ht4ezS7fbehgNOP?=
 =?us-ascii?Q?oF/GRKtN3QgqsajKovug1SEieJ8ZnYlOIjGDZGHiU5z/NyKfGLLd1IP6hRLd?=
 =?us-ascii?Q?oWSRlT3IdJCPSvtD1ZobsIoPK9Wr2gUNWyMjuc6HNtZYcUWdn2zHpIfEdtBP?=
 =?us-ascii?Q?SA0MQoJNjmmgEG/682mmRcHH675+VIG4fN06HQhUle/ikdnF6ckW9GBl4Zan?=
 =?us-ascii?Q?iCpkxFbQGN2IOsyJ/b7f4nsTnHHH6rQ1SGNUHMXPWAc3859ZJ2XUBagFsodA?=
 =?us-ascii?Q?UgLOa9VkHdecld8XLGrPgLzW0kV6zrCNAXOBCbn0U0cGYAIS1RuMHVksIChB?=
 =?us-ascii?Q?ugitoUuA/PJ52u2N4hYn2dbX/Ap1oAJNMHPfIuVXR8FvOMMXwjlc/M3NFs6X?=
 =?us-ascii?Q?FaDhhU/tqJ8LOoBHJOE0pM6IUSQN7aVixhu8qTfgHj3AoqvXhedPYsN5/XRr?=
 =?us-ascii?Q?IUzZHD0pptiGCIdwZ/x6+tXc2JM86KbvdjtF2YYcT7ikSid4mbCe7hrO7YUR?=
 =?us-ascii?Q?95eJXbztGCIjDXqb/ART3lsHvGnH59Mi5PHZhuxcsifPrDcVMrIYMz5MoYaQ?=
 =?us-ascii?Q?DioDCTa/X6oQnxwelScskHfLtebR3B+OEJ7b4YQJWEw6oOnirAe7MTcGgvFQ?=
 =?us-ascii?Q?CUl3b9ywVWj8qjeT75ZpVHB/M1KHn0Ty5G464LzXLwykCaGdHQHhhurByOB/?=
 =?us-ascii?Q?c3AHJJ/ny4kLbwrLO8tdw1qxbBF5OgIo8drmWTaP0RGE+HSm7aqXKxE+SRBe?=
 =?us-ascii?Q?fZsr2UvtwgwlePOINF2ect8vuEZghEB160Bx3rnNZGw/fEq6dGBrVHNEidLP?=
 =?us-ascii?Q?30ffEUccfjjRCUXxURxt4tKXHl8tb5gHY1Vvtn3T6RKHUbfBRfEtsL1ytd0U?=
 =?us-ascii?Q?K/QbkhHxSnsAlAIQDTNVEpvvHIKnTKeHIl1CDwxhgwSOh30j6FAwZhvMT2EH?=
 =?us-ascii?Q?XqqsQWH6ZGmH33QX+w//+K5RXh5tOIaYjC8KgwPGz9tQhkduoMLCmOUz7D0c?=
 =?us-ascii?Q?aNr84884m3c+XMrJBbGugV8LiewtJ1DkIVR1HuwdOuDvTosmr9HMAxFVs/2G?=
 =?us-ascii?Q?hsP/+lrHIM/QzP/CiZz0/8aDdYX95clS6SeGWYkgQL88xozhduFqd7b841PW?=
 =?us-ascii?Q?MSRF+nSI7GH9SRBGdDzM1HRVPNHm5FPA8FAkNyLaC/j5YflV+06zqqzrAR1X?=
 =?us-ascii?Q?xtAGYLJgSx+XOI867Z5iv77qfYQbIG6Nfi/Xt97n?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 78ffb492-64a0-4d31-023f-08dc758d6f31
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2024 09:49:09.7337
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: rJQS63KmS1MNLWfCoWavJLGKkB1X9UJ4RHcTookauOBDVbOhQDGqNIOmsOvnnGtSHQF3AeLeSHnJClVfwLQfgQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB5802
X-OriginatorOrg: intel.com

On Thu, May 16, 2024 at 04:38:12PM +0800, Tian, Kevin wrote:
> > From: Zhao, Yan Y <yan.y.zhao@intel.com>
> > Sent: Thursday, May 16, 2024 10:33 AM
> > 
> > On Wed, May 15, 2024 at 05:43:04PM -0300, Jason Gunthorpe wrote:
> > > On Wed, May 15, 2024 at 03:06:36PM +0800, Yan Zhao wrote:
> > >
> > > > > So it has to be calculated on closer to a page by page basis (really a
> > > > > span by span basis) if flushing of that span is needed based on where
> > > > > the pages came from. Only pages that came from a hwpt that is
> > > > > non-coherent can skip the flushing.
> > > > Is area by area basis also good?
> > > > Isn't an area either not mapped to any domain or mapped into all
> > domains?
> > >
> > > Yes, this is what the span iterator turns into in the background, it
> > > goes area by area to cover things.
> > >
> > > > But, yes, considering the limited number of non-coherent domains, it
> > appears
> > > > more robust and clean to always flush for non-coherent domain in
> > > > iopt_area_fill_domain().
> > > > It eliminates the need to decide whether to retain the area flag during a
> > split.
> > >
> > > And flush for pin user pages, so you basically always flush because
> > > you can't tell where the pages came from.
> > As a summary, do you think it's good to flush in below way?
> > 
> > 1. in iopt_area_fill_domains(), flush before mapping a page into domains
> > when
> >    iopt->noncoherent_domain_cnt > 0, no matter where the page is from.
> >    Record cache_flush_required in pages for unpin.
> > 2. in iopt_area_fill_domain(), pass in hwpt to check domain non-coherency.
> >    flush before mapping a page into a non-coherent domain, no matter where
> > the
> >    page is from.
> >    Record cache_flush_required in pages for unpin.
> > 3. in batch_unpin(), flush if pages->cache_flush_required before
> >    unpin_user_pages.
> 
> so above suggests a sequence similar to vfio_type1 does?
Similar. Except that in iopt_area_fill_domain(), flush is always performed to
non-coherent domains without checking pages->cache_flush_required, while in
vfio_iommu_replay(), flush can be skipped if dma->cache_flush_required is true.

This is because in vfio_type1, pages are mapped into domains in dma-by-dma basis,
but in iommufd, pages are mapped into domains in area-by-area basis.
Two areas are possible to be non-overlapping parts of an iopt_pages.
It's not right to skip flushing of pages in the second area if
pages->cache_flush_required is set to true by mapping pages in the first area.
It's also cumbersome to introduce and check another flag in area or to check
where pages came from before mapping them into a non-coherent domain.