Received-SPF: pass (google.com: domain of linux-kernel+bounces-180781-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: "Zhao, Yan Y" <yan.y.zhao@intel.com>, Jason Gunthorpe <jgg@nvidia.com>
CC: "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>, "alex.williamson@redhat.com"
	<alex.williamson@redhat.com>, "iommu@lists.linux.dev"
	<iommu@lists.linux.dev>, "pbonzini@redhat.com" <pbonzini@redhat.com>,
	"seanjc@google.com" <seanjc@google.com>, "dave.hansen@linux.intel.com"
	<dave.hansen@linux.intel.com>, "luto@kernel.org" <luto@kernel.org>,
	"peterz@infradead.org" <peterz@infradead.org>, "tglx@linutronix.de"
	<tglx@linutronix.de>, "mingo@redhat.com" <mingo@redhat.com>, "bp@alien8.de"
	<bp@alien8.de>, "hpa@zytor.com" <hpa@zytor.com>, "corbet@lwn.net"
	<corbet@lwn.net>, "joro@8bytes.org" <joro@8bytes.org>, "will@kernel.org"
	<will@kernel.org>, "robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>, "Liu, Yi L"
	<yi.l.liu@intel.com>
Subject: RE: [PATCH 5/5] iommufd: Flush CPU caches on DMA pages in
 non-coherent domains
Thread-Topic: [PATCH 5/5] iommufd: Flush CPU caches on DMA pages in
 non-coherent domains
Thread-Index: AQHaoEcDoWZ4XBbpTkCRP2AjSpAxzLGO9ZcAgAEq0wCAAFsyAIAEVmaAgAIPYoCAAQrnAIAA5B4AgABhsYCAAGX4EA==
Date: Thu, 16 May 2024 08:38:12 +0000
Message-ID: <BN9PR11MB5276C9F1281F689B32440B558CED2@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20240507061802.20184-1-yan.y.zhao@intel.com>
 <20240507062212.20535-1-yan.y.zhao@intel.com>
 <20240509141332.GP4650@nvidia.com>
 <Zj3UuHQe4XgdDmDs@yzhao56-desk.sh.intel.com>
 <20240510132928.GS4650@nvidia.com>
 <ZkHEsfaGAXuOFMkq@yzhao56-desk.sh.intel.com> <ZkN/F3dGKfGSdf/6@nvidia.com>
 <ZkRe/HeAIgscsYZw@yzhao56-desk.sh.intel.com> <ZkUeWAjHuvIhLcFH@nvidia.com>
 <ZkVwS8n7ARzKAbyW@yzhao56-desk.sh.intel.com>
In-Reply-To: <ZkVwS8n7ARzKAbyW@yzhao56-desk.sh.intel.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?/pJ50FedzGUidmQWqPl8HCQhk4wBoOLbAcQqmbQzym72MJPiV/BO5tiLVN8/?=
 =?us-ascii?Q?L2ExjQzbOw31+Byjy5zALPxkrIlMmZAp2CLwPoSq5Y32FcNVcyUNCaGooGrw?=
 =?us-ascii?Q?kMLTrjKmu1FfEZQEBv6QdRlQxvPR5cI0Yj6akQWKpK/jxnQ1pv3MX7ftp3yJ?=
 =?us-ascii?Q?YT/DWoOwVkubY8lohJSgm9azNsJHItL7uZDWZZAlb+yCpX6ds6eVIvCe49jv?=
 =?us-ascii?Q?g5bdOT1914nbVhW0gI4EUoHjGYOXYV4mBRJV3We1szMFIfGl9Ckyr8TXJyP4?=
 =?us-ascii?Q?ag0Rf6RizyX2NDrBVakjVjPYMPiID0XGjctt3OI7vb170gM2MiC1iQSkYnma?=
 =?us-ascii?Q?TOQusdlj8uZZfhaKBcSM8MbeatJDQGId/3MmSxL2aZw34e0HZhgrsxHCT5gx?=
 =?us-ascii?Q?gTGT40gm8knhXBEl2R5R25J/B8cUS4BG+3RDFt/Y5uQbfyaXnT2DXRjG50yC?=
 =?us-ascii?Q?W2CDy8ShPVv0NTSXH9cRvfTBT+jtEh+DTbUfy5xDvj6lZNXv70zGBei3WC4u?=
 =?us-ascii?Q?+W8otC0g1NvAk1w9iDEl5UZDx61DK1sfLLrbejBbNTKptjLXK2ecmXgYwcuV?=
 =?us-ascii?Q?xfQOwsh3fWzX+d6KuiBxebWNMRUxtiPwInjKmXMztvP09jQiqMS6OXC6Pt0s?=
 =?us-ascii?Q?EXW4S/ZlbPdv7K8HR+E5GXvN2nR64nUtLEXu1zHbpQtNG/6b6MmkBPAQVHOJ?=
 =?us-ascii?Q?+FClurw04hdkfy1gCdDt7vFakkpSYdkwTfm0Irm/bCObQRlo/9jXlLyPA4lj?=
 =?us-ascii?Q?MJoO5szS1XhW8x6hI5dKlzOen3D1g1QNCa3GoED/1uwdjBAjJvCFPrfT4J7K?=
 =?us-ascii?Q?PTD+Nsvj9H/6C8z1T3gJXYjO7HBO6NV46MO2zlFrfF9F0C8/YdeGocGV5hvT?=
 =?us-ascii?Q?BYUGJTug96o6aQqjuFIRNvBkHLxhQLWppxS2clwiwbw14qPQ7rI9RLFkRgQ3?=
 =?us-ascii?Q?aA1Odq1wX7DAASvJZQosFE+JOf6PQW+AWpudN7Rt2whBPm4R65BXr/Gf9Dez?=
 =?us-ascii?Q?njRv8r19QBzJWboqj3yp3wXUTVxKaVM+6IP1uHH9v7bPVZY/EEQ4nos0ZYln?=
 =?us-ascii?Q?2n3RZxFaN9y5WynEFnjVRqaUQdqlIcIs792EcrgmlEMQsMzK9pArsUHGi1Em?=
 =?us-ascii?Q?Dd5FWxLJijJ9j7UhEwUeZumbRTIrBq9n66AaiDAepOkyUaxOBT5IZkOScQoA?=
 =?us-ascii?Q?IB3FTLkZXdTpWtt+59yxhWdy7I5Yu06r2oXcrCUcszsEjJM92HXCzYWVNNqN?=
 =?us-ascii?Q?MIE2OHAN1CVYRAOO969nX4VwUKS3+rLn0NpHSi3OSj0ZZxmDQMKfaylHwmnm?=
 =?us-ascii?Q?Tzv+D0bls0etRBS6BeNNU+ewJaC5Qf7ItakrK7q4H6UPe00hb+5ElUCmx6AS?=
 =?us-ascii?Q?ZkQKDeG4p7deY7ah9QTNmKJenRJpWYOLr4OCJTU18f44q46e36aRS4CUPi30?=
 =?us-ascii?Q?SYWAUXzbq3VYlxEEskYHtMkYXWU/sxA7N80vgLXu12bq1ML61IK0BgHa1BX9?=
 =?us-ascii?Q?hNVlkM+wcVRuCYgkY9eHmIjX6kig0ZrrZpO6kcIbBUOk3vhevxFw7zbkirdH?=
 =?us-ascii?Q?t0j04+MgYZ+CpsV5SXiCnB4JF40DvIGDwkcGR5Zp?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d0d79b2d-ff88-471a-013c-08dc7583858c
X-MS-Exchange-CrossTenant-originalarrivaltime: 16 May 2024 08:38:12.1436
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: ul+BgK+4NTI/W//Y/WMuHoGmRNIYoDMn5/7WcMIPHmDH0kxxnOZz/frQY2rs/CUyn+8xzIlA7YCCqFdKdWrCiQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB7417
X-OriginatorOrg: intel.com

> From: Zhao, Yan Y <yan.y.zhao@intel.com>
> Sent: Thursday, May 16, 2024 10:33 AM
>=20
> On Wed, May 15, 2024 at 05:43:04PM -0300, Jason Gunthorpe wrote:
> > On Wed, May 15, 2024 at 03:06:36PM +0800, Yan Zhao wrote:
> >
> > > > So it has to be calculated on closer to a page by page basis (reall=
y a
> > > > span by span basis) if flushing of that span is needed based on whe=
re
> > > > the pages came from. Only pages that came from a hwpt that is
> > > > non-coherent can skip the flushing.
> > > Is area by area basis also good?
> > > Isn't an area either not mapped to any domain or mapped into all
> domains?
> >
> > Yes, this is what the span iterator turns into in the background, it
> > goes area by area to cover things.
> >
> > > But, yes, considering the limited number of non-coherent domains, it
> appears
> > > more robust and clean to always flush for non-coherent domain in
> > > iopt_area_fill_domain().
> > > It eliminates the need to decide whether to retain the area flag duri=
ng a
> split.
> >
> > And flush for pin user pages, so you basically always flush because
> > you can't tell where the pages came from.
> As a summary, do you think it's good to flush in below way?
>=20
> 1. in iopt_area_fill_domains(), flush before mapping a page into domains
> when
>    iopt->noncoherent_domain_cnt > 0, no matter where the page is from.
>    Record cache_flush_required in pages for unpin.
> 2. in iopt_area_fill_domain(), pass in hwpt to check domain non-coherency=
.
>    flush before mapping a page into a non-coherent domain, no matter wher=
e
> the
>    page is from.
>    Record cache_flush_required in pages for unpin.
> 3. in batch_unpin(), flush if pages->cache_flush_required before
>    unpin_user_pages.

so above suggests a sequence similar to vfio_type1 does?