Received-SPF: pass (google.com: domain of linux-kernel+bounces-180573-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Thu, 16 May 2024 10:32:43 +0800
From: Yan Zhao <yan.y.zhao@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
CC: <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <x86@kernel.org>,
	<alex.williamson@redhat.com>, <kevin.tian@intel.com>,
	<iommu@lists.linux.dev>, <pbonzini@redhat.com>, <seanjc@google.com>,
	<dave.hansen@linux.intel.com>, <luto@kernel.org>, <peterz@infradead.org>,
	<tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>, <hpa@zytor.com>,
	<corbet@lwn.net>, <joro@8bytes.org>, <will@kernel.org>,
	<robin.murphy@arm.com>, <baolu.lu@linux.intel.com>, <yi.l.liu@intel.com>
Subject: Re: [PATCH 5/5] iommufd: Flush CPU caches on DMA pages in
 non-coherent domains
Message-ID: <ZkVwS8n7ARzKAbyW@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20240507061802.20184-1-yan.y.zhao@intel.com>
 <20240507062212.20535-1-yan.y.zhao@intel.com>
 <20240509141332.GP4650@nvidia.com>
 <Zj3UuHQe4XgdDmDs@yzhao56-desk.sh.intel.com>
 <20240510132928.GS4650@nvidia.com>
 <ZkHEsfaGAXuOFMkq@yzhao56-desk.sh.intel.com>
 <ZkN/F3dGKfGSdf/6@nvidia.com>
 <ZkRe/HeAIgscsYZw@yzhao56-desk.sh.intel.com>
 <ZkUeWAjHuvIhLcFH@nvidia.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <ZkUeWAjHuvIhLcFH@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?fsa3QRi1risIFIl9XyhC0DUhTy5ZS6beHbXP36J9y6oWv0nEq8CfaVhwEcU5?=
 =?us-ascii?Q?KI/e39NPiOeGGKUQGRXuH2s1Yi8w0UVBZ06Gwn6UF3m79FLx4H93I2AfAFyr?=
 =?us-ascii?Q?YE4IU/CnH6b9RyO8i2cljOiCuPtKxm3wwVtPuzSvdqRTW1cddwh5+pevINPF?=
 =?us-ascii?Q?zkgBWRAcMkEov3VZz7MfkqozJYmpFCgdgg80fHizvp9XucQ4f8WS7hCKzmXT?=
 =?us-ascii?Q?unpealawXfhV5EVJscpyJD8F+mXN03T8VxNNETviOb3WLrChn01laf1eyy2f?=
 =?us-ascii?Q?mt10aseHwLLkEPXMbgkfhhJsy9B4GsL8U0Zo0+Pt1ZwKkMbgWBCGclUnjbfF?=
 =?us-ascii?Q?3fbzB1JyYtzV+NQ5W3tN+l1JJTq7oL/a7EReEopH6x98ID36FpUIAqNgtkw8?=
 =?us-ascii?Q?0RwQWWj/wlUkQGBMpE/CSteFG5ZsiGAoYw6sR2EwU/Z+pg7sW6e6WA4X35bG?=
 =?us-ascii?Q?ler9WOM4SQCJyLt9Iz1Wg2uiVhRK6ElM6BRCiY7vkWIzq9sijVr6btD2mBZd?=
 =?us-ascii?Q?gHPRiJ2nV4vswPSKp/W2ISH6HW9ww9vBkCuf/5cBHI61WJEfw7CQF9sdTP9V?=
 =?us-ascii?Q?QtHVYf9HlrBLrVk/lCfdZ54vinZPibnstYv7z6GrmERvl+W0nkWd5BFgbbXH?=
 =?us-ascii?Q?0o+p6IkX/yGRzINuqgNYGcLjHjmCDg3mygkrA/TJSbjPpvnepFxn7sqTZfNP?=
 =?us-ascii?Q?ZfVnQHQy17+DP7ay7mg7yOHKc5M8sSN2BJi1JQmVeVwXgLMfjMvryZnde/3I?=
 =?us-ascii?Q?OffKozd0Xfb9/tplZLAggOtJK8YK5Zbt4oZrismO61m3nAiBR2OATLFbeg7h?=
 =?us-ascii?Q?cfvJn63B+sifD5SYVXbCamSH7K+XB+QsXVWqvHBjwvfcPee4IY4hwmGeVzev?=
 =?us-ascii?Q?Gg/74+f9T6qD6VfQ1kXnS/mOPpOkQdJRgix5sLZ9hJSxCh94syA3GcePECAP?=
 =?us-ascii?Q?wPT9m+VpGSWYeu7Iz6UHF9suXR7wq7LdmVMTKlxuAn63SCKbtAJG66HnT8A4?=
 =?us-ascii?Q?0YJ4oOvXHtUI08wGq1Tv9rC6GkD37PSFMyDsPMHgKEJ8KcwzPEzj2uIB4UQi?=
 =?us-ascii?Q?K/F2MY+YXq9usYrTL/adf3Jxsu3q54pkakA8IwiJkFp+/zYkANdzbgSxcHo3?=
 =?us-ascii?Q?Xv6QS0nDAVvTQNR96geojjt6Ekw73wyea5BCk6GrsbeUO2Mp+07iwEPCkwAV?=
 =?us-ascii?Q?iaqdaB9NXoUPzF3KmkYVxxCui8QQ4Y9lpoP/GKTGu7PqSaE+8ZnpbwMgJdGz?=
 =?us-ascii?Q?ZIkyfU/E4zjIgo59CD3jxIlwGLNPNvd1SSaFmRPwcPfbjowBPBIzaOEUQ6Xx?=
 =?us-ascii?Q?pu3YYQC6TO8xXU2aZcbzVwb/GlNtSZp/Jtqz2mixA46J7wxIx5yjZpUv48nR?=
 =?us-ascii?Q?4VeSNqdPUkqDepl/lmxrH8EbMh30P3hPaZWaMNpn2pgewGBZDFdUsSHphbFd?=
 =?us-ascii?Q?Jleg4bqWaLvYgeoff4wqBt9JllHQQVvbnNzcz4KnM7NwEc97KcBC1Wj7PefW?=
 =?us-ascii?Q?8+4UkdKfPBlyX117t6bjxLpuWM/gAb9IIFwCaYMTzKMtRMOg8YI5hdczFvZX?=
 =?us-ascii?Q?itn8p65CVBuvexb/AwSXErcy03arOS3oKyqBFTMM?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c6eed315-6d63-4da2-2a8b-08dc7550941f
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2024 02:33:32.3974
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LloidLkxiN700r4xT5LmqLSRWzI/3fpcLtBbXoQ71dGmx6vDDg3AtQnP6RnJCStaN2YJc/mzmxYYTjO53bLR2g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB4873
X-OriginatorOrg: intel.com

On Wed, May 15, 2024 at 05:43:04PM -0300, Jason Gunthorpe wrote:
> On Wed, May 15, 2024 at 03:06:36PM +0800, Yan Zhao wrote:
> 
> > > So it has to be calculated on closer to a page by page basis (really a
> > > span by span basis) if flushing of that span is needed based on where
> > > the pages came from. Only pages that came from a hwpt that is
> > > non-coherent can skip the flushing.
> > Is area by area basis also good?
> > Isn't an area either not mapped to any domain or mapped into all domains?
> 
> Yes, this is what the span iterator turns into in the background, it
> goes area by area to cover things.
> 
> > But, yes, considering the limited number of non-coherent domains, it appears
> > more robust and clean to always flush for non-coherent domain in
> > iopt_area_fill_domain().
> > It eliminates the need to decide whether to retain the area flag during a split.
> 
> And flush for pin user pages, so you basically always flush because
> you can't tell where the pages came from.
As a summary, do you think it's good to flush in below way?

1. in iopt_area_fill_domains(), flush before mapping a page into domains when
   iopt->noncoherent_domain_cnt > 0, no matter where the page is from.
   Record cache_flush_required in pages for unpin.
2. in iopt_area_fill_domain(), pass in hwpt to check domain non-coherency.
   flush before mapping a page into a non-coherent domain, no matter where the
   page is from.
   Record cache_flush_required in pages for unpin.
3. in batch_unpin(), flush if pages->cache_flush_required before
   unpin_user_pages.