Received-SPF: pass (google.com: domain of linux-kernel+bounces-11345-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <cbd97b49-37b5-4445-a8b5-717b8ce99f59@intel.com>
Date: Tue, 26 Dec 2023 12:51:39 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v7 9/9] iommu/vt-d: Add iotlb flush for nested domain
Content-Language: en-US
To: "Tian, Kevin" <kevin.tian@intel.com>, "joro@8bytes.org" <joro@8bytes.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>, "jgg@nvidia.com"
	<jgg@nvidia.com>, "robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>
CC: "cohuck@redhat.com" <cohuck@redhat.com>, "eric.auger@redhat.com"
	<eric.auger@redhat.com>, "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>, "mjrosato@linux.ibm.com"
	<mjrosato@linux.ibm.com>, "chao.p.peng@linux.intel.com"
	<chao.p.peng@linux.intel.com>, "yi.y.sun@linux.intel.com"
	<yi.y.sun@linux.intel.com>, "peterx@redhat.com" <peterx@redhat.com>,
	"jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>, "Duan,
 Zhenzhong" <zhenzhong.duan@intel.com>, "joao.m.martins@oracle.com"
	<joao.m.martins@oracle.com>, "Zeng, Xin" <xin.zeng@intel.com>, "Zhao, Yan Y"
	<yan.y.zhao@intel.com>, "j.granados@samsung.com" <j.granados@samsung.com>
References: <20231221153948.119007-1-yi.l.liu@intel.com>
 <20231221153948.119007-10-yi.l.liu@intel.com>
 <BN9PR11MB5276A45F5355A6DAA8CBE5738C94A@BN9PR11MB5276.namprd11.prod.outlook.com>
From: Yi Liu <yi.l.liu@intel.com>
In-Reply-To: <BN9PR11MB5276A45F5355A6DAA8CBE5738C94A@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?L01RNUcxekU0TVNBZzF2bTZMWjhPNm9kdjd2WDBkQm1iWlJVSHlvZXpBcE1S?=
 =?utf-8?B?eFVuSGpSZjFVQnRvTkZDSGNwVVRPYkFheFdkeDBGMEk5dVFTWnRXODBrcnZT?=
 =?utf-8?B?T2JlZkN2YWhZdDYvUVpOZWtoOHhxbkxrVU1jZDRLVCtraHJyL3o4SnR5amZP?=
 =?utf-8?B?cUNwWXVXc3M5QnRpSEwxemZ6enpVWU51WXhqNVdVMVZkd29vWWYrOWR2ZDZj?=
 =?utf-8?B?bUNmZk1IZ0s0QkNndU1kUjErRmhqdjFEWVh4RzNEeW94Y1JxTGpzd1A2aDRy?=
 =?utf-8?B?QjBnRTBEWXF0NnovekJia00zQ1c1L3RiU1pQcE5uNTJLTzZPMmtjZG1ZZkVo?=
 =?utf-8?B?VjZEb09ZKzR0UHJMbW9aaHdFVjJOVktJcEhXZXpvRStZWFhjOGwzUzJmOUYw?=
 =?utf-8?B?Q1ZaME1POStmQmovRHJsMWZaRjNwYi8vUGRkT2tuVzlrS1R3TGxWSUgyVXVB?=
 =?utf-8?B?bnB0VnNPUDNDUjJWQW9KaFJoV1EvSEVxSDlUQlJUYUZFdDlmRTdqNUhhV1M0?=
 =?utf-8?B?OEVYb29LbE8zMEtpZ0V3dENCanZjQjgvTHVHcGszem9pR3RrU213VzFxa2NZ?=
 =?utf-8?B?Sjd6SFFyOVFyU01lUVN5eXBacXJBV1ZvclZQbCsxaEdhN2paKzZqdDlsVjJu?=
 =?utf-8?B?MUJYc3M5Vml0d0pOb1VHNno4S0tXUFlFWStxL0ptYldWTFYxQUtWZEoxV3RE?=
 =?utf-8?B?UEJoenZjR0J0bndCTTRLTzlvZ3RKSW4zUG1ub3I1K0JxNEttSm1WeUlvOERG?=
 =?utf-8?B?S28wdVhpVjhnME81RlJsMjYya3JaUUhMRFRLWm9ibDU4YTMwNFgzbzlTZWpD?=
 =?utf-8?B?Y3IweEhhR2N4WUtRaUpkc0grNXpXTFFTSUgyTVNQSGxaM0tLYnRrVXZWZytR?=
 =?utf-8?B?cVF5cXUzZEhYYzhabE83eFpySEZkUmdqU3Raa2dXUXk2L3VISlJ5K1ZtcGZu?=
 =?utf-8?B?VmFWc3hGdFdURVF0ajNQd2xIWExXQVpSVVc5TnlOc0R1RGJpcEFtRGdRd3ZY?=
 =?utf-8?B?ZnVVa20zYjlHY1BXd1ZBVnZwYnVSMjN0cnorbjg0QUk5MCtGbmUrdVcxaXdG?=
 =?utf-8?B?SmQ0Y21xa2w4V2srWGx4V3lBN2JNRVBaYWVYVklhYzRLMzREeFJnUHFvdElx?=
 =?utf-8?B?b1FEOXoyUk42K2tCeGpWTzdiaVpERENMdW5UWTIvcmtwb0pBQStmUjVoR2Rr?=
 =?utf-8?B?RUtLU2dTZm5wZytmVkU0d1ZjYWpHa1ZibGVkbDdqKzlNVlgvcnJXdXlVQ2Vx?=
 =?utf-8?B?dVp0TlNkQmk5Z2c3Z000WFNqK2w2bzhqRHMycE0yaXVNSFZ3NWpmdkZmMzNm?=
 =?utf-8?B?MTFKRUEyRGFyR2ZvR2txdDBMTVluNDdVV0lQZXJxNi9PL2VSZGtSRUJhbEdq?=
 =?utf-8?B?TCt1UE9JL3doV2t5R1pQcnp3QUN2L0I1VnZuYU4rSnlQa1Z2RHh0SzM4Y0lV?=
 =?utf-8?B?RkpIdlFySUMxOXBGdHlaU1Y2OE1LZXdXTUJBWmNDNUJyVU81N1NHQURoVGIv?=
 =?utf-8?B?NWpnSHRqMEhhTG4vNVRkcldWbTFOd01rbHRScDVRZjQweTNVbE41QzV0WTJO?=
 =?utf-8?B?TzR4MWRGZEdodjVyaURGdW9UUndKZHVLRFNncVJsMGdwK05GQU5nU0c1UXNH?=
 =?utf-8?B?U2JLYVovMmpsY01IQ29MKzdRd29BSUJDdGxldk1yQjRTWEx6Uk1zMkRlekRq?=
 =?utf-8?B?a3gycXY5a0pncFNxZWpCT1JCRU1NZlRRSm1VUlNRMmh2Z2E2Q0NNWFlhdmVy?=
 =?utf-8?B?UFJVMm9EUUxhUjhsdnF4MTBiZU1aM3cwVFdTM3FoenkxcFMxMG82VmRCMHly?=
 =?utf-8?B?Mk1jNkVLRzZDUFF4TTFJcDdCdGxEbEdEdlJYWEl5dFA0a2xlUW5aMk8vMU9v?=
 =?utf-8?B?UmF5UmZQS1pvZytYM1lPK0RrUGltZG5kSzdDYnlqeHlzaG90SE1lUGVJYlAy?=
 =?utf-8?B?Z2loeDdhakdITkkwRGQwUUdNaVN2UCtKaTVJekcrcWhuWG1uMXpYZlJhN1A2?=
 =?utf-8?B?ZFJicGlxYWszV2lKeEloMXB2ZFR4YjR2TzFtWFd2NjdGaWNWZGFGQ1ZvSUVl?=
 =?utf-8?B?c2V2L0tmeWloVHFxQyt3bUw0azMvRGU3V3VtbGduSFNZS05UVjA4d25ra3ZO?=
 =?utf-8?Q?GjLKC2I5qwIg40VjAvB31oPuX?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 6355fe0c-2610-49b0-38f9-08dc05cdf58a
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7529.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Dec 2023 04:48:53.2766
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: kxI+RNWOp21CSmOxdEfwt1uelzPK1kl7moswTkGaesCS4zjt8HxnEUE27Bi9y8lqVs8BYzXFtT/ubJsAcyvIJw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5451
X-OriginatorOrg: intel.com

On 2023/12/22 14:57, Tian, Kevin wrote:
>> From: Liu, Yi L <yi.l.liu@intel.com>
>> Sent: Thursday, December 21, 2023 11:40 PM
>>
>> +
>> +static void intel_nested_flush_cache(struct dmar_domain *domain, u64
>> addr,
>> +				     unsigned long npages, u32 *error)
>> +{
>> +	struct iommu_domain_info *info;
>> +	unsigned long i;
>> +	unsigned mask;
>> +	u32 fault = 0;
>> +
>> +	if (npages == U64_MAX)
>> +		mask = 64 - VTD_PAGE_SHIFT;
>> +	else
>> +		mask = ilog2(__roundup_pow_of_two(npages));
>> +
>> +	xa_for_each(&domain->iommu_array, i, info) {
>> +		nested_flush_pasid_iotlb(info->iommu, domain, addr,
>> npages, 0);
> 
> so IOMMU_VTD_INV_FLAGS_LEAF is defined but ignored?

yeah... it is. It is named as ih in the driver code. But it appears only
the below code is set ih. When calling iommu_flush_iotlb_psi(), the 5th
parameter (ih) may be true.

static int intel_iommu_memory_notifier(struct notifier_block *nb,
				       unsigned long val, void *v)
{
	struct memory_notify *mhp = v;
	unsigned long start_vpfn = mm_to_dma_pfn(mhp->start_pfn);
	unsigned long last_vpfn = mm_to_dma_pfn(mhp->start_pfn +
			mhp->nr_pages - 1);

	switch (val) {
	case MEM_GOING_ONLINE:
		if (iommu_domain_identity_map(si_domain,
					      start_vpfn, last_vpfn)) {
			pr_warn("Failed to build identity map for [%lx-%lx]\n",
				start_vpfn, last_vpfn);
			return NOTIFY_BAD;
		}
		break;

	case MEM_OFFLINE:
	case MEM_CANCEL_ONLINE:
		{
			struct dmar_drhd_unit *drhd;
			struct intel_iommu *iommu;
			LIST_HEAD(freelist);

			domain_unmap(si_domain, start_vpfn, last_vpfn, &freelist);

			rcu_read_lock();
			for_each_active_iommu(iommu, drhd)
				iommu_flush_iotlb_psi(iommu, si_domain,
					start_vpfn, mhp->nr_pages,
					list_empty(&freelist), 0);
			rcu_read_unlock();
			put_pages_list(&freelist);
		}
		break;
	}

	return NOTIFY_OK;
}

> 
>> +
>> +		if (domain->has_iotlb_device)
>> +			continue;
>> +
>> +		nested_flush_dev_iotlb(domain, addr, mask, &fault);
>> +		if (fault & (DMA_FSTS_ITE | DMA_FSTS_ICE))
>> +			break;
> 
> here you may add a note that we don't plan to forward invalidation
> queue error (i.e. IQE) to the caller as it's caused only by driver
> internal bug.

yes.

> 
>> +
>> +		if (!IS_ALIGNED(inv_entry.addr, VTD_PAGE_SIZE) ||
>> +		    ((inv_entry.npages == U64_MAX) && inv_entry.addr)) {
>> +			ret = -EINVAL;
>> +			break;
>> +		}
>> +
> 
> why is [non-zero-addr, U64_MAX] an error? Is it explicitly stated to
> be not supported by underlying helpers?

no such limitation by underlying helpers. But in such case, the 
addr+npages*PAGE_SIZE would exceed U64_MAX, this seems a bit
strange. But I'm fine to relax the check since the underlying helper
only checks npages when determining paid-selective or not.

-- 
Regards,
Yi Liu