Received-SPF: pass (google.com: domain of linux-kernel+bounces-11420-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <47433717-1c16-44de-b1d8-5957b4f8df06@intel.com>
Date: Tue, 26 Dec 2023 16:46:00 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v7 9/9] iommu/vt-d: Add iotlb flush for nested domain
Content-Language: en-US
From: Yi Liu <yi.l.liu@intel.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
CC: "Yang, Weijiang" <weijiang.yang@intel.com>, "joro@8bytes.org"
	<joro@8bytes.org>, "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"jgg@nvidia.com" <jgg@nvidia.com>, "robin.murphy@arm.com"
	<robin.murphy@arm.com>, "baolu.lu@linux.intel.com"
	<baolu.lu@linux.intel.com>, "cohuck@redhat.com" <cohuck@redhat.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "nicolinc@nvidia.com"
	<nicolinc@nvidia.com>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
	"chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
	"yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>, "peterx@redhat.com"
	<peterx@redhat.com>, "jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>, "Duan,
 Zhenzhong" <zhenzhong.duan@intel.com>, "joao.m.martins@oracle.com"
	<joao.m.martins@oracle.com>, "Zeng, Xin" <xin.zeng@intel.com>, "Zhao, Yan Y"
	<yan.y.zhao@intel.com>, "j.granados@samsung.com" <j.granados@samsung.com>
References: <20231221153948.119007-1-yi.l.liu@intel.com>
 <20231221153948.119007-10-yi.l.liu@intel.com>
 <f6302d8e-fd5f-45e1-8148-e5812c61f5c0@intel.com>
 <BN9PR11MB52766A289D2CA50F8BD802F18C94A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <D35102EE-F1E8-4888-8A5E-A1A723B3363D@intel.com>
 <BN9PR11MB527672402F30F701A5EA53028C94A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <65B6D347-5C85-4CB0-9CD8-1C914045B62B@intel.com>
In-Reply-To: <65B6D347-5C85-4CB0-9CD8-1C914045B62B@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OGhWSmV1aFpOcjFhanltVmw4ak1mN3dEYml4VGxLZlZuNmU1RjNoMEZwZitX?=
 =?utf-8?B?Ui9JTUhBZE1uc0RIeG9GSnhHL1ZMNExVcThwVU1GWW5FOHVjdkVJMmFnbGdi?=
 =?utf-8?B?YWtoRmFzb1NTUTQvWHJ4M0tLZGdSSXRTa1RUZXNKL2tkVDl0d1puejJ2c3JY?=
 =?utf-8?B?RHVuU2tCMmNjMmF4K3RHVVJLMElFZXplNEt1NFJtMU9zUUlxZW0wODlTMG5S?=
 =?utf-8?B?UHg3Q3dzQjduREQ2Q1BJQklEZnZMd2ZPZnJJTGZ0a1lJNS81emE3WncyRzBp?=
 =?utf-8?B?eFQyblRBbkx5eDM2YkIwanFTZ2xEbUdvamd1eW1jM0lJVktwaUNQV1gvQVhp?=
 =?utf-8?B?VWxDbWZmdm96R0pPZ1FIbHE2MHBTMG5aKzhicGs4SlFlVU0remVQWmg5TmN6?=
 =?utf-8?B?MjNBdjNLTXMzNGFaQllMRmM1ZXIxK2NKak5zeGVyM05ybVFTbGJzSVYrTFZl?=
 =?utf-8?B?WTczNGl2VkJsM2IrMnMrVGxlNm8rMEJ0QzFyZlYzU2tPODBVU3ZFNlMxRFNU?=
 =?utf-8?B?ZDQvRkpqZEFBMjZobUUvdWdldk81ZUVpYjg5UlNtcHdFUWRzY3M0L0kvd1dC?=
 =?utf-8?B?QXEzY0lHYUtHTFZvcHZOZTVBTjZHM00ycjFzcEk4bURndG9YYnlZdW9WdC8z?=
 =?utf-8?B?d3hibUpJbmFiRUl5L0I3NUZDTGluSW5uZ2FQT3ViRXBTdkNDN3pVMGxSeEtP?=
 =?utf-8?B?TFBVMi9GaFo1RzZKTFh2dTZGbWdBUU5VOGNPVkpHc2htYW8yS0E3Zjk1bWxH?=
 =?utf-8?B?TzlxM1RRVDQxU1J4cHFXSVJKanQwK0VZM1FyZUNtV2xJcDRPVkRUWUR3ejdN?=
 =?utf-8?B?UCtkbllsdE5VcjRYNEdVNE94L2tZVWwwT0F5UGlWeUwvZk9kR2ltQW9wR0tw?=
 =?utf-8?B?Q2tiS2EvdThVcllJUklMNFZCVDlwSWlpaXN6SEZzU0Q4RHpZbHRybzFBcHlE?=
 =?utf-8?B?THdpOFc3ZHhOUFB2ODZEVEV1dmFZVld0ektNeURYRENIbjUxVGZ1UzBkL05w?=
 =?utf-8?B?OXlUZkJUU0VESUZrTHpUU2h1UDMzSkpKelFPT1VMdXJXcExNejJhVHdma0Vs?=
 =?utf-8?B?SW85a3lLLzdPSnc3TUlNd0U1QlhNWUw5MnpTZmw2WklBZWlMZi9ZRitYc2Fx?=
 =?utf-8?B?SVZzRWNXMStGZXRRZEU2Rkc4b1dROXN2SFlEWWh2U1VLTWduNW9mM3BwcXlN?=
 =?utf-8?B?b01BWkUyOVB3WmNPaG9wYURYOWUzK3FJNmRYTU9BdEJiNFM0ZVhtamdKZmRX?=
 =?utf-8?B?ZG1LeUpzcnc3elZ4c0dWN25LMkdDaC9hazEwTzREWXlhaEZlWUVTV3ZONEl3?=
 =?utf-8?B?VzQ4ZnM4UllhOHlZaGNWeWZaaTh1UVMzMmJkamh3bVlMQVJubDJ6WHhqQVZD?=
 =?utf-8?B?WTBXNDNXV1pFUWI4ZUJqMWtRQlE4bjBVbVhzQnExWmpaOG0zSlJWKy9KYzN3?=
 =?utf-8?B?YU4rVmNOODMzRWF2ZGN4YjROWnE4V2ZzaC9jRzBURVBUdlJaNHNoeGJRc3pK?=
 =?utf-8?B?clNsQW9GU2VBMlNqN2Vac1FHa3pHSzd6TVFqUlE5QTZoaWkrUUZlWWw5RVlh?=
 =?utf-8?B?K1h6cmxNV1NZcytBVm9QbmRnTldIMTlSNEI2Z0lram80eERLNUZ4eUFHWTVN?=
 =?utf-8?B?dGpiejd3bldPVEh1UHRXd0FjdEgyMlBra1dNTTBiVCs1eUVZeTkwc2R1UXJv?=
 =?utf-8?B?ekl2MzVEV0l5M3oyd0xBQzlvWFlMSnVaWGNEOTE0N2ZxZEg3eXViZE9tSWky?=
 =?utf-8?B?cjZkQTU5RkVmSEtmTzhyTFY1T2JBUFRuQUt2c1hpM1VlUDZwaFFtU3pwWnZs?=
 =?utf-8?B?bzhtelgvYVQwYjEwU0l1d2toRXF6aExZazBreXdZMW12UjNMem1FTXEyRUVT?=
 =?utf-8?B?UHlicE5tMVA0VDM4clBXKy9xMjdiUGh4dUdnV0h3UUd5d0hhdC8ycisrRXU1?=
 =?utf-8?B?MlRRNGlEbGtkcHA2endJQzdOdW5PM3JOSHRoTkJWZkR2bWNodGxaMmVHRkY5?=
 =?utf-8?B?anpCS2x4VDl5bWdwTmZZeU5FeVRCMm9kVzh0V1F2Wk43ZTh1REM3TU1yT0VX?=
 =?utf-8?B?V0VGOG5PQUpNTFg4R2NXZkkyVDZMMWNOdXRiL2hGc3duQzFqcklCTUVFUEtZ?=
 =?utf-8?Q?o+z4X6rOsiwC7u+/v1EOP44Sm?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 359048cf-0b4f-4aa2-1e2c-08dc05eeb208
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7529.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Dec 2023 08:43:12.8655
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7FTrJ1/uPf+4sWKpnU3yxgVVCJFU2VtlfSByGtVqVzt89S5sX6X0B6Ib0VSWesFpYhcnwtNtooFc6c6cMkQKcw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6995
X-OriginatorOrg: intel.com

On 2023/12/22 19:59, Liu, Yi L wrote:
> 
>> On Dec 22, 2023, at 15:12, Tian, Kevin <kevin.tian@intel.com> wrote:
>>
>> ﻿
>>>
>>> From: Liu, Yi L <yi.l.liu@intel.com>
>>> Sent: Friday, December 22, 2023 3:02 PM
>>>
>>>
>>>>> On Dec 22, 2023, at 14:47, Tian, Kevin <kevin.tian@intel.com> wrote:
>>>>
>>>>
>>>>>
>>>>> From: Yang, Weijiang <weijiang.yang@intel.com>
>>>>> Sent: Friday, December 22, 2023 11:56 AM
>>>>>> +
>>>>>> +    xa_for_each(&domain->iommu_array, i, info) {
>>>>>> +        nested_flush_pasid_iotlb(info->iommu, domain, addr,
>>>>> npages, 0);
>>>>>> +
>>>>>> +        if (domain->has_iotlb_device)
>>>>>> +            continue;
>>>>>
>>>>> Shouldn't this be if (!domain->has_iotlb_device)?
>>>>
>>>> yes that is wrong.
>>>>
>>>> actually it's weird to put domain check in a loop of domain->iommu_array.
>>>>
>>>> that check along with devtlb flush should be done out of that loop.
>>>
>>> Maybe adding a bool, set it out of the loop, check the bool in the loop.
>>
>> the point is that dev iotlb doesn't rely on info->iommu:
>>
>>     nested_flush_dev_iotlb(domain, addr, mask, &fault);
>>
>> then why do it in the loop of info->iommu?
> 
> yes. It should have another device loop instead.

let me move the device tlb related code out of the info->iommu loop.

-- 
Regards,
Yi Liu