Received-SPF: pass (google.com: domain of linux-kernel+bounces-11508-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <54b6d7ed-c496-42d6-a4b2-50ba4c4b180c@intel.com>
Date: Tue, 26 Dec 2023 20:35:25 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v7 9/9] iommu/vt-d: Add iotlb flush for nested domain
Content-Language: en-US
From: Yi Liu <yi.l.liu@intel.com>
To: "Tian, Kevin" <kevin.tian@intel.com>, "joro@8bytes.org" <joro@8bytes.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>, "jgg@nvidia.com"
	<jgg@nvidia.com>, "robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>
CC: "cohuck@redhat.com" <cohuck@redhat.com>, "eric.auger@redhat.com"
	<eric.auger@redhat.com>, "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>, "mjrosato@linux.ibm.com"
	<mjrosato@linux.ibm.com>, "chao.p.peng@linux.intel.com"
	<chao.p.peng@linux.intel.com>, "yi.y.sun@linux.intel.com"
	<yi.y.sun@linux.intel.com>, "peterx@redhat.com" <peterx@redhat.com>,
	"jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>, "Duan,
 Zhenzhong" <zhenzhong.duan@intel.com>, "joao.m.martins@oracle.com"
	<joao.m.martins@oracle.com>, "Zeng, Xin" <xin.zeng@intel.com>, "Zhao, Yan Y"
	<yan.y.zhao@intel.com>, "j.granados@samsung.com" <j.granados@samsung.com>
References: <20231221153948.119007-1-yi.l.liu@intel.com>
 <20231221153948.119007-10-yi.l.liu@intel.com>
 <BN9PR11MB5276A45F5355A6DAA8CBE5738C94A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <cbd97b49-37b5-4445-a8b5-717b8ce99f59@intel.com>
In-Reply-To: <cbd97b49-37b5-4445-a8b5-717b8ce99f59@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NGlBRUZNbTJoaC9EZFEyU2RVMWFNeDV6VWNhMFBZQ3FrWC9zUkNhZmZsdUZ2?=
 =?utf-8?B?R0FvM3hkOUFLNENndklJemtnaE1yUW1tL1ZKb0haS0xpREVmZ3ZtZXJ2c2JQ?=
 =?utf-8?B?cURVaXJiUm1oT2xKM2UzQTdDQUNVb0ltMXBybFVrcnVVaExPRUh1VWl4ekpF?=
 =?utf-8?B?TVk4eUlIUm94TzEvdXg4M3VPQUtxOUo2WjZKLzFoNnBzbWE3bTFkeEJadXFk?=
 =?utf-8?B?T213N1dvZHB4SEd3SGZXL0o0aEUvZUhqNXhwTEg5N3NjVExTWWk2ZFhHOHlz?=
 =?utf-8?B?YUNhU1JWa1d6bDV3V1hWL1dJZURzRGtpLzJLQkhlK3RHNW43cERYazF2cWJW?=
 =?utf-8?B?K2x4UG1uRU04NW9JWXdya2hWMFFkZXVZNkQrcTJOTzladFFrTzd5L2o5bW5j?=
 =?utf-8?B?SEd2ZkI2VVlDUHI5V25Qd1BNdldXdXNPRkkxcm1rKy9SbkFZeTVPNUdONW9C?=
 =?utf-8?B?RjMxY3dSVzJVVWs1N1grb0d5MVJmeG5lbXZTcERsejRTYlhIay9lMUcrdWFw?=
 =?utf-8?B?WTdsZjllc09yaDJITlM0RWtwL0VRVkRDTWlpTTR4cU15TGpSK3Y0V0NTYmhK?=
 =?utf-8?B?ZGppMFMzenB0dHdYQTlYMXRkSC83MGFCckl4THViaVJybHlMWnNSQ3JlbG51?=
 =?utf-8?B?S2o1VHdBN3VBdy9aeFFkU05LTHJpS0VsSUdTRUNKdnptQ25OL2VFZkZGYlBG?=
 =?utf-8?B?M2dwN0dUMFlOckd3ZWdZaUNVYmFXcm56WXM4M2R6YXJuNWF2WDZUaDBZaE1Y?=
 =?utf-8?B?ek9kd2phdzN1Q0U5cmMvVzNMQ3NIMm5HeEJ5d2llck1xTEVmQkhrSjFQb050?=
 =?utf-8?B?d0JaaW4wWm1wTGNEd0ZBczlQaGZhSXN1VHVyT042NjhSbW9ONDhvMDVLaDdT?=
 =?utf-8?B?dmtHU1cwTEx5M3NxODc5Q3JHR0RKZmVQd3lDR1ZhYWJCUjVNdXd0S3VtWnBk?=
 =?utf-8?B?Tko3QVdUS2UzRXlUQnB3SElMQXAyS2VGQ09PQ0RuYkZVdDRHdWtPWTlOQmZM?=
 =?utf-8?B?VUdUQndLUmxkamNRYVJ2anBoTk5UbE5wdGp6V3NrYXZ5RGU5Y2FRYjdlSW5v?=
 =?utf-8?B?dnNicTBUTmhwN0pjSVczT3hYNng5QjBhOXppSzBERTExYjBsMFpaalVsd2tN?=
 =?utf-8?B?Tyt0dllrVTM5bmcrMmpyckZUMFBoZklGL0M2alVWQ1VZMW03STcya293N1RZ?=
 =?utf-8?B?OGMyQitzeVM4SzhUNmxaZmp6OEFteEgyRnF3b0M4aGtTSk80RjBlQ2lEYjZT?=
 =?utf-8?B?UzJPaEZjZFh1VzFKMUdOYlFVczFXOXBGeldpbjJnV2tVaXJBVDlLb25ESE1w?=
 =?utf-8?B?bXpWM05UeTVYTFpwTEh2TW1WTGV1Vnk2RVZuK0ZCdG54c1VibGIreXl6eHcz?=
 =?utf-8?B?WGtadEdLYXh1STdmNWdEczBXWTAvaGRlcTVaeUt0ZzVJejBsTTlzbHYzV2w5?=
 =?utf-8?B?Ni9ZV0RUSEpHWVdNMWV5eXcvem44QW9TQmM2ejAzRHZHRmJCbkQyVjR2LzRT?=
 =?utf-8?B?WmxPMmRXaEU2YndpbHdUQUE5OXpvWTdNTjdZZEJBNTdDUDBHMGc4YkRrbWhq?=
 =?utf-8?B?ajFBVWJHM0FZN0k3S3NncVd3Q0h4TlpMMllUczA2d09xMXYyaFpGbjFiZktt?=
 =?utf-8?B?bzdVT3dSc0pMbWRhdVU3K0JhUUZHUXQ2ZmUxblVydnVHeTBreFlDM2FVNW1i?=
 =?utf-8?B?YWJ4RTY4Sm83bHNpeHJnWE80S0taM0JvMHRLd1J3b2czblFoUnhXOUZYNkxh?=
 =?utf-8?B?UU5YZ1JaVkk0SXoxa0FGRUdwZ3pGcnpCeWx6YVhTMnQrKzZtS3hvdUJxcGoy?=
 =?utf-8?B?RzBwRWc4N0o4MS82QWx6U3I1OXR5amhaRGRYdksySjdwanZBRkNqYlVrTjBG?=
 =?utf-8?B?ajV2MlA3MExkZzN1Y0FBcDNNSWRmOHVabHljQ1pwQWJ1WTZJTldGb0l6YjdM?=
 =?utf-8?B?N1laeFpXbTVyamY0SThxUysxZHNlM2IvK0tSRU05MmNVSFBpWkliQTRQTTF2?=
 =?utf-8?B?WVNMcGNZdDBlcTJaR05wRGV4Qk1WWnNpUmlDK0tLdm1yYjNIYks0ZEZ5ZS9i?=
 =?utf-8?B?RFVYdTBBajIycFhnNXZjM0k5bFAyUE53Q3pQYWNleDZzUnkvN2xJd3B3SXRu?=
 =?utf-8?Q?9C1I7Nr8qe/SDuEw0ykEQM9mo?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1bd22870-f38d-4674-bcd4-08dc060ebe89
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7529.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Dec 2023 12:32:37.9408
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5bYxhzN28QN96rha613u3OGzjiYCMDWPAHSEoCgWOsSR76zgwyJybKL2rvO+2PoYNkN2oGrxMlBOdgeY7jayDA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR11MB6366
X-OriginatorOrg: intel.com

On 2023/12/26 12:51, Yi Liu wrote:
> On 2023/12/22 14:57, Tian, Kevin wrote:
>>> From: Liu, Yi L <yi.l.liu@intel.com>
>>> Sent: Thursday, December 21, 2023 11:40 PM
>>>
>>> +
>>> +static void intel_nested_flush_cache(struct dmar_domain *domain, u64
>>> addr,
>>> +                     unsigned long npages, u32 *error)
>>> +{
>>> +    struct iommu_domain_info *info;
>>> +    unsigned long i;
>>> +    unsigned mask;
>>> +    u32 fault = 0;
>>> +
>>> +    if (npages == U64_MAX)
>>> +        mask = 64 - VTD_PAGE_SHIFT;
>>> +    else
>>> +        mask = ilog2(__roundup_pow_of_two(npages));
>>> +
>>> +    xa_for_each(&domain->iommu_array, i, info) {
>>> +        nested_flush_pasid_iotlb(info->iommu, domain, addr,
>>> npages, 0);
>>
>> so IOMMU_VTD_INV_FLAGS_LEAF is defined but ignored?
> 
> yeah... it is. It is named as ih in the driver code. But it appears only
> the below code is set ih. When calling iommu_flush_iotlb_psi(), the 5th
> parameter (ih) may be true.
> 
> static int intel_iommu_memory_notifier(struct notifier_block *nb,
>                         unsigned long val, void *v)
> {
>      struct memory_notify *mhp = v;
>      unsigned long start_vpfn = mm_to_dma_pfn(mhp->start_pfn);
>      unsigned long last_vpfn = mm_to_dma_pfn(mhp->start_pfn +
>              mhp->nr_pages - 1);
> 
>      switch (val) {
>      case MEM_GOING_ONLINE:
>          if (iommu_domain_identity_map(si_domain,
>                            start_vpfn, last_vpfn)) {
>              pr_warn("Failed to build identity map for [%lx-%lx]\n",
>                  start_vpfn, last_vpfn);
>              return NOTIFY_BAD;
>          }
>          break;
> 
>      case MEM_OFFLINE:
>      case MEM_CANCEL_ONLINE:
>          {
>              struct dmar_drhd_unit *drhd;
>              struct intel_iommu *iommu;
>              LIST_HEAD(freelist);
> 
>              domain_unmap(si_domain, start_vpfn, last_vpfn, &freelist);
> 
>              rcu_read_lock();
>              for_each_active_iommu(iommu, drhd)
>                  iommu_flush_iotlb_psi(iommu, si_domain,
>                      start_vpfn, mhp->nr_pages,
>                      list_empty(&freelist), 0);
>              rcu_read_unlock();
>              put_pages_list(&freelist);
>          }
>          break;
>      }
> 
>      return NOTIFY_OK;
> }

I passed this flag to the intel_nested_flush_cache() now as the
helper accepts an ih parameter.

-- 
Regards,
Yi Liu