Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
Message-ID: <8ad28ad3-7f8f-4bcc-8ec2-bc5e9e998d2c@oracle.com>
Date:   Wed, 18 Oct 2023 16:16:37 -0400
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] iommu/amd: Do not flush IRTE when only updating isRun and
 destination fields
Content-Language: en-US
To:     Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
        linux-kernel@vger.kernel.org, iommu@lists.linux.dev
Cc:     joro@8bytes.org, mlevitsk@redhat.com, seanjc@google.com,
        vasant.hegde@amd.com, jon.grimm@amd.com, santosh.shukla@amd.com,
        joao.m.martins@oracle.com, boris.ostrovsky@oracle.com
References: <20231017144236.8287-1-suravee.suthikulpanit@amd.com>
From:   Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
In-Reply-To: <20231017144236.8287-1-suravee.suthikulpanit@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RFFhNUQxcGpmSmFqQnN0UHRhTUVzL2FLcjBTTjhybkg3N0p1WW9QY2NVZGhy?=
 =?utf-8?B?cjNQenZNenZsRUJseGtjVjJLYnY5Z0FDbzdRek5mbnYzWUwrbXNHLzREV0FW?=
 =?utf-8?B?LzFvWlBnaXZHdXFJRlNkZlc1VTBTSHBuMW5IUi9vakxUTlp3c1JCaGt0MTNu?=
 =?utf-8?B?dkVkSlo2blRsWFZqakx1R2Z4R1J1WGJJWENJdk5CMkNjNzE1TjJOczljNHJP?=
 =?utf-8?B?VFpNVnFZczV1UHQyR2FKR2ZuSWJOQXNETVNsY2h2dzJaTk5kYU5pdzQzLzlo?=
 =?utf-8?B?ajI3dnh0bm45UnRzV0oveWJoM0RLdzR0bGQvbENyam10M3NBRHNyQktnRUZ1?=
 =?utf-8?B?eWJlZVFqUU5IVWNiOTMvM1J3clJFY2JWbnloTnFrWWIwNlpNVkNlcUFFL3Ex?=
 =?utf-8?B?MHRBRlRHek1jSEZ2UzNFSzVId1BzaFZrM3RGRWttY04yN21qQmdVMVpYUUVz?=
 =?utf-8?B?Q21kT0ovbzV5LzNSSFltMnZRUkp4OU1Qd1lqc25TckU2YWk0V0tERzUvTmJI?=
 =?utf-8?B?ek9zcWFBdHdVaEh2TlE5N3p0NXhhc2lZTjR3Q3NrZklHUjM2OGRzUFltVmhU?=
 =?utf-8?B?RUM3eUVqMkMycllHNjhkcEIxaWc1RVFlOXRhcEFYM3M4MWdpMG5TRCtMVFVR?=
 =?utf-8?B?bGIzRWd0cmN6M29aQXNMN2dEQ3A1VjM0bFIveHpBSWNCZDA1T1E2RlFUclBG?=
 =?utf-8?B?d09INy9Pa3paeEU1VjhJMytuclZDYkVCNmIzTWRnT0dycGlOS3JNUVNGZnRJ?=
 =?utf-8?B?RVREVmFadEJsb1lLNWg1N08wQlYzQkIwNGRONFpGL0lMK0FvaXZTZ24yM1Yy?=
 =?utf-8?B?NDgxQm0zc0hNcGdxVlZ0ckVlL3Y5czNhRUM0akRkVkY5SVZuam5WT2wvNzFH?=
 =?utf-8?B?Zk1RUDNuT1dlZzIvVVhvMGdXUGdjbjlFNXdGTCt4c1JmbFo3dnZPMHBySVNq?=
 =?utf-8?B?bzNweXRZZTZqRm1tYWQ1dENNanIzS3RkRHpLYTNuRU91SmhpM2lMYnRvNjgz?=
 =?utf-8?B?WTJ5NWhQWURtczBvT1BDdFN6V0dCQTZ1NUMreFJRdXZubVc5V2huNUNaVVdM?=
 =?utf-8?B?Ulk5Z1M0N2JYTmN0MUhTTmRGbkNhcWxWcEFBVGRKTXp3U2ZpVDIyN1FLcFg0?=
 =?utf-8?B?SlljenhSSXBpTGM1Vlg5a1NoK0JsVnBydklaT3VnNDVOUHlpeHVjSm9YQzJV?=
 =?utf-8?B?UXgrVWJlZFlXRU11M3J3SUwvRUtxYk1PQnpFZVF3eGxEVHgvSitMUjlzS2Fa?=
 =?utf-8?B?SmxtcUs5VnF0UjhKTEdRN29BbllWZWdvUkpQc2x2YXdvRkJ3c2ZJeDRFNkJq?=
 =?utf-8?B?Wmk0RUxPZG4zNEFYRTRkK1Z2WFdZMTdoUGU4bDhPYnY2MVhIdFBlTm5samU4?=
 =?utf-8?B?KzBCbVNiRHBXTkhNQjNPdlF3Q3lodlRlRnlyR2hJQUNpRFg1M0lYQ21MelVM?=
 =?utf-8?B?cXJIVVVlZGlnTytLVFQxbGZaanR1Sjk2amRZYXgyT01qU2xXcnNFWXFkVXFt?=
 =?utf-8?B?SHROWmRJYk5NTFMxRWhaV01DQWNEUjRvZnlCU2FSRDNNbFhobG5EK3RpemlY?=
 =?utf-8?B?Z0ZOMkE5TVdKSXJOSmx2bThMS0crbXByVzhwcDFBMmROZWpCVFVrNFEzcGIr?=
 =?utf-8?B?T25UaUcxOHAzRVJZWUpEbjYrREZ6U3dLNE80NE94U2NJSlpkUWxKNVQyS3BS?=
 =?utf-8?B?MDBSbk9xdzg4anFJZ1FKOXhNTjIyTGhydnp6aU1DczZZaW02dFdrdW9IQlJp?=
 =?utf-8?B?TWg0T1gveTdqd0N1dFM4Z01QWWRzbG4rc2ZIZGZMMDY4aEZ5R1IrZWZWWUFQ?=
 =?utf-8?B?cC8yQm5aSlVyTFVib1VuZU5Od0QvUHJJZ1kyNklSZDJDVnZJTVd5S3JrZ21u?=
 =?utf-8?B?d2lWbzBnNFR1blRTWUZYZm0wQzlnb1Y5Y2hOK3FHYnVtZUFia2lKVGhhWXRl?=
 =?utf-8?B?VDM1bS9TYllVc1g0YzhETkJCWWNvcTNLRlR1eURPaHJmVVdtSmdKSG9nb2I5?=
 =?utf-8?B?TUlnaENFRFZSM2g1UVllWE5mRU1IYUg2OGoxU2R0V3lHbVhpM0dHZWs1QTMz?=
 =?utf-8?B?RlNVY3Qxd3c0S1FZb0NRUFZXekpvM0pNMlRYR25EYjZVbVc2a0dlYVQxbkdu?=
 =?utf-8?B?ZTFpcWFMM2J1SERMMnFQVjRCc1dqYjlVU3ErVzJDcDdwUGNtRlVFVzlQbzJO?=
 =?utf-8?B?K2c9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 6IwSTSMhHsxqyLBR31FMXHnIjQAN++AywgasgamEqlH8L5DijtfHuBXA/65li4N4ohzf7rKXKIDFpblh8O3qsWpUNykzS8e7drZdmrpkk+7O8yCSxdIBQalIShMjwhgHc/gUGsjh4h22ssg4bYU95HzL4KV5mUKvlH6AaehBmZpjXU53vVho8Hd5hJrqLnSy0fEPNN7waY0KTj9CKH5pFp0A3O1IiYMI+DNBysmpAzycKXvcZ0JFzrdgpeWzIfvFN7GWzAqtfkOcx/igF/W8K1Mc80k1ozAgDOLr5ZOxt886d5AF24CXYHGUiLaWUh3auEjRoG8mHZ8IsND3TK+mXCSJX+4PAI17CDciBJxqyVJp+oMx0G2euRrYU8qqJA1MFEx1/Rjwb3uK8qqvUdkiZSUz5oqH7KlQTHI45b7XOmgnEcBMG3MQCMLDcIg5xXiYSgb4F2pqVJsM1sBQt5itnv5JATeF9c4NNR4NSf1VIGcQGSqPlnJpG6+ra5kuaHSR6k2AE5ZyYpm8oAdPlBc66LEzmRrOdOfZkfm2C7taS49WJk5p+ieXe5rN3O52gMkjU8JQ7IJL9z96mNJYREZ5824YdNMeEGlG0AkAUdnxTJn5kZmYAqfwRZ2myJGBQu+M2ohp8zFJgaJDS7PjqbiFL9BETlhSB0f4YqKAALq/7yc/s5UUZsu/bs+ovN9xjnjbJp5hN2d2BAecKc892+Jq+c5IWbk/xzzjjQEE2IovpOCyczsWAfUiFbQuMTt0oKsunLlik/O+ngaaH+3b9k52i1gvO+QXUdJMs5DcJb01A4yKlIc3gMFiff6n9+xSQwthspUJniOUy9THiBuIbL9Qd1aYx2+CR5UWWGnUqJm7UfV5WEY+9B3xK0R6gJ3no5b5MJinteO3gSi8IKrkizuG3oDDq8xX7mEpiVY7uQpQ7ls=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3776e9c5-cb42-4213-381a-08dbd0172439
X-MS-Exchange-CrossTenant-AuthSource: DS7PR10MB5280.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 20:16:41.6087
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: TwB8r5yVk0kTCONHx7SLHy2/psl3Vil48CCfi2J70HOScn3BfwH5rHRat9UiPApaqQH9kuJKFCuMYamrirW3IahjZwBdRQeIc4N8YSH0g70=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5836
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-10-18_18,2023-10-18_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 phishscore=0
 mlxlogscore=999 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000
 definitions=main-2310180166
X-Proofpoint-ORIG-GUID: amzVVrVUwG2eEcGTWOzJuaOqwpVMv2Fm
X-Proofpoint-GUID: amzVVrVUwG2eEcGTWOzJuaOqwpVMv2Fm
X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 18 Oct 2023 13:17:13 -0700 (PDT)


On 10/17/23 10:42, Suravee Suthikulpanit wrote:
> According to the recent update in the AMD IOMMU spec [1], the IsRun and
> Destination fields of the Interrupt Remapping Table Entry (IRTE) are not
> cached by the IOMMU hardware.
> 
> Therefore, do not issue the INVALIDATE_INTERRUPT_TABLE command when
> updating IRTE[IsRun] and IRTE[Destination] when IRTE[GuestMode]=1, which
> should help improve IOMMU AVIC/x2AVIC performance.
> 
> References:
> [1] AMD IOMMU Spec Revision (Rev 3.08-PUB)
> (Link: https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/specifications/48882_IOMMU.pdf)
> 
> Cc: Joao Martins <joao.m.martins@oracle.com>
> Cc: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
> Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>

Built on top of v6.6-rc6 tag, and booted on EPYC Genoa host. Launched 380 vCPU guest with AVIC enabled and 16 VFs, and ran rds-ping workload that uses all of the VFs to send traffic. THis workload has been used in the past to stress IOMMU AVIC code paths. No issues found.

Tested-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>

> ---
>   drivers/iommu/amd/iommu.c | 20 ++++++++++++++++----
>   1 file changed, 16 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
> index 089886485895..d63590563d3e 100644
> --- a/drivers/iommu/amd/iommu.c
> +++ b/drivers/iommu/amd/iommu.c
> @@ -2970,8 +2970,8 @@ static int alloc_irq_index(struct amd_iommu *iommu, u16 devid, int count,
>   	return index;
>   }
>   
> -static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
> -			  struct irte_ga *irte)
> +static int __modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
> +			    struct irte_ga *irte)
>   {
>   	struct irq_remap_table *table;
>   	struct irte_ga *entry;
> @@ -2998,6 +2998,18 @@ static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
>   
>   	raw_spin_unlock_irqrestore(&table->lock, flags);
>   
> +	return 0;
> +}
> +
> +static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
> +			  struct irte_ga *irte)
> +{
> +	bool ret;
> +
> +	ret = __modify_irte_ga(iommu, devid, index, irte);
> +	if (ret)
> +		return ret;
> +
>   	iommu_flush_irt_and_complete(iommu, devid);
>   
>   	return 0;
> @@ -3681,8 +3693,8 @@ int amd_iommu_update_ga(int cpu, bool is_run, void *data)
>   	}
>   	entry->lo.fields_vapic.is_run = is_run;
>   
> -	return modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
> -			      ir_data->irq_2_irte.index, entry);
> +	return __modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
> +				ir_data->irq_2_irte.index, entry);
>   }
>   EXPORT_SYMBOL(amd_iommu_update_ga);
>   #endif