Received-SPF: pass (google.com: domain of linux-kernel+bounces-19119-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <1fbdeec3-58a6-44ec-ac0d-400676212f40@amd.com>
Date: Mon, 8 Jan 2024 13:49:25 +0700
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH 6/6] iommu/amd: Introduce nested translation support
Content-Language: en-US
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
 "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
 "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
 "joro@8bytes.org" <joro@8bytes.org>, "Liu, Yi L" <yi.l.liu@intel.com>,
 "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
 "eric.auger@redhat.com" <eric.auger@redhat.com>,
 "vasant.hegde@amd.com" <vasant.hegde@amd.com>,
 "jon.grimm@amd.com" <jon.grimm@amd.com>,
 "santosh.shukla@amd.com" <santosh.shukla@amd.com>,
 "Dhaval.Giani@amd.com" <Dhaval.Giani@amd.com>,
 "pandoh@google.com" <pandoh@google.com>,
 "loganodell@google.com" <loganodell@google.com>
References: <20231212160139.174229-1-suravee.suthikulpanit@amd.com>
 <20231212160139.174229-7-suravee.suthikulpanit@amd.com>
 <BN9PR11MB5276BA85118EECCCD89681EB8C93A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <d2cc19fc-f8c6-4b3a-ba88-5db947fb1604@amd.com>
 <20240105143703.GA50406@nvidia.com>
From: "Suthikulpanit, Suravee" <suravee.suthikulpanit@amd.com>
In-Reply-To: <20240105143703.GA50406@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?SDQzTXRjMm8vdHc4MGM5Sm95MzhlcEJuczBPVlo0VEdmRHZROU9JKzhmQ0t2?=
 =?utf-8?B?bjU5KzR0TW1CVDZUcm1xVzB6VTc0cEk2cW1RdkhEckh2VjRlWnNMRHp0ZUVs?=
 =?utf-8?B?ZlNWN1VuVGpvb0g5ai8yWG1RTThWS1pPNEZJcDN0VUpNRGFZb0VSZnZiVnRm?=
 =?utf-8?B?blNsK2gxamhMcndjT1VuOXd3MVllSDFYZ2xuOW1DR29Ob1Nnc0IyTXVtVWgy?=
 =?utf-8?B?Rm1TaEdJUU5RUEZYQlZBb2dyNTR0SEZqbXdNME1DSUI1U3VlUzVSUG5Xd3c5?=
 =?utf-8?B?VGhwRDI1MXluQlBYZnhMR28xL0dhaE1tcUcwM21BbDVHWVRBWkNHekZMUlYx?=
 =?utf-8?B?dWQyUHB6enh3UTZlOTBQMThRYzVudDZ4M2VzZXdITzlHNytBcHFUOTJiK0FK?=
 =?utf-8?B?cmhhZXRnS0dmS1VKZy9qYnRQVU5rdU1KNUdMNGhyNHBQM0JHNTg5QmY5UUhE?=
 =?utf-8?B?Y0c1UkNSOFhBTThybkZOWEM0aDU5OGFpYlZNYlJQQlVTUnZRUFZnR2Q1bGRz?=
 =?utf-8?B?MXhnK1BjVDlIWHNrbElxcldmcDgwbDhjNERraWNDNHIwOGhBZDAvb3doMlFu?=
 =?utf-8?B?Ty9TQU4rNGN6eXhQek5jeFNzUnE3ZzRFSm9CWHJKUmJJT0hrbzN2NUg1STRl?=
 =?utf-8?B?T2JrME1DVS9zR295Rkludi93dXBzSTRrbGlieHhPaEkvVW9OWm91OFdHalJn?=
 =?utf-8?B?bmJFallvSkd1aVNWL1AyS1h5aXlWbjlKMlkweWFTcFY4OTJIQ0lGVVExT2lN?=
 =?utf-8?B?ZVBGVGkzSTRiZi9LZi9uTDFKZUFOQVZRUTV6UUp3aDQwU3hGbWZyL3ZlK08v?=
 =?utf-8?B?dEhUb1UwMkRnU2JZZXFmTm4yU1pKQ09kdExzam1CMTVXeVZVWExjMThYcEdk?=
 =?utf-8?B?a0VCeVlTbEd4VjJwcmdrNm45RDN5ZGkxUjFPSjlzOHRqaElYRlVSS1BhUzhx?=
 =?utf-8?B?NUhsc3ExL0lFM0Z3M3JiTHVlM2J4Q01HekIybU9aN0FlL3doc3FOM1IyMmg4?=
 =?utf-8?B?RjNnNVBCR3ZjelJ0bVpnZjllRDV1TmtRMnI3UXJaZUwxOHE5bTJKOHZ1ZHRs?=
 =?utf-8?B?VzZaTnhubHNOOG4vTU9scGhlOExHNTFtb0poTUVFL0V2a0FhTlNuODhEb2k0?=
 =?utf-8?B?eEVJa2hFb2lSOGwzYldpaCtncW92UkpyNkUzT0F0RmtmeW8rY2F0ejNXNFFC?=
 =?utf-8?B?YnliRmtJc2VTZkoxcGt2a3ZNTU9pWjB3eFFCaWtXN2pNK24rR1REWmdUTGUy?=
 =?utf-8?B?MW15Sy9Ra2grT1U2UkdFNmlCS29IUFMrTG9JbUJhUENIeGQ2ckU0SWFtc3F5?=
 =?utf-8?B?YTBQcFU2c0JsbXBJT21uT3BzdjRvMFVIdXdRSzVWVkp3bGVxNVNiSTgxUmoz?=
 =?utf-8?B?cm85bUZkQ2cxdkVwSVRIZ280bGMvbDJieWpoWGtkT0Z5eVo0dFYxaEVCNDJt?=
 =?utf-8?B?WHdSNU9qbWszYmUxR3pCZmZSZTc3MFlvZTYzTDhvbXR6RzQvYUdxUXRTWDBL?=
 =?utf-8?B?Y3ZFR2dvZzZjNnhlY1hWRTduaml0L0FZOTZOMGI5OXIvUVlIYjBjSVhIa01j?=
 =?utf-8?B?RjE2N25iRnVYZUxsTGFOSG9RcDBHc0pNeTJEV2ZjQVB3b1dXYWpJNnpyOGd3?=
 =?utf-8?B?V2hoR2o1eWY3UkxPbnJMb1o5STZmdFROSXRtSnRVTUxoUjNOc21MbzhQMU1N?=
 =?utf-8?B?bUx2RE1TVFdJMmVRQzVHSkRTL1BqTkpJUnpxZDlqVFlLSmdaMU03WTN3ZjF0?=
 =?utf-8?B?S1dQc3MvckdHcW1Fb0lSQU1JbjdHWnFQaFh4aXZtM0FhOTVlbkJGOU9aV0hC?=
 =?utf-8?B?bkVwdnM3Uk04UWtDOWJVdlIzUzU0c1RkcG5vbm1TOFh5TjRYRW5kb2J0VHUy?=
 =?utf-8?B?Ky9qYU96QzY2SDVySWh0bTJRdXFVcjBWVWQxWjVCam45Y1VtYytVVnlpTmtQ?=
 =?utf-8?B?ck1oSDlOQ3FtUEh6U0s3OFdaTXlaT0xtbFA3VEY2UlpQK0xjZEUxeDBYMFNw?=
 =?utf-8?B?Y3VIY1dQdEZ3R0NjbkpKSlNmOW1wVUdaeXUrbTZFOGtqSmpaaXlHQXZrNW05?=
 =?utf-8?B?MXdlVk1jQTRENmlJY1VOdGJGNHErMVBhTFN2OERkS1FKQnA0N0ZKUGRGTWl3?=
 =?utf-8?Q?JeGcQiQ97tcy+x4oDakEeoxOo?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e4e9c580-ab01-41b1-0fd0-08dc1015f98a
X-MS-Exchange-CrossTenant-AuthSource: DM8PR12MB5445.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jan 2024 06:49:34.8051
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: UN5irFBDJ3R7gjEM8+L7IK2jDN3tvF9zqaA47YDWAn1RXY3WXXvaSRYnBDMNfDzdnhNagIKT1uWyf9bjSApyRg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB8821


On 1/5/2024 9:37 PM, Jason Gunthorpe wrote:
> On Fri, Jan 05, 2024 at 08:39:52PM +0700, Suthikulpanit, Suravee wrote:
>> Hi Kevin,
>>
>> On 12/15/2023 2:45 PM, Tian, Kevin wrote:
>>>> From: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
>>>> Sent: Wednesday, December 13, 2023 12:02 AM
>>>>
>>>> To support nested translation on AMD IOMMU, the driver needs to
>>>> program DTE[GCR3 Table Root Pointer] with the address provided by
>>>> the guest via struct iommu_hwpt_amd_v2, which is passed as a parameter
>>>> of the struct iommu_ops.domain_alloc_user() with the flag
>>>> IOMMU_HWPT_ALLOC_NEST_PARENT.
>>>>
>>>> Note that current implementation only support GCR3TRPMode for
>>>> nested translation, which uses GPA to program GCR3 Table Root Pointer.
>>>>
>>>
>>> means there is a plan to support another mode in the future or
>>> actually the nested translation requires GCR3TRPMode as a
>>> functional requirement? imho the point of GPA is assumed
>>> in the nested configuration in concept...
>>
>> On (older) system, which does not support GCR3TRPMode, the IOMMU driver
>> needs to program the device's DTE[GCR3 Table Root Pointer] field w/ SPA.
> 
> Meaning that on older systems the GCR3 Table Root Pointer is not
> translated by the parent v1 page table?

Correct.

>> When QEMU presents an AMD vIOMMU device to a guest, the guest programs the
>> guest DTE[GCR3 Table Root Pointer] with GPA. Then we need to :
>> 1. Traps the DTE write
>> 2. Translate the GPA->SPA
>> 3. Program DTE with SPA.
>>
>> With the GCR3TRPMode, we can skip step 2 above and directly program step 3
>> with GPA.
> 
> Do you want to support this? It will be hard to do because it is not
> just those three steps (which are easy) but that you have to somehow
> maintain coherence with any changes to the parent page table, so you
> have to hook the iommu_domain unmap as well...

I'm debating this. Let me get back on this part.

Thanks,
Suravee