Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <c3fbec52-1281-eb1c-40f8-588438146c14@amd.com>
Date:   Fri, 20 Jan 2023 09:12:26 -0600
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.2.1
Subject: Re: [PATCH 1/4] iommu/amd: Introduce Protection-domain flag VFIO
Content-Language: en-US
To:     Jason Gunthorpe <jgg@ziepe.ca>
Cc:     Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
        linux-kernel@vger.kernel.org, iommu@lists.linux.dev,
        joro@8bytes.org, robin.murphy@arm.com, thomas.lendacky@amd.com,
        vasant.hegde@amd.com, jon.grimm@amd.com
References: <20230110143137.54517-1-suravee.suthikulpanit@amd.com>
 <20230110143137.54517-2-suravee.suthikulpanit@amd.com>
 <Y8F53dzdebKLTlOy@ziepe.ca> <90762dee-1559-58ac-220d-a13635d5032e@amd.com>
 <Y8mBczFH/Hw6xot0@ziepe.ca>
From:   "Kalra, Ashish" <ashish.kalra@amd.com>
In-Reply-To: <Y8mBczFH/Hw6xot0@ziepe.ca>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TW1LTk93TmVmam5rTmZrRkw4L0MvNWpWZVpwd1RhQ3Fva0EvS1k0T0FDd0gz?=
 =?utf-8?B?YVNsaUQ2L25zTTgvd0FCUHhVVEVVeENRM1JINGRwS1dFUHk0aEZHQ1piMzc1?=
 =?utf-8?B?ekV6dEcwWW9aSFRFR2dRVmZOS2NVSHl5S1U3VmcwZ09oVml1WFBSUUU1RFhJ?=
 =?utf-8?B?aGJyVnFRRkI2SDVvNFFOc1RmYXBTdGRQMlArUVAzb3FZOGNiaGZSdkJwMDFN?=
 =?utf-8?B?Smg1cm52SkFiOXRpYnoxT1lQTWJ6RjNRcEFJVG5ZTCt1VUt2ZXVQRFhDS0hK?=
 =?utf-8?B?TWtob3hPa2FmdFkrWUtBTGZDT2VaY2puM0JPRVZQdjBIaE5xZ3RWQVczL1V2?=
 =?utf-8?B?eUtMSklQMForVFBuNmdZbTluRTkyZnVzQWlQVHl0LytsbzYyNnhJTFZpVTBl?=
 =?utf-8?B?L0dtL2crTi9YdCtrMzNrRlRRR2QwNnlKNGRESWR2My9DV20vUkVlRXpXa3RI?=
 =?utf-8?B?MW52ajg4M0ZiQm5ERlZpYnNud3A2SkJNQ1ArZENRcTNZYnRyaldzWFlQb05M?=
 =?utf-8?B?aStKUlRjWEl2SHdXdG9HalFsbmhDVEppNFJKcWxNWmkycUdod29QaVVXR0Mv?=
 =?utf-8?B?QmhoUnByVjk5Y1IwWmVBczUvWnFBVmRqTVBVdWlQZUYxYTVtbE1WQVQvS3Bw?=
 =?utf-8?B?d3hSUDh2cXdDM3kraVlwQk4rQW0vTkxlN2V4YjMzbGVvaWdEZFRlcnhUWml5?=
 =?utf-8?B?VWlzeUd0UnlQcHR1enU2MklmZjVzZFNwS214c0FoNzZMak05bHI1c1FIazZI?=
 =?utf-8?B?anhSQlBReEErOU55bEdmUDVYcnJ3VjhpaDZWeWFvc25FNkYrUEYwd1oxeW13?=
 =?utf-8?B?RjIvWUlUQm4xTXVjZUo1YnUweGRndmcvdjRITTJCWUVQazdVS3FyY2lyUmRI?=
 =?utf-8?B?MFRmZ0ZiRmlCMWVvMWR4Qjdwd0xaTlpFN0Q2WHVNLzVHbTZmV2NxSW9kM0tT?=
 =?utf-8?B?QXcxRXB4NmVQZUJ2a3UwVTM3UDZZV1FzM01IbVZzMWJOWmZ2c2ROem55cDdQ?=
 =?utf-8?B?VU1SQnFyeE4rSkRyelZxQVdvS3NWQ0JKSVVvbWtpNk9MVlo5SUN6UUppQ3dW?=
 =?utf-8?B?TSs1N0xlL29sSm9TdFlmYnFVODNweWpGRTdrWlNUY1lWN0Q0NWlDWnBmVXJI?=
 =?utf-8?B?cHluYThKVjhDMzNjbklYM28zKzZQempZVmJTcmhZRjBYWEhHVmE0bG5JbG5Y?=
 =?utf-8?B?M1c2WlM4K1hFZ0FFVml3QTl3UzRSem9Ld1NmOE9TOW5jTnZaYkdHZG50dm1a?=
 =?utf-8?B?U21IWmwxZWNoeUVEd1Y4anVYV09xbXpML1NhRnUwTS9YQ2s0TDJDQ1E2WEFW?=
 =?utf-8?B?eWtwTVRrZ3NjUStiZk45WGI5YzBNTFNWQzJmSmQrUTdPMlljN2F1VmRpcTVT?=
 =?utf-8?B?bHlVWVV6Si82b0VGOVBLNjFPcU5pb3RSR2c0b1ZoQStoZ0dzNm5TTDhWTlEv?=
 =?utf-8?B?VjhxL3VoZ2FWcnpTYVkvR3BpenQ4VGVFRlRENGdnSm1pdFgxUGx5YjcxM2o2?=
 =?utf-8?B?MEo1UTdsODM0YnVhN2U5QnBqVk5iWnVMcWloNzQ1em9ySmJ6ZUl3YW8xZkZE?=
 =?utf-8?B?K21zUVh4Q0VETlFPdEUyY2hUVUFxQzdCZHhPY3lmaVBUUTdjeXVKUmx6RHlR?=
 =?utf-8?B?Q3VyN0xHQjBMMzQyaGRpQlVpYnVSbDh6aWgrWWhEb0VMWmRHaTRSbjhNY1RQ?=
 =?utf-8?B?dGdEM3J2T1lIYmFoRW9Ma1hnSlFnV041WUNObmE2eXB2YU1HNFZDbGcxa3dJ?=
 =?utf-8?B?YmJiOTVwT0VlVDQ2eGVHMXh3ZVBHYnRWWjVySTJHT2RyaUpGTTQ2WEkxOVlK?=
 =?utf-8?B?aGhSZnV2Vkw0WWFSRmpFRkxhL3R5Vk9KU1ErQVM1YzZqWDdiWlk5V0tZUnR5?=
 =?utf-8?B?Y1FGbmJGZ0ZsQ3hJWGQySkhNSDFrbE80WXhTSUVFU3ZZdWdSN1NzbExDOFdp?=
 =?utf-8?B?dThBVEkrNUhIbHd6TnlRWjNjMkJXQkxHYWpUbnpROW9UL3h0bTY4M2puc2Uy?=
 =?utf-8?B?c29WeHhhTW5VRklKY3ZqSEoyN1dvUnhyOG1GZ0NnZUdGU3JnUEU2bHVXajkx?=
 =?utf-8?B?d0QyQzlEUUdLTTVGY3d4MTJxWnZ4T1FNYnNkN3FLNEoxQVZzTGFuVnJVbkZE?=
 =?utf-8?Q?ZAagNdxaMzIomiyEiFLHTUmP+?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 46630e71-35b0-402e-e40b-08dafaf8bef0
X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB2767.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jan 2023 15:12:29.0669
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5YwirKHKJzvyC8uITAUHCbdDX+0In+Mf+U6hL+YoXAeJAEUO1i7bYsD5v0IdPPb6M2Du18istrKBv2lyPg0m7Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4896
X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 1/19/2023 11:44 AM, Jason Gunthorpe wrote:
> On Thu, Jan 19, 2023 at 02:54:43AM -0600, Kalra, Ashish wrote:
>> Hello Jason,
>>
>> On 1/13/2023 9:33 AM, Jason Gunthorpe wrote:
>>> On Tue, Jan 10, 2023 at 08:31:34AM -0600, Suravee Suthikulpanit wrote:
>>>> Currently, to detect if a domain is enabled with VFIO support, the driver
>>>> checks if the domain has devices attached and check if the domain type is
>>>> IOMMU_DOMAIN_UNMANAGED.
>>>
>>> NAK
>>>
>>> If you need weird HW specific stuff like this then please implement it
>>> properly in iommufd, not try and randomly guess what things need from
>>> the domain type.
>>>
>>> All this confidential computing stuff needs a comprehensive solution,
>>> not some piecemeal mess. How can you even use a CC guest with VFIO in
>>> the upstream kernel? Hmm?
>>>
>>
>> Currently all guest devices are untrusted - whether they are emulated,
>> virtio or passthrough. In the current use case of VFIO device-passthrough to
>> an SNP guest, the pass-through device will perform DMA to un-encrypted or
>> shared guest memory, in the same way as virtio or emulated devices.
>>
>> This fix is prompted by an issue reported by Nvidia, they are trying to do
>> PCIe device passthrough to SNP guest. The memory allocated for DMA is
>> through dma_alloc_coherent() in the SNP guest and during DMA I/O an
>> RMP_PAGE_FAULT is observed on the host.
>>
>> These dma_alloc_coherent() calls map into page state change hypercalls into
>> the host to change guest page state from encrypted to shared in the RMP
>> table.
>>
>> Following is a link to issue discussed above:
>> https://github.com/AMDESE/AMDSEV/issues/109
> 
> Wow you should really write all of this in the commmit message
> 
>> Now, to set individual 4K entries to different shared/private
>> mappings in NPT or host page tables for large page entries, the RMP
>> and NPT/host page table large page entries are split to 4K pte’s.
> 
> Why are mappings to private pages even in the iommu in the first
> place - and how did they even get there?
>

You seem to be confusing between host/NPT page tables and IOMMU page 
tables.

There are no private page mappings in the IOMMU page tables, as i 
mentioned above currently all DMA to SNP guest is to/from shared memory.

> I thought the design for the private memory was walling it off in a
> memfd and making it un-gup'able?
> 
> This seems to be your actual problem, somehow the iommu is being
> loaded with private memory PFNs instead of only being loaded with
> shared PFNs when shared mappings are created?
> 

The IOMMU page tables are loaded with shared PFNs and not private memory 
PFNs.

> If the IOMMU mappings actually only extend to the legitimate shared
> pages then you don't have a problem with large IOPTEs spanning a
> mixture of page types.
> 
>> The fix is to force 4K page size for IOMMU page tables for SNP guests.
> 
> But even if you want to persue this as the fix, it should not be done
> in this way.
> 
>> This patch-set adds support to detect if a domain belongs to an SNP-enabled
>> guest. This way it can set default page size of a domain to 4K only for
>> SNP-enabled guest and allow non-SNP guest to use larger page size.
> 
> As I said, the KVM has nothing to do with the iommu and I want to
> laregly keep it that way.
> 
> If the VMM needs to request a 4k page size only iommu_domain because
> it is somehow mapping mixtures of private and public pages, 

Again, there is no mixture of private and public pages, the IOMMU only 
has shared page mappings.

Thanks,
Ashish

>then the
> VMM knows it is doing this crazy thing and it needs to ask iommufd
> directly for customized iommu_domain from the driver.
> 
> No KVM interconnection.
> 
> In fact, we already have a way to do this in iommufd generically, have
> the VMM set IOMMU_OPTION_HUGE_PAGES = 0.
> 
> Jason
>