Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <1dfaf07e-040e-848b-db7c-86a107fd5cb3@amd.com>
Date:   Mon, 16 May 2022 19:27:51 +0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.9.0
Subject: Re: [PATCH v2] iommu/amd: Set translation valid bit only when IO page
 tables are in used
Content-Language: en-US
To:     Joerg Roedel <joro@8bytes.org>
Cc:     linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org,
        vasant.hegde@amd.com, jon.grimm@amd.com
References: <20220509074815.11881-1-suravee.suthikulpanit@amd.com>
 <Yn5YIfzmerwAM30M@8bytes.org>
From:   Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
In-Reply-To: <Yn5YIfzmerwAM30M@8bytes.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VDBXaVI4YWM5anJyOGpQaGE5WGFsajZsZy9paC9hRlpCK0FEbVlhSHcyOGhn?=
 =?utf-8?B?QU56bG8ycU1HNk1DVzJNNTBQOU96N0VTMFhPOXdpRmlOdTBSOXJBeXlrNldS?=
 =?utf-8?B?UUJ5Z05DTXVFTHB0TDkydDF3WmpsOE9BU1BrYThBbWhXNm5BdXdKMDFSSnU3?=
 =?utf-8?B?TXZST3FMdFQ3Zy9adDFOK3Z1STY1Tk9nSHZMMGsyQnQ0bGd6cjlPUVlUNG5O?=
 =?utf-8?B?VUIzbTJ1Y3dzd3d4SWhNeDVNalRZWnRQczhIdGdqUGhicFNQMStieUg5U0Mz?=
 =?utf-8?B?U25MdFU0UmQrQUl5ZVZ1WkcwQVdub0Fqd1Z1RS9zd3ZCVm1tSXgzamR2d0h3?=
 =?utf-8?B?aHhwMkZLQ2s3NVZPV2tLV1pKOHptVUQwajhLOHorYzUxQkM3aXZJd0tuOHg4?=
 =?utf-8?B?OHhJVi9jM0xpWEFNQjFlM0NmMnFCYWdnb2V6VDRTTVpkOHBybmEweFdOYkJP?=
 =?utf-8?B?Y3pWTjdTSDN4T2ZWbXpqVG5MTGdVdTFsNnZmVlZUcGU2dXFyY29IcUk4Zk9i?=
 =?utf-8?B?cTIyaGZrUXJkTEhPOXhWdDZFcTNPVVBINlNlaDFBOXdNaE9MaVlYemUzREZJ?=
 =?utf-8?B?MmxJamNGSDlCbDNNVldlbU8xcE1YRjAwKzNlbG1RNU1BZDJ5bGo3dE0ya0k1?=
 =?utf-8?B?aS9ndTN1SE1TT2RJRHBsYXpNUXptbmRxWXpjSlV5Z01EaE91Q1l6U2lKU1Zl?=
 =?utf-8?B?b0c1bWEvUmhLNXppS3d6OGgxT1Q4eUcvbE41TzYzQlVwSVJwVDdoenVVSW5B?=
 =?utf-8?B?ZEF5RVdBNTN2L282MnNzWG91YjcxelFGZUVnM2RWTWVoemE1OXJJaGJnSVZk?=
 =?utf-8?B?eWJGcm5yRHBORnZMNVlJRi9tc0FRbU4zZDBsL21hRjdNeWFWcXZsWmhyVXJx?=
 =?utf-8?B?Q0FUdk8vVkNQU3BoZzNRVmxRazBJNmtNc3FwVTJ5UnNvZzVEUERxRWpIcXJ0?=
 =?utf-8?B?QkxIbmlhVHZaeGVFY3hldldxMm9tVnk1cjV4dzJNT2JBdHljV2l2UlNMdVUr?=
 =?utf-8?B?U1VuSmtscWZPUFkwVnl5WHlFKzZhQkgvcU9QQ2U2RjdwK25LQlM5VjRjZUsz?=
 =?utf-8?B?YndFRUdNejNnSTVpbnlpN0hXV3F2WEVaYXZjcFQyOWpTZVhKVlZTNWRWMmg2?=
 =?utf-8?B?VjRwSi9qVVkyTkNIMGNrYkJGL1hickdPNXBEaW9UTUZ4dFFwcHQxNjZnajRt?=
 =?utf-8?B?Y0JBS2NqcVE2ZDBUczhNMkVzM2N6OUZqUXJQc0dqU3NOeEVyUXZxYktKZTdN?=
 =?utf-8?B?eEVVeE5nZmhiZTlnK1hSMEJzRHZQaVZ5UUlqTzBlRzAvVjkvNUVId3hNWmJr?=
 =?utf-8?B?MnpaTlRZVlErTExYT1lPZUpkMXErdEd6amNJaHhRSEV1UFZnS01QNkJNT0Zk?=
 =?utf-8?B?N1kwUnU2RXRwbU15MVpuT1Fia29yQ04wNVR5N1kwR295WFF1MVdMc2l4MUJ5?=
 =?utf-8?B?dHFSS0RqSGxYWTRUZ3c1TTV6blVVWmhVVjVOOVl2THI2b0J5ajV0UTZOS1NK?=
 =?utf-8?B?QjBhL2tGOHNOaDIzYzBYamVpemtZUm9pcmpsdjdMYVZ4Z2VBZ3J6dHVLQjR2?=
 =?utf-8?B?eUQrQmRsRlg1S2I3TUVXMGVaTWQ2RzVKcnRCODBaOG44b1NhaXQrNW9pYnlK?=
 =?utf-8?B?OWc2Q0l0eXVSWVYxS1ViTy9rR2VXSGY5aFltajVYYnAvV3pIZVBLU0FBZ1or?=
 =?utf-8?B?SFp2dnhmc2dHRW9qanJBV1V0S1M5UUl2VlBzZGorTE92V0w4RCthSEFFUUh2?=
 =?utf-8?B?VEdmMVh4SVRjWG05RXhSK05UM0RDZk16Y2l2RnBXTGxmT25MTmpxbjdjSklw?=
 =?utf-8?B?b2hnL1JTcWU5WGhjTzZZQUMycDNFM3ZuS2o3akhkZm81OFVaNXJBVjVMUGlq?=
 =?utf-8?B?NjFnSEV4VU1zZjh4NUkrc0NHbGJUYzhRbTV3di9KckZtMGE1b291TEFNOTcw?=
 =?utf-8?B?RGRzc0hkM3Bud3RYRzloUTFmeTBXREVPUWdBYWxRK0VTbDlGWEtaRTBiT1FD?=
 =?utf-8?B?U2VGamI3RzNURjc0M1kvVmZXVG1zUVBjcWZCMWp5M1pEUWFSOVhoaHJKMG9K?=
 =?utf-8?B?S3Q1dnJQaVp5a3BpQkNhbGZ5YzVxaTNmM2E1eTJ1SjdJbkZURUF1QUxLWWJ4?=
 =?utf-8?B?YlQ3ZTAxSFFGdCtSVjdkdFpkdkY3OHZZaEFFRk5OVElzQzYyUytNUy9admQv?=
 =?utf-8?B?WktOV3ZrN0llcGNsTHkwa3M5US9Bb2RySS9DckluZTRWK1NNNkVaUjhicFYx?=
 =?utf-8?B?Qy9GdWVDWFBxQ0txY1pXUkRaYUYreVdDMzUvNXBPMTlFeWU4ZWxFN2ppUEJm?=
 =?utf-8?B?d2VCN1hxbHV4TFhORng0N0FlV1FIWmMzdG9ZZk9FYWtRTEprbzU1Zz09?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0b4da5bb-5f8b-4809-44bb-08da37378281
X-MS-Exchange-CrossTenant-AuthSource: DM8PR12MB5445.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2022 12:27:58.1457
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GWZ0fYpY5Klo5RWztONGyzlO4lQhIieqkW0/wvhgR9ACn5HUsrjCmyw7bTND/bZqidv5wj5dH98fAE+11genCQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5031
X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Joerg,

On 5/13/22 8:07 PM, Joerg Roedel wrote:
> On Mon, May 09, 2022 at 02:48:15AM -0500, Suravee Suthikulpanit wrote:
>> On AMD system with SNP enabled, IOMMU hardware checks the host translation
>> valid (TV) and guest translation valid (GV) bits in the device
>> table entry (DTE) before accessing the corresponded page tables.
>>
>> However, current IOMMU driver sets the TV bit for all devices
>> regardless of whether the host page table is in used.
>> This results in ILLEGAL_DEV_TABLE_ENTRY event for devices, which
>> do not the host page table root pointer set up.
> 
> Hmm, this sound weird. In the early AMD IOMMUs it was recommended to set
> TV=1 and V=1 and the rest to 0 to block all DMA from a device.
>
> I wonder how this triggers ILLEGAL_DEV_TABLE_ENTRY errors now. It is
> (was?) legal to set V=1 TV=1, mode=0 and leave the page-table empty.

Due to the new restriction (please see the IOMMU spec Rev 3.06-PUB - Apr 2021
https://www.amd.com/system/files/TechDocs/48882_IOMMU.pdf) where the use of
DTE[Mode]=0 is not supported on systems that are SNP-enabled (i.e. EFR[SNPSup]=1),
the IOMMU HW looks at the DTE[TV] bit to determine if it needs to handle the v1 page table.
When the HW encounters DTE entry with TV=1, V=1, Mode=0, it would generate
ILLEGAL_DEV_TABLE_ENTRY event.

Note: I am following up with HW folks for the updated document for this
specific detail.

Therefore, we need to modify IOMMU driver as following:

- For non-DMA devices (e.g. the IOAPIC devices), we need to
modify IOMMU driver to default to DTE[TV]=0. For Linux, this is equivalent
to DTE with domain ID 0.

- I am still trying to see what is the best way to force Linux to not allow
Mode=0 (i.e. iommu=pt mode). Any thoughts?

- Also, it seems that the current iommu v2 page table use case, where GVA->GPA=SPA
will no longer be supported on system w/ SNPSup=1. Any thoughts?

> When then IW=0 and IR=0, DMA is blocked. From what I remember this is a
> valid setting in a DTE.

Correct.

> Do you have an example DTE which triggers this error message?

This is specifically from the device representing an IOAPIC.

[  +0.000108] iommu ivhd0: AMD-Vi: Event logged [ILLEGAL_DEV_TABLE_ENTRY device=c0:00.1 pasid=0x00000 
address=0xfffffffdf8140000 flags=0x0008]
[  +0.000011] AMD-Vi: DTE[0]: 0000000000000003
[  +0.000003] AMD-Vi: DTE[1]: 0000000000000000
[  +0.000002] AMD-Vi: DTE[2]: 2008000100258013
[  +0.000001] AMD-Vi: DTE[3]: 0000000000000000

Best Regards,
Suravee