Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <daaf7a84-4204-48ca-e40c-7ba296b4789c@amd.com>
Date:   Tue, 14 Jun 2022 11:01:55 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.1
Subject: Re: [PATCH v12 19/46] x86/kernel: Make the .bss..decrypted section
 shared in RMP table
Content-Language: en-US
To:     Sean Christopherson <seanjc@google.com>,
        Michael Roth <michael.roth@amd.com>
Cc:     x86@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
        linux-efi@vger.kernel.org, platform-driver-x86@vger.kernel.org,
        linux-coco@lists.linux.dev, linux-mm@kvack.org,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Joerg Roedel <jroedel@suse.de>,
        "H. Peter Anvin" <hpa@zytor.com>, Ard Biesheuvel <ardb@kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Jim Mattson <jmattson@google.com>,
        Andy Lutomirski <luto@kernel.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Sergio Lopez <slp@redhat.com>, Peter Gonda <pgonda@google.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>,
        David Rientjes <rientjes@google.com>,
        Dov Murik <dovmurik@linux.ibm.com>,
        Tobin Feldman-Fitzthum <tobin@ibm.com>,
        Borislav Petkov <bp@alien8.de>,
        Vlastimil Babka <vbabka@suse.cz>,
        "Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>,
        Andi Kleen <ak@linux.intel.com>,
        "Dr . David Alan Gilbert" <dgilbert@redhat.com>,
        brijesh.ksingh@gmail.com, tony.luck@intel.com, marcorr@google.com,
        sathyanarayanan.kuppuswamy@linux.intel.com
References: <20220307213356.2797205-1-brijesh.singh@amd.com>
 <20220307213356.2797205-20-brijesh.singh@amd.com>
 <YqfabnTRxFSM+LoX@google.com> <YqistMvngNKEJu2o@google.com>
From:   Tom Lendacky <thomas.lendacky@amd.com>
In-Reply-To: <YqistMvngNKEJu2o@google.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?a0FiTG9oN0dmRldDSlBDNDJjT1g0Q3VoeFdESXVKclJ0ZUk4c0o2b3lwQXRY?=
 =?utf-8?B?TnJKSko2TlNVUHhuTG5yaGVRZk5ZcnFsZmxWZE1BaHZ0MW5sNjVnejZGM25t?=
 =?utf-8?B?QmQ3TmlwRlZsSlpjNjRNa2lsZFV0SHBHWWwxcm1nTUQxQ2d1WGNxdWhHVUN3?=
 =?utf-8?B?MFdEMldSLzh4dGxPMkUzN04wWFJsMTd1MlBSSVQ2bUp5NE1IK05QcERHSlpI?=
 =?utf-8?B?Rzc0OUd2d1pEL3greWZtQmJWOGNvTmNCNUtoUjVuMUlteXowVGFMTWtvNEpS?=
 =?utf-8?B?OUtDYklwNis4VHFoczdYU3dKN3Z0UlZTcFBXRDJia1IzRHk2NkxaeHdDVHov?=
 =?utf-8?B?NzlWNzVkWkpIOGV1KzV3YTMzQTRvQmhqbVZvb1BDcmFPaWpvejJBaC9Zakxn?=
 =?utf-8?B?bmhEVnNVekdrSzBqTEExTkJ2NWNTQ1lRa0NGbUFNNWZPeUwxeVRuZzZKQ0cz?=
 =?utf-8?B?aWV3NEZyMFVoL0Vvd3ZEM0hQRFl4a1h2Q1JsZWNVejRUbytJa29PUUdqT2My?=
 =?utf-8?B?RHJQZnhMT2Q1cHU5NWQ3VDRKWjJmZ2R6TElEakIvOHFNZjhFTkp2eGZNb1J3?=
 =?utf-8?B?Y1F1c2kzQTJFNGpIdlZVdUY0d3VDbHg2a3FQZUtEY3JIYnhzTmtuV2ZMcWp6?=
 =?utf-8?B?NUdoV2xiekJPb1NBZnlOOWZGZG1IckJjcHZiNThNQXB2MHgxNEZMb2lFZ3lZ?=
 =?utf-8?B?UUNwZ2x2WUc4TGIrdkZCaW91enZDb1RMQ0pDUUJvY2kxd2ppV1QycE9qZWxX?=
 =?utf-8?B?UUxudndwaVNVS2d1czVHc0UzTkRQQzZtT1J2ZjZJRGRidk5YNm8zMWhDZzdo?=
 =?utf-8?B?cHp0ZnV0YzVxMUxpc1Z6c0lwcklmNnlGSTl3Y0Q0akxVVlY1dC85OGZ1V01M?=
 =?utf-8?B?eUlVYzJpalVrcWlSczBSdkhGTDlDVE5MRHF2QUsrRWVjZFJCeGdEYlpwRlQv?=
 =?utf-8?B?Qml2UjVmc1hvcVpTTHZPUVduVVQzTjhGdmRrOEZOWjhBeExQdlY5Q1JiS0pT?=
 =?utf-8?B?N0dLc3AzZ1AxdS9qVmVzNkp5b2d3R3QzUDFMR2FiT2dWR0ZXV2w0WVBLTmZC?=
 =?utf-8?B?dm80UzNqbXhpMUZsTTFjSVRKTzNpa01Uek9kQkUyQStEaTZ0cERKQzR1b2ov?=
 =?utf-8?B?WEpRZE5RSUJMT2JLTkZ3QTl2cTg3aTNXYTBnTVZuaFo0dlBPcitSYVdlVGVY?=
 =?utf-8?B?Tlh4QnF3ZWZ0QjJhU0pYRnZJcEM5ekdla3NwWDQ0TXZqc1NuQmdCWHlobGg5?=
 =?utf-8?B?VnJ1TnNFNXdpWGdsb3UzL0p5SFJ2eFptUWpmcTBmc0cyZitrV0puSTZYRlh4?=
 =?utf-8?B?b2JVK1dHcEgxUWZoMUN4T2phVEVJL1A5MW9ybzBxRmFBdGFzOG5Ed1hka2R3?=
 =?utf-8?B?YUlIM3R2SkVsUFFnVUdnV3c3c2Z0MXE1UEdEV0ZHb2haaWRDYnoyNDBPazNj?=
 =?utf-8?B?S1BMTytqNVBaTnl4RkoxNmh3QWVUbmdCenM0eXIzN3dIQjBCOURtR3JiZlBp?=
 =?utf-8?B?UXBlZHZNTU94R2cyZlloajFqVEc2UFhDcVBrSlo5N3Z2UGpTcE5lQzZNNHB6?=
 =?utf-8?B?cjVHc1lEN0VRWjVjS2NjbWZHTTlmM2RUMjRNWGNZYlQrbG83aFE0S3JWYisy?=
 =?utf-8?B?NExJeHNLdjRZQXlyK080UUJveHBuV3JZT0YrUEVRalBna08xWHBPY3ZnTnls?=
 =?utf-8?B?Y1dWT0loS29QaS93TE4rdFczM3lxbDVvZ1lOTmErdFRqem1UNFkzS3VXZy9x?=
 =?utf-8?B?MEtJOHY3Qjc5Z3BVdDROelltTm9hZlJpekh4QnFwNTdEMWVBSFo2Y3hVd2xE?=
 =?utf-8?B?akxVcUlhTzd2N1dnVXVHM0g4Qk9UdElKblc2cndPOGNXbTVmL0tMLzZvUExh?=
 =?utf-8?B?WHpJMExneGFlcW80YjRicHZJMWdzU1Z0MEptc1FkV1VicU9EZ2lnS1h3Y2xD?=
 =?utf-8?B?eWIyZDlDZGpmSzhUQU1GbFlweDUzSytlNWtuV1B4V3ZKaDI2T3dBS291c21D?=
 =?utf-8?B?YURhL3FEL1FGWVFPZEcreVNLWDJNZ1pLZ0dFbDhjVkl1Q0F6OG9xZURUWFJ6?=
 =?utf-8?B?czJsRFUzeWQrWkVtdGVHUGZKTURBUHNnU3FrOGRIdFFWcWZjSHZxUEJTKy9k?=
 =?utf-8?B?VjdjUU9GNXVyV1NiTHZ4VmV3TTgxREIzekJXL0FxTDJhamdndENwa3c0VHRP?=
 =?utf-8?B?YWVkL0NML2xKVi9hanFXcEdqbHhSVDc1K0pRTFZ3clBDdDBEaW5KTTVRTzJU?=
 =?utf-8?B?UlB0Q3BKTFR2UmZxbU9SMTdyVDFJWG11NkZPUElqdGcvV0txcjJ2TTBNZzBP?=
 =?utf-8?B?Yy9laDJWcUZyS3VrQ0pVd1hlL1ZoWkIrZVNYRDdBemxlYkVoQk5VQT09?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 35a75ce2-192e-46fa-9f50-08da4e1f3748
X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5229.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jun 2022 16:02:00.6680
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: BzpN+PIMMrIGRP/h2pjnY23qVAu2OPoUInpQW54CylmNGOcISuGzkp6n8r79mDaXGlwghxMoYmb3GbfNexd/qQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5221
X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 6/14/22 10:43, Sean Christopherson wrote:
> On Tue, Jun 14, 2022, Sean Christopherson wrote:
>> s/Brijesh/Michael
>>
>> On Mon, Mar 07, 2022, Brijesh Singh wrote:
>>> The encryption attribute for the .bss..decrypted section is cleared in the
>>> initial page table build. This is because the section contains the data
>>> that need to be shared between the guest and the hypervisor.
>>>
>>> When SEV-SNP is active, just clearing the encryption attribute in the
>>> page table is not enough. The page state need to be updated in the RMP
>>> table.
>>>
>>> Signed-off-by: Brijesh Singh <brijesh.singh@amd.com>
>>> ---
>>>   arch/x86/kernel/head64.c | 13 +++++++++++++
>>>   1 file changed, 13 insertions(+)
>>>
>>> diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
>>> index 83514b9827e6..656d2f3e2cf0 100644
>>> --- a/arch/x86/kernel/head64.c
>>> +++ b/arch/x86/kernel/head64.c
>>> @@ -143,7 +143,20 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdv
>>>   	if (sme_get_me_mask()) {
>>>   		vaddr = (unsigned long)__start_bss_decrypted;
>>>   		vaddr_end = (unsigned long)__end_bss_decrypted;
>>> +
>>>   		for (; vaddr < vaddr_end; vaddr += PMD_SIZE) {
>>> +			/*
>>> +			 * On SNP, transition the page to shared in the RMP table so that
>>> +			 * it is consistent with the page table attribute change.
>>> +			 *
>>> +			 * __start_bss_decrypted has a virtual address in the high range
>>> +			 * mapping (kernel .text). PVALIDATE, by way of
>>> +			 * early_snp_set_memory_shared(), requires a valid virtual
>>> +			 * address but the kernel is currently running off of the identity
>>> +			 * mapping so use __pa() to get a *currently* valid virtual address.
>>> +			 */
>>> +			early_snp_set_memory_shared(__pa(vaddr), __pa(vaddr), PTRS_PER_PMD);
>>
>> This breaks SME on Rome and Milan when compiling with clang-13.  I haven't been
>> able to figure out exactly what goes wrong.  printk isn't functional at this point,
>> and interactive debug during boot on our test systems is beyond me.  I can't even
>> verify that the bug is specific to clang because the draconian build system for our
>> test systems apparently is stuck pointing at gcc-4.9.
>>
>> I suspect the issue is related to relocation and/or encrypting memory, as skipping
>> the call to early_snp_set_memory_shared() if SNP isn't active masks the issue.
>> I've dug through the assembly and haven't spotted a smoking gun, e.g. no obvious
>> use of absolute addresses.
>>
>> Forcing a VM through the same path doesn't fail.  I can't test an SEV guest at the
>> moment because INIT_EX is also broken.
> 
> The SEV INIT_EX was a PEBKAC issue.  An SEV guest boots just fine with a clang-built
> kernel, so either it's a finnicky relocation issue or something specific to SME.

I just built and booted 5.19-rc2 with clang-13 and SME enabled without issue:

[    4.118226] Memory Encryption Features active: AMD SME

Maybe something with your kernel config? Can you send me your config?

Thanks,
Tom

> 
>> The crash incurs a very, very slow reboot, and I was out of cycles to work on this
>> about three hours ago.  If someone on the AMD side can repro, it would be much
>> appreciated.