Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <8f85787a-1fa5-8f59-6047-b3a791291bfd@amd.com>
Date:   Wed, 3 May 2023 12:58:05 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.10.0
Subject: Re: [PATCH 0/6] efi/x86: Avoid legacy decompressor during EFI boot
Content-Language: en-US
From:   Tom Lendacky <thomas.lendacky@amd.com>
To:     Ard Biesheuvel <ardb@kernel.org>
Cc:     linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org,
        Evgeniy Baskov <baskov@ispras.ru>,
        Borislav Petkov <bp@alien8.de>,
        Andy Lutomirski <luto@kernel.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Alexey Khoroshilov <khoroshilov@ispras.ru>,
        Peter Jones <pjones@redhat.com>,
        Gerd Hoffmann <kraxel@redhat.com>,
        Dave Young <dyoung@redhat.com>,
        Mario Limonciello <mario.limonciello@amd.com>,
        Kees Cook <keescook@chromium.org>,
        "Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>,
        Linus Torvalds <torvalds@linux-foundation.org>
References: <20230424165726.2245548-1-ardb@kernel.org>
 <ecbcc066-3636-9620-8db9-2c066d16e717@amd.com>
 <CAMj1kXG8ZrdEUNTiSCet+aT+1ZyrbguFsGtNhVV+XgCssZgj9w@mail.gmail.com>
 <a6846282-95c8-274a-accc-ffa54205489c@amd.com>
In-Reply-To: <a6846282-95c8-274a-accc-ffa54205489c@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bElRdE9kMXpTU0dueEY0clovNFRxMlZRcGNQbVhPa0k3dnhic1NJNXd4bWtU?=
 =?utf-8?B?ZHFwdWhpaEwrNkplTFgwZUhuMFdSUCs1VmJrTUp0S1Q5NWhtbXBFdjB1dkxp?=
 =?utf-8?B?WmdDWWpQSTBrcHFjU3B5TUdOY1BhWFJzaWhIbU1wRjRFZ1lXWFltY0ZwTmIy?=
 =?utf-8?B?aTZZdHpla1FHRS9oSFBUOFZjU3pGRjUxaUVxSkxKYnJiNDZScXVRNDltVlkz?=
 =?utf-8?B?R29CMm5oQnZMUmRNaVRLUVNZek1rZmI2YW9lV3J0TUhtV2hKWGRHR1lCTzlo?=
 =?utf-8?B?TUxaeTR6KzBPR0JCdXM2cjFIK1RVVTZwWGhTbHNNN29PN0xIaGdhS0dHV2lo?=
 =?utf-8?B?SDFqZGxPWExsUUlNWXJrbVo3K3BTM3Q4REZPL0t6QndZTlg0SlFuc2pnQTln?=
 =?utf-8?B?OHhhdGkweEdKK2FhNWJVWXRNUmU5VW5FVVd0SG0vc2lTYkNORGFJVVMyM3V5?=
 =?utf-8?B?NVFOQXIyZUVTM0RqK1Byb3BCenNpR1BHTkpTYUVaYVNtMjg1WEs4YlZlTnBh?=
 =?utf-8?B?cnRnMW5QLysxVlY3NGdMb2xod0JVZWZ0MnlkQWZ6T2FkRUFVYm40NUFheE5N?=
 =?utf-8?B?OGwwN3YvejVQeUJDZGpINWlCQkRHczRiWlQ3YklLK2gvRlBGQmhscmY2ais1?=
 =?utf-8?B?ano3N1pvbThvM2xuczRWb1MwOTFCZ3p6WnM3WFVuRmxyOHN6T1dLTVkvN0k5?=
 =?utf-8?B?YnAvcnQwUnVPdkFjVGZHbXJTUGt3bXA4RmdrbWJpbkdrcEJlUjZVS3BFL25R?=
 =?utf-8?B?SHlMUkk2MmFQajJnTW83TVcwVDNkT2Q2ODNnbVAwTU85ZE1xcnd0bStPRUtq?=
 =?utf-8?B?T1hTNFpqTnI5OU8rSERNRzc1NnpKSS9DcXZDclpwaXRiNGsya2JRN0c2c0Zn?=
 =?utf-8?B?RVB3S2tCSy9zc3lOeTZuZjcwb3RrdWZPaVN2MmhSa25seEtkdnFyM3ZjMWtL?=
 =?utf-8?B?NWxKV01ZUStXbUtxc0Zaakpla1lQTmZGclZtZkErNFg4b2M1UFpEZWFVTXBs?=
 =?utf-8?B?eWpUYUtYS05MbG1oOFhZaUJXMTNoVFB6RTdtVitKUXZOM01DNE9nNStPaC9q?=
 =?utf-8?B?ajJoa3JNWFdYVC9BRWdQSTNYQm9ERGd4OFpGNitFZ29aZGsvK2tRbG45UkxK?=
 =?utf-8?B?eitVUDQ2NGJaa1F0bmxFWlE0eldRbHR2ZjZUNmNhT3IwbTMxbWdvWEh0QXZQ?=
 =?utf-8?B?N2h5dFlTYkM2L2llR2t3cWUyczRIdmV3d3RqZlp3YVZGaTd5TlhjbEcvME5C?=
 =?utf-8?B?cnF6T09Ua2srWlgyamlFN3lyR3lha2N3L1VWT09oUGhGcWgxSncwWjcrRlhJ?=
 =?utf-8?B?OE5ZNFNJYU0rSnBRZGV5M0tRQk9zdjZWK2M0cU1FbitRaFExS0pBM2ovRlZB?=
 =?utf-8?B?ZU9zMUhqRUV2eXE5VWExSldiQVdkYWV4YzBsL2dKamdlZ0xNekdyR0FvRDlG?=
 =?utf-8?B?QS9IaWF6WHZjY0pIa1E3YS8xZjduTEtKZU1SMUtzQURseE8rekhSekIvcnU4?=
 =?utf-8?B?aHBzWU5iYWdQQ01Bd0hoZWNJdjQ4emwxNFRTekNOSDBRait2SmlWR3NETDdz?=
 =?utf-8?B?N01ocGJoTXdQdTZYZkxxanVaSHVDT1NrT29TdlBac1VmbytzbkZEaGpXSk5W?=
 =?utf-8?B?L09EajBHUFduR09oT2FxWFFzOXU3RUpSSG5oQWR4RVNPenZuakpmd2g1ZUhV?=
 =?utf-8?B?cE8wZlVBYXEvaHFxSGNMMWMwY1Q5MWRvS1gzWE1ERlZ4ZGxPSG5nRm9QTUpO?=
 =?utf-8?B?c1BaRjRkdWhJOGZCZTVjOG9RTnFEMVB4WTU5V1hiM1BzVTQ1SlZTRjZqaUp0?=
 =?utf-8?B?OE1KTUNuSWZ1cUlUeWpOZlVSc2FGUElYYnpKenRLNjBFTlNTOFVZRU96cEk1?=
 =?utf-8?B?T0gzTEV4M2U5THNtWWRYclkrZXdBVEtBOWhzdFBHa2NhTUozQ29BLzBsbXBu?=
 =?utf-8?B?eUhOUU9Ob0VRNXNMTnVUQlFLNm9EOVI1RVVQaVlDTkVFa2ZkbXoxdHJOdUxy?=
 =?utf-8?B?VWVrSkZKaGZPMUFMVkRvRGo3NUltSFBtVVZVYkd4QUhFKzM2VXFoTFAxT0Ir?=
 =?utf-8?B?T3V3MW5GZG1iTkZ3QjM4RG8wUnZQWG9PcmVOcTkveFY5TW5yMjEyaHpZQmpF?=
 =?utf-8?Q?HwmmvzCpgGF/K+GcyHRzrFAQW?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b5b45b3e-e9bf-467f-727a-08db4bfff3b9
X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5229.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 May 2023 17:58:08.2081
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ubxLUOzVqVr7fI3XvaXT0r07trn2eMV3nWMeaXueY+3pKS6x/GtGUeV0qhW1wWW/m7cBfFcstN9WM3m50rvhlw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR12MB6595
X-Spam-Status: No, score=-5.4 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 5/2/23 11:08, Tom Lendacky wrote:
> On 5/2/23 08:39, Ard Biesheuvel wrote:
>> On Tue, 2 May 2023 at 15:37, Tom Lendacky <thomas.lendacky@amd.com> wrote:
>>>
>>> On 4/24/23 11:57, Ard Biesheuvel wrote:
>>>> This series is conceptually a combination of Evgeny's series [0] and
>>>> mine [1], both of which attempt to make the early decompressor code more
>>>> amenable to executing in the EFI environment with stricter handling of
>>>> memory permissions.
>>>>
>>>> My series [1] implemented zboot for x86, by getting rid of the entire
>>>> x86 decompressor, and replacing it with existing EFI code that does the
>>>> same but in a generic way. The downside of this is that only EFI boot is
>>>> supported, making it unviable for distros, which need to support BIOS
>>>> boot and hybrid EFI boot modes that omit the EFI stub.
>>>>
>>>> Evgeny's series [0] adapted the entire decompressor code flow to allow
>>>> it to execute in the EFI context as well as the bare metal context, and
>>>> this involves changes to the 1:1 mapping code and the page fault
>>>> handlers etc, none of which are really needed when doing EFI boot in the
>>>> first place.
>>>>
>>>> So this series attempts to occupy the middle ground here: it makes
>>>> minimal changes to the existing decompressor so some of it can be called
>>>> from the EFI stub. Then, it reimplements the EFI boot flow to decompress
>>>> the kernel and boot it directly, without relying on the trampoline code,
>>>> page table code or page fault handling code. This allows us to get rid
>>>> of quite a bit of unsavory EFI stub code, and replace it with two clear
>>>> invocations of the EFI firmware APIs to clear NX restrictions from
>>>> allocations that have been populated with executable code.
>>>>
>>>> The only code that is being reused is the decompression library itself,
>>>> along with the minimal ELF parsing that is required to copy the ELF
>>>> segments in place, and the relocation processing that fixes up absolute
>>>> symbol references to refer to the correct virtual addresses.
>>>>
>>>> Note that some of Evgeny's changes to clean up the PE/COFF header
>>>> generation will still be needed, but I've omitted those here for
>>>> brevity.
>>>
>>> I tried booting an SEV and an SEV-ES guest using this and both failed 
>>> to boot:
>>>
>>> EFI stub: WARNING: Decompression failed: Out of memory while allocating
>>> z_stream
>>>
>>> I'll have to take a closer look as to why, but it might be a couple of
>>> days before I can get to it.
>>>
>>
>> Thanks Tom.
>>
>> The internal malloc() seems to be failing, which is often caused by
>> BSS clearing problems. Could you elaborate a little bit on the boot
>> environment you are using here?
> 
> I'm using Qemu v7.2.1 as my VMM, Linux 6.3 with your series applied for my
> host/hypervisor and guest kernel and the current OVMF tree built using
> OvmfPkgX64.dsc.
> 
> I was originally using the current merge window Linux, but moved to the
> release version just to . With the release version SEV and SEV-ES still 
> fail to
> boot, but SEV actually #GPs now. And some of the register contents look
> like encrypted data:
> 
> ConvertPages: range 1000000 - 4FA1FFF covers multiple entries
> !!!! X64 Exception Type - 0D(#GP - General Protection)  CPU Apic ID - 
> 00000000 !!!!
> ExceptionData - 0000000000000000
> RIP  - 00000000597E71C1, CS  - 0000000000000038, RFLAGS - 0000000000210206
> RAX  - 1FBA02A45943B920, RCX - 0000000000AF7009, RDX - A9DAE761B64A1F1B
> RBX  - 1FBA02A45943B8C0, RSP - 000000007FD97320, RBP - 0000000002000000
> RSI  - 0000000001000000, RDI - 1FBA02A45943DE68
> R8   - 0000000003EF3C94, R9  - 0000000000000000, R10 - 000000007D7C6018
> R11  - 0000000000000000, R12 - 0000000001000000, R13 - 00000000597EDD98
> R14  - 0000000001000000, R15 - 000000007E0A5198
> DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
> GS   - 0000000000000030, SS  - 0000000000000030
> CR0  - 0000000080010033, CR2 - 0000000000000000, CR3 - 000000007FA01000
> CR4  - 0000000000000668, CR8 - 0000000000000000
> DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
> DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
> GDTR - 000000007F7DC000 0000000000000047, LDTR - 0000000000000000
> IDTR - 000000007F34C018 0000000000000FFF,   TR - 0000000000000000
> FXSAVE_STATE - 000000007FD96F80
> !!!! Find image based on IP(0x597E71C1) 
> /root/kernels/ovmf-build-X64/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Universal/Variable/RuntimeDxe/VariableRuntimeDxe/DEBUG/Variable
> RuntimeDxe.dll (ImageBase=0000000000D4792C, EntryPoint=0000000000D50CC3) !!!!
> 
> So, yes, probably an area of memory that was zeroes when mapped
> unencrypted, but wasn't cleared after changing the mapping to
> encrypted.

Yes, looks like a bss clearing issue. If I add __section(".data") to 
free_mem_ptr and free_mem_end_ptr in arch/x86/boot/compressed/misc.c and 
to malloc_ptr and malloc_cnt in include/linux/decompress/mm.h, then I can 
boot an SEV guest.

However, an SEV-ES guest is triple faulting. This looks to be because 
we're still on the EFI CS of 0x38 after switching GDTs in 
arch/x86/kernel/head_64.S by calling startup_64_setup_env(). Before 
switching to the kernel CS, we take a #VC (from CPUID calls in sme_enable) 
and things blow up on the iretq. Moving the block headed by the comment 
"Now switch to __KERNEL_CS so IRET works reliably" to just after calling 
startup_64_setup_env() fixes it and an SEV-ES guest can boot.

This worked before because I believe we switched off the EFI CS as part of 
the kernel decompressor support and so this bug wasn't exposed. But this 
needs to be fixed regardless of this series.

Thanks,
Tom

> 
> Thanks,
> Tom
> 
>