Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BED9CC76196 for ; Thu, 16 Mar 2023 13:53:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230416AbjCPNxy (ORCPT ); Thu, 16 Mar 2023 09:53:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58094 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230446AbjCPNxn (ORCPT ); Thu, 16 Mar 2023 09:53:43 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94236B5FDC; Thu, 16 Mar 2023 06:53:40 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 48C06B8219C; Thu, 16 Mar 2023 13:53:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F325AC4339B; Thu, 16 Mar 2023 13:53:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678974818; bh=ZFCg1R9x3TYU2MIKEu2nraQz7Cm1ZbqwW3c+AwfOoRs=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=sP0YqmZZDv7gq2cfQr+7tzUpA/rkkSJR9Xpu3AfU2joplN6rprXlpO9j64C1MiOSp a2gHHXkDJfNhMZrjPIYDkbVslyXsQ6p18ObepWz2JYUn7UP5B+efuLQxhGlLwgYVBO yPY0vBZ6tI/r6ZQ/MYZhdRksWXwBb/LQzxWISSD1BY2MPb/R+gmUeFRS0kUkZEDshQ uFDTzB17bWZbjkpxHIguHzv46ki7fGkU7HhqV4xkmKZDtmvARxV8VWDrDrcAJHbQZN zEeDFteffNMXqjlgpWOls8BKxorwVUHYTQ+bVY2VReLgquV30igkqPe89W2tfXUeRG ovkAzfEvmHZ6Q== Received: by mail-lf1-f41.google.com with SMTP id bi9so2443794lfb.12; Thu, 16 Mar 2023 06:53:37 -0700 (PDT) X-Gm-Message-State: AO0yUKWVNhnAMXdxLpEWgK3R6xNyCnCyFR1bbfX/gjNhd+lr3oSld+rn FE7Fs9rDMPj+jyhVfnOC7jX7iwnG8l1p5z4RH8c= X-Google-Smtp-Source: AK7set8aEVEP9Pqpoe8z0WN3pnvwA4tRVznAzzFiJbL5STPsavVMZe7yVpfRpYnltjdW55wDMKEycSwTyVwtTRI5UVk= X-Received: by 2002:a05:6512:304c:b0:4dc:7e56:9839 with SMTP id b12-20020a056512304c00b004dc7e569839mr4911045lfb.5.1678974816034; Thu, 16 Mar 2023 06:53:36 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ard Biesheuvel Date: Thu, 16 Mar 2023 14:53:24 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: kernel 6.2 stuck at boot (efi_call_rts) on arm64 To: Andrea Righi Cc: "Jason A. Donenfeld" , Paolo Pisati , linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, Darren Hart Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 16 Mar 2023 at 14:50, Andrea Righi wrote: > > On Thu, Mar 16, 2023 at 02:45:49PM +0100, Ard Biesheuvel wrote: > > On Thu, 16 Mar 2023 at 13:50, Andrea Righi wrote: > > > > > > On Thu, Mar 16, 2023 at 01:43:32PM +0100, Ard Biesheuvel wrote: > > > > On Thu, 16 Mar 2023 at 13:41, Andrea Righi wrote: > > > > > > > > > > On Thu, Mar 16, 2023 at 01:38:30PM +0100, Ard Biesheuvel wrote: > > > > > > On Thu, 16 Mar 2023 at 13:21, Ard Biesheuvel wrote: > > > > > > > > > > > > > > On Thu, 16 Mar 2023 at 12:34, Andrea Righi wrote: > > > > > > > > > > > > > > > > On Thu, Mar 16, 2023 at 11:18:21AM +0100, Ard Biesheuvel wrote: > > > > > > > > > On Thu, 16 Mar 2023 at 11:03, Andrea Righi wrote: > > > > > > > > > > > > > > > > > > > > On Thu, Mar 16, 2023 at 10:55:58AM +0100, Ard Biesheuvel wrote: > > > > > > > > > > > (cc Darren) > > > > > > > > > > > > > > > > > > > > > > On Thu, 16 Mar 2023 at 10:45, Andrea Righi wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 16, 2023 at 08:58:20AM +0100, Ard Biesheuvel wrote: > > > > > > > > > > > > > Hello Andrea, > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, 16 Mar 2023 at 08:54, Andrea Righi wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > > > the latest v6.2.6 kernel fails to boot on some arm64 systems, the kernel > > > > > > > > > > > > > > gets stuck and never completes the boot. On the console I see this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > [ 72.043484] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: > > > > > > > > > > > > > > [ 72.049571] rcu: 22-...0: (30 GPs behind) idle=b10c/1/0x4000000000000000 softirq=164/164 fqs=6443 > > > > > > > > > > > > > > [ 72.058520] (detected by 28, t=15005 jiffies, g=449, q=174 ncpus=32) > > > > > > > > > > > > > > [ 72.064949] Task dump for CPU 22: > > > > > > > > > > > > > > [ 72.068251] task:kworker/u64:5 state:R running task stack:0 pid:447 ppid:2 flags:0x0000000a > > > > > > > > > > > > > > [ 72.078156] Workqueue: efi_rts_wq efi_call_rts > > > > > > > > > > > > > > [ 72.082595] Call trace: > > > > > > > > > > > > > > [ 72.085029] __switch_to+0xbc/0x100 > > > > > > > > > > > > > > [ 72.088508] 0xffff80000fe83d4c > > > > > > > > > > > > > > > > > > > > > > > > > > > > After that, as a consequence, I start to get a lot of hung task timeout traces. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I tried to bisect the problem and I found that the offending commit is > > > > > > > > > > > > > > this one: > > > > > > > > > > > > > > > > > > > > > > > > > > > > e7b813b32a42 ("efi: random: refresh non-volatile random seed when RNG is initialized") > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've reverted this commit for now and everything works just fine, but I > > > > > > > > > > > > > > was wondering if the problem could be caused by a lack of entropy on > > > > > > > > > > > > > > these arm64 boxes or something else. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any suggestion? Let me know if you want me to do any specific test. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the report. > > > > > > > > > > > > > > > > > > > > > > > > > > This is most likely the EFI SetVariable() call going off into the > > > > > > > > > > > > > weeds and never returning. > > > > > > > > > > > > > > > > > > > > > > > > > > Is this an Ampere Altra system by any chance? Do you see it on > > > > > > > > > > > > > different types of hardware? > > > > > > > > > > > > > > > > > > > > > > > > This is: Ampere eMAG / Lenovo ThinkSystem HR330a. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Could you check whether SetVariable works on this system? E.g. by > > > > > > > > > > > > > updating the EFI boot timeout (sudo efibootmgr -t )? > > > > > > > > > > > > > > > > > > > > > > > > ubuntu@kuzzle:~$ sudo efibootmgr -t 10 > > > > > > > > > > > > ^C^C^C^C > > > > > > > > > > > > > > > > > > > > > > > > ^ Stuck there, so it really looks like SetVariable is the problem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Could you please share the output of > > > > > > > > > > > > > > > > > > > > > > dmidecode -s bios > > > > > > > > > > > dmidecode -s system-family > > > > > > > > > > > > > > > > > > > > $ sudo dmidecode -s bios-vendor > > > > > > > > > > LENOVO > > > > > > > > > > $ sudo dmidecode -s bios-version > > > > > > > > > > hve104r-1.15 > > > > > > > > > > $ sudo dmidecode -s bios-release-date > > > > > > > > > > 02/26/2021 > > > > > > > > > > $ sudo dmidecode -s bios-revision > > > > > > > > > > 1.15 > > > > > > > > > > $ sudo dmidecode -s system-family > > > > > > > > > > Lenovo ThinkSystem HR330A/HR350A > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > Mind checking if this patch fixes your issue as well? > > > > > > > > > > > > > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/commit/?h=altra-fix&id=77fa99dd4741456da85049c13ec31a148f5f5ac0 > > > > > > > > > > > > > > > > Unfortunately this doesn't seem to be enough, I'm still getting the same > > > > > > > > problem also with this patch applied. > > > > > > > > > > > > > > > > > > > > > > Thanks for trying. > > > > > > > > > > > > > > How about the last 3 patches on this branch? > > > > > > > > > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-smbios-altra-fix > > > > > > > > > > > > Actually, that may not match your hardware. > > > > > > > > > > > > Does your kernel log have a line like > > > > > > > > > > > > SMCCC: SOC_ID: ID = jep106:036b:0019 Revision = 0x00000102 > > > > > > > > > > > > ? > > > > > > > > > > $ sudo dmesg | grep "SMCCC: SOC_ID" > > > > > [ 5.320782] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping .... > > > > > > > > > > > > > Thanks. Could you share the entire dmidecode output somewhere? Or at > > > > least the type 4 record(s)? > > > > > > Sure, here's the full output of dmidecode: > > > https://pastebin.ubuntu.com/p/4ZmKmP2xTm/ > > > > > > > Thanks. I have updated my SMBIOS patches to take the processor version > > 'eMAG' into account, which appears to be what these boxes are using. > > > > I have updated the efi/urgent branch here with the latest versions. > > Mind giving them a spin? > > > > > > In the mean time, just for the record - could you please run this as well? > > > > hexdump -C /sys/firmware/dmi/entries/4-0/raw > > > > (as root) > > hm.. I don't have that in /sys/firmware/, this is what I have: > > # ls -l /sys/firmware/dmi/ > total 0 > drwxr-xr-x 2 root root 0 Mar 16 13:26 tables > # ls -l /sys/firmware/dmi/tables/ > total 0 > -r-------- 1 root root 5004 Mar 16 13:26 DMI > -r-------- 1 root root 24 Mar 16 13:26 smbios_entry_point > You'll need to load the dmi_sysfs module for that. But no big deal otherwise, I'm pretty sure the word order is the correct on on your system in any case (it decodes the value correctly in the next line)