Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
MIME-Version: 1.0
References: <1652860121-24092-1-git-send-email-quic_vivekuma@quicinc.com>
In-Reply-To: <1652860121-24092-1-git-send-email-quic_vivekuma@quicinc.com>
From:   Pasha Tatashin <pasha.tatashin@soleen.com>
Date:   Wed, 18 May 2022 12:55:18 -0400
Message-ID: <CA+CK2bDDphX6ANu_=8SPTy7gbB5aajtE3uVLGZrpaczZQKw4QA@mail.gmail.com>
Subject: Re: [RFC 0/6] Bootloader based hibernation
To:     Vivek Kumar <quic_vivekuma@quicinc.com>
Cc:     Jonathan Corbet <corbet@lwn.net>,
        Catalin Marinas <catalin.marinas@arm.com>,
        Will Deacon <will@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Marc Zyngier <maz@kernel.org>, Jens Axboe <axboe@kernel.dk>,
        "Rafael J. Wysocki" <rafael@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Linux Doc Mailing List <linux-doc@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Linux ARM <linux-arm-kernel@lists.infradead.org>,
        linux-block <linux-block@vger.kernel.org>,
        linux-pm@vger.kernel.org, linux-mm <linux-mm@kvack.org>,
        len.brown@intel.com, Pavel Machek <pavel@ucw.cz>,
        paulmck@kernel.org, Borislav Petkov <bp@suse.de>,
        Kees Cook <keescook@chromium.org>,
        Muchun Song <songmuchun@bytedance.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        damien.lemoal@opensource.wdc.com, Fuad Tabba <tabba@google.com>,
        Ard Biesheuvel <ardb@kernel.org>, tsoni@quicinc.com,
        quic_psodagud@quicinc.com, quic_svaddagi@quicinc.com
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Hi Vivek,

Interesting stuff.

On Wed, May 18, 2022 at 3:49 AM Vivek Kumar <quic_vivekuma@quicinc.com> wrote:
>
> Kernel Hibernation
>
> Linux Kernel has been already supporting hibernation, a process which
> involves freezing of all userspace tasks, followed by quiescing of all
> kernel device drivers and then a DDR snapshot is taken which is saved
> to disc-swap partition, after the save, the system can either shutdown
> or continue further. Generally during the next power cycle when kernel
> boots and after probing almost all of the drivers, in the late_init()
> part, it checks if a hibernation image is present in the specified swap
> slot, if a valid hibernation image is found, it superimposes the currently
> executing Kernel with an older kernel from the snapshot, moving further,
> it calls the restore of the drivers and unfreezes the userspace tasks.
> CONFIG_HIBERNATION and a designated swap partition needs to be present
> for to enable Hibernation.
>
> Bootloader Based Hibernation:
>
> Automotive usecases require better boot KPIs, Hence we are proposing a
> bootloader based hibernation restore. Purpose of bootloader based
> hibernation is to improve the overall boot time till the first display
> frame is seen on the screen or a camera application can be launched from
> userspace after the power on reset key is pressed. This RFC patchset
> implements a slightly tweaked version of hibernation in which the
> restoration of an older snapshot into DDR is being carried out from the
> bootloader (ABL) itself, by doing this we are saving some time
> (1 second measured on msm-4.14 Kernel) by not running a
> temporary kernel and figuring out the hibernation image at late_init().

I wonder where most of the time is spent? Is it initializing struct
pages? Potentially we could enlighten bootloader to determine whether
hibernation image is stored or not on the swap device, and change boot
parameter for the kernel accordingly. The booting kernel would know
from the very beginning of boot that it will eventually resume a
hibernated image, and therefore skip some of initilization parts, and
perhaps limit amount of memory that it initializes.

> In order to achieve the same bootloader checks for the hibernation
> image at a very early stage from swap partition, it parses the image and
> loads it in the DDR instead of loading boot image form boot partition.
> Since we are not running the temporary kernel,which would have done some
> basic ARM related setup like, MMU enablement, EL2 setup, CPU setup etc,

What boot loader is used? I suspect bootloader enables MMU to load the
hibernated image into memory, otherwise the performance would be very
poor. After the image is loaded, and prior to jumping into the entry
address of loaded image the MMU is probably disabled.

> entry point into hibernation snapshot image directly from bootloader is
> different, on similar lines, all device drivers are now re-programming
> the IO-mapped registers as part of the restore callback (which is
> triggered from the hibernation framework) to bring back the HW/SW sync.
>
> Other factors like, read-speed of the secondary storage device and
> organization of the hibernation image in the swap partition effects the
> total image restore time and the overall boot time. In our current
> implementation we have serialized the allocation of swap-partition's slots
> in kernel, so when hibernation image is being saved to disc, each page is
> not scattered across various swap-slot offsets, rather it in a serial
> manner. For example, if a DDR page at Page frame number 0x8005 is
> located at a swap-slot offset 50, the next valid DDR page at PFN 0x8005
> will be preset at the swap-slot offset 51. With this optimization in
> place, bootloader can utilize the max capacity of issuing a disc-read
> for reading a bigger chunk (~50 MBs at once) from the swap slot,
> and also parsing of the image becomes simpler as it is available
> contiguously.

This optimization seems generic enough that it would benefit both
types of resume: from bootloader and from kernel.

Thanks,
Pasha

>
>
>
> Vivek Kumar (6):
>   arm64: hibernate: Introduce new entry point to kernel
>   PM: Hibernate: Add option to disable disk offset randomization
>   block: gendisk: Add a new genhd capability flag
>   mm: swap: Add randomization check for swapon/off calls
>   Hibernate: Add check for pte_valid in saveable page
>   irqchip/gic-v3: Re-init GIC hardware upon hibernation restore
>
>  Documentation/admin-guide/kernel-parameters.txt |  11 ++
>  arch/arm64/kernel/hibernate.c                   |   9 ++
>  drivers/irqchip/irq-gic-v3.c                    | 138 ++++++++++++++++-
>  include/linux/blkdev.h                          |   1 +
>  kernel/power/snapshot.c                         |  43 ++++++++
>  kernel/power/swap.c                             |  12 +++
>  mm/swapfile.c                                   |   6 +-
>  7 files changed, 216 insertions(+), 4 deletions(-)
>
> --
> 2.7.4
>