Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751314AbdHIEDu (ORCPT ); Wed, 9 Aug 2017 00:03:50 -0400 Received: from conssluserg-06.nifty.com ([210.131.2.91]:39023 "EHLO conssluserg-06.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750750AbdHIEDs (ORCPT ); Wed, 9 Aug 2017 00:03:48 -0400 DKIM-Filter: OpenDKIM Filter v2.10.3 conssluserg-06.nifty.com v7943dEF004611 X-Nifty-SrcIP: [209.85.161.180] MIME-Version: 1.0 In-Reply-To: References: <4d6b511a-61d5-3c5e-a406-9f71d83670b6@linux.intel.com> <1d2c51f3-a655-2223-68a9-e6d700e7d8e1@intel.com> <412a3ab0-4ae5-b27a-0b2d-d2e03b27a999@intel.com> <93b9efde-397d-d7bb-787d-c28ff754b6a5@arm.com> From: Masahiro Yamada Date: Wed, 9 Aug 2017 13:03:37 +0900 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Suspend-resume failure on Intel Eagle Lake Core2Duo To: Marc Zyngier Cc: Thomas Gleixner , Tomi Sarvela , Martin Peres , Jeffy Chen , Linux Kernel Mailing List Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 23527 Lines: 551 Hi Marc, 2017-08-08 16:39 GMT+09:00 Marc Zyngier : > On 08/08/17 02:30, Masahiro Yamada wrote: >> Hi Marc, >> >> 2017-08-07 17:17 GMT+09:00 Marc Zyngier : >>> On 07/08/17 05:45, Masahiro Yamada wrote: >>>> Hi Marc, >>>> >>>> >>>> 2017-08-03 22:30 GMT+09:00 Marc Zyngier : >>>>> On 03/08/17 13:52, Masahiro Yamada wrote: >>>>>> Hi Marc, >>>>>> >>>>>> 2017-08-03 17:41 GMT+09:00 Marc Zyngier : >>>>>>> Hi Masahiro, >>>>>>> >>>>>>> On 03/08/17 08:32, Masahiro Yamada wrote: >>>>>>>> Hi. >>>>>>>> >>>>>>>> 2017-08-01 0:55 GMT+09:00 Thomas Gleixner : >>>>>>>>> On Mon, 31 Jul 2017, Tomi Sarvela wrote: >>>>>>>>>> On 31/07/17 18:06, Thomas Gleixner wrote: >>>>>>>>>>> Can you please remove the patch. And try the following: >>>>>>>>>>> >>>>>>>>>>> # echo N > /sys/module/printk/parameters/console_suspend >>>>>>>>>>> >>>>>>>>>>> # echo mem > /sys/power/state >>>>>>>>>>> >>>>>>>>>>> and log the output of the serial console. That way we might get a clue >>>>>>>>>>> where it gets stuck. >>>>>>>>>> >>>>>>>>>> I'm afraid it hangs right away. No response from SSH, no output to serial. >>>>>>>>> >>>>>>>>> What means hangs right away? Is there no output at all on the serial >>>>>>>>> console? Or does it just stop at some point? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> tglx >>>>>>>>> >>>>>>>> >>>>>>>> Sorry for jumping in. >>>>>>>> Finally, I found this thread. >>>>>>>> >>>>>>>> >>>>>>>> My environment is completely different (ARM64 board), >>>>>>>> I am also suffering from a hibernation problem >>>>>>>> since this commit. >>>>>>>> >>>>>>>> >>>>>>>> I get no response on the serial console >>>>>>>> after "Restarting tasks ... done." log message. >>>>>>>> >>>>>>>> >>>>>>>> By reverting bf22ff45bed6 ("genirq: Avoid unnecessary low level >>>>>>>> irq function calls", I can get hibernation working again. >>>>>>>> >>>>>>>> >>>>>>>> SW info: >>>>>>>> defconfig: arch/arm64/configs/defconfig >>>>>>>> DT : arch/arm64/boot/dts/socionext/uniphier-ld20-ref.dts >>>>>>>> PSCI : ARM Trusted Firmware >>>>>>>> >>>>>>>> >>>>>>>> SoC info: >>>>>>>> CPU : Cortex-A72 * 2 + Cortex-A53 * 2 >>>>>>>> irqchip : GICv3 (drivers/irq/irq-gic-v3.c) >>>>>>> >>>>>>> Let me take an educated guess: It feels like your firmware doesn't >>>>>>> save/restore the GIC context across suspend/resume. Is that something >>>>>>> you could check, assuming you have access to the firmware source code? >>>>>> >>>>>> Thanks for your comments. >>>>>> >>>>>> >>>>>> I do not know much about the manner of preserving GICv3 context. >>>>>> >>>>>> I can see this patch (rejected?) : >>>>>> https://patchwork.kernel.org/patch/9343061/ >>>>>> >>>>>> >>>>>> Is it something that should be completely cared by firmware >>>>>> instead of kernel? >>>>> >>>>> That was definitely the intention, but it looks like something that ATF >>>>> has only started supporting very recently: >>>>> >>>>> https://github.com/ARM-software/arm-trusted-firmware/pull/1047 >>>>> >>>>>> ARM Trusted Firmware (https://github.com/ARM-software/arm-trusted-firmware) >>>>>> is open source software, and I pushed my platform code to the upstream. >>>>>> >>>>>> So, yes, I (and everybody) can have access to the firmware source code. >>>>>> >>>>>> >>>>>> I am not sure how ATF saves the context during hibernation, though. >>>>> >>>>> See the above link. Is there any chance of you trying this into your >>>>> firmware? >>>>> >>>>> Thanks, >>>> >>>> Thanks for the pointer. >>>> >>>> >>>> Yes. I will try that once GIC-v3 context save/restore is supported in ATF. >>>> >>>> I think that will basically work for suspend-to-ram >>>> because all contexts including both non-secure and secure worlds will >>>> be retained in the main memory. >>>> >>>> However, I still do not understand how the context is preserved during >>>> the hibernation (suspend-to-disk). >>>> >>>> >>>> If my understanding is correct, hibernation on Linux works like follows: >>>> >>>> [1] Freeze all tasks >>>> [2] CPU_OFF for non-boot CPUs >>>> [3] Create a hibernation image >>>> [4] CPU_ON for non-boot CPUs >>>> [5] Write the hibernation image to the disk (=swap area) >>>> [6] SYSTEM_OFF >>>> >>>> >>>> IIUC, [5] only writes the context Linux takes care of (only non-secure). >>>> >>>> If so, where and how does the firmware write the GIC-v3 context >>>> to the disk? >>> >>> Gah, I completely missed the fact that you were talking about suspend to >>> disk, sorry about that. >>> >>> It is likely that some driver doesn't restore its state properly. Is >>> there any chance that you could pinpoint which device creates the issue? >>> >> >> I use eMMC to store the hibernation image, but >> I do not think eMMC driver is the cause of the issue. >> >> I guess the cause of the issue is GIC-v3 context is lost. > > It is not lost. The boot kernel has re-initialized its state. What is > missing is that one driver in your system fails to restore its own state > correctly, and relies on doing something such as enabling/disabling the > interrupt in its PM handler, things will start working again (probably > because they use the same PM callbacks functions for both suspend/resume > and hibernation). This is in no way a guarantee. > > Please try: > >> swapon -a >> echo test_resume > /sys/power/disk >> echo disk > /sys/power/state > > and let me know how this fares. > >> I am not an expert in this, so I will ask the ATF community >> about how ATF can support suspend-to-disk. > > As you pointed out, ARF is not involved at all in that context, so > that's pretty pointless. > I tried it based on Linus' tree. Commit: bfa738cf3dfa (Today's Linus tree HEAD) defconfig: arch/arm64/configs/defconfig DT: arch/arm64/boot/dts/socionext/uniphier-ld20-ref.dts I am using eMMC (/dev/mmcblk0) as a disk. The eMMC driver is in upstream (drivers/mmc/host/sdhci-cadence.c) "test_resume" seems to work. The full log is as follows: Starting kernel ... WARNING: BL31: cortex_a53: errata workaround for 835769 was missing! WARNING: BL31: cortex_a53: errata workaround for 843419 was missing! INFO: BL31: cortex_a53: errata workaround for 855873 was applied [ 0.000000] Booting Linux on physical CPU 0x0 [ 0.000000] Linux version 4.13.0-rc4-00052-gbfa738cf3dfa (masahiro@pug) (gcc version 6.1.1 20160711 (Linaro GCC 6.1-2016.08)) #445 SMP PREEMPT Wed Aug 9 12:48:33 JST 2017 [ 0.000000] Boot CPU: AArch64 Processor [410fd082] [ 0.000000] Machine model: UniPhier LD20 Reference Board [ 0.000000] efi: Getting EFI parameters from FDT: [ 0.000000] efi: UEFI not found. [ 0.000000] cma: Reserved 16 MiB at 0x00000000fec00000 [ 0.000000] NUMA: No NUMA configuration found [ 0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000013fffffff] [ 0.000000] NUMA: NODE_DATA [mem 0x13ffe2400-0x13ffe3eff] [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000080000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x000000013fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000080000000-0x000000013fffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000013fffffff] [ 0.000000] psci: probing for conduit method from DT. [ 0.000000] psci: PSCIv1.0 detected in firmware. [ 0.000000] psci: Using standard PSCI v0.2 function IDs [ 0.000000] psci: MIGRATE_INFO_TYPE not supported. [ 0.000000] percpu: Embedded 26 pages/cpu @ffff8000bff78000 s67352 r8192 d30952 u106496 [ 0.000000] Detected PIPT I-cache on CPU0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 774144 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: resume=/dev/mmcblk0 no_console_suspend [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] software IO TLB [mem 0xfac00000-0xfec00000] (64MB) mapped at [ffff80007ac00000-ffff80007ebfffff] [ 0.000000] Memory: 2963680K/3145728K available (9212K kernel code, 1130K rwdata, 4200K rodata, 1088K init, 400K bss, 165664K reserved, 16384K cma-reserved) [ 0.000000] Virtual kernel memory layout: [ 0.000000] modules : 0xffff000000000000 - 0xffff000008000000 ( 128 MB) [ 0.000000] vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000 (129022 GB) [ 0.000000] .text : 0xffff000008080000 - 0xffff000008980000 ( 9216 KB) [ 0.000000] .rodata : 0xffff000008980000 - 0xffff000008da0000 ( 4224 KB) [ 0.000000] .init : 0xffff000008da0000 - 0xffff000008eb0000 ( 1088 KB) [ 0.000000] .data : 0xffff000008eb0000 - 0xffff000008fcaa00 ( 1131 KB) [ 0.000000] .bss : 0xffff000008fcaa00 - 0xffff00000902ece0 ( 401 KB) [ 0.000000] fixed : 0xffff7dfffe7fd000 - 0xffff7dfffec00000 ( 4108 KB) [ 0.000000] PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000 ( 16 MB) [ 0.000000] vmemmap : 0xffff7e0000000000 - 0xffff800000000000 ( 2048 GB maximum) [ 0.000000] 0xffff7e0000000000 - 0xffff7e0003000000 ( 48 MB actual) [ 0.000000] memory : 0xffff800000000000 - 0xffff8000c0000000 ( 3072 MB) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Preemptible hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x000000005fe80000 [ 0.000000] arch_timer: cp15 timer(s) running at 50.00MHz (virt). [ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns [ 0.000003] sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns [ 0.000131] Console: colour dummy device 80x25 [ 0.000587] console [tty0] enabled [ 0.000628] Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=200000) [ 0.000651] pid_max: default: 32768 minimum: 301 [ 0.000708] Security Framework initialized [ 0.003117] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.004482] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.004561] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.004617] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.020049] ASID allocator initialised with 65536 entries [ 0.028039] Hierarchical SRCU implementation. [ 0.036324] EFI services will not be available. [ 0.044056] smp: Bringing up secondary CPUs ... [ 0.072170] Detected PIPT I-cache on CPU1 [ 0.072196] GICv3: CPU1: found redistributor 1 region 0:0x000000005fea0000 [ 0.072217] CPU1: Booted secondary processor [410fd082] [ 0.118125] CPU features: enabling workaround for ARM erratum 845719 [ 0.118132] Detected VIPT I-cache on CPU2 [ 0.118155] GICv3: CPU2: found redistributor 100 region 0:0x000000005fec0000 [ 0.118178] CPU2: Booted secondary processor [410fd034] [ 0.144235] Detected VIPT I-cache on CPU3 [ 0.144251] GICv3: CPU3: found redistributor 101 region 0:0x000000005fee0000 [ 0.144264] CPU3: Booted secondary processor [410fd034] [ 0.144338] smp: Brought up 1 node, 4 CPUs [ 0.144437] SMP: Total of 4 processors activated. [ 0.144449] CPU features: detected feature: GIC system register CPU interface [ 0.144461] CPU features: detected feature: 32-bit EL0 Support [ 0.144639] CPU: All CPU(s) started at EL1 [ 0.144667] alternatives: patching kernel code [ 0.145650] devtmpfs: initialized [ 0.146949] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.147003] futex hash table entries: 1024 (order: 5, 131072 bytes) [ 0.147364] pinctrl core: initialized pinctrl subsystem [ 0.147896] DMI not present or invalid. [ 0.148088] random: get_random_u32 called from bucket_table_alloc+0xf8/0x280 with crng_init=0 [ 0.148214] NET: Registered protocol family 16 [ 0.148830] random: fast init done [ 0.152312] cpuidle: using governor menu [ 0.152645] vdso: 2 pages (1 code @ ffff000008986000, 1 data @ ffff000008eb5000) [ 0.152680] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers. [ 0.153493] DMA: preallocated 256 KiB pool for atomic allocations [ 0.153823] Serial: AMBA PL011 UART driver [ 0.164789] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.174604] ACPI: Interpreter disabled. [ 0.175163] vgaarb: loaded [ 0.175378] SCSI subsystem initialized [ 0.176629] usbcore: registered new interface driver usbfs [ 0.176686] usbcore: registered new interface driver hub [ 0.176762] usbcore: registered new device driver usb [ 0.177205] pps_core: LinuxPPS API ver. 1 registered [ 0.177223] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.177266] PTP clock support registered [ 0.177370] EDAC MC: Ver: 3.0.0 [ 0.177638] dmi: Firmware registration failed. [ 0.177822] Advanced Linux Sound Architecture Driver Initialized. [ 0.178276] clocksource: Switched to clocksource arch_sys_counter [ 0.178375] VFS: Disk quotas dquot_6.6.0 [ 0.178421] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.178554] pnp: PnP ACPI: disabled [ 0.182345] NET: Registered protocol family 2 [ 0.182740] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.182937] TCP bind hash table entries: 32768 (order: 7, 524288 bytes) [ 0.183486] TCP: Hash tables configured (established 32768 bind 32768) [ 0.183644] UDP hash table entries: 2048 (order: 4, 65536 bytes) [ 0.183692] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes) [ 0.183818] NET: Registered protocol family 1 [ 0.184138] RPC: Registered named UNIX socket transport module. [ 0.184163] RPC: Registered udp transport module. [ 0.184178] RPC: Registered tcp transport module. [ 0.184192] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.184389] Unpacking initramfs... [ 0.240364] Freeing initrd memory: 1108K [ 0.240832] kvm [1]: HYP mode not available [ 0.241653] audit: initializing netlink subsys (disabled) [ 0.241824] audit: type=2000 audit(0.239:1): state=initialized audit_enabled=0 res=1 [ 0.242045] workingset: timestamp_bits=44 max_order=20 bucket_order=0 [ 0.247990] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 0.248572] NFS: Registering the id_resolver key type [ 0.248612] Key type id_resolver registered [ 0.248628] Key type id_legacy registered [ 0.248649] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 0.248812] 9p: Installing v9fs 9p2000 file system support [ 0.250783] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 0.250815] io scheduler noop registered [ 0.250988] io scheduler cfq registered (default) [ 0.251005] io scheduler mq-deadline registered [ 0.251021] io scheduler kyber registered [ 0.251698] uniphier-system-bus 58c00000.system-bus: could not find pctldev for node /soc@0/soc-glue@5f800000/pinctrl/system_bus_grp, deferring probe [ 0.261491] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.263260] 54006800.serial: ttyS0 at MMIO 0x54006800 (irq = 6, base_baud = 3676470) is a 16550A [ 1.091069] console [ttyS0] enabled [ 1.095058] SuperH (H)SCI(F) driver initialized [ 1.099835] msm_serial: driver initialized [ 1.104419] cacheinfo: Unable to detect cache hierarchy for CPU 0 [ 1.115648] loop: module loaded [ 1.119515] hisi_sas: driver version v1.6 [ 1.125404] libphy: Fixed MDIO Bus: probed [ 1.129973] tun: Universal TUN/TAP device driver, 1.6 [ 1.135741] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k [ 1.141610] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. [ 1.147598] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k [ 1.154592] igb: Copyright (c) 2007-2014 Intel Corporation. [ 1.160222] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k [ 1.168090] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. [ 1.174225] sky2: driver version 1.30 [ 1.178248] VFIO - User Level meta-driver version: 0.3 [ 1.184273] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.190836] ehci-pci: EHCI PCI platform driver [ 1.195329] ehci-platform: EHCI generic platform driver [ 1.200642] ehci-orion: EHCI orion driver [ 1.204724] ehci-exynos: EHCI EXYNOS driver [ 1.208981] ehci-msm: Qualcomm On-Chip EHCI Host Controller [ 1.214636] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 1.220887] ohci-pci: OHCI PCI platform driver [ 1.225387] ohci-platform: OHCI generic platform driver [ 1.230697] ohci-exynos: OHCI EXYNOS driver [ 1.235177] usbcore: registered new interface driver usb-storage [ 1.242531] i2c /dev entries driver [ 1.250270] sdhci: Secure Digital Host Controller Interface driver [ 1.256518] sdhci: Copyright(c) Pierre Ossman [ 1.261074] Synopsys Designware Multimedia Card Interface Driver [ 1.267620] sdhci-pltfm: SDHCI platform and OF driver helper [ 1.318318] mmc0: SDHCI controller on 5a000000.sdhc [5a000000.sdhc] using ADMA 64-bit [ 1.330068] ledtrig-cpu: registered to indicate activity on CPUs [ 1.336862] usbcore: registered new interface driver usbhid [ 1.342482] usbhid: USB HID core driver [ 1.348120] NET: Registered protocol family 17 [ 1.352689] 9pnet: Installing 9P2000 support [ 1.357074] Key type dns_resolver registered [ 1.362787] registered taskstats version 1 [ 1.370774] smsc911x 43f00000.ethernet: 43f00000.ethernet supply vdd33a not found, using dummy regulator [ 1.380376] smsc911x 43f00000.ethernet: 43f00000.ethernet supply vddvario not found, using dummy regulator [ 1.399995] mmc0: new HS200 MMC card at address 0001 [ 1.405310] mmcblk0: mmc0:0001 004G60 3.69 GiB [ 1.409971] mmcblk0boot0: mmc0:0001 004G60 partition 1 2.00 MiB [ 1.412556] libphy: smsc911x-mdio: probed [ 1.412947] smsc911x 43f00000.ethernet eth0: MAC Address: 6a:fe:ad:31:b1:51 [ 1.414166] 43fb0000.uart: ttyS1 at MMIO 0x43fb0000 (irq = 0, base_baud = 768000) is a 16550A [ 1.414785] hctosys: unable to open rtc device (rtc0) [ 1.440767] mmcblk0boot1: mmc0:0001 004G60 partition 2 2.00 MiB [ 1.446817] mmcblk0rpmb: mmc0:0001 004G60 partition 3 512 KiB [ 1.454926] ALSA device list: [ 1.457909] No soundcards found. [ 1.462026] Freeing unused kernel memory: 1088K Starting logging: OK Initializing random number generator... done. Starting network: OK Welcome to Buildroot buildroot login: root # mkswap /dev/mmcblk0 Setting up swapspace version 1, size = 3959418880 bytes # swapon /dev/mmcblk0 [ 36.214903] Adding 3866620k swap on /dev/mmcblk0. Priority:-1 extents:1 across:3866620k SS # echo test_resume > /sys/power/disk # cat /sys/power/disk shutdown reboot suspend [test_resume] # echo disk > /sys/power/state [ 71.137589] PM: Syncing filesystems ... [ 71.141579] PM: done. [ 71.143871] Freezing user space processes ... (elapsed 0.001 seconds) done. [ 71.151948] OOM killer disabled. [ 71.155295] PM: Preallocating image memory... done (allocated 47630 pages) [ 71.360608] PM: Allocated 190520 kbytes in 0.19 seconds (1002.73 MB/s) [ 71.367149] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 71.391858] Disabling non-boot CPUs ... [ 71.414571] CPU1: shutdown [ 71.417281] psci: CPU1 killed. [ 71.442560] CPU2: shutdown [ 71.445271] psci: CPU2 killed. [ 71.470963] CPU3: shutdown [ 71.473674] psci: CPU3 killed. [ 71.477083] PM: Creating hibernation image: [ 71.477083] PM: Need to copy 46899 pages [ 71.477083] PM: Hibernation image created (46899 pages copied) [ 71.477083] Suspended for 0.569 seconds [ 71.477110] Enabling non-boot CPUs ... [ 71.484903] Detected PIPT I-cache on CPU1 [ 71.488941] GICv3: CPU1: found redistributor 1 region 0:0x000000005fea0000 [ 71.495853] CPU1: Booted secondary processor [410fd082] [ 71.501314] CPU1 is up [ 71.503769] Detected VIPT I-cache on CPU2 [ 71.507802] GICv3: CPU2: found redistributor 100 region 0:0x000000005fec0000 [ 71.514882] CPU2: Booted secondary processor [410fd034] [ 71.520791] CPU2 is up [ 71.523242] Detected VIPT I-cache on CPU3 [ 71.527266] GICv3: CPU3: found redistributor 101 region 0:0x000000005fee0000 [ 71.534334] CPU3: Booted secondary processor [410fd034] [ 71.539807] CPU3 is up [ 71.543493] hibernate: Hibernating on CPU 0 [mpidr:0x0] [ 71.621511] PM: Using 3 thread(s) for compression. [ 71.621511] PM: Compressing and saving image data (46991 pages)... [ 71.632559] PM: Image saving progress: 0% [ 71.729828] PM: Image saving progress: 10% [ 71.844900] PM: Image saving progress: 20% [ 71.958230] PM: Image saving progress: 30% [ 72.036647] PM: Image saving progress: 40% [ 72.099759] PM: Image saving progress: 50% [ 72.160172] PM: Image saving progress: 60% [ 72.233590] PM: Image saving progress: 70% [ 72.299996] PM: Image saving progress: 80% [ 72.369865] PM: Image saving progress: 90% [ 72.454932] PM: Image saving progress: 100% [ 74.105739] PM: Image saving done. [ 74.109213] PM: Wrote 187964 kbytes in 2.47 seconds (76.09 MB/s) [ 74.115637] PM: S [ 74.116848] | [ 74.182660] PM: Using 3 thread(s) for decompression. [ 74.182660] PM: Loading and decompressing image data (46991 pages)... [ 74.194105] hibernate: Hibernated on CPU 0 [mpidr:0x0] [ 74.416966] PM: Image loading progress: 0% [ 74.667431] PM: Image loading progress: 10% [ 74.768269] PM: Image loading progress: 20% [ 74.863927] PM: Image loading progress: 30% [ 74.957324] PM: Image loading progress: 40% [ 75.050884] PM: Image loading progress: 50% [ 75.144371] PM: Image loading progress: 60% [ 75.230387] PM: Image loading progress: 70% [ 75.313445] PM: Image loading progress: 80% [ 75.403118] PM: Image loading progress: 90% [ 75.494369] PM: Image loading progress: 100% [ 75.499042] PM: Image loading done. [ 75.502549] PM: Read 187964 kbytes in 1.30 seconds (144.58 MB/s) [ 75.527824] Disabling non-boot CPUs ... [ 75.558505] CPU1: shutdown [ 75.561219] psci: CPU1 killed. [ 75.586483] CPU2: shutdown [ 75.589193] psci: CPU2 killed. [ 75.614940] CPU3: shutdown [ 75.617654] psci: CPU3 killed. [ 71.477083] Suspended for 9.134 seconds [ 71.477111] Enabling non-boot CPUs ... [ 71.484909] Detected PIPT I-cache on CPU1 [ 71.488946] GICv3: CPU1: found redistributor 1 region 0:0x000000005fea0000 [ 71.495857] CPU1: Booted secondary processor [410fd082] [ 71.501316] CPU1 is up [ 71.503777] Detected VIPT I-cache on CPU2 [ 71.507817] GICv3: CPU2: found redistributor 100 region 0:0x000000005fec0000 [ 71.514895] CPU2: Booted secondary processor [410fd034] [ 71.520841] CPU2 is up [ 71.523293] Detected VIPT I-cache on CPU3 [ 71.527315] GICv3: CPU3: found redistributor 101 region 0:0x000000005fee0000 [ 71.534384] CPU3: Booted secondary processor [410fd034] [ 71.539852] CPU3 is up [ 71.596208] OOM killer enabled. [ 71.599374] Restarting tasks ... done. # ls # uname -r 4.13.0-rc4-00052-gbfa738cf3dfa -- Best Regards Masahiro Yamada