Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <CAEJqkggcnW98Sk3BEBCCZf57Uwd9rdqD5Da0tmuTaNfkJN5kVg@mail.gmail.com>
 <CACVXFVNiRX+K8toexRfykazVnpC1sNYj2UbUtnqNeqkqUfs8TA@mail.gmail.com>
 <CAEJqkgii81G8_=ddVb92TFAcsr5LCEOi08aBfQ2RAu2UPAYpwg@mail.gmail.com> <20190917091806.GB15374@ming.t460p>
In-Reply-To: <20190917091806.GB15374@ming.t460p>
From:   Dexuan-Linux Cui <dexuan.linux@gmail.com>
Date:   Sun, 2 Feb 2020 15:32:17 -0800
Message-ID: <CAA42JLYMmHu7NV1L6CiacHtLS45NtCDa53VFji6Zj098B4z-gw@mail.gmail.com>
Subject: Re: nvme vs. hibernation ( again )
To:     Ming Lei <ming.lei@redhat.com>
Cc:     Gabriel C <nix.or.die@gmail.com>, Ming Lei <tom.leiming@gmail.com>,
        LKML <linux-kernel@vger.kernel.org>,
        linux-nvme <linux-nvme@lists.infradead.org>,
        Christoph Hellwig <hch@lst.de>,
        Dexuan Cui <decui@microsoft.com>, juhlee@microsoft.com,
        mikelley@microsoft.com
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Sep 17, 2019 at 2:58 AM Ming Lei <ming.lei@redhat.com> wrote:
>
> On Thu, Sep 12, 2019 at 11:24:23AM +0200, Gabriel C wrote:
> > Am Do., 12. Sept. 2019 um 02:51 Uhr schrieb Ming Lei <tom.leiming@gmail.com>:
> > >
> > > On Thu, Sep 12, 2019 at 12:27 AM Gabriel C <nix.or.die@gmail.com> wrote:
> > > >
> > > > Hi Christoph,
> > > >
> > > > I see this was already discussed in 2 threads:
> > > >
> > > >  https://lists.infradead.org/pipermail/linux-nvme/2019-April/023234.html
> > > >  https://lkml.org/lkml/2019/5/24/668
> > > >
> > > > but in latest git the issue still exists.
> > > >
> > > > I hit that on each resume on my Acer Nitro 5 (AN515-43-R8BF) Laptop.
> > > >
> > > > .....
> > > > Sep 11 16:16:30 nitro5 kernel: Freezing remaining freezable tasks ...
> > > > (elapsed 0.000 seconds) done.
> > > > Sep 11 16:16:30 nitro5 kernel: printk: Suspending console(s) (use
> > > > no_console_suspend to debug)
> > > > Sep 11 16:16:30 nitro5 kernel: WARNING: CPU: 0 PID: 882 at
> > > > kernel/irq/chip.c:210 irq_startup+0xe6/0xf0
> > > > Sep 11 16:16:30 nitro5 kernel: Modules linked in: af_packet bnep
> > > > amdgpu ath10k_pci ath10k_core ath mac80211 joydev uvcvideo
> > > > videobuf2_vmalloc videobuf2_memops edac_mce_amd videobuf2_v4l2
> > > > amd_iommu_v2 kvm_amd gpu_sched btusb snd_hda_codec_realtek ttm btrtl
> > > > btbcm btintel hid_multitouch ccp snd_hda_codec_generic nls_utf8
> > > > bluetooth drm_kms_helper hid_generic videobuf2_common ledtrig_audio
> > > > snd_hda_codec_hdmi nls_cp437 cfg80211 drm kvm snd_hda_intel vfat
> > > > videodev fat agpgart efi_pstore r8169 snd_hda_codec ecdh_generic
> > > > i2c_algo_bit realtek irqbypass pcspkr mc rfkill fb_sys_fops efivars
> > > > syscopyarea snd_hda_core ecc k10temp wmi_bmof sysfillrect tpm_crb
> > > > crc16 libphy i2c_piix4 libarc4 snd_hwdep hwmon sysimgblt tpm_tis
> > > > tpm_tis_core evdev ac tpm battery mac_hid i2c_designware_platform
> > > > pinctrl_amd i2c_designware_core rng_core acer_wireless button
> > > > acpi_cpufreq ppdev sch_fq_codel fuse snd_pcm_oss snd_mixer_oss snd_pcm
> > > > snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device
> > > > snd_timer snd soundcore lp parport_pc
> > > > Sep 11 16:16:30 nitro5 kernel:  parport xfs libcrc32c crc32c_generic
> > > > crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ahci
> > > > libahci libata xhci_pci xhci_hcd aesni_intel usbcore scsi_mod
> > > > aes_x86_64 crypto_simd cryptd glue_helper serio_raw i2c_hid hid video
> > > > i2c_core wmi dm_mirror dm_region_hash dm_log dm_mod unix sha1_ssse3
> > > > sha1_generic hmac ipv6 nf_defrag_ipv6 autofs4
> > > > Sep 11 16:16:30 nitro5 kernel: CPU: 0 PID: 882 Comm: kworker/u32:9 Not
> > > > tainted 5.3.0-rc8-00007-g3120b9a6a3f7-dirty #2
> > > > Sep 11 16:16:30 nitro5 kernel: Hardware name: Acer Nitro
> > > > AN515-43/Octavia_PKS, BIOS V1.05 08/07/2019
> > > > Sep 11 16:16:30 nitro5 kernel: Workqueue: events_unbound async_run_entry_fn
> > > > Sep 11 16:16:30 nitro5 kernel: RIP: 0010:irq_startup+0xe6/0xf0
> > > > Sep 11 16:16:30 nitro5 kernel: Code: e8 7f 3c 00 00 85 c0 0f 85 e3 09
> > > > 00 00 4c 89 e7 31 d2 4c 89 ee e8 1a cf ff ff 48 89 ef e8 b2 fe ff ff
> > > > 41 89 c4 e9 51 ff ff ff <0f> 0b eb b2 66 0f 1f 44 00 00 0f 1f 44 00 00
> > > > 55 48 89 fd 53 48 8b
> > > > Sep 11 16:16:30 nitro5 kernel: RSP: 0018:ffffbe9b00793c38 EFLAGS: 00010002
> > > > Sep 11 16:16:30 nitro5 kernel: RAX: 0000000000000010 RBX:
> > > > 0000000000000001 RCX: 0000000000000040
> > > > Sep 11 16:16:30 nitro5 kernel: RDX: 0000000000000000 RSI:
> > > > ffffffff9d1b8800 RDI: ffff9c9d9e136598
> > > > Sep 11 16:16:30 nitro5 kernel: RBP: ffff9c9d981e5400 R08:
> > > > 0000000000000000 R09: ffff9c9d9e8003f0
> > > > Sep 11 16:16:30 nitro5 kernel: R10: 0000000000000000 R11:
> > > > ffffffff9d057688 R12: 0000000000000001
> > > > Sep 11 16:16:30 nitro5 kernel: R13: ffff9c9d9e136598 R14:
> > > > 0000000000000000 R15: ffff9c9d9e346000
> > > > Sep 11 16:16:30 nitro5 kernel: FS:  0000000000000000(0000)
> > > > GS:ffff9c9da0800000(0000) knlGS:0000000000000000
> > > > Sep 11 16:16:30 nitro5 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > Sep 11 16:16:30 nitro5 kernel: CR2: 00005633ad8d0060 CR3:
> > > > 00000003db8d0000 CR4: 00000000003406f0
> > > > Sep 11 16:16:30 nitro5 kernel: Call Trace:
> > > > Sep 11 16:16:30 nitro5 kernel:  enable_irq+0x48/0x90
> > > > Sep 11 16:16:30 nitro5 kernel:  nvme_poll_irqdisable+0x20c/0x280
> > > > Sep 11 16:16:30 nitro5 kernel:  __nvme_disable_io_queues+0x19d/0x1d0
> > > > Sep 11 16:16:30 nitro5 kernel:  ? nvme_del_queue_end+0x20/0x20
> > > > Sep 11 16:16:30 nitro5 kernel:  nvme_dev_disable+0x15c/0x210
> > > > Sep 11 16:16:30 nitro5 kernel:  nvme_suspend+0x40/0x130
> > > > Sep 11 16:16:30 nitro5 kernel:  pci_pm_suspend+0x72/0x130
> > > > Sep 11 16:16:30 nitro5 kernel:  ? pci_pm_freeze+0xb0/0xb0
> > > > Sep 11 16:16:30 nitro5 kernel:  dpm_run_callback+0x29/0x120
> > > > Sep 11 16:16:30 nitro5 kernel:  __device_suspend+0x1b2/0x400
> > > > Sep 11 16:16:30 nitro5 kernel:  async_suspend+0x1b/0x90
> > > > Sep 11 16:16:30 nitro5 kernel:  async_run_entry_fn+0x37/0xe0
> > > > Sep 11 16:16:30 nitro5 kernel:  process_one_work+0x1d1/0x3a0
> > > > Sep 11 16:16:30 nitro5 kernel:  worker_thread+0x4a/0x3d0
> > > > Sep 11 16:16:30 nitro5 kernel:  kthread+0xf9/0x130
> > > > Sep 11 16:16:30 nitro5 kernel:  ? process_one_work+0x3a0/0x3a0
> > > > Sep 11 16:16:30 nitro5 kernel:  ? kthread_park+0x80/0x80
> > > > Sep 11 16:16:30 nitro5 kernel:  ret_from_fork+0x22/0x40
> > > > Sep 11 16:16:30 nitro5 kernel: ---[ end trace c598a86b44574730 ]---
> > > >
> > > > ...
> > > >
> > > > The patch from Dongli Zhang was rejected the time without any other fix
> > > > or work on this issue I could find.
> > > >
> > > > Are there any plans to fix that or any code to test?
> > >
> > > I guess the following patchset may address it:
> > >
> > > https://lore.kernel.org/linux-block/20190812134312.16732-1-ming.lei@redhat.com/
> > >
> >
> > Thx Ming Lei.
> >
> > Can you tell me on what branch/tree they are based on?
>
> The patchset can be applied cleanly on for-5.4/block.
>
> However, the warning can't be fixed, even though the issue and
> the patchset is closely related.
>
> The warning just means there isn't any online CPU in this managed IRQ's
> affinity. Once we can guarantee that there aren't any in-flight requests
> for dead hctx(no any online CPU in its affinity mask), the warning
> can be removed.
>
>
> thanks,
> Ming

Hi Ming,
I still see the call-trace with v5.5 (I passed through a NVMe
controller to a Linux VM running on Hyper-V and I'm testing VM
hibernation with some extra patches applied to the Hyper-V drivers).

It looks your Aug-2019 patchset
(https://lore.kernel.org/linux-block/20190812134312.16732-6-ming.lei@redhat.com/)
has not been accepted?

PS, this is the call-trace I'm seeing:

[  274.693172] PM: hibernation entry
[  274.718964] Filesystems sync: 0.022 seconds
[  274.718964] Freezing user space processes ... (elapsed 0.001 seconds) done.
[  274.720859] OOM killer disabled.
[  274.720873] PM: Marking nosave pages: [mem 0x00000000-0x00000fff]
[  274.720873] PM: Marking nosave pages: [mem 0x0009f000-0x000fffff]
[  274.720873] PM: Marking nosave pages: [mem 0x3fff0000-0xffffffff]
[  274.722536] PM: Marking nosave pages: [mem 0xfe0000000-0xfffffffff]
[  274.722539] PM: Basic memory bitmaps created
[  274.722539] PM: Preallocating image memory... done (allocated 837903 pages)
[  276.950530] PM: Allocated 3351612 kbytes in 2.22 seconds (1509.73 MB/s)
[  276.950530] Freezing remaining freezable tasks ... (elapsed 0.002
seconds) done.
[  276.956255] serial 00:04: disabled
[  277.014876] ------------[ cut here ]------------
[  277.014876] WARNING: CPU: 15 PID: 389 at kernel/irq/chip.c:210
irq_startup+0xe1/0xf0
[  277.014876] Modules linked in: ......
[  277.014876] CPU: 15 PID: 389 Comm: kworker/u258:2 Not tainted 5.5.0+ #1
[  277.014876] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS 090007  05/18/2018
[  277.014876] Workqueue: events_unbound async_run_entry_fn
[  277.014876] RIP: 0010:irq_startup+0xe1/0xf0
[  277.014876] Code: f6 4c 89 f7 e8 00 42 00 00 85 c0 75 21 4c 89 ee
31 d2 4c ......
[  277.014876] RSP: 0018:ffffb70e48043bc0 EFLAGS: 00010002
[  277.014876] RAX: 0000000000000080 RBX: ffff8e596e3a6c00 RCX: 0000000000000080
[  277.014876] RDX: 0000000000000000 RSI: ffffffff99c2b920 RDI: ffff8e599ad66d40
[  277.014876] RBP: ffffb70e48043be0 R08: 0000000000000000 R09: 0000000000000001
[  277.014876] R10: 0000000000000000 R11: ffffffff99a621a8 R12: 0000000000000001
[  277.014876] R13: ffff8e599ad66d40 R14: 0000000000000001 R15: ffff8e498b332000
[  277.014876] FS:  0000000000000000(0000) GS:ffff8e599efc0000(0000)
knlGS:0000000000000000
[  277.014876] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  277.014876] CR2: 000055eab6b36d80 CR3: 000000106e3b4000 CR4: 00000000003406e0
[  277.014876] Call Trace:
[  277.014877]  __enable_irq+0x52/0x60
[  277.014877]  enable_irq+0x4c/0x90
[  277.014877]  nvme_poll_irqdisable+0x376/0x380
[  277.014877]  ? wake_up_q+0xb0/0xb0
[  277.014877]  __nvme_disable_io_queues+0x1b5/0x200
[  277.014877]  ? nvme_del_queue_end+0x30/0x30
[  277.014877]  nvme_disable_io_queues+0x25/0x30
[  277.014877]  nvme_dev_disable+0x1b1/0x250
[  277.014877]  nvme_simple_suspend+0x2b/0x40
[  277.014877]  pci_pm_freeze+0x5c/0xd0
[  277.014877]  ? pci_pm_poweroff+0x100/0x100
[  277.014877]  dpm_run_callback+0x5a/0x170
[  277.014877]  __device_suspend+0x156/0x4f0
[  277.014877]  async_suspend+0x1f/0xa0
[  277.014877]  async_run_entry_fn+0x3c/0x150
[  277.014877]  process_one_work+0x1f7/0x3f0
[  277.014877]  worker_thread+0x34/0x400
[  277.014877]  kthread+0x121/0x140
[  277.014877]  ? process_one_work+0x3f0/0x3f0
[  277.014877]  ? kthread_park+0x90/0x90
[  277.014877]  ret_from_fork+0x22/0x40


Thanks,
Dexuan