Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <002901d5064d$42355ea0$c6a01be0$@lucidpixels.com>
In-Reply-To: <002901d5064d$42355ea0$c6a01be0$@lucidpixels.com>
From:   Justin Piszcz <jpiszcz@lucidpixels.com>
Date:   Thu, 9 May 2019 07:17:43 -0400
Message-ID: <CAO9zADwmyFZiiHPdggrxkyJ2jL8YOcBY9Wj35zGz5SYAvk+Vvg@mail.gmail.com>
Subject: Re: 5.1 kernel: khugepaged stuck at 100%
To:     LKML <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, May 9, 2019 at 5:54 AM Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
> Hello,
>
> Kernel: 5.1 (self-compiled, no modules)
> Arch: x86_64
> Distro: Debian Testing
>
> Issue: I was performing a dump of ext3 and ext4 filesystems and then
> restoring them to a separate volume (testing)-- afterwards I noticed that
> khugepaged is stuck at 100% CPU. It is currently still stuck at 100% CPU, is
> there anything I can do to debug what is happening prior to a reboot to work
> around the issue?  I have never seen this behavior prior to 5.1.
>
> $ cat /proc/cmdline
> auto BOOT_IMAGE=5.1.0-2 ro root=901 3w-sas.use_msi=1 nohugeiomap
> page_poison=1 pcie_aspm=off pti=on rootfstype=ext4 slub_debug=P
> zram.enabled=1 zram.num_devices=12 zswap.enabled=1 zswap.compressor=lz4
> zswap.zpool=z3fold
>
> $ 5.1 .config attached: config.txt.gz
>
> $ attachment: graphic.JPG -> graph of the processes, dark green ->
> khugepaged
>
> $ top
>
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
>    77 root      39  19       0      0      0 R 100.0   0.0  23:29.27
> khugepaged
>     1 root      20   0  171800   7832   4948 S   0.0   0.0   1:25.59 systemd
>     2 root      20   0       0      0      0 S   0.0   0.0   0:00.02
> kthreadd
>     3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
>     4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00
> rcu_par_gp
>     6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00
> kworker/0+
>     8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00
> mm_percpu+
>

Will disable zcache/zswap for now, but FWIW

From perf top:

   PerfTop:    1181 irqs/sec  kernel:83.1%  exact: 96.9% lost: 73/137
drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
-------------------------------------------------------------------------------

    54.56%  [kernel]            [k] compaction_alloc
    29.46%  [kernel]            [k] __pageblock_pfn_to_page
    10.66%  [kernel]            [k] nmi
     0.61%  [kernel]            [k] _cond_resched
     0.32%  [kernel]            [k] format_decode
     0.25%  [kernel]            [k] __lock_text_start
     0.20%  [kernel]            [k] __vma_adjust
     0.16%  perf                [.] __symbols__insert
     0.13%  perf                [.] rb_insert_color_cached

I tried dropping caches, etc, I will disable huge pages for now.

Some other posts:
https://bugzilla.redhat.com/show_bug.cgi?id=879801
https://lkml.org/lkml/2012/6/27/565
https://lkml.org/lkml/2011/11/9/252