Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
From:   Feng Li <lifeng1519@gmail.com>
Date:   Tue, 31 Jul 2018 15:54:08 +0800
Message-ID: <CAEK8JBCZ9HOHs9N6mdjg-q5pr4KZe3kWnrYKgW-huo8Yg9FZmQ@mail.gmail.com>
Subject: page faults affects the performance
To:     linux-kernel <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi experts,


When I use fio to write our distribution storage, I find a lot of page
faults that affect the performance.

My OS is CentOS7,  anyone could give suggestions on how to root cause
this issue? The CPU is running at 100%.

[root@58 15:50:57 ~]$uname -a
Linux 58 3.10.0-693.11.6.el7.smartx.1.x86_64 #1 SMP Tue Jan 16
22:09:10 CST 2018 x86_64 x86_64 x86_64 GNU/Linux

Thanks in advance.

This is the perf report from "perf record -g -p 9881"

+   64.68%     0.01%  zbs-chunkd       [k] system_call_fastpath

       =E2=96=92
+   42.86%     0.01%  zbs-chunkd       [k] inet_recvmsg

       =E2=96=92
+   42.84%     0.00%  zbs-chunkd       [k] sys_recvfrom

       =E2=96=92
+   42.83%     0.13%  zbs-chunkd       [k] tcp_recvmsg

       =E2=96=92
+   42.83%     0.01%  zbs-chunkd       [k] SYSC_recvfrom

       =E2=96=92
+   42.75%     0.02%  zbs-chunkd       [k] sock_recvmsg

       =E2=96=92
+   39.52%     0.09%  zbs-chunkd       [k] skb_copy_datagram_iovec

       =E2=96=92
-   31.97%     7.18%  zbs-chunkd       [k] page_fault

       =E2=96=92
   - 24.79% page_fault

       =E2=96=92
      + 24.78% do_page_fault

       =E2=96=92
   - 7.17% __libc_recv

       =E2=96=92
      - 7.17% system_call_fastpath

       =E2=96=92
           sys_recvfrom

       =E2=96=92
           SYSC_recvfrom

       =E2=96=92
           sock_recvmsg

       =E2=96=92
           inet_recvmsg

       =E2=96=92
           tcp_recvmsg

       =E2=96=92
           skb_copy_datagram_iovec

       =E2=96=92
           page_fault

       =E2=96=92


+   24.79%     0.04%  zbs-chunkd       [k] do_page_fault


This is the output of "perf record -e page-faults -ag":

Samples: 132K of event 'page-faults', Event count (approx.): 9249205
  Children      Self  Command          Shared Object                 Symbol
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] skb_copy_datagram_iovec
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] tcp_recvmsg
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] inet_recvmsg
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] sock_recvmsg
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] SYSC_recvfrom
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] sys_recvfrom
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] system_call_fastpath
+   78.92%    78.92%  zbs-chunkd       [kernel.kallsyms]
[k] copy_user_enhanced_fast_string
+   78.92%     0.00%  zbs-chunkd       libpthread-2.17.so
[.] __libc_recv


--=20
Thanks and Best Regards,
Feng Li(Alex)