Hi experts,
When I use fio to write our distribution storage, I find a lot of page
faults that affect the performance.
My OS is CentOS7, anyone could give suggestions on how to root cause
this issue? The CPU is running at 100%.
[root@58 15:50:57 ~]$uname -a
Linux 58 3.10.0-693.11.6.el7.smartx.1.x86_64 #1 SMP Tue Jan 16
22:09:10 CST 2018 x86_64 x86_64 x86_64 GNU/Linux
Thanks in advance.
This is the perf report from "perf record -g -p 9881"
+ 64.68% 0.01% zbs-chunkd [k] system_call_fastpath
▒
+ 42.86% 0.01% zbs-chunkd [k] inet_recvmsg
▒
+ 42.84% 0.00% zbs-chunkd [k] sys_recvfrom
▒
+ 42.83% 0.13% zbs-chunkd [k] tcp_recvmsg
▒
+ 42.83% 0.01% zbs-chunkd [k] SYSC_recvfrom
▒
+ 42.75% 0.02% zbs-chunkd [k] sock_recvmsg
▒
+ 39.52% 0.09% zbs-chunkd [k] skb_copy_datagram_iovec
▒
- 31.97% 7.18% zbs-chunkd [k] page_fault
▒
- 24.79% page_fault
▒
+ 24.78% do_page_fault
▒
- 7.17% __libc_recv
▒
- 7.17% system_call_fastpath
▒
sys_recvfrom
▒
SYSC_recvfrom
▒
sock_recvmsg
▒
inet_recvmsg
▒
tcp_recvmsg
▒
skb_copy_datagram_iovec
▒
page_fault
▒
+ 24.79% 0.04% zbs-chunkd [k] do_page_fault
This is the output of "perf record -e page-faults -ag":
Samples: 132K of event 'page-faults', Event count (approx.): 9249205
Children Self Command Shared Object Symbol
+ 78.92% 0.00% zbs-chunkd [kernel.kallsyms]
[k] skb_copy_datagram_iovec
+ 78.92% 0.00% zbs-chunkd [kernel.kallsyms]
[k] tcp_recvmsg
+ 78.92% 0.00% zbs-chunkd [kernel.kallsyms]
[k] inet_recvmsg
+ 78.92% 0.00% zbs-chunkd [kernel.kallsyms]
[k] sock_recvmsg
+ 78.92% 0.00% zbs-chunkd [kernel.kallsyms]
[k] SYSC_recvfrom
+ 78.92% 0.00% zbs-chunkd [kernel.kallsyms]
[k] sys_recvfrom
+ 78.92% 0.00% zbs-chunkd [kernel.kallsyms]
[k] system_call_fastpath
+ 78.92% 78.92% zbs-chunkd [kernel.kallsyms]
[k] copy_user_enhanced_fast_string
+ 78.92% 0.00% zbs-chunkd libpthread-2.17.so
[.] __libc_recv
--
Thanks and Best Regards,
Feng Li(Alex)