Hello,
On (01/30/18 11:48), Andrew Morton wrote:
> Subject: [Bug 198617] New: zswap causing random applications to crash
>
> https://bugzilla.kernel.org/show_bug.cgi?id=198617
>
> Bug ID: 198617
> Summary: zswap causing random applications to crash
> Product: Memory Management
> Version: 2.5
> Kernel Version: 4.14.15
> Hardware: x86-64
> OS: Linux
> Tree: Mainline
> Status: NEW
> Severity: normal
> Priority: P1
> Component: Page Allocator
> Assignee: [email protected]
> Reporter: [email protected]
> Regression: No
>
> https://bugs.freedesktop.org/show_bug.cgi?id=104709
> https://bugs.kde.org/show_bug.cgi?id=389542
>
> I did have zswap enabled for a long while, and a lot of wine games,
> plasmashell, xorg, kwin_x11 (and other) did crash randomly when reached 100% of
> physical ram and swap was like almost never used.
>
> I could esilly open a lot of browser tabs and the browser or xorg would fail
> every time.
>
> After disabling zswap no crashes at all.
>
> /etc/systemd/swap.conf
> zswap_enabled=1
> zswap_compressor=lz4 # lzo lz4
> zswap_max_pool_percent=25 # 1-99
> zswap_zpool=zbud # zbud z3fold
So I did a number of tests and I confirm that under memory pressure
with frontswap enabled I do see segfaults and memory corruptions in
random user space applications.
kernel: urxvt[338]: segfault at 20 ip 00007fc08889ae0d sp 00007ffc73a7fc40 error 6 in libc-2.26.so[7fc08881a000+1ae000]
#0 0x00007fc08889ae0d _int_malloc (libc.so.6)
#1 0x00007fc08889c2f3 malloc (libc.so.6)
#2 0x0000560e6004bff7 _Z14rxvt_wcstoutf8PKwi (urxvt)
#3 0x0000560e6005e75c n/a (urxvt)
#4 0x0000560e6007d9f1 _ZN16rxvt_perl_interp6invokeEP9rxvt_term9hook_typez (urxvt)
#5 0x0000560e6003d988 _ZN9rxvt_term9cmd_parseEv (urxvt)
#6 0x0000560e60042804 _ZN9rxvt_term6pty_cbERN2ev2ioEi (urxvt)
#7 0x0000560e6005c10f _Z17ev_invoke_pendingv (urxvt)
#8 0x0000560e6005cb55 ev_run (urxvt)
#9 0x0000560e6003b9b9 main (urxvt)
#10 0x00007fc08883af4a __libc_start_main (libc.so.6)
#11 0x0000560e6003f9da _start (urxvt)
kernel: urxvt[343]: segfault at 10 ip 00007fa56bd7d52b sp 00007ffc09783a40 error 4 in libc-2.26.so[7fa56bcfd000+1ae000]
#0 0x00007fa56bd7d52b _int_malloc (libc.so.6)
#1 0x00007fa56bd7f2f3 malloc (libc.so.6)
#2 0x00007fa56b3d6097 n/a (libxcb.so.1)
#3 0x00007fa56b3d64d8 n/a (libxcb.so.1)
#4 0x00007fa56c921b79 n/a (libX11.so.6)
#5 0x00007fa56c921ceb n/a (libX11.so.6)
#6 0x00007fa56c921fdd _XEventsQueued (libX11.so.6)
#7 0x00007fa56c913c49 XEventsQueued (libX11.so.6)
#8 0x000055b35cfc3262 _ZN12rxvt_display8flush_cbERN2ev7prepareEi (urxvt)
#9 0x000055b35cfc910f _Z17ev_invoke_pendingv (urxvt)
#10 0x000055b35cfc9c02 ev_run (urxvt)
#11 0x000055b35cfa89b9 main (urxvt)
#12 0x00007fa56bd1df4a __libc_start_main (libc.so.6)
#13 0x000055b35cfac9da _start (urxvt)
Stack trace of thread 351:
#0 0x00007f5baaee7860 raise (libc.so.6)
#1 0x00007f5baaee8ec9 abort (libc.so.6)
#2 0x00007f5baaf30849 __malloc_assert (libc.so.6)
#3 0x00007f5baaf34011 _int_malloc (libc.so.6)
#4 0x00007f5baaf352f3 malloc (libc.so.6)
#5 0x00007f5baaf71cad __alloc_dir (libc.so.6)
#6 0x00007f5baaf71dbd opendir_tail (libc.so.6)
#7 0x00007f5bab5bbac4 Perl_pp_open_dir (libperl.so)
#8 0x00007f5bab55fec6 Perl_runops_standard (libperl.so)
#9 0x00007f5bab4d9390 Perl_call_sv (libperl.so)
#10 0x00005611f097e190 _ZN16rxvt_perl_interp6invokeEP9rxvt_term9hook_typez (urxvt)
#11 0x00005611f0947acb _ZN9rxvt_term14init_resourcesEiPKPKc (urxvt)
#12 0x00005611f0948da8 _ZN9rxvt_term5init2EiPKPKc (urxvt)
#13 0x00005611f097a0af n/a (urxvt)
#14 0x00007f5bab568259 Perl_pp_entersub (libperl.so)
#15 0x00007f5bab55fec6 Perl_runops_standard (libperl.so)
#16 0x00007f5bab4d9390 Perl_call_sv (libperl.so)
#17 0x00005611f097e190 _ZN16rxvt_perl_interp6invokeEP9rxvt_term9hook_typez (urxvt)
#18 0x00005611f0939a77 _ZN9rxvt_term9key_pressER9XKeyEvent (urxvt)
#19 0x00005611f093d77a _ZN9rxvt_term4x_cbER7_XEvent (urxvt)
#20 0x00005611f09572e8 _ZN12rxvt_display8flush_cbERN2ev7prepareEi (urxvt)
#21 0x00005611f095d10f _Z17ev_invoke_pendingv (urxvt)
#22 0x00005611f095dc02 ev_run (urxvt)
#23 0x00005611f093c9b9 main (urxvt)
#24 0x00007f5baaed3f4a __libc_start_main (libc.so.6)
#25 0x00005611f09409da _start (urxvt)
and so on.
However, the problem is not specific to 4.14.15 or 4.14.11.
I manages to track it down to 4.14 merge window, so we are basically
looking at 4.14-rc0+
The bisect log looks as follows:
git bisect start
# bad: [2bd6bf03f4c1c59381d62c61d03f6cc3fe71f66e] Linux 4.14-rc1
git bisect bad 2bd6bf03f4c1c59381d62c61d03f6cc3fe71f66e
# good: [569dbb88e80deb68974ef6fdd6a13edb9d686261] Linux 4.13
git bisect good 569dbb88e80deb68974ef6fdd6a13edb9d686261
# good: [aae3dbb4776e7916b6cd442d00159bea27a695c1] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
git bisect good aae3dbb4776e7916b6cd442d00159bea27a695c1
# bad: [2f173d2688559a6f85643d38a2ad6f45eb420c42] KVM: x86: Fix immediate_exit handling for uninitialized AP
git bisect bad 2f173d2688559a6f85643d38a2ad6f45eb420c42
# bad: [d969443064abf2f51510559a5b01325eaabfcb1d] Merge tag 'sound-4.14-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect bad d969443064abf2f51510559a5b01325eaabfcb1d
# bad: [a0725ab0c7536076d5477264420ef420ebb64501] Merge branch 'for-4.14/block' of git://git.kernel.dk/linux-block
git bisect bad a0725ab0c7536076d5477264420ef420ebb64501
# bad: [f92e3da18b7d5941468040af962c201235148301] Merge branch 'efi-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad f92e3da18b7d5941468040af962c201235148301
# good: [1c9fe4409ce3e9c78b1ed96ee8ed699d4f03bf33] x86/mm: Document how CR4.PCIDE restore works
git bisect good 1c9fe4409ce3e9c78b1ed96ee8ed699d4f03bf33
# bad: [da99ecf117fce6570bd3989263d68ee0007e1249] mm: replace TIF_MEMDIE checks by tsk_is_oom_victim
git bisect bad da99ecf117fce6570bd3989263d68ee0007e1249
# good: [f7b68046873724129798c405e1a4e326b409c08f] mm: use find_get_pages_range() in filemap_range_has_page()
git bisect good f7b68046873724129798c405e1a4e326b409c08f
# bad: [824f973904a1108806fa0fbe15dc93ee9ecd9e0a] userfaultfd: selftest: enable testing of UFFDIO_ZEROPAGE for shmem
git bisect bad 824f973904a1108806fa0fbe15dc93ee9ecd9e0a
# good: [98cc093cba1e925eb34963dedb5f1684f1bdb2f4] block, THP: make block_device_operations.rw_page support THP
git bisect good 98cc093cba1e925eb34963dedb5f1684f1bdb2f4
# bad: [fe490cc0fe9e6ee48cc48bb5dc463bc5f0f1428f] mm, THP, swap: add THP swapping out fallback counting
git bisect bad fe490cc0fe9e6ee48cc48bb5dc463bc5f0f1428f
# good: [3e14a57b2416b7c94189b95baffd673cf5e0d0a3] memcg, THP, swap: support move mem cgroup charge for THP swapped out
git bisect good 3e14a57b2416b7c94189b95baffd673cf5e0d0a3
# good: [d6810d730022016d9c0f389452b86b035dba1492] memcg, THP, swap: make mem_cgroup_swapout() support THP
git bisect good d6810d730022016d9c0f389452b86b035dba1492
# bad: [bd4c82c22c367e068acb1ec9ec02be2fac3e09e2] mm, THP, swap: delay splitting THP after swapped out
git bisect bad bd4c82c22c367e068acb1ec9ec02be2fac3e09e2
# first bad commit: [bd4c82c22c367e068acb1ec9ec02be2fac3e09e2] mm, THP, swap: delay splitting THP after swapped out
The suspected first bad commit is:
bd4c82c22c367e068acb1ec9ec02be2fac3e09e2 is the first bad commit
commit bd4c82c22c367e068acb1ec9ec02be2fac3e09e2
Author: Huang Ying
Date: Wed Sep 6 16:22:49 2017 -0700
mm, THP, swap: delay splitting THP after swapped out
In this patch, splitting transparent huge page (THP) during swapping out
is delayed from after adding the THP into the swap cache to after
swapping out finishes. After the patch, more operations for the
anonymous THP reclaiming, such as writing the THP to the swap device,
removing the THP from the swap cache could be batched. So that the
performance of anonymous THP swapping out could be improved.
This is the second step for the THP swap support. The plan is to delay
splitting the THP step by step and avoid splitting the THP finally.
With the patchset, the swap out throughput improves 42% (from about
5.81GB/s to about 8.25GB/s) in the vm-scalability swap-w-seq test case
with 16 processes. At the same time, the IPI (reflect TLB flushing)
reduced about 78.9%. The test is done on a Xeon E5 v3 system. The swap
device used is a RAM simulated PMEM (persistent memory) device. To test
the sequential swapping out, the test case creates 8 processes, which
sequentially allocate and write to the anonymous pages until the RAM and
part of the swap device is used up.
Link: http://lkml.kernel.org/r/[email protected]
-ss
On (02/03/18 10:34), Sergey Senozhatsky wrote:
> so we are basically looking at 4.14-rc0+
[..]
> # first bad commit: [bd4c82c22c367e068acb1ec9ec02be2fac3e09e2] mm, THP, swap: delay splitting THP after swapped out
To re-confirm, disabling CONFIG_TRANSPARENT_HUGEPAGE fixes my 4.15.0-next
-ss
Hi, Sergey,
Thanks for reporting!
On Sat, Feb 3, 2018 at 9:34 AM, Sergey Senozhatsky
<[email protected]> wrote:
> Hello,
>
> On (01/30/18 11:48), Andrew Morton wrote:
>> Subject: [Bug 198617] New: zswap causing random applications to crash
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=198617
>>
>> Bug ID: 198617
>> Summary: zswap causing random applications to crash
>> Product: Memory Management
>> Version: 2.5
>> Kernel Version: 4.14.15
>> Hardware: x86-64
>> OS: Linux
>> Tree: Mainline
>> Status: NEW
>> Severity: normal
>> Priority: P1
>> Component: Page Allocator
>> Assignee: [email protected]
>> Reporter: [email protected]
>> Regression: No
>>
>> https://bugs.freedesktop.org/show_bug.cgi?id=104709
>> https://bugs.kde.org/show_bug.cgi?id=389542
>>
>> I did have zswap enabled for a long while, and a lot of wine games,
>> plasmashell, xorg, kwin_x11 (and other) did crash randomly when reached 100% of
>> physical ram and swap was like almost never used.
>>
>> I could esilly open a lot of browser tabs and the browser or xorg would fail
>> every time.
>>
>> After disabling zswap no crashes at all.
>>
>> /etc/systemd/swap.conf
>> zswap_enabled=1
>> zswap_compressor=lz4 # lzo lz4
>> zswap_max_pool_percent=25 # 1-99
>> zswap_zpool=zbud # zbud z3fold
>
>
> So I did a number of tests and I confirm that under memory pressure
> with frontswap enabled I do see segfaults and memory corruptions in
> random user space applications.
>
> kernel: urxvt[338]: segfault at 20 ip 00007fc08889ae0d sp 00007ffc73a7fc40 error 6 in libc-2.26.so[7fc08881a000+1ae000]
> #0 0x00007fc08889ae0d _int_malloc (libc.so.6)
> #1 0x00007fc08889c2f3 malloc (libc.so.6)
> #2 0x0000560e6004bff7 _Z14rxvt_wcstoutf8PKwi (urxvt)
> #3 0x0000560e6005e75c n/a (urxvt)
> #4 0x0000560e6007d9f1 _ZN16rxvt_perl_interp6invokeEP9rxvt_term9hook_typez (urxvt)
> #5 0x0000560e6003d988 _ZN9rxvt_term9cmd_parseEv (urxvt)
> #6 0x0000560e60042804 _ZN9rxvt_term6pty_cbERN2ev2ioEi (urxvt)
> #7 0x0000560e6005c10f _Z17ev_invoke_pendingv (urxvt)
> #8 0x0000560e6005cb55 ev_run (urxvt)
> #9 0x0000560e6003b9b9 main (urxvt)
> #10 0x00007fc08883af4a __libc_start_main (libc.so.6)
> #11 0x0000560e6003f9da _start (urxvt)
>
> kernel: urxvt[343]: segfault at 10 ip 00007fa56bd7d52b sp 00007ffc09783a40 error 4 in libc-2.26.so[7fa56bcfd000+1ae000]
> #0 0x00007fa56bd7d52b _int_malloc (libc.so.6)
> #1 0x00007fa56bd7f2f3 malloc (libc.so.6)
> #2 0x00007fa56b3d6097 n/a (libxcb.so.1)
> #3 0x00007fa56b3d64d8 n/a (libxcb.so.1)
> #4 0x00007fa56c921b79 n/a (libX11.so.6)
> #5 0x00007fa56c921ceb n/a (libX11.so.6)
> #6 0x00007fa56c921fdd _XEventsQueued (libX11.so.6)
> #7 0x00007fa56c913c49 XEventsQueued (libX11.so.6)
> #8 0x000055b35cfc3262 _ZN12rxvt_display8flush_cbERN2ev7prepareEi (urxvt)
> #9 0x000055b35cfc910f _Z17ev_invoke_pendingv (urxvt)
> #10 0x000055b35cfc9c02 ev_run (urxvt)
> #11 0x000055b35cfa89b9 main (urxvt)
> #12 0x00007fa56bd1df4a __libc_start_main (libc.so.6)
> #13 0x000055b35cfac9da _start (urxvt)
>
> Stack trace of thread 351:
> #0 0x00007f5baaee7860 raise (libc.so.6)
> #1 0x00007f5baaee8ec9 abort (libc.so.6)
> #2 0x00007f5baaf30849 __malloc_assert (libc.so.6)
> #3 0x00007f5baaf34011 _int_malloc (libc.so.6)
> #4 0x00007f5baaf352f3 malloc (libc.so.6)
> #5 0x00007f5baaf71cad __alloc_dir (libc.so.6)
> #6 0x00007f5baaf71dbd opendir_tail (libc.so.6)
> #7 0x00007f5bab5bbac4 Perl_pp_open_dir (libperl.so)
> #8 0x00007f5bab55fec6 Perl_runops_standard (libperl.so)
> #9 0x00007f5bab4d9390 Perl_call_sv (libperl.so)
> #10 0x00005611f097e190 _ZN16rxvt_perl_interp6invokeEP9rxvt_term9hook_typez (urxvt)
> #11 0x00005611f0947acb _ZN9rxvt_term14init_resourcesEiPKPKc (urxvt)
> #12 0x00005611f0948da8 _ZN9rxvt_term5init2EiPKPKc (urxvt)
> #13 0x00005611f097a0af n/a (urxvt)
> #14 0x00007f5bab568259 Perl_pp_entersub (libperl.so)
> #15 0x00007f5bab55fec6 Perl_runops_standard (libperl.so)
> #16 0x00007f5bab4d9390 Perl_call_sv (libperl.so)
> #17 0x00005611f097e190 _ZN16rxvt_perl_interp6invokeEP9rxvt_term9hook_typez (urxvt)
> #18 0x00005611f0939a77 _ZN9rxvt_term9key_pressER9XKeyEvent (urxvt)
> #19 0x00005611f093d77a _ZN9rxvt_term4x_cbER7_XEvent (urxvt)
> #20 0x00005611f09572e8 _ZN12rxvt_display8flush_cbERN2ev7prepareEi (urxvt)
> #21 0x00005611f095d10f _Z17ev_invoke_pendingv (urxvt)
> #22 0x00005611f095dc02 ev_run (urxvt)
> #23 0x00005611f093c9b9 main (urxvt)
> #24 0x00007f5baaed3f4a __libc_start_main (libc.so.6)
> #25 0x00005611f09409da _start (urxvt)
>
> and so on.
>
>
> However, the problem is not specific to 4.14.15 or 4.14.11.
>
> I manages to track it down to 4.14 merge window, so we are basically
> looking at 4.14-rc0+
>
> The bisect log looks as follows:
>
> git bisect start
> # bad: [2bd6bf03f4c1c59381d62c61d03f6cc3fe71f66e] Linux 4.14-rc1
> git bisect bad 2bd6bf03f4c1c59381d62c61d03f6cc3fe71f66e
> # good: [569dbb88e80deb68974ef6fdd6a13edb9d686261] Linux 4.13
> git bisect good 569dbb88e80deb68974ef6fdd6a13edb9d686261
> # good: [aae3dbb4776e7916b6cd442d00159bea27a695c1] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
> git bisect good aae3dbb4776e7916b6cd442d00159bea27a695c1
> # bad: [2f173d2688559a6f85643d38a2ad6f45eb420c42] KVM: x86: Fix immediate_exit handling for uninitialized AP
> git bisect bad 2f173d2688559a6f85643d38a2ad6f45eb420c42
> # bad: [d969443064abf2f51510559a5b01325eaabfcb1d] Merge tag 'sound-4.14-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
> git bisect bad d969443064abf2f51510559a5b01325eaabfcb1d
> # bad: [a0725ab0c7536076d5477264420ef420ebb64501] Merge branch 'for-4.14/block' of git://git.kernel.dk/linux-block
> git bisect bad a0725ab0c7536076d5477264420ef420ebb64501
> # bad: [f92e3da18b7d5941468040af962c201235148301] Merge branch 'efi-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> git bisect bad f92e3da18b7d5941468040af962c201235148301
> # good: [1c9fe4409ce3e9c78b1ed96ee8ed699d4f03bf33] x86/mm: Document how CR4.PCIDE restore works
> git bisect good 1c9fe4409ce3e9c78b1ed96ee8ed699d4f03bf33
> # bad: [da99ecf117fce6570bd3989263d68ee0007e1249] mm: replace TIF_MEMDIE checks by tsk_is_oom_victim
> git bisect bad da99ecf117fce6570bd3989263d68ee0007e1249
> # good: [f7b68046873724129798c405e1a4e326b409c08f] mm: use find_get_pages_range() in filemap_range_has_page()
> git bisect good f7b68046873724129798c405e1a4e326b409c08f
> # bad: [824f973904a1108806fa0fbe15dc93ee9ecd9e0a] userfaultfd: selftest: enable testing of UFFDIO_ZEROPAGE for shmem
> git bisect bad 824f973904a1108806fa0fbe15dc93ee9ecd9e0a
> # good: [98cc093cba1e925eb34963dedb5f1684f1bdb2f4] block, THP: make block_device_operations.rw_page support THP
> git bisect good 98cc093cba1e925eb34963dedb5f1684f1bdb2f4
> # bad: [fe490cc0fe9e6ee48cc48bb5dc463bc5f0f1428f] mm, THP, swap: add THP swapping out fallback counting
> git bisect bad fe490cc0fe9e6ee48cc48bb5dc463bc5f0f1428f
> # good: [3e14a57b2416b7c94189b95baffd673cf5e0d0a3] memcg, THP, swap: support move mem cgroup charge for THP swapped out
> git bisect good 3e14a57b2416b7c94189b95baffd673cf5e0d0a3
> # good: [d6810d730022016d9c0f389452b86b035dba1492] memcg, THP, swap: make mem_cgroup_swapout() support THP
> git bisect good d6810d730022016d9c0f389452b86b035dba1492
> # bad: [bd4c82c22c367e068acb1ec9ec02be2fac3e09e2] mm, THP, swap: delay splitting THP after swapped out
> git bisect bad bd4c82c22c367e068acb1ec9ec02be2fac3e09e2
> # first bad commit: [bd4c82c22c367e068acb1ec9ec02be2fac3e09e2] mm, THP, swap: delay splitting THP after swapped out
>
>
> The suspected first bad commit is:
>
> bd4c82c22c367e068acb1ec9ec02be2fac3e09e2 is the first bad commit
> commit bd4c82c22c367e068acb1ec9ec02be2fac3e09e2
> Author: Huang Ying
> Date: Wed Sep 6 16:22:49 2017 -0700
>
> mm, THP, swap: delay splitting THP after swapped out
>
> In this patch, splitting transparent huge page (THP) during swapping out
> is delayed from after adding the THP into the swap cache to after
> swapping out finishes. After the patch, more operations for the
> anonymous THP reclaiming, such as writing the THP to the swap device,
> removing the THP from the swap cache could be batched. So that the
> performance of anonymous THP swapping out could be improved.
>
> This is the second step for the THP swap support. The plan is to delay
> splitting the THP step by step and avoid splitting the THP finally.
>
> With the patchset, the swap out throughput improves 42% (from about
> 5.81GB/s to about 8.25GB/s) in the vm-scalability swap-w-seq test case
> with 16 processes. At the same time, the IPI (reflect TLB flushing)
> reduced about 78.9%. The test is done on a Xeon E5 v3 system. The swap
> device used is a RAM simulated PMEM (persistent memory) device. To test
> the sequential swapping out, the test case creates 8 processes, which
> sequentially allocate and write to the anonymous pages until the RAM and
> part of the swap device is used up.
>
> Link: http://lkml.kernel.org/r/[email protected]
Can you give me some detailed steps to reproduce this? Like the
kernel configuration file, swap configuration, etc. Any kernel
WARNING during testing? Can you reproduce this with a real swap
device instead of zswap?
Best Regards,
Huang, Ying
Hi,
On (02/04/18 22:21), huang ying wrote:
[..]
> >> After disabling zswap no crashes at all.
> >>
> >> /etc/systemd/swap.conf
> >> zswap_enabled=1
> >> zswap_compressor=lz4 # lzo lz4
> >> zswap_max_pool_percent=25 # 1-99
> >> zswap_zpool=zbud # zbud z3fold
> >
[..]
> Can you give me some detailed steps to reproduce this? Like the
> kernel configuration file, swap configuration, etc. Any kernel
> WARNING during testing? Can you reproduce this with a real swap
> device instead of zswap?
No warnings (at least no warnings with my .config). Tested it only with
zram based swap (I'm running swap-less x86 systems, so zram is the easiest
way). It seems it's THP + frontswap that makes things unstable, rather
than THP + swap.
Kernel zswap boot params:
zswap.enabled=1 zswap.compressor=lz4 zswap.max_pool_percent=10 zswap.zpool=zbud
Then I add a 4G zram swap and run a silly memory hogger. I don't think
you'll have any problems reproducing it, but just in case I attached my
.config
-ss
Sergey Senozhatsky <[email protected]> writes:
> Hi,
>
> On (02/04/18 22:21), huang ying wrote:
> [..]
>> >> After disabling zswap no crashes at all.
>> >>
>> >> /etc/systemd/swap.conf
>> >> zswap_enabled=1
>> >> zswap_compressor=lz4 # lzo lz4
>> >> zswap_max_pool_percent=25 # 1-99
>> >> zswap_zpool=zbud # zbud z3fold
>> >
> [..]
>> Can you give me some detailed steps to reproduce this? Like the
>> kernel configuration file, swap configuration, etc. Any kernel
>> WARNING during testing? Can you reproduce this with a real swap
>> device instead of zswap?
>
> No warnings (at least no warnings with my .config). Tested it only with
> zram based swap (I'm running swap-less x86 systems, so zram is the easiest
> way). It seems it's THP + frontswap that makes things unstable, rather
> than THP + swap.
>
> Kernel zswap boot params:
> zswap.enabled=1 zswap.compressor=lz4 zswap.max_pool_percent=10 zswap.zpool=zbud
>
> Then I add a 4G zram swap and run a silly memory hogger. I don't think
> you'll have any problems reproducing it, but just in case I attached my
> .config
Thanks for your information! I will try to reproduce it today!
Best Regards,
Huang, Ying
> -ss
Sergey Senozhatsky <[email protected]> writes:
> Hi,
>
> On (02/04/18 22:21), huang ying wrote:
> [..]
>> >> After disabling zswap no crashes at all.
>> >>
>> >> /etc/systemd/swap.conf
>> >> zswap_enabled=1
>> >> zswap_compressor=lz4 # lzo lz4
>> >> zswap_max_pool_percent=25 # 1-99
>> >> zswap_zpool=zbud # zbud z3fold
>> >
> [..]
>> Can you give me some detailed steps to reproduce this? Like the
>> kernel configuration file, swap configuration, etc. Any kernel
>> WARNING during testing? Can you reproduce this with a real swap
>> device instead of zswap?
>
> No warnings (at least no warnings with my .config). Tested it only with
> zram based swap (I'm running swap-less x86 systems, so zram is the easiest
> way). It seems it's THP + frontswap that makes things unstable, rather
> than THP + swap.
>
> Kernel zswap boot params:
> zswap.enabled=1 zswap.compressor=lz4 zswap.max_pool_percent=10 zswap.zpool=zbud
>
> Then I add a 4G zram swap and run a silly memory hogger. I don't think
> you'll have any problems reproducing it, but just in case I attached my
> .config
I have successfully reproduced the issue and find the problem. The
following patch fix the issue for me, can you try it?
Best Regards,
Huang, Ying
---------------------------------8<-------------------------------
From 4c52d531680f91572ebc6f4525a018e32a934ef0 Mon Sep 17 00:00:00 2001
From: Huang Ying <[email protected]>
Date: Mon, 5 Feb 2018 19:27:43 +0800
Subject: [PATCH] fontswap thp fix
---
mm/page_io.c | 2 +-
mm/vmscan.c | 16 +++++++++++++---
2 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/mm/page_io.c b/mm/page_io.c
index b41cf9644585..6dca817ae7a0 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -250,7 +250,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)
unlock_page(page);
goto out;
}
- if (frontswap_store(page) == 0) {
+ if (!PageTransHuge(page) && frontswap_store(page) == 0) {
set_page_writeback(page);
unlock_page(page);
end_page_writeback(page);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index bee53495a829..d1c1e00b08bb 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -55,6 +55,7 @@
#include <linux/swapops.h>
#include <linux/balloon_compaction.h>
+#include <linux/frontswap.h>
#include "internal.h"
@@ -1063,14 +1064,23 @@ static unsigned long shrink_page_list(struct list_head *page_list,
/* cannot split THP, skip it */
if (!can_split_huge_page(page, NULL))
goto activate_locked;
+ /*
+ * Split THP if frontswap enabled,
+ * because it cannot process THP
+ */
+ if (frontswap_enabled()) {
+ if (split_huge_page_to_list(
+ page, page_list))
+ goto activate_locked;
+ }
/*
* Split pages without a PMD map right
* away. Chances are some or all of the
* tail pages can be freed without IO.
*/
- if (!compound_mapcount(page) &&
- split_huge_page_to_list(page,
- page_list))
+ else if (!compound_mapcount(page) &&
+ split_huge_page_to_list(page,
+ page_list))
goto activate_locked;
}
if (!add_to_swap(page)) {
--
2.15.1
On (02/05/18 20:00), Huang, Ying wrote:
[..]
> I have successfully reproduced the issue and find the problem. The
> following patch fix the issue for me, can you try it?
That was quick ;)
> ---------------------------------8<-------------------------------
> From 4c52d531680f91572ebc6f4525a018e32a934ef0 Mon Sep 17 00:00:00 2001
> From: Huang Ying <[email protected]>
> Date: Mon, 5 Feb 2018 19:27:43 +0800
> Subject: [PATCH] fontswap thp fix
Seems to be fixing the problem on my x86 box. Executed several tests, no
crashes were observed. Can run more tests tomorrow.
============================================================================
Probably unrelated, but may be it is related: my X server used to hang
sometimes (rarely) which I suspect was/is caused by nouveau driver. It,
surprisingly, didn't hang this time around. Nouveau spitted a number
of backtraces, but X server managed to survive it. Any chance that
nouveau-X server thing was caused by THP?
[ 308.986648] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
[ 308.986653] nouveau 0000:01:00.0: swiotlb: coherent allocation failed, size=2097152
[ 308.986657] CPU: 5 PID: 343 Comm: Xorg Not tainted 4.15.0-next-20180205-dbg-00021-ga1282bf979c4-dirty #2480
[ 308.986659] Call Trace:
[ 308.986667] dump_stack+0x46/0x59
[ 308.986671] swiotlb_alloc_coherent+0x164/0x174
[ 308.986675] ttm_dma_pool_get_pages+0x16e/0x3ce
[ 308.986679] ttm_dma_populate+0x108/0x2af
[ 308.986681] ttm_tt_bind+0x32/0x57
[ 308.986684] ttm_bo_handle_move_mem+0x120/0x328
[ 308.986687] ? ttm_bo_mem_space+0x170/0x3a0
[ 308.986690] ttm_bo_validate+0x7b/0xd9
[ 308.986694] ? __mutex_trylock_or_owner+0x43/0x54
[ 308.986697] ttm_bo_init_reserved+0x31f/0x38a
[ 308.986700] ttm_bo_init+0x52/0x76
[ 308.986703] ? nouveau_bo_invalidate_caches+0x8/0x8
[ 308.986706] nouveau_bo_new+0x4a8/0x4c7
[ 308.986709] ? nouveau_bo_invalidate_caches+0x8/0x8
[ 308.986712] nouveau_gem_new+0x49/0xcc
[ 308.986715] nouveau_gem_ioctl_new+0x3e/0x9f
[ 308.986717] ? nouveau_gem_new+0xcc/0xcc
[ 308.986720] drm_ioctl_kernel+0x64/0xa0
[ 308.986723] drm_ioctl+0x1d6/0x2a8
[ 308.986725] ? nouveau_gem_new+0xcc/0xcc
[ 308.986729] ? _raw_spin_unlock_irq+0x13/0x24
[ 308.986732] ? free_swap_slot+0xad/0xc2
[ 308.986735] nouveau_drm_ioctl+0x71/0xa4
[ 308.986738] vfs_ioctl+0x1e/0x2b
[ 308.986741] do_vfs_ioctl+0x505/0x518
[ 308.986745] ? __fget+0x5d/0x67
[ 308.986747] SyS_ioctl+0x3e/0x5a
[ 308.986751] do_syscall_64+0x17f/0x196
[ 308.986754] entry_SYSCALL_64_after_hwframe+0x21/0x86
[ 308.986757] RIP: 0033:0x7f0395218d87
[ 308.986759] RSP: 002b:00007ffc210c4c48 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 308.986761] RAX: ffffffffffffffda RBX: 000056213e828350 RCX: 00007f0395218d87
[ 308.986764] RDX: 00007ffc210c4ca0 RSI: 00000000c0306480 RDI: 000000000000000b
[ 308.986765] RBP: 00007ffc210c4ca0 R08: 0000000000000004 R09: 0000000000000006
[ 308.986767] R10: 000056213e320010 R11: 0000000000000246 R12: 00000000c0306480
[ 308.986769] R13: 000000000000000b R14: 00007ffc210c4d60 R15: 000056213e35e770
[ 309.264408] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
[ 309.264412] nouveau 0000:01:00.0: swiotlb: coherent allocation failed, size=2097152
[ 309.264416] CPU: 4 PID: 343 Comm: Xorg Not tainted 4.15.0-next-20180205-dbg-00021-ga1282bf979c4-dirty #2480
[ 309.264418] Call Trace:
[ 309.264425] dump_stack+0x46/0x59
[ 309.264429] swiotlb_alloc_coherent+0x164/0x174
[ 309.264434] ttm_dma_pool_get_pages+0x16e/0x3ce
[ 309.264437] ttm_dma_populate+0x108/0x2af
[ 309.264440] ttm_tt_bind+0x32/0x57
[ 309.264442] ttm_bo_handle_move_mem+0x120/0x328
[ 309.264445] ? ttm_bo_mem_space+0x170/0x3a0
[ 309.264448] ttm_bo_validate+0x7b/0xd9
[ 309.264452] ? __mutex_trylock_or_owner+0x43/0x54
[ 309.264454] ttm_bo_init_reserved+0x31f/0x38a
[ 309.264457] ttm_bo_init+0x52/0x76
[ 309.264461] ? nouveau_bo_invalidate_caches+0x8/0x8
[ 309.264463] nouveau_bo_new+0x4a8/0x4c7
[ 309.264466] ? nouveau_bo_invalidate_caches+0x8/0x8
[ 309.264469] nouveau_gem_new+0x49/0xcc
[ 309.264471] nouveau_gem_ioctl_new+0x3e/0x9f
[ 309.264474] ? nouveau_gem_new+0xcc/0xcc
[ 309.264477] drm_ioctl_kernel+0x64/0xa0
[ 309.264479] drm_ioctl+0x1d6/0x2a8
[ 309.264482] ? nouveau_gem_new+0xcc/0xcc
[ 309.264485] nouveau_drm_ioctl+0x71/0xa4
[ 309.264489] vfs_ioctl+0x1e/0x2b
[ 309.264491] do_vfs_ioctl+0x505/0x518
[ 309.264495] ? __fget+0x5d/0x67
[ 309.264497] SyS_ioctl+0x3e/0x5a
[ 309.264500] do_syscall_64+0x17f/0x196
[ 309.264504] entry_SYSCALL_64_after_hwframe+0x21/0x86
[ 309.264506] RIP: 0033:0x7f0395218d87
[ 309.264508] RSP: 002b:00007ffc210c4c48 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 309.264511] RAX: ffffffffffffffda RBX: 000056213e828350 RCX: 00007f0395218d87
[ 309.264513] RDX: 00007ffc210c4ca0 RSI: 00000000c0306480 RDI: 000000000000000b
[ 309.264515] RBP: 00007ffc210c4ca0 R08: 0000000000000004 R09: 00007f03954ddad0
[ 309.264517] R10: 00007f03913786a1 R11: 0000000000000246 R12: 00000000c0306480
[ 309.264518] R13: 000000000000000b R14: 00007ffc210c4d60 R15: 000056213e35e770
-ss
On (02/05/18 21:39), Sergey Senozhatsky wrote:
> > I have successfully reproduced the issue and find the problem. The
> > following patch fix the issue for me, can you try it?
>
> That was quick ;)
>
> > ---------------------------------8<-------------------------------
> > From 4c52d531680f91572ebc6f4525a018e32a934ef0 Mon Sep 17 00:00:00 2001
> > From: Huang Ying <[email protected]>
> > Date: Mon, 5 Feb 2018 19:27:43 +0800
> > Subject: [PATCH] fontswap thp fix
>
> Seems to be fixing the problem on my x86 box. Executed several tests, no
> crashes were observed. Can run more tests tomorrow.
>
>
> ============================================================================
>
> Probably unrelated, but may be it is related: my X server used to hang
> sometimes (rarely) which I suspect was/is caused by nouveau driver. It,
> surprisingly, didn't hang this time around. Nouveau spitted a number
> of backtraces, but X server managed to survive it. Any chance that
> nouveau-X server thing was caused by THP?
No, wait. How could it be... I don't even use frontswap usually.
Sorry for the silly question.
-ss
On Mon, 5 Feb 2018 21:39:47 +0900 Sergey Senozhatsky <[email protected]> wrote:
> > ---------------------------------8<-------------------------------
> > From 4c52d531680f91572ebc6f4525a018e32a934ef0 Mon Sep 17 00:00:00 2001
> > From: Huang Ying <[email protected]>
> > Date: Mon, 5 Feb 2018 19:27:43 +0800
> > Subject: [PATCH] fontswap thp fix
>
> Seems to be fixing the problem on my x86 box. Executed several tests, no
> crashes were observed. Can run more tests tomorrow.
Thanks. I queued this for 4.16 with a cc:stable and a Fixes:
bd4c82c22c367e068 ("mm, THP, swap: delay splitting THP after swapped
out").
Can we please have a changelog ;)
Andrew Morton <[email protected]> writes:
> On Mon, 5 Feb 2018 21:39:47 +0900 Sergey Senozhatsky <[email protected]> wrote:
>
>> > ---------------------------------8<-------------------------------
>> > From 4c52d531680f91572ebc6f4525a018e32a934ef0 Mon Sep 17 00:00:00 2001
>> > From: Huang Ying <[email protected]>
>> > Date: Mon, 5 Feb 2018 19:27:43 +0800
>> > Subject: [PATCH] fontswap thp fix
>>
>> Seems to be fixing the problem on my x86 box. Executed several tests, no
>> crashes were observed. Can run more tests tomorrow.
>
> Thanks. I queued this for 4.16 with a cc:stable and a Fixes:
> bd4c82c22c367e068 ("mm, THP, swap: delay splitting THP after swapped
> out").
>
> Can we please have a changelog ;)
Sure! I will send out today.
Best Regards,
Huang, Ying
On (02/05/18 16:24), Andrew Morton wrote:
> On Mon, 5 Feb 2018 21:39:47 +0900 Sergey Senozhatsky <[email protected]> wrote:
>
> > > ---------------------------------8<-------------------------------
> > > From 4c52d531680f91572ebc6f4525a018e32a934ef0 Mon Sep 17 00:00:00 2001
> > > From: Huang Ying <[email protected]>
> > > Date: Mon, 5 Feb 2018 19:27:43 +0800
> > > Subject: [PATCH] fontswap thp fix
> >
> > Seems to be fixing the problem on my x86 box. Executed several tests, no
> > crashes were observed. Can run more tests tomorrow.
>
> Thanks. I queued this for 4.16 with a cc:stable and a Fixes:
> bd4c82c22c367e068 ("mm, THP, swap: delay splitting THP after swapped
> out").
>
> Can we please have a changelog ;)
Thanks!
Tested-by: Sergey Senozhatsky <[email protected]>
-ss