Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <0e64ed24-e676-6cfc-376f-f404e759f6f1@redhat.com>
Date:   Thu, 21 Apr 2022 12:21:10 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Subject: Re: [PATCH v3] vm_swappiness=0 should still try to avoid swapping
 anon memory
Content-Language: en-US
To:     Johannes Weiner <hannes@cmpxchg.org>
Cc:     linux-mm@kvack.org, akpm@linux-foundation.org,
        linux-kernel@vger.kernel.org, aquini@redhat.com,
        shakeelb@google.com, mhocko@suse.com, hakavlad@inbox.lv,
        David Hildenbrand <david@redhat.com>, llong@redhat.com
References: <20210809223740.59009-1-npache@redhat.com>
 <b5575449-08a2-2901-e7e1-a97150e4878c@redhat.com>
 <Yl8Dk+a1YDwupLMv@cmpxchg.org>
 <e495901c-f2f3-cc85-10a6-0d2944255b19@redhat.com>
 <YmASIHjTVndHHoL4@cmpxchg.org>
 <bc9f5209-5c59-c921-d85e-e2e54b2375db@redhat.com>
 <YmBUkNMVDQobgK4M@cmpxchg.org>
From:   Nico Pache <npache@redhat.com>
In-Reply-To: <YmBUkNMVDQobgK4M@cmpxchg.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 4/20/22 14:44, Johannes Weiner wrote:
>>
>> The larger issue is that our workload has regressed in performance.
>>
>> With V2 and swappiness=10 we are still seeing some swap, but very little tearing
>> down of THPs over time. With swappiness=0 it did some when swap but we are not
>> losings GBs of THPS (with your patch swappiness=0 has swap or THP issues on V2).
I meant to say `with your patch swappiness=0 does not swap or have thp issues on v2`
>>
>> With V1 and swappiness=(0|10)(with and without your patch), it swaps a ton and
>> ultimately leads to a significant amount of THP splitting. So the longer the
>> system/workload runs, the less likely we are to get THPs backing the guest and
>> the performance gain from THPs is lost.
> 
> I hate to ask, but is it possible this is a configuration issue
Im very glad you asked :)
> 
> One significant difference between V1 and V2 is that V1 has per-cgroup
> swappiness, which is inherited when the cgroup is created. So if you
> set sysctl vm.swappiness=0 after cgroups have been created, it will
> not update them. V2 cgroups do use vm.swappiness:

This is something I did not consider... Thank you for pointing that out!

The issue still occurs weather or not I set the swappiness value before the VM
boot. However this led me to find the icing on the cake :)

Even if I set vm.swappiness=0 at boot using sysctl.conf I was not considering
the fact that libvirtd was creating its own cgroup for the machines you start it
with... additionally it does not inherit the sysctl value (even when set at
boot)?!? How annoying...

The cgroups swappiness value is defaulted to 60. This to me seems wrong from a
libvirt/systemd POV. If the system is booted with swappiness=0 then why does the
(user|machine|system).splice cgroup ignore this value when it creates it cgroups
(see below).

I will have to dig a little further to find a cause/fix for this. This requires
the libvirt users to understand a number of intricacies that they really
shouldnt have to consider, and may lead to headaches like these ;P

Values of the memcgs created on boot (with sysctl.swappiness=0 on V1 boot)
------------------------------------------------------------------------
/sys/fs/cgroup/memory/memory.swappiness 				=0
/sys/fs/cgroup/memory/dev-hugepages.mount/memory.swappiness		=60
/sys/fs/cgroup/memory/dev-mqueue.mount/memory.swappiness		=60
/sys/fs/cgroup/memory/machine.slice/memory.swappiness			=60
/sys/fs/cgroup/memory/proc-sys-fs-binfmt_misc.mount/memory.swappiness	=0
/sys/fs/cgroup/memory/sys-fs-fuse-connections.mount/memory.swappiness	=0
/sys/fs/cgroup/memory/sys-kernel-config.mount/memory.swappiness		=0
/sys/fs/cgroup/memory/sys-kernel-debug.mount/memory.swappiness		=60
/sys/fs/cgroup/memory/sys-kernel-tracing.mount/memory.swappiness	=60
/sys/fs/cgroup/memory/system.slice/memory.swappiness			=60
/sys/fs/cgroup/memory/user.slice/memory.swappiness			=60

Some seem to inherit the cgroup/memory/memory.swappiness value and some do
not... This issue was brought up in a systemd issue with no solution or
documentation [1].

Libvirt in particular is using the machine.splice cgroup so it inherits the 60.
If i change that value to 0, then start the machine it now has swappiness 0.
$ echo 0 > /sys/fs/cgroup/memory/machine.slice/memory.swappiness
$ virsh start <guest-name>
$ cat /sys/fs/cgroup/memory/machine.slice/machine-qemu.scope/memory.swappiness
0

Thank you so much for your very insightful note that led to the real issue :)

> Thanks for verifying. I'll prepare a proper patch.

my issue with v1 vs v2 seems to go away with a much more sane value of
swappiness=10 on v1 (when actually set properly lol).

Also as per my results below, I actually dont think your patch caused much
change to my workload. Im not sure what happened the first time I ran it that
caused the swapping on v2 (before your patch)... perhaps I ran the older kernel
(~v5.14) that was still having issues with v2 or its the fact that the results
can differ between runs. sorry about that.

Here is the test results for your patch with V1 and V2 (swappiness=0/10):

Before Patch
-------------
-- V1(swappiness=0):
               total        used        free      shared  buff/cache   available
Mem:       264071432   257465704     1100160        4224     5505568     5064964
Swap:        4194300       47828     4146472

Node 0 AnonPages:      128068580 kB	Node 1 AnonPages:      128120400 kB
Node 0 AnonHugePages:  128012288 kB	Node 1 AnonHugePages:  128090112 kB
                                                               ^^^^^ no loss

-- V1(swappiness=10):
               total        used        free	  shared  buff/cache   available
Mem:	   264071432   257364436      972048        3972     5734948     5164520
Swap:        4194300      235028     3959272

Node 0 AnonPages:      128015404 kB     Node 1 AnonPages:      128002788 kB
Node 0 AnonHugePages:  128002048 kB     Node 1 AnonHugePages:  120576000 kB
                                                               ^^^^^ some loss

-- V2(swappiness=0):
               total        used        free	  shared  buff/cache   available
Mem:	   264071432   257609352      924692        4664     5537388     4921236
Swap:        4194300           0     4194300
                           ^^^^^ No Swap
Node 0 AnonPages:      128083104 kB     Node 1 AnonPages:      128180576 kB
Node 0 AnonHugePages:  128002048 kB     Node 1 AnonHugePages:  128124928 kB
                                                               ^^^^^ No loss

-- V2(swappiness=10):
               total        used        free	  shared  buff/cache   available
Mem:	   264071432   257407576      918124        4632     5745732     5101764
Swap:        4194300      220424     3973876
                           ^^^^^ Some Swap
Node 0 AnonPages:      128109700 kB	Node 1 AnonPages:      127918164 kB
Node 0 AnonHugePages:  128006144 kB	Node 1 AnonHugePages:  120569856 kB
                                                               ^^^^^ some loss

After Patch
-------------
-- V1:swappiness=0
               total        used        free	  shared  buff/cache   available
Mem:	   264071432   257538832      945276        4368     5587324     4991852
Swap:        4194300        9276     4185024

Node 0 AnonPages:      128133932 kB	Node 1 AnonPages:      128100540 kB
Node 0 AnonHugePages:  128047104 kB	Node 1 AnonHugePages:  128061440 kB


-- V1:swappiness=10
               total        used        free	  shared  buff/cache   available
Mem:	   264071432   257428564      969252        4384     5673616     5100824
Swap:        4194300      138936     4055364
                           ^^^^^ Some Swap
Node 0 AnonPages:      128161724 kB     Node 1 AnonPages:      127945368 kB
Node 0 AnonHugePages:  128043008 kB     Node 1 AnonHugePages:  120221696 kB
                                                               ^^^^^ some loss

-- V2(swappiness=0):
               total        used        free      shared  buff/cache   available
Mem:       264071432   257536896      927424        4664     5607112     4993184
Swap:        4194300           0     4194300

Node 0 AnonPages:      128145476 kB	Node 1 AnonPages:      128111908 kB
Node 0 AnonHugePages:  128026624 kB	Node 1 AnonHugePages:  128090112 kB

-- V2(swappiness=10):
               total        used        free	  shared  buff/cache   available
Mem:	   264071432   257423936     1007076        4548     5640420     5106544
Swap:        4194300	  156016     4038284

Node 0 AnonPages:      128133264 kB     Node 1 AnonPages:      127955952 kB
Node 0 AnonHugePages:  128018432 kB     Node 1 AnonHugePages:  122507264 kB
                                                           ^^^^ slightly better

The only notable difference between before/after your patch is that with your
patch the THP tearing was slightly better, resulting in an extra 2GB as seen in
the last result. This may just be noise.

I'll have to see if I can find a fix for this in either the kernel, libvirt, or
systemd, and will follow up if I do. If not this should at least be documented
correctly. Given the fact cgroupV1 is in limited support mode upstream, and
systemd's hesitancy to make changes for V1, we may how to go down our own
avenues to ensure our customers dont run into this issue.

Big Thanks!
-- Nico

[1] - https://github.com/systemd/systemd/issues/9276