Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Ivan Teterevkov <ivan.teterevkov@nutanix.com>
To:     Chris Down <chris@chrisdown.name>,
        Matthew Wilcox <willy@infradead.org>
CC:     David Rientjes <rientjes@google.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "mchehab+samsung@kernel.org" <mchehab+samsung@kernel.org>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "jpoimboe@redhat.com" <jpoimboe@redhat.com>,
        "pawan.kumar.gupta@linux.intel.com" 
        <pawan.kumar.gupta@linux.intel.com>,
        "jgross@suse.com" <jgross@suse.com>,
        "oneukum@suse.com" <oneukum@suse.com>,
        "linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>
Subject: RE: [PATCH] mm/vmscan: add vm_swappiness configuration knobs
Thread-Topic: [PATCH] mm/vmscan: add vm_swappiness configuration knobs
Thread-Index: AdX3zHNqbuFpQvKERxyPueA21j6FDgAD0RWAACPlSzAAAfzUAAAA7+kAACrp4sA=
Date:   Fri, 13 Mar 2020 10:49:32 +0000
Message-ID: <BL0PR02MB56011828432D343371088516E9FA0@BL0PR02MB5601.namprd02.prod.outlook.com>
References: <BL0PR02MB560167492CA4094C91589930E9FC0@BL0PR02MB5601.namprd02.prod.outlook.com>
 <alpine.DEB.2.21.2003111227230.171292@chino.kir.corp.google.com>
 <BL0PR02MB5601808F36BE202813E9D562E9FD0@BL0PR02MB5601.namprd02.prod.outlook.com>
 <20200312133636.GJ22433@bombadil.infradead.org>
 <20200312140326.GA1701917@chrisdown.name>
In-Reply-To: <20200312140326.GA1701917@chrisdown.name>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: nutanix.com does not designate
 permitted sender hosts)
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: c43d75cb-871b-4aff-9e8f-08d7c73c36f9
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Mar 2020 10:49:32.8949
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: bb047546-786f-4de1-bd75-24e5b6f79043
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: /PlyYGk1cUVwPYztCcXTJ/qkOkRypmX3GN9feTl3D3yoJIZu5c6y7AYGJp4PiitkRv7mPm7MBQiOI69ORWRPHSX0rJQBLsiZgiV4dHZSGA4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR02MB4642
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.572
 definitions=2020-03-13_04:2020-03-12,2020-03-13 signatures=0
X-Proofpoint-Spam-Reason: safe
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thurs, 12 Mar 2020, Chris Down wrote:

> Matthew Wilcox writes:
> >On Thu, Mar 12, 2020 at 12:48:22PM +0000, Ivan Teterevkov wrote:
> >> This is exactly what I'm trying to avoid: in some distros there is no
> >> way to tackle the configuration early enough, e.g. in systemd-based
> >> systems the systemd is the process that starts first and arranges
> >> memcg in a way it's configured, but unfortunately, it doesn't offer th=
e
> swappiness knob.
> >
> >This sounds like a systemd problem.  Have you talked to the systemd
> >people about fixing it in systemd?
>=20
> Hi there ;-)
>=20
> In general most of us maintaining cgroups in systemd run with cgroup v2, =
so this
> isn't a problem we run into in production. The swappiness controls in gen=
eral
> don't make a whole lot of sense being distributed hierarchically, so they=
've been
> phased out entirely in cgroup v2.
>=20
> If there had been a patch years ago implementing this in systemd we'd pro=
bably
> have accepted it, but cgroup v1 is dying and I am really not in favour of=
 adding
> more code to massage its rough edges. We already have enough problems
> generated by it already.
>=20
> However, the following kludge in tmpfiles.d should work to solve your
> immediate
> problem:
>=20
> 	w /sys/fs/cgroup/memory/system.slice/memory.swappiness - - - - value
>=20
> Taking my systemd hat off and putting my -mm hat on: let's not add more h=
acky
> APIs at cgroup v1's behest, or we'll be here until we're pushing up the d=
aisies.
>=20
> Thanks,
>=20
> Chris

The above approach doesn't work for me in el7 with systemd 219 or ubuntu 19
with systemd 242 because presumably the systemd-tmpfiles services start too
late. Please find the snippet at the bottom of the email.

The only approach that seems to work is to set up a service to run:

$ find /sys/fs/cgroup/memory/ -name memory.swappiness | while read -r name;=
 do echo 0 > "${name}"; done

I think this is quite ugly because there might be a race condition with
the systemd that could be creating the slices while the find is running.
One could suggest constraining the depth and going from top to the
bottom of the memcg but this still looks inherently unstable.

This is why I ended up with the vm_swappiness patch (which I don't
quite like myself) but this appears to be the only rock solid option
unless I've missed anything obvious. There is no doubt that cgroup v1
is due for replacement and vm_swappiness is frightening but they still
have certain advantages to employ until they are history.

$ systemctl --version
systemd 242 (242)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +=
GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PC=
RE2 default-hierarchy=3Dhybrid

$ cat /etc/lsb-release
DISTRIB_ID=3DUbuntu
DISTRIB_RELEASE=3D19.10
DISTRIB_CODENAME=3Deoan
DISTRIB_DESCRIPTION=3D"Ubuntu 19.10"

$ uname -a
Linux ubuntu 5.6.0-rc5-custom #1 SMP Wed Mar 11 14:59:15 GMT 2020 x86_64 x8=
6_64 x86_64 GNU/Linux

$ tail -1 /etc/sysctl.conf
vm.swappiness=3D10

$ cat /etc/tmpfiles.d/10-swap.conf
w /sys/fs/cgroup/memory/system.slice/memory.swappiness - - - - 20
w /sys/fs/cgroup/memory/user.slice/memory.swappiness   - - - - 30

$ find /sys/fs/cgroup/memory -name memory.swappiness | while read -r name; =
do cat "${name}"; done | sort | uniq -c
      1 10
     32 20
      6 30
     21 60

$ find /sys/fs/cgroup/memory -name memory.swappiness | while read -r name; =
do echo "${name}"; cat "${name}"; done | grep --before-context=3D1 60
/sys/fs/cgroup/memory/system.slice/systemd-udevd.service/memory.swappiness
60
--
/sys/fs/cgroup/memory/system.slice/sys-fs-fuse-connections.mount/memory.swa=
ppiness
60
/sys/fs/cgroup/memory/system.slice/snap-gnome\x2d3\x2d28\x2d1804-116.mount/=
memory.swappiness
60
/sys/fs/cgroup/memory/system.slice/snap-gnome\x2dlogs-81.mount/memory.swapp=
iness
60
/sys/fs/cgroup/memory/system.slice/sys-kernel-config.mount/memory.swappines=
s
60
--
/sys/fs/cgroup/memory/system.slice/snap-core-7917.mount/memory.swappiness
60
/sys/fs/cgroup/memory/system.slice/sys-kernel-debug.mount/memory.swappiness
60
--
/sys/fs/cgroup/memory/system.slice/snap-gnome\x2dcharacters-399.mount/memor=
y.swappiness
60
/sys/fs/cgroup/memory/system.slice/swapfile.swap/memory.swappiness
60
--
/sys/fs/cgroup/memory/system.slice/snap-gtk\x2dcommon\x2dthemes-1440.mount/=
memory.swappiness
60
/sys/fs/cgroup/memory/system.slice/snap-gnome\x2dcharacters-317.mount/memor=
y.swappiness
60
--
/sys/fs/cgroup/memory/system.slice/systemd-journald.service/memory.swappine=
ss
60
--
/sys/fs/cgroup/memory/system.slice/dev-mqueue.mount/memory.swappiness
60
--
/sys/fs/cgroup/memory/system.slice/snap-gtk\x2dcommon\x2dthemes-1353.mount/=
memory.swappiness
60
/sys/fs/cgroup/memory/system.slice/snap-core-8689.mount/memory.swappiness
60
--
/sys/fs/cgroup/memory/system.slice/snap-gnome\x2d3\x2d28\x2d1804-71.mount/m=
emory.swappiness
60
--
/sys/fs/cgroup/memory/system.slice/snap-core18-1668.mount/memory.swappiness
60
--
/sys/fs/cgroup/memory/system.slice/snap-gnome\x2dcalculator-501.mount/memor=
y.swappiness
60
--
/sys/fs/cgroup/memory/system.slice/dev-hugepages.mount/memory.swappiness
60
--
/sys/fs/cgroup/memory/system.slice/snap-gnome\x2dcalculator-544.mount/memor=
y.swappiness
60
--
/sys/fs/cgroup/memory/system.slice/snap-core18-1223.mount/memory.swappiness
60