Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <2E04DD7753BE0E4ABABF0B664610AD6F2620CAF7@dggeml528-mbx.china.huawei.com>
In-Reply-To: <2E04DD7753BE0E4ABABF0B664610AD6F2620CAF7@dggeml528-mbx.china.huawei.com>
From:   Shakeel Butt <shakeelb@google.com>
Date:   Tue, 21 Jul 2020 07:45:14 -0700
Message-ID: <CALvZod7xGNxJxTcJmo8mCVAgDkPdC5Pp12rhuBNsFsw-Yv=e+A@mail.gmail.com>
Subject: Re: PROBLEM: cgroup cost too much memory when transfer small files to tmpfs
To:     jingrui <jingrui@huawei.com>
Cc:     "tj@kernel.org" <tj@kernel.org>, Lizefan <lizefan@huawei.com>,
        "hannes@cmpxchg.org" <hannes@cmpxchg.org>,
        "mhocko@kernel.org" <mhocko@kernel.org>,
        "vdavydov.dev@gmail.com" <vdavydov.dev@gmail.com>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        caihaomin <caihaomin@huawei.com>,
        "Weiwei (N)" <wick.wei@huawei.com>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Jul 21, 2020 at 4:20 AM jingrui <jingrui@huawei.com> wrote:
>
> Cc: Johannes Weiner <hannes@cmpxchg.org> ; Michal Hocko <mhocko@kernel.org>; Vladimir Davydov <vdavydov.dev@gmail.com>
>
> Thanks.
>
> ---
> PROBLEM: cgroup cost too much memory when transfer small files to tmpfs.
>
> keywords: cgroup PERCPU/memory cost too much.
>
> description:
>
> We send small files from node-A to node-B tmpfs /tmp directory using sftp. On
> node-B the systemd configured with pam on like below.
>
> cat /etc/pam.d/password-auth | grep systemd
> -session     optional      pam_systemd.so
>
> So when transfer a file, a systemd session is created, that means a cgroup is
> created, then file saved at /tmp will associated with a cgroup object. After
> file transferred, session and cgroup-dir will be removed, but the file in /tmp
> still associated with the cgroup object.

Is there a way for you to re-use the cgroup instead of creating and
deleting cgroup for each individual file transfer session?

> The PERCPU memory in cgroup/css object
> cost a lot(about 0.5MB/per-cgroup-object) on 200/cpus machine.
>
> When lot of small files transferred to tmpfs, the cgroup/css object memory
> cost become huge in this scenes to be used.
>
> systemd related issue: https://github.com/systemd/systemd/issues/16499
>
> kernel version: 4.19+
>
> Problem:
>
> 1. Do we have any idea to descrease cgroup memory cost in this case?
> 2. When user remove cgroup directory, does it possible associated file memory to root cgroup?

No, the memory remains associated with the cgroup and the cgroup
becomes zombie on deletion.

> 3. Can we provide an option that do not associate memory with cgroup in tmpfs?

Only way, if you don't want to disable memcg, is to move the file
receiver process to root cgroup.