Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <ca5e57fd-4699-2cec-b328-3d6bac43c8ef@bytedance.com>
Date:   Tue, 6 Sep 2022 18:37:40 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.13.0
Subject: Re: [External] Re: [PATCH] cgroup/cpuset: Add a new isolated
 mems.policy type.
To:     Michal Hocko <mhocko@suse.com>
Cc:     hannes@cmpxchg.org, roman.gushchin@linux.dev,
        linux-kernel@vger.kernel.org, cgroups@vger.kernel.org,
        linux-mm@kvack.org, lizefan.x@bytedance.com,
        wuyun.abel@bytedance.com
References: <20220904040241.1708-1-hezhongkun.hzk@bytedance.com>
 <YxWbBYZKDTrkmlOe@dhcp22.suse.cz>
 <0e5f380b-9201-0f56-9144-ce8449491fc8@bytedance.com>
 <YxXUjvWmZoG9vVNV@dhcp22.suse.cz>
From:   Zhongkun He <hezhongkun.hzk@bytedance.com>
In-Reply-To: <YxXUjvWmZoG9vVNV@dhcp22.suse.cz>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

> On Mon 05-09-22 18:30:55, Zhongkun He wrote:
>> Hi Michal, thanks for your reply.
>>
>> The current 'mempolicy' is hierarchically independent. The default value of
>> the child is to inherit from the parent. The modification of the child
>> policy will not be restricted by the parent.
> 
> This breaks cgroup fundamental property of hierarchical enforcement of
> each property. And as such it is a no go.
> 
>> Of course, there are other options, such as the child's policy mode must be
>> the same as the parent's. node can be the subset of parent's, but the
>> interleave type will be complicated, that's why hierarchy independence is
>> used. It would be better if you have other suggestions?
> 
> Honestly, I am not really sure cgroup cpusets is a great fit for this
> usecase. It would be probably better to elaborate some more what are the
> existing shortcomings and what you would like to achieve. Just stating
> the syscall is a hard to use interface is not quite clear on its own.
> 
> Btw. have you noticed this question?
> 
>>> What is the hierarchical behavior of the policy? Say parent has a
>>> stronger requirement (say bind) than a child (prefer)?
>>>> How to use the mempolicy interface:
>>>> 	echo prefer:2 > /sys/fs/cgroup/zz/cpuset.mems.policy
>>>> 	echo bind:1-3 > /sys/fs/cgroup/zz/cpuset.mems.policy
>>>>           echo interleave:0,1,2,3 >/sys/fs/cgroup/zz/cpuset.mems.policy
>>>
>>> Am I just confused or did you really mean to combine all these
>>> together?
>

Hi Michal, thanks for your reply.

 >>Say parent has a stronger requirement (say bind) than a child(prefer)?

Yes, combine all these together. The parent's task will use 'bind', 
child's use 'prefer'.This is the current implementation, and we can 
discuss and modify it together if there are other suggestions.

1:Existing shortcomings

In our use case, the application and the control plane are two separate 
systems. When the application is created, it doesn't know how to use 
memory, and it doesn't care. The control plane will decide the memory 
usage policy based on different reasons (the attributes of the 
application itself, the priority, the remaining resources of the 
system). Currently, numactl is used to set it at program startup, and 
the child process will inherit the mempolicy. But we can't dynamically 
adjust the memory policy, except restart, the memory policy will not change.

2:Our goals

For the above reasons, we want to create a mempolicy at the cgroup 
level. Usually processes under a cgroup have the same priority and 
attributes, and we can dynamically adjust the memory allocation strategy 
according to the remaining resources of the system. For example, a 
low-priority cgroup uses the 'bind:2-3' policy, and a high-priority 
cgroup uses bind:0-1. When resources are insufficient, it will be 
changed to bind:3, bind:0-2 by control plane, etc.Further more, more 
mempolicy can be extended, such as allocating memory according to node 
weight, etc.

Thanks.