Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2;
Date:   Tue, 24 Oct 2023 18:13:49 +0200
From:   Michal =?utf-8?Q?Koutn=C3=BD?= <mkoutny@suse.com>
To:     Waiman Long <longman@redhat.com>
Cc:     Tejun Heo <tj@kernel.org>, Zefan Li <lizefan.x@bytedance.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Christian Brauner <brauner@kernel.org>,
        Jonathan Corbet <corbet@lwn.net>,
        Shuah Khan <shuah@kernel.org>, cgroups@vger.kernel.org,
        linux-kernel@vger.kernel.org, Juri Lelli <juri.lelli@redhat.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Giuseppe Scrivano <gscrivan@redhat.com>
Subject: Re: [PATCH v8 0/7] cgroup/cpuset: Support remote partitions
Message-ID: <agjgbmdi2yqegjk7p7m52yb3wxmr64ivohbra5wapcd3lwynpw@jjmx6dsboo53>
References: <20230905133243.91107-1-longman@redhat.com>
 <ahevhcy2aa7k3plmfvlepjehs6u3fun3j4oyskdz7axkhftlyi@zr3j473rciwi>
 <f75859e0-04d4-3da2-8df0-eb8841623a7c@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <f75859e0-04d4-3da2-8df0-eb8841623a7c@redhat.com>
Precedence: bulk

On Fri, Oct 13, 2023 at 12:03:18PM -0400, Waiman Long <longman@redhat.com> wrote:
> > [chain]
> >    root
> >    |                           \
> >    mid1a                        mid1b
> >     cpuset.cpus=0-1              cpuset.cpus=2-15
> >     cpuset.cpus.partition=root
> >    |
> >    mid2
> >     cpuset.cpus=0-1
> >     cpuset.cpus.partition=root
> >    |
> >    cont
> >     cpuset.cpus=0-1
> >     cpuset.cpus.partition=root
> In this case, the effective CPUs of both mid1a and mid2 will be empty. IOW,
> you can't have any task in these 2 cpusets.

I see, that is relevant to a threaded subtree only where the admin / app
can know how to distribute CPUs and place threads to internal nodes.

> For the remote case, you can have intermediate tasks in both mid1a and mid2
> as long as cpuset.cpus contains more CPUs than cpuset.cpus.exclusive.

It's obvious that cpuset.cpus.exclusive should be exclusive among
siblings.
Should it also be so along the vertical path?

  root
  |                           
  mid1a                       
   cpuset.cpus=0-2
   cpuset.cpus.exclusive=0    
  |
  mid2
   cpuset.cpus=0-2
   cpuset.cpus.exclusive=1
  |
  cont
   cpuset.cpus=0-2
   cpuset.cpus.exclusive=2
   cpuset.cpus.partition=root

IIUC, this should be a valid config regardless of cpuset.cpus.partition
setting on mid1a and mid2.
Whereas

  root
  |                           
  mid1a                       
   cpuset.cpus=0-2
   cpuset.cpus.exclusive=0    
  |
  mid2
   cpuset.cpus=0-2
   cpuset.cpus.exclusive=1-2
   cpuset.cpus.partition=root
  |
  cont
   cpuset.cpus=1-2
   cpuset.cpus.exclusive=1-2
   cpuset.cpus.partition=root

Here, I'm hesitating, will mid2 have any exclusively owned cpus?

(I have flashes of understading cpus.exclusive as being a more
expressive mechanism than partitions. OTOH, it seems non-intuitive when
both are combined, thus I'm asking to internalize it better.
Should partitions be deprecated for simplicty? They're still good to
provide the notification mechanism of invalidation.
cpuset.cpus.exclusive.effective don't have that.)

> They will be ready eventually. This requirement of remote partition actually
> came from our OpenShift team as the use of just local partition did not meet
> their need. They don't need access to exclusive CPUs in the parent cgroup
> layer for their management daemons. They do need to activate isolated
> partition in selected child cgroups to support our Telco customers to run
> workloads like DPDK.
> 
> So they will add the support to upstream Kubernetes.

Is it worth implementing anything touching (ancestral)
cpuset.cpus.partition then?

Thanks,
Michal