Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <563fd5e1-650a-e329-8aab-2fa1953a9f49@redhat.com>
Date:   Tue, 6 Jun 2023 16:11:02 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.7.1
Subject: Re: [RFC PATCH 0/5] cgroup/cpuset: A new "isolcpus" paritition
Content-Language: en-US
To:     Tejun Heo <tj@kernel.org>
Cc:     =?UTF-8?Q?Michal_Koutn=c3=bd?= <mkoutny@suse.com>,
        Zefan Li <lizefan.x@bytedance.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Jonathan Corbet <corbet@lwn.net>,
        Shuah Khan <shuah@kernel.org>, linux-kernel@vger.kernel.org,
        cgroups@vger.kernel.org, linux-doc@vger.kernel.org,
        linux-kselftest@vger.kernel.org,
        Juri Lelli <juri.lelli@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        Frederic Weisbecker <frederic@kernel.org>,
        Mrunal Patel <mpatel@redhat.com>,
        Ryan Phillips <rphillips@redhat.com>,
        Brent Rowsell <browsell@redhat.com>,
        Peter Hunt <pehunt@redhat.com>, Phil Auld <pauld@redhat.com>
References: <deb7b684-3d7c-b3ae-7b36-5b7ba2dd8001@redhat.com>
 <ZFUo5IYAIwTEKR4_@slm.duckdns.org>
 <759603dd-7538-54ad-e63d-bb827b618ae3@redhat.com>
 <405b2805-538c-790b-5bf8-e90d3660f116@redhat.com>
 <ZGvHUjOCjwat91Gq@slm.duckdns.org>
 <18793f4a-fd39-2e71-0b77-856afb01547b@redhat.com>
 <ZH4jfmypOXGJPu0D@slm.duckdns.org>
 <be64a569-4388-9dd9-3e06-36d716a54f6c@redhat.com>
 <ZH5FNc6wjlGPsaaO@slm.duckdns.org>
 <a2220c9f-7a8d-da82-ecc0-b39f3807408c@redhat.com>
 <ZH-P7X_yjnVfhy7b@slm.duckdns.org>
From:   Waiman Long <longman@redhat.com>
In-Reply-To: <ZH-P7X_yjnVfhy7b@slm.duckdns.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 6/6/23 15:58, Tejun Heo wrote:
> Hello, Waiman.
>
> On Mon, Jun 05, 2023 at 10:47:08PM -0400, Waiman Long wrote:
> ...
>> I had a different idea on the semantics of the cpuset.cpus.exclusive at the
>> beginning. My original thinking is that it was the actual exclusive CPUs
>> that are allocated to the cgroup. Now if we treat this as a hint of what
>> exclusive CPUs should be used and it becomes valid only if the cgroup can
> I wouldn't call it a hint. It's still hard allocation of the CPUs to the
> cgroups that own them. Setting up a partition requires exclusive CPUs and
> thus would depend on exclusive allocations set up accordingly.
>
>> become a valid partition. I can see it as a value that can be hierarchically
>> set throughout the whole cpuset hierarchy.
>>
>> So a transition to a valid partition is possible iff
>>
>> 1) cpuset.cpus.exclusive is a subset of cpuset.cpus and is a subset of
>> cpuset.cpus.exclusive of all its ancestors.
> Yes.
>
>> 2) If its parent is not a partition root, none of the CPUs in
>> cpuset.cpus.exclusive are currently allocated to other partitions. This the
> Not just that, the CPUs aren't available to cgroups which don't have them
> set in the .exclusive file. IOW, if a CPU is in cpus.exclusive of some
> cgroups, it shouldn't appear in cpus.effective of cgroups which don't have
> the CPU in their cpus.exclusive.
>
> So, .exclusive explicitly establishes exclusive ownership of CPUs and
> partitions depend on that with an implicit "turn CPUs exclusive" behavior in
> case the parent is a partition root for backward compatibility.
The current CPU exclusive behavior is limited to sibling cgroups only. 
Because of the hierarchical nature of cpu distribution, the set of 
exclusive CPUs have to appear in all its ancestors. When partition is 
enabled, we do a sibling exclusivity test at that point to verify that 
it is exclusive. It looks like you want to do an exclusivity test even 
when the partition isn't active. I can certainly do that when the file 
is being updated. However, it will fail the write if the exclusivity 
test fails just like the v1 cpuset.cpus.exclusive flag if you are OK 
with that.
>
>> same remote partition concept in my v2 patch. If its parent is a partition
>> root, part of its exclusive CPUs will be distributed to this child partition
>> like the current behavior of cpuset partition.
> Yes, similar in a sense. Please do away with the "once .reserve is used, the
> behavior is switched" part.

That behavior has been gone in my v2 patch.

> Instead, it can be sth like "if the parent is a
> partition root, cpuset implicitly tries to set all CPUs in its cpus file in
> its cpus.exclusive file" so that user-visible behavior stays unchanged
> depending on past history.

If parent is a partition root, auto reservation will be done and 
cpus.exclusive will be set automatically just like before. So existing 
applications using partition will not be affected.

Cheers,
Longman