Received-SPF: pass (google.com: domain of linux-kernel+bounces-126597-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <78c60269-5aee-45d7-8014-2c0188f972da@bursov.com>
Date: Mon, 1 Apr 2024 13:35:03 +0300
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2 3/3] docs: cgroup-v1: clarify that domain levels are
 system-specific
To: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Ingo Molnar <mingo@redhat.com>, Peter Zijlstra <peterz@infradead.org>,
 Juri Lelli <juri.lelli@redhat.com>,
 Vincent Guittot <vincent.guittot@linaro.org>,
 Dietmar Eggemann <dietmar.eggemann@arm.com>,
 Steven Rostedt <rostedt@goodmis.org>, Ben Segall <bsegall@google.com>,
 Mel Gorman <mgorman@suse.de>, Daniel Bristot de Oliveira
 <bristot@redhat.com>, Valentin Schneider <vschneid@redhat.com>,
 linux-kernel@vger.kernel.org
References: <cover.1711900396.git.vitaly@bursov.com>
 <af9562aac2e9208029aef1dd19c3b0e096dd42c7.1711900396.git.vitaly@bursov.com>
 <3d85926d-378a-4d5e-8303-92461bd3b100@linux.ibm.com>
Content-Language: en-US, ru-RU, uk-UA
From: Vitalii Bursov <vitaly@bursov.com>
In-Reply-To: <3d85926d-378a-4d5e-8303-92461bd3b100@linux.ibm.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit


On 01.04.24 07:05, Shrikanth Hegde wrote:
> 
> 
> On 3/31/24 9:31 PM, Vitalii Bursov wrote:
>> Add a clarification that domain levels are system-specific
>> and where to check for system details.
>>
>> Add CPU clusters to the scheduler domain levels table.
>>
>> Signed-off-by: Vitalii Bursov <vitaly@bursov.com>
>> ---
>>  Documentation/admin-guide/cgroup-v1/cpusets.rst | 16 +++++++++++-----
>>  1 file changed, 11 insertions(+), 5 deletions(-)
>>
>> diff --git a/Documentation/admin-guide/cgroup-v1/cpusets.rst b/Documentation/admin-guide/cgroup-v1/cpusets.rst
>> index 7d3415eea..d16a3967d 100644
>> --- a/Documentation/admin-guide/cgroup-v1/cpusets.rst
>> +++ b/Documentation/admin-guide/cgroup-v1/cpusets.rst
>> @@ -568,19 +568,25 @@ on the next tick.  For some applications in special situation, waiting
>>  
>>  The 'cpuset.sched_relax_domain_level' file allows you to request changing
>>  this searching range as you like.  This file takes int value which
>> -indicates size of searching range in levels ideally as follows,
>> +indicates size of searching range in levels approximately as follows,
>>  otherwise initial value -1 that indicates the cpuset has no request.
>>  
>>  ====== ===========================================================
>>    -1   no request. use system default or follow request of others.
>>     0   no search.
>>     1   search siblings (hyperthreads in a core).
>> -   2   search cores in a package.
>> -   3   search cpus in a node [= system wide on non-NUMA system]
>> -   4   search nodes in a chunk of node [on NUMA system]
>> -   5   search system wide [on NUMA system]
>> +   2   search cpu clusters
>> +   3   search cores in a package.
>> +   4   search cpus in a node [= system wide on non-NUMA system]
>> +   5   search nodes in a chunk of node [on NUMA system]
>> +   6   search system wide [on NUMA system]
> 
> I think above block of documentation need not change. SD_CLUSTER is a software 
> construct, not a sched domain per se. 
> 

I added "cpu clusters" because the original table:
====== ===========================================================
  -1   no request. use system default or follow request of others.
   0   no search.
   1   search siblings (hyperthreads in a core).
   2   search cores in a package.
   3   search cpus in a node [= system wide on non-NUMA system]
   4   search nodes in a chunk of node [on NUMA system]
   5   search system wide [on NUMA system]
====== ===========================================================
does not match to what I see on a few systems I checked.

AMD Ryzen and the same dual-CPU Intel server with NUMA disabled:
  level:0 - SMT
  level:2 - MC
  level:3 - PKG

Server with NUMA enabled:
  level:0 - SMT
  level:2 - MC
  level:5 - NUMA

So, for the relax level original table:
  1 -> enables 0 SMP -> OK
  2 -> enables 1 unknown -> does not enable cores in a package
  3 -> enables 2 MC -> OK for NUMA, but not system wide on non-NUMA system
  5 -> enables 4 unknown -> does not enable system wide on NUMA

The updated table
====== ===========================================================
  -1   no request. use system default or follow request of others.
   0   no search.
   1   search siblings (hyperthreads in a core).
   2   search cpu clusters
   3   search cores in a package.
   4   search cpus in a node [= system wide on non-NUMA system]
   5   search nodes in a chunk of node [on NUMA system]
   6   search system wide [on NUMA system]
====== ===========================================================
would work like this:
  1 -> enables 0 SMP -> OK
  2 -> enables 1 unknown -> does nothing new
  3 -> enables 2 MC -> OK, cores in a package for NUMA and non-NUMA system
  4 -> enables 3 PKG -> OK on non-NUMA system
  6 -> enables 5 NUMA -> OK

I think it would look more correct on "average" systems, but anyway,
please confirm and I'll remove the table update in an updated patch.

Thanks

> IMO the next paragraph that is added is good enough and the above change can be removed.

>>  ====== ===========================================================
>>  
>> +Not all levels can be present and values can change depending on the
>> +system architecture and kernel configuration. Check
>> +/sys/kernel/debug/sched/domains/cpu*/domain*/ for system-specific
>> +details.
>> +
>>  The system default is architecture dependent.  The system default
>>  can be changed using the relax_domain_level= boot parameter.
>>