Subject: Re: [PATCH 1/2] x86/CPU/AMD: Present package as die instead of socket
To: Borislav Petkov <bp@alien8.de>
References: <1498545653-6755-1-git-send-email-suravee.suthikulpanit@amd.com>
 <1498545653-6755-2-git-send-email-suravee.suthikulpanit@amd.com>
 <20170627104803.wlhsqhaylbeqod37@pd.tnic>
 <bb167d94-828b-2d5b-642a-d56e8db392d0@amd.com>
 <20170627134251.5ztfqqlm3rg6y7ql@pd.tnic>
Cc: x86@kernel.org, linux-kernel@vger.kernel.org, leo.duran@amd.com,
        yazen.ghannam@amd.com, Peter Zijlstra <peterz@infradead.org>
From: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Message-ID: <2f3bf9f2-cbca-ca72-219a-13c6faf9d314@amd.com>
Date: Tue, 27 Jun 2017 23:54:12 +0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0)
 Gecko/20100101 Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <20170627134251.5ztfqqlm3rg6y7ql@pd.tnic>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jun 2017 16:54:18.0771 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1734
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 3449
Lines: 90

Boris,

On 6/27/17 20:42, Borislav Petkov wrote:
> On Tue, Jun 27, 2017 at 08:07:10PM +0700, Suravee Suthikulpanit wrote:
>> What we are trying point out here is that (NUMA) "node" and "die" are the
>> same thing in most of AMD processors, not necessary trying to introduce
>> another term here.
>
> So don't use it then. The whole topology topic is confusing as it is to
> people so that every time I, for example, have to explain myself with an
> example when talking about it. Adding a "die" into the mix makes it more
> confusing, not less.
>
> So pick the terms, please, and document them properly so that we all are
> on the same page when talking about topology.
>
>
>> Yes. 4 packages (or 4 dies, or 4 NUMA nodes) in a socket.
>
> See above.
>
> I'd like to have the topology terminology all explained and written
> down, pls.
>

Sure, I will document the additional terms as you suggested once we agree on the 
direction.

>> However, SRAT/SLIT does not describe the DIE. So, using
>> x86_numa_in_packge_topology on multi-die Zen processor will result in
>> missing the DIE sched-domain for cpus within a die.
>
> What does "does not describe the DIE" mean exactly? How exactly you need
> to describe a die. And forget the die sched domain - first answer the
> question: how is the NUMA info in SRAT/SLIT insufficient for scheduling?
>
> Are you saying, you want to have all threads on a die belong to a
> separate scheduling entity?

Please see my comment below.....

>> Zen cpu0 (package-as-die)
>> domain0 00000000,00000001,00000000,00000001 (SMT)
>> domain1 00000000,0000000f,00000000,0000000f (MC ccx)
>> domain2 00000000,000000ff,00000000,000000ff (DIE)
>
> So this is 8 threads IINM.
>

Actually, the DIE sched-domain (domain2) has 16 threads (the cpumask is split 
between cpu 0-7 and 64-71 since the BIOS enumerate all T0 in the system first 
before T1).

> You want to have those 8 threads as a separate scheduling entity?
> But looking at this picture:
>
> Die (Dx) View :
>              ----------------------------
>          C0  | T0 T1 |    ||    | T0 T1 | C4
>              --------|    ||    |--------
>          C1  | T0 T1 | L3 || L3 | T0 T1 | C5
>              --------|    ||    |--------
>          C2  | T0 T1 | #0 || #1 | T0 T1 | C6
>              --------|    ||    |--------
>          C3  | T0 T1 |    ||    | T0 T1 | C7
>              ----------------------------
>
> That's 16 threads on a die.
>
> So are you trying to tell me that you want to have all threads sharing
> an L3 into a single scheduling domain? Is that it?
> Or do you want to have all threads on a die in a single scheduling
> domain?

The 8 threads sharing each L3 are already in the same sched-domain1 (MC CCX). 
So, cpu0 is in the same sched-domain1 as cpu1,2,3,64,65,66,67. Here, we need the 
DIE sched-domain because it represents all cpus that are in the same NUMA node 
(since we have one memory controller per DIE). IIUC, for Zen, w/o the DIE 
sched-domain, the scheduler could try to re-balance the tasks from one CCX 
(schedule group) to another CCX across NUMA node, and potentially causing 
unnecessary performance due to remote memory access.

Please note also that SRAT/SLIT information are used to derive the NUMA 
sched-domains, while the DIE sched-domain is non-NUMA sched-domain (derived from 
CPUID topology extension which is available on newer families).

Please let me know if I missing any other points.

Thanks,
Suravee