Received-SPF: pass (google.com: domain of linux-kernel+bounces-148172-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <7592f555-21f8-284a-dbc7-0a6ab4d42c0d@amd.com>
Date: Wed, 17 Apr 2024 14:18:46 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.13.1
Subject: Re: [PATCH v4 0/8] sched: Implement shared runqueue in fair.c
Content-Language: en-US
To: David Vernet <void@manifault.com>
Cc: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com,
 bsegall@google.com, mgorman@suse.de, bristot@redhat.com,
 vschneid@redhat.com, youssefesmat@google.com, joelaf@google.com,
 roman.gushchin@linux.dev, yu.c.chen@intel.com, gautham.shenoy@amd.com,
 aboorvad@linux.vnet.ibm.com, wuyun.abel@bytedance.com, tj@kernel.org,
 kernel-team@meta.com, linux-kernel@vger.kernel.org
References: <20231212003141.216236-1-void@manifault.com>
From: K Prateek Nayak <kprateek.nayak@amd.com>
In-Reply-To: <20231212003141.216236-1-void@manifault.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?TGhUK0NybDA2L0RQT0IwQkVHS2JIOVN3TXlYdHkrVEN4Y3BvMG1tUndWKzRz?=
 =?utf-8?B?NHEvYXM3dWFUUlRhdU5wcUVBbitrbDFFTkF3Qjd0MUhrMzNFYjNWdlgyNFhZ?=
 =?utf-8?B?bFlIVVpQQStjNXNuV2hpdWd0WU56cCtsb2Q0K1JYbURKcDNYSEtpamlzd2Zi?=
 =?utf-8?B?MkJWTjJqNFRwOWY3M2ZoNjcyQmFiUGY0dXhQSkZLZi9BWFFZamFydktGUVVW?=
 =?utf-8?B?d2ZsWjFXUHk0UXhuYVV3WWJ6VHFIMU9DVzZmUUs5TTIwOUhFODBKOGczRk95?=
 =?utf-8?B?Vk1hZiticy9ObUphWlpMWDZhbzFPL0VuMno3OW82Nis1RWxOeFY5aG9sODFH?=
 =?utf-8?B?SnhmY3h6WDhMdEkyS2kyL0VxbFBPTXVyaGhUM3hMSWZybjNPR1krWnRUM2k1?=
 =?utf-8?B?SGc3NmlkSzBuT2kxbkpad3pCR09CSU5rWm96Y3JrVUswMzdnUi9HUHhCdHBS?=
 =?utf-8?B?NG8rK01mWlVmREZSMGJxWUJNcHVVK0FpNDMxUXdrSk9NTTJSKzBYcmlpcWVP?=
 =?utf-8?B?TStYcUhLYlRMOTdkOVRNVFJQV3dMUE5oSzBLU2poOVlhYUNtakkvWnJHUWE3?=
 =?utf-8?B?TEkvSnBGWHF1Y0JXRFRTcnJNK3RJUEtWeG1HT1hJSW5YalorWCtzOVFJbitZ?=
 =?utf-8?B?TS9MbDNxUFVjSW56S0xvVGkxUGVJUVlyREd5VTFJNHFoN0pwajVLTkxObzBq?=
 =?utf-8?B?YTQ4NUoyK3d6RlJ2TExGK3pwdUppOHArT1diK1Bjc09BaXdyT1pPaGllaVlF?=
 =?utf-8?B?YTY2citHYm16RmlSTlRXc05uYUNTV2poOW02dEdHeXdVc1k2VmJqLy9jVEsy?=
 =?utf-8?B?TVZIQ2cvNnFhT0RqTDlZV2Q4Rmlma215c3NFNkdXeWFQWUFvN2t6bnNUWVF5?=
 =?utf-8?B?ODByaUVXR2ZybUJCR3g1L2p3eTliV0FMNER4MGJLQ1ZPcDRaTEpqSjhxTlZq?=
 =?utf-8?B?VWFHbExkaW9vaVo1TlpUZ0plT2NNMHFMZUg1c3ZXVDVvS0k5ZnFHSDIrSC9T?=
 =?utf-8?B?WlkxNUsva1JTVS9XWWlBOG02MXc5TjRQaW0wWjh1VjEzNTg1ekd3eEo1ZURG?=
 =?utf-8?B?TTl1OFJjQ0ZRSkUyWnd0RGtWblo5OVViN1lIM2ZjaEorVVcwZUJ0d3FtYnQx?=
 =?utf-8?B?YzZGTU5FUzBzR0x5NGdIb3lsRU8vSGUwOTNEM25KdFR3WFhHMDVPZUJLVHhW?=
 =?utf-8?B?Q21IUm5Ld1NUZjQ2MUU2MzgzQmJuSmMzZjA1a2VwT2FxTFRLQ2o5WGNpY1Fq?=
 =?utf-8?B?eHJzYlZGUGtpNk1ia2hNci9pVThFK1l5Yk5jWHh5SE1DT0hieTE0eW9MTWpv?=
 =?utf-8?B?dVNOdVRydTZFUXlDcUNUZ3JTWkY4VXdBcUZ0OXhKVGtYdGlGQjdZRmpNbEg3?=
 =?utf-8?B?bmkzalBWc0w0M2VzSTNqcWlUbU85M3lXRTBWRXpSczRNeXpnOUZBaGxyR3BR?=
 =?utf-8?B?REZpL2tFSFk3Y2MvSUN5ekl0dm5EOWpVbmR1dWlqQ2pza0dkdWNWRjhGOG4w?=
 =?utf-8?B?SUF2dzJPL1ZnOG1mcGZ3eDlObXo1NENsUzFOeGtlSzZZSytBdnBZcWNyOU5a?=
 =?utf-8?B?clNhQ0p5NStBVnRCL1J0TlVlMS9yOEJxSGt2QUVWV3oyelVXQXU5a21nY3Vo?=
 =?utf-8?B?RGN4ZVdmYXRNOGtrVC92VUY5bzJ4UWVtM2lrNy9oeTZNVnRTU3pDNFU1Y0k3?=
 =?utf-8?B?aFJnd2J3OGtGRjRMV3RqMndJMmp3OXU5N1U2Mll3N0VLT1BxT1J2bTlBV1VY?=
 =?utf-8?B?NCtmRmRtdFd4VTVyMmNpd2FBRnRjVGFMY2RVL1p6Q3BUcVpJMTNZMFA4K3l3?=
 =?utf-8?B?aFJTM003YXYvNklORCtLVEQxYmVnU3dWRmJWU0hCanptYlhzTGdlbjI5cGMr?=
 =?utf-8?B?Z0hkdUZrdlQ4Y24wclpuVE9xTE1GOHFpTG1DTjVwWXdpOU9PRXV2VXBjVDk3?=
 =?utf-8?B?a3gxbEcvRnVzL0tLZHNmRmkrMjcwRUtoaStvR0JmWXFvMGdqVEJKRW9tb054?=
 =?utf-8?B?SDhLWjc5RTk5TDNlSzU0NzZGYWVGaFNrNkFDOVRWVkRkemNlNmtha3hMR2lw?=
 =?utf-8?B?ODFIL2JHTEhEblpaeVdCZ0RSOGh0a093V2t4Q3V5RVZVdVJJNUx0WFpiamd2?=
 =?utf-8?Q?x0zDRlyvw6dmAGkAGyRHBFBdP?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 65e7997d-e31c-4405-993f-08dc5ebb37e8
X-MS-Exchange-CrossTenant-AuthSource: MW2PR12MB2379.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Apr 2024 08:48:57.2818
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: VwHqIelHM+6vdjvDHe6bnDs/TtaSHMu6wgB9F22K7BLMgf7gX0zIQlxXIfr71pikFfg3ytNtyQinc1PZF/PKQg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7152

Hello David,

On 12/12/2023 6:01 AM, David Vernet wrote:
> This is v4 of the shared runqueue patchset. This patch set is based off
> of commit 418146e39891 ("freezer,sched: Clean saved_state when restoring
> it during thaw") on the sched/core branch of tip.git.
> 
> In prior versions of this patch set, I was observing consistent and
> statistically significant wins for several benchmarks when this feature
> was enabled, such as kernel compile and hackbench. After rebasing onto
> the latest sched/core on tip.git, I'm no longer observing these wins,
> and in fact observe some performance loss with SHARED_RUNQ on hackbench.
> I ended up bisecting this to when EEVDF was merged.
> 
> As I mentioned in [0], our plan for now is to take a step back and
> re-evaluate how we want to proceed with this patch set. That said, I did
> want to send this out in the interim in case it could be of interest to
> anyone else who would like to continue to experiment with it.

I was doing a bunch of testing prior to OSPM in case folks wanted to
discuss the results. Leaving the results of SHARED_RUNQ runs from a
recent-ish tip below.

tl;dr

- I haven't dug deeper into the regressions but the most prominent one
  seems to be hackbench with lower number of groups but the picture
  flips with higher number of groups.

Other benchmarks behave more or less similar to the tip. I'll leave the
full results below: 

o System Details

- 3rd Generation EPYC System
- 2 x 64C/128T
- NPS1 mode

o Kernels

tip:			tip:sched/core at commit 8cec3dd9e593
			("sched/core: Simplify code by removing
			 duplicate #ifdefs")

shared_runq:		tip + this series

o Results

==================================================================
Test          : hackbench
Units         : Normalized time in seconds
Interpretation: Lower is better
Statistic     : AMean
==================================================================
Case:           tip[pct imp](CV)    shared_runq[pct imp](CV)
 1-groups     1.00 [ -0.00]( 1.80)     4.49 [-349.19](92.14)
 2-groups     1.00 [ -0.00]( 1.76)     1.02 [ -2.17](19.20)
 4-groups     1.00 [ -0.00]( 1.82)     0.86 [ 13.53]( 1.37)
 8-groups     1.00 [ -0.00]( 1.40)     0.91 [  8.73]( 2.39)
16-groups     1.00 [ -0.00]( 3.38)     0.91 [  9.47]( 2.39)


==================================================================
Test          : tbench
Units         : Normalized throughput
Interpretation: Higher is better
Statistic     : AMean
==================================================================
Clients:           tip[pct imp](CV)    shared_runq[pct imp](CV)
    1     1.00 [  0.00]( 0.44)     1.00 [ -0.39]( 0.53)
    2     1.00 [  0.00]( 0.39)     1.00 [ -0.16]( 0.57)
    4     1.00 [  0.00]( 0.40)     1.00 [ -0.07]( 0.69)
    8     1.00 [  0.00]( 0.16)     0.99 [ -0.67]( 0.45)
   16     1.00 [  0.00]( 3.00)     1.03 [  2.86]( 1.23)
   32     1.00 [  0.00]( 0.84)     1.00 [ -0.32]( 1.46)
   64     1.00 [  0.00]( 1.66)     0.98 [ -1.60]( 0.79)
  128     1.00 [  0.00]( 1.04)     1.01 [  0.57]( 0.59)
  256     1.00 [  0.00]( 0.26)     0.98 [ -1.91]( 2.48)
  512     1.00 [  0.00]( 0.15)     1.00 [  0.22]( 0.16)
 1024     1.00 [  0.00]( 0.20)     1.00 [ -0.37]( 0.02)


==================================================================
Test          : stream-10
Units         : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic     : HMean
==================================================================
Test:           tip[pct imp](CV)    shared_runq[pct imp](CV)
 Copy     1.00 [  0.00]( 6.19)     1.10 [  9.51]( 4.30)
Scale     1.00 [  0.00]( 6.47)     1.03 [  2.90]( 2.82)
  Add     1.00 [  0.00]( 6.50)     1.04 [  3.82]( 3.10)
Triad     1.00 [  0.00]( 5.70)     1.01 [  1.49]( 4.30)


==================================================================
Test          : stream-100
Units         : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic     : HMean
==================================================================
Test:           tip[pct imp](CV)    shared_runq[pct imp](CV)
 Copy     1.00 [  0.00]( 3.22)     1.04 [  3.67]( 2.41)
Scale     1.00 [  0.00]( 6.17)     1.03 [  2.75]( 1.63)
  Add     1.00 [  0.00]( 5.12)     1.02 [  2.42]( 2.10)
Triad     1.00 [  0.00]( 2.29)     1.01 [  1.11]( 1.59)


==================================================================
Test          : netperf
Units         : Normalized Througput
Interpretation: Higher is better
Statistic     : AMean
==================================================================
Clients:           tip[pct imp](CV)    shared_runq[pct imp](CV)
 1-clients     1.00 [  0.00]( 0.17)     0.99 [ -0.65]( 0.40)
 2-clients     1.00 [  0.00]( 0.49)     1.00 [ -0.17]( 0.27)
 4-clients     1.00 [  0.00]( 0.65)     1.00 [  0.09]( 0.69)
 8-clients     1.00 [  0.00]( 0.56)     1.00 [ -0.05]( 0.61)
16-clients     1.00 [  0.00]( 0.78)     1.00 [ -0.23]( 0.58)
32-clients     1.00 [  0.00]( 0.62)     0.98 [ -2.22]( 0.76)
64-clients     1.00 [  0.00]( 1.41)     0.96 [ -3.75]( 1.19)
128-clients    1.00 [  0.00]( 0.83)     0.98 [ -2.29]( 0.97)
256-clients    1.00 [  0.00]( 4.60)     0.96 [ -4.18]( 3.02)
512-clients    1.00 [  0.00](54.18)     0.99 [ -1.36](52.79)


==================================================================
Test          : schbench
Units         : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic     : Median
==================================================================
#workers:           tip[pct imp](CV)    shared_runq[pct imp](CV)
  1     1.00 [ -0.00](34.63)     1.40 [-40.00]( 2.38)
  2     1.00 [ -0.00]( 2.70)     1.08 [ -8.11]( 7.53)
  4     1.00 [ -0.00]( 4.70)     0.93 [  6.67]( 7.16)
  8     1.00 [ -0.00]( 5.09)     0.92 [  7.55](10.20)
 16     1.00 [ -0.00]( 5.08)     0.97 [  3.39]( 2.00)
 32     1.00 [ -0.00]( 2.91)     1.03 [ -3.33]( 2.22)
 64     1.00 [ -0.00]( 2.73)     0.99 [  1.04]( 3.43)
128     1.00 [ -0.00]( 7.89)     0.99 [  0.69]( 9.65)
256     1.00 [ -0.00](28.55)     0.92 [  7.94](19.85)
512     1.00 [ -0.00]( 2.11)     1.13 [-12.69]( 6.41)


==================================================================
Test          : DeathStarBench
Units         : Normalized throughput
Interpretation: Higher is better
Statistic     : Mean
==================================================================
Pinning      scaling     tip            shared_runq (pct imp)
 1CCD           1       1.00            1.01 (%diff: 1.45%)
 2CCD           2       1.00            1.01 (%diff: 1.71%)
 4CCD           4       1.00            1.01 (%diff: 1.66%)
 8CCD           8       1.00            1.00 (%diff: 0.63%)

--

> 
> [0]: https://lore.kernel.org/all/20231204193001.GA53255@maniforge/
> 
> v1 (RFC): https://lore.kernel.org/lkml/20230613052004.2836135-1-void@manifault.com/
> v2: https://lore.kernel.org/lkml/20230710200342.358255-1-void@manifault.com/
> v3: https://lore.kernel.org/all/20230809221218.163894-1-void@manifault.com/
> 
> [..snip..]
> 

I'll take a deeper look at the regressions soon. I'll update the thread
if I find anything interesting in the meantime.

--
Thanks and Regards,
Prateek