Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
Date:   Sat, 23 Sep 2023 14:35:17 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     David Vernet <void@manifault.com>
CC:     <linux-kernel@vger.kernel.org>, <peterz@infradead.org>,
        <mingo@redhat.com>, <juri.lelli@redhat.com>,
        <vincent.guittot@linaro.org>, <dietmar.eggemann@arm.com>,
        <rostedt@goodmis.org>, <bsegall@google.com>, <mgorman@suse.de>,
        <bristot@redhat.com>, <vschneid@redhat.com>, <tj@kernel.org>,
        <roman.gushchin@linux.dev>, <gautham.shenoy@amd.com>,
        <kprateek.nayak@amd.com>, <aaron.lu@intel.com>,
        <wuyun.abel@bytedance.com>, <kernel-team@meta.com>,
        <tim.c.chen@intel.com>
Subject: Re: [PATCH v3 7/7] sched: Shard per-LLC shared runqueues
Message-ID: <ZQ6HJW8TnmOWAKC6@chenyu5-mobl2>
References: <20230809221218.163894-1-void@manifault.com>
 <20230809221218.163894-8-void@manifault.com>
 <ZO7e5YaS71cXVxQN@chenyu5-mobl2>
 <20230831000147.GA506447@maniforge>
 <ZPBvN9W6RYOr51zQ@chenyu5-mobl2>
 <20230831191444.GD531917@maniforge>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230831191444.GD531917@maniforge>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?bw7Y4Gd5NuVfp24KIRbRC+eIAv2ibLQ61803ui/9fMha/4hEYGA1hkav48Kf?=
 =?us-ascii?Q?e+rVpnqtiF9hQv+Fsbf3lwFaQLMnnAAGHrQBcrCKsxWnu8m8+r7XwJtkCNuJ?=
 =?us-ascii?Q?o4SLm5hI6wtEomEvaPbpBLYtyhlGmv1Cg0HfZohjaod/PU4zGIn0huJj3Vdx?=
 =?us-ascii?Q?YalLrEU9OhIBtz+E+74kJvuJWqmc75+vTUhADb0WE6Hw9cHMUg/cGogMZcof?=
 =?us-ascii?Q?C1444ty7eXCMM0S9DcPy6LzIbomgsd42GfXkDzgmhEuIzzBo/oV6OpwEkTqd?=
 =?us-ascii?Q?+GxI381E7ie8FGacNa24MiIadSC0Errg42kKLJR1Gjw/GVgUqHSLau3LPXuZ?=
 =?us-ascii?Q?YZ6verJCDNolQha/tO4L+ErKVxjfk7f/qWzS9GuxTudqk7q1j5hDXku5TBQ6?=
 =?us-ascii?Q?DCAmf6gqVSwIGti/znp0E37GzE/SMQxg/nMFa6xUEz8vZhc+CWbJEwS5dRjX?=
 =?us-ascii?Q?Qf4inB8Ww3YE6W9foP1H9BTPgoFEyJdSDRiHNSjjdI/ksa69Rb6nWy5ScTSz?=
 =?us-ascii?Q?wHtTPV5uCA3P/yekMuUSgQENx7pxYW5tlm9cEWxqjD/pZd2tOCgXnjNTfM2d?=
 =?us-ascii?Q?BfkY9eUU7ukBe51qaWwHFGaoemEST3aqEwNFlVXYOjdwfsJxl/sXQICZ4jHI?=
 =?us-ascii?Q?oobqf8q2zdw1m3Tnx9H0zuxlW6DjqMSph90YJcEqIeXfM3QqOTY+HQXacDhY?=
 =?us-ascii?Q?GphYAM+Qw1GNxknZhrtgTIpNZzhUJxToxFBKcKUlMN3xfziSud3LC78247LA?=
 =?us-ascii?Q?SF5G3KwVyAqG6yeD1nK6LADxZiYkMX1yheMPqriyYVcDdsGrMZMVUSHUJSUJ?=
 =?us-ascii?Q?eMydpUjI8c8XXUvXdEorphPuyJEpPPNNzfbtrIVX4AMpCTDYQcWTdIbYagrE?=
 =?us-ascii?Q?a940RgqbhkIFc4HHHgGEr5cSk9zBjyIMeE5Oq2u9LRFXJF2vdiubP5MO19Mm?=
 =?us-ascii?Q?+aILfOgM+4fvNQy98mhakRaoGOuw1dqk+Fmp/0vTt/QBezKeUn2JcDp/bzxx?=
 =?us-ascii?Q?eedQFq9QCkUKtixLAMFAiZuXsgc+yhqGMRcwWGxfubDr6gXT/8QZKyEXZiXo?=
 =?us-ascii?Q?chnDI6lsiGSRGrKx/hKFJ5p09U3vtITeun1DIsGQZvVrrkNzArFqxZjQrxC1?=
 =?us-ascii?Q?TEUygmlzZSjmy9l0+cPXbs/d6mPlxtPKB3qQvJGUfrbaLW6EKTLSmv77zDBh?=
 =?us-ascii?Q?MJZRNo+h86DRybr+cyRF4dX0b3xugK2Ir4qofExpUypvpj2xi5neYV2ij8nC?=
 =?us-ascii?Q?Rr/9dAUIulq3p1HWThZGNdQB534PDi9nKPfNiRTkFdaR+m+JKPGQgShLIo8N?=
 =?us-ascii?Q?2cOkVLozl34QDLKNhYgF0qUxow7b7kvQPNvPwO5Tc5tgOaqhB0lMUJ0sdu1/?=
 =?us-ascii?Q?tHHEaF+MfOXqN9UNlPsFXWdirIDJwlr65AXtmES9sIWfmignTef/GgPE8OGe?=
 =?us-ascii?Q?beDmdjroRorbArOwJC4Rw9HkIVwr59O/R9H6luYkv1fWl8DoyejtmnSMOB35?=
 =?us-ascii?Q?gpmxpaBvqGeBfnl1KizLDTUtroYGlbYNWLuSJfgk5pB+ajGGwCP8qZgJiXC5?=
 =?us-ascii?Q?okzJg21uURQn1oEtKOAUhDBgmNjFbEeoY3VJMIER?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 19bc0c7c-a4af-46c2-ea18-08dbbbff4c1d
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2023 06:35:38.1784
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: w/VX16sox6Gqjsh8YVYY/fL127rF6zfg9fc2SGsrBJ639O620mpfyKPm48hLCTyhipwPfZHksjwmX7A6yX5GlQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5141
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 22 Sep 2023 23:36:15 -0700 (PDT)
X-Spam-Level: **

On 2023-08-31 at 14:14:44 -0500, David Vernet wrote:
> On Thu, Aug 31, 2023 at 06:45:11PM +0800, Chen Yu wrote:
> > On 2023-08-30 at 19:01:47 -0500, David Vernet wrote:
> > > On Wed, Aug 30, 2023 at 02:17:09PM +0800, Chen Yu wrote:
[snip...]
> > 
> > Let me run other benchmarks to see if others are sensitive to
> > the resource locality.
> 
> Great, thank you Chenyu.
> 
> FYI, I'll be on vacation for over a week starting later today, so my
> responses may be delayed.
> 
> Thanks in advance for working on this. Looking forward to seeing the
> results when I'm back at work.

Sorry for late result. I applied your latest patch set on top of upstream
6.6-rc2 Commit 27bbf45eae9c(I pulled the latest commit from upstream yesterday).
The good news is that, there is overall slight stable improvement on tbench,
and no obvious regression on other benchmarks is observed on Sapphire Rapids
with 224 CPUs:

tbench throughput
======
case            	load    	baseline(std%)	compare%( std%)
loopback        	56-threads	 1.00 (  0.85)	 +4.35 (  0.23)
loopback        	112-threads	 1.00 (  0.38)	 +0.91 (  0.05)
loopback        	168-threads	 1.00 (  0.03)	 +2.96 (  0.06)
loopback        	224-threads	 1.00 (  0.09)	 +2.95 (  0.05)
loopback        	280-threads	 1.00 (  0.12)	 +2.48 (  0.25)
loopback        	336-threads	 1.00 (  0.23)	 +2.54 (  0.14)
loopback        	392-threads	 1.00 (  0.53)	 +2.91 (  0.04)
loopback        	448-threads	 1.00 (  0.10)	 +2.76 (  0.07)

schbench  99.0th tail latency
========
case            	load    	baseline(std%)	compare%( std%)
normal          	1-mthreads	 1.00 (  0.32)	 +0.68 (  0.32)
normal          	2-mthreads	 1.00 (  1.83)	 +4.48 (  3.31)
normal          	4-mthreads	 1.00 (  0.83)	 -0.59 (  1.80)
normal          	8-mthreads	 1.00 (  4.47)	 -1.07 (  3.49)

netperf  throughput
=======
case            	load    	baseline(std%)	compare%( std%)
TCP_RR          	56-threads	 1.00 (  1.01)	 +1.37 (  1.41)
TCP_RR          	112-threads	 1.00 (  2.44)	 -0.94 (  2.63)
TCP_RR          	168-threads	 1.00 (  2.94)	 +3.22 (  4.63)
TCP_RR          	224-threads	 1.00 (  2.38)	 +2.83 (  3.62)
TCP_RR          	280-threads	 1.00 ( 66.07)	 -7.26 ( 78.95)
TCP_RR          	336-threads	 1.00 ( 21.92)	 -0.50 ( 21.48)
TCP_RR          	392-threads	 1.00 ( 34.31)	 -0.00 ( 33.08)
TCP_RR          	448-threads	 1.00 ( 43.33)	 -0.31 ( 43.82)
UDP_RR          	56-threads	 1.00 (  8.78)	 +3.84 (  9.38)
UDP_RR          	112-threads	 1.00 ( 14.15)	 +1.84 (  8.35)
UDP_RR          	168-threads	 1.00 (  5.10)	 +2.95 (  8.85)
UDP_RR          	224-threads	 1.00 ( 15.13)	 +2.76 ( 14.11)
UDP_RR          	280-threads	 1.00 ( 15.14)	 +2.14 ( 16.75)
UDP_RR          	336-threads	 1.00 ( 25.85)	 +1.64 ( 27.42)
UDP_RR          	392-threads	 1.00 ( 34.34)	 +0.40 ( 34.20)
UDP_RR          	448-threads	 1.00 ( 41.87)	 +1.41 ( 41.22)

We can have a re-run after the latest one is released.

thanks,
Chenyu