Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <1bd3bb9c-581e-2e18-aa41-ecf000e3686e@meta.com>
Date:   Fri, 21 Apr 2023 14:14:10 -0400
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.10.0
Subject: Re: schbench v1.0
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     David Vernet <void@manifault.com>, linux-kernel@vger.kernel.org,
        kernel-team@fb.com, Ingo Molnar <mingo@kernel.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        gautham.shenoy@amd.com
References: <bc85a40c-1ea0-9b57-6ba3-b920c436a02c@meta.com>
 <20230420150537.GC4253@hirez.programming.kicks-ass.net>
Content-Language: en-US
From:   Chris Mason <clm@meta.com>
In-Reply-To: <20230420150537.GC4253@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZnRIak9EUzhOTUF1NG5BVGVDM2xyNFZURUh6cWpFS2hrMmFNVnBIR0hHdFY2?=
 =?utf-8?B?YjJyQ2VSUUZhZ0l6U1pCR0U5OTRhdkU4aW1sZ01zVVFSeTVyWEp3eVlKZjRP?=
 =?utf-8?B?VHlneW1BVUJVeHZmQ2V0eDRrUFRZVWYxKzJFL253SmM3SzlWRW5WR2hLelZs?=
 =?utf-8?B?ODhVc0NNekRDa09CdzVIeUNhYlNRSmJ2YkdqcC80cTdxYmhyTS94Tno4d05n?=
 =?utf-8?B?L1VaRzRydWpFSlQza2pldTc5b2lYQXFFSjZ3dnlXek1wWkJMdXZXTDZQTDFl?=
 =?utf-8?B?c0pkSTcvZCtGQnVicmRpK0VVaTF1S3VLcGVTaWE5UFBlZU5jajNIVldMelNI?=
 =?utf-8?B?L0ltQ3ZaSFFCcDM0YUNBVS9IMDVYNUdLZ1ZURGc5d29mNHpPeHRkazBwaUVV?=
 =?utf-8?B?VFZIRHRWc0tQWC9XY2x5dUVGb0VUdnRrcW93YnJFeFFhaW80YjA5Tkcwb1VD?=
 =?utf-8?B?a1hBc1Yzdzh1QzVoMnQ5dDFUN0pUeWE2bWo4bmJxa0xTcGZIN1MxSVQwTHRl?=
 =?utf-8?B?eTFvelhoRmpCT2JCemY3SzU5MENOaE5hNjYzL3RQOVJzOCswSkpDZ2VBd2Vp?=
 =?utf-8?B?MVRyVGxKajJSR1grb3kwMk1xeXpFeFJYNC9mMU13SWFBc2pwS0lPM2dxaVdP?=
 =?utf-8?B?ZCtYY3RwMG0vYlhES3R6Z085bDdKcE9iOWtxWUc4N1JJaXowanR1VG5nV3N5?=
 =?utf-8?B?K1JWU1dlZm5KOS93b2ovWU9SMUt3NlVMZHJzRVZZclFTRHZRMm0rQ2dQU0Ft?=
 =?utf-8?B?eFZIcEJvdDl3cmhlL2hWT3BSU2xqOW84WlYvMDNTNjFGRkZ2Zm5yY205TmdP?=
 =?utf-8?B?Rnd1V1BZbmQ0K1RqWFhwZDh2ai9QWmw4a05RdUZ6VFVQMHhLT25aRlZUR00w?=
 =?utf-8?B?Z204NjhnY2g4Z214SWJYQVZJdndKbGtNeGlUZWlqS1FZaDVlYWQrT0l4UVZ4?=
 =?utf-8?B?NWc4TkV2c1h2bDVwa0kzNHVEcm15NmxHVFVOZVEzSHUzSld0elZ1bi9ENitu?=
 =?utf-8?B?RG02SDVXOU5hREpLa05TVW1tNzJqdjhqUVhMUVJFK0VxUzFRdFFDbHc5a3Ez?=
 =?utf-8?B?K1BiNW95Z1VaeFV1NHN2VjRva2Q2Y2hxQUZqb0NxU1FuVmJieFRGMlZreVVX?=
 =?utf-8?B?WUFIRXB0bjZUa3ZDUDFTMnBKOFFlekN5VTRkanNnb3hhY3RuYU02YUJ6S2E5?=
 =?utf-8?B?QUplNjkwSWh2MEttVFN2MEhsNTJIV3dkWVI2bEc2WlVJSERWaGZTeDlFbUtK?=
 =?utf-8?B?eVN6ODBocXEvMWJwSlR0NkgwcVpKSFM4aVpPUWsyK3RvOG1OSEZ0eGpLZWFC?=
 =?utf-8?B?RzlEdkVpZkJMZk40ZXVzVFRIbHFsZzJ0SXg2c2VYZW9QekhVQUZWc1daZTFN?=
 =?utf-8?B?bEpNelI0ZlpCVmlNWW4xeStJU2tJaVFiQWpSVVFkVzBvdjk1d29zQU1uaFFY?=
 =?utf-8?B?Q3ZhZTRSdFdlZHlvM0NDTy9ISmdGTSsvdC9JWEk3S1FoNDkvbUoxUkdnMkhK?=
 =?utf-8?B?cGYyUUxydWlwUkJmN2tMOVZ1YnphbC9NSnlvc2pjZWc0YjVIbGdYQ1ZGSnBN?=
 =?utf-8?B?dnhBSGo2amtUazFLVUQxWHVPaEI5SDl4VjFPVERzMFlQeW5HSXJPeHpFaUFq?=
 =?utf-8?B?RGxCeldnYlNtQ01PRHY2em1ZZjRwWFZmZzhtaVN4UEdkUUhZZVVEYmZrQmk5?=
 =?utf-8?B?a0VGaFNITnBKMTRkSG5iV0MvVjl2dVVkTTRIaW5mNVJFcXFUdzdhMUg3NDBp?=
 =?utf-8?B?bHROWURaWXV1SWNMWGNYUzFkWlkreVVRcHFHc1VhTDNmZ05URE11WW5WRURD?=
 =?utf-8?B?dTdWS0JyczVTMCs3OUp3Uy91VDJTZVZIN2Z4enlqZk05VGhmTi84VFBTc25V?=
 =?utf-8?B?NEpuemZzWlJzQndZV1FhWjlGNHJ1amFqSUxHbFc1QmpkckJWNnlZS1JEUk55?=
 =?utf-8?B?TC9IZk9OMW5TU1BCVENuOWRmZ1Z0NHFJN3NvSXpsUWZkVlc1bnc2Q2dDVTho?=
 =?utf-8?B?Y0VwTFovd1djc3c1dUthZ0RaQ2V1YVE4UEdsZ3czNWtwOEVnTlMycXFObmdv?=
 =?utf-8?B?bUJuUVltZ1FvYTlhMlhaYkNqKzYvRFVCMjJjRGZmK2puWEpLVGxlOFI5OVJk?=
 =?utf-8?B?WUtoS0ZVRnRSWmMzOUhTbEdDbTBBMDlwN2NJK0w4dmx1VVBRWWpOUlhkcXVP?=
 =?utf-8?B?UUE9PQ==?=
X-OriginatorOrg: meta.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d338820c-397c-4075-6998-08db429436ce
X-MS-Exchange-CrossTenant-AuthSource: MN2PR15MB4287.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Apr 2023 18:14:14.7518
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2/2p/WCMEtc37AeCzZtHEZJAi1NuNHSfHfWVlEHYqrVVGAwwIl71zkY+bzvdmzbO
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR15MB5597
X-Proofpoint-GUID: q7ExnEbQr10nb0EmFcpc4vPX-GjJU11r
X-Proofpoint-ORIG-GUID: q7ExnEbQr10nb0EmFcpc4vPX-GjJU11r
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22
 definitions=2023-04-21_08,2023-04-21_01,2023-02-09_01
X-Spam-Status: No, score=-6.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_PASS,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 4/20/23 11:05 AM, Peter Zijlstra wrote:
> On Mon, Apr 17, 2023 at 10:10:25AM +0200, Chris Mason wrote:
> 
>> F128 N10                EEVDF    Linus
>> Wakeup  (usec): 99.0th: 755      1,266
>> Request (usec): 99.0th: 25,632   22,304
>> RPS    (count): 50.0th: 4,280    4,376
>>
>> F128 N10 no-locking     EEVDF    Linus
>> Wakeup  (usec): 99.0th: 823      1,118
>> Request (usec): 99.0th: 17,184   14,192
>> RPS    (count): 50.0th: 4,440    4,456
> 
> With the below fixlet (against queue/sched/eevdf) on my measly IVB-EP
> (2*10*2):
> 
> ./schbench -F128 -n10 -C
> 
> Request Latencies percentiles (usec) runtime 30 (s) (153800 total samples)
> 	  90.0th: 6376       (35699 samples)
> 	* 99.0th: 6440       (9055 samples)
> 	  99.9th: 7048       (1345 samples)
> 
> CFS
> 
> schbench -m2 -F128 -n10	-r90	OTHER	BATCH
> Wakeup  (usec): 99.0th:		6600	6328
> Request (usec): 99.0th:		35904	14640
> RPS    (count): 50.0th:		5368	6104
> 

Peter and I went back and forth a bit and now schbench git has a few fixes:

- README.md updated

- warmup time defaults to zero (disabling warmup).  This was causing the
stats inconsistency Peter noticed below.

- RPS calculated more often.  Every second instead of every reporting
interval.

- thread count scaled to CPU count when -m is used.  The thread count is
per messenge thread, so when you use -m2 like Peter did in these runs,
he was ending up with 2xNUM_CPUs workers.  That's why his wakeup
latencies are so high, he had double the work that I did.

I'll experiment with some of the suggestions he made too.

-chris