Received: by 2002:a25:d7c1:0:0:0:0:0 with SMTP id o184csp4677672ybg; Tue, 29 Oct 2019 10:43:30 -0700 (PDT) X-Google-Smtp-Source: APXvYqyKRems54f5Gc3HhTZJyL6bFilwJtogz0YC8J7fAftHlvMmmtyYxf35DyWIrCull4b2NB4m X-Received: by 2002:a17:906:448e:: with SMTP id y14mr4625191ejo.136.1572371009965; Tue, 29 Oct 2019 10:43:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572371009; cv=none; d=google.com; s=arc-20160816; b=s0hJpKce7pN7LUwje5avtQ2GCbwCNZ/EHGtqN5330JSKyeJcXFuUmP0ybZ4VAM/9HA eJLQx4uJYYmOPphG+R9mfwMH84jyf2OQ4Vu2jnce9zTLFNmKkrTITWDgEgPSptN0RnKW b/l4cZgfC5VXm4f+AxZP5lZuSoLpY09l+gN1CjT1tsaeKtYBHNwZLownaOX7wzw+6RR2 tjBIThnxHn4TsB+YmrtLGXlp+wVWAL1w+HQhS58FvYIY23wynflcBU1AaT8JVJ06cD9a O7c80WJUF124lHJxsQvQGqcUUO50SQeWolQmVSFrPJz+ehzrFUj6pdltvNNGLN+q8d7F uvxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:user-agent:organization :references:in-reply-to:date:cc:to:from:subject:message-id; bh=WFWBcA3qoHHw58tXxU1pCvJzUtFK9tjZDoOUDi9E8EU=; b=K/SRh0wIxvt2fsZL4Z3z5uLcaSO0Ri6TKR3adFs5Fn+jIdCqG6m6SDiGxI+6xDOI5/ qS2pXURdMC518UwbQeIJYTG3NWn/sONtsbXFz37weagq5j9KphQJXcHAtlKGPYir8GNB zsgsv9GvuMXUBErOESDY0InapJ3q2tn1ur0UmqW9mNmzdqedf3mVHOlsLv35QbG7Cld9 PdU/WCzkFRFNOlbQZpy1ECoTrzO6oF9OL2ziyLkjqi+t1ZDbzPuiYIqpJArhkeRMAcsR oXIxJdzWc0y0MSLy0kp8IQWILYVE0OTEjC6ZEAOm93Lfv5JF1fAkbfjBY2qkKyreahsL /OIg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n21si8466362ejx.148.2019.10.29.10.43.05; Tue, 29 Oct 2019 10:43:29 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731877AbfJ2JSJ (ORCPT + 99 others); Tue, 29 Oct 2019 05:18:09 -0400 Received: from mx2.suse.de ([195.135.220.15]:40548 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1730066AbfJ2JSJ (ORCPT ); Tue, 29 Oct 2019 05:18:09 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 40957B21E; Tue, 29 Oct 2019 09:18:06 +0000 (UTC) Message-ID: Subject: Re: [RFC PATCH v3 00/16] Core scheduling v3 From: Dario Faggioli To: Aaron Lu , Aubrey Li Cc: Tim Chen , Julien Desfossez , "Li, Aubrey" , Subhra Mazumdar , Vineeth Remanan Pillai , Nishanth Aravamudan , Peter Zijlstra , Ingo Molnar , Thomas Gleixner , Paul Turner , Linus Torvalds , Linux List Kernel Mailing , =?ISO-8859-1?Q?Fr=E9d=E9ric?= Weisbecker , Kees Cook , Greg Kerr , Phil Auld , Valentin Schneider , Mel Gorman , Pawan Gupta , Paolo Bonzini Date: Tue, 29 Oct 2019 10:18:04 +0100 In-Reply-To: <277737d6034b3da072d3b0b808d2fa6e110038b0.camel@suse.com> References: <20190726152101.GA27884@sinkpad> <7dc86e3c-aa3f-905f-3745-01181a3b0dac@linux.intel.com> <20190802153715.GA18075@sinkpad> <69cd9bca-da28-1d35-3913-1efefe0c1c22@linux.intel.com> <20190911140204.GA52872@aaronlu> <7b001860-05b4-4308-df0e-8b60037b8000@linux.intel.com> <20190912120400.GA16200@aaronlu> <20190915141402.GA1349@aaronlu> <277737d6034b3da072d3b0b808d2fa6e110038b0.camel@suse.com> Organization: SUSE Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-RwOX30FLRoOswSa3sgcf" User-Agent: Evolution 3.34.1 MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --=-RwOX30FLRoOswSa3sgcf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, 2019-10-29 at 10:11 +0100, Dario Faggioli wrote: > On Sun, 2019-09-15 at 22:14 +0800, Aaron Lu wrote: > > I'm using the following branch as base which is v5.1.5 based: > > https://github.com/digitalocean/linux-coresched coresched-v3- > > v5.1.5- > > test > >=20 > > And I have pushed Tim's branch to: > > https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-tim > >=20 > > Mine: > > https://github.com/aaronlu/linux coresched-v3-v5.1.5-test- > > core_vruntime > >=20 > Hello, >=20 > As anticipated, I've been trying to follow the development of this > feature and, in the meantime, I have done some benchmarks. >=20 > I actually have a lot of data (and am planning for more), so I am > sending a few emails, each one with a subset of the numbers in it, > instead than just one which would be beyond giant! :-) >=20 SYSBENCHTHREAD =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D http://xenbits.xen.org/people/dariof/benchmarks/results/linux/core-sched/mm= tests/boxes/wayrath/coresched-email-5_sysbenchthread.txt v v = BM BM BM BM= BM BM BM-HT BM-noHT = HT noHT csc-HT csc_stallfix-HT= csc_tim-HT csc_vruntime-HT Amean 1 1.12 ( 0.00%) 1.11 ( 0.38%) 1.14 * -= 2.43%* 1.15 * -2.81%* 8.49 *-659.72%* 8.43 *-654.99%*= 8.42 *-653.96%* 8.38 *-649.74%* Amean 3 1.11 ( 0.00%) 1.11 ( 0.00%) 1.14 * -= 2.70%* 1.14 * -2.96%* 8.35 *-651.29%* 8.42 *-657.20%*= 8.34 *-650.77%* 8.41 *-656.56%* Amean 5 1.11 ( 0.00%) 1.11 ( 0.00%) 1.14 * -= 2.70%* 1.14 * -2.82%* 8.34 *-649.81%* 8.41 *-655.46%*= 8.32 *-647.75%* 8.34 *-649.42%* Amean 7 1.12 ( 0.00%) 1.11 ( 0.38%) 1.14 * -= 2.43%* 1.15 * -2.69%* 8.31 *-643.48%* 8.40 *-652.05%*= 8.41 *-653.20%* 8.33 *-645.65%* Amean 12 1.12 ( 0.00%) 1.12 ( 0.00%) 1.14 * -= 2.43%* 1.14 * -2.56%* 8.38 *-651.34%* 8.41 *-654.16%*= 8.32 *-645.71%* 8.38 *-651.09%* Amean 16 1.11 ( 0.00%) 1.11 ( -0.13%) 1.15 * -= 3.08%* 1.14 * -2.57%* 8.42 *-656.61%* 8.36 *-651.60%*= 8.31 *-646.73%* 8.39 *-654.30%* Stddev 1 0.01 ( 0.00%) 0.01 ( 49.47%) 0.01 ( 6= 4.27%) 0.01 ( 39.86%) 0.08 (-414.47%) 0.16 (-976.34%)= 0.01 ( 36.42%) 0.09 (-525.69%) Stddev 3 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( = 0.00%) 0.01 (-108.17%) 0.13 (-3211.60%) 0.10 (-2531.86= %) 0.06 (-1515.55%) 0.10 (-2467.10%) Stddev 5 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( = 0.00%) 0.01 ( -9.54%) 0.12 (-2271.92%) 0.10 (-1917.42= %) 0.09 (-1700.00%) 0.09 (-1725.38%) Stddev 7 0.01 ( 0.00%) 0.01 ( 20.53%) 0.01 ( 4= 3.80%) 0.01 ( 0.00%) 0.13 (-1255.69%) 0.10 (-988.21%= ) 0.03 (-247.93%) 0.08 (-762.68%) Stddev 12 0.01 ( 0.00%) 0.01 ( -24.03%) 0.00 ( 3= 7.98%) 0.01 ( 0.00%) 0.13 (-1577.68%) 0.10 (-1210.31= %) 0.10 (-1140.97%) 0.06 (-644.73%) Stddev 16 0.00 ( 0.00%) 0.01 ( -9.54%) 0.01 ( -5= 4.92%) 0.00 ( 22.54%) 0.15 (-3032.73%) 0.12 (-2401.20= %) 0.11 (-2206.51%) 0.07 (-1304.28%) v v = VM VM VM VM= VM VM VM-HT VM-noHT = HT noHT csc-HT csc_stallfix-HT= csc_tim-HT csc_vruntime-HT Amean 1 1.43 ( 0.00%) 1.15 * 19.56%* 1.15 * 1= 9.36%* 1.19 * 16.67%* 1.16 * 19.16%* 1.17 * 18.46%*= 1.16 * 18.66%* 1.16 * 18.86%* Amean 3 1.43 ( 0.00%) 1.16 * 19.16%* 1.16 * 1= 9.26%* 1.17 * 18.46%* 1.16 * 19.16%* 1.21 * 15.17%*= 1.16 * 18.86%* 1.21 * 15.27%* Amean 5 1.43 ( 0.00%) 1.15 * 19.28%* 1.18 * 1= 7.78%* 1.17 * 18.08%* 1.16 * 19.18%* 1.17 * 18.18%*= 1.16 * 19.08%* 1.18 * 17.58%* Amean 7 1.43 ( 0.00%) 1.16 * 19.26%* 1.16 * 1= 8.86%* 1.17 * 18.56%* 1.16 * 19.16%* 1.19 * 16.57%*= 1.15 * 19.36%* 1.15 * 19.36%* Amean 12 1.44 ( 0.00%) 1.16 * 19.44%* 1.16 * 1= 9.44%* 1.17 * 18.75%* 1.16 * 19.25%* 1.18 * 18.06%*= 1.16 * 19.15%* 1.16 * 19.44%* Amean 16 1.46 ( 0.00%) 1.16 * 20.37%* 1.16 * 2= 0.37%* 1.16 * 20.47%* 1.16 * 20.37%* 1.17 * 19.59%*= 1.17 * 20.08%* 1.16 * 20.37%* Stddev 1 0.00 ( 0.00%) 0.00 ( 0.00%) 0.01 ( -4= 1.42%) 0.05 (-1314.21%) 0.00 ( -29.10%) 0.00 ( -29.10%= ) 0.01 (-236.65%) 0.02 (-316.33%) Stddev 3 0.00 ( 0.00%) 0.00 ( -29.10%) 0.01 (-10= 8.17%) 0.03 (-773.69%) 0.01 (-100.00%) 0.09 (-2324.18%= ) 0.01 (-138.05%) 0.06 (-1378.74%) Stddev 5 0.00 ( 0.00%) 0.01 ( -99.00%) 0.05 ( -9= 9.00%) 0.02 ( -99.00%) 0.01 ( -99.00%) 0.01 ( -99.00%)= 0.01 ( -99.00%) 0.04 ( -99.00%) Stddev 7 0.00 ( 0.00%) 0.01 ( -41.42%) 0.01 (-13= 8.05%) 0.02 (-468.62%) 0.01 (-100.00%) 0.06 (-1600.00%= ) 0.01 (-108.17%) 0.01 ( -41.42%) Stddev 12 0.00 ( 0.00%) 0.00 ( 100.00%) 0.00 ( 10= 0.00%) 0.03 (-11031521092846084.00%) 0.00 (-2034518927425100.= 00%) 0.01 (-4169523056347680.00%) 0.01 (-2228703820443891.00%= ) 0.00 ( 100.00%) Stddev 16 0.05 ( 0.00%) 0.00 ( 92.31%) 0.00 ( 9= 2.31%) 0.00 ( 100.00%) 0.00 ( 92.31%) 0.01 ( 80.64%)= 0.01 ( 89.12%) 0.00 ( 92.31%) v v = VM VM VM VM = VM VM VM-v4-HT VM-v4-noHT = v4-HT v4-noHT v4-csc-HT v4-csc_stallfix-HT = v4-csc_tim-HT v4-csc_vruntime-HT Amean 1 1.43 ( 0.00%) 1.17 * 18.15%* 1.15 * 19= .64%* 1.15 * 19.44%* 1.16 * 19.14%* 1.17 * 18.34%* = 1.16 * 18.94%* 1.16 * 18.94%* Amean 3 1.43 ( 0.00%) 1.17 * 18.33%* 1.17 * 18= .73%* 1.15 * 19.52%* 1.15 * 19.52%* 1.19 * 16.93%* = 1.16 * 19.12%* 1.16 * 19.32%* Amean 5 1.43 ( 0.00%) 1.18 * 17.45%* 1.15 * 19= .44%* 1.15 * 19.54%* 1.17 * 18.15%* 1.18 * 17.55%* = 1.15 * 19.44%* 1.16 * 19.04%* Amean 7 1.43 ( 0.00%) 1.18 * 17.53%* 1.16 * 19= .32%* 1.15 * 19.72%* 1.17 * 18.33%* 1.17 * 18.13%* = 1.16 * 18.92%* 1.15 * 19.62%* Amean 8 1.43 ( 0.00%) 1.17 * 18.46%* 1.16 * 19= .16%* 1.16 * 18.96%* 1.17 * 18.46%* 1.18 * 17.86%* = 1.15 * 19.36%* 1.16 * 19.06%* Stddev 1 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 22= .54%) 0.01 ( -9.54%) 0.01 (-119.09%) 0.01 ( -67.33%) = 0.01 (-149.00%) 0.01 ( -84.39%) Stddev 3 0.01 ( 0.00%) 0.01 ( 12.29%) 0.02 (-105= .69%) 0.01 ( 0.00%) 0.01 ( 0.00%) 0.05 (-515.82%) = 0.01 ( -27.10%) 0.01 ( -41.42%) Stddev 5 0.00 ( 0.00%) 0.04 (-700.00%) 0.01 ( -61= .25%) 0.01 ( -54.92%) 0.04 (-717.31%) 0.01 ( -84.39%) = 0.01 ( -61.25%) 0.01 ( -18.32%) Stddev 7 0.01 ( 0.00%) 0.04 (-390.68%) 0.01 ( 3= .92%) 0.00 ( 51.96%) 0.04 (-366.58%) 0.01 ( 0.00%) = 0.02 (-151.15%) 0.01 ( 3.92%) Stddev 8 0.00 ( 0.00%) 0.00 ( -29.10%) 0.01 (-151= .66%) 0.02 (-383.05%) 0.05 (-1100.00%) 0.01 (-108.17%)= 0.01 ( -41.42%) 0.01 (-138.05%) v v = VMx2 VMx2 VMx2 VMx2= VMx2 VMx2 VMx2-HT VMx2-noHT = HT noHT csc-HT csc_stallfix-HT= csc_tim-HT csc_vruntime-HT Amean 1 1.16 ( 0.00%) 1.22 * -5.80%* 1.54 * -3= 2.84%* 1.22 * -5.31%* 2.83 (-144.32%) 1.35 * -16.42%*= 1.69 * -46.30%* 1.35 * -16.91%* Amean 3 1.16 ( 0.00%) 1.24 * -6.90%* 1.56 * -3= 4.11%* 1.24 * -6.53%* 1.89 * -63.05%* 1.95 * -68.35%*= 1.67 * -43.84%* 1.33 * -14.41%* Amean 5 1.17 ( 0.00%) 1.21 * -3.79%* 1.52 * -3= 0.48%* 1.23 * -5.75%* 1.43 * -22.89%* 1.85 ( -58.51%)= 1.71 * -46.88%* 1.35 * -15.91%* Amean 7 1.15 ( 0.00%) 1.23 * -6.81%* 1.54 * -3= 3.29%* 1.24 * -7.05%* 1.68 * -45.79%* 1.39 * -20.79%*= 2.09 * -80.69%* 1.30 * -12.62%* Amean 12 1.16 ( 0.00%) 1.25 * -7.25%* 1.69 * -4= 5.09%* 1.22 * -5.28%* 1.72 * -47.54%* 1.52 * -31.08%*= 2.08 * -78.87%* 1.32 * -13.14%* Amean 16 1.16 ( 0.00%) 1.23 * -6.14%* 1.68 * -4= 4.23%* 1.22 * -4.67%* 1.80 * -54.91%* 1.37 * -17.44%*= 2.18 * -87.10%* 1.28 * -9.95%* Stddev 1 0.01 ( 0.00%) 0.05 (-569.58%) 0.14 (-16= 93.97%) 0.02 (-169.26%) 3.19 (-42121.82%) 0.25 (-3271.= 57%) 0.26 (-3362.06%) 0.09 (-1072.60%) Stddev 3 0.01 ( 0.00%) 0.06 (-664.85%) 0.06 (-63= 0.95%) 0.03 (-259.56%) 0.84 (-10177.51%) 0.98 (-11852.= 56%) 0.45 (-5404.28%) 0.08 (-845.29%) Stddev 5 0.00 ( 0.00%) 0.02 (-393.96%) 0.17 (-33= 71.31%) 0.07 (-1423.81%) 0.32 (-6361.11%) 1.04 (-21129= .08%) 0.22 (-4383.53%) 0.06 (-1033.14%) Stddev 7 0.01 ( 0.00%) 0.07 (-1234.79%) 0.20 (-3= 727.10%) 0.08 (-1483.25%) 0.57 (-10594.70%) 0.19 (-345= 5.98%) 0.30 (-5454.88%) 0.08 (-1400.56%) Stddev 12 0.00 ( 0.00%) 0.08 (-1641.84%) 0.16 (-3= 078.68%) 0.03 (-524.50%) 0.50 (-10161.87%) 0.08 (-1634= .36%) 0.32 (-6423.80%) 0.07 (-1232.67%) Stddev 16 0.00 ( 0.00%) 0.05 (-996.36%) 0.20 (-39= 58.82%) 0.06 (-1045.43%) 0.60 (-12116.06%) 0.23 (-4642= .99%) 0.23 (-4594.04%) 0.05 (-842.34%) This is all quite bizarre. Judging from these numbers, the benchmark seems to be no sensitive to HT at all (while, e.g., sysbenchcpu was), when on baremetal. In VMs, at least in most cases, things are significantly *faster* with HT off. Which I know is something that can happen, I just was not expecting it from this workload. Also, core scheduling is a total disaster on baremetal. And its behavior in VM is an anti-pattern too. So, I guess I'll go trying to see if I did something wrong when configuring or running this particular benchmark. If that is not the case, then core scheduling has a serious issue when dealing with threads instead than with processes, I would say. --=20 Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <> (Raistlin Majere) --=-RwOX30FLRoOswSa3sgcf Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEES5ssOj3Vhr0WPnOLFkJ4iaW4c+4FAl24A8wACgkQFkJ4iaW4 c+6OEhAAjOuXmCdWnpoDKrqGSoAB/9GRpROpzbj+ec/Sb6zeb0PSr3JWoo5Lrx6H O79chza8kEQl6PaxTOqBDBDUwhuqMQy45PEzztdvkWIIvX/3N6rZW+tOgMWsUCuU 6Oh5Xb9W2z1FrUkL/unRNGGyq5dZDfhAyvvx9xA5DZKthI1vd4N/GSMvwXU4i+mw 9LI8i5hJ7IMImM5dcK8gQ82G8+PmbBcmA2MvfJcahP4B6ZQDfND1lLIHJXI+w5M2 CofCPpG+0uZt4gdOVTQOhagP6JJk7Z5HFdckQ0KNL7ux4PwHVPRyRe1wnhj0Nc0B Tal6En43nacDH97WcfoBn1Fy7q2Pj4ibXi2A8jWmRMTxayN4ImKLZnBvwmo+r6OM C39VOi/nvJcUWLU8xnodky5qIreYq+5QwNGv37oyLmBTM4TgwbOQSfir8IHpI5Ue /khnhtHM9RmIXGAcUgBGMGkHK7d+9JfC2DZV4DnvdSRNcTruCqcLkiGrlu6yzOaw uUdMS39ab3DTGmfDDr7ZgtnXlQ07ABXJbtto22w6gX+Y2rIDhVi/5eMnO2+haiAn 3IwVSCyPmlsvkR91yDjK3NMupidF60mgFR+trddxvktBdRS1u5TaOaflEeGkdIfi tPuzU1Dl7JxNfzXYjICtZnPmpba02mCTbOjozqZGiGrsfGeZbfE= =ZX3Z -----END PGP SIGNATURE----- --=-RwOX30FLRoOswSa3sgcf--