Received: by 2002:a25:d7c1:0:0:0:0:0 with SMTP id o184csp4186876ybg; Tue, 29 Oct 2019 03:31:06 -0700 (PDT) X-Google-Smtp-Source: APXvYqwIz8wRoUOaRZRmhd3SvlsdDwiH54pmUSPcC0Z1Z5pmFtCgqXGBp5UZGX1TfQnm8U8CxgON X-Received: by 2002:a17:907:42cc:: with SMTP id nz20mr2356911ejb.324.1572345066314; Tue, 29 Oct 2019 03:31:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572345066; cv=none; d=google.com; s=arc-20160816; b=aoGF6CWLUdt/xftAQwebYG1BEH4Xg/COupQ7hU6JXrELzBLG1GCimXcMju+/H7ou7X iFjcFxoMaQcPfD17fvGzMGPPIHEi7WUlw21hWrKpAxQ05gpTBV5sDcR7EBwJFHGKE1mX MPi+3Rk3+3hq1BfI1L0EFVFTkQ2a0D2JqY1nJsRDJXLuFNzBMv/ClNE3dGE/YxletoIX RZ5Hv0e+nK6E4YKTjhJqW4btwSsFs3EI0+X1vTZ9B38knI4lELWVA3Vp+7OTN4i8SZQG +kJrOYFnOLlleRRFN8QN4/90d6UQLxJolJL7D/MQEGVOWPvI7iHakpBkCXDA+YSD/2pt xdog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:user-agent:organization :references:in-reply-to:date:cc:to:from:subject:message-id; bh=im01JBZR8LdWD2rJ3MfcEEhAUo7rpewVpPgWb4WhVDQ=; b=DZopa0p2ASAkZ3bOfzbFYijuG+W+chbLRMgcmvmMp2jdYCQDuW39YWPiKLP9QnzG0d J/sspXHR60lDr9jHOmTx79QY2Oby2s6ks6l09IcxcshJBcuczEWy3g4v2u/R3g5L6uq6 r+oi7obnAxX31XauU6P7p9YQljxEVyEaf0U2FASpH1eQ2fKSy0SunWk4FnC5DXpTouXJ InhYa2XzOxabApSsQzTtJcDirZ2sM9edIgAeoilAe/u/89fUDAvPSW16xD8m9LDeq10u bpd2nSnpzLIcqKLvxZpAV52PPiKqn2g8ebJw7kdf1L8udagSKWXz8zCP05L3eeopBPIz kvDg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w17si10389280edc.152.2019.10.29.03.30.41; Tue, 29 Oct 2019 03:31:06 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731994AbfJ2JSy (ORCPT + 99 others); Tue, 29 Oct 2019 05:18:54 -0400 Received: from mx2.suse.de ([195.135.220.15]:40764 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726803AbfJ2JSx (ORCPT ); Tue, 29 Oct 2019 05:18:53 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 5003BB283; Tue, 29 Oct 2019 09:18:50 +0000 (UTC) Message-ID: <615958c46c850d284de76ffcf9f0dc35ad262066.camel@suse.com> Subject: Re: [RFC PATCH v3 00/16] Core scheduling v3 From: Dario Faggioli To: Aaron Lu , Aubrey Li Cc: Tim Chen , Julien Desfossez , "Li, Aubrey" , Subhra Mazumdar , Vineeth Remanan Pillai , Nishanth Aravamudan , Peter Zijlstra , Ingo Molnar , Thomas Gleixner , Paul Turner , Linus Torvalds , Linux List Kernel Mailing , =?ISO-8859-1?Q?Fr=E9d=E9ric?= Weisbecker , Kees Cook , Greg Kerr , Phil Auld , Valentin Schneider , Mel Gorman , Pawan Gupta , Paolo Bonzini Date: Tue, 29 Oct 2019 10:18:48 +0100 In-Reply-To: <277737d6034b3da072d3b0b808d2fa6e110038b0.camel@suse.com> References: <20190726152101.GA27884@sinkpad> <7dc86e3c-aa3f-905f-3745-01181a3b0dac@linux.intel.com> <20190802153715.GA18075@sinkpad> <69cd9bca-da28-1d35-3913-1efefe0c1c22@linux.intel.com> <20190911140204.GA52872@aaronlu> <7b001860-05b4-4308-df0e-8b60037b8000@linux.intel.com> <20190912120400.GA16200@aaronlu> <20190915141402.GA1349@aaronlu> <277737d6034b3da072d3b0b808d2fa6e110038b0.camel@suse.com> Organization: SUSE Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-vUJAtHRWv6ovXvFt0x+s" User-Agent: Evolution 3.34.1 MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --=-vUJAtHRWv6ovXvFt0x+s Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, 2019-10-29 at 10:11 +0100, Dario Faggioli wrote: > On Sun, 2019-09-15 at 22:14 +0800, Aaron Lu wrote: > > I'm using the following branch as base which is v5.1.5 based: > > https://github.com/digitalocean/linux-coresched coresched-v3- > > v5.1.5- > > test > >=20 > > And I have pushed Tim's branch to: > > https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-tim > >=20 > > Mine: > > https://github.com/aaronlu/linux coresched-v3-v5.1.5-test- > > core_vruntime > >=20 > Hello, >=20 > As anticipated, I've been trying to follow the development of this > feature and, in the meantime, I have done some benchmarks. >=20 > I actually have a lot of data (and am planning for more), so I am > sending a few emails, each one with a subset of the numbers in it, > instead than just one which would be beyond giant! :-) >=20 SYSBENCH =3D=3D=3D=3D=3D=3D=3D=3D http://xenbits.xen.org/people/dariof/benchmarks/results/linux/core-sched/mm= tests/boxes/wayrath/coresched-email-6_sysbench.txt v v = BM BM BM BM = BM BM BM-HT BM-noHT = HT noHT csc-HT csc_stallfix-HT = csc_tim-HT csc_vruntime-HT Hmean 1 235.81 ( 0.00%) 221.49 ( -6.07%) 245.28 ( 4= .01%) 230.53 ( -2.24%) 241.40 ( 2.37%) 225.00 ( -4.58%) = 225.50 ( -4.37%) 202.38 ( -14.18%) Hmean 4 273.77 ( 0.00%) 290.01 ( 5.93%) 292.47 ( 6= .83%) 261.76 ( -4.39%) 287.58 ( 5.04%) 281.30 ( 2.75%) = 274.21 ( 0.16%) 271.91 ( -0.68%) Hmean 7 346.60 ( 0.00%) 315.58 ( -8.95%) 345.38 ( -0= .35%) 349.29 ( 0.78%) 363.76 ( 4.95%) 349.09 ( 0.72%) = 355.69 ( 2.62%) 336.69 ( -2.86%) Hmean 8 343.17 ( 0.00%) 353.73 ( 3.08%) 409.04 ( 19= .19%) 411.31 ( 19.86%) 406.77 ( 18.53%) 306.33 ( -10.74%) = 393.70 ( 14.72%) 342.73 ( -0.13%) Stddev 1 44.93 ( 0.00%) 50.07 ( -11.44%) 25.05 ( 44= .24%) 39.22 ( 12.71%) 26.27 ( 41.54%) 42.77 ( 4.81%) = 43.63 ( 2.90%) 62.88 ( -39.95%) Stddev 4 16.03 ( 0.00%) 23.37 ( -45.76%) 23.77 ( -48= .25%) 22.40 ( -39.69%) 18.63 ( -16.19%) 14.37 ( 10.35%) = 9.34 ( 41.72%) 25.21 ( -57.23%) Stddev 7 22.88 ( 0.00%) 37.54 ( -64.07%) 26.57 ( -16= .16%) 38.50 ( -68.26%) 59.14 (-158.51%) 26.73 ( -16.83%) = 24.58 ( -7.43%) 32.93 ( -43.94%) Stddev 8 36.74 ( 0.00%) 36.60 ( 0.39%) 102.82 (-179= .86%) 93.56 (-154.65%) 77.33 (-110.47%) 36.16 ( 1.58%) = 44.27 ( -20.50%) 35.15 ( 4.33%) v v = VM VM VM VM = VM VM VM-HT VM-noHT = HT noHT csc-HT csc_stallfix-HT = csc_tim-HT csc_vruntime-HT Hmean 1 215.16 ( 0.00%) 225.80 * 4.95%* 205.74 ( -4= .38%) 200.61 ( -6.76%) 169.70 ( -21.13%) 168.84 ( -21.53%) = 157.27 ( -26.91%) 162.94 ( -24.27%) Hmean 4 163.44 ( 0.00%) 189.82 * 16.14%* 164.54 ( 0= .67%) 148.47 ( -9.16%) 40.62 * -75.15%* 53.14 * -67.49%* = 129.51 * -20.76%* 158.99 ( -2.72%) Hmean 7 162.74 ( 0.00%) 185.79 ( 14.17%) 211.79 ( 30= .14%) 186.92 ( 14.86%) 28.02 * -82.78%* 34.32 * -78.91%* = 130.01 ( -20.11%) 145.47 ( -10.61%) Hmean 8 240.19 ( 0.00%) 192.24 ( -19.96%) 192.87 * -19= .70%* 194.01 ( -19.23%) 16.92 * -92.95%* 30.55 * -87.28%* = 150.51 * -37.34%* 147.67 * -38.52%* Stddev 1 1.80 ( 0.00%) 4.14 (-129.80%) 6.04 (-234= .90%) 24.54 (-1261.18%) 61.54 (-3313.19%) 62.92 (-3389.48= %) 55.91 (-3000.67%) 58.29 (-3132.75%) Stddev 4 6.33 ( 0.00%) 14.22 (-124.51%) 7.04 ( -11= .07%) 13.77 (-117.30%) 13.23 (-108.83%) 5.07 ( 19.92%) = 15.73 (-148.38%) 14.04 (-121.61%) Stddev 7 24.70 ( 0.00%) 37.50 ( -51.78%) 35.59 ( -44= .07%) 29.96 ( -21.26%) 20.07 ( 18.77%) 21.06 ( 14.74%) = 24.97 ( -1.07%) 30.52 ( -23.52%) Stddev 8 23.23 ( 0.00%) 41.13 ( -77.03%) 27.30 ( -17= .49%) 41.67 ( -79.35%) 3.75 ( 83.88%) 12.62 ( 45.68%) = 55.87 (-140.49%) 37.12 ( -59.76%) v v = VM VM VM VM = VM VM VM-v4-HT VM-v4-noHT = v4-HT v4-noHT v4-csc-HT v4-csc_stallfix-HT = v4-csc_tim-HT v4-csc_vruntime-HT Hmean 1 216.12 ( 0.00%) 310.43 ( 43.63%) 168.80 ( -21= .90%) 196.04 ( -9.29%) 168.27 ( -22.14%) 188.53 ( -12.77%) = 176.57 ( -18.30%) 180.12 ( -16.66%) Hmean 3 161.91 ( 0.00%) 160.80 ( -0.69%) 160.33 ( -0= .97%) 175.36 ( 8.31%) 51.27 * -68.34%* 52.18 * -67.77%* = 137.95 ( -14.80%) 166.12 ( 2.60%) Hmean 4 156.44 ( 0.00%) 196.25 * 25.45%* 165.19 ( 5= .60%) 199.78 * 27.71%* 50.67 * -67.61%* 40.42 * -74.16%* = 175.03 ( 11.88%) 172.66 * 10.37%* Stddev 1 4.67 ( 0.00%) 100.18 (-2043.42%) 50.61 (-98= 2.87%) 141.33 (-2923.68%) 69.08 (-1377.96%) 70.09 (-1399.5= 1%) 47.32 (-912.42%) 42.67 (-813.02%) Stddev 3 12.62 ( 0.00%) 8.76 ( 30.60%) 13.18 ( -4= .38%) 9.42 ( 25.41%) 3.57 ( 71.72%) 28.47 (-125.51%) = 21.08 ( -67.00%) 19.37 ( -53.48%) Stddev 4 8.52 ( 0.00%) 8.54 ( -0.17%) 3.09 ( 63= .70%) 13.84 ( -62.42%) 24.33 (-185.50%) 9.30 ( -9.10%) = 16.60 ( -94.73%) 9.64 ( -13.07%) v v = VMx2 VMx2 VMx2 VMx2 = VMx2 VMx2 VMx2-HT VMx2-noHT = HT noHT csc-HT csc_stallfix-HT = csc_tim-HT csc_vruntime-HT Hmean 1 168.87 ( 0.00%) 154.79 ( -8.34%) 190.28 ( 12= .68%) 151.18 ( -10.48%) 136.73 ( -19.03%) 36.63 * -78.31%* = 83.21 ( -50.73%) 124.42 ( -26.32%) Hmean 4 163.65 ( 0.00%) 87.90 * -46.29%* 119.37 * -27= .06%* 87.94 * -46.26%* 26.96 * -83.53%* 24.08 * -85.29%* = 54.15 * -66.91%* 63.80 * -61.01%* Hmean 7 181.60 ( 0.00%) 89.10 * -50.93%* 148.16 ( -18= .41%) 75.71 * -58.31%* 16.98 * -90.65%* 23.92 * -86.83%* = 57.28 * -68.46%* 66.10 * -63.60%* Hmean 8 198.98 ( 0.00%) 94.24 * -52.64%* 141.96 ( -28= .65%) 96.62 * -51.44%* 23.22 * -88.33%* 29.24 * -85.30%* = 80.10 * -59.74%* 80.36 * -59.61%* Stddev 1 61.59 ( 0.00%) 44.71 ( 27.41%) 52.14 ( 15= .33%) 44.61 ( 27.56%) 90.12 ( -46.32%) 42.53 ( 30.94%) = 38.32 ( 37.79%) 43.03 ( 30.12%) Stddev 4 8.65 ( 0.00%) 21.74 (-151.41%) 21.18 (-144= .98%) 22.51 (-160.27%) 19.72 (-128.07%) 4.38 ( 49.38%) = 2.68 ( 68.95%) 14.40 ( -66.55%) Stddev 7 17.94 ( 0.00%) 15.14 ( 15.62%) 29.94 ( -66= .88%) 17.30 ( 3.54%) 26.23 ( -46.20%) 5.17 ( 71.17%) = 15.98 ( 10.95%) 18.43 ( -2.72%) Stddev 8 38.45 ( 0.00%) 19.68 ( 48.82%) 44.14 ( -14= .78%) 28.84 ( 25.00%) 10.64 ( 72.33%) 11.65 ( 69.71%) = 22.63 ( 41.15%) 16.00 ( 58.39%) Core scheduling does not seem to be able to handle sysbench well, yet. In this case, things are not to bad on baremtal (and the best performing coresched variant is again the one with Tim's patches). But things go bad when running the banchmark in VMs, where core scheduling almost always loses against no HyperThreading, even in the overcommitted case. For virt. cases, it's also not straightforward to tell which set of patches are best, as some runs are in favours of Tim's, some others of vruntime's. --=20 Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <> (Raistlin Majere) --=-vUJAtHRWv6ovXvFt0x+s Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEES5ssOj3Vhr0WPnOLFkJ4iaW4c+4FAl24A/gACgkQFkJ4iaW4 c+7bshAAyPnbrjHmeXxO5catKxzdwPUwXYeG4Vsu/EbJG8Ze3X/0ES5tPYrGCyfK Zx9e5QY3cpCn/fCAebjwoMGtXK3l4FrSl3XlHokY+lGvTbZbKu/qHqkM4Ds0Ep99 729r+CPdAPQZve+2hcRSaHC7QinFu/9xLObaJiMpmWwDfx18nyGMwbxv0VaIknY4 gYK12V0+fpwkwE7as7BQbRX0fxzQPzC8OK32iNJfeQX6UZEODntr3SUxYiiqZ5vd IsGVCOAtJyVJtCxU0dhcju04rf6gEXV+PtWkfR1Kb3Aq8J0V+G0kY56bwWcqydPd 7QnUYBuxt4hCGYM0WbqBXIrhjq8hUrVLsctp3m5DX7M1e5aZubmBeBsFwf4Tjrga TmW7oNYG2NOm+MklBlnwKfLe7Y/q+0CCV74MH/jaVETNibD3lCMUEOxfw8ykqdem x6qNg/iDXpYu8mAdSixPcTWhm2sQXhiqosdzmJoCyE4B5wR6N29Yre3WtydWSE3V gfWVs6ILR1nGQZKhHDZRILbkwjwjIahHl6p0FXfKoLzr3SDJNV/KkUOoZkzXPUZi xIo9FOak7As7mDCYeUg2Aj3bne6prPEH9ZB8G3YftQp8MpucMXP9y0kkf2br2hLw 7MRWDuXPHWwuaf7uHB+hHvJSC3W/IBoILTu9WodrKVAPK0z1Uk4= =Z/4b -----END PGP SIGNATURE----- --=-vUJAtHRWv6ovXvFt0x+s--