Received: by 2002:ac0:b08d:0:0:0:0:0 with SMTP id l13csp1802400imc; Fri, 22 Feb 2019 11:27:43 -0800 (PST) X-Google-Smtp-Source: AHgI3IYQspSrG4385YbAOhXBaLo0Cz3fEnAZjN8falxNFqMi7QYhqrqmeC9j3ScGht0AZnmwDVsi X-Received: by 2002:a63:4962:: with SMTP id y34mr5553703pgk.425.1550863663374; Fri, 22 Feb 2019 11:27:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550863663; cv=none; d=google.com; s=arc-20160816; b=yTsQQz4cquvxQ3yay3Cc/TS7bHRa3KNiYbuwKDHfi5hPslz7FtWMo+W/h37JGp95Yg 9Djxlln+KnQMlbvvUQucJIt2kqAgZfKr4pRypACjWddy3ZCJ3/75hkoWWvvc94ZS+O8M CwcRcH7Lpat4OekLNIUA5mmtjQlmcsDFjzoWSMWwpp/pPtwKU2VS52Jv6sc3hU7VxlTU X1i+blnj/649WsBT+1Pj5U6GIgpF0XTx/AZy8KQLx5krVo/6DOJywmP1/gcAWmIti4vS cxbpyNnlL/nDn05QuFD0BlIq12Xo13aXcAClmooUa+QzSCRPK4VcsaYtHh9M0uncSWF5 LOiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:subject:autocrypt:openpgp:from:references:cc:to; bh=IIEq17/dqAOfle2c1uwGAXcmIFsq5nN/8EVjOSKXSOU=; b=Skwiv+9w1I9fsCqfPIN39D81TwdWt0zb3e6ss/jmXXeGxE1PtA55YdimZUbVg4NNbs lW7YtQ0jsiSHzK6wk1Ep+BCVTcJEQm4eUON6AM1HguR2Svi8RWZokXTVkbv9I+mZzQ04 MJ+lrhZVkDfTb9hw9G0P2dEG8sCqtdfhB9p96wEeqwHSDemuGixUiAYmy1trVqyw92IR syRFazJC+5fddgn3nMcEZD5neWin0EAzIDtgDZauZ4eVGFKIJuW76vSkCQCcNgPnErgb nmIgFETFWOhawMf7frJO5Sq0qxxoVBFSl4T5V/6pmsPcM03Gh7MeDKDWifMG1Egmdtxd VjnQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o8si2098606pfk.94.2019.02.22.11.27.27; Fri, 22 Feb 2019 11:27:43 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726320AbfBVT06 (ORCPT + 99 others); Fri, 22 Feb 2019 14:26:58 -0500 Received: from mga09.intel.com ([134.134.136.24]:24907 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725878AbfBVT06 (ORCPT ); Fri, 22 Feb 2019 14:26:58 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Feb 2019 11:26:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,400,1544515200"; d="scan'208";a="128608438" Received: from schen9-desk.jf.intel.com (HELO [10.54.74.162]) ([10.54.74.162]) by orsmga003.jf.intel.com with ESMTP; 22 Feb 2019 11:26:57 -0800 To: Peter Zijlstra , Paolo Bonzini Cc: Linus Torvalds , Ingo Molnar , Thomas Gleixner , Paul Turner , Linux List Kernel Mailing , subhra.mazumdar@oracle.com, =?UTF-8?B?RnLDqWTDqXJpYyBXZWlzYmVja2Vy?= , Kees Cook , kerrnel@google.com References: <20190218165620.383905466@infradead.org> <20190218204020.GV32494@hirez.programming.kicks-ass.net> <407b6589-1801-20b5-e3b7-d7458370cfc0@redhat.com> <20190222142030.GA32494@hirez.programming.kicks-ass.net> From: Tim Chen Openpgp: preference=signencrypt Autocrypt: addr=tim.c.chen@linux.intel.com; prefer-encrypt=mutual; keydata= mQINBE6ONugBEAC1c8laQ2QrezbYFetwrzD0v8rOqanj5X1jkySQr3hm/rqVcDJudcfdSMv0 BNCCjt2dofFxVfRL0G8eQR4qoSgzDGDzoFva3NjTJ/34TlK9MMouLY7X5x3sXdZtrV4zhKGv 3Rt2osfARdH3QDoTUHujhQxlcPk7cwjTXe4o3aHIFbcIBUmxhqPaz3AMfdCqbhd7uWe9MAZX 7M9vk6PboyO4PgZRAs5lWRoD4ZfROtSViX49KEkO7BDClacVsODITpiaWtZVDxkYUX/D9OxG AkxmqrCxZxxZHDQos1SnS08aKD0QITm/LWQtwx1y0P4GGMXRlIAQE4rK69BDvzSaLB45ppOw AO7kw8aR3eu/sW8p016dx34bUFFTwbILJFvazpvRImdjmZGcTcvRd8QgmhNV5INyGwtfA8sn L4V13aZNZA9eWd+iuB8qZfoFiyAeHNWzLX/Moi8hB7LxFuEGnvbxYByRS83jsxjH2Bd49bTi XOsAY/YyGj6gl8KkjSbKOkj0IRy28nLisFdGBvgeQrvaLaA06VexptmrLjp1Qtyesw6zIJeP oHUImJltjPjFvyfkuIPfVIB87kukpB78bhSRA5mC365LsLRl+nrX7SauEo8b7MX0qbW9pg0f wsiyCCK0ioTTm4IWL2wiDB7PeiJSsViBORNKoxA093B42BWFJQARAQABtDRUaW0gQ2hlbiAo d29yayByZWxhdGVkKSA8dGltLmMuY2hlbkBsaW51eC5pbnRlbC5jb20+iQI+BBMBAgAoAhsD BgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAUCXFIuxAUJEYZe0wAKCRCiZ7WKota4STH3EACW 1jBRzdzEd5QeTQWrTtB0Dxs5cC8/P7gEYlYQCr3Dod8fG7UcPbY7wlZXc3vr7+A47/bSTVc0 DhUAUwJT+VBMIpKdYUbvfjmgicL9mOYW73/PHTO38BsMyoeOtuZlyoUl3yoxWmIqD4S1xV04 q5qKyTakghFa+1ZlGTAIqjIzixY0E6309spVTHoImJTkXNdDQSF0AxjW0YNejt52rkGXXSoi IgYLRb3mLJE/k1KziYtXbkgQRYssty3n731prN5XrupcS4AiZIQl6+uG7nN2DGn9ozy2dgTi smPAOFH7PKJwj8UU8HUYtX24mQA6LKRNmOgB290PvrIy89FsBot/xKT2kpSlk20Ftmke7KCa 65br/ExDzfaBKLynztcF8o72DXuJ4nS2IxfT/Zmkekvvx/s9R4kyPyebJ5IA/CH2Ez6kXIP+ q0QVS25WF21vOtK52buUgt4SeRbqSpTZc8bpBBpWQcmeJqleo19WzITojpt0JvdVNC/1H7mF 4l7og76MYSTCqIKcLzvKFeJSie50PM3IOPp4U2czSrmZURlTO0o1TRAa7Z5v/j8KxtSJKTgD lYKhR0MTIaNw3z5LPWCCYCmYfcwCsIa2vd3aZr3/Ao31ZnBuF4K2LCkZR7RQgLu+y5Tr8P7c e82t/AhTZrzQowzP0Vl6NQo8N6C2fcwjSrkCDQROjjboARAAx+LxKhznLH0RFvuBEGTcntrC 3S0tpYmVsuWbdWr2ZL9VqZmXh6UWb0K7w7OpPNW1FiaWtVLnG1nuMmBJhE5jpYsi+yU8sbMA 5BEiQn2hUo0k5eww5/oiyNI9H7vql9h628JhYd9T1CcDMghTNOKfCPNGzQ8Js33cFnszqL4I N9jh+qdg5FnMHs/+oBNtlvNjD1dQdM6gm8WLhFttXNPn7nRUPuLQxTqbuoPgoTmxUxR3/M5A KDjntKEdYZziBYfQJkvfLJdnRZnuHvXhO2EU1/7bAhdz7nULZktw9j1Sp9zRYfKRnQdIvXXa jHkOn3N41n0zjoKV1J1KpAH3UcVfOmnTj+u6iVMW5dkxLo07CddJDaayXtCBSmmd90OG0Odx cq9VaIu/DOQJ8OZU3JORiuuq40jlFsF1fy7nZSvQFsJlSmHkb+cDMZDc1yk0ko65girmNjMF hsAdVYfVsqS1TJrnengBgbPgesYO5eY0Tm3+0pa07EkONsxnzyWJDn4fh/eA6IEUo2JrOrex O6cRBNv9dwrUfJbMgzFeKdoyq/Zwe9QmdStkFpoh9036iWsj6Nt58NhXP8WDHOfBg9o86z9O VMZMC2Q0r6pGm7L0yHmPiixrxWdW0dGKvTHu/DH/ORUrjBYYeMsCc4jWoUt4Xq49LX98KDGN dhkZDGwKnAUAEQEAAYkCJQQYAQIADwIbDAUCXFIulQUJEYZenwAKCRCiZ7WKota4SYqUEACj P/GMnWbaG6s4TPM5Dg6lkiSjFLWWJi74m34I19vaX2CAJDxPXoTU6ya8KwNgXU4yhVq7TMId keQGTIw/fnCv3RLNRcTAapLarxwDPRzzq2snkZKIeNh+WcwilFjTpTRASRMRy9ehKYMq6Zh7 PXXULzxblhF60dsvi7CuRsyiYprJg0h2iZVJbCIjhumCrsLnZ531SbZpnWz6OJM9Y16+HILp iZ77miSE87+xNa5Ye1W1ASRNnTd9ftWoTgLezi0/MeZVQ4Qz2Shk0MIOu56UxBb0asIaOgRj B5RGfDpbHfjy3Ja5WBDWgUQGgLd2b5B6MVruiFjpYK5WwDGPsj0nAOoENByJ+Oa6vvP2Olkl gQzSV2zm9vjgWeWx9H+X0eq40U+ounxTLJYNoJLK3jSkguwdXOfL2/Bvj2IyU35EOC5sgO6h VRt3kA/JPvZK+6MDxXmm6R8OyohR8uM/9NCb9aDw/DnLEWcFPHfzzFFn0idp7zD5SNgAXHzV PFY6UGIm86OuPZuSG31R0AU5zvcmWCeIvhxl5ZNfmZtv5h8TgmfGAgF4PSD0x/Bq4qobcfaL ugWG5FwiybPzu2H9ZLGoaRwRmCnzblJG0pRzNaC/F+0hNf63F1iSXzIlncHZ3By15bnt5QDk l50q2K/r651xphs7CGEdKi1nU0YJVbQxJQ== Subject: Re: [RFC][PATCH 00/16] sched: Core scheduling Message-ID: <786668c1-fb52-508c-e916-f86707a1d791@linux.intel.com> Date: Fri, 22 Feb 2019 11:26:56 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 In-Reply-To: <20190222142030.GA32494@hirez.programming.kicks-ass.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2/22/19 6:20 AM, Peter Zijlstra wrote: > On Fri, Feb 22, 2019 at 01:17:01PM +0100, Paolo Bonzini wrote: >> On 18/02/19 21:40, Peter Zijlstra wrote: >>> On Mon, Feb 18, 2019 at 09:49:10AM -0800, Linus Torvalds wrote: >>>> On Mon, Feb 18, 2019 at 9:40 AM Peter Zijlstra wrote: >>>>> >>>>> However; whichever way around you turn this cookie; it is expensive and nasty. >>>> >>>> Do you (or anybody else) have numbers for real loads? >>>> >>>> Because performance is all that matters. If performance is bad, then >>>> it's pointless, since just turning off SMT is the answer. >>> >>> Not for these patches; they stopped crashing only yesterday and I >>> cleaned them up and send them out. >>> >>> The previous version; which was more horrible; but L1TF complete, was >>> between OK-ish and horrible depending on the number of VMEXITs a >>> workload had. >>> >>> If there were close to no VMEXITs, it beat smt=off, if there were lots >>> of VMEXITs it was far far worse. Supposedly hosting people try their >>> very bestest to have no VMEXITs so it mostly works for them (with the >>> obvious exception of single VCPU guests). >> >> If you are giving access to dedicated cores to guests, you also let them >> do PAUSE/HLT/MWAIT without vmexits and the host just thinks it's a CPU >> bound workload. >> >> In any case, IIUC what you are looking for is: >> >> 1) take a benchmark that *is* helped by SMT, this will be something CPU >> bound. >> >> 2) compare two runs, one without SMT and without core scheduler, and one >> with SMT+core scheduler. >> >> 3) find out whether performance is helped by SMT despite the increased >> overhead of the core scheduler >> >> Do you want some other load in the host, so that the scheduler actually >> does do something? Or is the point just that you show that the >> performance isn't affected when the scheduler does not have anything to >> do (which should be obvious, but having numbers is always better)? > > Well, what _I_ want is for all this to just go away :-) > > Tim did much of testing last time around; and I don't think he did > core-pinning of VMs much (although I'm sure he did some of that). I'm Yes. The last time around I tested basic scenarios like: 1. single VM pinned on a core 2. 2 VMs pinned on a core 3. system oversubscription (no pinning) In general, CPU bound benchmarks and even things without too much I/O causing lots of VMexits perform better with HT than without for Peter's last patchset. > still a complete virt noob; I can barely boot a VM to save my life. > > (you should be glad to not have heard my cursing at qemu cmdline when > trying to reproduce some of Tim's results -- lets just say that I can > deal with gpg) > > I'm sure he tried some oversubscribed scenarios without pinning. We did try some oversubscribed scenarios like SPECVirt, that tried to squeeze tons of VMs on a single system in over subscription mode. There're two main problems in the last go around: 1. Workload with high rate of Vmexits (SpecVirt is one) were a major source of pain when we tried Peter's previous patchset. The switch from vcpus to qemu and back in previous version of Peter's patch requires some coordination between the hyperthread siblings via IPI. And for workload that does this a lot, the overhead quickly added up. For Peter's new patch, this overhead hopefully would be reduced and give better performance. 2. Load balancing is quite tricky. Peter's last patchset did not have load balancing for consolidating compatible running threads. I did some non-sophisticated load balancing to pair vcpus up. But the constant vcpu migrations overhead probably ate up any improvements from better load pairing. So I didn't get much improvement in the over-subscription case when turning on load balancing to consolidate the VCPUs of the same VM. We'll probably have to try out this incarnation of Peter's patch and see how well the load balancing works. I'll try to line up some benchmarking folks to do some tests. Tim