Received: by 2002:a05:6a10:a0d1:0:0:0:0 with SMTP id j17csp1885256pxa; Mon, 3 Aug 2020 01:26:41 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw9cKuOMM+YO+21L1abt+8k1gq4Ke7jfp7QdATIWVVkKCFPtIfb3QzaF2TzE3A8UJD8DEip X-Received: by 2002:a17:906:fc10:: with SMTP id ov16mr16134419ejb.171.1596443201053; Mon, 03 Aug 2020 01:26:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1596443201; cv=none; d=google.com; s=arc-20160816; b=Q4NH2Zu0DwfsV+gAePgvwWGL1GzWueYq0cB268UXIJn5CbFX5/HI+5eaUbC4/dwFrO isKjYBlWDwKQ+hDM+fBG7eRKAuaa8EWugouT79/hJwF/i8ek6CQFsg0Au4w61we5RR43 b3vofcvobZrXRX3r6ODFDQrnaDSRwXoN16c9NpFDehETis6xJEZjhZjEO6PoYqfae1Lc DcIICaBe5uD9PfF/2KnaivqKYs2OY2MlIvHqOSxlKyzK+OgqFgQKbr0NEYO3ptYwm/vF aW0Be6MiXwD3UDOvG69PDMvEsO9REcky1nMZdpBlB5MmXuEh3Sf+eOp96Jgm3oLBcYwS YfGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:references:cc:to:subject:from:ironport-sdr:ironport-sdr; bh=nz4lSXazseWaha0RNtyKC42psS/hEOF8Ks7tcrqnSII=; b=Vt6mzwR4mUG4F0aUr6ZJCZCaXR1Inph8dQiQ2HUDRF9IE3jWkVtZynKznkLuELbIYo /EWFsB1TvqpiDUX7QYhWRPlFq+oJG2TR3KMXsAOPMWQsUlLLvTlfm5Loig4m3nKOT5N1 6H1QId2ArIr7vRrPXdPG0XyBMrOjS3jPvEmdNUlGzw2qrTsjN966HOoGHG5yUD+KwG8N tVGjtXkLe4pWdY1arDIj+wXpIHUcLBi/LDN32dHlJsj3MnoQyNElzHBVEBbpnonfVmgd HcHCbfkZAmk06CnaJEGMsixqD9ts2zZXZnY63MnOKFN9o9yzS28aC1J8ar3magazOApx Yzhg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id r9si10313235ejp.629.2020.08.03.01.26.18; Mon, 03 Aug 2020 01:26:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725980AbgHCIXd (ORCPT + 99 others); Mon, 3 Aug 2020 04:23:33 -0400 Received: from mga07.intel.com ([134.134.136.100]:34766 "EHLO mga07.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725965AbgHCIXc (ORCPT ); Mon, 3 Aug 2020 04:23:32 -0400 IronPort-SDR: 5jktLlD9grkGgOvxBtzh8TIXl61whsEF41wavJj2w9CH2R0OlWOjR+yN/FhH7+v22+Qrvs1CXU +GZHO2SbRdng== X-IronPort-AV: E=McAfee;i="6000,8403,9701"; a="216487932" X-IronPort-AV: E=Sophos;i="5.75,429,1589266800"; d="scan'208";a="216487932" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Aug 2020 01:23:32 -0700 IronPort-SDR: IqKQYllmsmUv1SF5B7jY+UiuAiPWzm98dCbhaC3Q5ZJE0AmQILR9xC90WHYbW93Slw25btj1Fr iSAkhYogKByg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,429,1589266800"; d="scan'208";a="329927012" Received: from cli6-desk1.ccr.corp.intel.com (HELO [10.239.161.135]) ([10.239.161.135]) by FMSMGA003.fm.intel.com with ESMTP; 03 Aug 2020 01:23:26 -0700 From: "Li, Aubrey" Subject: Re: [RFC PATCH 00/16] Core scheduling v6 To: viremana@linux.microsoft.com, Nishanth Aravamudan , Julien Desfossez , Peter Zijlstra , Tim Chen , mingo@kernel.org, tglx@linutronix.de, pjt@google.com, torvalds@linux-foundation.org Cc: linux-kernel@vger.kernel.org, subhra.mazumdar@oracle.com, fweisbec@gmail.com, keescook@chromium.org, kerrnel@google.com, Phil Auld , Aaron Lu , Aubrey Li , Valentin Schneider , Mel Gorman , Pawan Gupta , Paolo Bonzini , Joel Fernandes , joel@joelfernandes.org, vineethrp@gmail.com, Chen Yu , Christian Brauner , "Ning, Hongyu" , =?UTF-8?B?YmVuYmppYW5nKOiSi+W9qik=?= References: Message-ID: <6d0f9fc0-2e34-f559-29bc-4143e6d3f751@linux.intel.com> Date: Mon, 3 Aug 2020 16:23:26 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2020/7/1 5:32, Vineeth Remanan Pillai wrote: > Sixth iteration of the Core-Scheduling feature. > > Core scheduling is a feature that allows only trusted tasks to run > concurrently on cpus sharing compute resources (eg: hyperthreads on a > core). The goal is to mitigate the core-level side-channel attacks > without requiring to disable SMT (which has a significant impact on > performance in some situations). Core scheduling (as of v6) mitigates > user-space to user-space attacks and user to kernel attack when one of > the siblings enters the kernel via interrupts. It is still possible to > have a task attack the sibling thread when it enters the kernel via > syscalls. > > By default, the feature doesn't change any of the current scheduler > behavior. The user decides which tasks can run simultaneously on the > same core (for now by having them in the same tagged cgroup). When a > tag is enabled in a cgroup and a task from that cgroup is running on a > hardware thread, the scheduler ensures that only idle or trusted tasks > run on the other sibling(s). Besides security concerns, this feature > can also be beneficial for RT and performance applications where we > want to control how tasks make use of SMT dynamically. > > This iteration is mostly a cleanup of v5 except for a major feature of > pausing sibling when a cpu enters kernel via nmi/irq/softirq. Also > introducing documentation and includes minor crash fixes. > > One major cleanup was removing the hotplug support and related code. > The hotplug related crashes were not documented and the fixes piled up > over time leading to complex code. We were not able to reproduce the > crashes in the limited testing done. But if they are reroducable, we > don't want to hide them. We should document them and design better > fixes if any. > > In terms of performance, the results in this release are similar to > v5. On a x86 system with N hardware threads: > - if only N/2 hardware threads are busy, the performance is similar > between baseline, corescheduling and nosmt > - if N hardware threads are busy with N different corescheduling > groups, the impact of corescheduling is similar to nosmt > - if N hardware threads are busy and multiple active threads share the > same corescheduling cookie, they gain a performance improvement over > nosmt. > The specific performance impact depends on the workload, but for a > really busy database 12-vcpu VM (1 coresched tag) running on a 36 > hardware threads NUMA node with 96 mostly idle neighbor VMs (each in > their own coresched tag), the performance drops by 54% with > corescheduling and drops by 90% with nosmt. > We found uperf(in cgroup) throughput drops by ~50% with corescheduling. The problem is, uperf triggered a lot of softirq and offloaded softirq service to *ksoftirqd* thread. - default, ksoftirqd thread can run with uperf on the same core, we saw 100% CPU utilization. - coresched enabled, ksoftirqd's core cookie is different from uperf, so they can't run concurrently on the same core, we saw ~15% forced idle. I guess this kind of performance drop can be replicated by other similar (a lot of softirq activities) workloads. Currently core scheduler picks cookie-match tasks for all SMT siblings, does it make sense we add a policy to allow cookie-compatible task running together? For example, if a task is trusted(set by admin), it can work with kernel thread. The difference from corescheduling disabled is that we still have user to user isolation. Thanks, -Aubrey