Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752397AbdIFI7X (ORCPT ); Wed, 6 Sep 2017 04:59:23 -0400 Received: from mail-pg0-f53.google.com ([74.125.83.53]:35428 "EHLO mail-pg0-f53.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752306AbdIFI7Q (ORCPT ); Wed, 6 Sep 2017 04:59:16 -0400 X-Google-Smtp-Source: ADKCNb7dAs9I5w140Uv+CzH5Vb3cTpVS3DXXEHsj/TVq8HpfqJGHLBlgtidNIpivTA5cYaWDFipzxA== Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: Abysmal scheduler performance in Linus' tree? From: Andy Lutomirski X-Mailer: iPhone Mail (14G60) In-Reply-To: <20170906082520.xgvo3hewje7jvdyo@hirez.programming.kicks-ass.net> Date: Wed, 6 Sep 2017 01:59:14 -0700 Cc: Andy Lutomirski , Ingo Molnar , "linux-kernel@vger.kernel.org" Message-Id: <877A43A3-AC8F-4D7C-88E4-8E3D36B1DAFA@amacapital.net> References: <20170906082520.xgvo3hewje7jvdyo@hirez.programming.kicks-ass.net> To: Peter Zijlstra Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by nfs id v868xo15015645 Content-Length: 4425 Lines: 74 > On Sep 6, 2017, at 1:25 AM, Peter Zijlstra wrote: > >> On Tue, Sep 05, 2017 at 10:13:39PM -0700, Andy Lutomirski wrote: >> I'm running e7d0c41ecc2e372a81741a30894f556afec24315 from Linus' tree >> today, and I'm seeing abysmal scheduler performance. Running make -j4 >> ends up with all the tasks on CPU 3 most of the time (on my >> 4-logical-thread laptop). taskset -c 0 whatever puts whatever on CPU >> 0, but plain while true; do true; done puts the infinite loop on CPU 3 >> right along with the make -j4 tasks. >> >> This is on Fedora 26, and I don't think I'm doing anything weird. >> systemd has enabled the cpu controller, but it doesn't seem to have >> configured anything or created any non-root cgroups. >> >> Just a heads up. I haven't tried to diagnose it at all. > > "make O=defconfig-build -j80" results in: > > %Cpu0 : 90.7 us, 9.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu1 : 88.7 us, 11.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu2 : 93.5 us, 6.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu3 : 86.8 us, 13.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu4 : 89.7 us, 10.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu5 : 96.3 us, 3.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu6 : 95.3 us, 4.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu7 : 94.4 us, 5.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu8 : 91.7 us, 8.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu9 : 94.3 us, 5.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu10 : 90.7 us, 9.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu11 : 96.2 us, 3.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu12 : 91.5 us, 8.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu13 : 90.6 us, 9.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu14 : 97.2 us, 2.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu15 : 89.7 us, 10.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu16 : 90.6 us, 9.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu17 : 93.4 us, 6.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu18 : 90.6 us, 9.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu19 : 92.5 us, 7.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu20 : 94.4 us, 5.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu21 : 90.7 us, 9.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu22 : 92.5 us, 7.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu23 : 90.7 us, 9.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu24 : 91.6 us, 8.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu25 : 93.5 us, 6.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu26 : 93.4 us, 5.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.9 si, 0.0 st > %Cpu27 : 92.5 us, 7.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu28 : 92.5 us, 7.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu29 : 88.8 us, 11.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu30 : 90.6 us, 9.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu31 : 93.5 us, 6.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu32 : 93.5 us, 6.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu33 : 93.4 us, 6.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu34 : 90.7 us, 9.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu35 : 93.5 us, 6.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu36 : 90.7 us, 9.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu37 : 97.2 us, 2.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu38 : 92.5 us, 7.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > %Cpu39 : 92.6 us, 7.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st > > Do you have a .config somewhere? I'll attach tomorrow. I'll also test in a VM. > Are you running with the systemd? Is it > creating cpu cgroups? Yes systemd, no cgroups. > > Any specifics on your setup? On further fiddling, I only see this after a suspend and resume cycle.