Received: by 2002:a05:6a10:f347:0:0:0:0 with SMTP id d7csp994726pxu; Wed, 2 Dec 2020 08:27:36 -0800 (PST) X-Google-Smtp-Source: ABdhPJwbCYA33jPbAEOJdiwqamJmAc3RpyyaZ5aYPiwx/Wy6He+1FZKIpROdauohJkG3x76LwSRl X-Received: by 2002:a17:906:c7d9:: with SMTP id dc25mr598264ejb.138.1606926456172; Wed, 02 Dec 2020 08:27:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1606926456; cv=none; d=google.com; s=arc-20160816; b=rUenBWkaLZ8LOf4+h2gcp8HgIJ3T/vKwOR4KpxxlTR1UKF0dFCcvPkLxPAspiaXPaQ ZsrbLWlwYMD7xoHlYx34QuSGc358thot/yx0pFvfrA41s8YWAC9ZMlVtjtDHRhk7D4zU S+wY9DiXcoOoffwz0Z7EJINz8oC6Kdi9aMuFQw1NfEN1j6Wc783eDn9oifqVhwm07WQs C2KvAfkzHxAmjlc31P+HJolIEx1l/ylfnLry/Frj8zPNOrdv+B9ipxA/MESLtCnE5IHv Y+bz+GrcPbtt0xpbThA2kUpHaSJJn9sAVBiaE24tDuTJcp4AVfUTRQ0PbzLoNZ3EZeSp gfcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:in-reply-to :references:mime-version:dkim-signature; bh=zkzWh1n8apsb00YEXyZll9FC6kSVnRaWL8x+YUkOQho=; b=c19Pu0FXkbYLr/sk28oAB+G1ZbUmo2tCCJ+EHVWTm/ChTYlu4niZxK1UTKU6LoSbOI sLlf49tA5XPFdBZP/DRIpp1h1mifuh4Bj/DfOwxOfaz4jgmJp+Mh0WEmVPDjeoNaMLFZ Gp+xClFgxeJXpX7NOvNVtPPkE2r9hvL4HYiWUDvOUdcV2Vev/I2SfPonnbL3smciZ+rz 7RTiT4/Uav/Bg+GaW29shWWerxzWyVavzYkarfXVfsEteaAHNSD9sk8nr9UdghIRapeB KRGSQt6cezN6uGlkFC+x8dJj6HD5vd8dvLH5YD8cE9+Jy67nCGgwGVOrtycojLA7Tq/j Bg8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KZLPsRnO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id y9si290960edq.205.2020.12.02.08.27.12; Wed, 02 Dec 2020 08:27:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KZLPsRnO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389072AbgLBQXy (ORCPT + 99 others); Wed, 2 Dec 2020 11:23:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388999AbgLBQXx (ORCPT ); Wed, 2 Dec 2020 11:23:53 -0500 Received: from mail-lf1-x144.google.com (mail-lf1-x144.google.com [IPv6:2a00:1450:4864:20::144]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3FA0C0613CF for ; Wed, 2 Dec 2020 08:23:12 -0800 (PST) Received: by mail-lf1-x144.google.com with SMTP id d8so5723854lfa.1 for ; Wed, 02 Dec 2020 08:23:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zkzWh1n8apsb00YEXyZll9FC6kSVnRaWL8x+YUkOQho=; b=KZLPsRnOwomBLzlBoH7n1dnG+b0g/3ya9uDJsTghBWnNcTc54eODzPHInC0a9fMP3x hxxf3Lal0n0O0WTvUY5coWMubP9nC7hhs+JmiO2Xe/T6eurKm6CYlcn4bXldUPrKcqyz rpL/AlwSXaRadcySApnPZPAJtf1km/Smoml0kiiNE8SevdK8Uco4dMDq0PebTQoLDllY AoQ/pw/svxuApE0wJft3EMaWcv/AlKpk1BWON9m/ZJX3JXPoo7ftAjqG749EzsbAc9sa BQkS5QUfjDF1HTF7ZNFrv0kTfIFcWEMmGHnJsSY/dBcPVNSTin/T9NysuSMuDuvYRTye +gNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zkzWh1n8apsb00YEXyZll9FC6kSVnRaWL8x+YUkOQho=; b=t+FY+Ax4zcQpAKAXU3WNkeXfP+/3XFPbTcASNqve7mQkm0RjcKELa8zVz6HI/P3ja1 a8hEAJW/uUq8GSPIgHNE9CA2rkfAyanKGpF603rASvepI660uwe1jD2SqgFqS6Tvwwyi 044x9NpS4JfFDSna/vzWxq6VpwP/29cK+IpxfeVZBE7YX/Q8fzS/Sxebnu/yTmUWbX5t Pf+NcAsZG9urLFvRQomzfYrfbbj3KyHzbdEOgqrw5e1Mr1m9B8KSoMa9cTYuco2C6vp3 80Hu2sVVDpjL6i7aVjZR33KatnEEbxmJFdDxLZix9i5NmZtnf42ZgrstD7VnIyX92dat Xvew== X-Gm-Message-State: AOAM532y2SMTf8/9YsCt0jUbKGi/mqkBGSw4j4blhG70eE4baIgEShJt wCDGAVUydYiWF3iPLaQOoTJy+uvLl6lTY7Qjw79uXg== X-Received: by 2002:a19:8347:: with SMTP id f68mr1679092lfd.83.1606926191127; Wed, 02 Dec 2020 08:23:11 -0800 (PST) MIME-Version: 1.0 References: <20201125090923.GA3723@shao2-debian> <6fef3fc7-be18-92e5-c622-add6decb88c4@linux.intel.com> <20201126121351.GJ3371@techsingularity.net> In-Reply-To: From: Vincent Guittot Date: Wed, 2 Dec 2020 17:22:59 +0100 Message-ID: Subject: Re: [sched/fair] 8d86968ac3: netperf.Throughput_tps -29.5% regression To: "Li, Aubrey" Cc: Mel Gorman , kernel test robot , 0day robot , Mel Gorman , Qais Yousef , Valentin Schneider , Jiang Biao , Tim Chen , LKML , lkp@lists.01.org, "Huang, Ying" , "Tang, Feng" , zhengjun.xing@intel.com, Ingo Molnar , Peter Zijlstra , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Aubrey Li , Chen Yu Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 2 Dec 2020 at 15:30, Li, Aubrey wrote: > > Hi Mel, > > On 2020/11/26 20:13, Mel Gorman wrote: > > On Thu, Nov 26, 2020 at 02:57:07PM +0800, Li, Aubrey wrote: > >> Hi Robot, > >> > >> On 2020/11/25 17:09, kernel test robot wrote: > >>> Greeting, > >>> > >>> FYI, we noticed a -29.5% regression of netperf.Throughput_tps due to commit: > >>> > >>> > >>> commit: 8d86968ac36ea5bff487f70b5ffc252a87d44c51 ("[RFC PATCH v4] sched/fair: select idle cpu from idle cpumask for task wakeup") > >>> url: https://github.com/0day-ci/linux/commits/Aubrey-Li/sched-fair-select-idle-cpu-from-idle-cpumask-for-task-wakeup/20201118-115145 > >>> base: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git 09162bc32c880a791c6c0668ce0745cf7958f576 > >> > >> I tried to replicate this on my side on a 192 threads(with SMT) machine as well and didn't see the regression. > >> > >> nr_threads v5.9.8 +patch > >> 96(50%) 1 (+/- 2.499%) 1.007672(+/- 3.0872%) > >> > >> I also tested another 100% case and see similar improvement as what I saw on uperf benchmark > >> > >> nr_threads v5.9.8 +patch > >> 192(100%) 1 (+/- 45.32%) 1.864917(+/- 23.29%) > >> > >> My base is v5.9.8 BTW. > >> > >>> ip: ipv4 > >>> runtime: 300s > >>> nr_threads: 50% > >>> cluster: cs-localhost > >>> test: UDP_RR > >>> cpufreq_governor: performance > >>> ucode: 0x5003003 > >>> > > > > Note that I suspect that regressions with this will be tricky to reproduce > > because it'll depend on the timing of when the idle mask gets updated. With > > this configuration there are 50% "threads" which likely gets translates > > into 1 client/server per thread or 100% of CPUs active but as it's a > > ping-pong workload, the pairs are rapidly idling for very short periods. > > I tried to replicate this regression but no solid fruit found. I tried 30 times > 300s 50%.netperf running, all the data are better than the default data. The only > interesting thing I found is an option CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_32B=y, > but it performs different on different machines. In case anything I missed, > do you have any suggestions to replicate this regression? > > > > > If the idle mask is not getting cleared then select_idle_cpu() is > > probably returning immediately. select_idle_core() is almost certainly > > failing so that just leaves select_idle_smt() to find a potentially idle > > CPU. That's a limited search space so tasks may be getting stacked and > > missing CPUs that are idling for short periods. > > Vincent suggested we decouple idle cpumask from short idle(stop tick) and > set it every time the CPU enters idle, I'll make this change in V6. This v6 behavior is much more conservative regarding the idle cpumask and should restore the regression that appeared with V4 > > > > > On the flip side, I expect cases like hackbench to benefit because it > > can saturate a machine to such a degree that select_idle_cpu() is a waste > > of time. > > Yes, I believe that's also why I saw uperf/netperf improvement at high > load levels. > > > > > That said, I haven't followed the different versions closely. I know v5 > > got a lot of feedback so will take a closer look at v6. Fundamentally > > though I expect that using the idle mask will be a mixed bag. At low > > utilisation or over-saturation, it'll be a benefit. At the point where > > the machine is almost fully busy, some workloads will benefit (lightly > > communicating workloads that occasionally migrate) and others will not > > (ping-pong workloads looking for CPUs that are idle for very brief > > periods). > > Do you have any interested workload [matrix] I can do the measurement? > > > > > It's tricky enough that it might benefit from a sched_feat() check that > > is default true so it gets tested. For regressions that show up, it'll > > be easy enough to ask for the feature to be disabled to see if it fixes > > it. Over time, that might give an idea of exactly what sort of workloads > > benefit and what suffers. > > Okay, I'll add a sched_feat() for this feature. > > > > > Note that the cost of select_idle_cpu() can also be reduced by enabling > > SIS_AVG_CPU so it would be interesting to know if the idle mask is superior > > or inferior to SIS_AVG_CPU for workloads that show regressions. > > > > Thanks, > -Aubrey