Received: by 2002:a05:6a10:5bc5:0:0:0:0 with SMTP id os5csp929314pxb; Wed, 27 Oct 2021 15:26:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz+x6/bhSqoSpNRmom+tQtFJLT9L0DYFwAazTtGztRyt8pW7kAYbF/EQPLOASXynhusApg/ X-Received: by 2002:a17:90b:3843:: with SMTP id nl3mr419544pjb.147.1635373582092; Wed, 27 Oct 2021 15:26:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1635373582; cv=none; d=google.com; s=arc-20160816; b=kbBGoWA1QD0Q70w08PCjKvEplZohlJFLnTBKXtA5Z2EEmI0NkDVyMnY8x0+hMm7SPw i7cvKHlAk9bqsh1bFGTjgZ0fsHxflp9V9dsD6F8ZfCcoazSyiZ7WpcbG7sjZ2Y5OmM0i enagE0dHhMwqvjRZa5fmogTzq12Lkj7ab9gVprw8ZhK+QnwPu8kIWunYUin/cxxMwNni h8lM3eK6cTboqehTRq1xN61x9g1fpSlBAoQFTKD6mlZvtjDQ2uFUL8fsH1MPgPJE6c1f Ivs4IlikBHDvLub1CpaBKp10a2K4OGdlxEZJbQvyIRYbQUzTHs/YNjB20ImIcqPylKbm ZFiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:in-reply-to :references:mime-version:dkim-signature; bh=7v97GZsBc3GwS31cv/kL7gwWb5rRYExQxWI8jr90dtE=; b=bMIsAAqjGy7wMd7OMCjM8YwDf3SFE4SG1PsELtVJ8n5zi0WpjngDhay2HFipJ1QQV9 1n8jnnzyeuxYu9CvPsnAi3UkEUCQogjEMMvPB14j+zZQRM/bP4kEV5ATQ3zOjbTv26XH dvT06+RE2zD39K0Jo/VHJKnrKSQ6vWgT1LypTC0dmRSwSxRsuCrVv/nH8ah3PJn0qSZZ VmVEOblfajknRon0bLFHzMcfhd8Uyc0AldPVISKG8utulHLaXokkSTNBA0Z201BeUDpm p1v1vwrxaNZEBwfhHWzHoKO8/Y1N8M9b6a6eiOsJP7aAIZ33Oxf3YPs/G38qdKouA7eC C41Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hyOREpcW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id lb12si1996730pjb.80.2021.10.27.15.26.10; Wed, 27 Oct 2021 15:26:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hyOREpcW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232108AbhJ0IwO (ORCPT + 98 others); Wed, 27 Oct 2021 04:52:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239598AbhJ0IwM (ORCPT ); Wed, 27 Oct 2021 04:52:12 -0400 Received: from mail-yb1-xb34.google.com (mail-yb1-xb34.google.com [IPv6:2607:f8b0:4864:20::b34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6AA3C061570 for ; Wed, 27 Oct 2021 01:49:47 -0700 (PDT) Received: by mail-yb1-xb34.google.com with SMTP id r184so4444159ybc.10 for ; Wed, 27 Oct 2021 01:49:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=7v97GZsBc3GwS31cv/kL7gwWb5rRYExQxWI8jr90dtE=; b=hyOREpcWPyBwMLLPOzWmyNTD/+xsgrEQPeGMBwnMqQGMFdaXyt1qHpn7hRNwgc1q6m bwhLZlxJEDBEED6IsmT5bFPNzdkMjpJolvGL/0kuzQXNK2lI/XMAB9pnLwO63Qd/bilz ZAnT7/ijMsfNzGnXcbH9upiv2mkdij4oI3wCazjfi7ifZnMUfOcSxlsQCy0f6+IdW0zX hvJI6iCqsz35tJke6P6XFsJ5IB6hD92Y+mgvDedAEvxWPgvm8BWfyIPXgjtDtnggzfu5 UJm9zD39UZuV7I85Eu9fidBeDXv+tgzkLjiiBI+rPQ75r+ei+D2AaS1V8c6WcAB1a/Rk sX6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=7v97GZsBc3GwS31cv/kL7gwWb5rRYExQxWI8jr90dtE=; b=Ue3H5O7bw0PEW7SVminYLJp4lwEifGUBl17gRTtYBhvNgUytK2LD8pM42LVZ5lC/1b hmo5K5sN2jR+8W1oudwox94qOJDCudNVQ0/rZbk6eqe6jwso7m7G8+IeU811ef7iVPRB 0c8SmqUifNwnfEzs9/yrzEa7N2THPqzT6I0tZAF2/3P5t04s4MmWjCxk91MJ1zH3jRMc geRHcNpbT3oM6WsMIVTyRE0T3Ee+3z/B/uJ3+e3iwkRM+7IwSy5APKrArce22lJ/pzRf HoAriYwSvoiLYMkAWDHAjaF1qqj/fJSKJBQRP0zkcYo38wc8OccnLx/bCDLX04AOwfDw FVOg== X-Gm-Message-State: AOAM531RTAaY5ev9eKFFfckjEV1NzLRE2dAlHD9UEbwYu7VD0/YbfoxL vwr29z9eofJ6Jli3khoWft8ksnTx4+iR5JVmqV4NaA== X-Received: by 2002:a25:2d49:: with SMTP id s9mr30209604ybe.546.1635324587023; Wed, 27 Oct 2021 01:49:47 -0700 (PDT) MIME-Version: 1.0 References: <20211019123537.17146-1-vincent.guittot@linaro.org> <7128695d64e9161637b67315b5beb51c4accdc82.camel@linux.intel.com> In-Reply-To: <7128695d64e9161637b67315b5beb51c4accdc82.camel@linux.intel.com> From: Vincent Guittot Date: Wed, 27 Oct 2021 10:49:35 +0200 Message-ID: Subject: Re: [PATCH v3 0/5] Improve newidle lb cost tracking and early abort To: Tim Chen Cc: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 26 Oct 2021 at 19:25, Tim Chen wrote: > > On Tue, 2021-10-19 at 14:35 +0200, Vincent Guittot wrote: > > This patchset updates newidle lb cost tracking and early abort: > > > > The time spent running update_blocked_averages is now accounted in > > the 1st > > sched_domain level. This time can be significant and move the cost of > > newidle lb above the avg_idle time. > > > > The decay of max_newidle_lb_cost is modified to start only when the > > field > > has not been updated for a while. Recent update will not be decayed > > immediatlybut only after a while. > > > > The condition of an avg_idle lower than sysctl_sched_migration_cost > > has > > been removed as the 500us value is quite large and prevent > > opportunity to > > pull task on the newly idle CPU for at least 1st domain levels. > > > > Monitoring sd->max_newidle_lb_cost on cpu0 of a Arm64 system > > THX2 (2 nodes * 28 cores * 4 cpus) during the benchmarks gives the > > following results: > > min avg max > > SMT: 1us 33us 273us - this one includes the update of blocked > > load > > MC: 7us 49us 398us > > NUMA: 10us 45us 158us > > > > > > Some results for hackbench -l $LOOPS -g $group : > > group tip/sched/core + this patchset > > 1 15.189(+/- 2%) 14.987(+/- 2%) +1% > > 4 4.336(+/- 3%) 4.322(+/- 5%) +0% > > 16 3.654(+/- 1%) 2.922(+/- 3%) +20% > > 32 3.209(+/- 1%) 2.919(+/- 3%) +9% > > 64 2.965(+/- 1%) 2.826(+/- 1%) +4% > > 128 2.954(+/- 1%) 2.993(+/- 8%) -1% > > 256 2.951(+/- 1%) 2.894(+/- 1%) +2% > > > > tbench and reaim have not shown any difference > > > > Vincent, > > Our benchmark team tested the patches for our OLTP benchmark > on a 2 socket Cascade Lake > with 28 cores/socket. It is a smaller configuration > than the 2 socket Ice Lake we hae tested previously that has 40 > cores/socket so the overhead on update_blocked_averages is smaller > (~4%). > > Here's a summary of the results: > Relative Performance > (higher better) > 5.15 rc4 vanilla (cgroup disabled) 100% > 5.15 rc4 vanilla (cgroup enabled) 96% > patch v2 96% > patch v3 96% > > We didn't see much change in performance from the patch set. Thanks for testing. I was not expecting this patch to fix all your problems but at least those which have been highlighted during previous exchanges. Few problems still remain in your case if I'm not wrong: There is a patch that ensures that rq->next_balance is never set in the past. IIRC, you also mentioned in a previous thread that the idle LB happening during the tick just after the new idle balance was a problem in your case. Which is probably linked to your figures below > > Looking at the profile on update_blocked_averages a bit more, > the majority of the call to update_blocked_averages > happens in run_rebalance_domain. And we are not > including that cost of update_blocked_averages for > run_rebalance_domains in our current patch set. I think > the patch set should account for that too. nohz_newidle_balance keeps using sysctl_sched_migration_cost to trigger a _nohz_idle_balance(cpu_rq(cpu), NOHZ_STATS_KICK, CPU_IDLE); This would probably benefit to take into account the cost of update_blocked_averages instead > > > 0.60% 0.00% 3 [kernel.vmlinux] [k] run_rebalance_domains - - > | > --0.59%--run_rebalance_domains > | > --0.57%--update_blocked_averages > > Thanks. > > Tim >