Received: by 2002:a05:6a10:206:0:0:0:0 with SMTP id 6csp392987pxj; Fri, 14 May 2021 06:09:03 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxRjoj859dvNglPb7SXa+NGtjdWhjUEBpzPLrYaOIFu2FLS47dlXNoSFp0gneL715rpUkwe X-Received: by 2002:a6b:4e15:: with SMTP id c21mr31386069iob.116.1620997743165; Fri, 14 May 2021 06:09:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1620997743; cv=none; d=google.com; s=arc-20160816; b=N+XfKzAtga9OF/a5yaVJJ8lL1FEgchYoPOmJmzr+bb7u/mf1EUr7HWftQuJ4Ttd9xe A7Vmo/6o02+9RUZkDfnjKDNwznHog8zzTgW768/HBb5e7EK7iSvdtiRtBWfkjzSqoP++ ES+MHkGzpkmheMKqnN4q/WGZ0U35SqTSGwWSFk/YAQs9FelHBJeQJ5WqzrgtX6HnK0Lm Gzc4+nxr63ZaKuXbPUNKQEcglXZcGL2JjpsRUSLs3nyXO57gcHIaeSOfF99c7S6OWxHk vjb2QbymF37UKrurv6jcUDA83PYMHefP7sFnxoPSS+Hdia0N4jEKAF3mheFSf1s79Y1r Ul/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:content-language :in-reply-to:mime-version:user-agent:date:message-id:from:references :cc:to:subject:ironport-sdr:ironport-sdr; bh=ZZZRP0Zfq7b2yvUPLug0M/oZ7r4/NiE9hH3a5OvAsrI=; b=P46FvwgeVZRkjryn176pJfO0MAfH+v4V2RtTDVuc36T8s1EkTK8P7ocNGkIx9I2Reb wsy/gkXVNJG/7i6BOq+EMJ+4QXsAVfJh//1vCnAyQIrX46w5ld09PZx+j9sLHKQv2kfQ JYzpMa/viLtL8Rs7WswMHazA50srF1Bcf5i1e2zE5Gb7kGD9QYH2S1t3tIwC1/Ix24dT QutnK1CAWH/LEFgtTuGRk8chDfxgx5LSeErLtN8sgEOHibEraIUS107w3hVuyQFEvno6 hqpMOZJTkL1ygI6r6HXWt5RBP/O5mjslsOrIbtyD27k1XFzTizqLjdMvG0ouJCUbkb78 RMLQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id w14si8544419ilu.141.2021.05.14.06.08.49; Fri, 14 May 2021 06:09:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232091AbhENENq (ORCPT + 99 others); Fri, 14 May 2021 00:13:46 -0400 Received: from mga01.intel.com ([192.55.52.88]:44441 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231171AbhENENp (ORCPT ); Fri, 14 May 2021 00:13:45 -0400 IronPort-SDR: rVhVUzAY4F+N5X7tT5Lh38+zZPoWBLLjxtxCivMHxOYxgz3a4mfisELiEOCGS5ifNxFVAU+s22 bcQui/ow95oA== X-IronPort-AV: E=McAfee;i="6200,9189,9983"; a="221117157" X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="221117157" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2021 21:11:50 -0700 IronPort-SDR: hYRIpW6aBE6MiqIDrdQnVlOo012hB3FulWhH36IyRxk+56BPaqBvkld8Dj9vmNapbRIuUwZNso mmE/0tQ0AEBg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="456942727" Received: from aubrey-app.sh.intel.com (HELO [10.239.53.25]) ([10.239.53.25]) by fmsmga004.fm.intel.com with ESMTP; 13 May 2021 21:11:47 -0700 Subject: Re: [PATCH v2 6/8] sched/idle: Move busy_cpu accounting to idle callback To: Srikar Dronamraju Cc: Ingo Molnar , Peter Zijlstra , LKML , Mel Gorman , Rik van Riel , Thomas Gleixner , Valentin Schneider , Vincent Guittot , Dietmar Eggemann , Gautham R Shenoy , Parth Shah References: <20210506164543.90688-1-srikar@linux.vnet.ibm.com> <20210506164543.90688-7-srikar@linux.vnet.ibm.com> <47d29f1d-cea6-492a-5125-85db6bce0fa7@linux.intel.com> <20210513073112.GV2633526@linux.vnet.ibm.com> From: Aubrey Li Message-ID: <5823f298-6fae-5a73-3ab8-f708d90a7e52@linux.intel.com> Date: Fri, 14 May 2021 12:11:50 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <20210513073112.GV2633526@linux.vnet.ibm.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 5/13/21 3:31 PM, Srikar Dronamraju wrote: > * Aubrey Li [2021-05-12 16:08:24]: > >> On 5/7/21 12:45 AM, Srikar Dronamraju wrote: >>> Currently we account nr_busy_cpus in no_hz idle functions. >>> There is no reason why nr_busy_cpus should updated be in NO_HZ_COMMON >>> configs only. Also scheduler can mark a CPU as non-busy as soon as an >>> idle class task starts to run. Scheduler can then mark a CPU as busy >>> as soon as its woken up from idle or a new task is placed on it's >>> runqueue. >> >> IIRC, we discussed this before, if a SCHED_IDLE task is placed on the >> CPU's runqueue, this CPU should be still taken as a wakeup target. >> > > Yes, this CPU is still a wakeup target, its only when this CPU is busy, that > we look at other CPUs > >> Also, for those frequent context-switching tasks with very short idle, >> it's expensive for scheduler to mark idle/busy every time, that's why >> my patch only marks idle every time and marks busy ratelimited in >> scheduler tick. >> > > I have tried few tasks with very short idle times and updating nr_busy > everytime, doesnt seem to be impacting. Infact, it seems to help in picking > the idler-llc more often. > How many CPUs in your LLC? This is a system with 192 CPUs, 4 nodes and each node has 48 CPUs in LLC domain. It looks like for netperf both TCP and UDP cases have the notable change under 2 x overcommit, it may be not interesting though. hackbench(48 tasks per group) ========= case load baseline(std%) compare%( std%) process-pipe group-1 1.00 ( 6.74) -4.61 ( 8.97) process-pipe group-2 1.00 ( 36.84) +11.53 ( 26.35) process-pipe group-3 1.00 ( 24.97) +12.21 ( 19.05) process-pipe group-4 1.00 ( 18.27) -2.62 ( 17.60) process-pipe group-8 1.00 ( 4.33) -2.22 ( 3.08) process-sockets group-1 1.00 ( 7.88) -20.26 ( 15.97) process-sockets group-2 1.00 ( 5.38) -19.41 ( 9.25) process-sockets group-3 1.00 ( 4.22) -5.70 ( 3.00) process-sockets group-4 1.00 ( 1.44) -1.80 ( 0.79) process-sockets group-8 1.00 ( 0.44) -2.86 ( 0.06) threads-pipe group-1 1.00 ( 5.43) -3.69 ( 3.59) threads-pipe group-2 1.00 ( 18.00) -2.69 ( 16.79) threads-pipe group-3 1.00 ( 21.72) -9.01 ( 21.34) threads-pipe group-4 1.00 ( 21.58) -6.43 ( 16.26) threads-pipe group-8 1.00 ( 3.05) -0.15 ( 2.31) threads-sockets group-1 1.00 ( 14.51) -5.35 ( 13.85) threads-sockets group-2 1.00 ( 3.97) -24.15 ( 4.40) threads-sockets group-3 1.00 ( 4.97) -9.05 ( 2.46) threads-sockets group-4 1.00 ( 1.98) -3.44 ( 0.49) threads-sockets group-8 1.00 ( 0.37) -2.13 ( 0.20) netperf ======= case load baseline(std%) compare%( std%) TCP_RR thread-48 1.00 ( 3.84) -2.20 ( 3.83) TCP_RR thread-96 1.00 ( 5.22) -4.97 ( 3.90) TCP_RR thread-144 1.00 ( 7.97) -0.75 ( 4.39) TCP_RR thread-192 1.00 ( 3.03) -0.67 ( 4.40) TCP_RR thread-384 1.00 ( 22.27) -14.15 ( 36.28) UDP_RR thread-48 1.00 ( 2.08) -0.39 ( 2.29) UDP_RR thread-96 1.00 ( 2.48) -4.26 ( 16.06) UDP_RR thread-144 1.00 ( 49.50) -3.28 ( 34.86) UDP_RR thread-192 1.00 ( 6.39) +8.07 ( 88.15) UDP_RR thread-384 1.00 ( 31.54) -12.76 ( 35.98)