Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 14 Aug 2023 11:14:32 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Zhang Rui <rui.zhang@intel.com>
CC:     <mingo@redhat.com>, <peterz@infradead.org>,
        <vincent.guittot@linaro.org>, <linux-kernel@vger.kernel.org>,
        <tj@kernel.org>, <srinivas.pandruvada@intel.com>
Subject: Re: [PATCH] sched/fair: Skip cpus with no sched domain attached
 during NOHZ idle balance
Message-ID: <20230814031432.GA574314@ziqianlu-dell>
References: <20230804090858.7605-1-rui.zhang@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230804090858.7605-1-rui.zhang@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?xblB2yaUIndIUjDB3LMpT+tV8yodCtU14hQFVzcP5/462tF3tnqewW7euAHv?=
 =?us-ascii?Q?CJTbGIteKgJzp/k524DG4Kft3W/l9uA8Bq4fRbhKgOq/bV1cDrBXk/yAn164?=
 =?us-ascii?Q?vFGMnGycCYmM9FjvMVFQi0kJfs25wqybMXmbOrxJEIIGkdwtDIY9nrCsuBJt?=
 =?us-ascii?Q?BxAigga22zwFE0Kgk5sOQLQmofJ3FCssjZRkWChMGy8JaBUWVXoxdGgBN4hA?=
 =?us-ascii?Q?1dOCTJ8+4jWkCxmpQSJT7gg4+CI3w+oj/qBMYBWXgeI0LKIOv4zFJ/GFVpra?=
 =?us-ascii?Q?2c1bRz5/PQK7EjsWoPboixWBwxRBKAuIMi8oNfaeTGIds+iTGIDSkGoLyjWz?=
 =?us-ascii?Q?qFCMYsRuICly5FjzO29UcI7QT+lHZZWrEtqhJ+vvGPjdMTDt4r91qR73cEfs?=
 =?us-ascii?Q?fEcc4lOuy7meBHQdIjH2hc9bN0p3GBEDpkLJLzUZiQT+S/z/mpk+HBcsDJaN?=
 =?us-ascii?Q?woES4I3d/yk90bAZFvYKrsfdFORKg9XdQb4HAlZ+dhsIhtEVyflp0t9RsnXj?=
 =?us-ascii?Q?4uEa0QZHfJ1l57hLXdLUhtM6Hj9/6h6Dct3f1eCG84Q+LnRXNMXkYoYdnROQ?=
 =?us-ascii?Q?mIl25uMVnqn7arJKWq1vvechuTh5s00aLw7VMSvplkl9ShVnN+GCBqw5Kxl3?=
 =?us-ascii?Q?yG4Eph4KU+bzJs2CbEC+LSj9wh/x5jVaTKBpt5V/AvIKsCcI2MPRWOAgKnBZ?=
 =?us-ascii?Q?Hdqka5WsJXtwq6L6sAumgkT/smUGHople18iXkBer+K4KU4WaQVchCfrkMSP?=
 =?us-ascii?Q?kj5obKTk3Q/uWeaaTD6rKEYKAkbZYpH7gws2obZdZ99yd6oiBw17kmtOYaI+?=
 =?us-ascii?Q?urpXm6GxPqJYt7JuMvkTYd3MiSlIeD0cOWWSbrCbGq4b/NFFko6A7HWY/+JU?=
 =?us-ascii?Q?9UzqJIx4n6UFA+nwqXihS0GOcnVE8+IjVK0DK2X7p1YKcbXaqENG5yYzmJjc?=
 =?us-ascii?Q?+/7e9S8C+yfbz9wZkdyvrW09AJF4rkMcBDX+txwazpyLdpbPNy54IrIraF3M?=
 =?us-ascii?Q?7mn42N24PUdkJVG+oMm0vPVzdAVYM4iH9vm8rBwoylGP7yUEuQzJfckBjzKh?=
 =?us-ascii?Q?mTmuyYa2n+EcrjA/qffa5kHHHUfwf48YmFCuljg3pZG5OkCDIDJhMC0IrKfT?=
 =?us-ascii?Q?IGVnZpo2Hv/DUyErn+C5m6/mrh0acFhByVex3cVQymfqaKPt3cklSJJopdAZ?=
 =?us-ascii?Q?cmqY5EWPpz/0OCTFUYW+iXwstg/5MpT8WRDsZdre6wyXmzc/6moYK9RClOkU?=
 =?us-ascii?Q?Vzco3JenLA1pqADnojLWGF5FerN8ri4/L1IMX48Ti7eqxxMLMaEQgT5B9air?=
 =?us-ascii?Q?ECG73tHnWtWUseJTwLSGOWA5FIKIt5PpbWuGbImPyErVYN1Vh2bPBjveEO19?=
 =?us-ascii?Q?cjNmYZb6GzRbtT7FTRx6ZuasEayPKMb0JE1NIBIfhiNUWC64Pxn/Ou/A0snA?=
 =?us-ascii?Q?IwdXav40G0mHYGDxfHITnAC+8zk+p+29H534l/IehYZ87/gf/9vIEN3QAUBg?=
 =?us-ascii?Q?R1Q+WOP1IflUYGXJ1g7Tbc9eLDwgetl5JO+EwhZj8aj7fE2wMG2ppvi8Bpxz?=
 =?us-ascii?Q?cOccNOUWG1GYk+Eh6/9o5lMSRUvjrI3PJKy25rZk?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1a764ceb-9d50-4407-c299-08db9c749b17
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2023 03:14:43.9571
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: J9yG8xf4LjDR9BIXAo3gV72yQaYikrbE2YGq0gNv05gfBk6MeBYLCWChxxjpDSs3OfLd8sai6eDGlN4bwND51Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB8364
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
        RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Rui,

On Fri, Aug 04, 2023 at 05:08:58PM +0800, Zhang Rui wrote:
> Problem statement
> -----------------
> When using cgroup isolated partition to isolate cpus including cpu0, it
> is observed that cpu0 is woken up frequenctly but doing nothing. This is
> not good for power efficiency.
> 
> <idle>-0     [000]   616.491602: hrtimer_cancel:       hrtimer=0xffff8e8fdf623c10
> <idle>-0     [000]   616.491608: hrtimer_start:        hrtimer=0xffff8e8fdf623c10 function=tick_sched_timer/0x0 expires=615996000000 softexpires=615996000000
> <idle>-0     [000]   616.491616: rcu_utilization:      Start context switch
> <idle>-0     [000]   616.491618: rcu_utilization:      End context switch
> <idle>-0     [000]   616.491637: tick_stop:            success=1 dependency=NONE
> <idle>-0     [000]   616.491637: hrtimer_cancel:       hrtimer=0xffff8e8fdf623c10
> <idle>-0     [000]   616.491638: hrtimer_start:        hrtimer=0xffff8e8fdf623c10 function=tick_sched_timer/0x0 expires=616420000000 softexpires=616420000000
> 
> The above pattern repeats every one or multiple ticks, results in total
> 2000+ wakeups on cpu0 in 60 seconds, when running workload on the
> cpus that are not in the isolated partition.
> 
> Rootcause
> ---------
> In NOHZ mode, an active cpu either sends an IPI or touches the idle
> cpu's polling flag to wake it up, so that the idle cpu can pull tasks
> from the busy cpu. The logic for selecting the target cpu is to use the
> first idle cpu that presents in both nohz.idle_cpus_mask and
> housekeeping_cpumask.
> 
> In the above scenario, when cpu0 is in the cgroup isolated partition,
> its sched domain is deteched, but it is still available in both of the
> above cpumasks. As a result, cpu0

I saw in nohz_balance_enter_idle(), if a cpu is isolated, it will not
set itself in nohz.idle_cpus_mask and thus should not be chosen as
ilb_cpu. I wonder what's stopping this from working?

> 1. is always selected when kicking idle load balance
> 2. is woken up from the idle loop
> 3. calls __schedule() but cannot find any task to pull because it is not
>    in any sched_domain, thus it does nothing and reenters idle.
> 
> Solution
> --------
> Fix the problem by skipping cpus with no sched domain attached during
> NOHZ idle balance.
> 
> Signed-off-by: Zhang Rui <rui.zhang@intel.com>
> ---
>  kernel/sched/fair.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index b3e25be58e2b..ea3185a46962 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -11340,6 +11340,9 @@ static inline int find_new_ilb(void)
>  		if (ilb == smp_processor_id())
>  			continue;
>  
> +		if (unlikely(on_null_domain(cpu_rq(ilb))))
> +			continue;
> +
>  		if (idle_cpu(ilb))
>  			return ilb;
>  	}
> -- 
> 2.34.1
>