Received: by 2002:a6b:500f:0:0:0:0:0 with SMTP id e15csp327176iob; Wed, 18 May 2022 03:07:51 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzI2b4ZlMVvkT8tD9/ZOW6/Colmt3F1XJ7uAFal1UnerwKpHYN/CtdKy7fT/nE3/i7GGOZ4 X-Received: by 2002:a63:5d1d:0:b0:3c6:480:1ab7 with SMTP id r29-20020a635d1d000000b003c604801ab7mr23157583pgb.228.1652868471402; Wed, 18 May 2022 03:07:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1652868471; cv=none; d=google.com; s=arc-20160816; b=j8YMH6MEw5rYIWoTh7m8iK9gqpiKVaCSCGGCEQXolWzURvjQyHppOCdaddDrEmJcUz 3eQL1wNYLIiICAScoU2V72jDqeOhm+LykWCuYUOd2PWrrsZBUW7Mp2OWjU+ciR/HR+mE Wxp5SuShjQmrN1TA+npqiRm6uwje38z5GaIl5M8G0mGYclQOp59kbieEjMLQSTmLicf4 FVvQnjqHg5GgcjCIQ16oDBF9rzRxu7lq+zVz2zhBONUAZGi+Qu5p8lOYQJEGaa2TL2AM J04dByIxEz2b3SQXGEg7Ga7Ad0ber2CVCsJph0mR1X9HR4fINr6mwv5c+aOjiS1yMUN3 1EDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=WMIQhBjGQ1AyewOOSm+elcUwgB8iEM4v1qmvWm/lR2g=; b=tiGC4xANlAUQuwFLpEJ9CUjc4fY+vV+3AO/yUkbT9+zYnJm6dn594Q4zjJxBRXgDud AAL34DpTtGFo+UmnxtYG+xnJ3rRtXaYQvWFKZsYRSIJ5RZhmTD95erE3ZMQxAr3826B/ W6cQ0hQX5MvhccoJcW6d4f+M3noZWfK6zE9+R5VeN/4mVEZ1xmDRNJVPUIShnhWrvNhb DzRyKXV1bl0IWvkZOJrH48ALNYo1jNJEuj3o90p1AlpmDLBPBOExW4TIoBNtxVFSZPp/ W19I8IKVvCr+MYW5TRqInc/+mKfYwqPO+rMk+RdwFOSMd6mxQpVWSL+Ui3QPI5ncp7Vn CKVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=XEvnqIPk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [2620:137:e000::1:18]) by mx.google.com with ESMTPS id c21-20020a63d155000000b0039db795524csi2147501pgj.42.2022.05.18.03.07.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 May 2022 03:07:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=XEvnqIPk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 2ACF820F52; Wed, 18 May 2022 02:49:20 -0700 (PDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234699AbiERJnG (ORCPT + 99 others); Wed, 18 May 2022 05:43:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234846AbiERJlb (ORCPT ); Wed, 18 May 2022 05:41:31 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 681F31C123 for ; Wed, 18 May 2022 02:41:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=WMIQhBjGQ1AyewOOSm+elcUwgB8iEM4v1qmvWm/lR2g=; b=XEvnqIPkN9Db3ze+a+4kzYnbOO 36gqGPjKECEBvvuiZf3efaGVETypr6da7xy2QUE8K9VsQMWKM9brv0hHVBnIgbCDZ8hm4CdhoHC3K bsDPx205shVP5QeBFg4y55I1Mb8n6RE7jKC2P6NyVNL7tcL3UxmL8ND4yUAgfBn4/Dr0CWfMU6Qv/ sXGr4eWbTmf9y6hMHOv/G1s7VAbFBs3MntLSnHkJUebbCfkC6cVzT4jIJnTwaAkO1dLAHK3Bdz9rc +Um+VridYpNhK1iAMGG0qcTfX9mlWjMXNtOe7vucFfTH6LMNla9br7o5coiJe7ti/SdVPEQDIKpIi 2oJO5MMQ==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=worktop.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrGB8-001V4E-Ux; Wed, 18 May 2022 09:41:15 +0000 Received: by worktop.programming.kicks-ass.net (Postfix, from userid 1000) id 5F2D5980DFF; Wed, 18 May 2022 11:41:12 +0200 (CEST) Date: Wed, 18 May 2022 11:41:12 +0200 From: Peter Zijlstra To: Mel Gorman Cc: Ingo Molnar , Vincent Guittot , Valentin Schneider , Aubrey Li , LKML Subject: Re: [PATCH 4/4] sched/numa: Adjust imb_numa_nr to a better approximation of memory channels Message-ID: <20220518094112.GE10117@worktop.programming.kicks-ass.net> References: <20220511143038.4620-1-mgorman@techsingularity.net> <20220511143038.4620-5-mgorman@techsingularity.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220511143038.4620-5-mgorman@techsingularity.net> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, May 11, 2022 at 03:30:38PM +0100, Mel Gorman wrote: > For a single LLC per node, a NUMA imbalance is allowed up until 25% > of CPUs sharing a node could be active. One intent of the cut-off is > to avoid an imbalance of memory channels but there is no topological > information based on active memory channels. Furthermore, there can > be differences between nodes depending on the number of populated > DIMMs. > > A cut-off of 25% was arbitrary but generally worked. It does have a severe > corner cases though when an parallel workload is using 25% of all available > CPUs over-saturates memory channels. This can happen due to the initial > forking of tasks that get pulled more to one node after early wakeups > (e.g. a barrier synchronisation) that is not quickly corrected by the > load balancer. The LB may fail to act quickly as the parallel tasks are > considered to be poor migrate candidates due to locality or cache hotness. > > On a range of modern Intel CPUs, 12.5% appears to be a better cut-off > assuming all memory channels are populated and is used as the new cut-off > point. A minimum of 1 is specified to allow a communicating pair to > remain local even for CPUs with low numbers of cores. For modern AMDs, > there are multiple LLCs and are not affected. Can the hardware tell us about memory channels?