Received-SPF: pass (google.com: domain of linux-kernel+bounces-28281-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
From: Eric Mackay <eric.mackay@oracle.com>
To: mark.rutland@arm.com
Cc: Jonathan.Cameron@huawei.com, Matteo.Carlini@arm.com,
        Valentin.Schneider@arm.com, akpm@linux-foundation.org,
        anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@linux.com,
        dave.kleikamp@oracle.com, linux-arm-kernel@lists.infradead.org,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        linux@armlinux.org.uk, robin.murphy@arm.com,
        vanshikonda@os.amperecomputing.com, yang@os.amperecomputing.com
Subject: Re: [PATCH] ARM64: Dynamically allocate cpumasks and increase supported CPUs to 512
Date: Tue, 16 Jan 2024 13:06:33 -0800
Message-Id: <20240116210633.116278-1-eric.mackay@oracle.com>
In-Reply-To: <ZaZ_xK5QHVPSD7N6@FVFF77S0Q05N>
References: <ZaZ_xK5QHVPSD7N6@FVFF77S0Q05N>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?0oHC4iWlOQ8VnTD0Tb0gS4Jf8DXYmRutUsjREAikFjFAk5CDXdQYDLab7Q/4?=
 =?us-ascii?Q?hulstMSGDtIkZVZDgKNSaYyTN3/1iOOUGAQnwBG3AgTzXz9uhp6zoE21Aihk?=
 =?us-ascii?Q?dxl7R+ZA4nl251l5XuMygvIaM8sut6ti7TWwLXHF5Rh0zSOcw72gcbNvuoCZ?=
 =?us-ascii?Q?AdtjB6ZLYJVUOUYlwVET208lVrAR8yW9DW1pU/hSv7BHYLmkhzXQcxKN3xHS?=
 =?us-ascii?Q?JCwwD+G7QrvdVhjI+GYk8zVa8cJbReZPTPs+TKNHF0K7H3iy9xXlCQOoEDoL?=
 =?us-ascii?Q?8o2GzxYquU02UWKEGtgsV+XdQgF3z58TZdY5GNE/G6QEzJAkk0WYCEX4DGsU?=
 =?us-ascii?Q?+MqvHFzwnWZoiqTcaLdlfjOQHW7tLSM9/XFXzcN+6JQbkd5h7Lnis5hOMDON?=
 =?us-ascii?Q?yzFyF/9nmbd75X9zzW5rfHN7rPd0eqyxgqdjEMA9n03OQYVnvkMVLxUWMJpC?=
 =?us-ascii?Q?Dwjd3OazTrzpsZqog6ZJsnnFcqB54aQFMvauuSJPs3b1zQBXg/ze8D8xwOz1?=
 =?us-ascii?Q?wFqnSsvPJBNarc2ar5p/l+1JcgwH4DY74R0LW849oMkz1lFdSFrrQDAqrIF6?=
 =?us-ascii?Q?tPNt/Wr8gvuh/AEbER4BUUHsgGLcrNCUWJCrkN/YX3hwH9fzL+q1RCdOVjk4?=
 =?us-ascii?Q?TOeoIBz9OFSUfTckUXMEshfCEHl1JVjKxgT6fWyfEYWjWTClxn3kCfP9D9At?=
 =?us-ascii?Q?RV2i3o/3e4IuaxBWbx7gj5b1X0hdGJAyuhduAgHdAF6Vio5XgCVQOAUGOuf2?=
 =?us-ascii?Q?/17FA/mKBgyQKeT6irUaH7cbWSipZJMfDnf43utLSO4qTWiyX5UhF2u+uzAu?=
 =?us-ascii?Q?YOFL8YPqI8A+JMvax4kXLnzqiJ7d+i8mig1Ot7MCVj9UCgWvzc4qzBALG8OU?=
 =?us-ascii?Q?B+/bwcn5ERZ60lJmXS1FusRggrS1x9HYn/Qrkq1Xi/amJhvLPcSjJouEt/LK?=
 =?us-ascii?Q?hjDEjj5Vnfm/4ZjtQGfCazTDP4n6Qz9sSjOQ0n4yrEWn0nd5gxcgtIOjVLjS?=
 =?us-ascii?Q?FEcrGwR5txNZA4Nvmwhmy3NdGPXjxwxZDOM2NDsO7k8JsqomZFZQ7mWjR9cV?=
 =?us-ascii?Q?T9PJTOC+wqELxGPQSGnXxKDlSVcveLzvc9XDUbwLTD7SsaTAxag8qsxmAwW6?=
 =?us-ascii?Q?xAkfwn4br313tdHu3fEbCJax4j4sN3UNUjX/40pNKvxjcYRR59UXl4ueRihb?=
 =?us-ascii?Q?NaiuonjS41ciUf7/taPpAaojXtDSRqX0lYzR99ee/xwsuev83tGzcWMRvaHJ?=
 =?us-ascii?Q?39+I+TMulwK/STHZGEaOSUYDLVBf6dhZL2xpIBMYNx8QPS0hAMo0nEtMUsPr?=
 =?us-ascii?Q?bD4j4cVyZyaE2kRtbTU8f+SdRCOm3/DpX34aMpaX6WCpDXyutdwhpJ5ECDWx?=
 =?us-ascii?Q?PAZgSYvSjesR7fe7DfFhj2PoSYkSRqmrEH3Q0+KYALT/ZGgGNytMOFFayqGJ?=
 =?us-ascii?Q?BJQ3qHu7JMdAaOgaWIfY0IWNCZ64wF1LgkffMK1rmuTHQVI4dH6aTPy3gb/T?=
 =?us-ascii?Q?MYaTerBlNePp6G1Pj9XskzYiX5smi2AWN5vgysodxE0RI6wiDryTc5bdUuC+?=
 =?us-ascii?Q?vHip/zf0gc44ig2WuIoK9nGQmIyOCmex6E0JSdbDdVPT21HIpPy43L1QSdFL?=
 =?us-ascii?Q?Ow=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	KXEiNCdNdJ0quKPpYW/bgPhG8rFzNBxpaPJJ1/vDRe7RjyTqItmo690sHbFHJM+5xAwR5pIc0Xno+gV06xa/wMzSq6f+N0sNu/hT6rkJC5/RKsKMsAMwul7bYEfjrAeqLN41RkovuvElYFXBbozj2fgF9Q4EOv/vGn8yiGLGdjLywGvBqAeo4uNlax4jZ0PL7PuFdSWbNFWNE1iNLEsiS4afOVcRs6xmd2VHDGyjLE7XyG9amtw6cZoP0OoRfT5yaGCo2QLmfnD6h9MZbfMc66umrbDTiwTyC471SyTli/BRHJy0hq4eiRAzE0kj0Q1NY9gW7Yl7+w0KJADYewywxOSbZT3SyiB0+YGVMwuMJV78++nqmuTFetkgsJC8YmEeBB9w6P484rU1WY50GZLC2OybndNc7qy1xI1lHEFR+hLjaO1ch/Kt/H40+c9sw3otcf2CiHNquqriBGezf3ERpr7mTDwZzpF+YcGcvYkpDjoJplu5NM1W1yvw7wLkbIAn7epnoddmLGGH2EjAlNsytfkkamWWcqHslhLbXFCBmBdkxFGpOKoqK/EVwWoOUdRZ28VOOWSpSe5gE1wBZGiCWRhRxw0YcbD7N7kbwWHwAI4=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6eae8154-4292-452c-2873-08dc16d706e7
X-MS-Exchange-CrossTenant-AuthSource: CO1PR10MB4418.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jan 2024 21:06:37.2020
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fvBB3NeFok/zjAGiDiUxfQWRfb1BrFLUUKxjnigVNBIptpgd+U+2jSHcctcLFrcX8jGqE0IrcI7rpP8Zt4I5yw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB7261
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-01-16_12,2024-01-16_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 adultscore=0 phishscore=0
 mlxlogscore=999 suspectscore=0 bulkscore=0 mlxscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2401160166
X-Proofpoint-ORIG-GUID: B5iHWVGiMJzcz8orXyfLMWoFsmopzjYz
X-Proofpoint-GUID: B5iHWVGiMJzcz8orXyfLMWoFsmopzjYz

On Tue, Jan 16, 2024 1:08:20PM +0000, Mark Rutland wrote:
> On Mon, Jan 15, 2024 at 03:39:00PM +0000, Russell King (Oracle) wrote:
> > On Thu, Dec 14, 2023 at 04:05:56PM -0800, Christoph Lameter (Ampere) wrote:
> > > +# Determines the placement of cpumasks.
> > > +#
> > > +# With CPUMASK_OFFSTACK the cpumasks are dynamically allocated.
> > > +# Useful for machines with lots of core because it avoids increasing
> > > +# the size of many of the data structures in the kernel.
> > > +#
> > > +# If this is off then the cpumasks have a static sizes and are
> > > +# embedded within data structures.
> > > +#
> > > +config CPUMASK_OFFSTACK
> > > +	def_bool y
> > > +	depends on NR_CPUS > 256
> > 
> > Should that be ">= 256" ?
> 
> I don't think that ">= 256" makes sense. Note that since the cpumasks are
> arrays of unsigned long, they're chunked into groups of 64 bits:
> 
>     2 to  64 cpus:  1 x unsigned long =>  8 bytes
>    65 to 128 cpus:  2 x unsigned long => 16 bytes
>   129 to 192 cpus:  3 x unsigned long => 24 bytes
>   193 to 256 cpus:  4 x unsigned long => 32 bytes
>   257 to 320 cpus:  5 x unsigned long => 40 bytes
> 
> ... and so if a mask for 256 CPUs is too big to go in the stack, so is any mask
> for 193+ CPUs, and so ">= 256" should be clamped down to ">= 193" or "> 192".
> The boundary should be just after a multiple of 64.
> 
> How did we choose 256 specifically? I note that x86-64 allows 512 CPUs before
> requiring CPUMASK_OFFSTACK, and I see that powerpc selects CPUMASK_OFFSTACK
> when NR_CPUS >= 8192.
> 
> Mark.

The suggestion for >= 256 may have been a zero-index/one-index mixup.

It seems > 256 was chosen as the cutoff simply because it preserves existing behavior.
The patch description seems to imply there was pushback from distro maintainers on just increasing
the default NR_CPUS.

The existing default value of 256 is probably already a strain for smaller ARM systems, to which
x86-64 isn't a reasonable comparison. I'm not sure what the reaction to increasing from 64 to 256
in 2019 was like, but picking a pivot point for CPUMASK_OFFSTACK beyond 256 may skew the balance
even less in favor of smaller ARM systems.