Received-SPF: pass (google.com: domain of linux-kernel+bounces-71783-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
From: Michael Kelley <mhklinux@outlook.com>
To: Saurabh Singh Sengar <ssengar@linux.microsoft.com>
CC: "kys@microsoft.com" <kys@microsoft.com>, "haiyangz@microsoft.com"
	<haiyangz@microsoft.com>, "wei.liu@kernel.org" <wei.liu@kernel.org>,
	"decui@microsoft.com" <decui@microsoft.com>, "linux-hyperv@vger.kernel.org"
	<linux-hyperv@vger.kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "ssengar@microsoft.com"
	<ssengar@microsoft.com>
Subject: RE: [PATCH] Drivers: hv: Kconfig: select CPUMASK_OFFSTACK for Hyper-V
Thread-Topic: [PATCH] Drivers: hv: Kconfig: select CPUMASK_OFFSTACK for
 Hyper-V
Thread-Index: AQHaYOHvK5zyzBkwkUeCyKTO1/OqarEOuT2wgAD6DICAAj61sA==
Date: Mon, 19 Feb 2024 17:53:25 +0000
Message-ID:
 <SN6PR02MB41573212FCDBDF58305881ECD4512@SN6PR02MB4157.namprd02.prod.outlook.com>
References: <1708092603-14504-1-git-send-email-ssengar@linux.microsoft.com>
 <SN6PR02MB415758508A7BE89E0CFBD976D4532@SN6PR02MB4157.namprd02.prod.outlook.com>
 <20240218071727.GA19702@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
In-Reply-To:
 <20240218071727.GA19702@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?us-ascii?Q?8ba9BTlYh2H/rm8GLM2izt93X56BWrl3aeJmTZWCznrFH5M8CzbWnlAcg5n/?=
 =?us-ascii?Q?GRl3QNFyrrIroIr+Ldy2BIAvZ1VeHustxoURW07ou9exI+/JO9GFulaUZkzK?=
 =?us-ascii?Q?TdrBh1h/o/jW6NdsxwC03SvRS09ZAC8rLC8iekhVM1WZlHkbWdG5UMGVuEJJ?=
 =?us-ascii?Q?HJSkXlSBIj/KplTZZwB9SGW5ORfBDVZgeY69I56xZtYXe6Xb96CHSEPEC4E1?=
 =?us-ascii?Q?EHprSJU50ZZixOt/kbtW2ySUxdxnkU48qKCJ76Zi/DXqpUYIi7aMPV13JhKK?=
 =?us-ascii?Q?1vk1zfv10IoFoizsuzlgkDdhAgBizLF3UNuTVcjgxmKCDta1TOwPg4rs+ysT?=
 =?us-ascii?Q?lLb5QHdUV1umTSvwIpeqgQYCA3bSc0KCnh91Yml6AL+DFiTSJAAEl70G4lnR?=
 =?us-ascii?Q?QW/TymrHXUJAkoZff9PNn1HnSJINzsxAoOchJXMsPpIKEI0/HTwdKkhkwhRk?=
 =?us-ascii?Q?7PClv2BUx+aldBld8mKXRpcVWBN2VnRLOR225sMQYxaB8apd9GnD6IC3rPsY?=
 =?us-ascii?Q?Cy+H+MHslVZueFryznH0Ci2nvjfqhP4URenjneoiKkxBgbJDV8/xczuD1mn5?=
 =?us-ascii?Q?tidnI/LVFqSAzg4ujcgQY7yZY8ANUK0GazhcBXx50ykFUCIg2N//XVMp5GIw?=
 =?us-ascii?Q?6UIeZ2PS2KJJk/inul7WXENecorp0sZI412nj/+p4fy1sF+RTETKz/aQ5WVB?=
 =?us-ascii?Q?casgq5HmaMCcORe5phf+yQFanLPPz4ASDTB2aBDDCSkl7Bxw/E1DVZBEV9iF?=
 =?us-ascii?Q?w2rYNdP+6PB9naQC3W9UB7Jmuv1Hr9nl8ID1D6Bde0/Wpl69sW4PXtQ2dODH?=
 =?us-ascii?Q?AV8G5pS05ejmUqfe1gybmPbbAj00GeLID0226tyXiKXhF9dKTTFbN0GtT7nz?=
 =?us-ascii?Q?tZ45Zdf++swP39oL1L525PyGppVMjhavs5iGugtWaXKTRJKski+r4pPZb4Jz?=
 =?us-ascii?Q?XF1E9KkXSq6Fj9Y+NOtabbeQF37XA+a6OlHvwWZy9SZvL/DBUF/AKe6b/dfc?=
 =?us-ascii?Q?0aTbH/6QfrAAmam0tlp8i6LYs3aWi7ULdwojdA2MD3hdPadGBWV2x8YgqlwM?=
 =?us-ascii?Q?Ig6yhItqcPprp8xA6K4aWoCQxOP0PsYmzqmWYZKB8Hkp3fcCmsL3YHK0Sfn6?=
 =?us-ascii?Q?DTu0tcpu8MnRW9jSpqy3VYVYGsP/eU/WWrHAhP88jmu3yqO+26p8m80D5uej?=
 =?us-ascii?Q?XK7wcB7ZbvyvfNHgLMpCE+WjA0C2UHslSgMCTA=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-OriginatorOrg: outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SN6PR02MB4157.namprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-CrossTenant-Network-Message-Id: beca78f6-3bf0-4815-4fdc-08dc3173abe6
X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Feb 2024 17:53:25.4372
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR02MB7197

From: Saurabh Singh Sengar <ssengar@linux.microsoft.com> Sent: Saturday, Fe=
bruary 17, 2024 11:17 PM
> > >
> > > diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig
> > > index 0024210..bc3f496 100644
> > > --- a/drivers/hv/Kconfig
> > > +++ b/drivers/hv/Kconfig
> > > @@ -9,6 +9,7 @@ config HYPERV
> > >  	select PARAVIRT
> > >  	select X86_HV_CALLBACK_VECTOR if X86
> > >  	select OF_EARLY_FLATTREE if OF
> > > +	select CPUMASK_OFFSTACK
> > >  	help
> > >  	  Select this option to run Linux as a Hyper-V client operating
> > >  	  system.
> > > --
> > > 1.8.3.1
> > >
> >
> > I'm not sure that enabling CPUMASK_OFFSTACK for Hyper-V
> > guests is the right thing to do, as there's additional runtime
> > cost when CPUMASK_OFFSTACK is enabled.  I agree that for
> > the most general case, you want NR_CPUS to be 2048, which
> > requires CPUMASK_OFFSTACK.  But it would be legitimate to
> > build a kernel with NR_CPUS set to something like 64 or 256
> > for a more limited Hyper-V guest use case, and to not want to
> > incur the cost of CPUMASK_OFFSTACK.
> >
> > You could consider doing something like this:
> >
> > 	select CPUMASK_OFFSTACK if NR_CPUS > 512
>=20
> Thanks for your review.
>=20
> This was my first thought as well, but for x86, NR_CPUS itself depends
> on CPUMASK_OFFSTACK and this creates some kind of circular dependency
> and doesn't work effectively.
>=20
> Here are few key points to note:
>=20
> 1. In ARM64 as well for enabling CPUMASK_OFFSTACK we need to enable
>    DEBUG_PER_CPU_MAPS and that will have additional overhead.
>    This dependency is for all the archs. There was an earlier attempt
>    to decouple it: https://lore.kernel.org/lkml/20220412231508.32629-1-li=
bo.chen@oracle.com/=20
>=20
> 2. However, for ARM64, NR_CPUS doesn't have dependency on CPUMASK_OFFSTAC=
K.
>    In ARM64 NR_CPUS is quite independent from any policy, we can choose a=
ny
>    value for NR_CPUS freely, things are simple. This problem specificaly
>    to be solved for x86.
>=20
> 3. If we have to select more then 512 CPUs on x86, CPUMASK_OFFSTACK
>    needto be enabled, so this additional runtime cost is unavoidable
>    for NR_CPUS > 512. There is no way today to enable CPUMASK_OFFSTACK
>    apart from enabling MAXSMP or DEBUG_PER_CPU_MAPS. Both of these
>    options we don't want to use.
>=20
> I agree that we possibly don't want to enable this option for HyperV VMs
> where NR_CPUS < 512. I have two thoughts here:
>=20
> 1. Enable it only for VTL platforms, as current requirement for minimal k=
ernel
>    is only for VTL platforms only.
>=20
> 2. Fix this for all of x86. I couldn't find any reson why CPUMASK_OFFSTAC=
K
>    dependency is there on x86 for having more than 512 CPUs. What is spec=
ial
>    in x86 to have this restriction ? If there is no reason we should rela=
x
>    the restriction of CPUMASK_OFFSTACK for NR_CPUs similar to ARM and oth=
er
>    archs.
>=20

You've done some deeper research than I did. :-(  What a mess.

ARM64 seems to have it right.  On x86, the dependency between NR_CPUS
and CPUMASK_OFFSTACK seems to flow the wrong direction. I would think
you would select NR_CPUS first, and then if the number is large, select
CPUMASK_OFFSTACK.

And the display of CPUMASK_OFFSTACK in config tools should not be
dependent on DEBUG_PER_CPU_MAPS.   It should be easy to independently
select CPUMASK_OFFSTACK (modulo architectures that don't support it).
In the Libo Chen thread, I don't understand the reluctance to make
CPUMASK_OFFSTACK independent of DEBUG_PER_CPU_MAPS.

I don't have any great suggestions for the path forward. :-(  Maybe
revive the Libo Chen thread, with a better justification for removing
the dependency between CPUMASK_OFFSTACK and
DEBUG_PER_CPU_MAPS?  Or at least clarify why the dependency
should be kept?

Michael