From:   Nathan Lynch <nathanl@linux.ibm.com>
To:     Laurent Dufour <ldufour@linux.ibm.com>, mpe@ellerman.id.au,
        npiggin@gmail.com, christophe.leroy@csgroup.eu,
        Srikar Dronamraju <srikar@linux.ibm.com>
Cc:     linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org,
        Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Subject: Re: [PATCH] powerpc/pseries/cpuhp: respect current SMT when adding
 new CPU
In-Reply-To: <20230213124510.12651-1-ldufour@linux.ibm.com>
References: <20230213124510.12651-1-ldufour@linux.ibm.com>
Date:   Mon, 13 Feb 2023 08:46:50 -0600
Message-ID: <87ilg5aahx.fsf@linux.ibm.com>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: bulk

Laurent Dufour <ldufour@linux.ibm.com> writes:
> When a new CPU is added, the kernel is activating all its threads. This
> leads to weird, but functional, result when adding CPU on a SMT 4 system
> for instance.
>
> Here the newly added CPU 1 has 8 threads while the other one has 4 threads
> active (system has been booted with the 'smt-enabled=4' kernel option):
>
> ltcden3-lp12:~ # ppc64_cpu --info
> Core   0:    0*    1*    2*    3*    4     5     6     7
> Core   1:    8*    9*   10*   11*   12*   13*   14*   15*
>
> There is no SMT value in the kernel. It is possible to run unbalanced LPAR
> with 2 threads for a CPU, 4 for another one, and 5 on the latest.
>
> To work around this possibility, and assuming that the LPAR run with the
> same number of threads for each CPU, which is the common case,

I am skeptical at best of baking that assumption into this code. Mixed
SMT modes within a partition doesn't strike me as an unreasonable
possibility for some use cases. And if that's wrong, then we should just
add a global smt value instead of using heuristics.

> the number
> of active threads of the CPU doing the hot-plug operation is computed. Only
> that number of threads will be activated for the newly added CPU.
>
> This way on a LPAR running in SMT=4, newly added CPU will be running 4
> threads, which is what a end user would expect.

I could see why most users would prefer this new behavior. But surely
some users have come to expect the existing behavior, which has been in
place for years, and developed workarounds that might be broken by this
change?

I would suggest that to handle this well, we need to give user space
more ability to tell the kernel what actions to take on added cores, on
an opt-in basis.

This could take the form of extending the DLPAR sysfs command set:

Option 1 - Add a flag that tells the kernel not to online any threads at
all; user space will online the desired threads later.

Option 2 - Add an option that tells the kernel which SMT mode to apply.