Aarch32 compatibility mode is enabled at compile time through
CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support
(for both processes and syscalls) be enabled at boot time using
a kernel parameter. Also, it provides a mean for distributions
to set their own default without sacrificing compatibility support,
that is users can override default behaviour through the kernel
parameter.
*** Notes about syscall management ***
VBAR_EL1 register, which holds the exception table address,
is setup very early in the boot process, before parse_early_param().
This means that it's not possible to access boot parameter before
setting the register. Also, setting the aforementioned register
for secondary cpus is done later in the boot flow.
Several ways to work around this has been considered, among which:
* resetting VBAR_EL1 to point to one of two vector tables (the
former with 32-bit exceptions handler enabled and the latter
pointing to unhandled stub, just as if CONFIG_COMPAT is enabled)
depending on the proposed boot parameter. This has the disadvantage
to produce a somewhat messy patchset involving several lines,
has higher cognitive load since there are at least three places
where the register is getting changed (not near to each other),
and have implications on other code segments (namely kpti, kvm
and vdso), requiring special care.
* patching the vector table contents once the early param is available.
This has most of the implications of the previous option
(except maybe not impacting other code segments), plus it sounds
a little 'hackish'.
The chosen approach involves conditional executing 32-bit syscalls
depending on the parameter value. This of course results in a
little performance loss, but has the following advantages:
* all the cons from previously explained alternatives are solved
* users of 32-bit apps on 64-bit kernel are already suffering from
performance losses due to 32-bit apps not fully leveraging the 64-bit
processor, so they are already aware of this
* users of 32-bit apps on 64-bit kernel are believed
to be a minority and most of the time there are sources available
to be recompiled for 64-bit as a workaround for better performance
It worth mentioning that users of 64-bit apps are, of course,
unaffected.
Based on the work from Nikolay Borisov, see:
Link: https://lkml.org/lkml/2023/6/23/387
Andrea della Porta (4):
arm64: Introduce aarch32_enabled()
arm64/process: Make loading of 32bit processes depend on
aarch32_enabled()
arm64/entry-common: Make Aarch32 syscalls' availability depend on
aarch32_enabled()
arm64: Make Aarch32 emulation boot time configurable
.../admin-guide/kernel-parameters.txt | 7 ++++
arch/arm64/Kconfig | 9 +++++
arch/arm64/include/asm/compat.h | 12 +++++++
arch/arm64/kernel/entry-common.c | 33 +++++++++++++++++--
arch/arm64/kernel/process.c | 2 +-
5 files changed, 59 insertions(+), 4 deletions(-)
--
2.35.3
Distributions would like to reduce their attack surface as much as
possible but at the same time they'd want to retain flexibility to
cater to a variety of legacy software. This stems from the conjecture
that compat layer is likely rarely tested and could have latent
security bugs. Ideally distributions will set their default policy
and also give users the ability to override it as appropriate.
To enable this use case, introduce CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED
compile time option, which controls whether 32bit processes/syscalls
should be allowed or not. This option is aimed mainly at distributions
to set their preferred default behavior in their kernels.
To allow users to override the distro's policy, introduce the
'aarch32_emulation' parameter which allows overriding
CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED state at boot time.
Signed-off-by: Andrea della Porta <[email protected]>
---
Documentation/admin-guide/kernel-parameters.txt | 7 +++++++
arch/arm64/Kconfig | 9 +++++++++
arch/arm64/kernel/entry-common.c | 8 +++++++-
3 files changed, 23 insertions(+), 1 deletion(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0a1731a0f0ef..a41c5e6f5d2e 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -1,3 +1,10 @@
+ aarch32_emulation= [ARM64]
+ Format: <bool>
+ When true, allows loading 32-bit programs and executing
+ 32-bit syscalls, essentially overriding
+ AARCH32_EMULATION_DEFAULT_DISABLED at boot time. when false,
+ unconditionally disables AARCH32 emulation.
+
acpi= [HW,ACPI,X86,ARM64,RISCV64]
Advanced Configuration and Power Interface
Format: { force | on | off | strict | noirq | rsdt |
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b10515c0200b..66c4cb273550 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1725,6 +1725,15 @@ config SETEND_EMULATION
If unsure, say Y
endif # ARMV8_DEPRECATED
+config AARCH32_EMULATION_DEFAULT_DISABLED
+ bool "Aarch32 emulation disabled by default"
+ default n
+ depends on COMPAT
+ help
+ Make Aarch32 emulation disabled by default. This prevents loading 32-bit
+ processes and access to 32-bit syscalls.
+
+ If unsure, leave it to its default value.
endif # COMPAT
menu "ARMv8.1 architectural features"
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 32761760d9dd..07f2b4e632b8 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -897,7 +897,13 @@ asmlinkage void noinstr el0t_32_error_handler(struct pt_regs *regs)
__el0_error_handler_common(regs);
}
-bool __aarch32_enabled __ro_after_init = true;
+bool __aarch32_enabled __ro_after_init = !IS_ENABLED(CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED);
+
+static int aarch32_emulation_override_cmdline(char *arg)
+{
+ return kstrtobool(arg, &__aarch32_enabled);
+}
+early_param("aarch32_emulation", aarch32_emulation_override_cmdline);
#else /* CONFIG_COMPAT */
UNHANDLED(el0t, 32, sync)
UNHANDLED(el0t, 32, irq)
--
2.35.3
Major aspect of Aarch32 emulation is the ability to load 32bit
processes.
That's currently decided (among others) by compat_elf_check_arch().
Make the macro use aarch32_enabled() to decide if Aarch32 compat is
enabled before loading a 32bit process.
Signed-off-by: Andrea della Porta <[email protected]>
---
arch/arm64/kernel/process.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 657ea273c0f9..96832f1ec3ee 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -601,7 +601,7 @@ unsigned long arch_align_stack(unsigned long sp)
#ifdef CONFIG_COMPAT
int compat_elf_check_arch(const struct elf32_hdr *hdr)
{
- if (!system_supports_32bit_el0())
+ if (!system_supports_32bit_el0() || !aarch32_enabled())
return false;
if ((hdr)->e_machine != EM_ARM)
--
2.35.3
Another major aspect of supporting running of 32bit processes is the
ability to access 32bit syscalls. Such syscalls can be invoked by
using the svc instruction.
If Aarch32 emulation is disabled ensure that calling svc results
in the same behavior as if CONFIG_COMPAT has not been enabled (i.e.
a kernel panic).
Signed-off-by: Andrea della Porta <[email protected]>
---
arch/arm64/kernel/entry-common.c | 25 ++++++++++++++++++++++---
1 file changed, 22 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 69ff9b8c0bde..32761760d9dd 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -802,6 +802,11 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
}
#ifdef CONFIG_COMPAT
+UNHANDLED(el0t, 32, sync_ni)
+UNHANDLED(el0t, 32, irq_ni)
+UNHANDLED(el0t, 32, fiq_ni)
+UNHANDLED(el0t, 32, error_ni)
+
static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
{
enter_from_user_mode(regs);
@@ -821,6 +826,11 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
{
+ if (!aarch32_enabled()) {
+ el0t_32_sync_ni_handler(regs);
+ return;
+ }
+
unsigned long esr = read_sysreg(esr_el1);
switch (ESR_ELx_EC(esr)) {
@@ -865,17 +875,26 @@ asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
asmlinkage void noinstr el0t_32_irq_handler(struct pt_regs *regs)
{
- __el0_irq_handler_common(regs);
+ if (!aarch32_enabled())
+ el0t_32_irq_ni_handler(regs);
+ else
+ __el0_irq_handler_common(regs);
}
asmlinkage void noinstr el0t_32_fiq_handler(struct pt_regs *regs)
{
- __el0_fiq_handler_common(regs);
+ if (!aarch32_enabled())
+ el0t_32_fiq_ni_handler(regs);
+ else
+ __el0_fiq_handler_common(regs);
}
asmlinkage void noinstr el0t_32_error_handler(struct pt_regs *regs)
{
- __el0_error_handler_common(regs);
+ if (!aarch32_enabled())
+ el0t_32_error_ni_handler(regs);
+ else
+ __el0_error_handler_common(regs);
}
bool __aarch32_enabled __ro_after_init = true;
--
2.35.3
Hi,
On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote:
> Aarch32 compatibility mode is enabled at compile time through
> CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support
> (for both processes and syscalls) be enabled at boot time using
> a kernel parameter. Also, it provides a mean for distributions
> to set their own default without sacrificing compatibility support,
> that is users can override default behaviour through the kernel
> parameter.
I proposed something similar in the past:
https://lkml.kernel.org/linux-fsdevel/[email protected]/
bu the conclusion there (see the reply from Kees) was that it was better
to either use existing seccomp mechanisms or add something to control
which binfmts can be loaded.
Will
On Wed, Oct 18, 2023, at 14:27, Will Deacon wrote:
> Hi,
>
> On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote:
>> Aarch32 compatibility mode is enabled at compile time through
>> CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support
>> (for both processes and syscalls) be enabled at boot time using
>> a kernel parameter. Also, it provides a mean for distributions
>> to set their own default without sacrificing compatibility support,
>> that is users can override default behaviour through the kernel
>> parameter.
>
> I proposed something similar in the past:
>
> https://lkml.kernel.org/linux-fsdevel/[email protected]/
>
> bu the conclusion there (see the reply from Kees) was that it was better
> to either use existing seccomp mechanisms or add something to control
> which binfmts can be loaded.
Right, I was going to reply along the same lines here: x86 is
a bit of a special case that needs this, but I believe all the
other architectures already guard the compat syscall execution
on test_thread_flag(TIF_32BIT) that is only set by the compat
binfmt loader.
Doing the reverse is something that has however come up in the
past several times and that could be interesting: In order to
run userspace emulation (qemu-user, fex, ...) we may want to
allow calling syscalls and ioctls for foreign ABIs in a native
task, and at that point having a mechanism to control this
capability globally or per task would be useful as well.
The compat mode (arm32 on arm64) is the easiest case here, but the
same thing could be done for emulating the very subtle architecture
differences (x86-64 on arm64, arm64 on x86_64, arm32 on x86-compat,
or any of the above on riscv or loongarch).
Arnd
On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote:
> Aarch32 compatibility mode is enabled at compile time through
> CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support
> (for both processes and syscalls) be enabled at boot time using
> a kernel parameter. Also, it provides a mean for distributions
> to set their own default without sacrificing compatibility support,
> that is users can override default behaviour through the kernel
> parameter.
Can you elaborate on *why* people want such a policy?
> *** Notes about syscall management ***
> VBAR_EL1 register, which holds the exception table address,
> is setup very early in the boot process, before parse_early_param().
> This means that it's not possible to access boot parameter before
> setting the register. Also, setting the aforementioned register
> for secondary cpus is done later in the boot flow.
> Several ways to work around this has been considered, among which:
>
> * resetting VBAR_EL1 to point to one of two vector tables (the
> former with 32-bit exceptions handler enabled and the latter
> pointing to unhandled stub, just as if CONFIG_COMPAT is enabled)
> depending on the proposed boot parameter. This has the disadvantage
> to produce a somewhat messy patchset involving several lines,
> has higher cognitive load since there are at least three places
> where the register is getting changed (not near to each other),
> and have implications on other code segments (namely kpti, kvm
> and vdso), requiring special care.
>
> * patching the vector table contents once the early param is available.
> This has most of the implications of the previous option
> (except maybe not impacting other code segments), plus it sounds
> a little 'hackish'.
>
> The chosen approach involves conditional executing 32-bit syscalls
> depending on the parameter value.
Why does the compat syscall path need to do anything?
On arm64 it's not possible to issue compat syscalls from a native 64-bit task.
If you prevent the loading of AArch32 binaries, none of the compat syscalls
will be reachable at all.
That's the proper way to implement this, and we already have logic for that as
part of the mismatched AArch32 support.
> This of course results in a little performance loss, but has the following
> advantages:
A performance loss for what relative to what?
How much of a performance loss?
Mark.
> * all the cons from previously explained alternatives are solved
> * users of 32-bit apps on 64-bit kernel are already suffering from
> performance losses due to 32-bit apps not fully leveraging the 64-bit
> processor, so they are already aware of this
> * users of 32-bit apps on 64-bit kernel are believed
> to be a minority and most of the time there are sources available
> to be recompiled for 64-bit as a workaround for better performance
>
> It worth mentioning that users of 64-bit apps are, of course,
> unaffected.
>
> Based on the work from Nikolay Borisov, see:
> Link: https://lkml.org/lkml/2023/6/23/387
>
> Andrea della Porta (4):
> arm64: Introduce aarch32_enabled()
> arm64/process: Make loading of 32bit processes depend on
> aarch32_enabled()
> arm64/entry-common: Make Aarch32 syscalls' availability depend on
> aarch32_enabled()
> arm64: Make Aarch32 emulation boot time configurable
>
> .../admin-guide/kernel-parameters.txt | 7 ++++
> arch/arm64/Kconfig | 9 +++++
> arch/arm64/include/asm/compat.h | 12 +++++++
> arch/arm64/kernel/entry-common.c | 33 +++++++++++++++++--
> arch/arm64/kernel/process.c | 2 +-
> 5 files changed, 59 insertions(+), 4 deletions(-)
>
> --
> 2.35.3
>
On Wed, Oct 18, 2023 at 01:13:20PM +0200, Andrea della Porta wrote:
> Major aspect of Aarch32 emulation is the ability to load 32bit
> processes.
> That's currently decided (among others) by compat_elf_check_arch().
>
> Make the macro use aarch32_enabled() to decide if Aarch32 compat is
> enabled before loading a 32bit process.
>
> Signed-off-by: Andrea della Porta <[email protected]>
Why can't you make system_supports_32bit_el0() take the option into account
instead?
Mark.
> ---
> arch/arm64/kernel/process.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
> index 657ea273c0f9..96832f1ec3ee 100644
> --- a/arch/arm64/kernel/process.c
> +++ b/arch/arm64/kernel/process.c
> @@ -601,7 +601,7 @@ unsigned long arch_align_stack(unsigned long sp)
> #ifdef CONFIG_COMPAT
> int compat_elf_check_arch(const struct elf32_hdr *hdr)
> {
> - if (!system_supports_32bit_el0())
> + if (!system_supports_32bit_el0() || !aarch32_enabled())
> return false;
>
> if ((hdr)->e_machine != EM_ARM)
> --
> 2.35.3
>
On Wed, Oct 18, 2023 at 01:13:21PM +0200, Andrea della Porta wrote:
> Another major aspect of supporting running of 32bit processes is the
> ability to access 32bit syscalls. Such syscalls can be invoked by
> using the svc instruction.
>
> If Aarch32 emulation is disabled ensure that calling svc results
> in the same behavior as if CONFIG_COMPAT has not been enabled (i.e.
> a kernel panic).
It's not "emulation" it's directly supported by the hardware.
>
> Signed-off-by: Andrea della Porta <[email protected]>
> ---
> arch/arm64/kernel/entry-common.c | 25 ++++++++++++++++++++++---
> 1 file changed, 22 insertions(+), 3 deletions(-)
>
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index 69ff9b8c0bde..32761760d9dd 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -802,6 +802,11 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
> }
>
> #ifdef CONFIG_COMPAT
> +UNHANDLED(el0t, 32, sync_ni)
> +UNHANDLED(el0t, 32, irq_ni)
> +UNHANDLED(el0t, 32, fiq_ni)
> +UNHANDLED(el0t, 32, error_ni)
IRQ, FIQ, and SError are not syscalls, so the commit title is bad.
> +
> static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
> {
> enter_from_user_mode(regs);
> @@ -821,6 +826,11 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
>
> asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
> {
> + if (!aarch32_enabled()) {
> + el0t_32_sync_ni_handler(regs);
> + return;
> + }
Why do we have to do this at all?
If we don't have AArch32 tasks, these paths are unreachable. Why do we need to
check that they aren't called?
Mark.
> +
> unsigned long esr = read_sysreg(esr_el1);
>
> switch (ESR_ELx_EC(esr)) {
> @@ -865,17 +875,26 @@ asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
>
> asmlinkage void noinstr el0t_32_irq_handler(struct pt_regs *regs)
> {
> - __el0_irq_handler_common(regs);
> + if (!aarch32_enabled())
> + el0t_32_irq_ni_handler(regs);
> + else
> + __el0_irq_handler_common(regs);
> }
>
> asmlinkage void noinstr el0t_32_fiq_handler(struct pt_regs *regs)
> {
> - __el0_fiq_handler_common(regs);
> + if (!aarch32_enabled())
> + el0t_32_fiq_ni_handler(regs);
> + else
> + __el0_fiq_handler_common(regs);
> }
>
> asmlinkage void noinstr el0t_32_error_handler(struct pt_regs *regs)
> {
> - __el0_error_handler_common(regs);
> + if (!aarch32_enabled())
> + el0t_32_error_ni_handler(regs);
> + else
> + __el0_error_handler_common(regs);
> }
>
> bool __aarch32_enabled __ro_after_init = true;
> --
> 2.35.3
>
On Wed, Oct 18, 2023 at 01:13:22PM +0200, Andrea della Porta wrote:
> Distributions would like to reduce their attack surface as much as
> possible but at the same time they'd want to retain flexibility to
> cater to a variety of legacy software. This stems from the conjecture
> that compat layer is likely rarely tested and could have latent
> security bugs. Ideally distributions will set their default policy
> and also give users the ability to override it as appropriate.
>
> To enable this use case, introduce CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED
> compile time option, which controls whether 32bit processes/syscalls
> should be allowed or not. This option is aimed mainly at distributions
> to set their preferred default behavior in their kernels.
>
> To allow users to override the distro's policy, introduce the
> 'aarch32_emulation' parameter which allows overriding
> CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED state at boot time.
>
> Signed-off-by: Andrea della Porta <[email protected]>
> ---
> Documentation/admin-guide/kernel-parameters.txt | 7 +++++++
> arch/arm64/Kconfig | 9 +++++++++
> arch/arm64/kernel/entry-common.c | 8 +++++++-
> 3 files changed, 23 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 0a1731a0f0ef..a41c5e6f5d2e 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -1,3 +1,10 @@
> + aarch32_emulation= [ARM64]
> + Format: <bool>
> + When true, allows loading 32-bit programs and executing
> + 32-bit syscalls, essentially overriding
> + AARCH32_EMULATION_DEFAULT_DISABLED at boot time. when false,
> + unconditionally disables AARCH32 emulation.
Can we please drop the 'emulation' part of the name? We don't use that
terminology on arm64 for regular execution of compat tasks, and only use that
to refer to true emulation of deprecated instructions.
We already have the 'allow_mismatched_32bit_el0' option; can we please us a
name that aligns with that? e.g. 'allow_32bit_el0=false' to disable support.
Mark.
> +
> acpi= [HW,ACPI,X86,ARM64,RISCV64]
> Advanced Configuration and Power Interface
> Format: { force | on | off | strict | noirq | rsdt |
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index b10515c0200b..66c4cb273550 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -1725,6 +1725,15 @@ config SETEND_EMULATION
> If unsure, say Y
> endif # ARMV8_DEPRECATED
>
> +config AARCH32_EMULATION_DEFAULT_DISABLED
> + bool "Aarch32 emulation disabled by default"
> + default n
> + depends on COMPAT
> + help
> + Make Aarch32 emulation disabled by default. This prevents loading 32-bit
> + processes and access to 32-bit syscalls.
> +
> + If unsure, leave it to its default value.
> endif # COMPAT
>
> menu "ARMv8.1 architectural features"
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index 32761760d9dd..07f2b4e632b8 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -897,7 +897,13 @@ asmlinkage void noinstr el0t_32_error_handler(struct pt_regs *regs)
> __el0_error_handler_common(regs);
> }
>
> -bool __aarch32_enabled __ro_after_init = true;
> +bool __aarch32_enabled __ro_after_init = !IS_ENABLED(CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED);
> +
> +static int aarch32_emulation_override_cmdline(char *arg)
> +{
> + return kstrtobool(arg, &__aarch32_enabled);
> +}
> +early_param("aarch32_emulation", aarch32_emulation_override_cmdline);
> #else /* CONFIG_COMPAT */
> UNHANDLED(el0t, 32, sync)
> UNHANDLED(el0t, 32, irq)
> --
> 2.35.3
>
On 13:27 Wed 18 Oct , Will Deacon wrote:
> Hi,
>
> On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote:
> > Aarch32 compatibility mode is enabled at compile time through
> > CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support
> > (for both processes and syscalls) be enabled at boot time using
> > a kernel parameter. Also, it provides a mean for distributions
> > to set their own default without sacrificing compatibility support,
> > that is users can override default behaviour through the kernel
> > parameter.
>
> I proposed something similar in the past:
>
> https://lkml.kernel.org/linux-fsdevel/[email protected]/
>
> bu the conclusion there (see the reply from Kees) was that it was better
> to either use existing seccomp mechanisms or add something to control
> which binfmts can be loaded.
>
> Will
I see. Seccomp sounds like a really good idea, since just blocking the compat
binfmt would not avoid the call to 32-bit syscalls per se: it's true that
ARM64 enforce the transition from A64 to A32 only on exception return and
PSTATE.nRW flag can change only from EL1, maybe though some exploitation
may arise in the future to do just that (I'm not aware of any or come up
with a proof off the top of my head, but I can't exclude it either). So,
assuming by absurd a switch to A32 is feasible, the further step of embedding
A32 instruction in a A64 ELF executable is a breeze. Hence blocking the
syscall (and not only the binfmt loading) could prove necessary. I know all
of this is higly speculative right now, maybe it's worth thinking nonetheless.
Andrea
On 14:44 Wed 18 Oct , Arnd Bergmann wrote:
> On Wed, Oct 18, 2023, at 14:27, Will Deacon wrote:
> > Hi,
> >
> > On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote:
> >> Aarch32 compatibility mode is enabled at compile time through
> >> CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support
> >> (for both processes and syscalls) be enabled at boot time using
> >> a kernel parameter. Also, it provides a mean for distributions
> >> to set their own default without sacrificing compatibility support,
> >> that is users can override default behaviour through the kernel
> >> parameter.
> >
> > I proposed something similar in the past:
> >
> > https://lkml.kernel.org/linux-fsdevel/[email protected]/
> >
> > bu the conclusion there (see the reply from Kees) was that it was better
> > to either use existing seccomp mechanisms or add something to control
> > which binfmts can be loaded.
>
> Right, I was going to reply along the same lines here: x86 is
> a bit of a special case that needs this, but I believe all the
> other architectures already guard the compat syscall execution
> on test_thread_flag(TIF_32BIT) that is only set by the compat
> binfmt loader.
Are you referring to the fact that x86 can switch at will between 32- and 64-
bit code?
Regarding the TIF_32BIT flag, thanks for the head-up. I still believe though
that this mechanism can somehow break down in the future, since prohibiting
32 bit executable loading *and* blocking 32 bit compat syscall are two
separate path of execution, held together by the architecture prohibiting
to switch to A32 instructions by design. Breaking the first rule and embedding
wisely crafted A32 instruction in an executable is easy, while the difficult
part is finding some 'reentrancy' to be able to do the execution state switch,
as pinted out in https://lore.kernel.org/lkml/ZTD0DAes-J-YQ2eu@apocalypse/.
I agree it's highly speculative and not something to be concerned right
now, it's just a head up, should the need arise in the future.
> Doing the reverse is something that has however come up in the
> past several times and that could be interesting: In order to
> run userspace emulation (qemu-user, fex, ...) we may want to
> allow calling syscalls and ioctls for foreign ABIs in a native
> task, and at that point having a mechanism to control this
> capability globally or per task would be useful as well.
>
> The compat mode (arm32 on arm64) is the easiest case here, but the
> same thing could be done for emulating the very subtle architecture
> differences (x86-64 on arm64, arm64 on x86_64, arm32 on x86-compat,
> or any of the above on riscv or loongarch).
>
> Arnd
Really interesting, Since it's more related to emulation needs (my patch
has another focus due to the fact that A64 can execute A32 natively),
I'll take a look at this separately.
Andrea
On Thu, Oct 19, 2023, at 12:52, Andrea della Porta wrote:
> On 14:44 Wed 18 Oct , Arnd Bergmann wrote:
>> On Wed, Oct 18, 2023, at 14:27, Will Deacon wrote:
>>
>> Right, I was going to reply along the same lines here: x86 is
>> a bit of a special case that needs this, but I believe all the
>> other architectures already guard the compat syscall execution
>> on test_thread_flag(TIF_32BIT) that is only set by the compat
>> binfmt loader.
>
> Are you referring to the fact that x86 can switch at will between 32- and 64-
> bit code?
No.
> Regarding the TIF_32BIT flag, thanks for the head-up. I still believe though
> that this mechanism can somehow break down in the future, since prohibiting
> 32 bit executable loading *and* blocking 32 bit compat syscall are two
> separate path of execution, held together by the architecture prohibiting
> to switch to A32 instructions by design. Breaking the first rule and embedding
> wisely crafted A32 instruction in an executable is easy, while the difficult
> part is finding some 'reentrancy' to be able to do the execution state switch,
> as pinted out in https://lore.kernel.org/lkml/ZTD0DAes-J-YQ2eu@apocalypse/.
> I agree it's highly speculative and not something to be concerned right
> now, it's just a head up, should the need arise in the future.
There are (at least) five separate aspects to compat mode that are easy
to mix up:
1. Instruction decoding -- switching between the modes supported by the
CPU (A64/A32/T32)
2. Word size -- what happens to the upper 32 bits of a register in
an arithmetic operation
3. Personality -- Which architecture string gets returned by the
uname syscall (aarch64 vs armv8) as well as the format of
/proc/cpuinfo
4. system call entry points -- how a process calls into native or
compat syscalls, or possibly foreign OS emulation
5. Binary format -- elf32 vs elf64 executables
On most architectures with compat mode, 4. and 5. are fundamentally
tied together today: a compat task can only call compat syscalls
and a native task can only call native syscalls. x86 is the exception
here, as it uses different instructions (int80, syscall, sysenter)
and picks the syscall table based on that instruction.
I think 1. and 2. are also always tied to 5 on arm, but this is
not necessarily true for other architectures. 3. used to be tied
to 5 on some architectures in the past, but should be independent
now.
>> Doing the reverse is something that has however come up in the
>> past several times and that could be interesting: In order to
>> run userspace emulation (qemu-user, fex, ...) we may want to
>> allow calling syscalls and ioctls for foreign ABIs in a native
>> task, and at that point having a mechanism to control this
>> capability globally or per task would be useful as well.
>>
>> The compat mode (arm32 on arm64) is the easiest case here, but the
>> same thing could be done for emulating the very subtle architecture
>> differences (x86-64 on arm64, arm64 on x86_64, arm32 on x86-compat,
>> or any of the above on riscv or loongarch).
>
> Really interesting, Since it's more related to emulation needs (my patch
> has another focus due to the fact that A64 can execute A32 natively),
> I'll take a look at this separately.
A64 mode (unlike some other architectures, notably mips64) cannot
execute A32 or T32 instructions without a mode switch, the three are
entirely incompatible on the binary level.
Many ARMv8-CPUs support both Aarch64 mode and Aarch32 (A32/T32),
but a lot of the newer ones (e.g. Apple M1/M2, Cortex-R82 or
Cortex-A715) only do Aarch64 and need user-space emulation to run
32-bit binaries.
Arnd
On 13:52 Wed 18 Oct , Mark Rutland wrote:
> On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote:
> > Aarch32 compatibility mode is enabled at compile time through
> > CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support
> > (for both processes and syscalls) be enabled at boot time using
> > a kernel parameter. Also, it provides a mean for distributions
> > to set their own default without sacrificing compatibility support,
> > that is users can override default behaviour through the kernel
> > parameter.
>
> Can you elaborate on *why* people want such a policy?
>
Formerly, the reason was to reduce kernel attack surface by excluding
compat syscall, wherever applicable. Much less important but still a point,
I would also say this could be a good chance to get rid of somewhat old
and stale 32-bit libraries and programs, but this is of course debatable.
> > *** Notes about syscall management ***
> > VBAR_EL1 register, which holds the exception table address,
> > is setup very early in the boot process, before parse_early_param().
> > This means that it's not possible to access boot parameter before
> > setting the register. Also, setting the aforementioned register
> > for secondary cpus is done later in the boot flow.
> > Several ways to work around this has been considered, among which:
> >
> > * resetting VBAR_EL1 to point to one of two vector tables (the
> > former with 32-bit exceptions handler enabled and the latter
> > pointing to unhandled stub, just as if CONFIG_COMPAT is enabled)
> > depending on the proposed boot parameter. This has the disadvantage
> > to produce a somewhat messy patchset involving several lines,
> > has higher cognitive load since there are at least three places
> > where the register is getting changed (not near to each other),
> > and have implications on other code segments (namely kpti, kvm
> > and vdso), requiring special care.
> >
> > * patching the vector table contents once the early param is available.
> > This has most of the implications of the previous option
> > (except maybe not impacting other code segments), plus it sounds
> > a little 'hackish'.
> >
> > The chosen approach involves conditional executing 32-bit syscalls
> > depending on the parameter value.
>
> Why does the compat syscall path need to do anything?
I probably didn't catch your point here, compat syscall does not need to do
anything and they do not (just like they works right now with CONFIG_COMPAT
alone), except for the conditional instruction that excludes them at runtime.
Of course this conditional *is* doing something and somewhat redundant if
compat is disabled, but in this scenario I think it's unavoidable.
>
> On arm64 it's not possible to issue compat syscalls from a native 64-bit task.
> If you prevent the loading of AArch32 binaries, none of the compat syscalls
> will be reachable at all.
>
> That's the proper way to implement this, and we already have logic for that as
> part of the mismatched AArch32 support.
>
> > This of course results in a little performance loss, but has the following
> > advantages:
>
> A performance loss for what relative to what?
of a compat syscall as it is now enabling CONFIG_COMPAT vs the patched
syscall handlers that need a further conditional instruction to check
whether comapt is enabled or not.
>
> How much of a performance loss?
I did not take measurement yet since it was just a qualitative consideration
more than a quantitative one, also considering that chances are that it would
affect just very little population. The conditional instruction time taken
to execute is reasonably near to negligible if compared to any syscall execution.
On 13:52 Wed 18 Oct , Mark Rutland wrote:
> On Wed, Oct 18, 2023 at 01:13:20PM +0200, Andrea della Porta wrote:
> > Major aspect of Aarch32 emulation is the ability to load 32bit
> > processes.
> > That's currently decided (among others) by compat_elf_check_arch().
> >
> > Make the macro use aarch32_enabled() to decide if Aarch32 compat is
> > enabled before loading a 32bit process.
> >
> > Signed-off-by: Andrea della Porta <[email protected]>
>
> Why can't you make system_supports_32bit_el0() take the option into account
> instead?
>
I may be wrong here, but it seems to me that system_supports_32bit_el0()
answers teh question "can this system supports compat execution?" rather than
"do I want this system to run any compat execution?". That's the point of
aarch32_enabled(), to state whether we want teh system to run A32 code or not,
regardless of the system supporting it (of course, if the system does not
support A32 in EL0, this is a no-no, but that's another story).
Andrea
On 13:57 Wed 18 Oct , Mark Rutland wrote:
> On Wed, Oct 18, 2023 at 01:13:21PM +0200, Andrea della Porta wrote:
> > Another major aspect of supporting running of 32bit processes is the
> > ability to access 32bit syscalls. Such syscalls can be invoked by
> > using the svc instruction.
> >
> > If Aarch32 emulation is disabled ensure that calling svc results
> > in the same behavior as if CONFIG_COMPAT has not been enabled (i.e.
> > a kernel panic).
>
> It's not "emulation" it's directly supported by the hardware.
You're right. I also struggled to use this label but I just reused the same
name from Nikolai's patchset for x86, in the hope that the option would be
more recognizable (something like 'ARCH_emulation' that could be used maybe
for other platforms as well), but I agree with you that this is highly
misleading. I will change it to something more straightforward.
>
> >
> > Signed-off-by: Andrea della Porta <[email protected]>
> > ---
> > arch/arm64/kernel/entry-common.c | 25 ++++++++++++++++++++++---
> > 1 file changed, 22 insertions(+), 3 deletions(-)
> >
> > diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> > index 69ff9b8c0bde..32761760d9dd 100644
> > --- a/arch/arm64/kernel/entry-common.c
> > +++ b/arch/arm64/kernel/entry-common.c
> > @@ -802,6 +802,11 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
> > }
> >
> > #ifdef CONFIG_COMPAT
> > +UNHANDLED(el0t, 32, sync_ni)
> > +UNHANDLED(el0t, 32, irq_ni)
> > +UNHANDLED(el0t, 32, fiq_ni)
> > +UNHANDLED(el0t, 32, error_ni)
>
> IRQ, FIQ, and SError are not syscalls, so the commit title is bad.
Agreed. I'll call them exceptions.
>
> > +
> > static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
> > {
> > enter_from_user_mode(regs);
> > @@ -821,6 +826,11 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
> >
> > asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
> > {
> > + if (!aarch32_enabled()) {
> > + el0t_32_sync_ni_handler(regs);
> > + return;
> > + }
>
> Why do we have to do this at all?
>
> If we don't have AArch32 tasks, these paths are unreachable. Why do we need to
> check that they aren't called?
>
> Mark.
Agreed. Please see also my previous comments here:
https://lore.kernel.org/lkml/ZTEKabxNdegsbxyv@apocalypse/
https://lore.kernel.org/lkml/ZTD0DAes-J-YQ2eu@apocalypse/
but again, that's only speculative as of now, so we can ignore that part.
Andrea
On 14:02 Wed 18 Oct , Mark Rutland wrote:
> On Wed, Oct 18, 2023 at 01:13:22PM +0200, Andrea della Porta wrote:
> > Distributions would like to reduce their attack surface as much as
> > possible but at the same time they'd want to retain flexibility to
> > cater to a variety of legacy software. This stems from the conjecture
> > that compat layer is likely rarely tested and could have latent
> > security bugs. Ideally distributions will set their default policy
> > and also give users the ability to override it as appropriate.
> >
> > To enable this use case, introduce CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED
> > compile time option, which controls whether 32bit processes/syscalls
> > should be allowed or not. This option is aimed mainly at distributions
> > to set their preferred default behavior in their kernels.
> >
> > To allow users to override the distro's policy, introduce the
> > 'aarch32_emulation' parameter which allows overriding
> > CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED state at boot time.
> >
> > Signed-off-by: Andrea della Porta <[email protected]>
> > ---
> > Documentation/admin-guide/kernel-parameters.txt | 7 +++++++
> > arch/arm64/Kconfig | 9 +++++++++
> > arch/arm64/kernel/entry-common.c | 8 +++++++-
> > 3 files changed, 23 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > index 0a1731a0f0ef..a41c5e6f5d2e 100644
> > --- a/Documentation/admin-guide/kernel-parameters.txt
> > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > @@ -1,3 +1,10 @@
> > + aarch32_emulation= [ARM64]
> > + Format: <bool>
> > + When true, allows loading 32-bit programs and executing
> > + 32-bit syscalls, essentially overriding
> > + AARCH32_EMULATION_DEFAULT_DISABLED at boot time. when false,
> > + unconditionally disables AARCH32 emulation.
>
> Can we please drop the 'emulation' part of the name? We don't use that
> terminology on arm64 for regular execution of compat tasks, and only use that
> to refer to true emulation of deprecated instructions.
>
> We already have the 'allow_mismatched_32bit_el0' option; can we please us a
> name that aligns with that? e.g. 'allow_32bit_el0=false' to disable support.
>
Sure, 'allow_mismatched_32bit_el0' will do. I'll prepare a patch accordingly.
Andrea
On Thu, Oct 19, 2023 at 02:38:32PM +0200, Andrea della Porta wrote:
> On 13:52 Wed 18 Oct , Mark Rutland wrote:
> > On Wed, Oct 18, 2023 at 01:13:20PM +0200, Andrea della Porta wrote:
> > > Major aspect of Aarch32 emulation is the ability to load 32bit
> > > processes.
> > > That's currently decided (among others) by compat_elf_check_arch().
> > >
> > > Make the macro use aarch32_enabled() to decide if Aarch32 compat is
> > > enabled before loading a 32bit process.
> > >
> > > Signed-off-by: Andrea della Porta <[email protected]>
> >
> > Why can't you make system_supports_32bit_el0() take the option into account
> > instead?
> >
>
> I may be wrong here, but it seems to me that system_supports_32bit_el0()
> answers teh question "can this system supports compat execution?" rather than
> "do I want this system to run any compat execution?". That's the point of
> aarch32_enabled(), to state whether we want teh system to run A32 code or not,
> regardless of the system supporting it (of course, if the system does not
> support A32 in EL0, this is a no-no, but that's another story).
That's what the implementation does today, but we're really using it as a "do
we intend for 32-bit EL0 to work?" predicate, and generally the
system_supports_${FEATURE}() helpers are affected by the combination of actual
HW support, kernel config options, *and* kernel command line options. For
example, system_supports_sve() is affected by both CONFIG_ARM64_SVE and the
"arm64.nosve" command line option.
Thanks,
Mark.
On 15:27 Thu 19 Oct , Mark Rutland wrote:
> On Thu, Oct 19, 2023 at 02:38:32PM +0200, Andrea della Porta wrote:
> > On 13:52 Wed 18 Oct , Mark Rutland wrote:
> > > On Wed, Oct 18, 2023 at 01:13:20PM +0200, Andrea della Porta wrote:
> > > > Major aspect of Aarch32 emulation is the ability to load 32bit
> > > > processes.
> > > > That's currently decided (among others) by compat_elf_check_arch().
> > > >
> > > > Make the macro use aarch32_enabled() to decide if Aarch32 compat is
> > > > enabled before loading a 32bit process.
> > > >
> > > > Signed-off-by: Andrea della Porta <[email protected]>
> > >
> > > Why can't you make system_supports_32bit_el0() take the option into account
> > > instead?
> > >
> >
> > I may be wrong here, but it seems to me that system_supports_32bit_el0()
> > answers teh question "can this system supports compat execution?" rather than
> > "do I want this system to run any compat execution?". That's the point of
> > aarch32_enabled(), to state whether we want teh system to run A32 code or not,
> > regardless of the system supporting it (of course, if the system does not
> > support A32 in EL0, this is a no-no, but that's another story).
>
> That's what the implementation does today, but we're really using it as a "do
> we intend for 32-bit EL0 to work?" predicate, and generally the
> system_supports_${FEATURE}() helpers are affected by the combination of actual
> HW support, kernel config options, *and* kernel command line options. For
> example, system_supports_sve() is affected by both CONFIG_ARM64_SVE and the
> "arm64.nosve" command line option.
>
> Thanks,
> Mark.
Many thanks for the explanation, then inserting aach32_enabled() in
system_supports_32bit_el0() is the way to go.
Andrea
Hi Andrea,
kernel test robot noticed the following build warnings:
[auto build test WARNING on arm64/for-next/core]
[also build test WARNING on arm-perf/for-next/perf arm/for-next arm/fixes kvmarm/next soc/for-next linus/master v6.6-rc6 next-20231020]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Andrea-della-Porta/arm64-Introduce-aarch32_enabled/20231018-191517
base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
patch link: https://lore.kernel.org/r/88bdea628a13747bff32c0c3055d6d6ef7264d96.1697614386.git.andrea.porta%40suse.com
patch subject: [PATCH 3/4] arm64/entry-common: Make Aarch32 syscalls' availability depend on aarch32_enabled()
config: arm64-randconfig-003-20231023 (https://download.01.org/0day-ci/archive/20231023/[email protected]/config)
compiler: aarch64-linux-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231023/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> arch/arm64/kernel/entry-common.c:805:11: warning: no previous prototype for 'el0t_32_sync_ni_handler' [-Wmissing-prototypes]
805 | UNHANDLED(el0t, 32, sync_ni)
| ^~~~
arch/arm64/kernel/entry-common.c:302:25: note: in definition of macro 'UNHANDLED'
302 | asmlinkage void noinstr el##_##regsize##_##vector##_handler(struct pt_regs *regs) \
| ^~
>> arch/arm64/kernel/entry-common.c:806:11: warning: no previous prototype for 'el0t_32_irq_ni_handler' [-Wmissing-prototypes]
806 | UNHANDLED(el0t, 32, irq_ni)
| ^~~~
arch/arm64/kernel/entry-common.c:302:25: note: in definition of macro 'UNHANDLED'
302 | asmlinkage void noinstr el##_##regsize##_##vector##_handler(struct pt_regs *regs) \
| ^~
>> arch/arm64/kernel/entry-common.c:807:11: warning: no previous prototype for 'el0t_32_fiq_ni_handler' [-Wmissing-prototypes]
807 | UNHANDLED(el0t, 32, fiq_ni)
| ^~~~
arch/arm64/kernel/entry-common.c:302:25: note: in definition of macro 'UNHANDLED'
302 | asmlinkage void noinstr el##_##regsize##_##vector##_handler(struct pt_regs *regs) \
| ^~
>> arch/arm64/kernel/entry-common.c:808:11: warning: no previous prototype for 'el0t_32_error_ni_handler' [-Wmissing-prototypes]
808 | UNHANDLED(el0t, 32, error_ni)
| ^~~~
arch/arm64/kernel/entry-common.c:302:25: note: in definition of macro 'UNHANDLED'
302 | asmlinkage void noinstr el##_##regsize##_##vector##_handler(struct pt_regs *regs) \
| ^~
vim +/el0t_32_sync_ni_handler +805 arch/arm64/kernel/entry-common.c
803
804 #ifdef CONFIG_COMPAT
> 805 UNHANDLED(el0t, 32, sync_ni)
> 806 UNHANDLED(el0t, 32, irq_ni)
> 807 UNHANDLED(el0t, 32, fiq_ni)
> 808 UNHANDLED(el0t, 32, error_ni)
809
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki