2019-06-20 00:33:35

by Nick Desaulniers

[permalink] [raw]
Subject: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

Generated via:
$ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig
$ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig
<enable CONFIG_RANDOMIZE_BASE aka KASLR>
$ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make savedefconfig
$ mv defconfig arch/arm64/configs/defconfig

Removes explicit enablement of:
CONFIG_TI_SCI_PROTOCOL
CONFIG_TI_MESSAGE_MANAGER
CONFIG_SOC_TI
CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND

Signed-off-by: Nick Desaulniers <[email protected]>
---
arch/arm64/configs/defconfig | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index 4d583514258c..54d35e847836 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -68,6 +68,7 @@ CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_XEN=y
CONFIG_COMPAT=y
+CONFIG_RANDOMIZE_BASE=y
CONFIG_HIBERNATION=y
CONFIG_WQ_POWER_EFFICIENT_DEFAULT=y
CONFIG_ARM_CPUIDLE=y
@@ -86,7 +87,6 @@ CONFIG_ARM_TEGRA186_CPUFREQ=y
CONFIG_ARM_SCPI_PROTOCOL=y
CONFIG_RASPBERRYPI_FIRMWARE=y
CONFIG_INTEL_STRATIX10_SERVICE=y
-CONFIG_TI_SCI_PROTOCOL=y
CONFIG_EFI_CAPSULE_LOADER=y
CONFIG_IMX_SCU=y
CONFIG_IMX_SCU_PD=y
@@ -191,7 +191,6 @@ CONFIG_PCIE_QCOM=y
CONFIG_PCIE_ARMADA_8K=y
CONFIG_PCIE_KIRIN=y
CONFIG_PCIE_HISI_STB=y
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_HISILICON_LPC=y
@@ -365,8 +364,8 @@ CONFIG_SPI_PL022=y
CONFIG_SPI_ROCKCHIP=y
CONFIG_SPI_QUP=y
CONFIG_SPI_S3C64XX=y
-CONFIG_SPI_SPIDEV=m
CONFIG_SPI_SUN6I=y
+CONFIG_SPI_SPIDEV=m
CONFIG_SPMI=y
CONFIG_PINCTRL_SINGLE=y
CONFIG_PINCTRL_MAX77620=y
@@ -658,7 +657,6 @@ CONFIG_ARM_MHU=y
CONFIG_IMX_MBOX=y
CONFIG_PLATFORM_MHU=y
CONFIG_BCM2835_MBOX=y
-CONFIG_TI_MESSAGE_MANAGER=y
CONFIG_QCOM_APCS_IPC=y
CONFIG_ROCKCHIP_IOMMU=y
CONFIG_TEGRA_IOMMU_SMMU=y
@@ -696,9 +694,7 @@ CONFIG_ARCH_TEGRA_210_SOC=y
CONFIG_ARCH_TEGRA_186_SOC=y
CONFIG_ARCH_TEGRA_194_SOC=y
CONFIG_ARCH_K3_AM6_SOC=y
-CONFIG_SOC_TI=y
CONFIG_TI_SCI_PM_DOMAINS=y
-CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y
CONFIG_EXTCON_USB_GPIO=y
CONFIG_EXTCON_USBC_CROS_EC=y
CONFIG_MEMORY=y
--
2.22.0.410.gd8fdbe21b5-goog


2019-06-20 07:47:57

by Will Deacon

[permalink] [raw]
Subject: Re: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

Hi Nick,

On Wed, Jun 19, 2019 at 05:32:42PM -0700, Nick Desaulniers wrote:
> Generated via:
> $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig
> $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig
> <enable CONFIG_RANDOMIZE_BASE aka KASLR>
> $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make savedefconfig
> $ mv defconfig arch/arm64/configs/defconfig

Hmm, I'm in two minds about whether we want this on by default. On the plus
side, it gets us extra testing coverage, although the /vast/ majority of
firmware implementations I run into either don't pass a seed or don't
provide a working EFI_RNG. Perhaps that's just a chicken-and-egg problem
which can be solved if we shout loud enough when we fail to randomize; we'll
also eventually be in a better position when CPUs start implementing the
v8.5 RNG instructions (but don't hold your breath unless you have an
unusually high lung capacity).

On the flip side, I worry that it could make debugging more difficult, but I
don't know whether that's a genuine concern or not. I'm assuming you've
debugged your fair share of crashes from KASLR-enabled kernels; how bad is
it? (I'm thinking of the case where somebody mails you part of a panic log
and a .config).

Irrespective of the above, I know Catalin was running into issues with his
automated tests where the kernel would die silently during early boot with
some seeds. That's a bit rubbish if it's still the case -- Catalin?

Finally, I know that (K)ASLR can be a bit controversial amongst security
folks, with some seeing it as purely a smoke-and-mirrors game with no
tangible benefits other than making us feel better about ourselves. Is it
still the case that it can be trivially bypassed, or do you see it actually
preventing some attacks in production?

Sorry for the barrage of questions, but I think enabling this one by default
is quite a significant thing to do and probably deserves a bit of scrutiny
beforehand.

Cheers,

Will

2019-06-20 08:19:34

by Ard Biesheuvel

[permalink] [raw]
Subject: Re: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

On Thu, 20 Jun 2019 at 09:47, Will Deacon <[email protected]> wrote:
>
> Hi Nick,
>
> On Wed, Jun 19, 2019 at 05:32:42PM -0700, Nick Desaulniers wrote:
> > Generated via:
> > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig
> > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig
> > <enable CONFIG_RANDOMIZE_BASE aka KASLR>
> > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make savedefconfig
> > $ mv defconfig arch/arm64/configs/defconfig
>
> Hmm, I'm in two minds about whether we want this on by default. On the plus
> side, it gets us extra testing coverage, although the /vast/ majority of
> firmware implementations I run into either don't pass a seed or don't
> provide a working EFI_RNG. Perhaps that's just a chicken-and-egg problem
> which can be solved if we shout loud enough when we fail to randomize; we'll
> also eventually be in a better position when CPUs start implementing the
> v8.5 RNG instructions (but don't hold your breath unless you have an
> unusually high lung capacity).
>

For testing coverage purposes, exercising the relocation machinery etc
even on no-kaslr boots would be beneficial imo. (The kernel is
relocated once for non-kaslr boots and twice for kaslr boots on kaslr
capable kernels)

> On the flip side, I worry that it could make debugging more difficult, but I
> don't know whether that's a genuine concern or not. I'm assuming you've
> debugged your fair share of crashes from KASLR-enabled kernels; how bad is
> it? (I'm thinking of the case where somebody mails you part of a panic log
> and a .config).
>

When you are debugging using GDB, it can get a bit tedious, since you
have to pass the offset when you load the symbols. However, in that
case, you can just pass 'nokaslr' unless you are debugging something
that is affected by the randomization.

For reading backtraces etc, nothing really changes, since we get
symbol+nnn/mmm entries (and the full panic log prints the KASLR offset
as well, in case it matters)

> Irrespective of the above, I know Catalin was running into issues with his
> automated tests where the kernel would die silently during early boot with
> some seeds. That's a bit rubbish if it's still the case -- Catalin?
>

Yes, it would be good if we could fix that. In fact, I would argue
that having this change in would have increased the likelihood that
someone else would have spotted it and fixed it :-)

In fact, given how many Android phones are running this code: Nick,
can you check if there are any KASLR related kernel fixes that haven't
been upstreamed?

> Finally, I know that (K)ASLR can be a bit controversial amongst security
> folks, with some seeing it as purely a smoke-and-mirrors game with no
> tangible benefits other than making us feel better about ourselves. Is it
> still the case that it can be trivially bypassed, or do you see it actually
> preventing some attacks in production?
>
> Sorry for the barrage of questions, but I think enabling this one by default
> is quite a significant thing to do and probably deserves a bit of scrutiny
> beforehand.
>

I think it is mostly controversial among non-security folks, who think
that every mitigation by itself should be bullet proof. Security folks
tend to think more about how each layer reduces the attack surface,
hopefully resulting in a secure system when all layers are enabled.

So KASLR is known to be broken unless you enable KPTI as well, so that
is something we could take into account. I.e., mitigations that don't
reduce the attack surface at all are just pointless complexity, which
should obviously be avoided.

Another thing to note is that the runtime cost of KASLR is ~zero, with
the exception of the module PLTs. However, the latter could do with
some additional coverage as well, so in summary, I think enabling this
is a good thing. Otherwise, we could disable full module randomization
so that the module PLT code doesn't get used in practice.

Acked-by: Ard Biesheuvel <[email protected]>

2019-06-21 20:28:20

by Nick Desaulniers

[permalink] [raw]
Subject: Re: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

On Thu, Jun 20, 2019 at 1:17 AM Ard Biesheuvel
<[email protected]> wrote:
>
> On Thu, 20 Jun 2019 at 09:47, Will Deacon <[email protected]> wrote:
>
> > On Wed, Jun 19, 2019 at 05:32:42PM -0700, Nick Desaulniers wrote:
> > > Generated via:
> > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig
> > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig
> > > <enable CONFIG_RANDOMIZE_BASE aka KASLR>
> > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make savedefconfig
> > > $ mv defconfig arch/arm64/configs/defconfig

FWIW, it looks like CONFIG_RANDOMIZE_BASE is set in x86 defconfigs.

> >
> > Hmm, I'm in two minds about whether we want this on by default. On the plus
> > side, it gets us extra testing coverage, although the /vast/ majority of
> > firmware implementations I run into either don't pass a seed or don't
> > provide a working EFI_RNG. Perhaps that's just a chicken-and-egg problem
> > which can be solved if we shout loud enough when we fail to randomize; we'll
> > also eventually be in a better position when CPUs start implementing the
> > v8.5 RNG instructions (but don't hold your breath unless you have an
> > unusually high lung capacity).
> >
>
> For testing coverage purposes, exercising the relocation machinery etc
> even on no-kaslr boots would be beneficial imo. (The kernel is
> relocated once for non-kaslr boots and twice for kaslr boots on kaslr
> capable kernels)

Reminds me of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fd9dde6abcb9bfe6c6bee48834e157999f113971
(but maybe unrelated?).

>
> > On the flip side, I worry that it could make debugging more difficult, but I
> > don't know whether that's a genuine concern or not. I'm assuming you've
> > debugged your fair share of crashes from KASLR-enabled kernels; how bad is
> > it? (I'm thinking of the case where somebody mails you part of a panic log
> > and a .config).

I don't recall specific cases where KASLR made debugging difficult. I
went and spoke to our stability team that debugs crash reports from
the field. My understanding is that we capture full ramdumps. They
have a lot of custom tooling for debugging, but they did not recall
ever having to disable KASLR to debug further. We've had KASLR
enabled since I think the 2016 Pixel 1, so I assume their tooling
accounts for the seed/offset.

I think if a full ramdump of the kernel image is loaded into GDB with
the matching kernel image it "just works" but could be mistaken. For
external developers, "nokaslr" boot time param is pretty standard.

> In fact, given how many Android phones are running this code: Nick,
> can you check if there are any KASLR related kernel fixes that haven't
> been upstreamed?

I spoke with the android common kernel team that's trying to burn down
their out of tree patches. I triple checked a doc they had where they
had audited every last patch, looking for for KASLR and
CONFIG_RANDOMIZE_BASE. I also triple checked our internal bug tracker
for burning down the out of tree patches. Finally I'm scanning each
branch of our android-common trees via `git log --all --grep
<KASLR|CONFIG_RANDOMIZE_BASE>`. I haven't found anything yet, and the
team doesn't expect any out of tree patches related to that feature.
Sorry for not responding sooner, but I'm still going through our 4.4,
4.9, 4.14, and 4.19 branches.

I know I put some backports into an unreleased device's kernel to
support LLD (backports of some of your patches). All of those patches
are now upstream in mainline, and LLD has its bugs fixed. In fact, I
think the backports I did of your patches that broke
CONFIG_RANDOMIZE_BASE were actually to support LLD (not Clang) (sorry
again for that). I'm currently upgrading AOSP's prebuilt version of
Clang. Once that exists, I'll clean up that device's kernel.

If you know of any android vendor relying on out of tree hacks to
KASLR, and they're getting them from android common kernels, please
shoot me a quick email. Every out of tree patch is a burden IMO.

Also, for internal code reviews of backports, I've been much
better/more aggressive about the backports being sent to stable (when
they aren't large features) and then pulling them in from there
(sorry, Sami).

>
> > Finally, I know that (K)ASLR can be a bit controversial amongst security
> > folks, with some seeing it as purely a smoke-and-mirrors game with no
> > tangible benefits other than making us feel better about ourselves. Is it
> > still the case that it can be trivially bypassed, or do you see it actually
> > preventing some attacks in production?
> >
> > Sorry for the barrage of questions, but I think enabling this one by default
> > is quite a significant thing to do and probably deserves a bit of scrutiny
> > beforehand.
> >
>
> I think it is mostly controversial among non-security folks, who think
> that every mitigation by itself should be bullet proof. Security folks
> tend to think more about how each layer reduces the attack surface,
> hopefully resulting in a secure system when all layers are enabled.

+ Kees, Sami, Jeff
It's a relatively low cost part of our defense in depth strategy.
Maybe (Kees, Sami, Jeff) have more thoughts?

>
> So KASLR is known to be broken unless you enable KPTI as well, so that
> is something we could take into account. I.e., mitigations that don't
> reduce the attack surface at all are just pointless complexity, which
> should obviously be avoided.

(Note to Sami + Jeff if they had KPTI on their radar)

>
> Another thing to note is that the runtime cost of KASLR is ~zero, with
> the exception of the module PLTs. However, the latter could do with
> some additional coverage as well, so in summary, I think enabling this
> is a good thing. Otherwise, we could disable full module randomization
> so that the module PLT code doesn't get used in practice.
>
> Acked-by: Ard Biesheuvel <[email protected]>

Olof mentioned on IRC that I should resend without the other defconfig
changes. Do others have thoughts on that?

--
Thanks,
~Nick Desaulniers

2019-06-21 21:01:54

by Kees Cook

[permalink] [raw]
Subject: Re: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

On Fri, Jun 21, 2019 at 1:28 PM Nick Desaulniers
<[email protected]> wrote:
>
> On Thu, Jun 20, 2019 at 1:17 AM Ard Biesheuvel
> <[email protected]> wrote:
> > I think it is mostly controversial among non-security folks, who think
> > that every mitigation by itself should be bullet proof. Security folks
> > tend to think more about how each layer reduces the attack surface,
> > hopefully resulting in a secure system when all layers are enabled.
>
> + Kees, Sami, Jeff
> It's a relatively low cost part of our defense in depth strategy.
> Maybe (Kees, Sami, Jeff) have more thoughts?

Right -- the thought is that it provides more benefit than
complication. It is hardly a perfect defense, but it does provide
building blocks to more interesting situations. For example, once
execute-only memory is more common, KASLR + XOM means there is a not
insignificant defense against automated ROP. And KASLR is a general
precursor to fine-grained KASLR (i.e. randomizing on function).

> > So KASLR is known to be broken unless you enable KPTI as well, so that
> > is something we could take into account. I.e., mitigations that don't
> > reduce the attack surface at all are just pointless complexity, which
> > should obviously be avoided.
>
> (Note to Sami + Jeff if they had KPTI on their radar)

I prefer that KPTI always stay enabled. :)

> > Another thing to note is that the runtime cost of KASLR is ~zero, with
> > the exception of the module PLTs. However, the latter could do with
> > some additional coverage as well, so in summary, I think enabling this
> > is a good thing. Otherwise, we could disable full module randomization
> > so that the module PLT code doesn't get used in practice.
> >
> > Acked-by: Ard Biesheuvel <[email protected]>

Reviewed-by: Kees Cook <[email protected]>

--
Kees Cook

2019-06-24 09:57:30

by Catalin Marinas

[permalink] [raw]
Subject: Re: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

On Thu, Jun 20, 2019 at 08:46:58AM +0100, Will Deacon wrote:
> On Wed, Jun 19, 2019 at 05:32:42PM -0700, Nick Desaulniers wrote:
> > Generated via:
> > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig
> > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig
> > <enable CONFIG_RANDOMIZE_BASE aka KASLR>
> > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make savedefconfig
> > $ mv defconfig arch/arm64/configs/defconfig
[...]
> Irrespective of the above, I know Catalin was running into issues with his
> automated tests where the kernel would die silently during early boot with
> some seeds. That's a bit rubbish if it's still the case -- Catalin?

I stopped seeing these failures about 1-2 months ago, not sure exactly
what changed. I'm ok to enable KASLR in defconfig.

--
Catalin

2019-06-24 10:03:16

by Will Deacon

[permalink] [raw]
Subject: Re: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

Hi Nick, Kees, Ard,

Thanks for the responses.

On Fri, Jun 21, 2019 at 01:27:45PM -0700, Nick Desaulniers wrote:
> On Thu, Jun 20, 2019 at 1:17 AM Ard Biesheuvel
> <[email protected]> wrote:
> > On Thu, 20 Jun 2019 at 09:47, Will Deacon <[email protected]> wrote:
> > > On the flip side, I worry that it could make debugging more difficult, but I
> > > don't know whether that's a genuine concern or not. I'm assuming you've
> > > debugged your fair share of crashes from KASLR-enabled kernels; how bad is
> > > it? (I'm thinking of the case where somebody mails you part of a panic log
> > > and a .config).
>
> I don't recall specific cases where KASLR made debugging difficult. I
> went and spoke to our stability team that debugs crash reports from
> the field. My understanding is that we capture full ramdumps. They
> have a lot of custom tooling for debugging, but they did not recall
> ever having to disable KASLR to debug further. We've had KASLR
> enabled since I think the 2016 Pixel 1, so I assume their tooling
> accounts for the seed/offset.
>
> I think if a full ramdump of the kernel image is loaded into GDB with
> the matching kernel image it "just works" but could be mistaken. For
> external developers, "nokaslr" boot time param is pretty standard.
>
> > In fact, given how many Android phones are running this code: Nick,
> > can you check if there are any KASLR related kernel fixes that haven't
> > been upstreamed?
>
> I spoke with the android common kernel team that's trying to burn down
> their out of tree patches. I triple checked a doc they had where they
> had audited every last patch, looking for for KASLR and
> CONFIG_RANDOMIZE_BASE. I also triple checked our internal bug tracker
> for burning down the out of tree patches. Finally I'm scanning each
> branch of our android-common trees via `git log --all --grep
> <KASLR|CONFIG_RANDOMIZE_BASE>`. I haven't found anything yet, and the
> team doesn't expect any out of tree patches related to that feature.
> Sorry for not responding sooner, but I'm still going through our 4.4,
> 4.9, 4.14, and 4.19 branches.

Thanks for having a look. It could be that we've fixed the issue Catalin was
running into in the past -- he was going to see if the problem persists with
mainline, since it was frequent enough that it was causing us to ignore the
results from our testing infrastructure when RANDOMIZE_BASE=y.

> > So KASLR is known to be broken unless you enable KPTI as well, so that
> > is something we could take into account. I.e., mitigations that don't
> > reduce the attack surface at all are just pointless complexity, which
> > should obviously be avoided.
>
> (Note to Sami + Jeff if they had KPTI on their radar)

I mean, we could have RANDOMIZE_BASE select UNMAP_KERNEL_AT_EL0 if you like?
The latter is already default y and hidden behind EXPERT.

> > Another thing to note is that the runtime cost of KASLR is ~zero, with
> > the exception of the module PLTs. However, the latter could do with
> > some additional coverage as well, so in summary, I think enabling this
> > is a good thing. Otherwise, we could disable full module randomization
> > so that the module PLT code doesn't get used in practice.
> >
> > Acked-by: Ard Biesheuvel <[email protected]>
>
> Olof mentioned on IRC that I should resend without the other defconfig
> changes. Do others have thoughts on that?

That's not a bad idea. If you do that, feel free to add my Ack to the one
adding RANDOMIZE_BASE=y:

Acked-by: Will Deacon <[email protected]>

Will

2019-06-24 10:12:38

by Ard Biesheuvel

[permalink] [raw]
Subject: Re: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

On Mon, 24 Jun 2019 at 11:57, Will Deacon <[email protected]> wrote:
>
> Hi Nick, Kees, Ard,
>
> Thanks for the responses.
>
> On Fri, Jun 21, 2019 at 01:27:45PM -0700, Nick Desaulniers wrote:
> > On Thu, Jun 20, 2019 at 1:17 AM Ard Biesheuvel
> > <[email protected]> wrote:
> > > On Thu, 20 Jun 2019 at 09:47, Will Deacon <[email protected]> wrote:
> > > > On the flip side, I worry that it could make debugging more difficult, but I
> > > > don't know whether that's a genuine concern or not. I'm assuming you've
> > > > debugged your fair share of crashes from KASLR-enabled kernels; how bad is
> > > > it? (I'm thinking of the case where somebody mails you part of a panic log
> > > > and a .config).
> >
> > I don't recall specific cases where KASLR made debugging difficult. I
> > went and spoke to our stability team that debugs crash reports from
> > the field. My understanding is that we capture full ramdumps. They
> > have a lot of custom tooling for debugging, but they did not recall
> > ever having to disable KASLR to debug further. We've had KASLR
> > enabled since I think the 2016 Pixel 1, so I assume their tooling
> > accounts for the seed/offset.
> >
> > I think if a full ramdump of the kernel image is loaded into GDB with
> > the matching kernel image it "just works" but could be mistaken. For
> > external developers, "nokaslr" boot time param is pretty standard.
> >
> > > In fact, given how many Android phones are running this code: Nick,
> > > can you check if there are any KASLR related kernel fixes that haven't
> > > been upstreamed?
> >
> > I spoke with the android common kernel team that's trying to burn down
> > their out of tree patches. I triple checked a doc they had where they
> > had audited every last patch, looking for for KASLR and
> > CONFIG_RANDOMIZE_BASE. I also triple checked our internal bug tracker
> > for burning down the out of tree patches. Finally I'm scanning each
> > branch of our android-common trees via `git log --all --grep
> > <KASLR|CONFIG_RANDOMIZE_BASE>`. I haven't found anything yet, and the
> > team doesn't expect any out of tree patches related to that feature.
> > Sorry for not responding sooner, but I'm still going through our 4.4,
> > 4.9, 4.14, and 4.19 branches.
>
> Thanks for having a look. It could be that we've fixed the issue Catalin was
> running into in the past -- he was going to see if the problem persists with
> mainline, since it was frequent enough that it was causing us to ignore the
> results from our testing infrastructure when RANDOMIZE_BASE=y.
>

I had no idea this was the case. I can look into it if we are still
seeing failures.

> > > So KASLR is known to be broken unless you enable KPTI as well, so that
> > > is something we could take into account. I.e., mitigations that don't
> > > reduce the attack surface at all are just pointless complexity, which
> > > should obviously be avoided.
> >
> > (Note to Sami + Jeff if they had KPTI on their radar)
>
> I mean, we could have RANDOMIZE_BASE select UNMAP_KERNEL_AT_EL0 if you like?
> The latter is already default y and hidden behind EXPERT.
>

IIRC, when KASLR is enabled (and we have a seed), we override the
runtime decision to out out of KPTI, and so even uarchs that don't
require the Meltdown mitigations it provides will still be using it.

So I'd be fine with just adding a note to the UNMAP_KERNEL_AT_EL0
Kconfig help text that even non-affected uarchs have a use for it if
KASLR is enabled, but given that it is already behind EXPERT, I don't
think more hand holding is necessary.

> > > Another thing to note is that the runtime cost of KASLR is ~zero, with
> > > the exception of the module PLTs. However, the latter could do with
> > > some additional coverage as well, so in summary, I think enabling this
> > > is a good thing. Otherwise, we could disable full module randomization
> > > so that the module PLT code doesn't get used in practice.
> > >
> > > Acked-by: Ard Biesheuvel <[email protected]>
> >
> > Olof mentioned on IRC that I should resend without the other defconfig
> > changes. Do others have thoughts on that?
>
> That's not a bad idea. If you do that, feel free to add my Ack to the one
> adding RANDOMIZE_BASE=y:
>
> Acked-by: Will Deacon <[email protected]>
>
> Will

2019-06-24 21:18:55

by Nick Desaulniers

[permalink] [raw]
Subject: [PATCH v2] arm64: defconfig: enable CONFIG_RANDOMIZE_BASE

For testing coverage and improved defense in depth, enable KASLR by
default.

Acked-by: Ard Biesheuvel <[email protected]>
Acked-by: Will Deacon <[email protected]>
Reviewed-by: Kees Cook <[email protected]>
Suggested-by: Arnd Bergmann <[email protected]>
Suggested-by: Olof Johansson <[email protected]>
Signed-off-by: Nick Desaulniers <[email protected]>
---
Changes v1 -> v2:
* drop other hunks as per Olof and Will
* Collect Acks/Reviewed-by
* Add Arnd and Olof's suggested by
* rewrite commit message

arch/arm64/configs/defconfig | 1 +
1 file changed, 1 insertion(+)

diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index 4d583514258c..a7cbf7cd84b4 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -68,6 +68,7 @@ CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_XEN=y
CONFIG_COMPAT=y
+CONFIG_RANDOMIZE_BASE=y
CONFIG_HIBERNATION=y
CONFIG_WQ_POWER_EFFICIENT_DEFAULT=y
CONFIG_ARM_CPUIDLE=y
--
2.22.0.410.gd8fdbe21b5-goog

2019-06-25 07:59:04

by Catalin Marinas

[permalink] [raw]
Subject: Re: [PATCH v2] arm64: defconfig: enable CONFIG_RANDOMIZE_BASE

On Mon, Jun 24, 2019 at 10:58:50AM -0700, Nick Desaulniers wrote:
> For testing coverage and improved defense in depth, enable KASLR by
> default.
>
> Acked-by: Ard Biesheuvel <[email protected]>
> Acked-by: Will Deacon <[email protected]>
> Reviewed-by: Kees Cook <[email protected]>
> Suggested-by: Arnd Bergmann <[email protected]>
> Suggested-by: Olof Johansson <[email protected]>
> Signed-off-by: Nick Desaulniers <[email protected]>

Thanks. I'll queue this for 5.3.

--
Catalin

2019-06-25 19:17:51

by Catalin Marinas

[permalink] [raw]
Subject: Re: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

On Mon, Jun 24, 2019 at 12:06:18PM +0200, Ard Biesheuvel wrote:
> On Mon, 24 Jun 2019 at 11:57, Will Deacon <[email protected]> wrote:
> > Thanks for having a look. It could be that we've fixed the issue Catalin was
> > running into in the past -- he was going to see if the problem persists with
> > mainline, since it was frequent enough that it was causing us to ignore the
> > results from our testing infrastructure when RANDOMIZE_BASE=y.
>
> I had no idea this was the case. I can look into it if we are still
> seeing failures.

I've seen the panic below with 5.2-rc1, defconfig + RANDOMIZE_BASE=y in
a guest on TX2. It takes a few tries to trigger just with kaslr,
enabling lots of other DEBUG_* options makes the failures more
deterministic. I can't really say it's kaslr's fault here, only that I
used to consistently get it in this configuration. For some reason, I
can no longer reproduce it on arm64 for-next/core (or maybe it just
takes more tries and my script doesn't catch this).

The fault is in the ip_tables module, the __this_cpu_read in
xt_write_recseq_begin() inlined in ipt_do_table(). The disassembled
sequence in my build:

0000000000000188 <ipt_do_table>:
...
258: d538d080 mrs x0, tpidr_el1
25c: aa1303f9 mov x25, x19
260: b8606b34 ldr w20, [x25, x0]

# modprobe iptable_filter
[ 45.618896] Unable to handle kernel paging request at virtual address ffffeda0ffbe1388
[ 45.620545] Mem abort info:
[ 45.621035] ESR = 0x96000005
[ 45.621559] Exception class = DABT (current EL), IL = 32 bits
[ 45.622527] SET = 0, FnV = 0
[ 45.623032] EA = 0, S1PTW = 0
[ 45.623684] Data abort info:
[ 45.624192] ISV = 0, ISS = 0x00000005
[ 45.624849] CM = 0, WnR = 0
[ 45.625361] swapper pgtable: 4k pages, 48-bit VAs, pgdp = 0000000026cb760e
[ 45.626537] [ffffeda0ffbe1388] pgd=00000000bfffa003, pud=0000000000000000
[ 45.627864] Internal error: Oops: 96000005 [#1] PREEMPT SMP
[ 45.628844] Modules linked in: iptable_filter cfg80211 rfkill 8021q garp crct10dif_ce stp mrp llc ip_tables x_tables ipv6
[ 45.630700] Process kworker/u8:5 (pid: 173, stack limit = 0x0000000047fc7e17)
[ 45.632056] CPU: 3 PID: 173 Comm: kworker/u8:5 Not tainted 5.2.0-rc1 #1
[ 45.633377] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 45.634752] Workqueue: rpciod rpc_async_schedule
[ 45.635684] pstate: 80400005 (Nzcv daif +PAN -UAO)
[ 45.636674] pc : ipt_do_table+0xd8/0x4b8 [ip_tables]
[ 45.637686] lr : ipt_do_table+0xc4/0x4b8 [ip_tables]
[ 45.638685] sp : ffff000010aa35a0
[ 45.639358] x29: ffff000010aa35a0 x28: ffffeda1f7eb0ce8
[ 45.640424] x27: ffffeda1e69f1380 x26: ffff5d2c2314a288
[ 45.641493] x25: ffff5d2c0cfbf388 x24: ffffeda1fb5a9000
[ 45.642556] x23: ffff5d2c2314a220 x22: 0000000000000003
[ 45.643636] x21: ffffeda1f79c15d8 x20: ffffeda1e69f1780
[ 45.644727] x19: ffff5d2c0cfbf388 x18: 0000000000000000
[ 45.645814] x17: 0000000000000000 x16: ffff5d2d0bc16d78
[ 45.646908] x15: 0000000000000000 x14: 0900000017ea6502
[ 45.647994] x13: 014a023b4751727f x12: 0000000000002238
[ 45.649060] x11: ffff5d2d0e0de000 x10: 0000000000004000
[ 45.650117] x9 : 000000000000000e x8 : ffffeda1fb5a9000
[ 45.651178] x7 : ffffeda1f7eb0c00 x6 : 00000000000000e8
[ 45.652263] x5 : 0000000000000080 x4 : 00009074f2c22000
[ 45.653286] x3 : 0000000000000200 x2 : ffffeda1fc41c440
[ 45.654314] x1 : ffff5d2c2314a000 x0 : 00009074f2c22000
[ 45.655382] Call trace:
[ 45.655896] ipt_do_table+0xd8/0x4b8 [ip_tables]
[ 45.656825] iptable_filter_hook+0x1c/0x28 [iptable_filter]
[ 45.657938] nf_hook_slow+0x44/0xe8
[ 45.658647] __ip_local_out+0x150/0x250
[ 45.659421] ip_local_out+0x20/0x80
[ 45.660142] __ip_queue_xmit+0x1b0/0x540
[ 45.660937] ip_queue_xmit+0x10/0x18
[ 45.661661] __tcp_transmit_skb+0x50c/0xad8
[ 45.662494] tcp_write_xmit+0x6cc/0x1010
[ 45.663294] __tcp_push_pending_frames+0x38/0xc0
[ 45.664236] tcp_push+0x150/0x170
[ 45.664912] tcp_sendmsg_locked+0xaa0/0xc40
[ 45.665754] tcp_sendmsg+0x34/0x58
[ 45.666442] inet_sendmsg+0x48/0x210
[ 45.667175] sock_sendmsg+0x18/0x30
[ 45.667889] xs_sendpages+0xfc/0x2d8
[ 45.668610] xs_tcp_send_request+0x9c/0x1b0
[ 45.669454] xprt_transmit+0x100/0x5b8
[ 45.670210] call_transmit+0x8c/0xa0
[ 45.670938] __rpc_execute+0xbc/0x580
[ 45.671687] rpc_async_schedule+0x28/0x48
[ 45.672493] process_one_work+0x244/0x680
[ 45.673306] worker_thread+0x40/0x3f0
[ 45.674053] kthread+0x128/0x130
[ 45.674710] ret_from_fork+0x10/0x18
[ 45.675436] Code: b0f4f3d3 910e2273 d538d080 aa1303f9 (b8606b34)
[ 45.676665] ---[ end trace 9d1f75fe6e41c0d7 ]---
[ 45.677590] Kernel panic - not syncing: Fatal exception in interrupt
[ 45.678851] SMP: stopping secondary CPUs
[ 45.679670] Kernel Offset: 0x5d2cfb600000 from 0xffff000010000000
[ 45.680882] PHYS_OFFSET: 0xffff925ec0000000
[ 45.681727] CPU features: 0x002,23800438
[ 45.682518] Memory Limit: none
[ 45.683143] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

--
Catalin

2019-06-25 19:18:00

by Ard Biesheuvel

[permalink] [raw]
Subject: Re: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

On Tue, 25 Jun 2019 at 17:39, Catalin Marinas <[email protected]> wrote:
>
> On Mon, Jun 24, 2019 at 12:06:18PM +0200, Ard Biesheuvel wrote:
> > On Mon, 24 Jun 2019 at 11:57, Will Deacon <[email protected]> wrote:
> > > Thanks for having a look. It could be that we've fixed the issue Catalin was
> > > running into in the past -- he was going to see if the problem persists with
> > > mainline, since it was frequent enough that it was causing us to ignore the
> > > results from our testing infrastructure when RANDOMIZE_BASE=y.
> >
> > I had no idea this was the case. I can look into it if we are still
> > seeing failures.
>
> I've seen the panic below with 5.2-rc1, defconfig + RANDOMIZE_BASE=y in
> a guest on TX2. It takes a few tries to trigger just with kaslr,
> enabling lots of other DEBUG_* options makes the failures more
> deterministic. I can't really say it's kaslr's fault here, only that I
> used to consistently get it in this configuration. For some reason, I
> can no longer reproduce it on arm64 for-next/core (or maybe it just
> takes more tries and my script doesn't catch this).
>
> The fault is in the ip_tables module, the __this_cpu_read in
> xt_write_recseq_begin() inlined in ipt_do_table(). The disassembled
> sequence in my build:
>
> 0000000000000188 <ipt_do_table>:
> ...
> 258: d538d080 mrs x0, tpidr_el1
> 25c: aa1303f9 mov x25, x19
> 260: b8606b34 ldr w20, [x25, x0]
>

This was fixed recently by

arm64/kernel: kaslr: reduce module randomization range to 2 GB

(and arm64/module: deal with ambiguity in PRELxx relocation ranges to
some extent)

2019-06-25 19:19:48

by Catalin Marinas

[permalink] [raw]
Subject: Re: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

On Tue, Jun 25, 2019 at 05:42:49PM +0200, Ard Biesheuvel wrote:
> On Tue, 25 Jun 2019 at 17:39, Catalin Marinas <[email protected]> wrote:
> > On Mon, Jun 24, 2019 at 12:06:18PM +0200, Ard Biesheuvel wrote:
> > > On Mon, 24 Jun 2019 at 11:57, Will Deacon <[email protected]> wrote:
> > > > Thanks for having a look. It could be that we've fixed the issue Catalin was
> > > > running into in the past -- he was going to see if the problem persists with
> > > > mainline, since it was frequent enough that it was causing us to ignore the
> > > > results from our testing infrastructure when RANDOMIZE_BASE=y.
> > >
> > > I had no idea this was the case. I can look into it if we are still
> > > seeing failures.
> >
> > I've seen the panic below with 5.2-rc1, defconfig + RANDOMIZE_BASE=y in
> > a guest on TX2. It takes a few tries to trigger just with kaslr,
> > enabling lots of other DEBUG_* options makes the failures more
> > deterministic. I can't really say it's kaslr's fault here, only that I
> > used to consistently get it in this configuration. For some reason, I
> > can no longer reproduce it on arm64 for-next/core (or maybe it just
> > takes more tries and my script doesn't catch this).
> >
> > The fault is in the ip_tables module, the __this_cpu_read in
> > xt_write_recseq_begin() inlined in ipt_do_table(). The disassembled
> > sequence in my build:
> >
> > 0000000000000188 <ipt_do_table>:
> > ...
> > 258: d538d080 mrs x0, tpidr_el1
> > 25c: aa1303f9 mov x25, x19
> > 260: b8606b34 ldr w20, [x25, x0]
>
> This was fixed recently by
>
> arm64/kernel: kaslr: reduce module randomization range to 2 GB
>
> (and arm64/module: deal with ambiguity in PRELxx relocation ranges to
> some extent)

Thanks. This explains it.

And another weird case that triggers only with 64K pages, KASan and
KASLR combination (guest on TX2). My test script modprobes all the
modules it finds installed (including some test kernel modules like lock
torture). At some point during modprobing, vmalloc trips over the
WARN_ON(!pte_none(*pte)) in vmap_pte_range():

# modprobe <some random module>
[ 341.631037] WARNING: CPU: 3 PID: 737 at mm/vmalloc.c:153 vmap_page_range_noflush+0x300/0x410
[ 341.632433] Modules linked in: pwm_fan altera_freeze_bridge fpga_region fpga_bridge reset_meson_audio_arb cros_ec_baro cros_ec_sensors rockchip_saradc cros_ec_light_prox industrialio_triggered_buffer kfifo_buf cros_ec_sensors_core spi_meson_spifc spi_meson_spicc spi_bcm2835 spi_
[ 341.632533] pwm_sun4i pwm_rcar aes_neon_bs chacha_neon chacha_generic sha3_generic aes_neon_blk sm3_generic sha512_arm64 test_udelay backtracetest locktorture torture br_netfilter bridge ip6table_mangle ip6table_nat nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tab
[ 341.661768] CPU: 3 PID: 737 Comm: modprobe Tainted: G W 5.2.0-rc3-00015-gd1b79ba6a070 #82
[ 341.663644] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 341.665065] pstate: 20400005 (nzCv daif +PAN -UAO)
[ 341.666027] pc : vmap_page_range_noflush+0x300/0x410
[ 341.667015] lr : map_vm_area+0x8c/0xd8
[ 341.667806] sp : ffffb618f4877690
[ 341.668488] x29: ffffb618f4877690 x28: ffff200026cc0008
[ 341.669552] x27: dfff200000000000 x26: ffff0400023f0000
[ 341.670611] x25: ffff200026bea818 x24: ffffb6193fd411f8
[ 341.671718] x23: ffff040002400000 x22: ffffb6193fd50000
[ 341.672769] x21: ffff040002400000 x20: ffff040004d7d503
[ 341.673836] x19: 0000000000000000 x18: ffffb618f11d65b0
[ 341.674903] x17: 0000000000002036 x16: ffff200024320da8
[ 341.675979] x15: 0000000000001000 x14: 00000000000001ba
[ 341.677043] x13: 0000000000054f18 x12: ffff0ffdb0c7a6b7
[ 341.678093] x11: 1fffeffdb0c7a6b6 x10: 1ffff6c327fa823f
[ 341.679197] x9 : ffff0400223f0000 x8 : 000036193fd50000
[ 341.680246] x7 : ffff200026bea000 x6 : 0000000000000001
[ 341.681362] x5 : 00000000f2f2f200 x4 : ffff200027c236c8
[ 341.682459] x3 : ffff0800023f0000 x2 : ffff7fed863d3580
[ 341.683517] x1 : 000036193fd411f8 x0 : 00e000005f880f93
[ 341.684582] Call trace:
[ 341.685089] vmap_page_range_noflush+0x300/0x410
[ 341.686050] map_vm_area+0x8c/0xd8
[ 341.686742] __vmalloc_node_range+0x4e4/0x718
[ 341.687608] kasan_module_alloc+0xb4/0x168
[ 341.688440] module_alloc+0xc0/0x240
[ 341.689171] load_module+0x18ac/0x65c0
[ 341.689970] __se_sys_finit_module+0x138/0x148
[ 341.690849] __arm64_sys_finit_module+0x68/0x98
[ 341.691765] el0_svc_handler+0x14c/0x360
[ 341.692539] el0_svc+0x8/0xc
[ 341.693167] ---[ end trace 1e09f33962be79c8 ]---

From this point, the kernel gets into some weird state with a few other
page faults at check_memory_region+0x64/0x1c8. Eventually, during
poweroff, I get the splat below. It seems that even though vzalloc()
allocated, the memory isn't there (which is not surprising given the
above warning.

[ 1016.168708] Unable to handle kernel paging request at virtual address ffff0400021c200a
[ 1016.170104] Mem abort info:
[ 1016.170598] ESR = 0x96000007
[ 1016.172011] Exception class = DABT (current EL), IL = 32 bits
[ 1016.173079] SET = 0, FnV = 0
[ 1016.173614] EA = 0, S1PTW = 0
[ 1016.174177] Data abort info:
[ 1016.174688] ISV = 0, ISS = 0x00000007
[ 1016.175513] CM = 0, WnR = 0
[ 1016.176064] swapper pgtable: 64k pages, 48-bit VAs, pgdp=000000005dec0000
[ 1016.177251] [ffff0400021c200a] pgd=00000000bfd50003, pud=00000000bfd50003, pmd=00000000bfd40003, pte=0000000000000000
[ 1016.179155] Internal error: Oops: 96000007 [#4] PREEMPT SMP
[ 1016.180122] Modules linked in: ghash_generic lcd pwm_fan altera_freeze_bridge fpga_region fpga_bridge reset_meson_audio_arb cros_ec_baro cros_ec_sensors rockchip_saradc cros_ec_light_prox industrialio_triggered_buffer kfifo_buf cros_ec_sensors_core spi_meson_spifc spi_meson_spic
[ 1016.180226] pwm_bcm2835 pwm_tegra pwm_sun4i pwm_rcar aes_neon_bs chacha_neon chacha_generic sha3_generic aes_neon_blk sm3_generic sha512_arm64 test_udelay backtracetest locktorture torture br_netfilter bridge ip6table_mangle ip6table_nat nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 i
[ 1016.212762] CPU: 1 PID: 1 Comm: systemd Tainted: G D W 5.2.0-rc3-00015-gd1b79ba6a070 #82
[ 1016.214653] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 1016.216081] pstate: 20400005 (nzCv daif +PAN -UAO)
[ 1016.217152] pc : n_tty_open+0x54/0x148
[ 1016.217931] lr : n_tty_open+0x18/0x148
[ 1016.218781] sp : ffffb618e0707660
[ 1016.219505] x29: ffffb618e0707660 x28: ffffb618e0fe7b00
[ 1016.220662] x27: 0000000000000000 x26: 0000000000020101
[ 1016.221758] x25: 1ffff6c31f6dbd68 x24: 0000000000000000
[ 1016.222866] x23: ffffb618f3069058 x22: 0000000000000000
[ 1016.223987] x21: 000000010002bb56 x20: ffff200010e10000
[ 1016.225097] x19: ffffb618f3069000 x18: 0000000000000000
[ 1016.226200] x17: 0000000000000000 x16: 0000000000000000
[ 1016.227296] x15: 0000000000000000 x14: 0000000000000000
[ 1016.228364] x13: 1ffff6c31c0e0f93 x12: ffff0ffdb0c7f1df
[ 1016.229479] x11: 1fffeffdb0c7f1de x10: 1ffff6c31c04d0e1
[ 1016.230594] x9 : ffff200030e10000 x8 : 000036193fff0000
[ 1016.231700] x7 : ffff200026bea000 x6 : 0000000000000008
[ 1016.232812] x5 : 00000000f2f2f200 x4 : ffff200027c236c8
[ 1016.233930] x3 : ffff240010e10000 x2 : 1fffe400021c200a
[ 1016.235043] x1 : dfff200000000000 x0 : ffff200010e10050
[ 1016.236149] Call trace:
[ 1016.236693] n_tty_open+0x54/0x148
[ 1016.237409] tty_ldisc_open.isra.1+0x94/0x128
[ 1016.238334] tty_ldisc_setup+0x40/0xd0
[ 1016.239117] tty_init_dev+0x180/0x3a0
[ 1016.239906] tty_open+0x438/0x780
[ 1016.240627] chrdev_open+0x194/0x408
[ 1016.241385] do_dentry_open+0x2ec/0xbf0
[ 1016.242189] vfs_open+0x8c/0xc0
[ 1016.242862] path_openat+0x694/0x3310
[ 1016.243641] do_filp_open+0x13c/0x200
[ 1016.244402] do_sys_open+0x1dc/0x320
[ 1016.245181] __arm64_sys_openat+0x88/0xc8
[ 1016.246031] el0_svc_handler+0x14c/0x360
[ 1016.246867] el0_svc+0x8/0xc
[ 1016.247472] Code: d2c40001 f2fbffe1 d343fc02 f944c2b5 (38e16841)
[ 1016.248794] ---[ end trace 1e09f33962be79ce ]---
[ 1016.366872] printk: systemd: 46 output lines suppressed due to ratelimiting
[ 1016.369964] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[ 1016.371385] SMP: stopping secondary CPUs
[ 1016.372226] Kernel Offset: 0x141d0000 from 0xffff200010000000
[ 1016.373378] PHYS_OFFSET: 0xffffc9e780000000
[ 1016.374242] CPU features: 0x0002,23800438
[ 1016.375049] Memory Limit: none
[ 1016.375702] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

--
Catalin

2019-06-25 19:23:14

by Ard Biesheuvel

[permalink] [raw]
Subject: Re: [PATCH] arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE

On Tue, 25 Jun 2019 at 18:03, Catalin Marinas <[email protected]> wrote:
>
> On Tue, Jun 25, 2019 at 05:42:49PM +0200, Ard Biesheuvel wrote:
> > On Tue, 25 Jun 2019 at 17:39, Catalin Marinas <[email protected]> wrote:
> > > On Mon, Jun 24, 2019 at 12:06:18PM +0200, Ard Biesheuvel wrote:
> > > > On Mon, 24 Jun 2019 at 11:57, Will Deacon <[email protected]> wrote:
> > > > > Thanks for having a look. It could be that we've fixed the issue Catalin was
> > > > > running into in the past -- he was going to see if the problem persists with
> > > > > mainline, since it was frequent enough that it was causing us to ignore the
> > > > > results from our testing infrastructure when RANDOMIZE_BASE=y.
> > > >
> > > > I had no idea this was the case. I can look into it if we are still
> > > > seeing failures.
> > >
> > > I've seen the panic below with 5.2-rc1, defconfig + RANDOMIZE_BASE=y in
> > > a guest on TX2. It takes a few tries to trigger just with kaslr,
> > > enabling lots of other DEBUG_* options makes the failures more
> > > deterministic. I can't really say it's kaslr's fault here, only that I
> > > used to consistently get it in this configuration. For some reason, I
> > > can no longer reproduce it on arm64 for-next/core (or maybe it just
> > > takes more tries and my script doesn't catch this).
> > >
> > > The fault is in the ip_tables module, the __this_cpu_read in
> > > xt_write_recseq_begin() inlined in ipt_do_table(). The disassembled
> > > sequence in my build:
> > >
> > > 0000000000000188 <ipt_do_table>:
> > > ...
> > > 258: d538d080 mrs x0, tpidr_el1
> > > 25c: aa1303f9 mov x25, x19
> > > 260: b8606b34 ldr w20, [x25, x0]
> >
> > This was fixed recently by
> >
> > arm64/kernel: kaslr: reduce module randomization range to 2 GB
> >
> > (and arm64/module: deal with ambiguity in PRELxx relocation ranges to
> > some extent)
>
> Thanks. This explains it.
>
> And another weird case that triggers only with 64K pages, KASan and
> KASLR combination (guest on TX2). My test script modprobes all the
> modules it finds installed (including some test kernel modules like lock
> torture). At some point during modprobing, vmalloc trips over the
> WARN_ON(!pte_none(*pte)) in vmap_pte_range():
>

When KASAN and KASLR are both enabled, modules are allocated in the
dedicated module window, since the vmalloc space is already shadowed
by KASAN zero pages and so the modules must be kept out of it. since
they have their own real shadow pages that are allocated on demand.

Looking at the backtrace, it seems like the failure may be due to the
shadow space clashing, probably because the top of the module region
exceeds MODULES_END.

Does the below help at all? (patch soup courtesy of gmail, apologies)


diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c
index f713e2fc4d75..7e94e1f948c9 100644
--- a/arch/arm64/kernel/module.c
+++ b/arch/arm64/kernel/module.c
@@ -32,6 +32,7 @@

void *module_alloc(unsigned long size)
{
+ u64 module_alloc_end = module_alloc_base + MODULES_VSIZE;
gfp_t gfp_mask = GFP_KERNEL;
void *p;

@@ -39,9 +40,11 @@ void *module_alloc(unsigned long size)
if (IS_ENABLED(CONFIG_ARM64_MODULE_PLTS))
gfp_mask |= __GFP_NOWARN;

+ if (IS_ENABLED(CONFIG_KASAN))
+ module_alloc_end = MODULES_END;
+
p = __vmalloc_node_range(size, MODULE_ALIGN, module_alloc_base,
- module_alloc_base + MODULES_VSIZE,
- gfp_mask, PAGE_KERNEL_EXEC, 0,
+ module_alloc_end, gfp_mask, PAGE_KERNEL_EXEC, 0,
NUMA_NO_NODE, __builtin_return_address(0));

if (!p && IS_ENABLED(CONFIG_ARM64_MODULE_PLTS) &&