Received-SPF: pass (google.com: domain of linux-kernel+bounces-180261-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Wed, 15 May 2024 11:30:06 -0700
From: Kees Cook <kees@kernel.org>
To: Ard Biesheuvel <ardb@kernel.org>, "Chaney, Ben" <bchaney@akamai.com>,
 Kees Cook <keescook@chromium.org>
CC: "gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
 "linux-efi@vger.kernel.org" <linux-efi@vger.kernel.org>,
 "stable@vger.kernel.org" <stable@vger.kernel.org>,
 "bp@alien8.de" <bp@alien8.de>,
 "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
 "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
 "mingo@redhat.com" <mingo@redhat.com>,
 "tglx@linutronix.de" <tglx@linutronix.de>,
 "Tottenham, Max" <mtottenh@akamai.com>, "Hunt, Joshua" <johunt@akamai.com>,
 "Galaxy, Michael" <mgalaxy@akamai.com>
Subject: Re: Regression in 6.1.81: Missing memory in pmem device
User-Agent: K-9 Mail for Android
In-Reply-To: <CAMj1kXE2ZvaKout=nSfv08Hn5yvf8SRGhQeTikZcUeQOmyDgnw@mail.gmail.com>
References: <FA5F6719-8824-4B04-803E-82990E65E627@akamai.com> <CAMj1kXE2ZvaKout=nSfv08Hn5yvf8SRGhQeTikZcUeQOmyDgnw@mail.gmail.com>
Message-ID: <742E72A5-4792-4B72-B556-22929BBB1AD9@kernel.org>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable


On May 15, 2024 10:42:49 AM PDT, Ard Biesheuvel <ardb@kernel=2Eorg> wrote:
>(cc Kees)
>
>On Wed, 15 May 2024 at 19:32, Chaney, Ben <bchaney@akamai=2Ecom> wrote:
>>
>> Hello,
>>                 I encountered an issue when upgrading to 6=2E1=2E89 fro=
m 6=2E1=2E77=2E This upgrade caused a breakage in emulated persistent memor=
y=2E Significant amounts of memory are missing from a pmem device:
>>
>> fdisk -l /dev/pmem*
>> Disk /dev/pmem0: 355=2E9 GiB, 382117871616 bytes, 746323968 sectors
>> Units: sectors of 1 * 512 =3D 512 bytes
>> Sector size (logical/physical): 512 bytes / 4096 bytes
>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>>
>> Disk /dev/pmem1: 25=2E38 GiB, 27246198784 bytes, 53215232 sectors
>> Units: sectors of 1 * 512 =3D 512 bytes
>> Sector size (logical/physical): 512 bytes / 4096 bytes
>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>>
>>         The memmap parameter that created these pmem devices is =E2=80=
=9Cmemmap=3D364416M!28672M,367488M!419840M=E2=80=9D, which should cause a m=
uch larger amount of memory to be allocated to /dev/pmem1=2E The amount of =
missing memory and the device it is missing from is randomized on each rebo=
ot=2E There is some amount of memory missing in almost all cases, but not 1=
00% of the time=2E Notably, the memory that is missing from these devices i=
s not reclaimed by the system for general use=2E This system in question ha=
s 768GB of memory split evenly across two NUMA nodes=2E
>>
>>         When the error occurs, there are also the following error messa=
ges showing up in dmesg:
>>
>> [    5=2E318317] nd_pmem namespace1=2E0: [mem 0x5c2042c000-0x5ff7ffffff=
 flags 0x200] misaligned, unable to map
>> [    5=2E335073] nd_pmem: probe of namespace1=2E0 failed with error -95
>>
>>         Bisection implicates 2dfaeac3f38e4e550d215204eedd97a061fdc118 a=
s the patch that first caused the issue=2E I believe the cause of the issue=
 is that the EFI stub is randomizing the location of the decompressed kerne=
l without accounting for the memory map, and it is clobbering some of the m=
emory that has been reserved for pmem=2E
>>
>
>Does using 'nokaslr' on the kernel command line work around this?
>
>I think in this particular case, we could just disable physical KASLR
>(but retain virtual KASLR) if memmap=3D appears on the kernel command
>line, on the basis that emulated persistent memory is somewhat of a
>niche use case, and physical KASLR is not as important as virtual
>KASLR (which shouldn't be implicated in this)=2E

Yeah, that seems reasonable to me=2E As long as we put a notice to dmesg t=
hat physical ASLR was disabled due to memmap's physical reservation=2E If t=
his usage becomes more common, we should find a better way, though=2E=20

This reminds me a bit of the work Steve has been exploring:
https://lore=2Ekernel=2Eorg/all/20240509163310=2E2aa0b2e1@rorschach=2Eloca=
l=2Ehome/


--=20
Kees Cook