Received-SPF: pass (google.com: domain of linux-kernel+bounces-121543-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Wed, 27 Mar 2024 10:33:20 -0500
From: Steve Wahl <steve.wahl@hpe.com>
To: "Eric W. Biederman" <ebiederm@xmission.com>
Cc: Steve Wahl <steve.wahl@hpe.com>, Russ Anderson <rja@hpe.com>,
        Ingo Molnar <mingo@kernel.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Andy Lutomirski <luto@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>,
        Borislav Petkov <bp@alien8.de>, x86@kernel.org,
        "H. Peter Anvin" <hpa@zytor.com>, linux-kernel@vger.kernel.org,
        Linux regressions mailing list <regressions@lists.linux.dev>,
        Pavin Joseph <me@pavinjoseph.com>, stable@vger.kernel.org,
        Eric Hagberg <ehagberg@gmail.com>, Simon Horman <horms@verge.net.au>,
        Dave Young <dyoung@redhat.com>, Sarah Brofeldt <srhb@dbc.dk>,
        Dimitri Sivanich <sivanich@hpe.com>
Subject: Re: [PATCH] x86/mm/ident_map: Use full gbpages in identity maps
 except on UV platform.
Message-ID: <ZgQ8Ej-MLlNJR6wn@swahl-home.5wahls.com>
References: <20240322162135.3984233-1-steve.wahl@hpe.com>
 <ZgABC1oQ9YJW6Bw3@gmail.com>
 <20240325020334.GA10309@hpe.com>
 <87o7b273p2.fsf@email.froward.int.ebiederm.org>
 <ZgHTXvCQr6ycbVzp@swahl-home.5wahls.com>
 <87r0fv6ddb.fsf@email.froward.int.ebiederm.org>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87r0fv6ddb.fsf@email.froward.int.ebiederm.org>

On Wed, Mar 27, 2024 at 07:57:52AM -0500, Eric W. Biederman wrote:
> Steve Wahl <steve.wahl@hpe.com> writes:
> 
> > On Mon, Mar 25, 2024 at 10:04:41AM -0500, Eric W. Biederman wrote:
> >> Russ Anderson <rja@hpe.com> writes:
> >> > Steve can certainly merge his two patches and resubmit, to replace the
> >> > reverted original patch.  He should be on in the morning to speak for
> >> > himself.
> >> 
> >> I am going to push back and suggest that this is perhaps a bug in the
> >> HPE UV systems firmware not setting up the cpus memory type range
> >> registers correctly.
> >> 
> >> Unless those systems are using new fangled cpus that don't have 16bit
> >> and 32bit support, and don't implement memory type range registers,
> >> I don't see how something that only affects HPE UV systems could be
> >> anything except an HPE UV specific bug.
> >
> > Eric,
> >
> > I took the time to communicate with others in the company who know
> > this stuff better than I do before replying on this.
> >
> > One of the problems with using the MTRRs for this is that there are
> > simply not enough of them.  The MTRRs size/alignment requirements mean
> > that more than one entry would be required per reserved region, and we
> > need one reserved region per socket on systems that currently can go
> > up to 32 sockets.  (In case you would think to ask, the reserved
> > regions also cannot be made contiguous.)
> >
> > So MTRRs will not work to keep speculation out of our reserved memory
> > regions.
> >
> > Let me know if you need more information from us on this.
> 
> Thanks for this.
> 
> Do you know if there are enough MTRRs for the first 4GB?

I don't personally know all the details of how BIOS chooses to place
things, but I suspect that might be true.  The restricted spaces
usually end up at the end of the address range for a particular node,
and 4GB would be in the early part of node 0.  If the conversation
develops further along these lines, I can find out more definitively.

> I am curious if kexec should even consider going into 32bit mode without
> page tables or even into 16bit mode on such a system.  Or if such a
> system will always require using page tables.

Unless I'm mistaken, wouldn't that put a pretty heavy restriction on
where the kdump kernel could be located?  Or the target region for
KASLR?  

> If you don't have enough MTRRs on a big NUMA system I think it is
> perfectly understandable, to need to use the page tables.
> 
> Please include this the fact that splitting GBpages is necessary because
> of a lack of MTRRs in the change description.

OK.

> Given that it is the lack of MTRRs on a large NUMA system that make the
> change necessary.   The goes from a pure bug fix change to a change to
> accommodate systems without enough MTRRs.
> 
> That information makes it more understandable why older systems (at
> least in the case of kexec) might not be ok with the change.  As for
> older systems their MTRRs are sufficient and thus they can use fewer
> page table entries.  Allowing for use of larger TLB entries.

That last paragraph doesn't match what I think is happening.

At least from my point of view, that some systems aren't OK with the
change has nothing to do with MTRRs or TLB page size.  They simply
require the extra "slop" of GB pages, implicitly adding a full GB of
space around any smaller space requested by map_acpi_tables().

The systems that failed with my original change also failed on earlier
kernels when nogbpages was added to the kernel command line.  That
creates the identity map using 2M pages for everything, with no GB
page "slop".   I'm pretty sure these systems will continue to fail with
"nogbpages" enabled.

For one debug-kernel cycle on Pavin's system I added in hard-coded
requests to explicitly add back in the areas that not being sloppy had
excluded, and that brought kexec back to functioning; which further
proves my point.  

I wanted to be sure you understood this in case it has any effect on
what you think should be done.

--> Steve

-- 
Steve Wahl, Hewlett Packard Enterprise