Received-SPF: pass (google.com: domain of linux-kernel+bounces-117792-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Mon, 25 Mar 2024 14:15:27 -0500
From: Steve Wahl <steve.wahl@hpe.com>
To: Dave Hansen <dave.hansen@intel.com>
Cc: Steve Wahl <steve.wahl@hpe.com>, Dave Hansen <dave.hansen@linux.intel.com>,
        Andy Lutomirski <luto@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>,
        Borislav Petkov <bp@alien8.de>, x86@kernel.org,
        "H. Peter Anvin" <hpa@zytor.com>, linux-kernel@vger.kernel.org,
        Linux regressions mailing list <regressions@lists.linux.dev>,
        Pavin Joseph <me@pavinjoseph.com>, stable@vger.kernel.org,
        Eric Hagberg <ehagberg@gmail.com>, Simon Horman <horms@verge.net.au>,
        Eric Biederman <ebiederm@xmission.com>, Dave Young <dyoung@redhat.com>,
        Sarah Brofeldt <srhb@dbc.dk>, Russ Anderson <rja@hpe.com>,
        Dimitri Sivanich <sivanich@hpe.com>
Subject: Re: [PATCH] x86/mm/ident_map: Use full gbpages in identity maps
 except on UV platform.
Message-ID: <ZgHNT1bYV8KNRr1G@swahl-home.5wahls.com>
References: <20240322162135.3984233-1-steve.wahl@hpe.com>
 <8cc9e238-fa70-402f-9990-f7e391b367a9@intel.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <8cc9e238-fa70-402f-9990-f7e391b367a9@intel.com>

I understand the original has been reverted, and that my way forward
is likely to combine the original patch and the fix into a combined
patch (possibly a set).  I still think some of this conversation will
be valuable to creation of that patch, so continuing to reply.  More
below.

On Fri, Mar 22, 2024 at 04:29:27PM -0700, Dave Hansen wrote:
> On 3/22/24 09:21, Steve Wahl wrote:
> > Some systems have ACPI tables that don't include everything that needs
> > to be mapped for a successful kexec.  These systems rely on identity
> > maps that include the full gigabyte surrounding any smaller region
> > requested for kexec success.  Without this, they fail to kexec and end
> > up doing a full firmware reboot.
> 
> I'm still missing something here.  Which ACPI tables are we talking
> about?  What don't they map?  I normally don't think of ACPI _tables_ as
> "mapping" things.

I'm refering to the memory areas that are mapped in machine_kexec_64.c
in the function map_acpi_tables.  These appear to be e820 table
entries that have a type of E820_TYPE_ACPI which the kernel marks as
IORES_DESC_ACPI_TABLES, or a type of E820_TYPE_NVS that the kernel
marks as IORES_DESC_ACPI_NV_STORAGE.

The name of the function that maps them is why I refer to them as ACPI
tables.  Sorry if that is inaccurate.

> It seems like there's a theory that some ACPI table isn't mapped, but
> looking through the discussion so far I don't see a smoking gun.

I think I'm saying more that the ACPI table doesn't list everything
that needs to be mapped, not that the table itself isn't mapped.  Not
sure if that changes your picture or not.

My debuging exchanges with Pavin showed that the regions mapped within
the map_acpi_tables function were the ones that left uncovered holes
in the identity map if you don't overshoot what's requested by using
full gbpages for everything.

For his system only, I manually added hardcoded regions corresponding
to the holes that got left by using 2M pages instead of GB pages, and
kexec succeeded.

Having the list of holes-not-covered (IIRC, four of them), I could
have persued which particular holes cause kexec to fail, but I did not
because I couldn't think of a way to make use of that information.
Even knowing which additional addresses need coverage for this
particular machine, I have no way of knowing what is in those regions,
nor how to generalize to what is needed on other machines.

> Let's say the kernel has a bug and the kernel was actively not
> mapping something that it should have mapped.  The oversized 1GB
> mappings made the bug harder to hit.  If that's the case, we'll just
> be adding a hack which papers over the bug instead of fixing it
> properly.

I hope you agree that by reverting, we have now papered over that bug,
just in a different way.

If a patch that leaves this papered over except for UV systems won't
be acceptable -- that's what I intend to do, just combining my two
patches -- please let me know what my way forward should be.

> I'm kind of leaning to say that we should just revert d794734c9bbf and
> have the UV folks go back to the nogbpages until we get this properly
> sorted.

Being larger memory systems (for example, 32 socket Sapphire Rapids
systems with a full set of RAM on each socket), UV probably suffers
the most from having an extra 4K per GiB to create the identity map.

> > @@ -10,6 +10,7 @@ struct x86_mapping_info {
> >  	unsigned long page_flag;	 /* page flag for PMD or PUD entry */
> >  	unsigned long offset;		 /* ident mapping offset */
> >  	bool direct_gbpages;		 /* PUD level 1GB page support */
> > +	bool direct_gbpages_always;	 /* use 1GB pages exclusively */
> >  	unsigned long kernpg_flag;	 /* kernel pagetable flag override */
> >  };
> 
> But let's at least talk about this patch in case we decide to go forward
> with it.  We've really got two things:
> 
> 1. Can the system use gbpages in the first place?
> 2. Do the gbpages need to be exact (UV) or sloppy (everything else)?
> 
> I wouldn't refer to this at all as "always" use gbpages.  It's really a
> be-sloppy-and-paper-over-bugs mode.  They might be kernel bugs or
> firmware bugs, but they're bugs _somewhere_ right?

Do you have a concise suggestion of what you'd call it?  I could use
*_sloppy if you'd like, but I don't care much for the way that
reads.

Thanks for your time,

--> Steve Wahl
-- 
Steve Wahl, Hewlett Packard Enterprise