Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <20190828194226.GA29967@swahl-linux> <CAKwvOdn0=7YabPD-5EUwkSoJgWjdYHY2mirM2LUz0TxZTBOf_Q@mail.gmail.com>
 <20190828221048.GB29967@swahl-linux>
In-Reply-To: <20190828221048.GB29967@swahl-linux>
From:   Nick Desaulniers <ndesaulniers@google.com>
Date:   Wed, 28 Aug 2019 15:22:13 -0700
Message-ID: <CAKwvOdnaxZuJHpbmMzdtKSZD10m3Rd52FdHeq2gvkas3XVmk7w@mail.gmail.com>
Subject: Re: Purgatory compile flag changes apparently causing Kexec
 relocation overflows
To:     Steve Wahl <steve.wahl@hpe.com>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        LKML <linux-kernel@vger.kernel.org>, russ.anderson@hpe.com,
        dimitri.sivanich@hpe.com, mike.travis@hpe.com
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, Aug 28, 2019 at 3:14 PM Steve Wahl <steve.wahl@hpe.com> wrote:
>
> On Wed, Aug 28, 2019 at 02:51:21PM -0700, Nick Desaulniers wrote:
> > On Wed, Aug 28, 2019 at 12:42 PM Steve Wahl <steve.wahl@hpe.com> wrote:
> > >
> > > Please CC me on responses to this.
> > >
> > > I normally would do more diligence on this, but the timing is such
> > > that I think it's better to get this out sooner.
> > >
> > > With the tip of the tree from https://github.com/torvalds/linux.git  (a
> > > few days old, most recent commit fetched is
> > > bb7ba8069de933d69cb45dd0a5806b61033796a3), I'm seeing "kexec: Overflow
> > > in relocation type 11 value 0x11fffd000" when I try to load a crash
> > > kernel with kdump. This seems to be caused by commit
> > > 059f801a937d164e03b33c1848bb3dca67c0b04, which changed the compiler
> > > flags used to compile purgatory.ro, apparently creating 32 bit
> > > relocations for things that aren't necessarily reachable with a 32 bit
> > > reference.  My guess is this only occurs when the crash kernel is
> > > located outside 32-bit addressable physical space.
> > >
> > > I have so far verified that the problem occurs with that commit, and
> > > does not occur with the previous commit.  For this commit, Thomas
> > > Gleixner mentioned a few of the changed flags should have been looked
> > > at twice.  I have not gone so far as to figure out which flags cause
> > > the problem.
> > >
> > > The hardware in use is a HPE Superdome Flex with 48 * 32GiB dimms
> > > (total 1536 GiB).
> > >
> > > One example of the exact error messages seen:
> > >
> > > 019-08-28T13:42:39.308110-05:00 uv4test14 kernel: [   45.137743] kexec: Overflow in relocation type 11 value 0x17f7affd000
> > > 2019-08-28T13:42:39.308123-05:00 uv4test14 kernel: [   45.137749] kexec-bzImage64: Loading purgatory failed
> >
> > Thanks for the report and sorry for the breakage.  Can you please send
> > me more information for how to precisely reproduce the issue?  I'm
> > happy to look into fixing it.
>
> Here's the details I know might be important:
>
> Since this appears to be a problem with the result of a relocation not
> fitting within 32 bits, I think the location chosen to place the crash
> kernel needs to be above 4GiB; so you need a machine with more memory
> than that.
>
> At the moment I'm running SLES 12 sp 4 as the rest of the
> environment.  rpm says kdump is kdump-0.8.16-9.2.x86_64.  I've fetched
> the kernel sources and compiled directly on this system.  I believe I
> copied the kernel config from the SLES kernel and did a make
> olddefconfig for configuration.  Made and installed the kernel from
> the kernel tree.
>
> crashkernel=512M,high is set on the command line.
>
> As the system boots, and systemd initializes kdump, it tries to load
> the crash kernel, I believe through
> /usr/lib/systemd/system/kdump.service running /lib/kdump/load.sh
> --update.
>
> Once that completes, 'systemctl status kdump' indicates a failure, and
> dmesg | grep kexec shows the error messages mentioned above.
>
> > Let me go dig up the different listed flags.  Steve, it may be fastest
> > for you to test re-adding them in your setup to see which one is
> > important.
>
> I will work through that tomorrow and let you know what I find.
>
> > Tglx, if you want to revert the above patches, I'm ok with that.  It's
> > important that we fix the issue eventually that my patches were meant
> > to address, but precisely *when* it's solved isn't critical; our
> > kernels can carry out of tree patches for now until the issue is
> > completely resolved worst case.

One point that might be more useful first would be, is a revert of:

commit b059f801a937 ("x86/purgatory: Use CFLAGS_REMOVE rather than
reset KBUILD_CFLAGS")

good enough, or must:

commit 4ce97317f41d ("x86/purgatory: Do not use __builtin_memcpy and
__builtin_memset")

be reverted additionally?  They were part of a 2 patch patchset.  I
would prefer tglx to revert as few patches as necessary if possible
(to avoid "revert of revert" soup), and I doubt the latter patch needs
to be reverted.  (Even more preferential would be a fix, with no
reverts, but whichever).
-- 
Thanks,
~Nick Desaulniers