Date:   Tue, 24 Jan 2023 01:21:24 +0000
From:   Sean Christopherson <seanjc@google.com>
To:     Erdem Aktas <erdemaktas@google.com>
Cc:     "Maciej S. Szmigiero" <mail@maciej.szmigiero.name>,
        Ackerley Tng <ackerleytng@google.com>,
        linux-kselftest@vger.kernel.org, pbonzini@redhat.com,
        isaku.yamahata@intel.com, sagis@google.com, afranji@google.com,
        runanwang@google.com, shuah@kernel.org, drjones@redhat.com,
        maz@kernel.org, bgardon@google.com, jmattson@google.com,
        dmatlack@google.com, peterx@redhat.com, oupton@google.com,
        ricarkol@google.com, yang.zhong@intel.com, wei.w.wang@intel.com,
        xiaoyao.li@intel.com, pgonda@google.com, marcorr@google.com,
        eesposit@redhat.com, borntraeger@de.ibm.com, eric.auger@redhat.com,
        wangyanan55@huawei.com, aaronlewis@google.com, vkuznets@redhat.com,
        pshier@google.com, axelrasmussen@google.com,
        zhenzhong.duan@intel.com, like.xu@linux.intel.com,
        linux-kernel@vger.kernel.org, kvm@vger.kernel.org
Subject: Re: [RFC PATCH v3 08/31] KVM: selftests: Require GCC to realign
 stacks on function entry
Message-ID: <Y88ylDFfMQNcUEw7@google.com>
References: <20230121001542.2472357-1-ackerleytng@google.com>
 <20230121001542.2472357-9-ackerleytng@google.com>
 <Y8sxjppvEnm4IBWG@google.com>
 <CAAYXXYy7=ZTCZ1LQ3_Sy39ju_xG5++dTrxi+DKGcbpJ5VJ3OuQ@mail.gmail.com>
 <99a36eed-e4e5-60ec-0f88-a33d1842a0d6@maciej.szmigiero.name>
 <Y87XnYZx1qzZOLKR@google.com>
 <CAAYXXYyqDJx4=cSy3kp7vX4VF+5z_Rtm6wPM8_o9BmHkB_T-kg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAAYXXYyqDJx4=cSy3kp7vX4VF+5z_Rtm6wPM8_o9BmHkB_T-kg@mail.gmail.com>
Precedence: bulk

On Mon, Jan 23, 2023, Erdem Aktas wrote:
> On Mon, Jan 23, 2023 at 10:53 AM Sean Christopherson <seanjc@google.com> wrote:
> >
> > On Mon, Jan 23, 2023, Maciej S. Szmigiero wrote:
> > > On 23.01.2023 19:30, Erdem Aktas wrote:
> > > > On Fri, Jan 20, 2023 at 4:28 PM Sean Christopherson <seanjc@google.com> wrote:
> > > > >
> > > > > On Sat, Jan 21, 2023, Ackerley Tng wrote:
> > > > > > Some SSE instructions assume a 16-byte aligned stack, and GCC compiles
> > > > > > assuming the stack is aligned:
> > > > > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=40838. This combination
> > > > > > results in a #GP in guests.
> > > > > >
> > > > > > Adding this compiler flag will generate an alternate prologue and
> > > > > > epilogue to realign the runtime stack, which makes selftest code
> > > > > > slower and bigger, but this is okay since we do not need selftest code
> > > > > > to be extremely performant.
> > > > >
> > > > > Huh, I had completely forgotten that this is why SSE is problematic.  I ran into
> > > > > this with the base UPM selftests and just disabled SSE.  /facepalm.
> > > > >
> > > > > We should figure out exactly what is causing a misaligned stack.  As you've noted,
> > > > > the x86-64 ABI requires a 16-byte aligned RSP.  Unless I'm misreading vm_arch_vcpu_add(),
> > > > > the starting stack should be page aligned, which means something is causing the
> > > > > stack to become unaligned at runtime.  I'd rather hunt down that something than
> > > > > paper over it by having the compiler force realignment.
> > > >
> > > > Is not it due to the 32bit execution part of the guest code at boot
> > > > time. Any push/pop of 32bit registers might make it a 16-byte
> > > > unaligned stack.
> > >
> > > 32-bit stack needs to be 16-byte aligned, too (at function call boundaries) -
> > > see [1] chapter 2.2.2 "The Stack Frame"
> >
> > And this showing up in the non-TDX selftests rules that out as the sole problem;
> > the selftests stuff 64-bit mode, i.e. don't have 32-bit boot code.
> 
> Thanks Maciej and Sean for the clarification. I was suspecting the
> hand-coded assembly part that we have for TDX tests but  it being
> happening in the non-TDX selftests disproves it.

Not necessarily, it could be both.  Goofs in the handcoded assembly and PEBKAC
on my end :-)