Received-SPF: pass (google.com: domain of linux-kernel+bounces-198756-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Precedence: bulk
Mime-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8
Date: Mon, 03 Jun 2024 18:06:34 +1000
Message-Id: <D1Q874JKY23A.3LB0GHUQQEEAW@gmail.com>
Cc: <mpe@ellerman.id.au>, <christophe.leroy@csgroup.eu>,
 <aneesh.kumar@kernel.org>, <naveen.n.rao@linux.ibm.com>, <corbet@lwn.net>,
 <linuxppc-dev@lists.ozlabs.org>, <kvm@vger.kernel.org>,
 <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
 <stable@vger.kernel.org>
Subject: Re: [PATCH v1 RESEND] arch/powerpc/kvm: Fix doorbell emulation by
 adding DPDES support
From: "Nicholas Piggin" <npiggin@gmail.com>
To: "Gautam Menghani" <gautam@linux.ibm.com>
References: <20240522084949.123148-1-gautam@linux.ibm.com>
 <D1Q54PY40E3B.22QS5DMQRA58N@gmail.com>
 <r74chlv6bgs5csvuf4nxxtylmgartvibftp3xuztyfuynqetp5@ythddpzo6yfi>
In-Reply-To: <r74chlv6bgs5csvuf4nxxtylmgartvibftp3xuztyfuynqetp5@ythddpzo6yfi>

On Mon Jun 3, 2024 at 5:09 PM AEST, Gautam Menghani wrote:
> On Mon, Jun 03, 2024 at 03:42:22PM GMT, Nicholas Piggin wrote:
> > On Wed May 22, 2024 at 6:49 PM AEST, Gautam Menghani wrote:
> > > Doorbell emulation is broken for KVM on PowerVM guests as support for
> > > DPDES was not added in the initial patch series. Due to this, a KVM o=
n
> > > PowerVM guest cannot be booted with the XICS interrupt controller as
> > > doorbells are to be setup in the initial probe path when using XICS
> > > (pSeries_smp_probe()). Add DPDES support in the host KVM code to fix
> > > doorbell emulation.
> >=20
> > This is broken when the KVM guest has SMT > 1? Or is it broken for SMT=
=3D1
> > as well? Can you explain a bit more of what breaks if it's the latter?
>
> Yes, doorbells are only setup when we use SMT>1 and interrupt controller
> is XICS. So without this patch, L2 KOP guest with XICS IC mode and SMT>1=
=20
> does not boot. SMT 1 is fine in all cases.

Okay good. Make that clear in the changelog, ideally if you can give
a recipe the reader is able to recreate is good too, (e.g., run the
guest machine with -smp 4,threads=3D4 and xive=3Doff boot parameter).

> > > Fixes: 6ccbbc33f06a ("KVM: PPC: Add helper library for Guest State Bu=
ffers")
> > > Cc: stable@vger.kernel.org
> > > Signed-off-by: Gautam Menghani <gautam@linux.ibm.com>
> > > ---
> > > v1 -> v1 resend:
> > > 1. Add the stable tag
> > >
> > >  Documentation/arch/powerpc/kvm-nested.rst     |  4 +++-
> > >  arch/powerpc/include/asm/guest-state-buffer.h |  3 ++-
> > >  arch/powerpc/include/asm/kvm_book3s.h         |  1 +
> > >  arch/powerpc/kvm/book3s_hv.c                  | 14 +++++++++++++-
> > >  arch/powerpc/kvm/book3s_hv_nestedv2.c         |  7 +++++++
> > >  arch/powerpc/kvm/test-guest-state-buffer.c    |  2 +-
> > >  6 files changed, 27 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/Documentation/arch/powerpc/kvm-nested.rst b/Documentatio=
n/arch/powerpc/kvm-nested.rst
> > > index 630602a8aa00..5defd13cc6c1 100644
> > > --- a/Documentation/arch/powerpc/kvm-nested.rst
> > > +++ b/Documentation/arch/powerpc/kvm-nested.rst
> > > @@ -546,7 +546,9 @@ table information.
> > >  +--------+-------+----+--------+----------------------------------+
> > >  | 0x1052 | 0x08  | RW |   T    | CTRL                             |
> > >  +--------+-------+----+--------+----------------------------------+
> > > -| 0x1053-|       |    |        | Reserved                         |
> > > +| 0x1053 | 0x08  | RW |   T    | DPDES                            |
> > > ++--------+-------+----+--------+----------------------------------+
> > > +| 0x1054-|       |    |        | Reserved                         |
> > >  | 0x1FFF |       |    |        |                                  |
> > >  +--------+-------+----+--------+----------------------------------+
> > >  | 0x2000 | 0x04  | RW |   T    | CR                               |
> > > diff --git a/arch/powerpc/include/asm/guest-state-buffer.h b/arch/pow=
erpc/include/asm/guest-state-buffer.h
> > > index 808149f31576..d107abe1468f 100644
> > > --- a/arch/powerpc/include/asm/guest-state-buffer.h
> > > +++ b/arch/powerpc/include/asm/guest-state-buffer.h
> > > @@ -81,6 +81,7 @@
> > >  #define KVMPPC_GSID_HASHKEYR			0x1050
> > >  #define KVMPPC_GSID_HASHPKEYR			0x1051
> > >  #define KVMPPC_GSID_CTRL			0x1052
> > > +#define KVMPPC_GSID_DPDES			0x1053
> > > =20
> > >  #define KVMPPC_GSID_CR				0x2000
> > >  #define KVMPPC_GSID_PIDR			0x2001
> > > @@ -110,7 +111,7 @@
> > >  #define KVMPPC_GSE_META_COUNT (KVMPPC_GSE_META_END - KVMPPC_GSE_META=
_START + 1)
> > > =20
> > >  #define KVMPPC_GSE_DW_REGS_START KVMPPC_GSID_GPR(0)
> > > -#define KVMPPC_GSE_DW_REGS_END KVMPPC_GSID_CTRL
> > > +#define KVMPPC_GSE_DW_REGS_END KVMPPC_GSID_DPDES
> > >  #define KVMPPC_GSE_DW_REGS_COUNT \
> > >  	(KVMPPC_GSE_DW_REGS_END - KVMPPC_GSE_DW_REGS_START + 1)
> > > =20
> > > diff --git a/arch/powerpc/include/asm/kvm_book3s.h b/arch/powerpc/inc=
lude/asm/kvm_book3s.h
> > > index 3e1e2a698c9e..10618622d7ef 100644
> > > --- a/arch/powerpc/include/asm/kvm_book3s.h
> > > +++ b/arch/powerpc/include/asm/kvm_book3s.h
> > > @@ -594,6 +594,7 @@ static inline u##size kvmppc_get_##reg(struct kvm=
_vcpu *vcpu)		\
> > > =20
> > > =20
> > >  KVMPPC_BOOK3S_VCORE_ACCESSOR(vtb, 64, KVMPPC_GSID_VTB)
> > > +KVMPPC_BOOK3S_VCORE_ACCESSOR(dpdes, 64, KVMPPC_GSID_DPDES)
> > >  KVMPPC_BOOK3S_VCORE_ACCESSOR_GET(arch_compat, 32, KVMPPC_GSID_LOGICA=
L_PVR)
> > >  KVMPPC_BOOK3S_VCORE_ACCESSOR_GET(lpcr, 64, KVMPPC_GSID_LPCR)
> > >  KVMPPC_BOOK3S_VCORE_ACCESSOR_SET(tb_offset, 64, KVMPPC_GSID_TB_OFFSE=
T)
> > > diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_h=
v.c
> > > index 35cb014a0c51..cf285e5153ba 100644
> > > --- a/arch/powerpc/kvm/book3s_hv.c
> > > +++ b/arch/powerpc/kvm/book3s_hv.c
> > > @@ -4116,6 +4116,11 @@ static int kvmhv_vcpu_entry_nestedv2(struct kv=
m_vcpu *vcpu, u64 time_limit,
> > >  	int trap;
> > >  	long rc;
> > > =20
> > > +	if (vcpu->arch.doorbell_request) {
> > > +		vcpu->arch.doorbell_request =3D 0;
> > > +		kvmppc_set_dpdes(vcpu, 1);
> > > +	}
> >=20
> > This probably looks okay... hmm, is the v1 KVM emulating doorbells
> > correctly for SMT L2 guests? I wonder if doorbell emulation isn't
> > broken there too because the L1 code looks to be passing in vc->dpdes
> > but all the POWER9 emulation code uses doorbell_request.
> >=20
>
> Yes launching SMT L2 on V1 API fails with a kernel Oops, I'll see if I
> can fix that as well.

Okay then I didn't miss something. Thanks v1 fix would be good too.
I think it should look something like putting doorbell_request into
hv_guest_state->dpdes that make the H_ENTER_NESTED call with.

For v1 you will need to restore the state back from dpdes back to
doorbell_request on exit, because the L0 doesn't keep it for you.

Thanks,
Nick