Message-ID: <a53b6402cefdef7645d1771a8b74782689b4e6dc.camel@linux.ibm.com>
Subject: Re: [PATCH 07/32] s390/pci: externalize the SIC operation controls
 and routine
From:   Niklas Schnelle <schnelle@linux.ibm.com>
To:     Matthew Rosato <mjrosato@linux.ibm.com>,
        Christian Borntraeger <borntraeger@linux.ibm.com>,
        linux-s390@vger.kernel.org
Cc:     alex.williamson@redhat.com, cohuck@redhat.com,
        farman@linux.ibm.com, pmorel@linux.ibm.com, hca@linux.ibm.com,
        gor@linux.ibm.com, gerald.schaefer@linux.ibm.com,
        agordeev@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com,
        imbrenda@linux.ibm.com, vneethv@linux.ibm.com,
        oberpar@linux.ibm.com, freude@linux.ibm.com, thuth@redhat.com,
        pasic@linux.ibm.com, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org
Date:   Wed, 08 Dec 2021 16:59:07 +0100
In-Reply-To: <eea46eb2-c14e-3bc1-d8e4-b6b28c677fe2@linux.ibm.com>
References: <20211207205743.150299-1-mjrosato@linux.ibm.com>
         <20211207205743.150299-8-mjrosato@linux.ibm.com>
         <bc3b60f7-833d-6d50-dcd0-b102a190c69d@linux.ibm.com>
         <614215b5aa14102c7b43913b234463199401a156.camel@linux.ibm.com>
         <eea46eb2-c14e-3bc1-d8e4-b6b28c677fe2@linux.ibm.com>
Content-Type: text/plain; charset="UTF-8"
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Wed, 2021-12-08 at 10:33 -0500, Matthew Rosato wrote:
> On 12/8/21 8:53 AM, Niklas Schnelle wrote:
> > On Wed, 2021-12-08 at 14:09 +0100, Christian Borntraeger wrote:
> > > Am 07.12.21 um 21:57 schrieb Matthew Rosato:
> > > > A subsequent patch will be issuing SIC from KVM -- export the necessary
> > > > routine and make the operation control definitions available from a header.
> > > > Because the routine will now be exported, let's swap the purpose of
> > > > zpci_set_irq_ctrl and __zpci_set_irq_ctrl, leaving the latter as a static
> > > > within pci_irq.c only for SIC calls that don't specify an iib.
> > > 
> > > Maybe it would be simpler to export the __ version instead of renaming everything.
> > > Whatever Niklas prefers.
> > 
> > See below I think it's just not worth it having both variants at all.
> > 
> > > > Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> > > > ---
> > > >    arch/s390/include/asm/pci_insn.h | 17 +++++++++--------
> > > >    arch/s390/pci/pci_insn.c         |  3 ++-
> > > >    arch/s390/pci/pci_irq.c          | 28 ++++++++++++++--------------
> > > >    3 files changed, 25 insertions(+), 23 deletions(-)
> > > > 
> > > > diff --git a/arch/s390/include/asm/pci_insn.h b/arch/s390/include/asm/pci_insn.h
> > > > index 61cf9531f68f..5331082fa516 100644
> > > > --- a/arch/s390/include/asm/pci_insn.h
> > > > +++ b/arch/s390/include/asm/pci_insn.h
> > > > @@ -98,6 +98,14 @@ struct zpci_fib {
> > > >    	u32 gd;
> > > >    } __packed __aligned(8);
> > > >    
> > > > +/* Set Interruption Controls Operation Controls  */
> > > > +#define	SIC_IRQ_MODE_ALL		0
> > > > +#define	SIC_IRQ_MODE_SINGLE		1
> > > > +#define	SIC_IRQ_MODE_DIRECT		4
> > > > +#define	SIC_IRQ_MODE_D_ALL		16
> > > > +#define	SIC_IRQ_MODE_D_SINGLE		17
> > > > +#define	SIC_IRQ_MODE_SET_CPU		18
> > > > +
> > > >    /* directed interruption information block */
> > > >    struct zpci_diib {
> > > >    	u32 : 1;
> > > > @@ -134,13 +142,6 @@ int __zpci_store(u64 data, u64 req, u64 offset);
> > > >    int zpci_store(const volatile void __iomem *addr, u64 data, unsigned long len);
> > > >    int __zpci_store_block(const u64 *data, u64 req, u64 offset);
> > > >    void zpci_barrier(void);
> > > > -int __zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib);
> > > > -
> > > > -static inline int zpci_set_irq_ctrl(u16 ctl, u8 isc)
> > > > -{
> > > > -	union zpci_sic_iib iib = {{0}};
> > > > -
> > > > -	return __zpci_set_irq_ctrl(ctl, isc, &iib);
> > > > -}
> > > > +int zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib);
> > 
> > Since the __zpci_set_irq_ctrl() was already non static/inline the above
> > inline to non-inline change shouldn't make a performance difference.
> > 
> > Looking at this makes me wonder though. Wouldn't it make sense to just
> > have the zpci_set_irq_ctrl() function inline in the header. Its body is
> > a single instruction inline asm plus a test_facility(). The latter by
> > the way I think also looks rather out of place there considering we
> > call zpci_set_irq_ctrl() in the interrupt handler and facilities can't
> > go away so it's pretty silly to check for it on every single
> > interrupt.. unless I'm totally missing something.
> 
> This test_facility isn't new to this patch

Yeah I got that part, your patch just made me look.

> , it was added via
> 
> commit 48070c73058be6de9c0d754d441ed7092dfc8f12
> Author: Christian Borntraeger <borntraeger@de.ibm.com>
> Date:   Mon Oct 30 14:38:58 2017 +0100
> 
>      s390/pci: do not require AIS facility
> 
> It looks like in the past, we would not even initialize zpci at all if 
> AIS wasn't available.  With this, we initialize PCI but only do the SIC 
> when we have AIS, which makes sense.

Ah yes I guess that is the something I was missing. I was wondering why
that wasn't just tested for during init.

> 
> So for this patch, the sane thing to do is probably just keep the 
> test_facility() in place and move to header, inline.

Yes sounds good.

> 
> Maybe there's a subsequent optimization to be made (setup a static key 
> like have_mio vs doing test_facility all the time?)

Yeah, looking again more closely at test_facilities() it's probably not
that expensive either I'll do some tests. Maybe we can also just add a
comment and a normal unlikely() macro since with this series KVM would
also support AIS, correct?

> 

---8<---