Message-ID: <eea46eb2-c14e-3bc1-d8e4-b6b28c677fe2@linux.ibm.com>
Date:   Wed, 8 Dec 2021 10:33:21 -0500
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.3.0
Subject: Re: [PATCH 07/32] s390/pci: externalize the SIC operation controls
 and routine
Content-Language: en-US
To:     Niklas Schnelle <schnelle@linux.ibm.com>,
        Christian Borntraeger <borntraeger@linux.ibm.com>,
        linux-s390@vger.kernel.org
Cc:     alex.williamson@redhat.com, cohuck@redhat.com,
        farman@linux.ibm.com, pmorel@linux.ibm.com, hca@linux.ibm.com,
        gor@linux.ibm.com, gerald.schaefer@linux.ibm.com,
        agordeev@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com,
        imbrenda@linux.ibm.com, vneethv@linux.ibm.com,
        oberpar@linux.ibm.com, freude@linux.ibm.com, thuth@redhat.com,
        pasic@linux.ibm.com, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org
References: <20211207205743.150299-1-mjrosato@linux.ibm.com>
 <20211207205743.150299-8-mjrosato@linux.ibm.com>
 <bc3b60f7-833d-6d50-dcd0-b102a190c69d@linux.ibm.com>
 <614215b5aa14102c7b43913b234463199401a156.camel@linux.ibm.com>
From:   Matthew Rosato <mjrosato@linux.ibm.com>
In-Reply-To: <614215b5aa14102c7b43913b234463199401a156.camel@linux.ibm.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 12/8/21 8:53 AM, Niklas Schnelle wrote:
> On Wed, 2021-12-08 at 14:09 +0100, Christian Borntraeger wrote:
>> Am 07.12.21 um 21:57 schrieb Matthew Rosato:
>>> A subsequent patch will be issuing SIC from KVM -- export the necessary
>>> routine and make the operation control definitions available from a header.
>>> Because the routine will now be exported, let's swap the purpose of
>>> zpci_set_irq_ctrl and __zpci_set_irq_ctrl, leaving the latter as a static
>>> within pci_irq.c only for SIC calls that don't specify an iib.
>>
>> Maybe it would be simpler to export the __ version instead of renaming everything.
>> Whatever Niklas prefers.
> 
> See below I think it's just not worth it having both variants at all.
> 
>>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>>> ---
>>>    arch/s390/include/asm/pci_insn.h | 17 +++++++++--------
>>>    arch/s390/pci/pci_insn.c         |  3 ++-
>>>    arch/s390/pci/pci_irq.c          | 28 ++++++++++++++--------------
>>>    3 files changed, 25 insertions(+), 23 deletions(-)
>>>
>>> diff --git a/arch/s390/include/asm/pci_insn.h b/arch/s390/include/asm/pci_insn.h
>>> index 61cf9531f68f..5331082fa516 100644
>>> --- a/arch/s390/include/asm/pci_insn.h
>>> +++ b/arch/s390/include/asm/pci_insn.h
>>> @@ -98,6 +98,14 @@ struct zpci_fib {
>>>    	u32 gd;
>>>    } __packed __aligned(8);
>>>    
>>> +/* Set Interruption Controls Operation Controls  */
>>> +#define	SIC_IRQ_MODE_ALL		0
>>> +#define	SIC_IRQ_MODE_SINGLE		1
>>> +#define	SIC_IRQ_MODE_DIRECT		4
>>> +#define	SIC_IRQ_MODE_D_ALL		16
>>> +#define	SIC_IRQ_MODE_D_SINGLE		17
>>> +#define	SIC_IRQ_MODE_SET_CPU		18
>>> +
>>>    /* directed interruption information block */
>>>    struct zpci_diib {
>>>    	u32 : 1;
>>> @@ -134,13 +142,6 @@ int __zpci_store(u64 data, u64 req, u64 offset);
>>>    int zpci_store(const volatile void __iomem *addr, u64 data, unsigned long len);
>>>    int __zpci_store_block(const u64 *data, u64 req, u64 offset);
>>>    void zpci_barrier(void);
>>> -int __zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib);
>>> -
>>> -static inline int zpci_set_irq_ctrl(u16 ctl, u8 isc)
>>> -{
>>> -	union zpci_sic_iib iib = {{0}};
>>> -
>>> -	return __zpci_set_irq_ctrl(ctl, isc, &iib);
>>> -}
>>> +int zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib);
> 
> Since the __zpci_set_irq_ctrl() was already non static/inline the above
> inline to non-inline change shouldn't make a performance difference.
> 
> Looking at this makes me wonder though. Wouldn't it make sense to just
> have the zpci_set_irq_ctrl() function inline in the header. Its body is
> a single instruction inline asm plus a test_facility(). The latter by
> the way I think also looks rather out of place there considering we
> call zpci_set_irq_ctrl() in the interrupt handler and facilities can't
> go away so it's pretty silly to check for it on every single
> interrupt.. unless I'm totally missing something.

This test_facility isn't new to this patch, it was added via

commit 48070c73058be6de9c0d754d441ed7092dfc8f12
Author: Christian Borntraeger <borntraeger@de.ibm.com>
Date:   Mon Oct 30 14:38:58 2017 +0100

     s390/pci: do not require AIS facility

It looks like in the past, we would not even initialize zpci at all if 
AIS wasn't available.  With this, we initialize PCI but only do the SIC 
when we have AIS, which makes sense.

So for this patch, the sane thing to do is probably just keep the 
test_facility() in place and move to header, inline.

Maybe there's a subsequent optimization to be made (setup a static key 
like have_mio vs doing test_facility all the time?)

> 
>>>    
>>>    #endif
>>> diff --git a/arch/s390/pci/pci_insn.c b/arch/s390/pci/pci_insn.c
>>> index 28d863aaafea..d1a8bd43ce26 100644
>>> --- a/arch/s390/pci/pci_insn.c
>>> +++ b/arch/s390/pci/pci_insn.c
>>> @@ -97,7 +97,7 @@ int zpci_refresh_trans(u64 fn, u64 addr, u64 range)
>>>    }
>>>    
>>>    /* Set Interruption Controls */
>>> -int __zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib)
>>> +int zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib)
>>>    {
>>>    	if (!test_facility(72))
>>>    		return -EIO;
>>> @@ -108,6 +108,7 @@ int __zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib)
>>>    
>>>    	return 0;
>>>    }
>>> +EXPORT_SYMBOL_GPL(zpci_set_irq_ctrl);
>>>    
>>>    /* PCI Load */
>>>    static inline int ____pcilg(u64 *data, u64 req, u64 offset, u8 *status)
>>> diff --git a/arch/s390/pci/pci_irq.c b/arch/s390/pci/pci_irq.c
>>> index dfd4f3276a6d..6b29e39496d1 100644
>>> --- a/arch/s390/pci/pci_irq.c
>>> +++ b/arch/s390/pci/pci_irq.c
>>> @@ -15,13 +15,6 @@
>>>    
>>>    static enum {FLOATING, DIRECTED} irq_delivery;
>>>    
>>> -#define	SIC_IRQ_MODE_ALL		0
>>> -#define	SIC_IRQ_MODE_SINGLE		1
>>> -#define	SIC_IRQ_MODE_DIRECT		4
>>> -#define	SIC_IRQ_MODE_D_ALL		16
>>> -#define	SIC_IRQ_MODE_D_SINGLE		17
>>> -#define	SIC_IRQ_MODE_SET_CPU		18
>>> -
>>>    /*
>>>     * summary bit vector
>>>     * FLOATING - summary bit per function
>>> @@ -145,6 +138,13 @@ static int zpci_set_irq_affinity(struct irq_data *data, const struct cpumask *de
>>>    	return IRQ_SET_MASK_OK;
>>>    }
>>>    
>>> +static inline int __zpci_set_irq_ctrl(u16 ctl, u8 isc)
>>> +{
>>> +	union zpci_sic_iib iib = {{0}};
>>> +
>>> +	return zpci_set_irq_ctrl(ctl, isc, &iib);
>>> +}
>>> +
> 
> I would be totally fine and slighlt prefer to have the 0 iib repeated
> at those 3 call sites that don't need it. On first glance that should
> come out to pretty much the same number of lines of code and it removes
> the potential confusion of swapping the __ prefixed and non-prefixed
> variants. What do you think?

Sure, I can do that.

> 
>>>    static struct irq_chip zpci_irq_chip = {
>>>    	.name = "PCI-MSI",
>>>    	.irq_unmask = pci_msi_unmask_irq,
>>> @@ -165,7 +165,7 @@ static void zpci_handle_cpu_local_irq(bool rescan)
>>>
> ---8<---
>