Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <f309313b-2bf0-f740-e372-06f20069e7e6@linux.ibm.com>
Date:   Wed, 25 May 2022 09:23:19 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.0
Subject: Re: [PATCH v8 17/22] vfio-pci/zdev: add open/close device hooks
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     linux-s390@vger.kernel.org, alex.williamson@redhat.com,
        cohuck@redhat.com, schnelle@linux.ibm.com, farman@linux.ibm.com,
        pmorel@linux.ibm.com, borntraeger@linux.ibm.com, hca@linux.ibm.com,
        gor@linux.ibm.com, gerald.schaefer@linux.ibm.com,
        agordeev@linux.ibm.com, svens@linux.ibm.com, frankja@linux.ibm.com,
        david@redhat.com, imbrenda@linux.ibm.com, vneethv@linux.ibm.com,
        oberpar@linux.ibm.com, freude@linux.ibm.com, thuth@redhat.com,
        pasic@linux.ibm.com, pbonzini@redhat.com, corbet@lwn.net,
        kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-doc@vger.kernel.org
References: <20220524185907.140285-1-mjrosato@linux.ibm.com>
 <20220524185907.140285-18-mjrosato@linux.ibm.com>
 <20220524210815.GB1343366@nvidia.com>
From:   Matthew Rosato <mjrosato@linux.ibm.com>
In-Reply-To: <20220524210815.GB1343366@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 5/24/22 5:08 PM, Jason Gunthorpe wrote:
> On Tue, May 24, 2022 at 02:59:02PM -0400, Matthew Rosato wrote:
>> During vfio-pci open_device, pass the KVM associated with the vfio group
>> (if one exists).  This is needed in order to pass a special indicator
>> (GISA) to firmware to allow zPCI interpretation facilities to be used
>> for only the specific KVM associated with the vfio-pci device.  During
>> vfio-pci close_device, unregister the notifier.
>>
>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>>   arch/s390/include/asm/pci.h      |  2 ++
>>   drivers/vfio/pci/vfio_pci_core.c | 11 ++++++++++-
>>   drivers/vfio/pci/vfio_pci_zdev.c | 27 +++++++++++++++++++++++++++
>>   include/linux/vfio_pci_core.h    | 12 ++++++++++++
>>   4 files changed, 51 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
>> index 85eb0ef9d4c3..67fbce1ea0c9 100644
>> +++ b/arch/s390/include/asm/pci.h
>> @@ -5,6 +5,7 @@
>>   #include <linux/pci.h>
>>   #include <linux/mutex.h>
>>   #include <linux/iommu.h>
>> +#include <linux/notifier.h>
>>   #include <linux/pci_hotplug.h>
>>   #include <asm-generic/pci.h>
>>   #include <asm/pci_clp.h>
>> @@ -195,6 +196,7 @@ struct zpci_dev {
>>   	struct s390_domain *s390_domain; /* s390 IOMMU domain data */
>>   	struct kvm_zdev *kzdev;
>>   	struct mutex kzdev_lock;
>> +	struct notifier_block nb; /* vfio notifications */
> 
> This is obsolete now right? Same for the #include ?

Of course, I forgot to remove them...  Will do

> 
>> @@ -418,6 +424,9 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
>>   
>>   	vdev->needs_reset = true;
>>   
>> +	if (vfio_pci_zdev_release(vdev))
>> +		pci_info(pdev, "%s: Couldn't restore zPCI state\n", __func__);
>> +
>>   	/*
>>   	 * If we have saved state, restore it.  If we can reset the device,
>>   	 * even better.  Resetting with current state seems better than
>> diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_zdev.c
>> index ea4c0d2b0663..d0df85c8b204 100644
>> +++ b/drivers/vfio/pci/vfio_pci_zdev.c
>> @@ -11,6 +11,7 @@
>>   #include <linux/uaccess.h>
>>   #include <linux/vfio.h>
>>   #include <linux/vfio_zdev.h>
>> +#include <linux/kvm_host.h>
>>   #include <asm/pci_clp.h>
>>   #include <asm/pci_io.h>
>>   
>> @@ -136,3 +137,29 @@ int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
>>   
>>   	return ret;
>>   }
>> +
>> +int vfio_pci_zdev_open(struct vfio_pci_core_device *vdev)
>> +{
>> +	struct zpci_dev *zdev = to_zpci(vdev->pdev);
>> +
>> +	if (!zdev)
>> +		return -ENODEV;
>> +
>> +	if (!vdev->vdev.kvm)
>> +		return 0;
>> +
>> +	return kvm_s390_pci_register_kvm(zdev, vdev->vdev.kvm);
>> +}
>> +
>> +int vfio_pci_zdev_release(struct vfio_pci_core_device *vdev)
>> +{
>> +	struct zpci_dev *zdev = to_zpci(vdev->pdev);
> 
> Keeping these functions named open_device/close_device wouuld probably
> be clearer

Agreed, will rename vfio_pci_zdev_{open,close}_device

> 
>> +	if (!zdev)
>> +		return -ENODEV;
>> +
>> +	if (!vdev->vdev.kvm)
>> +		return 0;
>> +
>> +	return kvm_s390_pci_unregister_kvm(zdev);
>> +}
> 
> Again this cannot fail, you should make it return void, not ignore the
> failure - or at least push the ignoring the failure down to the place
> that is causing this.

I'll move that handling into kvm_s390_pci_unregister_kvm and make 
vfio_pci_zdev_close_device return void / will remove the check for the 
rc in vfio_pci_core_disable.

> 
> Otherwise it looks fine to me, thanks
> 

Thanks