Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Fri, 30 Apr 2021 08:29:40 -0600
From:   Alex Williamson <alex.williamson@redhat.com>
To:     Yicong Yang <yangyicong@hisilicon.com>
Cc:     <cohuck@redhat.com>, <kvm@vger.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        "Zengtao (B)" <prime.zeng@hisilicon.com>,
        Linuxarm <linuxarm@huawei.com>
Subject: Re: [Question] Indefinitely block in the host when remove the PF
 driver
Message-ID: <20210430082940.4b0e0397@redhat.com>
In-Reply-To: <c9466e2c-385d-8298-d03c-80dcfc359f52@hisilicon.com>
References: <c9466e2c-385d-8298-d03c-80dcfc359f52@hisilicon.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Fri, 30 Apr 2021 15:57:47 +0800
Yicong Yang <yangyicong@hisilicon.com> wrote:

> When I try to remove the PF driver in the host, the process will be blocked
> if the related VF of the device is added in the Qemu as an iEP.
> 
> here's what I got in the host:
> 
> [root@localhost 0000:75:00.0]# rmmod hisi_zip
> [99760.571352] vfio-pci 0000:75:00.1: Relaying device request to user (#0)
> [99862.992099] vfio-pci 0000:75:00.1: Relaying device request to user (#10)
> [...]
> 
> and in the Qemu:
> 
> estuary:/$ lspci -tv
> -[0000:00]-+-00.0  Device 1b36:0008
>            +-01.0  Device 1af4:1000
>            +-02.0  Device 1af4:1009
>            \-03.0  Device 19e5:a251 <----- the related VF device
> estuary:/$ qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> [...]
> 
> The rmmod process will be blocked until I kill the Qemu process. That's the only way if I
> want to end the rmmod.
> 
> So my question is: is such block reasonable? If the VF devcie is occupied or doesn't
> support hotplug in the Qemu, shouldn't we fail the rmmod and return something like -EBUSY
> rather than make the host blocked indefinitely?

Where would we return -EBUSY?  pci_driver.remove() returns void.
Without blocking, I think our only option would be to kill the user
process.
 
> Add the VF under a pcie root port will avoid this. Is it encouraged to always
> add the VF under a pcie root port rather than directly add it as an iEP?

Releasing a device via the vfio request interrupt is always a
cooperative process currently, the VM needs to be configured such that
the device is capable of being unplugged and the guest needs to respond
to the ejection request.  Thanks,

Alex