Received-SPF: pass (google.com: domain of linux-kernel+bounces-107497-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
From: Mikhail Malyshev <mike.malyshev@gmail.com>
To: alex.williamson@redhat.com,
	jgg@ziepe.ca,
	yi.l.liu@intel.com,
	kevin.tian@intel.com,
	tglx@linutronix.de,
	reinette.chatre@intel.com,
	stefanha@redhat.com
Cc: abhsahu@nvidia.com,
	kvm@vger.kernel.org,
	linux-kernel@vger.kernel.org,
	Mikhail Malyshev <mike.malyshev@gmail.com>
Subject: [PATCH 0/1] Reenable runtime PM for dynamically unbound devices
Date: Tue, 19 Mar 2024 12:04:09 +0000
Message-Id: <20240319120410.1477713-1-mike.malyshev@gmail.com>
Precedence: bulk
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

When trying to run a VM with PCI passthrough of intel-eth-pci ETH device
QEMU fails with "Permission denied" error. This happens only if
intel-eth-pci driver is dynamically unbound from the device using
"echo -n $DEV > /sys/bus/pci/drivers/stmmac/unbind" command. If
"vfio-pci.ids=..." is used to bind the device to vfio-pci driver and the
device is never probed by intel-eth-pci driver the problem does not occur.

When intel-eth-pci driver is dynamically unbound from the device
remove()
  intel_eth_pci_remove()
    stmmac_dvr_remove()
      pm_runtime_disable();

Later when QEMU tries to get the device file descriptor by calling
VFIO_GROUP_GET_DEVICE_FD ioctl pm_runtime_resume_and_get returns -EACCES.
It happens because dev->power.disable_depth == 1 .

vfio_group_fops_unl_ioctl(VFIO_GROUP_GET_DEVICE_FD)
  vfio_group_ioctl_get_device_fd()
    vfio_device_open()
      ret = device->ops->open_device()
        vfio_pci_open_device()
          vfio_pci_core_enable()
              ret = pm_runtime_resume_and_get();

This behavior was introduced by
commit 7ab5e10eda02 ("vfio/pci: Move the unused device into low power state with runtime PM")

This may be the case for any driver calling pm_runtime_disable() in its
remove() callback.

The case when a runtime PM may be disable for a device is not handled so we
call pm_runtime_enable() in vfio_pci_core_register_device to re-enable it.

Mikhail Malyshev (1):
  vfio/pci: Reenable runtime PM for dynamically unbound devices

 drivers/vfio/pci/vfio_pci_core.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

--
2.34.1