Hi Ixgbe maintainers,
We met an issue that the VF of 82599 cannot be enabled after remove and rescan the PF device.
The PCI hierarchy on our platform is like:
[...]
+-[0000:80]-+-00.0-[81]--
| +-04.0-[82]--
| +-08.0-[83]--+-00.0 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection
| | \-00.1 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection
| \-10.0-[84]--
[...]
We operated like below:
[root@localhost ~]# cat /sys/class/net/enp131s0f0/device/sriov_numvfs
0
[root@localhost ~]# echo 1 > /sys/class/net/enp131s0f0/device/sriov_numvfs # enable 1 VF
[root@localhost ~]# echo 1 > /sys/bus/pci/devices/0000:83:00.0/remove # remove the PF
[root@localhost ~]# echo 1 > /sys/bus/pci/rescan # rescan the PF
[root@localhost ~]# cat /sys/class/net/enp131s0f0/device/sriov_numvfs
0
[root@localhost ~]# echo 1 > /sys/class/net/enp131s0f0/device/sriov_numvfs # attemp to enable the VF
[ 433.568996] ixgbe 0000:83:00.0 enp131s0f0: SR-IOV enabled with 1 VFs
[ 433.639027] ixgbe 0000:83:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 433.652932] ixgbe 0000:83:00.0: can't enable 1 VFs (bus 84 out of range of [bus 83])
[ 433.661228] ixgbe 0000:83:00.0: Failed to enable PCI sriov: -12
-bash: echo: write error: Cannot allocate memory
A further investigation shows that the SRIOV offset changed after the rescan, so we cannot find
an available PCI bus (it's already occupied) for the VF device:
Before the remove:
[root@localhost ~]# lspci -vvs 83:00.0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
VF offset: 128, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000280000804000 (64-bit, prefetchable)
Region 3: Memory at 0000280000904000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
After the rescan:
[root@localhost ~]# lspci -vvs 83:00.0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy- 10BitTagReq-
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
VF offset: 384, stride: 2, Device ID: 10ed
^^^^^^^^^^^^^^
offset has changed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000280000804000 (64-bit, prefetchable)
Region 3: Memory at 0000280000904000 (64-bit, prefetchable)
We don't know why the SRIOV offset and stride changed and is there anything wrong. Any help on how
to illustrate or fix this is highly appreciated! Please let us know if more information is needed.
Thanks,
Yicong
Hi Yicong,
VF offset depends on set of ARIHeirarchy(+/-) in the PCI config.
After Power cycle this bit is set to (+). When we force the port removal
# echo 1 > /sys/bus/pci/devices/0000\:04\:00.0/remove
and port rescan
# echo 1 > /sys/bus/pci/rescan
this cause that the ARIHierarchy is set to (-) and the offet is set to 384.
Look into the "Intel® 82599 10 GbE Controller Datasheet",
chapter 7.10.2.6.1.1 ARI Mode. In mode non-ARI "1" is added to the bus and
that cause you've got an error.
During boot sequence when all physical function (PF) are initialized,
pci driver in first attempt set ARI on the first encountered PF and ignore
other. When we remove that first encountered PF which has ARI enabled
in initializaion stage, performing rescan causes that the pci driver only
take into account existing earlier PF. In result pci driver doesn't set ARI
on any of them and the offset is set to 384.
static int sriov_init(struct pci_dev *dev, int pos)
{
ctrl = 0;
list_for_each_entry(pdev, &dev->bus->devices, bus_list)
if (pdev->is_physfn) <------
goto found;
pdev = NULL;
if (pci_ari_enabled(dev->bus))
ctrl |= PCI_SRIOV_CTRL_ARI;
found:
...
It is not the problem related to the device driver, but with pci driver,
as a workaround you may use one of these options:
1. remove all PF belonging to bus, before attempt to rescan.
2. disable ARI in grub add "noari" to the kernel parameters.
# noari do not use PCIe ARI.
--
BR,
Radek