Hello,
I installed kvm and tried to use SR-IOV virtualizaton for 82599EB(Intel XT-520 T2) dual port card with latest ixgbe driver(version:3.11.33) ,
kernel2.6.32-279.14.1(OS:Centos6.3) ,after configuration and reboot
It seems that only first port of the card's VFs works,second port of the card's VFs didn't work
I found some errors in /var/log messages:
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: not enough MMIO resources for SR-IOV
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (unregistered net_device): Failed to enable PCI sriov: -12
How to fix it,any help would be greatly appreciated.
below is the related information:
Server: R710
OS: centos6.3
NIC: X520-T2(dual port)
kernel: 2.6.32-279.14.1.el6.x86_64
BIOSVersion: 6.3.0(latest)
BIOS:Inter VT/VT-d or SR-IOV(enabled)
ixgbe:3.11.33(latest)
ixgbevf:2.7.12(latest)
grub config:intel_iommu=on appended
/var/log/messages:
Nov 8 14:56:54 12 kernel: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.11.33
Nov 8 14:56:54 12 kernel: Copyright (c) 1999-2012 Intel Corporation.
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: PCI INT B -> GSI 50 (level, low) -> IRQ 50
Nov 8 14:56:54 12 kernel: ixgbe: I/O Virtualization (IOV) set to 2
Nov 8 14:56:54 12 kernel: ixgbe: 0000:07:00.0: ixgbe_check_options: FCoE Offload feature enabled
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: (unregistered net_device): SR-IOV enabled with 2 VFs
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: FCoE offload feature is not available. Disabling FCoE offload feature
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: (PCI Express:5.0GT/s:Width x8) 68:05:ca:0c:7a:e2
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: MAC: 2, PHY: 2, PBA No: G21371-003
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: Enabled Features: RxQ: 1 TxQ: 1 LRO
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: IOV: VF 0 is enabled mac 0E:15:B9:26:B3:7D
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: IOV: VF 1 is enabled mac 32:69:2D:16:B9:40
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: Intel(R) 10 Gigabit Network Connection
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Nov 8 14:56:54 12 kernel: ixgbe: I/O Virtualization (IOV) set to 2
Nov 8 14:56:54 12 kernel: ixgbe: 0000:07:00.1: ixgbe_check_options: FCoE Offload feature enabled
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: not enough MMIO resources for SR-IOV
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (unregistered net_device): Failed to enable PCI sriov: -12
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: FCoE offload feature is not available. Disabling FCoE offload feature
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (PCI Express:5.0GT/s:Width x8) 68:05:ca:0c:7a:e3
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: MAC: 2, PHY: 2, PBA No: G21371-003
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: Enabled Features:
RxQ: 16 TxQ: 16 FdirHash RSC
Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: Intel(R) 10
Gigabit Network Connection
# dmesg |grep -E 'DMA|IOMMU'
ACPI: DMAR 00000000bf3b3668 001C0 (v01 DELL PE_SC3 00000001 DELL 00000001)
DMA 0x00000001 -> 0x00001000
DMA32 0x00001000 -> 0x00100000
DMA zone: 56 pages used for memmap
DMA zone: 102 pages reserved
DMA zone: 3839 pages, LIFO batch:0
DMA32 zone: 14280 pages used for memmap
DMA32 zone: 764849 pages, LIFO batch:31
Intel-IOMMU: enabled
DMAR: Host address width 40
DMAR: DRHD base: 0x000000fed90000 flags: 0x1
IOMMU fed90000: ver 1:0 cap c90780106f0462 ecap f020fe
DMAR: RMRR base: 0x000000bf4c8000 end: 0x000000bf4dffff
DMAR: RMRR base: 0x000000bf4b1000 end: 0x000000bf4bffff
DMAR: RMRR base: 0x000000bf4a1000 end: 0x000000bf4a1fff
DMAR: RMRR base: 0x000000bf4a3000 end: 0x000000bf4a3fff
DMAR: RMRR base: 0x000000bf4a5000 end: 0x000000bf4a5fff
DMAR: RMRR base: 0x000000bf4a7000 end: 0x000000bf4a7fff
DMAR: RMRR base: 0x000000bf4c0000 end: 0x000000bf4c0fff
DMAR: RMRR base: 0x000000bf4c2000 end: 0x000000bf4c2fff
DMAR: ATSR flags: 0x0
DMAR: Device scope device [0000:00:1a.02] not found
DMAR: Device scope device [0000:00:1a.02] not found
DMAR: Device scope device [0000:00:1d.02] not found
DMAR: Device scope device [0000:00:1d.02] not found
IOMMU 0xfed90000: using Queued invalidation
IOMMU: Setting RMRR:
IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4c2000 - 0xbf4c3000]
IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4c0000 - 0xbf4c1000]
IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4a7000 - 0xbf4a8000]
IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4a5000 - 0xbf4a6000]
IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4a3000 - 0xbf4a4000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4a1000 - 0xbf4a2000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4c8000 - 0xbf4e0000]
IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4c8000 - 0xbf4e0000]
IOMMU: Prepare 0-16MiB unity mapping for LPC
IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0x1000000]
PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
ata1: SATA max UDMA/133 cmd 0xdc10 ctl 0xdc08 bmdma 0xdc20 irq 23
ata2: SATA max UDMA/133 cmd 0xdc18 ctl 0xdc0c bmdma 0xdc28 irq 23
ata1.00: ATAPI: TSSTcorp DVD-ROM SN-108BB, D150, max UDMA/100
ata1.00: configured for UDMA/100
# lspci|grep Virtual
07:10.0 Ethernet controller: Intel Corporation 82599 Virtual Function (rev 01)
07:10.2 Ethernet controller: Intel Corporation 82599 Virtual Function (rev 01)
#lspci|grep '82599EB'
07:00.0 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN
Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN
Network Connection (rev 01)
# lspci -n|grep 07:00
07:00.0 0200: 8086:151c (rev 01)
07:00.1 0200: 8086:151c (rev 01)
# lspci -n -d 8086:151c -vvv|grep -i width
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
#lspci -vvv -s 07:00
07:00.0 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-T2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 50
Region 0: Memory at de400000 (64-bit, non-prefetchable) [size=2M]
Region 2: I/O ports at ecc0 [size=32]
Region 4: Memory at de3f8000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at de200000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [e0] Vital Product Data
Unknown small resource type 06, will not decode more.
Capabilities: [100] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-0c-7a-e2
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 2, Function Dependency Link: 00
VF offset: 128, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000d5900000 (64-bit, prefetchable)
Region 3: Memory at 00000000d5a00000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
Kernel modules: ixgbe
07:00.1 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-T2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 40
Region 0: Memory at de600000 (64-bit, non-prefetchable) [size=2M]
Region 2: I/O ports at ece0 [size=32]
Region 4: Memory at de3fc000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at d5800000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [e0] Vital Product Data
Unknown small resource type 06, will not decode more.
Capabilities: [100] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-0c-7a-e2
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 2, Function Dependency Link: 01
VF offset: 128, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000d5b00000 (64-bit, prefetchable)
Region 3: Memory at 00000000d5b00000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
Kernel modules: ixgbe
#modinfo ixgbe
filename: /lib/modules/2.6.32-279.14.1.el6.x86_64/kernel/drivers/net/ixgbe/ixgbe.ko
version: 3.11.33
license: GPL
description: Intel(R) 10 Gigabit PCI Express Network Driver
author: Intel Corporation, <[email protected]>
srcversion: ADDCBFA6FAFE8A0299074F8
alias: pci:v00008086d0000154Asv*sd*bc*sc*i*
alias: pci:v00008086d00001557sv*sd*bc*sc*i*
alias: pci:v00008086d0000154Fsv*sd*bc*sc*i*
alias: pci:v00008086d0000154Dsv*sd*bc*sc*i*
alias: pci:v00008086d00001528sv*sd*bc*sc*i*
alias: pci:v00008086d000010F8sv*sd*bc*sc*i*
alias: pci:v00008086d0000151Csv*sd*bc*sc*i*
alias: pci:v00008086d00001529sv*sd*bc*sc*i*
alias: pci:v00008086d0000152Asv*sd*bc*sc*i*
alias: pci:v00008086d000010F9sv*sd*bc*sc*i*
alias: pci:v00008086d00001514sv*sd*bc*sc*i*
alias: pci:v00008086d00001507sv*sd*bc*sc*i*
alias: pci:v00008086d000010FBsv*sd*bc*sc*i*
alias: pci:v00008086d00001517sv*sd*bc*sc*i*
alias: pci:v00008086d000010FCsv*sd*bc*sc*i*
alias: pci:v00008086d000010F7sv*sd*bc*sc*i*
alias: pci:v00008086d00001508sv*sd*bc*sc*i*
alias: pci:v00008086d000010DBsv*sd*bc*sc*i*
alias: pci:v00008086d000010F4sv*sd*bc*sc*i*
alias: pci:v00008086d000010E1sv*sd*bc*sc*i*
alias: pci:v00008086d000010F1sv*sd*bc*sc*i*
alias: pci:v00008086d000010ECsv*sd*bc*sc*i*
alias: pci:v00008086d000010DDsv*sd*bc*sc*i*
alias: pci:v00008086d0000150Bsv*sd*bc*sc*i*
alias: pci:v00008086d000010C8sv*sd*bc*sc*i*
alias: pci:v00008086d000010C7sv*sd*bc*sc*i*
alias: pci:v00008086d000010C6sv*sd*bc*sc*i*
alias: pci:v00008086d000010B6sv*sd*bc*sc*i*
depends: dca
vermagic: 2.6.32-279.14.1.el6.x86_64 SMP mod_unload modversions
parm: InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int)
parm: IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)
parm: MQ:Disable or enable Multiple Queues, default 1 (array of int)
parm: DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)
parm: RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)
parm: VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable (default=8) (array of int)
parm: max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)
parm: L2LBen:L2 Loopback Enable: 0 = disable, 1 = enable (default) (array of int)
parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int)
parm: LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)
parm: LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)
parm: LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)
parm: LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)
parm: LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)
parm: FdirPballoc:Flow Director packet buffer allocation level:
1 = 8k hash filters or 2k perfect filters
2 = 16k hash filters or 4k perfect filters
3 = 32k hash filters or 8k perfect filters (array of int)
parm: AtrSampleRate:Software ATR Tx packet sample rate (array of int)
parm: FCoE:Disable or enable FCoE Offload, default 1 (array of int)
parm: LRO:Large Receive Offload (0,1), default 1 = on (array of int)
parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)
# modinfo ixgbevf
filename: /lib/modules/2.6.32-279.14.1.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko
version: 2.7.12
license: GPL
description: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver
author: Intel Corporation, <[email protected]>
srcversion: 4D00B4A5B811D6D07D8BD15
alias: pci:v00008086d00001515sv*sd*bc*sc*i*
alias: pci:v00008086d000010EDsv*sd*bc*sc*i*
depends:
vermagic: 2.6.32-279.14.1.el6.x86_64 SMP mod_unload modversions
parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (956-488281), default 8000, use 1 for dynamic (array of int)
# dmidecode -t bios
# dmidecode 2.11
SMBIOS 2.6 present.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Dell Inc.
Version: 6.3.0
Release Date: 07/24/2012
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 4096 kB
Characteristics:
ISA is supported
PCI is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
EDD is supported
Japanese floppy for Toshiba 1.2 MB is supported (int 13h)
5.25"/360 kB floppy services are supported (int 13h)
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
BIOS boot specification is supported
Function key-initiated network boot is supported
Targeted content distribution is supported
BIOS Revision: 6.3
Handle 0x0D00, DMI type 13, 22 bytes
BIOS Language Information
Language Description Format: Long
Installable Languages: 1
en|US|iso8859-1
Currently Installed Language: en|US|iso8859-1
Jason Gao
--
On 11/08/2012 03:15 AM, pkill.2012 wrote:
> Hello,
>
> I installed kvm and tried to use SR-IOV virtualizaton for 82599EB(Intel XT-520 T2) dual port card with latest ixgbe driver(version:3.11.33) ,
> kernel2.6.32-279.14.1(OS:Centos6.3) ,after configuration and reboot
> It seems that only first port of the card's VFs works,second port of the card's VFs didn't work
> I found some errors in /var/log messages:
>
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: not enough MMIO resources for SR-IOV
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (unregistered net_device): Failed to enable PCI sriov: -12
>
> How to fix it,any help would be greatly appreciated.
>
> below is the related information:
> Server: R710
> OS: centos6.3
> NIC: X520-T2(dual port)
> kernel: 2.6.32-279.14.1.el6.x86_64
> BIOSVersion: 6.3.0(latest)
> BIOS:Inter VT/VT-d or SR-IOV(enabled)
> ixgbe:3.11.33(latest)
> ixgbevf:2.7.12(latest)
> grub config:intel_iommu=on appended
>
> /var/log/messages:
> Nov 8 14:56:54 12 kernel: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.11.33
> Nov 8 14:56:54 12 kernel: Copyright (c) 1999-2012 Intel Corporation.
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: PCI INT B -> GSI 50 (level, low) -> IRQ 50
> Nov 8 14:56:54 12 kernel: ixgbe: I/O Virtualization (IOV) set to 2
> Nov 8 14:56:54 12 kernel: ixgbe: 0000:07:00.0: ixgbe_check_options: FCoE Offload feature enabled
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: (unregistered net_device): SR-IOV enabled with 2 VFs
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: FCoE offload feature is not available. Disabling FCoE offload feature
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: (PCI Express:5.0GT/s:Width x8) 68:05:ca:0c:7a:e2
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: MAC: 2, PHY: 2, PBA No: G21371-003
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: Enabled Features: RxQ: 1 TxQ: 1 LRO
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: IOV: VF 0 is enabled mac 0E:15:B9:26:B3:7D
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: IOV: VF 1 is enabled mac 32:69:2D:16:B9:40
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: Intel(R) 10 Gigabit Network Connection
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: PCI INT A -> GSI 40 (level, low) -> IRQ 40
> Nov 8 14:56:54 12 kernel: ixgbe: I/O Virtualization (IOV) set to 2
> Nov 8 14:56:54 12 kernel: ixgbe: 0000:07:00.1: ixgbe_check_options: FCoE Offload feature enabled
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: not enough MMIO resources for SR-IOV
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (unregistered net_device): Failed to enable PCI sriov: -12
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: FCoE offload feature is not available. Disabling FCoE offload feature
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (PCI Express:5.0GT/s:Width x8) 68:05:ca:0c:7a:e3
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: MAC: 2, PHY: 2, PBA No: G21371-003
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: Enabled Features:
> RxQ: 16 TxQ: 16 FdirHash RSC
> Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: Intel(R) 10
> Gigabit Network Connection
>
> # dmesg |grep -E 'DMA|IOMMU'
> ACPI: DMAR 00000000bf3b3668 001C0 (v01 DELL PE_SC3 00000001 DELL 00000001)
> DMA 0x00000001 -> 0x00001000
> DMA32 0x00001000 -> 0x00100000
> DMA zone: 56 pages used for memmap
> DMA zone: 102 pages reserved
> DMA zone: 3839 pages, LIFO batch:0
> DMA32 zone: 14280 pages used for memmap
> DMA32 zone: 764849 pages, LIFO batch:31
> Intel-IOMMU: enabled
> DMAR: Host address width 40
> DMAR: DRHD base: 0x000000fed90000 flags: 0x1
> IOMMU fed90000: ver 1:0 cap c90780106f0462 ecap f020fe
> DMAR: RMRR base: 0x000000bf4c8000 end: 0x000000bf4dffff
> DMAR: RMRR base: 0x000000bf4b1000 end: 0x000000bf4bffff
> DMAR: RMRR base: 0x000000bf4a1000 end: 0x000000bf4a1fff
> DMAR: RMRR base: 0x000000bf4a3000 end: 0x000000bf4a3fff
> DMAR: RMRR base: 0x000000bf4a5000 end: 0x000000bf4a5fff
> DMAR: RMRR base: 0x000000bf4a7000 end: 0x000000bf4a7fff
> DMAR: RMRR base: 0x000000bf4c0000 end: 0x000000bf4c0fff
> DMAR: RMRR base: 0x000000bf4c2000 end: 0x000000bf4c2fff
> DMAR: ATSR flags: 0x0
> DMAR: Device scope device [0000:00:1a.02] not found
> DMAR: Device scope device [0000:00:1a.02] not found
> DMAR: Device scope device [0000:00:1d.02] not found
> DMAR: Device scope device [0000:00:1d.02] not found
> IOMMU 0xfed90000: using Queued invalidation
> IOMMU: Setting RMRR:
> IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4c2000 - 0xbf4c3000]
> IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4c0000 - 0xbf4c1000]
> IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4a7000 - 0xbf4a8000]
> IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4a5000 - 0xbf4a6000]
> IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4a3000 - 0xbf4a4000]
> IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4a1000 - 0xbf4a2000]
> IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4b1000 - 0xbf4c0000]
> IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4b1000 - 0xbf4c0000]
> IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4b1000 - 0xbf4c0000]
> IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4b1000 - 0xbf4c0000]
> IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4c8000 - 0xbf4e0000]
> IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4c8000 - 0xbf4e0000]
> IOMMU: Prepare 0-16MiB unity mapping for LPC
> IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0x1000000]
> PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
> ata1: SATA max UDMA/133 cmd 0xdc10 ctl 0xdc08 bmdma 0xdc20 irq 23
> ata2: SATA max UDMA/133 cmd 0xdc18 ctl 0xdc0c bmdma 0xdc28 irq 23
> ata1.00: ATAPI: TSSTcorp DVD-ROM SN-108BB, D150, max UDMA/100
> ata1.00: configured for UDMA/100
>
> # lspci|grep Virtual
> 07:10.0 Ethernet controller: Intel Corporation 82599 Virtual Function (rev 01)
> 07:10.2 Ethernet controller: Intel Corporation 82599 Virtual Function (rev 01)
>
> #lspci|grep '82599EB'
> 07:00.0 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN
> Network Connection (rev 01)
> 07:00.1 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN
> Network Connection (rev 01)
>
> # lspci -n|grep 07:00
> 07:00.0 0200: 8086:151c (rev 01)
> 07:00.1 0200: 8086:151c (rev 01)
>
> # lspci -n -d 8086:151c -vvv|grep -i width
> LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
> LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
> LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>
> #lspci -vvv -s 07:00
> 07:00.0 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
> Subsystem: Intel Corporation Ethernet Server Adapter X520-T2
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Interrupt: pin B routed to IRQ 50
> Region 0: Memory at de400000 (64-bit, non-prefetchable) [size=2M]
> Region 2: I/O ports at ecc0 [size=32]
> Region 4: Memory at de3f8000 (64-bit, non-prefetchable) [size=16K]
> Expansion ROM at de200000 [disabled] [size=1M]
> Capabilities: [40] Power Management version 3
> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> Address: 0000000000000000 Data: 0000
> Masking: 00000000 Pending: 00000000
> Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
> Vector table: BAR=4 offset=00000000
> PBA: BAR=4 offset=00002000
> Capabilities: [a0] Express (v2) Endpoint, MSI 00
> DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
> MaxPayload 256 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
> LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
> ClockPM- Surprise- LLActRep- BwNot-
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
> DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-
> LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> Compliance De-emphasis: -6dB
> LnkSta2: Current De-emphasis Level: -6dB
> Capabilities: [e0] Vital Product Data
> Unknown small resource type 06, will not decode more.
> Capabilities: [100] Advanced Error Reporting
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
> Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-0c-7a-e2
> Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
> ARICap: MFVC- ACS-, Next Function: 1
> ARICtl: MFVC- ACS-, Function Group: 0
> Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
> IOVCap: Migration-, Interrupt Message Number: 000
> IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
> IOVSta: Migration-
> Initial VFs: 64, Total VFs: 64, Number of VFs: 2, Function Dependency Link: 00
> VF offset: 128, stride: 2, Device ID: 10ed
> Supported Page Size: 00000553, System Page Size: 00000001
> Region 0: Memory at 00000000d5900000 (64-bit, prefetchable)
> Region 3: Memory at 00000000d5a00000 (64-bit, prefetchable)
> VF Migration: offset: 00000000, BIR: 0
> Kernel driver in use: ixgbe
> Kernel modules: ixgbe
> 07:00.1 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
> Subsystem: Intel Corporation Ethernet Server Adapter X520-T2
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Interrupt: pin A routed to IRQ 40
> Region 0: Memory at de600000 (64-bit, non-prefetchable) [size=2M]
> Region 2: I/O ports at ece0 [size=32]
> Region 4: Memory at de3fc000 (64-bit, non-prefetchable) [size=16K]
> Expansion ROM at d5800000 [disabled] [size=1M]
> Capabilities: [40] Power Management version 3
> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> Address: 0000000000000000 Data: 0000
> Masking: 00000000 Pending: 00000000
> Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
> Vector table: BAR=4 offset=00000000
> PBA: BAR=4 offset=00002000
> Capabilities: [a0] Express (v2) Endpoint, MSI 00
> DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
> MaxPayload 256 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
> LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
> ClockPM- Surprise- LLActRep- BwNot-
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
> DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-
> LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> Compliance De-emphasis: -6dB
> LnkSta2: Current De-emphasis Level: -6dB
> Capabilities: [e0] Vital Product Data
> Unknown small resource type 06, will not decode more.
> Capabilities: [100] Advanced Error Reporting
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
> Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-0c-7a-e2
> Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
> ARICap: MFVC- ACS-, Next Function: 0
> ARICtl: MFVC- ACS-, Function Group: 0
> Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
> IOVCap: Migration-, Interrupt Message Number: 000
> IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
> IOVSta: Migration-
> Initial VFs: 64, Total VFs: 64, Number of VFs: 2, Function Dependency Link: 01
> VF offset: 128, stride: 2, Device ID: 10ed
> Supported Page Size: 00000553, System Page Size: 00000001
> Region 0: Memory at 00000000d5b00000 (64-bit, prefetchable)
> Region 3: Memory at 00000000d5b00000 (64-bit, prefetchable)
> VF Migration: offset: 00000000, BIR: 0
> Kernel driver in use: ixgbe
> Kernel modules: ixgbe
>
> #modinfo ixgbe
> filename: /lib/modules/2.6.32-279.14.1.el6.x86_64/kernel/drivers/net/ixgbe/ixgbe.ko
> version: 3.11.33
> license: GPL
> description: Intel(R) 10 Gigabit PCI Express Network Driver
> author: Intel Corporation, <[email protected]>
> srcversion: ADDCBFA6FAFE8A0299074F8
> alias: pci:v00008086d0000154Asv*sd*bc*sc*i*
> alias: pci:v00008086d00001557sv*sd*bc*sc*i*
> alias: pci:v00008086d0000154Fsv*sd*bc*sc*i*
> alias: pci:v00008086d0000154Dsv*sd*bc*sc*i*
> alias: pci:v00008086d00001528sv*sd*bc*sc*i*
> alias: pci:v00008086d000010F8sv*sd*bc*sc*i*
> alias: pci:v00008086d0000151Csv*sd*bc*sc*i*
> alias: pci:v00008086d00001529sv*sd*bc*sc*i*
> alias: pci:v00008086d0000152Asv*sd*bc*sc*i*
> alias: pci:v00008086d000010F9sv*sd*bc*sc*i*
> alias: pci:v00008086d00001514sv*sd*bc*sc*i*
> alias: pci:v00008086d00001507sv*sd*bc*sc*i*
> alias: pci:v00008086d000010FBsv*sd*bc*sc*i*
> alias: pci:v00008086d00001517sv*sd*bc*sc*i*
> alias: pci:v00008086d000010FCsv*sd*bc*sc*i*
> alias: pci:v00008086d000010F7sv*sd*bc*sc*i*
> alias: pci:v00008086d00001508sv*sd*bc*sc*i*
> alias: pci:v00008086d000010DBsv*sd*bc*sc*i*
> alias: pci:v00008086d000010F4sv*sd*bc*sc*i*
> alias: pci:v00008086d000010E1sv*sd*bc*sc*i*
> alias: pci:v00008086d000010F1sv*sd*bc*sc*i*
> alias: pci:v00008086d000010ECsv*sd*bc*sc*i*
> alias: pci:v00008086d000010DDsv*sd*bc*sc*i*
> alias: pci:v00008086d0000150Bsv*sd*bc*sc*i*
> alias: pci:v00008086d000010C8sv*sd*bc*sc*i*
> alias: pci:v00008086d000010C7sv*sd*bc*sc*i*
> alias: pci:v00008086d000010C6sv*sd*bc*sc*i*
> alias: pci:v00008086d000010B6sv*sd*bc*sc*i*
> depends: dca
> vermagic: 2.6.32-279.14.1.el6.x86_64 SMP mod_unload modversions
> parm: InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int)
> parm: IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)
> parm: MQ:Disable or enable Multiple Queues, default 1 (array of int)
> parm: DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)
> parm: RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)
> parm: VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable (default=8) (array of int)
> parm: max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)
> parm: L2LBen:L2 Loopback Enable: 0 = disable, 1 = enable (default) (array of int)
> parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int)
> parm: LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)
> parm: LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)
> parm: LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)
> parm: LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)
> parm: LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)
> parm: FdirPballoc:Flow Director packet buffer allocation level:
> 1 = 8k hash filters or 2k perfect filters
> 2 = 16k hash filters or 4k perfect filters
> 3 = 32k hash filters or 8k perfect filters (array of int)
> parm: AtrSampleRate:Software ATR Tx packet sample rate (array of int)
> parm: FCoE:Disable or enable FCoE Offload, default 1 (array of int)
> parm: LRO:Large Receive Offload (0,1), default 1 = on (array of int)
> parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)
>
> # modinfo ixgbevf
> filename: /lib/modules/2.6.32-279.14.1.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko
> version: 2.7.12
> license: GPL
> description: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver
> author: Intel Corporation, <[email protected]>
> srcversion: 4D00B4A5B811D6D07D8BD15
> alias: pci:v00008086d00001515sv*sd*bc*sc*i*
> alias: pci:v00008086d000010EDsv*sd*bc*sc*i*
> depends:
> vermagic: 2.6.32-279.14.1.el6.x86_64 SMP mod_unload modversions
> parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (956-488281), default 8000, use 1 for dynamic (array of int)
>
>
> # dmidecode -t bios
> # dmidecode 2.11
> SMBIOS 2.6 present.
> Handle 0x0000, DMI type 0, 24 bytes
> BIOS Information
> Vendor: Dell Inc.
> Version: 6.3.0
> Release Date: 07/24/2012
> Address: 0xF0000
> Runtime Size: 64 kB
> ROM Size: 4096 kB
> Characteristics:
> ISA is supported
> PCI is supported
> PNP is supported
> BIOS is upgradeable
> BIOS shadowing is allowed
> Boot from CD is supported
> Selectable boot is supported
> EDD is supported
> Japanese floppy for Toshiba 1.2 MB is supported (int 13h)
> 5.25"/360 kB floppy services are supported (int 13h)
> 5.25"/1.2 MB floppy services are supported (int 13h)
> 3.5"/720 kB floppy services are supported (int 13h)
> 8042 keyboard services are supported (int 9h)
> Serial services are supported (int 14h)
> CGA/mono video services are supported (int 10h)
> ACPI is supported
> USB legacy is supported
> BIOS boot specification is supported
> Function key-initiated network boot is supported
> Targeted content distribution is supported
> BIOS Revision: 6.3
> Handle 0x0D00, DMI type 13, 22 bytes
> BIOS Language Information
> Language Description Format: Long
> Installable Languages: 1
> en|US|iso8859-1
> Currently Installed Language: en|US|iso8859-1
>
> Jason Gao
> --
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
Adding e100-devel mailing list
> The BIOS in your machine doesn't support SR-IOV. You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.
very thanks Greg,my server Dell R710 with latest BIOS version and
option for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused
that Does R710 provide full support for SR-IOV, kernel or ixgbe
driver's bug? but I'm not sure where the problem lies,anyone has any
experience about this? .
[+ linux-pci, Yinghai]
On Thu, Nov 8, 2012 at 8:59 PM, Jason Gao <[email protected]> wrote:
>> The BIOS in your machine doesn't support SR-IOV. You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.
>
> very thanks Greg,my server Dell R710 with latest BIOS version and
> option for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused
> that Does R710 provide full support for SR-IOV, kernel or ixgbe
> driver's bug? but I'm not sure where the problem lies,anyone has any
> experience about this? .
Linux normally uses the resource assignments done by the BIOS, but it
is possible for the kernel to reassign those. We don't have good
automatic support for that yet, but on a recent upstream kernel, you
can try "pci=realloc". I doubt this option is in CentOS 6.3, though.
If an upstream kernel with "pci=realloc" still doesn't work, please
post the entire dmesg log.
On Fri, Nov 9, 2012 at 7:26 AM, Bjorn Helgaas <[email protected]> wrote:
> [+ linux-pci, Yinghai]
>
> On Thu, Nov 8, 2012 at 8:59 PM, Jason Gao <[email protected]> wrote:
>>> The BIOS in your machine doesn't support SR-IOV. You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.
>>
>> very thanks Greg,my server Dell R710 with latest BIOS version and
>> option for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused
>> that Does R710 provide full support for SR-IOV, kernel or ixgbe
>> driver's bug? but I'm not sure where the problem lies,anyone has any
>> experience about this? .
>
> Linux normally uses the resource assignments done by the BIOS, but it
> is possible for the kernel to reassign those. We don't have good
> automatic support for that yet, but on a recent upstream kernel, you
> can try "pci=realloc". I doubt this option is in CentOS 6.3, though.
>
> If an upstream kernel with "pci=realloc" still doesn't work, please
> post the entire dmesg log.
Even better, upstream kernel will enable realloc automatically if it
finds some SRIOV BAR is not assigned.
related logic is in pci_realloc_detect().
BTW, we may need another patch about realloc.
http://git.kernel.org/?p=linux/kernel/git/yinghai/linux-yinghai.git;a=patch;h=afb9975bddd92b17c2f9859676ce521e2e014997
>From afb9975bddd92b17c2f9859676ce521e2e014997 Mon Sep 17 00:00:00 2001
From: Yinghai Lu <[email protected]>
Date: Wed, 3 Oct 2012 19:15:57 -0700
Subject: [PATCH] PCI: Add root bus children dev's res to fail list
We can stop trying according to try number now and do not need to use
root_bus checking as stop sign anymore.
In extreme case we could need to reallocate resource for device just
under root bus.
Signed-off-by: Yinghai Lu <[email protected]>
---
drivers/pci/setup-bus.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 1e808ca..9e625d84 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -283,7 +283,7 @@ static void
assign_requested_resources_sorted(struct list_head *head,
idx = res - &dev_res->dev->resource[0];
if (resource_size(res) &&
pci_assign_resource(dev_res->dev, idx)) {
- if (fail_head && !pci_is_root_bus(dev_res->dev->bus)) {
+ if (fail_head) {
/*
* if the failed res is for ROM BAR, and it will
* be enabled later, don't add it to the list
--
1.7.7.6
> -----Original Message-----
> From: Jason Gao [mailto:[email protected]]
> Sent: Thursday, November 08, 2012 8:00 PM
> To: Rose, Gregory V
> Cc: Kirsher, Jeffrey T; linux-kernel; netdev; kvm; e1000-
> [email protected]
> Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources
> for SR-IOV)
>
> > The BIOS in your machine doesn't support SR-IOV. You'll need to ask the
> manufacturer for a BIOS upgrade, if in fact one is available. Sometimes
> they're not.
>
> very thanks Greg,my server Dell R710 with latest BIOS version and option
> for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused that Does
> R710 provide full support for SR-IOV, kernel or ixgbe driver's bug? but
> I'm not sure where the problem lies,anyone has any
> experience about this? .
I use a Dell R710 for all my SR-IOV testing and it works fine but I had to acquire a special BIOS from Dell to get it to work. As I said, you'll want to contact them and make sure you've got the correct BIOS.
The error you're seeing, "not enough MMIO resources" has been an issue with the BIOS 100% of the times that we've run into it. In any case, it is NOT a driver bug. The driver has nothing to do with allocation of MMIO space reservation for VF devices. The message you're seeing pops up when the driver calls pci_enable_sriov(). There is nothing the driver can do to force that call to be successful.
I've added Sibai Li to this response. She should be able to get the BIOS version string off of one of our working Dell R710s.
Sibai,
Could you please reply to this message with that information?
Thanks,
- Greg
Jason,
DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is v.6.3.0 and release date is 07/24/2012
Please check if you configured intel_iommu=on in the grub.conf file.
If you did, check your kernel .config file under Device Drivers-> IOMMU Hardware support->enable Support for Intel IOMMU using DMA remapping Devices, enable Intel DMA Remapping Devices by Default, enable Support for Interrupt Remapping.
Thanks,
Sibai
> -----Original Message-----
> From: Rose, Gregory V
> Sent: Friday, November 09, 2012 9:16 AM
> To: Jason Gao; Li, Sibai
> Cc: Kirsher, Jeffrey T; linux-kernel; netdev; kvm
> Subject: RE: SR-IOV problem with Intel 82599EB (not enough MMIO resources
> for SR-IOV)
>
> > -----Original Message-----
> > From: Jason Gao [mailto:[email protected]]
> > Sent: Thursday, November 08, 2012 8:00 PM
> > To: Rose, Gregory V
> > Cc: Kirsher, Jeffrey T; linux-kernel; netdev; kvm; e1000-
> > [email protected]
> > Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO
> > resources for SR-IOV)
> >
> > > The BIOS in your machine doesn't support SR-IOV. You'll need to ask
> > > the
> > manufacturer for a BIOS upgrade, if in fact one is available.
> > Sometimes they're not.
> >
> > very thanks Greg,my server Dell R710 with latest BIOS version and
> > option for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused
> > that Does
> > R710 provide full support for SR-IOV, kernel or ixgbe driver's bug?
> > but I'm not sure where the problem lies,anyone has any
> > experience about this? .
>
> I use a Dell R710 for all my SR-IOV testing and it works fine but I had to acquire a
> special BIOS from Dell to get it to work. As I said, you'll want to contact them
> and make sure you've got the correct BIOS.
>
> The error you're seeing, "not enough MMIO resources" has been an issue with
> the BIOS 100% of the times that we've run into it. In any case, it is NOT a driver
> bug. The driver has nothing to do with allocation of MMIO space reservation for
> VF devices. The message you're seeing pops up when the driver calls
> pci_enable_sriov(). There is nothing the driver can do to force that call to be
> successful.
>
> I've added Sibai Li to this response. She should be able to get the BIOS version
> string off of one of our working Dell R710s.
>
> Sibai,
>
> Could you please reply to this message with that information?
>
> Thanks,
>
> - Greg
On 11/09/2012 10:26 AM, Bjorn Helgaas wrote:
> [+ linux-pci, Yinghai]
>
> On Thu, Nov 8, 2012 at 8:59 PM, Jason Gao<[email protected]> wrote:
>>> The BIOS in your machine doesn't support SR-IOV. You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.
>>
>> very thanks Greg,my server Dell R710 with latest BIOS version and
>> option for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused
>> that Does R710 provide full support for SR-IOV, kernel or ixgbe
>> driver's bug? but I'm not sure where the problem lies,anyone has any
>> experience about this? .
>
> Linux normally uses the resource assignments done by the BIOS, but it
> is possible for the kernel to reassign those. We don't have good
> automatic support for that yet, but on a recent upstream kernel, you
> can try "pci=realloc". I doubt this option is in CentOS 6.3, though.
>
Try moving the device into a different slot.
You may be trying it in a non-ARI slot in the 710; that is a problem
in RHEL6 (needing to realloc bus &/or mem-space).
A non-ARI slot will want to use one bus number per VF.... which will
be problematic.
I know I've seen ixgbe's (& their vfs's) working on a dell 710;
but they may also be one of those systems that has a slot off the ich10
with no ARI support.
> If an upstream kernel with "pci=realloc" still doesn't work, please
> post the entire dmesg log.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
I'm very sorry for delayed reply.now SR-IOV works for me in Centos
6.3,thank all of you.
On Fri, Nov 9, 2012 at 11:26 PM, Bjorn Helgaas <[email protected]> wrote:
> Linux normally uses the resource assignments done by the BIOS, but it
> is possible for the kernel to reassign those. We don't have good
> automatic support for that yet, but on a recent upstream kernel, you
> can try "pci=realloc". I doubt this option is in CentOS 6.3, though
Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.
On Sat, Nov 10, 2012 at 2:08 AM, Li, Sibai <[email protected]> wrote:
> DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is v.6.3.0 and release date is 07/24/2012
> Please check if you configured intel_iommu=on in the grub.conf file.
> If you did, check your kernel .config file under Device Drivers-> IOMMU Hardware support->enable Support for Intel IOMMU using DMA remapping Devices, enable Intel DMA Remapping Devices by Default, enable Support for Interrupt Remapping.
thank you Sibai,Our server "Dell R710",its BIOS version is just
v.6.3.0 and release date is 07/24/2012,and I also configured
intel_iommu=on in the grub.conf file,but I can't find these IOMMU
options in "Device Drivers" in my kernel(2.6.32-279) .config file ,
btw my os is Centos 6.3(RHEL6.3),although the problem solved,I'd like
to know what's your os version ,kernel version?
> -----Original Message-----
> From: Jason Gao [mailto:[email protected]]
> Sent: Tuesday, November 13, 2012 5:38 AM
> To: [email protected]; Rose, Gregory V; Li, Sibai
> Cc: [email protected]; Kirsher, Jeffrey T; linux-kernel; netdev; kvm; e1000-
> [email protected]; [email protected]; Yinghai Lu
> Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources
> for SR-IOV)
>
> I'm very sorry for delayed reply.now SR-IOV works for me in Centos 6.3,thank all
> of you.
>
>
> On Fri, Nov 9, 2012 at 11:26 PM, Bjorn Helgaas <[email protected]> wrote:
> > Linux normally uses the resource assignments done by the BIOS, but it
> > is possible for the kernel to reassign those. We don't have good
> > automatic support for that yet, but on a recent upstream kernel, you
> > can try "pci=realloc". I doubt this option is in CentOS 6.3, though
>
> Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.
>
>
>
> On Sat, Nov 10, 2012 at 2:08 AM, Li, Sibai <[email protected]> wrote:
> > DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is
> > v.6.3.0 and release date is 07/24/2012 Please check if you configured
> intel_iommu=on in the grub.conf file.
> > If you did, check your kernel .config file under Device Drivers-> IOMMU
> Hardware support->enable Support for Intel IOMMU using DMA remapping
> Devices, enable Intel DMA Remapping Devices by Default, enable Support for
> Interrupt Remapping.
>
> thank you Sibai,Our server "Dell R710",its BIOS version is just
> v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in
> the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my
> kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the
> problem solved,I'd like to know what's your os version ,kernel version?
I am using RHEL6.3 with unstable kernel 3.7.0-rc
On 11/13/2012 11:04 AM, Li, Sibai wrote:
>
>
>> -----Original Message-----
>> From: Jason Gao [mailto:[email protected]]
>> Sent: Tuesday, November 13, 2012 5:38 AM
>> To: [email protected]; Rose, Gregory V; Li, Sibai
>> Cc: [email protected]; Kirsher, Jeffrey T; linux-kernel; netdev; kvm; e1000-
>> [email protected]; [email protected]; Yinghai Lu
>> Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources
>> for SR-IOV)
>>
>> I'm very sorry for delayed reply.now SR-IOV works for me in Centos 6.3,thank all
>> of you.
>>
>>
>> On Fri, Nov 9, 2012 at 11:26 PM, Bjorn Helgaas<[email protected]> wrote:
>>> Linux normally uses the resource assignments done by the BIOS, but it
>>> is possible for the kernel to reassign those. We don't have good
>>> automatic support for that yet, but on a recent upstream kernel, you
>>> can try "pci=realloc". I doubt this option is in CentOS 6.3, though
>>
>> Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.
>>
>>
>>
>> On Sat, Nov 10, 2012 at 2:08 AM, Li, Sibai<[email protected]> wrote:
>>> DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is
>>> v.6.3.0 and release date is 07/24/2012 Please check if you configured
>> intel_iommu=on in the grub.conf file.
>>> If you did, check your kernel .config file under Device Drivers-> IOMMU
>> Hardware support->enable Support for Intel IOMMU using DMA remapping
>> Devices, enable Intel DMA Remapping Devices by Default, enable Support for
>> Interrupt Remapping.
>>
>> thank you Sibai,Our server "Dell R710",its BIOS version is just
>> v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in
>> the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my
Sibai is referring to kernel config options. RHEL6.3 has the IOMMU options built into
the kernel, but not enabled by default -- have to add 'intel_iommu=on' to the kernel
cmdline to enable IOMMU. SRIOV support (CONFIG_IOV) is built into the RHEL6.3 kernel as well.
>> kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the
>> problem solved,I'd like to know what's your os version ,kernel version?
>
> I am using RHEL6.3 with unstable kernel 3.7.0-rc
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Nov 13, 2012 at 8:04 AM, Li, Sibai <[email protected]> wrote:
>
>>
>> Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.
>>
>> thank you Sibai,Our server "Dell R710",its BIOS version is just
>> v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in
>> the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my
>> kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the
>> problem solved,I'd like to know what's your os version ,kernel version?
>
> I am using RHEL6.3 with unstable kernel 3.7.0-rc
that means that config has
CONFIG_PCI_REALLOC_ENABLE_AUTO=y
So you don't need to append "pci=realloc"
Yinghai
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of
> Yinghai Lu
> Sent: Tuesday, November 13, 2012 10:17 AM
> To: Li, Sibai
> Cc: Jason Gao; [email protected]; Rose, Gregory V; [email protected];
> Kirsher, Jeffrey T; linux-kernel; netdev; kvm; [email protected];
> [email protected]
> Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources
> for SR-IOV)
>
> On Tue, Nov 13, 2012 at 8:04 AM, Li, Sibai <[email protected]> wrote:
> >
> >>
> >> Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for
> me.
> >>
> >> thank you Sibai,Our server "Dell R710",its BIOS version is just
> >> v.6.3.0 and release date is 07/24/2012,and I also configured
> >> intel_iommu=on in the grub.conf file,but I can't find these IOMMU
> >> options in "Device Drivers" in my
> >> kernel(2.6.32-279) .config file , btw my os is Centos
> >> 6.3(RHEL6.3),although the problem solved,I'd like to know what's your os
> version ,kernel version?
> >
> > I am using RHEL6.3 with unstable kernel 3.7.0-rc
>
> that means that config has
> CONFIG_PCI_REALLOC_ENABLE_AUTO=y
>
> So you don't need to append "pci=realloc"
>
> Yinghai
Never append "pci=realloc" for both kernel 2.6.32.279 and kernel 3.5.0 above.
On Tue, Nov 13, 2012 at 10:25 AM, Li, Sibai <[email protected]> wrote:
>
> Never append "pci=realloc" for both kernel 2.6.32.279 and kernel 3.5.0 above.
well, can you both post boot log with "debug ignore_loglevel" ?