With virtio, in multiqueue case, each queue IRQ is normally
bound to a different CPU using the affinity mask.
This works fine on x86_64 but totally ignored on pseries.
This is not obvious at first look because irqbalance is doing
some balancing to improve that.
It appears that the "managed" flag set in the MSI entry
is never copied to the system IRQ entry.
This series passes the affinity mask from rtas_setup_msi_irqs()
to irq_domain_alloc_descs() by adding an affinity parameter to
irq_create_mapping().
The first patch adds the parameter (no functional change), the
second patch passes the actual affinity mask to irq_create_mapping()
in rtas_setup_msi_irqs().
For instance, with 32 CPUs VM and 32 queues virtio-scsi interface:
... -smp 32 -device virtio-scsi-pci,id=virtio_scsi_pci0,num_queues=32
for IRQ in $(grep virtio2-request /proc/interrupts |cut -d: -f1); do
for file in /proc/irq/$IRQ/ ; do
echo -n "IRQ: $(basename $file) CPU: " ; cat $file/smp_affinity_list
done
done
Without the patch (and without irqbalanced)
IRQ: 268 CPU: 0-31
IRQ: 269 CPU: 0-31
IRQ: 270 CPU: 0-31
IRQ: 271 CPU: 0-31
IRQ: 272 CPU: 0-31
IRQ: 273 CPU: 0-31
IRQ: 274 CPU: 0-31
IRQ: 275 CPU: 0-31
IRQ: 276 CPU: 0-31
IRQ: 277 CPU: 0-31
IRQ: 278 CPU: 0-31
IRQ: 279 CPU: 0-31
IRQ: 280 CPU: 0-31
IRQ: 281 CPU: 0-31
IRQ: 282 CPU: 0-31
IRQ: 283 CPU: 0-31
IRQ: 284 CPU: 0-31
IRQ: 285 CPU: 0-31
IRQ: 286 CPU: 0-31
IRQ: 287 CPU: 0-31
IRQ: 288 CPU: 0-31
IRQ: 289 CPU: 0-31
IRQ: 290 CPU: 0-31
IRQ: 291 CPU: 0-31
IRQ: 292 CPU: 0-31
IRQ: 293 CPU: 0-31
IRQ: 294 CPU: 0-31
IRQ: 295 CPU: 0-31
IRQ: 296 CPU: 0-31
IRQ: 297 CPU: 0-31
IRQ: 298 CPU: 0-31
IRQ: 299 CPU: 0-31
With the patch:
IRQ: 265 CPU: 0
IRQ: 266 CPU: 1
IRQ: 267 CPU: 2
IRQ: 268 CPU: 3
IRQ: 269 CPU: 4
IRQ: 270 CPU: 5
IRQ: 271 CPU: 6
IRQ: 272 CPU: 7
IRQ: 273 CPU: 8
IRQ: 274 CPU: 9
IRQ: 275 CPU: 10
IRQ: 276 CPU: 11
IRQ: 277 CPU: 12
IRQ: 278 CPU: 13
IRQ: 279 CPU: 14
IRQ: 280 CPU: 15
IRQ: 281 CPU: 16
IRQ: 282 CPU: 17
IRQ: 283 CPU: 18
IRQ: 284 CPU: 19
IRQ: 285 CPU: 20
IRQ: 286 CPU: 21
IRQ: 287 CPU: 22
IRQ: 288 CPU: 23
IRQ: 289 CPU: 24
IRQ: 290 CPU: 25
IRQ: 291 CPU: 26
IRQ: 292 CPU: 27
IRQ: 293 CPU: 28
IRQ: 294 CPU: 29
IRQ: 295 CPU: 30
IRQ: 299 CPU: 31
This matches what we have on an x86_64 system.
Laurent Vivier (2):
genirq: add an affinity parameter to irq_create_mapping()
powerpc/pseries: pass MSI affinity to irq_create_mapping()
arch/arc/kernel/intc-arcv2.c | 4 ++--
arch/arc/kernel/mcip.c | 2 +-
arch/arm/common/sa1111.c | 2 +-
arch/arm/mach-s3c/irq-s3c24xx.c | 3 ++-
arch/arm/plat-orion/gpio.c | 2 +-
arch/mips/ath25/ar2315.c | 4 ++--
arch/mips/ath25/ar5312.c | 4 ++--
arch/mips/lantiq/irq.c | 2 +-
arch/mips/pci/pci-ar2315.c | 3 ++-
arch/mips/pic32/pic32mzda/time.c | 2 +-
arch/mips/ralink/irq.c | 2 +-
arch/powerpc/kernel/pci-common.c | 2 +-
arch/powerpc/kvm/book3s_xive.c | 2 +-
arch/powerpc/platforms/44x/ppc476.c | 4 ++--
arch/powerpc/platforms/cell/interrupt.c | 4 ++--
arch/powerpc/platforms/cell/iommu.c | 3 ++-
arch/powerpc/platforms/cell/pmu.c | 2 +-
arch/powerpc/platforms/cell/spider-pic.c | 2 +-
arch/powerpc/platforms/cell/spu_manage.c | 6 +++---
arch/powerpc/platforms/maple/pci.c | 2 +-
arch/powerpc/platforms/pasemi/dma_lib.c | 5 +++--
arch/powerpc/platforms/pasemi/msi.c | 2 +-
arch/powerpc/platforms/pasemi/setup.c | 4 ++--
arch/powerpc/platforms/powermac/pci.c | 2 +-
arch/powerpc/platforms/powermac/pic.c | 2 +-
arch/powerpc/platforms/powermac/smp.c | 2 +-
arch/powerpc/platforms/powernv/opal-irqchip.c | 5 +++--
arch/powerpc/platforms/powernv/pci.c | 2 +-
arch/powerpc/platforms/powernv/vas.c | 2 +-
arch/powerpc/platforms/ps3/interrupt.c | 2 +-
arch/powerpc/platforms/pseries/ibmebus.c | 2 +-
arch/powerpc/platforms/pseries/msi.c | 2 +-
arch/powerpc/sysdev/fsl_mpic_err.c | 2 +-
arch/powerpc/sysdev/fsl_msi.c | 2 +-
arch/powerpc/sysdev/mpic.c | 3 ++-
arch/powerpc/sysdev/mpic_u3msi.c | 2 +-
arch/powerpc/sysdev/xics/xics-common.c | 2 +-
arch/powerpc/sysdev/xive/common.c | 2 +-
arch/sh/boards/mach-se/7343/irq.c | 2 +-
arch/sh/boards/mach-se/7722/irq.c | 2 +-
arch/sh/boards/mach-x3proto/gpio.c | 2 +-
arch/xtensa/kernel/perf_event.c | 2 +-
arch/xtensa/kernel/smp.c | 2 +-
arch/xtensa/kernel/time.c | 2 +-
drivers/ata/pata_macio.c | 2 +-
drivers/base/regmap/regmap-irq.c | 2 +-
drivers/bus/moxtet.c | 2 +-
drivers/clocksource/ingenic-timer.c | 2 +-
drivers/clocksource/timer-riscv.c | 2 +-
drivers/extcon/extcon-max8997.c | 3 ++-
drivers/gpio/gpio-bcm-kona.c | 2 +-
drivers/gpio/gpio-brcmstb.c | 2 +-
drivers/gpio/gpio-davinci.c | 2 +-
drivers/gpio/gpio-em.c | 3 ++-
drivers/gpio/gpio-grgpio.c | 2 +-
drivers/gpio/gpio-mockup.c | 2 +-
drivers/gpio/gpio-mpc8xxx.c | 2 +-
drivers/gpio/gpio-mvebu.c | 2 +-
drivers/gpio/gpio-tb10x.c | 2 +-
drivers/gpio/gpio-tegra.c | 2 +-
drivers/gpio/gpio-wm831x.c | 2 +-
drivers/gpio/gpiolib.c | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 3 ++-
drivers/gpu/ipu-v3/ipu-common.c | 2 +-
drivers/hid/hid-rmi.c | 2 +-
drivers/i2c/busses/i2c-cht-wc.c | 2 +-
drivers/i2c/i2c-core-base.c | 2 +-
drivers/i2c/muxes/i2c-mux-pca954x.c | 2 +-
drivers/ide/pmac.c | 2 +-
drivers/iio/dummy/iio_dummy_evgen.c | 3 ++-
drivers/input/rmi4/rmi_bus.c | 2 +-
drivers/irqchip/irq-ath79-misc.c | 3 ++-
drivers/irqchip/irq-bcm2835.c | 3 ++-
drivers/irqchip/irq-csky-mpintc.c | 2 +-
drivers/irqchip/irq-eznps.c | 2 +-
drivers/irqchip/irq-mips-gic.c | 8 +++++---
drivers/irqchip/irq-mmp.c | 4 ++--
drivers/irqchip/irq-versatile-fpga.c | 2 +-
drivers/irqchip/irq-vic.c | 2 +-
drivers/macintosh/macio_asic.c | 2 +-
drivers/memory/omap-gpmc.c | 2 +-
drivers/mfd/ab8500-core.c | 2 +-
drivers/mfd/arizona-irq.c | 5 +++--
drivers/mfd/db8500-prcmu.c | 2 +-
drivers/mfd/mfd-core.c | 2 +-
drivers/mfd/stmpe.c | 5 +++--
drivers/mfd/tc3589x.c | 2 +-
drivers/mfd/tps6586x.c | 2 +-
drivers/mfd/wm8994-irq.c | 5 +++--
drivers/misc/cxl/irq.c | 2 +-
drivers/misc/ocxl/afu_irq.c | 2 +-
drivers/misc/ocxl/link.c | 2 +-
drivers/net/dsa/mv88e6xxx/chip.c | 2 +-
drivers/net/dsa/mv88e6xxx/global2.c | 2 +-
drivers/net/dsa/qca/ar9331.c | 2 +-
drivers/net/dsa/rtl8366rb.c | 3 ++-
drivers/net/ethernet/ibm/ibmvnic.c | 4 ++--
drivers/net/usb/lan78xx.c | 2 +-
drivers/pci/controller/pci-ftpci100.c | 2 +-
drivers/pci/controller/pci-tegra.c | 2 +-
drivers/pci/controller/pcie-rcar-host.c | 2 +-
drivers/pci/controller/pcie-xilinx-cpm.c | 4 ++--
drivers/pci/controller/pcie-xilinx.c | 2 +-
drivers/pinctrl/mediatek/mtk-eint.c | 2 +-
drivers/pinctrl/nomadik/pinctrl-abx500.c | 3 ++-
drivers/pinctrl/pinctrl-at91-pio4.c | 3 ++-
drivers/pinctrl/pinctrl-rockchip.c | 2 +-
drivers/pinctrl/samsung/pinctrl-samsung.c | 2 +-
drivers/pinctrl/sunxi/pinctrl-sunxi.c | 2 +-
drivers/power/supply/lp8788-charger.c | 2 +-
drivers/rtc/rtc-lp8788.c | 2 +-
drivers/rtc/rtc-max8997.c | 3 ++-
drivers/rtc/rtc-max8998.c | 3 ++-
drivers/scsi/cxlflash/ocxl_hw.c | 2 +-
drivers/ssb/driver_gpio.c | 4 ++--
drivers/staging/hikey9xx/hi6421-spmi-pmic.c | 2 +-
drivers/staging/octeon-usb/octeon-hcd.c | 2 +-
drivers/tty/hvc/hvsi.c | 2 +-
drivers/tty/serial/pmac_zilog.c | 6 +++---
drivers/watchdog/octeon-wdt-main.c | 2 +-
include/linux/irqdomain.h | 3 ++-
kernel/irq/irqdomain.c | 8 +++++---
sound/soc/codecs/rt5677.c | 2 +-
123 files changed, 171 insertions(+), 146 deletions(-)
--
2.28.0
On Tue, Nov 24, 2020 at 09:03:06PM +0100, Laurent Vivier wrote:
> With virtio, in multiqueue case, each queue IRQ is normally
> bound to a different CPU using the affinity mask.
>
> This works fine on x86_64 but totally ignored on pseries.
>
> This is not obvious at first look because irqbalance is doing
> some balancing to improve that.
>
> It appears that the "managed" flag set in the MSI entry
> is never copied to the system IRQ entry.
>
> This series passes the affinity mask from rtas_setup_msi_irqs()
> to irq_domain_alloc_descs() by adding an affinity parameter to
> irq_create_mapping().
>
> The first patch adds the parameter (no functional change), the
> second patch passes the actual affinity mask to irq_create_mapping()
> in rtas_setup_msi_irqs().
>
> For instance, with 32 CPUs VM and 32 queues virtio-scsi interface:
>
> ... -smp 32 -device virtio-scsi-pci,id=virtio_scsi_pci0,num_queues=32
>
> for IRQ in $(grep virtio2-request /proc/interrupts |cut -d: -f1); do
> for file in /proc/irq/$IRQ/ ; do
> echo -n "IRQ: $(basename $file) CPU: " ; cat $file/smp_affinity_list
> done
> done
>
> Without the patch (and without irqbalanced)
>
> IRQ: 268 CPU: 0-31
> IRQ: 269 CPU: 0-31
> IRQ: 270 CPU: 0-31
> IRQ: 271 CPU: 0-31
> IRQ: 272 CPU: 0-31
> IRQ: 273 CPU: 0-31
> IRQ: 274 CPU: 0-31
> IRQ: 275 CPU: 0-31
> IRQ: 276 CPU: 0-31
> IRQ: 277 CPU: 0-31
> IRQ: 278 CPU: 0-31
> IRQ: 279 CPU: 0-31
> IRQ: 280 CPU: 0-31
> IRQ: 281 CPU: 0-31
> IRQ: 282 CPU: 0-31
> IRQ: 283 CPU: 0-31
> IRQ: 284 CPU: 0-31
> IRQ: 285 CPU: 0-31
> IRQ: 286 CPU: 0-31
> IRQ: 287 CPU: 0-31
> IRQ: 288 CPU: 0-31
> IRQ: 289 CPU: 0-31
> IRQ: 290 CPU: 0-31
> IRQ: 291 CPU: 0-31
> IRQ: 292 CPU: 0-31
> IRQ: 293 CPU: 0-31
> IRQ: 294 CPU: 0-31
> IRQ: 295 CPU: 0-31
> IRQ: 296 CPU: 0-31
> IRQ: 297 CPU: 0-31
> IRQ: 298 CPU: 0-31
> IRQ: 299 CPU: 0-31
>
> With the patch:
>
> IRQ: 265 CPU: 0
> IRQ: 266 CPU: 1
> IRQ: 267 CPU: 2
> IRQ: 268 CPU: 3
> IRQ: 269 CPU: 4
> IRQ: 270 CPU: 5
> IRQ: 271 CPU: 6
> IRQ: 272 CPU: 7
> IRQ: 273 CPU: 8
> IRQ: 274 CPU: 9
> IRQ: 275 CPU: 10
> IRQ: 276 CPU: 11
> IRQ: 277 CPU: 12
> IRQ: 278 CPU: 13
> IRQ: 279 CPU: 14
> IRQ: 280 CPU: 15
> IRQ: 281 CPU: 16
> IRQ: 282 CPU: 17
> IRQ: 283 CPU: 18
> IRQ: 284 CPU: 19
> IRQ: 285 CPU: 20
> IRQ: 286 CPU: 21
> IRQ: 287 CPU: 22
> IRQ: 288 CPU: 23
> IRQ: 289 CPU: 24
> IRQ: 290 CPU: 25
> IRQ: 291 CPU: 26
> IRQ: 292 CPU: 27
> IRQ: 293 CPU: 28
> IRQ: 294 CPU: 29
> IRQ: 295 CPU: 30
> IRQ: 299 CPU: 31
>
> This matches what we have on an x86_64 system.
Makes sense to me. FWIW
Acked-by: Michael S. Tsirkin <[email protected]>
> Laurent Vivier (2):
> genirq: add an affinity parameter to irq_create_mapping()
> powerpc/pseries: pass MSI affinity to irq_create_mapping()
>
> arch/arc/kernel/intc-arcv2.c | 4 ++--
> arch/arc/kernel/mcip.c | 2 +-
> arch/arm/common/sa1111.c | 2 +-
> arch/arm/mach-s3c/irq-s3c24xx.c | 3 ++-
> arch/arm/plat-orion/gpio.c | 2 +-
> arch/mips/ath25/ar2315.c | 4 ++--
> arch/mips/ath25/ar5312.c | 4 ++--
> arch/mips/lantiq/irq.c | 2 +-
> arch/mips/pci/pci-ar2315.c | 3 ++-
> arch/mips/pic32/pic32mzda/time.c | 2 +-
> arch/mips/ralink/irq.c | 2 +-
> arch/powerpc/kernel/pci-common.c | 2 +-
> arch/powerpc/kvm/book3s_xive.c | 2 +-
> arch/powerpc/platforms/44x/ppc476.c | 4 ++--
> arch/powerpc/platforms/cell/interrupt.c | 4 ++--
> arch/powerpc/platforms/cell/iommu.c | 3 ++-
> arch/powerpc/platforms/cell/pmu.c | 2 +-
> arch/powerpc/platforms/cell/spider-pic.c | 2 +-
> arch/powerpc/platforms/cell/spu_manage.c | 6 +++---
> arch/powerpc/platforms/maple/pci.c | 2 +-
> arch/powerpc/platforms/pasemi/dma_lib.c | 5 +++--
> arch/powerpc/platforms/pasemi/msi.c | 2 +-
> arch/powerpc/platforms/pasemi/setup.c | 4 ++--
> arch/powerpc/platforms/powermac/pci.c | 2 +-
> arch/powerpc/platforms/powermac/pic.c | 2 +-
> arch/powerpc/platforms/powermac/smp.c | 2 +-
> arch/powerpc/platforms/powernv/opal-irqchip.c | 5 +++--
> arch/powerpc/platforms/powernv/pci.c | 2 +-
> arch/powerpc/platforms/powernv/vas.c | 2 +-
> arch/powerpc/platforms/ps3/interrupt.c | 2 +-
> arch/powerpc/platforms/pseries/ibmebus.c | 2 +-
> arch/powerpc/platforms/pseries/msi.c | 2 +-
> arch/powerpc/sysdev/fsl_mpic_err.c | 2 +-
> arch/powerpc/sysdev/fsl_msi.c | 2 +-
> arch/powerpc/sysdev/mpic.c | 3 ++-
> arch/powerpc/sysdev/mpic_u3msi.c | 2 +-
> arch/powerpc/sysdev/xics/xics-common.c | 2 +-
> arch/powerpc/sysdev/xive/common.c | 2 +-
> arch/sh/boards/mach-se/7343/irq.c | 2 +-
> arch/sh/boards/mach-se/7722/irq.c | 2 +-
> arch/sh/boards/mach-x3proto/gpio.c | 2 +-
> arch/xtensa/kernel/perf_event.c | 2 +-
> arch/xtensa/kernel/smp.c | 2 +-
> arch/xtensa/kernel/time.c | 2 +-
> drivers/ata/pata_macio.c | 2 +-
> drivers/base/regmap/regmap-irq.c | 2 +-
> drivers/bus/moxtet.c | 2 +-
> drivers/clocksource/ingenic-timer.c | 2 +-
> drivers/clocksource/timer-riscv.c | 2 +-
> drivers/extcon/extcon-max8997.c | 3 ++-
> drivers/gpio/gpio-bcm-kona.c | 2 +-
> drivers/gpio/gpio-brcmstb.c | 2 +-
> drivers/gpio/gpio-davinci.c | 2 +-
> drivers/gpio/gpio-em.c | 3 ++-
> drivers/gpio/gpio-grgpio.c | 2 +-
> drivers/gpio/gpio-mockup.c | 2 +-
> drivers/gpio/gpio-mpc8xxx.c | 2 +-
> drivers/gpio/gpio-mvebu.c | 2 +-
> drivers/gpio/gpio-tb10x.c | 2 +-
> drivers/gpio/gpio-tegra.c | 2 +-
> drivers/gpio/gpio-wm831x.c | 2 +-
> drivers/gpio/gpiolib.c | 2 +-
> drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 3 ++-
> drivers/gpu/ipu-v3/ipu-common.c | 2 +-
> drivers/hid/hid-rmi.c | 2 +-
> drivers/i2c/busses/i2c-cht-wc.c | 2 +-
> drivers/i2c/i2c-core-base.c | 2 +-
> drivers/i2c/muxes/i2c-mux-pca954x.c | 2 +-
> drivers/ide/pmac.c | 2 +-
> drivers/iio/dummy/iio_dummy_evgen.c | 3 ++-
> drivers/input/rmi4/rmi_bus.c | 2 +-
> drivers/irqchip/irq-ath79-misc.c | 3 ++-
> drivers/irqchip/irq-bcm2835.c | 3 ++-
> drivers/irqchip/irq-csky-mpintc.c | 2 +-
> drivers/irqchip/irq-eznps.c | 2 +-
> drivers/irqchip/irq-mips-gic.c | 8 +++++---
> drivers/irqchip/irq-mmp.c | 4 ++--
> drivers/irqchip/irq-versatile-fpga.c | 2 +-
> drivers/irqchip/irq-vic.c | 2 +-
> drivers/macintosh/macio_asic.c | 2 +-
> drivers/memory/omap-gpmc.c | 2 +-
> drivers/mfd/ab8500-core.c | 2 +-
> drivers/mfd/arizona-irq.c | 5 +++--
> drivers/mfd/db8500-prcmu.c | 2 +-
> drivers/mfd/mfd-core.c | 2 +-
> drivers/mfd/stmpe.c | 5 +++--
> drivers/mfd/tc3589x.c | 2 +-
> drivers/mfd/tps6586x.c | 2 +-
> drivers/mfd/wm8994-irq.c | 5 +++--
> drivers/misc/cxl/irq.c | 2 +-
> drivers/misc/ocxl/afu_irq.c | 2 +-
> drivers/misc/ocxl/link.c | 2 +-
> drivers/net/dsa/mv88e6xxx/chip.c | 2 +-
> drivers/net/dsa/mv88e6xxx/global2.c | 2 +-
> drivers/net/dsa/qca/ar9331.c | 2 +-
> drivers/net/dsa/rtl8366rb.c | 3 ++-
> drivers/net/ethernet/ibm/ibmvnic.c | 4 ++--
> drivers/net/usb/lan78xx.c | 2 +-
> drivers/pci/controller/pci-ftpci100.c | 2 +-
> drivers/pci/controller/pci-tegra.c | 2 +-
> drivers/pci/controller/pcie-rcar-host.c | 2 +-
> drivers/pci/controller/pcie-xilinx-cpm.c | 4 ++--
> drivers/pci/controller/pcie-xilinx.c | 2 +-
> drivers/pinctrl/mediatek/mtk-eint.c | 2 +-
> drivers/pinctrl/nomadik/pinctrl-abx500.c | 3 ++-
> drivers/pinctrl/pinctrl-at91-pio4.c | 3 ++-
> drivers/pinctrl/pinctrl-rockchip.c | 2 +-
> drivers/pinctrl/samsung/pinctrl-samsung.c | 2 +-
> drivers/pinctrl/sunxi/pinctrl-sunxi.c | 2 +-
> drivers/power/supply/lp8788-charger.c | 2 +-
> drivers/rtc/rtc-lp8788.c | 2 +-
> drivers/rtc/rtc-max8997.c | 3 ++-
> drivers/rtc/rtc-max8998.c | 3 ++-
> drivers/scsi/cxlflash/ocxl_hw.c | 2 +-
> drivers/ssb/driver_gpio.c | 4 ++--
> drivers/staging/hikey9xx/hi6421-spmi-pmic.c | 2 +-
> drivers/staging/octeon-usb/octeon-hcd.c | 2 +-
> drivers/tty/hvc/hvsi.c | 2 +-
> drivers/tty/serial/pmac_zilog.c | 6 +++---
> drivers/watchdog/octeon-wdt-main.c | 2 +-
> include/linux/irqdomain.h | 3 ++-
> kernel/irq/irqdomain.c | 8 +++++---
> sound/soc/codecs/rt5677.c | 2 +-
> 123 files changed, 171 insertions(+), 146 deletions(-)
>
> --
> 2.28.0
>