Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <452d97ed-459f-7936-99e4-600380608615@huawei.com>
Date:   Mon, 7 Mar 2022 13:48:11 +0000
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.6.1
Subject: Re: PCI MSI issue for maxcpus=1
To:     Marc Zyngier <maz@kernel.org>
CC:     Thomas Gleixner <tglx@linutronix.de>,
        chenxiang <chenxiang66@hisilicon.com>,
        Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "liuqi (BA)" <liuqi115@huawei.com>, <wangxiongfeng2@huawei.com>,
        David Decotigny <decot@google.com>
References: <78615d08-1764-c895-f3b7-bfddfbcbdfb9@huawei.com>
 <87a6g8vp8k.wl-maz@kernel.org>
 <19d55cdf-9ef7-e4a3-5ae5-0970f0d7751b@huawei.com>
 <87v8yjyjc0.wl-maz@kernel.org> <87k0ey9122.wl-maz@kernel.org>
 <5f529b4e-1f6c-5a7d-236c-09ebe3a7db29@huawei.com>
 <1cbe7daa-8003-562b-06fa-5a50f7ee6ed2@huawei.com>
 <87a6e4tnkm.wl-maz@kernel.org>
From:   John Garry <john.garry@huawei.com>
In-Reply-To: <87a6e4tnkm.wl-maz@kernel.org>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi Marc,

> 
> diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
> index 2bdfce5edafd..97e9eb9aecc6 100644
> --- a/kernel/irq/msi.c
> +++ b/kernel/irq/msi.c
> @@ -823,6 +823,19 @@ static int msi_init_virq(struct irq_domain *domain, int virq, unsigned int vflag
>   	if (!(vflags & VIRQ_ACTIVATE))
>   		return 0;
>   
> +	if (!(vflags & VIRQ_CAN_RESERVE)) {
> +		/*
> +		 * If the interrupt is managed but no CPU is available
> +		 * to service it, shut it down until better times.
> +		 */
> +		if (irqd_affinity_is_managed(irqd) &&
> +		    !cpumask_intersects(irq_data_get_affinity_mask(irqd),
> +					cpu_online_mask)) {
> +			irqd_set_managed_shutdown(irqd);
> +			return 0;
> +		}
> +	}
> +
>   	ret = irq_domain_activate_irq(irqd, vflags & VIRQ_CAN_RESERVE);
>   	if (ret)
>   		return ret;
> 

Yeah, that seems to solve the issue. I will test it a bit more.

We need to check the isolcpus cmdline issue as well - wang xiongfeng, 
please assist here. I assume that this feature just never worked for 
arm64 since it was added.

> With this in place, I get the following results (VM booted with 4
> vcpus and maxcpus=1, the virtio device is using managed interrupts):
> 
> root@debian:~# cat /proc/interrupts
>             CPU0
>   10:       2298     GICv3  27 Level     arch_timer
>   12:         84     GICv3  33 Level     uart-pl011
>   49:          0     GICv3  41 Edge      ACPI:Ged
>   50:          0   ITS-MSI 16384 Edge      virtio0-config
>   51:       2088   ITS-MSI 16385 Edge      virtio0-req.0
>   52:          0   ITS-MSI 16386 Edge      virtio0-req.1
>   53:          0   ITS-MSI 16387 Edge      virtio0-req.2
>   54:          0   ITS-MSI 16388 Edge      virtio0-req.3
>   55:      11641   ITS-MSI 32768 Edge      xhci_hcd
>   56:          0   ITS-MSI 32769 Edge      xhci_hcd
> IPI0:         0       Rescheduling interrupts
> IPI1:         0       Function call interrupts
> IPI2:         0       CPU stop interrupts
> IPI3:         0       CPU stop (for crash dump) interrupts
> IPI4:         0       Timer broadcast interrupts
> IPI5:         0       IRQ work interrupts
> IPI6:         0       CPU wake-up interrupts
> Err:          0
> root@debian:~# echo 1 >/sys/devices/system/cpu/cpu2/online
> root@debian:~# cat /proc/interrupts
>             CPU0       CPU2
>   10:       2530         90     GICv3  27 Level     arch_timer
>   12:        103          0     GICv3  33 Level     uart-pl011
>   49:          0          0     GICv3  41 Edge      ACPI:Ged
>   50:          0          0   ITS-MSI 16384 Edge      virtio0-config
>   51:       2097          0   ITS-MSI 16385 Edge      virtio0-req.0
>   52:          0          0   ITS-MSI 16386 Edge      virtio0-req.1
>   53:          0         12   ITS-MSI 16387 Edge      virtio0-req.2
>   54:          0          0   ITS-MSI 16388 Edge      virtio0-req.3
>   55:      13487          0   ITS-MSI 32768 Edge      xhci_hcd
>   56:          0          0   ITS-MSI 32769 Edge      xhci_hcd
> IPI0:        38         45       Rescheduling interrupts
> IPI1:         3          3       Function call interrupts
> IPI2:         0          0       CPU stop interrupts
> IPI3:         0          0       CPU stop (for crash dump) interrupts
> IPI4:         0          0       Timer broadcast interrupts
> IPI5:         0          0       IRQ work interrupts
> IPI6:         0          0       CPU wake-up interrupts
> Err:          0
> 

Out of interest, is the virtio managed interrupts support just in your 
sandbox? You did mention earlier in the thread that you were considering 
adding this feature.

Thanks,
John