Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Sun, 16 Jan 2022 12:07:59 +0000
Message-ID: <87v8yjyjc0.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     John Garry <john.garry@huawei.com>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        chenxiang <chenxiang66@hisilicon.com>,
        Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "liuqi (BA)" <liuqi115@huawei.com>
Subject: Re: PCI MSI issue for maxcpus=1
In-Reply-To: <19d55cdf-9ef7-e4a3-5ae5-0970f0d7751b@huawei.com>
References: <78615d08-1764-c895-f3b7-bfddfbcbdfb9@huawei.com>
        <87a6g8vp8k.wl-maz@kernel.org>
        <19d55cdf-9ef7-e4a3-5ae5-0970f0d7751b@huawei.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

On Fri, 07 Jan 2022 11:24:38 +0000,
John Garry <john.garry@huawei.com> wrote:
> 
> Hi Marc,
> 
> >> So it's the driver call to pci_alloc_irq_vectors_affinity() which
> >> errors [1]:
> >> 
> >> [    9.619070] hisi_sas_v3_hw: probe of 0000:74:02.0 failed with error -2
> > Can you log what error is returned from pci_alloc_irq_vectors_affinity()?
> 
> -EINVAL
> 
> > 
> >> Some details:
> >> - device supports 32 MSI
> >> - min and max msi for that function is 17 and 32, respect.
> > This 17 is a bit odd, owing to the fact that MultiMSI can only deal
> > with powers of 2. You will always allocate 32 in this case. Not sure
> > why that'd cause an issue though. Unless...
> 
> Even though 17 is the min, we still try for nvec=32 in
> msi_capability_init() as possible CPUs is 96.
> 
> > 
> >> - affd pre and post are 16 and 0, respect.
> >> 
> >> I haven't checked to see what the issue is yet and I think that the
> >> pci_alloc_irq_vectors_affinity() usage is ok...
> > ... we really end-up with desc->nvec_used == 32 and try to activate
> > past vector 17 (which is likely to fail). Could you please check this?
> 
> Yeah, that looks to fail. Reason being that in the GIC ITS driver when
> we try to activate the irq for this managed interrupt all cpus in the
> affinity mask are offline. Calling its_irq_domain_activate() ->
> its_select_cpu() it gives cpu=nr_cpu_ids. The affinity mask for that
> interrupt is 24-29.

I guess that for managed interrupts, it shouldn't matter, as these
interrupts should only be used when the relevant CPUs come online.

Would something like below help? Totally untested, as I don't have a
Multi-MSI capable device that I can plug in a GICv3 system (maybe I
should teach that to a virtio device...).

Thanks,

	M.

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index d25b7a864bbb..850407294adb 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1632,6 +1632,10 @@ static int its_select_cpu(struct irq_data *d,
 			cpumask_and(tmpmask, tmpmask, cpumask_of_node(node));
 
 		cpu = cpumask_pick_least_loaded(d, tmpmask);
+
+		/* If all the possible CPUs are offline, just pick a victim. */
+		if (cpu == nr_cpu_ids)
+			cpu = cpumask_pick_least_loaded(d, irq_data_get_affinity_mask(d));
 	}
 out:
 	free_cpumask_var(tmpmask);

-- 
Without deviation from the norm, progress is not possible.