Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 25 Jan 2022 13:31:38 +0000
Message-ID: <87sftc6ix1.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Xiongfeng Wang <wangxiongfeng2@huawei.com>
Cc:     <tglx@linutronix.de>, <linux-kernel@vger.kernel.org>,
        <guohanjun@huawei.com>, John Garry <john.garry@huawei.com>
Subject: Re: [PATCH] irqchip/gic-v3-its: Select housekeeping CPUs preferentially for managed IRQs
In-Reply-To: <12ac7447-34dc-8497-b608-ada5a2ba17c4@huawei.com>
References: <20220124073440.88598-1-wangxiongfeng2@huawei.com>
        <87a6fl8jgb.wl-maz@kernel.org>
        <12ac7447-34dc-8497-b608-ada5a2ba17c4@huawei.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

On Tue, 25 Jan 2022 12:49:20 +0000,
Xiongfeng Wang <wangxiongfeng2@huawei.com> wrote:
> 
> Hi Marc,
> 
> On 2022/1/24 19:24, Marc Zyngier wrote:
> > + John Garry, as he was reporting issues around the same piece of code[1]
> > 
> > On Mon, 24 Jan 2022 07:34:40 +0000,
> > Xiongfeng Wang <wangxiongfeng2@huawei.com> wrote:
> >>
> >> When using kernel parameter 'isolcpus=managed_irq,xxxx' to bind the
> >> managed IRQs to housekeeping CPUs, the effective_affinity sometimes
> >> still contains the non-housekeeping CPUs.
> >>
> >> irq_do_set_affinity() passes the housekeeping cpumask to
> >> chip->irq_set_affinity(), but ITS driver select CPU according to
> >> irq_common_data->affinity. While 'irq_common_data->affinity' is updated
> >> after chip->irq_set_affinity() is called in irq_do_set_affinity(). Also
> >> 'irq_common_data->affinity' may contains non-housekeeping CPUs. I found
> >> the below link explaining the reason.
> >>   https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2267032.html
> >>
> >> To modify CPU selecting logic to prefer housekeeping CPUs, select CPU
> >> from the input cpumask parameter first. If none of it is online, then
> >> select CPU from 'irq_common_data->affinity'.
> >>
> >> Signed-off-by: Xiongfeng Wang <wangxiongfeng2@huawei.com>
> >> ---
> >>  drivers/irqchip/irq-gic-v3-its.c | 5 ++++-
> >>  1 file changed, 4 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> >> index d25b7a864bbb..17c15d3b2784 100644
> >> --- a/drivers/irqchip/irq-gic-v3-its.c
> >> +++ b/drivers/irqchip/irq-gic-v3-its.c
> >> @@ -1624,7 +1624,10 @@ static int its_select_cpu(struct irq_data *d,
> >>  
> >>  		cpu = cpumask_pick_least_loaded(d, tmpmask);
> >>  	} else {
> >> -		cpumask_and(tmpmask, irq_data_get_affinity_mask(d), cpu_online_mask);
> >> +		cpumask_and(tmpmask, aff_mask, cpu_online_mask);
> >> +		if (cpumask_empty(tmpmask))
> >> +			cpumask_and(tmpmask, irq_data_get_affinity_mask(d),
> >> +				    cpu_online_mask);
> > 
> > I think that the online_cpu_mask logical and is a bit wrong. A managed
> > interrupt should be able to target an offline CPU:
> > 
> > diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> > index eb0882d15366..0cea46bdaf99 100644
> > --- a/drivers/irqchip/irq-gic-v3-its.c
> > +++ b/drivers/irqchip/irq-gic-v3-its.c
> > @@ -1620,7 +1620,7 @@ static int its_select_cpu(struct irq_data *d,
> >  
> >  		cpu = cpumask_pick_least_loaded(d, tmpmask);
> >  	} else {
> > -		cpumask_and(tmpmask, irq_data_get_affinity_mask(d), cpu_online_mask);
> > +		cpumask_copy(tmpmask, aff_mask);
> >  
> >  		/* If we cannot cross sockets, limit the search to that node */
> >  		if ((its_dev->its->flags & ITS_FLAGS_WORKAROUND_CAVIUM_23144) &&
> 
> I have tested the above modification with 'maxcpus=1' kernel parameter and got
> the following CallTrace.
> 
> [   14.679493][    T5] pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS
> BTYPE=--)
> [   14.687114][    T5] pc : lpi_update_config+0xe0/0x300
> [   14.692146][    T5] lr : lpi_update_config+0x3c/0x300

That's a problem similar to what John was seeing: the CPU isn't there,
and a lot of stuff goes very wrong in the absence of a CPU targeted by
a managed interrupt.

> > We still have an issue when the system hasn't booted with all its
> > CPUs, as the corresponding collections aren't initialised and we
> > end-up in a rather bad place.
> 
> Shall we fix this 'effective CPU of managed IRQs is not housekeeping
> CPU' issue first, or we will wait until the 'maxcpus=1' issue is
> fixed.

I this we need to address this first. There is no point in only half
fixing it.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.