Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH] x86/apic/vector: Fix ordering in vector assignment
To:     Thomas Gleixner <tglx@linutronix.de>,
        LKML <linux-kernel@vger.kernel.org>
Cc:     x86@kernel.org, Shung-Hsi Yu <shung-hsi.yu@suse.com>,
        Ming Lei <ming.lei@redhat.com>, Peter Xu <peterx@redhat.com>
References: <87ft4djtyp.fsf@nanos.tec.linutronix.de>
From:   Prarit Bhargava <prarit@redhat.com>
Message-ID: <c5bb2239-dd40-03b2-d9d7-dace7499172d@redhat.com>
Date:   Fri, 11 Dec 2020 07:24:50 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.6.1
MIME-Version: 1.0
In-Reply-To: <87ft4djtyp.fsf@nanos.tec.linutronix.de>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 12/10/20 3:18 PM, Thomas Gleixner wrote:
> Prarit reported that depending on the affinity setting the
> 
>  ' irq $N: Affinity broken due to vector space exhaustion.'
> 
> message is showing up in dmesg, but the vector space on the CPUs in the
> affinity mask is definitely not exhausted.
> 
> Shung-Hsi provided traces and analysis which pinpoints the problem:
> 
> The ordering of trying to assign an interrupt vector in
> assign_irq_vector_any_locked() is simply wrong if the interrupt data has a
> valid node assigned. It does:
> 
>  1) Try the intersection of affinity mask and node mask
>  2) Try the node mask
>  3) Try the full affinity mask
>  4) Try the full online mask
> 
> Obviously #2 and #3 are in the wrong order as the requested affinity
> mask has to take precedence.
> 
> In the observed cases #1 failed because the affinity mask did not contain
> CPUs from node 0. That made it allocate a vector from node 0, thereby
> breaking affinity and emitting the misleading message.
> 
> Revert the order of #2 and #3 so the full affinity mask without the node
> intersection is tried before actually affinity is broken.
> 
> If no node is assigned then only the full affinity mask and if that fails
> the full online mask is tried.
> 
> Fixes: d6ffc6ac83b1 ("x86/vector: Respect affinity mask in irq descriptor")
> Reported-by: Shung-Hsi Yu <shung-hsi.yu@suse.com>
> Reported-by: Prarit Bhargava <prarit@redhat.com>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Tested-by: Shung-Hsi Yu <shung-hsi.yu@suse.com>
> Cc: stable@vger.kernel.org
> ---
>  arch/x86/kernel/apic/vector.c |   24 ++++++++++++++----------
>  1 file changed, 14 insertions(+), 10 deletions(-)
> 
> --- a/arch/x86/kernel/apic/vector.c
> +++ b/arch/x86/kernel/apic/vector.c
> @@ -273,20 +273,24 @@ static int assign_irq_vector_any_locked(
>  	const struct cpumask *affmsk = irq_data_get_affinity_mask(irqd);
>  	int node = irq_data_get_node(irqd);
>  
> -	if (node == NUMA_NO_NODE)
> -		goto all;
> -	/* Try the intersection of @affmsk and node mask */
> -	cpumask_and(vector_searchmask, cpumask_of_node(node), affmsk);
> -	if (!assign_vector_locked(irqd, vector_searchmask))
> -		return 0;
> -	/* Try the node mask */
> -	if (!assign_vector_locked(irqd, cpumask_of_node(node)))
> -		return 0;
> -all:
> +	if (node != NUMA_NO_NODE) {
> +		/* Try the intersection of @affmsk and node mask */
> +		cpumask_and(vector_searchmask, cpumask_of_node(node), affmsk);
> +		if (!assign_vector_locked(irqd, vector_searchmask))
> +			return 0;
> +	}
> +
>  	/* Try the full affinity mask */
>  	cpumask_and(vector_searchmask, affmsk, cpu_online_mask);
>  	if (!assign_vector_locked(irqd, vector_searchmask))
>  		return 0;
> +
> +	if (node != NUMA_NO_NODE) {
> +		/* Try the node mask */
> +		if (!assign_vector_locked(irqd, cpumask_of_node(node)))
> +			return 0;
> +	}
> +
>  	/* Try the full online mask */
>  	return assign_vector_locked(irqd, cpu_online_mask);
>  }
> 

Tested-and-Reviewed-by: Prarit Bhargava <prarit@redhat.com>

P.