Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <6410d0d5-bd88-64d5-6ae7-cca33746b302@quicinc.com>
Date:   Fri, 8 Apr 2022 11:34:17 -0600
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.6.0
Content-Language: en-US
To:     Thomas Gleixner <tglx@linutronix.de>
CC:     Marc Zyngier <maz@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
From:   Jeffrey Hugo <quic_jhugo@quicinc.com>
Subject: x86 multi-msi w/o iommu
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi Thomas,

I'd like to get multi-MSI without IOMMU working on x86.  I'm hoping you 
could help me understand the current state of things, and if there is a 
path toward enabling this.

This is an overly simplistic assessment, but this reportedly works in 
other OSes, so it looks like Linux is broken in comparison.

In my investigation so far, the failure stems from 
x86_vector_alloc_irqs() in arch/x86/kernel/apic/vector.c [1].  If we 
need a contiguous allocation of more than one irq, the allocation 
immediately fails:

	/* Currently vector allocator can't guarantee contiguous allocations */
	if ((info->flags & X86_IRQ_ALLOC_CONTIGUOUS_VECTORS) && nr_irqs > 1)
		return -ENOSYS;


As I'm sure you are aware, this only impacts MSI without IOMMU as both 
MSI-X and MSI with IOMMU can handle a discontinuous allocation (the flag 
is not set for either of those cases).

That check was added back in 2015 with [2].  In 2017, it looks like you 
refactored the allocator to the irq_matrix component [3].  However, the 
limitation remains to today.

Digging a bit further, it looks like the internal function 
matrix_alloc_area() [4] is capable of doing a contiguous allocation, but 
all the callers of that via the public api hardcode num to 1.  I 
wouldn't say it would be trivial to refactor the irq_matrix public api 
to do a contigious range allocation, and refactor 
x86_vector_alloc_irqs() to do that based on 
X86_IRQ_ALLOC_CONTIGUOUS_VECTORS, but since it seems like that hasn't 
been tackled in 5-7 years (depending on how you look at the history), I 
suspect I'm missing something.

Since I'm poking my nose in an area of the kernel that I haven't gone 
poking into before, I'm guessing there is a lot I don't know.  Would you 
kindly educate me on what the concerns might be?

I look forward to your expertise and opinions.

Thanks

-Jeff


[1]: 
https://elixir.bootlin.com/linux/v5.18-rc1/source/arch/x86/kernel/apic/vector.c#L542
[2]: 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/x86/kernel/apic/vector.c?h=v5.18-rc1&id=b5dc8e6c21e7ffba0246bf39cea97805c142bf85
[3]: 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/x86/kernel/apic/vector.c?h=v5.18-rc1&id=69cde0004a4b5cfc7d1cec4ef9ce4cf4e26142f0
[4]: 
https://elixir.bootlin.com/linux/v5.18-rc1/source/kernel/irq/matrix.c#L110