Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 10 Aug 2021 14:57:06 +0100
Message-ID: <8735rhbdxp.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Sunil Muthuswamy <sunilmut@microsoft.com>
Cc:     Robin Murphy <robin.murphy@arm.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "catalin.marinas@arm.com" <catalin.marinas@arm.com>,
        "will@kernel.org" <will@kernel.org>,
        Michael Kelley <mikelley@microsoft.com>,
        Boqun Feng <Boqun.Feng@microsoft.com>,
        KY Srinivasan <kys@microsoft.com>,
        Arnd Bergmann <arnd@arndb.de>,
        Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Subject: Re: [EXTERNAL] Re: [RFC 1/1] irqchip/gic-v3-its: Add irq domain and chip for Direct LPI without ITS
In-Reply-To: <MW4PR21MB20022D4903CDD8F989C30C50C0F79@MW4PR21MB2002.namprd21.prod.outlook.com>
References: <MW4PR21MB2002BA20FA4A8DAA1C27F4C6C0199@MW4PR21MB2002.namprd21.prod.outlook.com>
        <87a6mt2jke.wl-maz@kernel.org>
        <MW4PR21MB2002E51429F7E13B61A34FFEC0E89@MW4PR21MB2002.namprd21.prod.outlook.com>
        <87tuka24kj.wl-maz@kernel.org>
        <MW4PR21MB20027EAC23E8053210364E2BC0F09@MW4PR21MB2002.namprd21.prod.outlook.com>
        <b2dea108-9166-dc9d-abd0-d22491f78568@arm.com>
        <87r1f9wooc.wl-maz@kernel.org>
        <MW4PR21MB2002860BAED850B2B52D6E2BC0F19@MW4PR21MB2002.namprd21.prod.outlook.com>
        <87wnp0b86y.wl-maz@kernel.org>
        <MW4PR21MB2002ABD170CC9786CDF5D493C0F39@MW4PR21MB2002.namprd21.prod.outlook.com>
        <87o8a81boi.wl-maz@kernel.org>
        <MW4PR21MB200286E2CF4DA229D7059598C0F69@MW4PR21MB2002.namprd21.prod.outlook.com>
        <87pmunasik.wl-maz@kernel.org>
        <MW4PR21MB20022D4903CDD8F989C30C50C0F79@MW4PR21MB2002.namprd21.prod.outlook.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

On Tue, 10 Aug 2021 02:10:40 +0100,
Sunil Muthuswamy <sunilmut@microsoft.com> wrote:
> 
> On Monday, August 9, 2021 2:15 AM,
> Marc Zyngier <maz@kernel.org> wrote:
> [...]
> > If you plug directly into the GICv3 layer, I'd rather you inject SPIs,
> > just like any other non-architectural MSI controller. You can directly
> > interface with the ACPI GSI layer for that, without any need to mess
> > with the GICv3 internals. The SPI space isn't very large, but still
> > much larger than the equivalent x86 space (close to 1000).
> > 
> > If time is of the essence, I suggest you go the SPI way. For anything
> > involving LPIs, I really want to see a firmware spec that works for
> > everyone as opposed to a localised Hyper-V hack.
> > 
> Ok, thanks. Before we commit to anything, I would like to make sure
> that I am on the same page in terms of your description. With that in
> mind, I have few questions. Hopefully, these should settle the matter.
> 1. If we go with the SPI route, then the way I envision it is that the 
>     Hyper-V vPCI driver will implement an IRQ chip, which will take
>     care of allocating & managing the SPI interrupt for Hyper-V vPCI.
>     This IRQ chip will parent itself to the architectural GIC IRQ chip for
>     general interrupt management. Does that match with your
>     understanding/suggestion as well?

Yes.

> 
> 2. In the above, how will Hyper-V vPCI module discover the
>     architectural GIC IRQ domain generically for virtual devices that
>     are not firmware enumerated? Today, the GIC v3 IRQ domain is
>     not exported and the general 'irq_find_xyz' APIs only work for
>     firmware enumerated devices (i.e. something that has a fwnode
>     handle).

You don't need to discover it with ACPI. You simply instantiate your
own irqdomain using acpi_irq_create_hierarchy(), which will do the
right thing. Your PCI driver will have to create its own fwnode out of
thin air (there is an API for that), and call into this function to
plumb everything.

> 
> 3. Longer term, if we implement LPIs (with an ITS or Direct LPI), to
>     be able to support all scenarios such as Live Migration, the
>     Hyper-V virtual PCI driver would like to be able to control the
>     MSI address & data that gets programmed on the device
>    (i.e. .irq_compose_msi_msg). We can use the architectural
>    methods for everything else. Does that fit into the realm of
>    what would be acceptable upstream?

I cannot see how this works. The address has to match that of the
virtual HW you target (whether this is a redistributor or an ITS), and
the data is only meaningful in that context. And it really shouldn't
matter at all, as I expect you don't let the guest directly write to
the PCI MSI-X table.

If you let the guest have access direct to that table (which seems to
contradict your "live migration" argument), then your best bet is to
use provide a skeletal IOMMU implementation, and get
iommu_dma_compose_msi_msg() to do the remapping. But frankly, that's
horrible and I fully expect the IOMMU people to push back (and that
still doesn't give you any control over the data, only the address).

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.