Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Thomas Gleixner <tglx@linutronix.de>
To:     "Dey\, Megha" <megha.dey@intel.com>,
        Jason Gunthorpe <jgg@mellanox.com>
Cc:     Marc Zyngier <maz@kernel.org>,
        "Jiang\, Dave" <dave.jiang@intel.com>,
        "vkoul\@kernel.org" <vkoul@kernel.org>,
        "bhelgaas\@google.com" <bhelgaas@google.com>,
        "rafael\@kernel.org" <rafael@kernel.org>,
        "gregkh\@linuxfoundation.org" <gregkh@linuxfoundation.org>,
        "hpa\@zytor.com" <hpa@zytor.com>,
        "alex.williamson\@redhat.com" <alex.williamson@redhat.com>,
        "Pan\, Jacob jun" <jacob.jun.pan@intel.com>,
        "Raj\, Ashok" <ashok.raj@intel.com>,
        "Liu\, Yi L" <yi.l.liu@intel.com>,
        "Lu\, Baolu" <baolu.lu@intel.com>,
        "Tian\, Kevin" <kevin.tian@intel.com>,
        "Kumar\, Sanjay K" <sanjay.k.kumar@intel.com>,
        "Luck\, Tony" <tony.luck@intel.com>,
        "Lin\, Jing" <jing.lin@intel.com>,
        "Williams\, Dan J" <dan.j.williams@intel.com>,
        "kwankhede\@nvidia.com" <kwankhede@nvidia.com>,
        "eric.auger\@redhat.com" <eric.auger@redhat.com>,
        "parav\@mellanox.com" <parav@mellanox.com>,
        "Hansen\, Dave" <dave.hansen@intel.com>,
        "netanelg\@mellanox.com" <netanelg@mellanox.com>,
        "shahafs\@mellanox.com" <shahafs@mellanox.com>,
        "yan.y.zhao\@linux.intel.com" <yan.y.zhao@linux.intel.com>,
        "pbonzini\@redhat.com" <pbonzini@redhat.com>,
        "Ortiz\, Samuel" <samuel.ortiz@intel.com>,
        "Hossain\, Mona" <mona.hossain@intel.com>,
        "dmaengine\@vger.kernel.org" <dmaengine@vger.kernel.org>,
        "linux-kernel\@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "x86\@kernel.org" <x86@kernel.org>,
        "linux-pci\@vger.kernel.org" <linux-pci@vger.kernel.org>,
        "kvm\@vger.kernel.org" <kvm@vger.kernel.org>
Subject: RE: [PATCH RFC v2 02/18] irq/dev-msi: Add support for a new DEV_MSI irq domain
In-Reply-To: <c6a1c065ab9b46bbaf9f5713462085a5@intel.com>
References: <159534667974.28840.2045034360240786644.stgit@djiang5-desk3.ch.intel.com> <159534734833.28840.10067945890695808535.stgit@djiang5-desk3.ch.intel.com> <878sfbxtzi.wl-maz@kernel.org> <20200722195928.GN2021248@mellanox.com> <96a1eb5ccc724790b5404a642583919d@intel.com> <20200805221548.GK19097@mellanox.com> <70465fd3a7ae428a82e19f98daa779e8@intel.com> <20200805225330.GL19097@mellanox.com> <630e6a4dc17b49aba32675377f5a50e0@intel.com> <20200806001927.GM19097@mellanox.com> <c6a1c065ab9b46bbaf9f5713462085a5@intel.com>
Date:   Thu, 06 Aug 2020 19:10:05 +0200
Message-ID: <87tuxfhf9u.fsf@nanos.tec.linutronix.de>
MIME-Version: 1.0
Content-Type: text/plain
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Megha,

"Dey, Megha" <megha.dey@intel.com> writes:

>> -----Original Message-----
>> From: Jason Gunthorpe <jgg@mellanox.com>
<SNIP>
>> Subject: Re: [PATCH RFC v2 02/18] irq/dev-msi: Add support for a new DEV_MSI
>> irq domain

can you please fix your mail client not to copy the whole header of the
mail you are replying to into the mail body?

>> > > Well, I had suggested to pass in the parent struct device, but it
>> Oops, I was thinking of platform_msi_domain_alloc_irqs() not
>> create_device_domain()
>> 
>> ie call it in the device driver that wishes to consume the extra MSIs.
>> 
>> Is there a harm if each device driver creates a new irq_domain for its use?
>
> Well, the only harm is if we want to reuse the irq domain.

You cannot reuse the irq domain if you create a domain per driver. The
way how hierarchical domains work is:

vector --- DMAR-MSI
       |
       |-- ....
       |
       |-- IR-0 --- IO/APIC-0
       |        | 
       |        |-- IO/APIC-1
       |        |
       |        |-- PCI/MSI-0
       |        |
       |        |-- HPET/MSI-0
       |
       |-- IR-1 --- PCI/MSI-1
       |        |

The outermost domain is what the actual device driver uses. I.e. for
PCI-MSI it's the msi domain which is associated to the bus the device is
connected to. Each domain has its own interrupt chip instance and its
own data set.

Domains of the same type share the code, but neither the data nor the
interrupt chip instance.

Also there is a strict parent child relationship in terms of resources.
Let's look at PCI.

PCI/MSI-0 depends on IR-0 which depends on the vector domain. That's
reflecting both the flow of the interrupt and the steps required for
various tasks, e.g. allocation/deallocation and also interrupt chip
operations. In order to allocate a PCI/MSI interrupt in domain PCI/MSI-0
a slot in the remapping unit and a vector needs to be allocated.

If you disable interrupt remapping all the outermost domains in the
scheme above become childs of the vector domain.

So if we look at DEV/MSI as a infrastructure domain then the scheme
looks like this:

vector --- DMAR-MSI
       |
       |-- ....
       |
       |-- IR-0 --- IO/APIC-0
       |        | 
       |        |-- IO/APIC-1
       |        |
       |        |-- PCI/MSI-0
       |        |
       |        |-- HPET/MSI-0
       |        |
       |        |-- DEV/MSI-0
       |
       |-- IR-1 --- PCI/MSI-1
       |        |
       |        |-- DEV/MSI-1


But if you make it per device then you have multiple DEV/MSI domains per
IR unit.

What's the right thing to do?

If the DEV/MSI domain has it's own per IR unit resource management, then
you need one per IR unit.

If the resource management is solely per device then having a domain per
device is the right choice.

Thanks,

        tglx