From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Thomas Gleixner <tglx@linutronix.de>,
        Jason Gunthorpe <jgg@nvidia.com>
CC:     "Jiang, Dave" <dave.jiang@intel.com>,
        Logan Gunthorpe <logang@deltatee.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Marc Zygnier <maz@kernel.org>,
        Alex Williamson <alex.williamson@redhat.com>,
        "Dey, Megha" <megha.dey@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jon Mason <jdmason@kudzu.us>, Allen Hubbe <allenbh@gmail.com>,
        "linux-ntb@googlegroups.com" <linux-ntb@googlegroups.com>,
        "linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
        Heiko Carstens <hca@linux.ibm.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>,
        "x86@kernel.org" <x86@kernel.org>, Joerg Roedel <jroedel@suse.de>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>
Subject: RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Thread-Topic: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Thread-Index: AQHX4y1VNaiO68omHEet2J9DWrVARKwa1YEAgAAp0QCAABreAIAAIi2AgAE8MoCAAAd1gIAACyoAgAAPcQCAADDDAIAAp1uAgABn9QCAACTxgIAAAdGAgAAbd4CAAA94gIAABpSAgAt9SoCAADlJAIAAgb+AgABGMACAAAc8gIAAE+oAgAAmHACAAFJ4oIAAcxMAgAFJwrA=
Date:   Sat, 11 Dec 2021 08:06:36 +0000
Message-ID: <BN9PR11MB5276AD88A1D1A1AA313E228C8C729@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <8c2262ba-173e-0007-bc4c-94ec54b2847d@intel.com>
 <87pmqg88xq.ffs@tglx> <df00b87e-00dc-d998-8b64-46b16dba46eb@intel.com>
 <87k0go8432.ffs@tglx> <f4cc305b-a329-6d27-9fca-b74ebc9fa0c1@intel.com>
 <878rx480fk.ffs@tglx>
 <BN9PR11MB52765F2EF8420C60FD5945D18C709@BN9PR11MB5276.namprd11.prod.outlook.com>
 <87sfv2yy19.ffs@tglx> <20211209162129.GS6385@nvidia.com>
 <878rwtzfh1.ffs@tglx> <20211209205835.GZ6385@nvidia.com>
 <8735n1zaz3.ffs@tglx> <87sfv1xq3b.ffs@tglx>
 <BN9PR11MB527619B099061B3814EB40408C719@BN9PR11MB5276.namprd11.prod.outlook.com>
 <87lf0sy7xd.ffs@tglx>
In-Reply-To: <87lf0sy7xd.ffs@tglx>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?TsgFBzavXdHiXThggHYvSi/xt+Ng+1jTWFoJQx6uaqQKakuwGDQ567cacjau?=
 =?us-ascii?Q?texhdPW92aWD68jliOwTD3iZGtWsZEXVnSsT2i752vIvEmXn5NsRq9Mym+Mk?=
 =?us-ascii?Q?8ENLN0vKL8A+FO+kDS9ZHjygwWuycV1pYtrp+35XnnweJ7X9zKeLotC6fzM4?=
 =?us-ascii?Q?eXT4jP8DZXMRJ/ztYDsfQI014wFjfKxKbwSBbM7yY0l2SxMI1GkBy9W9Gvlw?=
 =?us-ascii?Q?SDGI6lIf+f3RVl5ar+Q0no+tQZIN7PQO3VfA12aphKmu22ysRWcdWCiXRDqF?=
 =?us-ascii?Q?p1FFp9EdyD8w39es+9fOW+rdl/CBLIb2uBjsSgY8ONVPvL3HrvACgHEJHFoe?=
 =?us-ascii?Q?9APDLpN0LYSR94MXwhFDUlt3lFG3eVmXDiQMNtuoHIMySe5GWnpkor/nF50s?=
 =?us-ascii?Q?w7IvL2iuxTyF8nx3CV3xBuwzQzjyULO9/lugkZYC42eIDxzWu6mZdRJTNgPy?=
 =?us-ascii?Q?wgH6D5CdKa69K6qH/c6NJ+pNE+FPljAWEFJBX9OIfYiGpPh2kxKOvE0QPw9Y?=
 =?us-ascii?Q?19LdzimIG8JswZrj1KjXd9E/XRclNSiRhNaQhxbuoddaB1k6fx9TgIaMNZ7m?=
 =?us-ascii?Q?XU0g8xGwL0kt5aIeUiPFMaWZ7+1qmMTZdUy4tWEQr0AJQnoa2ywCMaxkLohf?=
 =?us-ascii?Q?V4MlL00VCUeFfbWwAIjxianMzt9+c486FtrFecgdhnhez4c8xthcbea0UUEf?=
 =?us-ascii?Q?lKpu9zNcBXh9Z7xd81TI3dowybdVXaRrAo0tT9RroS66nASTV4LcEnNX5KP7?=
 =?us-ascii?Q?XvRUas3cWm4UhYJ6PgEUckA/mjnylYGuQRMY9LeanFzPKx6KZf/VOagLfwVa?=
 =?us-ascii?Q?Mmx9rV1qznI9hspKc9yVdLqiXMPM9zMPzor6ogj/XfPjDEAiQJNDOkenqzQl?=
 =?us-ascii?Q?N0QerGBNIF/C8tl8lzAl1HMG03yp6hnW4LJpDM+sZYXJLgD3aP54oYKrTy7d?=
 =?us-ascii?Q?YG+TsiygdOXnIVyO2oHAw1kqhiQuWSUf6m0MskLl8LVcoPNDOU1BMyZkJG8c?=
 =?us-ascii?Q?NdqB+w3HWumrDf1K76rctPL4+5IeYlj5nCi6RCt+4KEIJURRA8RmFf+DlcEh?=
 =?us-ascii?Q?SpC/fsUdH0R0IgwlDW4+P+wcvzPtXH6nYMvR/DqTqisz2DFOFZl30GkPopli?=
 =?us-ascii?Q?LtRgYCgz9jCUn+sHYhd9G535ehy0oLS6LJpCQnP9DgdJ4yh61gla3pqFbS/s?=
 =?us-ascii?Q?2UyWfcICoJc5zhcIVUSZhbc1SW74EkJgb61GWWk0oi7r2aiEW5Jf7tCn0FUM?=
 =?us-ascii?Q?nekBps9ieHsrlWT3au0QOlCzh8CNGI52F/PjhoQdeV/8UOHvBVjA831EYUXY?=
 =?us-ascii?Q?w+ptO/3RHvBZivegtvECF/CugBVGv5GXDcG+4j9+RKZuM9sJZxh3DgDwAE7F?=
 =?us-ascii?Q?52F7umerPGptF2c3TvgGx+o0/0QIliqZUEso87Is65f0oXxPfUPUTlFT8YPc?=
 =?us-ascii?Q?+kMKcqsgyWHXzS5tvu4j2D5ktNfWWyPP5yJF5tvkTC4iV1efPy1tU88koGPI?=
 =?us-ascii?Q?z3D2JyNkySXpnBnLuU8t6A6OpRXNpTgdw4remPAeYG0vLBUuWT+Gl9+eLCr1?=
 =?us-ascii?Q?9cZL/zIQbPLsAGbrN8hc9hqEQttTdnBqpxEJIAUJqU95Yfx+7fxXMDOmBN8l?=
 =?us-ascii?Q?Qg=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b953aac0-9cde-4f96-d012-08d9bc7d276d
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Dec 2021 08:06:36.4692
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Ghq/f9EJPdoLe2ekr4h9pOtrW3oDDEK+WqdPDQCC8tWD4nPRZScmflVOX/hPLQxB1fU509Fzut5blRkqdz4O/A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR11MB3572
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Thomas Gleixner <tglx@linutronix.de>
> Sent: Friday, December 10, 2021 8:13 PM
>=20
> >>   5) It's not possible for the kernel to reliably detect whether it is
> >>      running on bare metal or not. Yes we talked about heuristics, but
> >>      that's something I really want to avoid.
> >
> > How would the hypercall mechanism avoid such heuristics?
>=20
> The availability of IR remapping where the irqdomain which is provided
> by the remapping unit signals that it supports this new scheme:
>=20
>                     |--IO/APIC
>                     |--MSI
>      vector -- IR --|--MSI-X
>                     |--IMS
>=20
> while the current scheme is:
>=20
>                     |--IO/APIC
>      vector -- IR --|--PCI/MSI[-X]
>=20
> or
>=20
>                     |--IO/APIC
>      vector --------|--PCI/MSI[-X]
>=20
> So in the new scheme the IR domain will advertise new features which are
> not available on older kernels. The availability of these new features
> is the indicator for the interrupt subsystem and subsequently for PCI
> whether IMS is supported or not.
>=20
> Bootup either finds an IR unit or not. In the bare metal case that's the
> usual hardware/firmware detection. In the guest case it's the
> availability of vIR including the required hypercall protocol.

Given we have vIR already, there are three scenarios:

1) Bare metal: IR (no hypercall, for sure)
2) VM: vIR (no hypercall, today)
3) VM: vIR (hypercall, tomorrow)

IMS should be allowed only for 1) and 3).

But how to differentiate 2) from 1) if no guest heuristics?

btw I checked Qemu history to find vIR was introduced in 2016:

commit 1121e0afdcfa0cd40e36bd3acff56a3fac4f70fd
Author: Peter Xu <peterx@redhat.com>
Date:   Thu Jul 14 13:56:13 2016 +0800

    x86-iommu: introduce "intremap" property

    Adding one property for intel-iommu devices to specify whether we shoul=
d
    support interrupt remapping. By default, IR is disabled. To enable it,
    we should use (take Intel IOMMU as example):

      -device intel_iommu,intremap=3Don

    This property can be shared by Intel and future AMD IOMMUs.

    Signed-off-by: Peter Xu <peterx@redhat.com>
    Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

>=20
> > Then Qemu needs to find out the GSI number for the vIRTE handle.
> > Again Qemu doesn't have such information since it doesn't know
> > which MSI[-X] entry points to this handle due to no trap.
> >
> > This implies that we may also need carry device ID, #msi entry, etc.
> > in the hypercall, so Qemu can associate the virtual routing info
> > to the right [irqfd, gsi].
> >
> > In your model the hypercall is raised by IR domain. Do you see
> > any problem of finding those information within IR domain?
>=20
> IR has the following information available:
>=20
>    Interrupt type
>     - MSI:   Device, index and number of vectors
>     - MSI-X: Device, index
>     - IMS:   Device, index
>=20
>   Target APIC/vector pair
>=20
> IMS: The index depends on the storage type:
>=20
>      For storage in device memory, e.g. IDXD, it's the array index.
>=20
>      For storage in system memory, the index is a software artifact.
>=20
> Does that answer your question?
>=20

Yes.

Thanks
Kevin