From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Thomas Gleixner <tglx@linutronix.de>
CC:     "Jiang, Dave" <dave.jiang@intel.com>,
        Logan Gunthorpe <logang@deltatee.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Marc Zygnier <maz@kernel.org>,
        Alex Williamson <alex.williamson@redhat.com>,
        "Dey, Megha" <megha.dey@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jon Mason <jdmason@kudzu.us>, Allen Hubbe <allenbh@gmail.com>,
        "linux-ntb@googlegroups.com" <linux-ntb@googlegroups.com>,
        "linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
        Heiko Carstens <hca@linux.ibm.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>,
        "x86@kernel.org" <x86@kernel.org>, Joerg Roedel <jroedel@suse.de>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>
Subject: RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Thread-Topic: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Thread-Index: AQHX4y1VNaiO68omHEet2J9DWrVARKwa1YEAgAAp0QCAABreAIAAIi2AgAE8MoCAAAd1gIAACyoAgAAPcQCAADDDAIAAp1uAgABn9QCAACTxgIAAAdGAgAAbd4CAAA94gIAABpSAgAt9SoCAADlJAIAAgb+AgABGMACAAAc8gIAAsDnw
Date:   Fri, 10 Dec 2021 07:36:12 +0000
Message-ID: <BN9PR11MB5276599F467AD5EAC935A79E8C719@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <8c2262ba-173e-0007-bc4c-94ec54b2847d@intel.com>
 <87pmqg88xq.ffs@tglx> <df00b87e-00dc-d998-8b64-46b16dba46eb@intel.com>
 <87k0go8432.ffs@tglx> <f4cc305b-a329-6d27-9fca-b74ebc9fa0c1@intel.com>
 <878rx480fk.ffs@tglx>
 <BN9PR11MB52765F2EF8420C60FD5945D18C709@BN9PR11MB5276.namprd11.prod.outlook.com>
 <87sfv2yy19.ffs@tglx> <20211209162129.GS6385@nvidia.com>
 <878rwtzfh1.ffs@tglx> <20211209205835.GZ6385@nvidia.com>
In-Reply-To: <20211209205835.GZ6385@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
dlp-version: 11.6.200.16
dlp-product: dlpe-windows
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Wmb2C6oQBnZDq0fxSZ84ZZn/PRr1Vx17p/I32hO0lFF1oFigSADhb4HpaBdb?=
 =?us-ascii?Q?sWOhwEiyaICpgl+ujhIIwmKe6Jk+HAV9EsEIIJBMb63EW5vVh0Ffc2/l6YDp?=
 =?us-ascii?Q?8Nj19/bLckVyKkIpr0etXlMi9Jemh36CPGz7NWWtUHP65hDJxgNadkpuLsU9?=
 =?us-ascii?Q?eLOODCEvrBIfuzr4uN8Yz8+p97w5bf9ROV9+rYuz6U1guIi4irawGlQzwE4M?=
 =?us-ascii?Q?k+cWsWf/Eoi6sQ5X4v6O9JSLUj8+3PX5xUdcpbDHPhZy9dvLfNeyO1rEv6fb?=
 =?us-ascii?Q?9/zMrXESKg+Dpz7L5RFKuR7sTQDpaGgIkfZJ6w2/a//OEKtHaHxdHXDUci5B?=
 =?us-ascii?Q?PvdY358maYWlbfp4EkB2NDB5yB7kyxA1+Xi2uogNmXbp/KgxlPDnIPAlwHYn?=
 =?us-ascii?Q?ysLdNvIqpPri06rPfz+OV3HeSv0+RPjtUb+s4UkRhooAT2l70w5hZIW9Ayl8?=
 =?us-ascii?Q?GhfF4kJ7EyMJn6hX6e/6WdHV4V7y7/U1IF8TvSoxtnO+QU+KU8e5efeyhk2R?=
 =?us-ascii?Q?dV2gLO/odK8gU2a/XTTD7lzANKzogy74QfuRgoCT2dSoCUzgp11yBzDqV26B?=
 =?us-ascii?Q?rUsodh/I+TsxtCEKFlPLjeNm6ocWUQ4ZQ6s8LYF79u249hxOEIkUiqmE7eKZ?=
 =?us-ascii?Q?1fWg7kC37jyv5KnAjtx5mV/jDXOMc5zF5mDGvIqS9zpL1vC/NPeL+pn11LFv?=
 =?us-ascii?Q?C+hAGgxRWqmWClO9eB5d3B2ZB+0mk2wru1AJLKlXIDd3S574zrDrhiV724vq?=
 =?us-ascii?Q?BKDJ5IMvojGw2290eWKjn2ma3rOnFX5kTiAGHAzT3qlK874jHlVvGm2/d6ze?=
 =?us-ascii?Q?Kk3PffRVrAHkOe6PiqXw5jeqRhU25lJyl/Nn+Z0rTDeL7dPYDKxEPCxWYBcq?=
 =?us-ascii?Q?a4XJeVgT8YQqQ7SfKXbCm6SRT6RPXVWDDMQd+vHFlAzWUvV+iH4h8Afz0ZZo?=
 =?us-ascii?Q?aA3vD5IspE54bxrCvwAhGecUXIv0pHeEipGrZiwl6YewYuS8CQNdBa+da/Ro?=
 =?us-ascii?Q?JS7KQwL9FEXJreBhutES7keSFFjuZkDLcJ8/aTn3s6vVgloKXO4AbY7dHSvo?=
 =?us-ascii?Q?nKIQEAXnFhGHdc4lVVqM6X6gJHq+gqWaqxEfKaPasOQiFYBhHTJeVl97Cihq?=
 =?us-ascii?Q?YrKAeiBSsFe4nvk1/D5JVsL4Z2cHFapMdYWh0aSXUE+dLAB8E7Ws1IGps4+O?=
 =?us-ascii?Q?PVQlncAZwMcLwW9fjAlHQH8EFfajzFsFCL4u1AcddYUzjTACNtpE3tEAqSHo?=
 =?us-ascii?Q?qOJD3vfH7lS1zCNmJpeKGV4tjyP/SSwMoDa9J63j69siYkDLZppktYL0aWxv?=
 =?us-ascii?Q?sam6Yc4JzTr02o9LmCGVCxi2/ygdZGQXAnSIeX6fmIMO26/AIpBdHL/3ksMG?=
 =?us-ascii?Q?+qHCF29Weh4qmphKdov7dB3fgBzhnmjqrLcLU6hGYwTzE7xvg1MbOcf1eHhE?=
 =?us-ascii?Q?ek2SAO0arTppU5VI+p5KuDw5zIK73IUkrJWujJuzKmIqCvt9jphamaaGFaIX?=
 =?us-ascii?Q?VFj83GiWgrU/CD0V4EoHkdq0vuPwDs+lTEunIGZ5EG725/ZnArWYxSV5guCK?=
 =?us-ascii?Q?aEfETK8ZGpkftPNlxIDq7/4G7WDGSyBCgoFTLQ5ToBHbYPYBG7Nrp2MFBwVl?=
 =?us-ascii?Q?ugLNzeSyq5fNq3prMCFbmA4=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 96020c36-a142-44b3-4b85-08d9bbafbdce
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Dec 2021 07:36:12.6465
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: d5XyiUZhVgoGIPDkzzxU1E3ZeqQdwLaczRXq1rj6UoyRfucFHCTHVxw6BZRLXXNt04o4Bz4zswtD317I+psKSg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR11MB3652
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Friday, December 10, 2021 4:59 AM
>=20
> On Thu, Dec 09, 2021 at 09:32:42PM +0100, Thomas Gleixner wrote:
> > On Thu, Dec 09 2021 at 12:21, Jason Gunthorpe wrote:
> > > On Thu, Dec 09, 2021 at 09:37:06AM +0100, Thomas Gleixner wrote:
> > > If we keep the MSI emulation in the hypervisor then MSI !=3D IMS.  Th=
e
> > > MSI code needs to write a addr/data pair compatible with the emulatio=
n
> > > and the IMS code needs to write an addr/data pair from the
> > > hypercall. Seems like this scenario is best avoided!
> > >
> > > From this perspective I haven't connected how virtual interrupt
> > > remapping helps in the guest? Is this a way to provide the hypercall
> > > I'm imagining above?
> >
> > That was my thought to avoid having different mechanisms.
> >
> > The address/data pair is computed in two places:
> >
> >   1) Activation of an interrupt
> >   2) Affinity setting on an interrupt
> >
> > Both configure the IRTE when interrupt remapping is in place.
> >
> > In both cases a vector is allocated in the vector domain and based on
> > the resulting target APIC / vector number pair the IRTE is
> > (re)configured.
> >
> > So putting the hypercall into the vIRTE update is the obvious
> > place. Both activation and affinity setting can fail and propagate an
> > error code down to the originating caller.
> >
> > Hmm?
>=20
> Okay, I think I get it. Would be nice to have someone from intel
> familiar with the vIOMMU protocols and qemu code remark what the
> hypervisor side can look like.
>=20
> There is a bit more work here, we'd have to change VFIO to somehow
> entirely disconnect the kernel IRQ logic from the MSI table and
> directly pass control of it to the guest after the hypervisor IOMMU IR
> secures it. ie directly mmap the msi-x table into the guest
>=20

It's supported already:

/*
 * The MSIX mappable capability informs that MSIX data of a BAR can be mmap=
ped
 * which allows direct access to non-MSIX registers which happened to be wi=
thin
 * the same system page.
 *
 * Even though the userspace gets direct access to the MSIX data, the exist=
ing
 * VFIO_DEVICE_SET_IRQS interface must still be used for MSIX configuration=
.
 */
#define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE      3

IIRC this was introduced for PPC when a device has MSI-X in the same BAR as
other MMIO registers. Trapping MSI-X leads to performance downgrade on
accesses to adjacent registers. MSI-X can be mapped by userspace because
PPC already uses a hypercall mechanism for interrupt. Though unclear about
the detail it sounds a similar usage as proposed here.

Thanks
Kevin