Date:   Thu, 9 Dec 2021 16:58:35 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        Logan Gunthorpe <logang@deltatee.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Marc Zygnier <maz@kernel.org>,
        Alex Williamson <alex.williamson@redhat.com>,
        "Dey, Megha" <megha.dey@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jon Mason <jdmason@kudzu.us>, Allen Hubbe <allenbh@gmail.com>,
        "linux-ntb@googlegroups.com" <linux-ntb@googlegroups.com>,
        "linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
        Heiko Carstens <hca@linux.ibm.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>,
        "x86@kernel.org" <x86@kernel.org>, Joerg Roedel <jroedel@suse.de>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>
Subject: Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Message-ID: <20211209205835.GZ6385@nvidia.com>
References: <8c2262ba-173e-0007-bc4c-94ec54b2847d@intel.com>
 <87pmqg88xq.ffs@tglx>
 <df00b87e-00dc-d998-8b64-46b16dba46eb@intel.com>
 <87k0go8432.ffs@tglx>
 <f4cc305b-a329-6d27-9fca-b74ebc9fa0c1@intel.com>
 <878rx480fk.ffs@tglx>
 <BN9PR11MB52765F2EF8420C60FD5945D18C709@BN9PR11MB5276.namprd11.prod.outlook.com>
 <87sfv2yy19.ffs@tglx>
 <20211209162129.GS6385@nvidia.com>
 <878rwtzfh1.ffs@tglx>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <878rwtzfh1.ffs@tglx>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?RGnNdxm3ODL9Lj9rmlJhmr6W/sE9lIRuiqSpM31NWtci3finTvgujJVEKysP?=
 =?us-ascii?Q?1SmrtVbhI5yyfP1uRWXFfpiUzHYTALqiV8USQIcx8yKizHZgoKSCN9FuGacF?=
 =?us-ascii?Q?Uu7q6VotlLTQLnno6DwtvMMY0PvqkgtTymdIqQFZ20fLMxMbOSLEFGfAogvq?=
 =?us-ascii?Q?Xf4QYtWZ2DzgsdnfdaPVD50fVJvblWTIXdroVNDaUrgfGNMQdq/i2+FX6lDs?=
 =?us-ascii?Q?sYyTXrSHryu03USidqoa/UBXETj4LRAipEvbi2ttLMlT/uNLS3/2Uxnon38K?=
 =?us-ascii?Q?RW27Eowr3uNwGtrNAztymB7OSu4Ec+ABMp+Vhy+L0zpMWvCymhcG2+NyOkyd?=
 =?us-ascii?Q?YmXFi+zaivJMnYkEDuU8ukPLKO6miyFaUSRx33IgPLvS9Cc5LCe9YRImhstB?=
 =?us-ascii?Q?vv8EhFhOMM4k+gKibvmZcBlgT7vWCtmDzq715GMZ5MPwqjO1URgtE3R58nBk?=
 =?us-ascii?Q?HZI7GhDuwM/K/yiNm9T/+bNloTICaqWO6Kb8heHnhVKIHm+KmFjcxiY7V7x/?=
 =?us-ascii?Q?oA55mDx7n6yqaEM3TfHKNCrEI/RdZ7bQsebU1KWhfUcib07F0mIQp4Zb7qBi?=
 =?us-ascii?Q?hGNBexoIXkOrbijxJwEtZBC9sgzT/Qww74PuVgOgKEESSVW5ZUflHEU0LXwe?=
 =?us-ascii?Q?d7e+npw7P2HOKySYIMNQ/ar1+fHOsHNbBtzg4/NXEB69C38yBToiBvp2zjbP?=
 =?us-ascii?Q?kiY81gRhn2MnIg6Ins27of7NDUgJIybLZCfOs/sjT1f9MAseSLBY9Oit2wvw?=
 =?us-ascii?Q?GjkcTGgbIClOQFcpsc34HDKWodUKp9Prh7IYCt2mniJy37pGbjsZSIj95iyW?=
 =?us-ascii?Q?hdXdWsBDPEAU5F4QOiaWM6M4FMMOAKB7UkBuQRw6wy/rVli6c0AjP2Co9Klb?=
 =?us-ascii?Q?RPeYo8+AC8ICqV+u/ZW+p9y+2FqAJ0JFap/i0i7aGAFurXUXjvsR3DAlCl92?=
 =?us-ascii?Q?T/UZGoSpTuKs7IhYOby0z0sAhWX/dgl/JITPG1mlTv2/mYxLh8McQgt63FeP?=
 =?us-ascii?Q?AIXhh3VLR5RLhZFjh4uJeJBt1ow1zqkrS7MvB/7T38ry1Msj33l1zvSKtU/h?=
 =?us-ascii?Q?7xBfR1HxCxn+1E8X3tzKCZ7cdgUv4L/ZOujyagizPrRYKeadNhlnDs64EiBQ?=
 =?us-ascii?Q?XvC/+P3HtnCvXt2+TUpNsS6V85zyykBm3nduv3l6juWkUxqj4EeQOgOo1bSm?=
 =?us-ascii?Q?/J+RQGOU4w1WHbMPf35mHoR0qyhpBO9GX3hp8HiBOgqemoRaPvt/rVzMbTbD?=
 =?us-ascii?Q?YORO6tvJrf4cYCangWgIWiBWke8U3f9DwHbP4X9oaIqocrKNhmDZ3ehZZXac?=
 =?us-ascii?Q?zKyy4hXgd5aapGAE+WkvqgkVDh3JGmhNUswn1ja8sOT3CUAhSWJgqaaerMg+?=
 =?us-ascii?Q?Nghq/voG+MVCyU/bYbrYrHIB3dBJBx/8Zqe84eQXos7lGlC9H1E5VEYD+q0L?=
 =?us-ascii?Q?2VaVcIei953J1F5VqcnM4WzB/TlzF+zAnKaelXQY9q+URuvwhW20k2OeUH7z?=
 =?us-ascii?Q?p+Mwin3+4/pTJ7bvPiYhJXQqzvz/ZCVpwZW0URDBtPa1vCnFV9p9R1nKrN8y?=
 =?us-ascii?Q?jTP/S3YXHJU4P7qA1PE=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fc8ccbdb-d4c7-4eac-a345-08d9bb56aae2
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Dec 2021 20:58:35.9915
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: E4Qw5mKrSRJfZ7zj0ngG3x5ixYTjty9weeD7GOeS3Fv97ES6XIVOAzwzgZ8k8Thn
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5143
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Dec 09, 2021 at 09:32:42PM +0100, Thomas Gleixner wrote:
> On Thu, Dec 09 2021 at 12:21, Jason Gunthorpe wrote:
> > On Thu, Dec 09, 2021 at 09:37:06AM +0100, Thomas Gleixner wrote:
> > If we keep the MSI emulation in the hypervisor then MSI != IMS.  The
> > MSI code needs to write a addr/data pair compatible with the emulation
> > and the IMS code needs to write an addr/data pair from the
> > hypercall. Seems like this scenario is best avoided!
> >
> > From this perspective I haven't connected how virtual interrupt
> > remapping helps in the guest? Is this a way to provide the hypercall
> > I'm imagining above?
> 
> That was my thought to avoid having different mechanisms.
> 
> The address/data pair is computed in two places:
> 
>   1) Activation of an interrupt
>   2) Affinity setting on an interrupt
> 
> Both configure the IRTE when interrupt remapping is in place.
> 
> In both cases a vector is allocated in the vector domain and based on
> the resulting target APIC / vector number pair the IRTE is
> (re)configured.
> 
> So putting the hypercall into the vIRTE update is the obvious
> place. Both activation and affinity setting can fail and propagate an
> error code down to the originating caller.
> 
> Hmm?

Okay, I think I get it. Would be nice to have someone from intel
familiar with the vIOMMU protocols and qemu code remark what the
hypervisor side can look like.

There is a bit more work here, we'd have to change VFIO to somehow
entirely disconnect the kernel IRQ logic from the MSI table and
directly pass control of it to the guest after the hypervisor IOMMU IR
secures it. ie directly mmap the msi-x table into the guest

Jason