Date:   Sun, 12 Dec 2021 19:42:06 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        Logan Gunthorpe <logang@deltatee.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Marc Zygnier <maz@kernel.org>,
        Alex Williamson <alex.williamson@redhat.com>,
        "Dey, Megha" <megha.dey@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jon Mason <jdmason@kudzu.us>, Allen Hubbe <allenbh@gmail.com>,
        "linux-ntb@googlegroups.com" <linux-ntb@googlegroups.com>,
        "linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
        Heiko Carstens <hca@linux.ibm.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>,
        "x86@kernel.org" <x86@kernel.org>, Joerg Roedel <jroedel@suse.de>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>
Subject: Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Message-ID: <20211212234206.GO6385@nvidia.com>
References: <87sfv2yy19.ffs@tglx>
 <20211209162129.GS6385@nvidia.com>
 <878rwtzfh1.ffs@tglx>
 <20211209205835.GZ6385@nvidia.com>
 <8735n1zaz3.ffs@tglx>
 <87sfv1xq3b.ffs@tglx>
 <BN9PR11MB527619B099061B3814EB40408C719@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20211210123938.GF6385@nvidia.com>
 <BN9PR11MB5276B2584F928B4BFD4573428C729@BN9PR11MB5276.namprd11.prod.outlook.com>
 <87lf0qvfze.ffs@tglx>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87lf0qvfze.ffs@tglx>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?7qs/JYJvTCIiPEhbM7KQfQcilivWDeDAZAyL9wFnOG9am2T8pfjam5qw8PNv?=
 =?us-ascii?Q?3uAiwKPOk4Zo3Nzsx1YuRyDzrI6HcakpBZua8TQd2/OsrCfanKSwMl39JMHv?=
 =?us-ascii?Q?CUKzzJf+Qhc6vuQftzPNJUsh5lbaRy1uMn1wjCV64ItDp/BoswHQ+DX5fkbX?=
 =?us-ascii?Q?OP6WeQS71SH8AOLCHL+vbxEVh0/Us2PW6pbWZHt+WGNaHggFcFfNnN+z8RC9?=
 =?us-ascii?Q?hRBHp4DyK4gmMbJtCXgXDWdu7I0oW3DlN1JTymEaAVaDAfnLmn36gJ/lzRwO?=
 =?us-ascii?Q?Pv78ras//8usBfs6sH+Z6Cnn/F/OSp6tM5StajRoBV5T0INFzTrD5MJvv0cF?=
 =?us-ascii?Q?AvtaqOHTjmTK0ktzFTqnN61YB7sOJJo29jBW9AIpbWmf3fjDYe3EMIqiuGUs?=
 =?us-ascii?Q?IotpcI6LEF3BgqQLRXVQvhGia2rSUO9bQ6QLk2wtJt7DkhUBu5wPCL9T3f7c?=
 =?us-ascii?Q?VVLSLIhGfOyMGM0AWDYgI60NpnK5D4KIJaTtnp+3UQklMJ204QNEVB7UB9bu?=
 =?us-ascii?Q?wy3rIwn6w5RuOCN3k5/D6dzJtLA6AMkoLQa94viZwokJdQkB4kZOVXxSmG6b?=
 =?us-ascii?Q?Di/0U1AZvfR/RJBrYnsfYjje/a4Kol7BKJHTDFkXNv7awxEkasZD6NPxhpUA?=
 =?us-ascii?Q?+I5Vq/hVpygKBz9j6jnu61GFx5v9hkcDi5dPNr15UzZpEcdyYFG049naQubK?=
 =?us-ascii?Q?BR+k1AWOaxRBPLNyiiPXgZO1fIqcWY+IIUITkvTVrzPP71fVWpFZgOk8Vgxj?=
 =?us-ascii?Q?rUX79hmlE4BU3bINx2ACs1kMcjkNIaYl+/C80/+Lg41OVWyhCEWDv1wBesGt?=
 =?us-ascii?Q?h+jVyYb9NtK/bzVdnJfrQW0mi2ByRxe6DdhSCg1MFjp0HCnIoaNq5izuZy+1?=
 =?us-ascii?Q?t2q5ufjsEukwlo1Cnh94P0Trkhvq/WRD+6n2f/URDlgxzdOfWPUfwvMLv8UI?=
 =?us-ascii?Q?s07hz0PDc6/2+3HKbv5W2sos62d9bpjL96bwRwgU6VPF4DiuFdx7iAG2N2CL?=
 =?us-ascii?Q?TcoYP4L1C1IzRt/4raJlTdWcGC3MKxOIvJpVQGDnPrsEUA39tQOeV19T1YGQ?=
 =?us-ascii?Q?Yf8mM7qbFVibJttmQRsM4Qiv7VZ7fxE0o1IdAoT7y8MmwDKZoEmmHt2uHOUg?=
 =?us-ascii?Q?qP2jwmSXcmVvF1hn0XMCq5BKBA86BhDUYkhWFfjyktoEkaWQ8GwhN0BO0Uxr?=
 =?us-ascii?Q?J+VcHYdGvKTKOtKQELRdijjfBq8cjFGQACvgAlQccnkkEP+6DueiQ9qBb80G?=
 =?us-ascii?Q?l6nUXb0LVHZWtMtxUIYCq1GYhJA2531BMdvDQHAy151VmaSSCwtz4ZjOfa5w?=
 =?us-ascii?Q?9NEcZ7nz3P+U9+dVzCf0v/jwmJ1bHAb8SgRgsafgRKjTU/Y0yk0sXvrx1Nas?=
 =?us-ascii?Q?FGZNFOav/WnAWgv49yVp+cb5u+i5fKeco3pb8mTu4/TsGDKwX4XqtueyydJC?=
 =?us-ascii?Q?5b+tS0BikpjzHhz8iiIZs4lBBQFssf44mC3rfjbxPN7fIMirTfop17T3aF8M?=
 =?us-ascii?Q?C4kIxe+w7/sk69jvll5DwyD5W8gfKZzTuow3A1XkYd8MdRpTzCuZfoQZdcU6?=
 =?us-ascii?Q?vCmREsxMCGff4zGbsLY=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 75bcb45a-5a9d-43ae-14d1-08d9bdc91dcc
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2021 23:42:53.7032
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GqeEBk9m/aMKOxlbaNkfd5Jax0mGlr2RTRFiSnOPQNnzc/ILQoUU3jgAKHGXBH2M
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5189
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Sun, Dec 12, 2021 at 01:12:05AM +0100, Thomas Gleixner wrote:

>      PCI/MSI and PCI/MSI-X are just implementations of IMS
> 
> Not more, not less. The fact that they have very strict rules about the
> storage space and the fact that they are mutually exclusive does not
> change that at all.

And the mess we have is that virtualiation broke this
design. Virtualization made MSI/MSI-X special!

I am wondering if we just need to bite the bullet and force the
introduction of a new ACPI flag for the APIC that says one of:
- message addr/data pairs work correctly (baremetal)
- creating message addr/data pairs need to use a hypercall protocol
- property not defined so assume only MSI/MSI-X/etc work.

Intel was originally trying to do this with the 'IMS enabled' PCI
Capability block, but a per PCI device capability is in the wrong
layer.

Jason