From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Thomas Gleixner <tglx@linutronix.de>,
        Jason Gunthorpe <jgg@nvidia.com>
CC:     "Jiang, Dave" <dave.jiang@intel.com>,
        Logan Gunthorpe <logang@deltatee.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Marc Zygnier <maz@kernel.org>,
        Alex Williamson <alex.williamson@redhat.com>,
        "Dey, Megha" <megha.dey@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jon Mason <jdmason@kudzu.us>, Allen Hubbe <allenbh@gmail.com>,
        "linux-ntb@googlegroups.com" <linux-ntb@googlegroups.com>,
        "linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
        Heiko Carstens <hca@linux.ibm.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>,
        "x86@kernel.org" <x86@kernel.org>, "Rodel, Jorg" <jroedel@suse.de>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>
Subject: RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Thread-Topic: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Thread-Index: AQHX4y1VNaiO68omHEet2J9DWrVARKwa1YEAgAAp0QCAABreAIAAIi2AgAE8MoCAAAd1gIAACyoAgAAPcQCAADDDAIAAp1uAgABn9QCAACTxgIAAAdGAgAAbd4CAAA94gIAABpSAgAt9SoCAADlJAIAAgb+AgABGMACAAAc8gIAAE+oAgAAmHACAAFJ4oIAAem8AgAE/1RCAARP4gIAAHS7w
Date:   Sun, 12 Dec 2021 02:14:10 +0000
Message-ID: <BL1PR11MB5271BFC6B2218CF7E9151EE88C739@BL1PR11MB5271.namprd11.prod.outlook.com>
References: <f4cc305b-a329-6d27-9fca-b74ebc9fa0c1@intel.com>
 <878rx480fk.ffs@tglx>
 <BN9PR11MB52765F2EF8420C60FD5945D18C709@BN9PR11MB5276.namprd11.prod.outlook.com>
 <87sfv2yy19.ffs@tglx> <20211209162129.GS6385@nvidia.com>
 <878rwtzfh1.ffs@tglx> <20211209205835.GZ6385@nvidia.com>
 <8735n1zaz3.ffs@tglx> <87sfv1xq3b.ffs@tglx>
 <BN9PR11MB527619B099061B3814EB40408C719@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20211210123938.GF6385@nvidia.com>
 <BN9PR11MB5276B2584F928B4BFD4573428C729@BN9PR11MB5276.namprd11.prod.outlook.com>
 <87lf0qvfze.ffs@tglx>
In-Reply-To: <87lf0qvfze.ffs@tglx>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Cv23W3v7Eu2Hl87hNWWiml7cNgydQKZoL/iuVXPBI0QowyCV2/e2wyBHL4QJ?=
 =?us-ascii?Q?x+IEcu9fn4vY12Y4+zgokgZN9n42S4uAKhQgsZCWVxydvSyhZwLGdbL3OV8j?=
 =?us-ascii?Q?u7BbFN349eWeO+WifAknxTHr8lK17vlNEEQCinEdy2MG0qf5a0A7jA2+Db4Y?=
 =?us-ascii?Q?Zjp07PZsqoRSo+7zsNwwzajsDdknzs/quMnJp9NyiIrROTjuiDyg9PKCggmF?=
 =?us-ascii?Q?1cm2n+vbo65If6e2VPtWZR7MUay1F79UEZb0hasjeI4FAgs8UKZkl8obZSV9?=
 =?us-ascii?Q?2R0lDqpXKCGqvVQhm+hn3DN72XoV0Fe2fEWgziEczi6Nm7ncujyWA1A/o2az?=
 =?us-ascii?Q?VjEUWwLIxLW0WEFA9EanpZX1f5TT3d+xkNkOJa60JYTb+liRkyhLB40CQmfs?=
 =?us-ascii?Q?1VyvpLHwoMDahKiEII4MSz+7lfM8aLeLK26Xkg1npm1WofKwpG2EicbhkqYp?=
 =?us-ascii?Q?dfb/8hUgLqGZHUr+c25nEFUj86s5ow5e5ywzlCh0dFh8gjoZeCqRik93tDuV?=
 =?us-ascii?Q?nFk5kdmf+A5Tkvkd+lHTLtIsfhrfpBS5T4NdSrhHEDP6cWzHYdQ/6X9KoG+U?=
 =?us-ascii?Q?uRKMAxDnD0EM5PWKudtVYH6H0YRmXXUu/ENiVQmEMVVYXyQH2qpyMJRxrCEr?=
 =?us-ascii?Q?HJo4+/YZd949iD1ipoGH2R5KSpDL41k1ZBxNTio2o/js/dGoAClBLLQDB2NL?=
 =?us-ascii?Q?+r3F4tyP2t57oMQMF/32QQ7bC5+Ik9NRTf+x2zoJ6BuFpo5BL7wfkZMEKi4d?=
 =?us-ascii?Q?VS9e/vcbCG24GMEuV5nJdI+yiJIE9aR2bCetwBBR44sbV/MFn3QulyEimXt2?=
 =?us-ascii?Q?khHTSZgN38dfivYcNYMgqkWuY5kt3QJsmxz+vhz6qLRy+qPvHoupVbX9QWWn?=
 =?us-ascii?Q?1R9J1pXOwKPfpDI0JVV9TaomreVqkLZ5/hJwnT4n6G6FwX9rmsQE+IXWVHZJ?=
 =?us-ascii?Q?swn/EUPIt/GCYH0qKr6m9pBuj77LsFTHox2mwkpxJZMTjrJAbR/aKNYNEYIL?=
 =?us-ascii?Q?C86SEERFU7ls5TAeueeIv8SiTD93MObf3GlWKxa0jxbKTfxapjNNSlfNSXkB?=
 =?us-ascii?Q?3QSAlYrcJjEjKiUuAQDORNSbKte1nOFE2sIrHDii0AFZPiK3XPYErN2a0gJK?=
 =?us-ascii?Q?xsK2vLfqGfJGRdftl4A8vhYgFXjWzYEJ6gpJ0Qtuehi+Vz+OYQZB+KuWEiMJ?=
 =?us-ascii?Q?E13DnQmD5EH7TpXsjC9BnlBBKLBT1c1QFJehWfEybXbKMOyX2u87miEQFiat?=
 =?us-ascii?Q?WDweu9vZNdcyv6g3fxcAmBlIaqOUbEiP1+2i+w/6taKFoa32a+XfSDLHkCPF?=
 =?us-ascii?Q?SGEMIEkHae6XSpjS/RV8C2QZY/QEEpj55iMypqPaswgpOK6KxlGJmbSsN1y4?=
 =?us-ascii?Q?9lodfdtEbSWGBfyOIZMPbwayswRXo7M1BocD7nzk5N9wE9dlfMCJd7XuRB8P?=
 =?us-ascii?Q?JmJPbBmQnaD1Nz7uzPJzif7nVIuy6+OlQ3i6jeB0JOve5pBCDCLOm7yM660k?=
 =?us-ascii?Q?aDjVy0HL0p+19OthRLjzI94PSyaGU8TbPnUXVB1XEGeeeRo3b0uzUMCaddjB?=
 =?us-ascii?Q?UUUSDnxLwVUPmOyL96tZ0+AJNTt05BaiPQ42AIYvLCaWFCjCRPfa3iqFiNHI?=
 =?us-ascii?Q?Lqp2LZN3LlPwYa1f6tMmk4E=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5271.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 36ed7bef-3646-4702-9e5b-08d9bd151581
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Dec 2021 02:14:10.1166
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: YvZ5GNjNY6Lh7Zu9nQOj7GDimjUi0rQZw/rP6PLzKUJPmQScM/IArpyhAxVxCHxccTwcvtQMnRqqNcPW8rsvHA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB4016
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi, Thomas,

> From: Thomas Gleixner <tglx@linutronix.de>
> Sent: Sunday, December 12, 2021 8:12 AM
>=20
> Kevin,
>=20
> On Sat, Dec 11 2021 at 07:52, Kevin Tian wrote:
> >> From: Jason Gunthorpe <jgg@nvidia.com>
> >> > Then Qemu needs to find out the GSI number for the vIRTE handle.
> >> > Again Qemu doesn't have such information since it doesn't know
> >> > which MSI[-X] entry points to this handle due to no trap.
> >>
> >> No this is already going wrong. qemu *cannot* know the MSI information
> >> because there is no MSI information for IMS.
> >
> > I haven't thought of IMS at this step. The IR approach applies to
> > all types of interrupt storages, thus I'm more interested in how it
> > affect the storages which are already virtualized today (MSI[-X]
> > in my thought practice).
>=20
> They are not any different. As I explained several times now IMS is
> nothing new at all. It existed since the invention of Message Signaled
> interrupts. Why?
>=20
> The principle behind Message Signaled Interrupts is:
>=20
>     Device writes DATA to ADDRESS which raises an interrupt in a CPU
>=20
> Message Signaled Interrupts obviously need some place to store the
> ADDRESS/DATA pair so that the device can use it for raising an
> interrupt, i.e. an
>=20
>    Interrupt Message Store, short IMS.
>=20
> PCI/MSI was the first implementation of this and the storage was defined
> to be at a specified and therefore uniform and device independent place.
>=20
> PCI/MSI-X followed the same approch. While it solved quite some of the
> shortcomings of PCI/MSI it still has a specificed and uniform and device
> independent place to store the message (ADDRESS/DATA pair)
>=20
> Now the PCI wizards figured out that PCI/MSI[-X] is not longer up to the
> task for various reasons and came up with the revolutionary new concept
> of IMS, aka Interrupt Message Store. where the device defines where the
> message is stored.
>=20
> IOW, this is coming back full circle to the original problem of where to
> store the message, i.e. the ADDRESS/DATA pair so that the device can
> raise an interrupt in a CPU, which requires - drum roll - an
>=20
>    Interrupt Message Store, short IMS.
>=20
> So you simply have to look at it from a pure MSI (not PCI/MSI) point
> of view:
>=20
>    MSI at the conceptual level requires storage for the ADDRESS/DATA
>    pair at some place so that the device or the compute unit embedded in
>    the device can write DATA to ADDRESS.
>=20
> That's it. Not more, not less.
>=20
> When you look at it from this perspective, then you'll realize that
>=20
>      PCI/MSI and PCI/MSI-X are just implementations of IMS
>=20
> Not more, not less. The fact that they have very strict rules about the
> storage space and the fact that they are mutually exclusive does not
> change that at all.
>=20
> That's where a lot of the confusion comes from. If you go back to all
> the IDXD/IMS discussions which happened over time then you'll figure out
> that _all_ of us where coming from the same wrong assumption:
>=20
>     IMS is new and it's just another exclusive variant of PCI/MSI and
>     PCi/MSI-X.
>=20
> It took _all_ of us quite some time to realize that we need to look at
> it from the other way around.
>=20
> There was surely some other conceptual confusion vs. subdevices, queues
> and whatever involved which contributed to that. Water under the bridge.
>=20
> Coming back to your initial question:
>=20
> > I haven't thought of IMS at this step. The IR approach applies to
> > all types of interrupt storages, thus I'm more interested in how it
> > affect the storages which are already virtualized today (MSI[-X]
> > in my thought practice).
>=20
> Stop focussing on implementation details. Focus on the general concept
> instead. See above.
>=20

I have no any problem on understanding above relational. This has
been acked multiple times in my previous replies.

I just continue the thought practice along that direction to see what
the host flow will be like (step by step). Looking at the current=20
implementation is just one necessary step in my thought practice to=20
help refine the picture. When I found something which may be=20
worth being aligned then I shared to avoid follow a wrong direction=20
too far.

If both of your think it simply adds noise to this discussion, I can
surely hold back and focus on 'concept' only.

Thanks
Kevin