From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Thomas Gleixner <tglx@linutronix.de>,
        Jason Gunthorpe <jgg@nvidia.com>
CC:     Logan Gunthorpe <logang@deltatee.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>,
        "Marc Zygnier" <maz@kernel.org>,
        Alex Williamson <alex.williamson@redhat.com>,
        "Dey, Megha" <megha.dey@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jon Mason <jdmason@kudzu.us>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        Allen Hubbe <allenbh@gmail.com>,
        "linux-ntb@googlegroups.com" <linux-ntb@googlegroups.com>,
        "linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
        Heiko Carstens <hca@linux.ibm.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>,
        "x86@kernel.org" <x86@kernel.org>, Joerg Roedel <jroedel@suse.de>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>
Subject: RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Thread-Topic: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Thread-Index: AQHX4y1VNaiO68omHEet2J9DWrVARKwa1YEAgAAp0QCAABreAIAAIi2AgAE8MoCAAAd1gIAACyoAgAAPcQCAADDDAIAAp1uAgAAttoCAAEzjgIAACsMAgAAjl4CAAD2IAIAA6NQAgABcawCAAAmigIAAKimAgAAjYoCAAPMeAIAAGgMAgAis5HCAAEEdAIAANO7A
Date:   Thu, 9 Dec 2021 12:17:36 +0000
Message-ID: <BN9PR11MB5276BDE2AC58ADA7A66CAFBA8C709@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <87y2548byw.ffs@tglx> <20211201181406.GM4670@nvidia.com>
 <87mtlk84ae.ffs@tglx> <87r1av7u3d.ffs@tglx>
 <20211202135502.GP4670@nvidia.com> <87wnkm6c77.ffs@tglx>
 <20211202200017.GS4670@nvidia.com> <87o85y63m8.ffs@tglx>
 <20211203003749.GT4670@nvidia.com> <877dcl681d.ffs@tglx>
 <20211203164104.GX4670@nvidia.com>
 <BN9PR11MB52766CC46D3D4865308F61D98C709@BN9PR11MB5276.namprd11.prod.outlook.com>
 <87pmq6ywu3.ffs@tglx>
In-Reply-To: <87pmq6ywu3.ffs@tglx>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?uLF1CwV6V8M5HScw9bmsKMT1lg1nITA2RB0ia5+ikTAo1etS6FbmWaG5l6k8?=
 =?us-ascii?Q?77OXelpxNVi08nHDCT45RPsCxJ+1uD5fBHWVOWVMVkP1DyU/HwCwtg8V6Txg?=
 =?us-ascii?Q?kAZrcDQajTzOr4frDrk0uhmowFrLPHhSTYgl5Oct4oO3dUO9yn6BRL/OaOJ1?=
 =?us-ascii?Q?VysYLPqqanQyQYJo+05wim+yNqzLw8muq/FXwvS3pQy3POSVODdEWfoUWNtC?=
 =?us-ascii?Q?C685opss8fUzgU7PPHT21H7Qxn/G+1ly1Dy+HxLKR1sg+ZOm01aRcLLa6nMM?=
 =?us-ascii?Q?YYjfTqpzXSgFM2BXw15RbB6Xw3O8Uzu+FC1WTMU0ofbBSLWfBSp1PZgrZoQa?=
 =?us-ascii?Q?fPj2HCtji9/VfA0IucMawpZ9j8rE/pIPf1P9SBJ1lhKNkiJcfXKII3P06c1E?=
 =?us-ascii?Q?p2gF9e19823HvZA2IdqW/3QMj4n/iTs2IWyB3irjvYKxKY6Zk+OZ0Dtbzd8q?=
 =?us-ascii?Q?ZgHH7JVBG4XE1xN7vsD1+uqpt650A5ob/Lb4/aeyLjP+Dxhl4DGuNFoGsueB?=
 =?us-ascii?Q?S76+3hDWu4nQ+yfqB8Midx7yW4zDExK1vFlEOe7XYzyKUJCYB3/eFB9gajWZ?=
 =?us-ascii?Q?9QkeL48UThJPvVUxwdMPh3vYPoOkMqXhRQ8PhgfNkki23FMgo0F5UTinADbq?=
 =?us-ascii?Q?48BlUccA9jr8Hhap+3GAu6/V23YBMwQupKYcHiY8tLOz7np7qMM4Fyzv20lM?=
 =?us-ascii?Q?X4xJJgANBvUFeHlewFH8l0Sa3uc8LqLwt1t4YTty/h4LZWKN4D8Ohn5MI6Ur?=
 =?us-ascii?Q?NjUTdNwmVlwp7W5SdsT3pqGTxQ2+EyTuGxVr1OnL8qSV8pW+t8T4QNHng4Ta?=
 =?us-ascii?Q?bfjmEMaD0RXo4V20j088aQYS+T+yAQol9BeO2rUCMaTq4U45Yrw704B3BWQY?=
 =?us-ascii?Q?Ywc00vVQIT1Qqr2YNWn1QCtjl+Jk6IPEcHpQee4y8/2z2S/nDbGjD2VK/GsV?=
 =?us-ascii?Q?fz5m8NQOvNe6FK5X1PwGo1TIwztIt7ia5ePBJsvE1KFMd6t5WkW208exjR9K?=
 =?us-ascii?Q?Mf1R7fAiBt++9I+LVKLDwuwEFpjB5kyZTYwJ/YNx9kWOnuNofuBIi7tH4aLN?=
 =?us-ascii?Q?SaVVoEiFGE5HkMgtnZ/snIRlEam/yDoVMwfVTkzudCakX5EXzjny2mhwkwvs?=
 =?us-ascii?Q?uFFy8QTMe4dc7qSJOz+kQQeeXHv8qGX6S88SaYqiFq+SOkkkh38plbvkguCX?=
 =?us-ascii?Q?B3IWnlD/UClpyiine/bgL1ljqkxOAIABN9kAxxUu9tmZs9hlZbx4ZYrNtQ/c?=
 =?us-ascii?Q?SQzKzdf3hJgOSKWsf2SiR/rplYodc6qKgOHLMxdNvugxhDNaQDZGn4MX1I28?=
 =?us-ascii?Q?xm6NKGEm7Ny+3qcCQmV7+602UV/Z+3Yg5gD3WODhbMbXbwPHsfLmi3k+WRxd?=
 =?us-ascii?Q?PVBng1wIeRjvdoG2I3vV3FF8bG6Yvh744Y51IL2YoIZCVgoeQlxXByl6r7li?=
 =?us-ascii?Q?g55FccTqIqWBRT6eqTmtXQ2J4A0ptzR1NNKkJPNSafd6fDApUik+ZLL1swy+?=
 =?us-ascii?Q?Tt+1ncuO4JRzj6oJx6MGTnuu4oaUGighax2W71JXkGx3VlYfWz2rUD1EyKMc?=
 =?us-ascii?Q?wuHjLtrYxzM5MhMszUelDaCre+/jRUI+6nfuFs/OlwLDJuZCooz+BPWcv7W5?=
 =?us-ascii?Q?VoPUMB6pE1JvlW2CYVKozbE=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7f722c3d-9a20-4273-6a50-08d9bb0de2de
X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Dec 2021 12:17:36.3627
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: +Qsrb30MAVYx6VhBpnpcL+psk7W7Qo3E/JCMLq2xouGr+xlSZiwHKz6g6pZ2ftY+6mjsp/jkmPGNY+Y4+FKa4A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR11MB3714
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Thomas Gleixner <tglx@linutronix.de>
> Sent: Thursday, December 9, 2021 5:03 PM
>=20
> Kevin,
>=20
> On Thu, Dec 09 2021 at 06:26, Kevin Tian wrote:
> >> From: Jason Gunthorpe <jgg@nvidia.com>
> >> I don't know of any use case for more than one interrupt on a queue,
> >> and if it did come up I'd probably approach it by making the queue
> >> handle above also specify the 'queue relative HW index'
> >
> > We have such use case with IDXD.
> >
> > Basically the IDXD queue allows software to put an interrupt handle
> > (the index of MSI-X or IMS entry) in the submitted descriptor. Upon
> > completion of the descriptor the hardware finds the specified entry
> > and then generate interrupt to notify software.
> >
> > Conceptually descriptors submitted to a same queue can use different
> > handles, implying one queue can be associated to multiple interrupts.
>=20
> I think you are looking at that from the internal implementation details
> of IDXD. But you can just model it in an IDXD implementation agnostic
> way:
>=20
>      ENQCMD(PASID, IMS-ENTRY,.....)

Not exactly IMS-ENTRY. MSI-ENTRY also works.

>=20
> implies an on demand allocation of a virtual queue, which is deallocated
> when the command completes. The PASID and IMS-ENTRY act as the 'queue'
> identifier.
>=20
> The implementation detail of IDXD that it executes these computations on
> an internal shared workqueue does not change that.
>=20
> Such a workqueue can only execute one enqueued command at a time,
> which
> means that during the execution of a particular command that IDXD
> internal workqueue represents the 'virtual queue' which is identified by
> the unique PASID/IMS-ENTRY pair.
>=20
> No?
>=20

While it's one way of looking at this model do we want to actually
create some objects to represent this 'virtual queue' concept? that
implies each ENQCMD must be moderated to create such short-lifespan
objects and I'm not sure the benefit of doing so.

If not then from driver p.o.v it's still one queue resource and driver=20
needs to manage its association with multiple interrupt entries and=20
PASIDs when it's connected to multiple clients.

Thanks
Kevin