Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Alex Williamson <alex.williamson@redhat.com>
CC:     "Chatre, Reinette" <reinette.chatre@intel.com>,
        "jgg@nvidia.com" <jgg@nvidia.com>,
        "yishaih@nvidia.com" <yishaih@nvidia.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Liu, Jing2" <jing2.liu@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "tom.zanussi@linux.intel.com" <tom.zanussi@linux.intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "patches@lists.linux.dev" <patches@lists.linux.dev>
Subject: RE: [RFC PATCH V3 00/26] vfio/pci: Back guest interrupts from
 Interrupt Message Store (IMS)
Thread-Topic: [RFC PATCH V3 00/26] vfio/pci: Back guest interrupts from
 Interrupt Message Store (IMS)
Thread-Index: AQHaCPdEtaV8qeHO6EmMpiKLQ58cQ7BjhMFwgAJFzACAAIkGYIAADu6QgAEuhgCAAKTv8A==
Date:   Fri, 3 Nov 2023 07:23:13 +0000
Message-ID: <BN9PR11MB5276BCEA3275EC7203E06FDA8CA5A@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <cover.1698422237.git.reinette.chatre@intel.com>
        <BL1PR11MB52710EAB683507AD7FAD6A5B8CA0A@BL1PR11MB5271.namprd11.prod.outlook.com>
        <20231101120714.7763ed35.alex.williamson@redhat.com>
        <BN9PR11MB52769292F138F69D8717BE8D8CA6A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20231102151352.1731de78.alex.williamson@redhat.com>
In-Reply-To: <20231102151352.1731de78.alex.williamson@redhat.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?zuoMe5GrDshVgEpzRT5xmMEns9Jupd/7WX4/HtDPgiBtFw7kP9D974IZ1t1x?=
 =?us-ascii?Q?YaAG7IqdJoyokMkrKrVCSwkjdoEXcUS/B+MP78cMRA49Kei5Uh5/9Ynq3lCc?=
 =?us-ascii?Q?d3/cRIm3CD1lXzNer55c7wl6BMarEqM6/x/I8ZRD2UeGKT6DnZOebh6GXaqR?=
 =?us-ascii?Q?ewOOCO63shdK+AWJoQ4X5faUjJncO2RTY3gDlFrNOZm1VmRwIQAWVx8rTR0o?=
 =?us-ascii?Q?WISTT78Db/n0X+pJht0a6jascsSBSl7b/N354xbpRJfI/VJrRGykw3HP/oq/?=
 =?us-ascii?Q?lwsHtlpj6hdSaufp0Et+d8MEJD970Z5cWQPreLh8xod+AituWIzId8DUxXj5?=
 =?us-ascii?Q?ln0UMjrqyio99Y00VhoT3SRHKh2YqUdPNZx9SMZlFl//KZS44m96sasqiaYp?=
 =?us-ascii?Q?Ynmic1gJjd9jqi8ZQRvHXudlUEXKQZ+BEC2ekOXxcJh17Db2Ku8CIX2L2eKy?=
 =?us-ascii?Q?gWEw6AgGuOycDtcYs41a1zl2sw5iAWXjxK/oclAGxta/JlIroV6MdlDjtYEF?=
 =?us-ascii?Q?zzgqcc+OFSVkkvNikPrCYzcYgPP8DWKLkSzSlkDYniMVu+mTL0ELr4X9tDI6?=
 =?us-ascii?Q?Jtt+pKq1HkhDhAyM7sGG0p/3JHRSvc9gptl5AT0kMsarx97aMHoKU2X9Z5L1?=
 =?us-ascii?Q?0mDZF3HjIvHBQFlwrM8hs20HbtdxJwHzOs6/TWu5f7im98Zujd+/h27RY/Ft?=
 =?us-ascii?Q?wOOXfL8sqvGqpagmrHJ0fJHV2UISAs+EZZZ+AyF5cMft7GBSaiq1k8+uKpD7?=
 =?us-ascii?Q?h8jZ/j/fp7lefXxn1e7oLAV0BwZcCuNjzhAzQhZXRBtGrN3OplPLYfCXcv++?=
 =?us-ascii?Q?h9kVQdm6yLWpfQRJ6O4/FqKS55QTwIqAq7zjrvUOainnTEIz1hhdqQySsit0?=
 =?us-ascii?Q?PWN/DYckW2DaSh+gqgQxbfM5tBEjtLkenBKVd7XkHcdmuR6pxkCWaoSB8fUo?=
 =?us-ascii?Q?fA3VmgSSf4H5RWo2ilhqBAyHkQZpvku8B2bR5eQ0BlTiFj1qDVzIa0P/N2OB?=
 =?us-ascii?Q?0JkPKq5s0FvMgRu9coJfBKbROw6ZDUhenMLPzkIjS9RxmvgmV2qCAdhwthfm?=
 =?us-ascii?Q?IlKzJ+jXAGpOD91oqLaOQPMRtCG5aVAivA7z/lcMr182ki6mHL6w3Qz0EXwC?=
 =?us-ascii?Q?c/0y3sXk5+Nkg++7aVyA+JNaURFu89dF2sAO22MqByfQIG/v+RLV43jC3aXR?=
 =?us-ascii?Q?6seDFe0d6F7j4m/J/uCOe9Iy69uJWu0TMujWUzGC5Pg5a1110NDwxt4Qtytn?=
 =?us-ascii?Q?iOjqEBvHW1+fJGUkGDORvFKVqEHAPs1KLoNAOjWhbdd8BXoq5XLc3OVUXJp4?=
 =?us-ascii?Q?JBkVaX8+j5AHlsvxDbDHHLL7cxgBJmteu7OkLoT7JyhHdMx5+md/5DUmMLR/?=
 =?us-ascii?Q?eHUGaDZ+3MXu5i7wNZvz1u7gMxmHk+qcORtQ3osbhniYQzy5XM8IAN6/UYgj?=
 =?us-ascii?Q?Lb28kSN+ElAJzXtofREH9goIZW4QfOIlGxzesI6HR4ofoSIKUCbulrS4bJtP?=
 =?us-ascii?Q?s/RdmBdB860EaCDee15yXCsiw0oBxokwQUny8IMDtSOZ63Zngr7YXZ0ayjsa?=
 =?us-ascii?Q?xpRQ9NqOc9UH3GKmQXaaEyKWCZ6zzEnoapKBL+Nj?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9326f7f1-7296-4cef-8f2a-08dbdc3dbd79
X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Nov 2023 07:23:13.2423
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: dmOlffm5o4j59bUZR4YCAsm3o9N86mIDTUCmKgLkK6C7bW+UL5UCYMYXoT94u46Xbz5ltwvD8Uin3SvT2DWX+g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB8704
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 03 Nov 2023 00:23:25 -0700 (PDT)

> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Friday, November 3, 2023 5:14 AM
>=20
> On Thu, 2 Nov 2023 03:14:09 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
>=20
> > > From: Tian, Kevin
> > > Sent: Thursday, November 2, 2023 10:52 AM
> > >
> > > >
> > > > Without an in-tree user of this code, we're just chopping up code f=
or
> > > > no real purpose.  There's no reason that a variant driver requiring=
 IMS
> > > > couldn't initially implement their own SET_IRQS ioctl.  Doing that
> > >
> > > this is an interesting idea. We haven't seen a real usage which wants
> > > such MSI emulation on IMS for variant drivers. but if the code is
> > > simple enough to demonstrate the 1st user of IMS it might not be
> > > a bad choice. There are additional trap-emulation required in the
> > > device MMIO bar (mostly copying MSI permission entry which contains
> > > PASID info to the corresponding IMS entry). At a glance that area
> > > is 4k-aligned so should be doable.
> > >
> >
> > misread the spec. the MSI-X permission table which provides
> > auxiliary data to MSI-X table is not 4k-aligned. It sits in the 1st
> > 4k page together with many other registers. emulation of them
> > could be simple with a native read/write handler but not sure
> > whether any of them may sit in a hot path to affect perf due to
> > trap...
>=20
> I'm not sure if you're referring to a specific device spec or the PCI
> spec, but the PCI spec has long included an implementation note
> suggesting alignment of the MSI-X vector table and pba and separation
> from CSRs, and I see this is now even more strongly worded in the 6.0
> spec.
>=20
> Note though that for QEMU, these are emulated in the VMM and not
> written through to the device.  The result of writes to the vector
> table in the VMM are translated to vector use/unuse operations, which
> we see at the kernel level through SET_IRQS ioctl calls.  Are you
> expecting to get PASID information written by the guest through the
> emulated vector table?  That would entail something more than a simple
> IMS backend to MSI-X frontend.  Thanks,
>=20

I was referring to IDXD device spec. Basically it allows a process to
submit a descriptor which contains a completion interrupt handle.
The handle is the index of a MSI-X entry or IMS entry allocated by
the idxd driver. To mark the association between application and
related handles the driver records the PASID of the application
in an auxiliary structure for MSI-X (called MSI-X permission table)
or directly in the IMS entry. This additional info includes whether
an MSI-X/IMS entry has PASID enabled and if yes what is the PASID
value to be checked against the descriptor.

As you said virtualizing MSI-X table itself is via SET_IRQS and it's
4k aligned. Then we also need to capture guest updates to the MSI-X
permission table and copy the PASID information into the
corresponding IMS entry when using the IMS backend. It's MSI-X
permission table not 4k aligned then trapping it will affect adjacent
registers.

My quick check in idxd spec doesn't reveal an real impact in perf
critical path. Most registers are configuration/control registers
accessed at driver init time and a few interrupt registers related
to errors or administrative purpose.