Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: QFdTjsHgktUlDfjHPfgtCR6PX2cbtdrymlreUwxcotOIbmavcYDZGc6QeNvGJgKm+rLo/x2pTj
 nVctNaoOHC1w==
IronPort-SDR: tQgp+gEEQwDIgqwL7YDNAHHoKKbcJ6Q+fPgSumUZdLkBun1epexIboQwsn4WHcxUTRg/0w94d4
 JTw/qoiyQW2w==
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Alex Williamson <alex.williamson@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>
CC:     Jason Gunthorpe <jgg@nvidia.com>,
        "Dey, Megha" <megha.dey@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Pan, Jacob jun" <jacob.jun.pan@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>, "Lu, Baolu" <baolu.lu@intel.com>,
        "Williams, Dan J" <dan.j.williams@intel.com>,
        "Luck, Tony" <tony.luck@intel.com>,
        "Kumar, Sanjay K" <sanjay.k.kumar@intel.com>,
        LKML <linux-kernel@vger.kernel.org>, KVM <kvm@vger.kernel.org>,
        Kirti Wankhede <kwankhede@nvidia.com>,
        "Peter Zijlstra" <peterz@infradead.org>,
        Marc Zyngier <maz@kernel.org>,
        "Bjorn Helgaas" <helgaas@kernel.org>
Subject: RE: Virtualizing MSI-X on IMS via VFIO
Thread-Topic: Virtualizing MSI-X on IMS via VFIO
Thread-Index: AddnMs7+4GfLhTceT8q8tdV8716lmQAZ7UiAAAoHBgAAICMUgAARvpSQ
Date:   Thu, 24 Jun 2021 00:00:37 +0000
Message-ID: <MWHPR11MB18864420ACE88E060203F7818C079@MWHPR11MB1886.namprd11.prod.outlook.com>
References: <20210622131217.76b28f6f.alex.williamson@redhat.com>
        <87o8bxcuxv.ffs@nanos.tec.linutronix.de>
 <20210623091935.3ab3e378.alex.williamson@redhat.com>
In-Reply-To: <20210623091935.3ab3e378.alex.williamson@redhat.com>
Accept-Language: en-US
Content-Language: en-US
dlp-version: 11.5.1.3
dlp-product: dlpe-windows
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?+KKHj/U4bm3fznX4Oj7pgVHU7am5Lyy1DMZ2gP2IDOMhyX+5ztefrnNcOekq?=
 =?us-ascii?Q?qO4Qm5SPv8p7/6wBkz3dLxB8AkR5qa0lfGc1ert836sdKsi9K1RCfGfw0e9Y?=
 =?us-ascii?Q?QIK0WDVwnjke+R3U+F6wzaWSBEdaKY39IjXjZWVdDHQIjnKzftlokHd8NSMD?=
 =?us-ascii?Q?l/E8vmLmNcT/knzBghdGXqAZ2aC6XIo4eScykh6ehF+q/KJuut5wsauzS1cC?=
 =?us-ascii?Q?yQdrnJ3N9RRhz+6iaUe+vlU75bW7MEkVjw9iNqO23STAP6pHXDVcdcvkV+Ba?=
 =?us-ascii?Q?3p6hpz5iDT8rBYhg70ID9jbIQP0XOy5peJrkg7vxmxhxeNEp/QV9n1jfZLZy?=
 =?us-ascii?Q?clS4s+R3uM2qQOpGHoIJP1gQYWP4ND1hGxJ9zIW1zl9iCUGM6uqQDk88xFAM?=
 =?us-ascii?Q?K0x4OFOy8itH/GPp0iDj926caBoXsJ43bPknPbX+7yuM2hyEYm6yDa9ztC9d?=
 =?us-ascii?Q?PMRMnaIxntTUu7qnyostsiVe/GjZVWxAft0ZysALybpc9/0OHVhelAp0HMGl?=
 =?us-ascii?Q?trrrJ2uAz3JQHpXG/g7/pmYbDvV9q+vDqBmJm5zy0LqyOV+2tcFNjF1lIXSN?=
 =?us-ascii?Q?QCAoBFOgYGqoUpKJ5QXSTEDQQ9Lp5VAHk9v+NII4Lh9erGyKpy3AIpkUavdc?=
 =?us-ascii?Q?rOQ4R0ubNRyfz81eDWuNvYqHfDE1g7Vc+MbXPGvIoGbgjSGg26SILq5eTk8j?=
 =?us-ascii?Q?mzavG73/5kcpOmPd5VA6cnxXPUgm4oLzoCUGL95FEo9tlLNNiAZcBt+FPg9R?=
 =?us-ascii?Q?wORQj+YTozjF3L5A+J/cHrhmBcVaw7ufZJJYVuxbvV9QMyn0AnM34Wy+dwlu?=
 =?us-ascii?Q?XUxYOVrOFJIZJ1vWjLRRYwmRGmXDNJWd397H5oUt0i3++N3uWZDf4+6PWIMO?=
 =?us-ascii?Q?L7x3wOEta9Oj3/gL2bMzos5UA9CmM2IjUE94lE5xVeB9LK5BwekaxZFO9HBm?=
 =?us-ascii?Q?BxTIIdJbuUQXV8zJtblip1/epWMBb4IMA2Ddmj50KdrkXKb306buVbLGkp2P?=
 =?us-ascii?Q?HvbHyp4gfXWCE1/MvUSt9f5RJhyZOz/IHL+z7hXvte1LjZkhX4kzrycCsPT3?=
 =?us-ascii?Q?Wtgqzj/Hlq1/7Y8A1O2qEY+HsxSEgGKACmPCdVBGTdUM+XrT2TzJVpSeNrlM?=
 =?us-ascii?Q?wqBbyxHpPmU2vAjQSXBknU402SJfhZuUzbUmgZgyZ9C7C2gOdEhdM/kF+kyA?=
 =?us-ascii?Q?WqL/RaFeh/X7PmiouNdqHnTfhh30y64HEedRtWb52SDyHE7RRAbt6LhpEz3k?=
 =?us-ascii?Q?D6/uEcSYAHy+QfWI/Ks/NmunU4yvesQt0l/XUERquqTWAhzucHEC0hrkK20U?=
 =?us-ascii?Q?IKXtxd82mAq2R5HLOtJGxmbH?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MWHPR11MB1886.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d7e8a6d4-dcb5-4919-85bc-08d936a318ca
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Jun 2021 00:00:37.1101
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: WasvzdC6ylQ4U3UXMNmM/lJbOKVBK0/olx1iuA9VgKJwwiiUX7SRHMsZoA/4xYz8ChpanGzuW6R0OL+ERoqvqA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR11MB4962
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Wednesday, June 23, 2021 11:20 PM
>
[...]
 > > So the only downside today of allocating more MSI-X vectors than
> > necessary is memory consumption for the irq descriptors.
>=20
> As above, this is a QEMU policy of essentially trying to be a good
> citizen and allocate only what we can infer the guest is using.  What's
> a good way for QEMU, or any userspace, to know it's running on a host
> where vector exhaustion is not an issue?

In my proposal a new command (VFIO_DEVICE_ALLOC_IRQS) is
introduced to separate allocation from enabling. The availability
of this command could be the indicator whether vector=20
exhaustion is not an issue now?

> >
> > So no, we are not going to proliferate this complete ignorance of how
> > MSI-X actually works and just cram another "feature" into code which is
> > known to be incorrect.
>=20
> Some of the issues of virtualizing MSI-X are unsolvable without
> creating a new paravirtual interface, but obviously we want to work
> with existing drivers and unmodified guests, so that's not an option.
>=20
> To work with what we've got, the vfio API describes the limitation of
> the host interfaces via the VFIO_IRQ_INFO_NORESIZE flag.  QEMU then
> makes a choice in an attempt to better reflect what we can infer of the
> guest programming of the device to incrementally enable vectors.  We

It's a surprise to me that Qemu even doesn't look at this flag today after
searching its code...

> could a) work to provide host kernel interfaces that allow us to remove
> that noresize flag and b) decide whether QEMU's usage policy can be
> improved on kernels where vector exhaustion is no longer an issue.

Thomas can help confirm but looks noresize limitation is still there.=20
b) makes more sense since Thomas thinks vector exhaustion is not=20
an issue now (except one minor open about irte).

Thanks
Kevin