Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: n7ldQHDoanCFoBsmpybmt/jlNvtG0O9nd2rFJr76c/8zpY/L3GWJjEDLk3fGduOjYWwLqvh+Yt
 Kh4aBzSyZehw==
IronPort-SDR: IR+qu0Q4QcWy1+J31thSeCmmzqnIRzFdJ7FlONM87ZPg0biFHeJ/YnNig7sY11FRUwM5yLwyaB
 3nY7mpmkRAdg==
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Thomas Gleixner <tglx@linutronix.de>
CC:     Alex Williamson <alex.williamson@redhat.com>,
        "Dey, Megha" <megha.dey@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Pan, Jacob jun" <jacob.jun.pan@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>, "Lu, Baolu" <baolu.lu@intel.com>,
        "Williams, Dan J" <dan.j.williams@intel.com>,
        "Luck, Tony" <tony.luck@intel.com>,
        "Kumar, Sanjay K" <sanjay.k.kumar@intel.com>,
        LKML <linux-kernel@vger.kernel.org>, KVM <kvm@vger.kernel.org>,
        Kirti Wankhede <kwankhede@nvidia.com>,
        "Peter Zijlstra" <peterz@infradead.org>,
        Marc Zyngier <maz@kernel.org>,
        "Bjorn Helgaas" <helgaas@kernel.org>
Subject: RE: Virtualizing MSI-X on IMS via VFIO
Thread-Topic: Virtualizing MSI-X on IMS via VFIO
Thread-Index: AddnMs7+4GfLhTceT8q8tdV8716lmQAZ7UiAAAoHBgAACsXtAAAX4LwAAABVr4AADozDgA==
Date:   Wed, 23 Jun 2021 23:41:09 +0000
Message-ID: <MWHPR11MB188642E26DB4D5C541E5DCF18C089@MWHPR11MB1886.namprd11.prod.outlook.com>
References: <20210622131217.76b28f6f.alex.williamson@redhat.com>
 <87o8bxcuxv.ffs@nanos.tec.linutronix.de>
 <MWHPR11MB1886811339F7873A8E34549A8C089@MWHPR11MB1886.namprd11.prod.outlook.com>
 <87bl7wczkp.ffs@nanos.tec.linutronix.de>
 <20210623164109.GL2371267@nvidia.com>
In-Reply-To: <20210623164109.GL2371267@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
dlp-version: 11.5.1.3
dlp-product: dlpe-windows
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?FhOBc757L0+06lpEF8nsrKnvZNqczOMzrXFb1I2DYz7XdrC0bXEG5QpC8wIS?=
 =?us-ascii?Q?vSTapUaA7K13XGm6AL1lZPALPnvkyeI+2BG276eLTOFyHgIy0IBaUwD3XA84?=
 =?us-ascii?Q?4gWTd6ALYeFq7D9C5lxBefzhJLDZudwy1iC2wXOZrZPU7fj3M370vbSk/7Xv?=
 =?us-ascii?Q?R/IHursGISe8fSUbl7CBshIVgTBKoHtAPSwGoqEC3cnNBJw07RlaQ4my1MU0?=
 =?us-ascii?Q?aZUs2bNlD9sg4oeOqRIwM8TwKAEI/nUeWQrTAEjGLO/0WLbjMz0xPIyybgwS?=
 =?us-ascii?Q?AfDsptXnj3bX0qRIh9zIK+A4vg3YlevfdBU3l7JawEqoTPzgDGrNgMbG8Qwa?=
 =?us-ascii?Q?YaMgQCeUsm5czXr7jJ1n6ABLFz5efC2f44baV2whqV+hGhi1VVIR+rfwk9/N?=
 =?us-ascii?Q?m8q8gBe9MNXsEZ7Q0SCE8x2tgq5cDq80Wn526fSeLboUMDr1QbMnyVhttu7s?=
 =?us-ascii?Q?5mdHFhReh+qqko8e28+jMP0Ab77+rsrdDDtN7sb4Oek+AfJizZseaLn8VmoV?=
 =?us-ascii?Q?BGuCn+x3a3TB08TIQygxsKCxTy3g/Hl5REFq5RZp37gvCmvR7K34JGS87qV0?=
 =?us-ascii?Q?xDCVADYlgnth53DyJqwCmjPa+l04SYFFExSCjyXGOVxtuA+M4Cqlge1IrjuQ?=
 =?us-ascii?Q?DaSOAqDlaaVnHLfcCBUXhwv6UUQPXZ8qhTolpRYQFy/tYQf8EfiwdV8k6/7b?=
 =?us-ascii?Q?iBgsepr9aso4ryJpbz2zyC6dej6GVglHEbGYCWRhRkiW+rh0Vna4CkCQJsUv?=
 =?us-ascii?Q?ELG+xE6Jehx3nBQC+5zVo6SBKza7dC7iP5/Dwip36dQUGHSHVNU2WvL6Jdzt?=
 =?us-ascii?Q?ZypaWVL4cf/pMp0+kJ7Kypv4ijO54OWyB7RhxJTNLUXMt0cJPXOnuYnzElPt?=
 =?us-ascii?Q?ZJEVlmKAaR61ltBCBprcH65ZEmR3dM4J4Ztnz+YYAe9hACSzZOIcGWE0HZjh?=
 =?us-ascii?Q?DPSr80YS+TRuevSOs+YxtmRrjGpMXQ0w9G7zQOcqExjkTy7hQFea5Gl/XFHz?=
 =?us-ascii?Q?mf+vQCjZFm94VxVq6Ixu27+/i5ZUwEApZPP4e67xOh6bNpvtpgO/p0NniN+5?=
 =?us-ascii?Q?QN8KEiX0uEMnrjq/y6RJmkW3jLEq3LIERPqMZtwgjNYcuSFbVvd8arT9AUbS?=
 =?us-ascii?Q?bL7DPTaxibGbJxxUQu/ZQu8P1zfkWWqyfEObLwqTELp2ERupo3CaCqf+Bmbs?=
 =?us-ascii?Q?6kbBJkoSEOh3Rs5alfv0GgveJG2t8QiWJmhcc8WqoUf8JVwy0VNNyClcORWN?=
 =?us-ascii?Q?K/C2hUxtsJcd6KV1vvj2Ai9gladHn2KnmkTTuNQGPbU+LDn4kMmN9CXZywET?=
 =?us-ascii?Q?B+fe2BRJ8j50kNZhpIYCVn2p?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MWHPR11MB1886.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b26ddc81-d6b8-4608-2cd2-08d936a060aa
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Jun 2021 23:41:09.2179
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: sHZAZI+uwCB0Mge7/+TIamQEPh+UBqDtf5IGzb/aOessM9SgQHkUKvc8kNEoHRD3lnUTRM/1U3pcH2/RnyFdgQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR11MB1696
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, June 24, 2021 12:41 AM
>=20
> On Wed, Jun 23, 2021 at 06:31:34PM +0200, Thomas Gleixner wrote:
>=20
> > So IMO creating a proper paravirt interface is the right approach.  It
> > avoids _all_ of the trouble and will be necessary anyway once you want
> > to support devices which store the message/pasid in system memory and
> > not in on-device memory.
>=20
> I think this is basically where we got to in the other earlier
> discussion with using IMS natively in VMs - it can't be done
> generically without a new paravirt interface.
>=20
> The guest needs a paravirt interface to program the IOMMU to route MSI
> vectors to the guest's vAPIC and then the guest itself can deliver an
> addr/data pair directly to the HW.
>=20
> In this mode qemu would not emulate MSI at all so will avoid all the
> problems you identified.

No emulation for PF/VF.

But emulation might be required for mdev for two reasons:

1)   the ims entries for mdevs are collapsed together;
2)   there are other fields in ims entry which cannot allow guest to
      control, e.g. PASID;

>=20
> How to build that and provide backwards compat is an open
> question. Instead that thread went into blocking IMS on VM situations..
>=20
> Jason