Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Parav Pandit <parav@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Joerg Roedel <joro@8bytes.org>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Lu Baolu <baolu.lu@linux.intel.com>,
        David Woodhouse <dwmw2@infradead.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "Alex Williamson (alex.williamson@redhat.com)" 
        <alex.williamson@redhat.com>, Jason Wang <jasowang@redhat.com>
CC:     Eric Auger <eric.auger@redhat.com>,
        Jonathan Corbet <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.org>,
        David Gibson <david@gibson.dropbear.id.au>,
        Kirti Wankhede <kwankhede@nvidia.com>,
        Robin Murphy <robin.murphy@arm.com>
Subject: RE: [RFC] /dev/ioasid uAPI proposal
Thread-Topic: [RFC] /dev/ioasid uAPI proposal
Thread-Index: AddSzQ970oLnVHLeQca/ysPD8zMJZwEO8mWg
Date:   Tue, 1 Jun 2021 17:30:51 +0000
Message-ID: <PH0PR12MB54811863B392C644E5365446DC3E9@PH0PR12MB5481.namprd12.prod.outlook.com>
References: <MWHPR11MB1886422D4839B372C6AB245F8C239@MWHPR11MB1886.namprd11.prod.outlook.com>
In-Reply-To: <MWHPR11MB1886422D4839B372C6AB245F8C239@MWHPR11MB1886.namprd11.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: fadf3c45-f0d2-40c4-79fa-08d9252300a9
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jun 2021 17:30:51.2983
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: FpkGxyUXwWXPh+RzItkB1EUQUbiDeZ00GxValMX7t6o8DLYzLA56sQfgczmsXnQbaqMh59fE8MlAoQbDyU2N+w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB5419
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Tian, Kevin <kevin.tian@intel.com>
> Sent: Thursday, May 27, 2021 1:28 PM

> 5.6. I/O page fault
> +++++++++++++++
>=20
> (uAPI is TBD. Here is just about the high-level flow from host IOMMU driv=
er
> to guest IOMMU driver and backwards).
>=20
> -   Host IOMMU driver receives a page request with raw fault_data {rid,
>     pasid, addr};
>=20
> -   Host IOMMU driver identifies the faulting I/O page table according to
>     information registered by IOASID fault handler;
>=20
> -   IOASID fault handler is called with raw fault_data (rid, pasid, addr)=
, which
>     is saved in ioasid_data->fault_data (used for response);
>=20
> -   IOASID fault handler generates an user fault_data (ioasid, addr), lin=
ks it
>     to the shared ring buffer and triggers eventfd to userspace;
>=20
> -   Upon received event, Qemu needs to find the virtual routing informati=
on
>     (v_rid + v_pasid) of the device attached to the faulting ioasid. If t=
here are
>     multiple, pick a random one. This should be fine since the purpose is=
 to
>     fix the I/O page table on the guest;
>=20
> -   Qemu generates a virtual I/O page fault through vIOMMU into guest,
>     carrying the virtual fault data (v_rid, v_pasid, addr);
>=20
Why does it have to be through vIOMMU?
For a VFIO PCI device, have you considered to reuse the same PRI interface =
to inject page fault in the guest?
This eliminates any new v_rid.
It will also route the page fault request and response through the right vf=
io device.

> -   Guest IOMMU driver fixes up the fault, updates the I/O page table, an=
d
>     then sends a page response with virtual completion data (v_rid, v_pas=
id,
>     response_code) to vIOMMU;
>=20
What about fixing up the fault for mmu page table as well in guest?
Or you meant both when above you said "updates the I/O page table"?

It is unclear to me that if there is single nested page table maintained or=
 two (one for cr3 references and other for iommu).
Can you please clarify?

> -   Qemu finds the pending fault event, converts virtual completion data
>     into (ioasid, response_code), and then calls a /dev/ioasid ioctl to
>     complete the pending fault;
>=20
For VFIO PCI device a virtual PRI request response interface is done, it ca=
n be generic interface among multiple vIOMMUs.

> -   /dev/ioasid finds out the pending fault data {rid, pasid, addr} saved=
 in
>     ioasid_data->fault_data, and then calls iommu api to complete it with
>     {rid, pasid, response_code};
>