Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: W8dwnoC3Ef+gRckGim0xE7QE2XsAF5eYXMD6Hg+EFtDEbCErTSB1U0S3GkW2diDPwd7vyfQAHY
 cbgjatBMaflg==
IronPort-SDR: RpzmoFbVcB3/7jBBKvKhN7a3bKQ1xqp8B0dgSkc0X2Za19nGO/BPiHw0GetAE21NsoS/pr6H5T
 8HXnEnuk9DwA==
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>, "Liu, Yi L" <yi.l.liu@intel.com>
CC:     Jean-Philippe Brucker <jean-philippe@linaro.org>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>,
        LKML <linux-kernel@vger.kernel.org>,
        "Joerg Roedel" <joro@8bytes.org>,
        Lu Baolu <baolu.lu@linux.intel.com>,
        "David Woodhouse" <dwmw2@infradead.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
        Tejun Heo <tj@kernel.org>, Li Zefan <lizefan@huawei.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        "Jean-Philippe Brucker" <jean-philippe@linaro.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Eric Auger <eric.auger@redhat.com>,
        "Jonathan Corbet" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>
Subject: RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation
 APIs
Thread-Topic: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and
 allocation APIs
Thread-Index: AQHXDZub0CDG82VGXUmLEYgvMuPs4KqKkLIAgACg94CAAC72gIAAD04AgAADogCAAErUgIAETQEAgAOaswCAB7C3gIAAhxQwgADYDACAATCSAIAAVGMAgAELsQCAACiXgIAAVBgAgAAaMwCAAAESAIAABy2AgAAgOYCAAPrDQA==
Date:   Fri, 2 Apr 2021 07:30:23 +0000
Message-ID: <MWHPR11MB18866AB35A13A139262347FD8C7A9@MWHPR11MB1886.namprd11.prod.outlook.com>
References: <MWHPR11MB188639EE54B48B0E1321C8198C7D9@MWHPR11MB1886.namprd11.prod.outlook.com>
 <20210330132830.GO2356281@nvidia.com>
 <BN6PR11MB40688F5AA2323AB8CC8E65E7C37C9@BN6PR11MB4068.namprd11.prod.outlook.com>
 <20210331124038.GE1463678@nvidia.com>
 <BN6PR11MB406854CAE9D7CE86BEAB3E23C37B9@BN6PR11MB4068.namprd11.prod.outlook.com>
 <BN6PR11MB40687428F0D0F3B5F13EA3E0C37B9@BN6PR11MB4068.namprd11.prod.outlook.com>
 <YGW27KFt9eQB9X2z@myrica>
 <BN6PR11MB4068171CD1D4B823515F7EFBC37B9@BN6PR11MB4068.namprd11.prod.outlook.com>
 <20210401134236.GF1463678@nvidia.com>
 <BN6PR11MB4068C4DE7AF43D44DE70F4C1C37B9@BN6PR11MB4068.namprd11.prod.outlook.com>
 <20210401160337.GJ1463678@nvidia.com>
In-Reply-To: <20210401160337.GJ1463678@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
dlp-version: 11.5.1.3
dlp-product: dlpe-windows
dlp-reaction: no-action
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 9d3acda4-b60d-463c-c69a-08d8f5a92d9c
X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Apr 2021 07:30:23.4473
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: FnL68DT2dmjkZ9ZHAS7l/HX6TYnEer/DSYVDnEv4No1nmrcxtfL/mO1NQ8lpnkzf6lmNKZi3rg/x9A8psAROgQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR11MB1343
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Friday, April 2, 2021 12:04 AM
>=20
> On Thu, Apr 01, 2021 at 02:08:17PM +0000, Liu, Yi L wrote:
>=20
> > DMA page faults are delivered to root-complex via page request message
> and
> > it is per-device according to PCIe spec. Page request handling flow is:
> >
> > 1) iommu driver receives a page request from device
> > 2) iommu driver parses the page request message. Get the RID,PASID,
> faulted
> >    page and requested permissions etc.
> > 3) iommu driver triggers fault handler registered by device driver with
> >    iommu_report_device_fault()
>=20
> This seems confused.
>=20
> The PASID should define how to handle the page fault, not the driver.
>=20
> I don't remember any device specific actions in ATS, so what is the
> driver supposed to do?
>=20
> > 4) device driver's fault handler signals an event FD to notify userspac=
e to
> >    fetch the information about the page fault. If it's VM case, inject =
the
> >    page fault to VM and let guest to solve it.
>=20
> If the PASID is set to 'report page fault to userspace' then some
> event should come out of /dev/ioasid, or be reported to a linked
> eventfd, or whatever.
>=20
> If the PASID is set to 'SVM' then the fault should be passed to
> handle_mm_fault
>=20
> And so on.
>=20
> Userspace chooses what happens based on how they configure the PASID
> through /dev/ioasid.
>=20
> Why would a device driver get involved here?
>=20
> > Eric has sent below series for the page fault reporting for VM with pas=
sthru
> > device.
> > https://lore.kernel.org/kvm/20210223210625.604517-5-
> eric.auger@redhat.com/
>=20
> It certainly should not be in vfio pci. Everything using a PASID needs
> this infrastructure, VDPA, mdev, PCI, CXL, etc.
>=20

This touches an interesting fact:

The fault may be triggered in either 1st-level or 2nd-level page table,=20
when nested translation is enabled (in vSVA case). The 1st-level is bound=20
by the user space, which therefore needs to receive the fault event. The=20
2nd-level is managed by VFIO (or vDPA), which needs to fix the fault in=20
kernel (e.g. find HVA per faulting GPA, call handle_mm_fault and map=20
GPA->HPA to IOMMU). Yi's current proposal lets VFIO to register the=20
device fault handler, which then forward the event through /dev/ioasid=20
to userspace only if it is a 1st-level fault. Are you suggesting a pgtable-
centric fault reporting mechanism to separate handlers in each level,=20
i.e. letting VFIO register handler only for 2nd-level fault and then /dev/
ioasid register handler for 1st-level fault?

Thanks
Kevin