Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: mzSf6m1/q8A4jELRkWi3O7LkDGEHogczDhZX1VJJONasJubgrqKVfMiAezC8avSdM5ImmfRbvF
 0D5eRQD1oXHw==
IronPort-SDR: 0vXiZJNks+gu+pagwTwxvojkIplwjDYbCV7P4hZw3G2qVwRg1YwHNZH5/isj8LnKONxSUfQrJS
 DPwFXQN/I1jA==
From:   "Liu, Yi L" <yi.l.liu@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "Tian, Kevin" <kevin.tian@intel.com>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Joerg Roedel <joro@8bytes.org>,
        Lu Baolu <baolu.lu@linux.intel.com>,
        David Woodhouse <dwmw2@infradead.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
        Tejun Heo <tj@kernel.org>, Li Zefan <lizefan@huawei.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        "Jean-Philippe Brucker" <jean-philippe@linaro.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Eric Auger <eric.auger@redhat.com>,
        "Jonathan Corbet" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>
Subject: RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation
 APIs
Thread-Topic: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and
 allocation APIs
Thread-Index: AQHXDZuc1/oaTFwUU0WheDxUmLBHiqqKkLIAgACg94CAAC72gIAAD04AgAADogCAAErUgIAETQEAgAOaswCAB7C3gIAAmFuAgADGxQCAAR7PEIAAZiYAgAECgOCAACyd0A==
Date:   Thu, 1 Apr 2021 07:04:01 +0000
Message-ID: <BN6PR11MB40687428F0D0F3B5F13EA3E0C37B9@BN6PR11MB4068.namprd11.prod.outlook.com>
References: <20210319124645.GP2356281@nvidia.com> <YFSqDNJ5yagk4eO+@myrica>
 <20210319135432.GT2356281@nvidia.com> <20210319112221.5123b984@jacob-builder>
 <20210322120300.GU2356281@nvidia.com> <20210324120528.24d82dbd@jacob-builder>
 <20210329163147.GG2356281@nvidia.com>
 <MWHPR11MB188639EE54B48B0E1321C8198C7D9@MWHPR11MB1886.namprd11.prod.outlook.com>
 <20210330132830.GO2356281@nvidia.com>
 <BN6PR11MB40688F5AA2323AB8CC8E65E7C37C9@BN6PR11MB4068.namprd11.prod.outlook.com>
 <20210331124038.GE1463678@nvidia.com>
 <BN6PR11MB406854CAE9D7CE86BEAB3E23C37B9@BN6PR11MB4068.namprd11.prod.outlook.com>
In-Reply-To: <BN6PR11MB406854CAE9D7CE86BEAB3E23C37B9@BN6PR11MB4068.namprd11.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
dlp-reaction: no-action
dlp-version: 11.5.1.3
dlp-product: dlpe-windows
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: f3fc5b80-d554-4d64-a93e-08d8f4dc540b
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Apr 2021 07:04:01.2008
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: WoQIXHUV1uvjsfD4C1ZPjOsG6mUq9XO+uaAA2drywO/APf3+2e4GQ2ab0aVVEL0tB302cjaoVJ6rzcthfCFa+g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5354
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jason,

> From: Liu, Yi L <yi.l.liu@intel.com>
> Sent: Thursday, April 1, 2021 12:39 PM
>=20
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, March 31, 2021 8:41 PM
> >
> > On Wed, Mar 31, 2021 at 07:38:36AM +0000, Liu, Yi L wrote:
> >
> > > The reason is /dev/ioasid FD is per-VM since the ioasid allocated to
> > > the VM should be able to be shared by all assigned device for the VM.
> > > But the SVA operations (bind/unbind page table, cache_invalidate)
> should
> > > be per-device.
> >
> > It is not *per-device* it is *per-ioasid*
> >
> > And as /dev/ioasid is an interface for controlling multiple ioasid's
> > there is no issue to also multiplex the page table manipulation for
> > multiple ioasids as well.
> >
> > What you should do next is sketch out in some RFC the exactl ioctls
> > each FD would have and show how the parts I outlined would work and
> > point out any remaining gaps.
> >
> > The device FD is something like the vfio_device FD from VFIO, it has
> > *nothing* to do with PASID beyond having a single ioctl to authorize
> > the device to use the PASID. All control of the PASID is in
> > /dev/ioasid.
>=20
> good to see this reply. Your idea is much clearer to me now. If I'm getti=
ng
> you correctly. I think the skeleton is something like below:
> f
> 1) userspace opens a /dev/ioasid, meanwhile there will be an ioasid
>    allocated and a per-ioasid context which can be used to do bind page
>    table and cache invalidate, an ioasid FD returned to userspace.
> 2) userspace passes the ioasid FD to VFIO, let it associated with a devic=
e
>    FD (like vfio_device FD).
> 3) userspace binds page table on the ioasid FD with the page table info.
> 4) userspace unbinds the page table on the ioasid FD
> 5) userspace de-associates the ioasid FD and device FD
>=20
> Does above suit your outline?
>=20
> If yes, I still have below concern and wish to see your opinion.
> - the ioasid FD and device association will happen at runtime instead of
>   just happen in the setup phase.
> - how about AMD and ARM's vSVA support? Their PASID allocation and page
> table
>   happens within guest. They only need to bind the guest PASID table to
> host.
>   Above model seems unable to fit them. (Jean, Eric, Jacob please feel fr=
ee
>   to correct me)
> - this per-ioasid SVA operations is not aligned with the native SVA usage
>   model. Native SVA bind is per-device.

After reading your reply in https://lore.kernel.org/linux-iommu/20210331123=
801.GD1463678@nvidia.com/#t
So you mean /dev/ioasid FD is per-VM instead of per-ioasid, so above skelet=
on
doesn't suit your idea. I draft below skeleton to see if our mind is the
same. But I still believe there is an open on how to fit ARM and AMD's
vSVA support in this the per-ioasid SVA operation model. thoughts?

+-----------------------------+--------------------------------------------=
---+
|      userspace              |               kernel space                 =
   |
+-----------------------------+--------------------------------------------=
---+
| ioasid_fd =3D                 | /dev/ioasid does below:                  =
     |
| open("/dev/ioasid", O_RDWR);|   struct ioasid_fd_ctx {                   =
   |
|                             |       struct list_head ioasid_list;        =
   |
|                             |       ...                                  =
   |
|                             |   } ifd_ctx; // ifd_ctx is per ioasid_fd   =
   |
+-----------------------------+--------------------------------------------=
---+
| ioctl(ioasid_fd,            | /dev/ioasid does below:                    =
   |
|       ALLOC, &ioasid);      |   struct ioasid_data {                     =
   |
|                             |       ioasid_t ioasid;                     =
   |
|                             |       struct list_head device_list;        =
   |
|                             |       struct list_head next;               =
   |
|                             |       ...                                  =
   |
|                             |   } id_data; // id_data is per ioasid      =
   |
|                             |                                            =
   |
|                             |   list_add(&id_data.next,                  =
   |
|                             |            &ifd_ctx.ioasid_list);          =
   |
+-----------------------------+--------------------------------------------=
---+
| ioctl(device_fd,            | VFIO does below:                           =
   |
|       DEVICE_ALLOW_IOASID,  | 1) get ioasid_fd, check if ioasid_fd is val=
id |
|       ioasid_fd,            | 2) check if ioasid is allocated from ioasid=
_fd|
|       ioasid);              | 3) register device/domain info to /dev/ioas=
id |
|                             |    tracked in id_data.device_list          =
   |
|                             | 4) record the ioasid in VFIO's per-device  =
   |
|                             |    ioasid list for future security check   =
   |
+-----------------------------+--------------------------------------------=
---+
| ioctl(ioasid_fd,            | /dev/ioasid does below:                    =
   |
|       BIND_PGTBL,           | 1) find ioasid's id_data                   =
   |
|       pgtbl_data,           | 2) loop the id_data.device_list and tell io=
mmu|
|       ioasid);              |    give ioasid access to the devices       =
   |
+-----------------------------+--------------------------------------------=
---+
| ioctl(ioasid_fd,            | /dev/ioasid does below:                    =
   |
|       UNBIND_PGTBL,         | 1) find ioasid's id_data                   =
   |
|       ioasid);              | 2) loop the id_data.device_list and tell io=
mmu|
|                             |    clear ioasid access to the devices      =
   |
+-----------------------------+--------------------------------------------=
---+
| ioctl(device_fd,            | VFIO does below:                           =
   |
|      DEVICE_DISALLOW_IOASID,| 1) check if ioasid is associated in VFIO's =
   |
|       ioasid_fd,            |    device ioasid list.                     =
   |
|       ioasid);              | 2) unregister device/domain info from      =
   |
|                             |    /dev/ioasid, clear in id_data.device_lis=
t  |
+-----------------------------+--------------------------------------------=
---+
| ioctl(ioasid_fd,            | /dev/ioasid does below:                    =
   |
|       FREE, ioasid);        |  list_del(&id_data.next);                  =
   |
+-----------------------------+--------------------------------------------=
---+

Regards,
Yi Liu