Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     David Gibson <david@gibson.dropbear.id.au>
CC:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "hch@lst.de" <hch@lst.de>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "jgg@nvidia.com" <jgg@nvidia.com>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "Tian, Jun J" <jun.j.tian@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>
Subject: RE: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE
Thread-Topic: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE
Thread-Index: AQHXrSGPLoYXtOF3o0iA7Cse+/LM66u9vAMAgBLi5DCAAXeHgIAAGqnwgBEwZ4CAAuw5IA==
Date:   Wed, 27 Oct 2021 02:32:57 +0000
Message-ID: <BN9PR11MB54330217565C687A7275AE458C859@BN9PR11MB5433.namprd11.prod.outlook.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-12-yi.l.liu@intel.com> <YVamgnMzuv3TCQiX@yekko>
 <BN9PR11MB5433E3BE7550BBF176636F8A8CB79@BN9PR11MB5433.namprd11.prod.outlook.com>
 <YWe5U0fL3t+ldXC2@yekko>
 <BN9PR11MB54331C7936675EEE27C209948CB89@BN9PR11MB5433.namprd11.prod.outlook.com>
 <YXY7A+UQWC4gbIJc@yekko>
In-Reply-To: <YXY7A+UQWC4gbIJc@yekko>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?U2QPF8Nq0hkSU+FDavWDcv4RWtYskh07DjO5YsKfNAr09k9htknGh6lBveCw?=
 =?us-ascii?Q?FKf+f0EryLLhS61BPACob43j35NHtd8ah5AiRMsJTopa+lLVwwhpJnSYYvF3?=
 =?us-ascii?Q?s8EHJDi6LRWiL2q48hocRC+8SyH74ctadNp11jXgG+n3DCkbeg1lgzV83+bw?=
 =?us-ascii?Q?lqyV3KMKrFSC+tJSz4s/VWtqMi4oz9iUKCWeYVbu8z9esBp2dNLUt4jGa+m1?=
 =?us-ascii?Q?0UwGer1Yw8FbH+4Ecz2iRIO30+L5xV/+Louq/2kjrkm/f2bcUL0eGDO9b4kg?=
 =?us-ascii?Q?Rd7cJkautQXwfa9YAOGPRS8WpZlcmYNK5qicPBDotiU01lHMkTXUjUpGcgPD?=
 =?us-ascii?Q?sQQ9yI/hAoX8ACrrj/ie4xK67EBsU1Ze0RoIUVrYtsEQKQsmb51uraW2khCW?=
 =?us-ascii?Q?L0W5QTgr1n2MjFiBU2yNF8cdUazhgR4W4jtUlsGez0TilifqkVMpEPK1p+Fw?=
 =?us-ascii?Q?VMQ6eifybU+IyzBteyiStT7NIYCMLONbxCTkdFaxIeM6+rnjSX/sduadmPq6?=
 =?us-ascii?Q?tBXVc8wgmyX/5k4unMoFWnwj5MQJQCftCCBtHb3EREhroJEGc/8/I4RlECEq?=
 =?us-ascii?Q?o5gt8UzVk+y6fIzpvP5eSJhQenv2nUUFIOnH0EL8PVkaWJ0k9/QpeW7Zpx+B?=
 =?us-ascii?Q?qGY68NTILhjaSCdtwqheISJVGIcGw2TCa1bPPHKEzhiHzgDiOJzpBAsN0ekR?=
 =?us-ascii?Q?emrDIacfCq/vrPJ4gsRDmrp/bQ3kugkSZdd0Mu/A+3nBSuAd2LHR0ygwgfoK?=
 =?us-ascii?Q?AFjcGlryWeaf5RUZZwdDQrwWa3vbKSMi0Bg6CxLOsLnQvDBlcXCevZMJ5fuI?=
 =?us-ascii?Q?vWsTystsaeiwhk59F5QGTcyqMrv4kmDlkA6+LToEBEd/HRmbdCc6+RuShGYO?=
 =?us-ascii?Q?KE5s/J6aexfJdhpHdh9gjrK5Lv+wHqtHgomgJSnOKrC6+XCkjV6TIgiLTtPf?=
 =?us-ascii?Q?Cm1QgjJO1e4QG05eNq9auOAVHmMpBaHvhyRiuDjCCI4QrpdVsSWVJyb6MBPb?=
 =?us-ascii?Q?AGL8JLlv3kuO81MmH4LfUpTqB6+8zMKK9ljAi6hi3NH6Obmoh3gFMrvftSXz?=
 =?us-ascii?Q?C8h5dDl4aJfReuZ6CGTYnCT8edd6CMd7ppZGjEsCiWplMY4aUar8X7zBThdZ?=
 =?us-ascii?Q?tpvhgMpaTYliq/6Lr8ExvHMDKFetaPY6qVCINEwqSziVNQg1ggE30l447orf?=
 =?us-ascii?Q?2hIkCy0i08WK/OVfiuv1kDCnETprjacSH0ocCoQN45tnB9ErR1l6qREekJQE?=
 =?us-ascii?Q?tI5NV9yPi9gLg5VFLYj2pOxe9WvtrzzpRLPNtY+cZR7wWoTOxqGHhESFFshX?=
 =?us-ascii?Q?bONVv2k2J6vxHb1wcnNQOQbTsNe4/zj2JXfWMFz7kqE4TY779Lbo3QA7ygSX?=
 =?us-ascii?Q?6kxnTS+6SqiZsYT8HO1usfsv3n7LmqgnEPwqrX1lSOxbniOZ82U2gymzdx2P?=
 =?us-ascii?Q?VedUdf4flCKUsl48VwSLKeKY2U83WDUIogKzy67LEYvLXs3I3iie/8el3eiJ?=
 =?us-ascii?Q?LmmDQKFneJvQPGsi9Xyb9aZ3FPQ/WJnweKGGgdCasacqfG+RxkhdjdwU1SjQ?=
 =?us-ascii?Q?0ODB0jmSbaRYVjvvqDicNsj9EZhgC5iaLQRmSmRzt6km0oZhKME8sClWMgbK?=
 =?us-ascii?Q?ZQnAjaWhjdxTnc5rkOZmKJM=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5433.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a7afd310-80df-4fd5-d87a-08d998f21647
X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Oct 2021 02:32:57.1287
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: doqkeg13V4D7agqnFiIxNrAMDrol8c4RvlJh1FK/oLL9sdW0jPShhgwtRk3xCCSYOfaLPyLwQnrGbKMWTTuSxg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1268
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: David Gibson <david@gibson.dropbear.id.au>
> Sent: Monday, October 25, 2021 1:05 PM
>=20
> > > > For above cases a [base, max] hint can be provided by the user per
> > > > Jason's recommendation.
> > >
> > > Provided at which stage?
> >
> > IOMMU_IOASID_ALLOC
>=20
> Ok.  I have mixed thoughts on this.  Doing this at ALLOC time was my
> first instict as well.  However with Jason's suggestion that any of a
> number of things could disambiguate multiple IOAS attached to a
> device, I wonder if it makes more sense for consistency to put base
> address at attach time, as with PASID.

In that case the base address provided at attach time is used as an=20
address space ID similar to PASID, which imho is orthogonal to the=20
generic [base, size] info for IOAS itself. The 2nd base sort of becomes=20
an offset on top of the first base in ppc case.

> >
> > regarding live migration with vfio devices, it's still in early stage. =
there
> > are tons of compatibility check opens to be addressed before it can
> > be widely deployed. this might just add another annoying open to that
> > long list...
>=20
> So, yes, live migration with VFIO is limited, unfortunately this
> still affects us even if we don't (currently) have VFIO devices.  The
> problem arises from the combination of two limitations:
>=20
> 1) Live migration means that we can't dynamically select guest visible
> IOVA parameters at qemu start up time.  We need to get consistent
> guest visible behaviour for a given set of qemu options, so that we
> can migrate between them.
>=20
> 2) Device hotplug means that we don't know if a PCI domain will have
> VFIO devices on it when we start qemu.  So, we don't know if host
> limitations on IOVA ranges will affect the guest or not.
>=20
> Together these mean that the best we can do is to define a *fixed*
> (per machine type) configuration based on qemu options only.  That is,
> defined by the guest platform we're trying to present, only, never
> host capabilities.  We can then see if that configuration is possible
> on the host and pass or fail.  It's never safe to go the other
> direction and take host capabilities and present those to the guest.
>=20

That is just one userspace policy. We don't want to design a uAPI
just for a specific userspace implementation. In concept the=20
userspace could:

1)  use DMA-API like map/unmap i.e. letting IOVA address space
    managed by the kernel;

    * suitable for simple applications e.g. dpdk.

2)  manage IOVA address space with *fixed* layout:

    * fail device passthrough at MAP_DMA if conflict is detected
      between mapped range and device specific IOVA holes

    * suitable for VM when live migration is highly concerned

    * potential problem with vIOMMU since the guest is unaware
      of host constraints thus undefined behavior may occur if
      guest IOVA addresses happens to overlap with host IOVA holes.

        * ppc is special as you need to claim guest IOVA ranges in
          the host. But it's not the case for other emulated IOMMUs.

3)  manage IOVA address space with host constraints:

    * create IOVA layout by combining qemu options and IOVA holes=20
      of all boot-time passthrough devices

    * reject hotplugged device if it has conflicting IOVA holes with
      the initial IOVA layout

    * suitable for vIOMMU since host constraints can be further=20
      reported to the guest

    * suitable for VM w/o live migration requirement, e.g. in many
      client virtualization scenarios

    * suboptimal with VM live migration with compatibility limitation

Overall the proposed uAPI will provide:

1)  a simple DMA-API-like mapping protocol for kernel managed IOVA
    address space:

2)  a vfio-like mapping protocol for user managed IOVA address space:

    a) check IOVA conflict in MAP_DMA ioctl;
    b) allows the user to query available IOVA ranges;

Then it's totally user policy on how it wants to utilize those ioctls.

Thanks
Kevin