Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: RE: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma
 interfaces
Thread-Topic: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma
 interfaces
Thread-Index: AQHXrSF9WDj+Z+DU+UqMJ7tQK2EdmauuvK2AgACKnbCAALw6gIAAElKg
Date:   Wed, 22 Sep 2021 13:56:34 +0000
Message-ID: <BN9PR11MB5433CD8A4A4D14E2FD4A15AD8CA29@BN9PR11MB5433.namprd11.prod.outlook.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-7-yi.l.liu@intel.com>
 <20210921170943.GS327412@nvidia.com>
 <BN9PR11MB5433DA330D4583387B59AA7F8CA29@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210922123931.GI327412@nvidia.com>
In-Reply-To: <20210922123931.GI327412@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?GogA2ankaD3NLzw0ru18TeIZWaX0QX6UPBVDHyBN3j2uf/+kBl1ZesdgOkis?=
 =?us-ascii?Q?qL0MDvhnpg0kQsukzEwYfEfj5TBBNjOvSMhNeNKTg3qKfYT6eSBK79qUOUM3?=
 =?us-ascii?Q?qnGo7XeWScPohXqnfu+d8g9YiC6BNUIdLri6e0QQlHWYnmdJ7LzU9nOM4GCS?=
 =?us-ascii?Q?O9JQX/tBQipuS9SxK7GI6x0+zgug7eeAVZkl205Q+c4Pc+gDxUrZtp47Kf7+?=
 =?us-ascii?Q?TAVkY7doyFrppSc6LgQginpJRiHWWHhav9UQn8x0nbNZDXfjvcWRhwnCzKtc?=
 =?us-ascii?Q?Q3FFDD8ArwebmCRyzEAjn+1+97nx6lg3fHdcDqPYDvuCtcZlsipMF8ohaqE0?=
 =?us-ascii?Q?9usgev1UdC2LvrNlEXVRBUJ2Z60YTxcM9jS8TQ10zOj9on+57fNpzuVRruUN?=
 =?us-ascii?Q?AoypB4F8d1g32ZibLrJxdWSKpsZ6rEEwtc7Jyaf+pyQMyirQJRI+8VPIswjS?=
 =?us-ascii?Q?8KjzcoIG02LIOk4VnpomRUXVhlex1onAhqDogGYkLafBERgSGmQxZcfBPD3N?=
 =?us-ascii?Q?TvLNDkm7qgOl5JR9Ty2L+kebKFSFPEp/wGrwAJJpak5tVf6JL+7gsa4asdIl?=
 =?us-ascii?Q?gQ+ScWDt+VPwta/fUsN04xk0TCrrz0h3DicLCtenSdrLjKKU0ZOIDPDCVTmR?=
 =?us-ascii?Q?pjMv7G4/3MDmPJXl9jkTyGNF+rj+vFCKCfdPNI7FQcDvgcTjPYjMzIja6zE0?=
 =?us-ascii?Q?lTEE+mv8uj7+qKY8ssTggqnidEebBD+r3UCToBeoMvVXG8dBCyDSI+h3duaz?=
 =?us-ascii?Q?XFGU2ld6WRfEyFHurfp2Zc+a1tduXv2IZRJo+XE8+CtR16dom7qJeNc+Btz3?=
 =?us-ascii?Q?VNLLORFs2e7WtDRxnQvIZibdjqoMaZ3NCYldTa4S/Ddaiua55V3frQgM6Pvj?=
 =?us-ascii?Q?tdgmiapa1hhOzh47oH3vykzQIVNYd5GKV5ataBBq2z2IR+QzDee0yMseEl0n?=
 =?us-ascii?Q?gM8hR+r6E4r8VoiKJsKFbpQ7iDo79fQolY1kde2WtGhrKZp5Sm6tjL9LTlgP?=
 =?us-ascii?Q?11NemDqdhbWiMEYraeWuE7tZqJ9hhspUpHZYB2EazSmT2KoWpVIbEPdpL8ds?=
 =?us-ascii?Q?k/QUG3IuLHy7ZIScK8H1+2sU0JJoDVMV5v8h5cwFVxkvqkP1D7qKe+fY3xrj?=
 =?us-ascii?Q?1E7xK5SuARE+N1c4WbKaAtrahmyc1B4eBGk8IFgEwSD+RoYFfcuspl583aTF?=
 =?us-ascii?Q?5uz4UtF7evrCCL8rvedSsk/QWaVdL9/48dLs7thSwYCKlz07ccaM9nmwjZQS?=
 =?us-ascii?Q?GUKqpFfThtUvkPHOFYV50NRQeXwJNC4eLcK+TQ85vs8SE1MDv7jmVGRssXBR?=
 =?us-ascii?Q?MiUGLRIfiCDGwU1rZaZoxRLr?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5433.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6e6841ab-aeb0-4785-6265-08d97dd0ca59
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Sep 2021 13:56:34.9259
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: ovuGQkKIy8zqDkX+wmT/amqO0tAASeESk7AhDi4w/bVeRWoeMNve1Xu1fqG4xTylCriS/kZh2bzEipAUW1fUBQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1938
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, September 22, 2021 8:40 PM
>=20
> On Wed, Sep 22, 2021 at 01:47:05AM +0000, Tian, Kevin wrote:
>=20
> > > IIRC in VFIO the container is the IOAS and when the group goes to
> > > create the device fd it should simply do the
> > > iommu_device_init_user_dma() followed immediately by a call to bind
> > > the container IOAS as your #3.
> >
> > a slight correction.
> >
> > to meet vfio semantics we could do init_user_dma() at group attach
> > time and then call binding to container IOAS when the device fd
> > is created. This is because vfio requires the group in a security conte=
xt
> > before the device is opened.
>=20
> Is it? Until a device FD is opened the group fd is kind of idle, right?

yes, then there is no user-tangible difference between init_user_dma()
at group attach time vs. doing it when opening fd(). But the latter does
require more change than the former, as it also needs the vfio iommu=20
driver to provide a .device_attach callback.=20

What's in my mind now is to keep existing group attach sequence=20
which further calls a group-version init_user_dma(). Then when=20
device fd is created, just create a iommu_dev object and switch to
normal fops.=20

>=20
> > > Ie the basic flow would see the driver core doing some:
> >
> > Just double confirm. Is there concern on having the driver core to
> > call iommu functions?
>=20
> It is always an interesting question, but I'd say iommu is
> foundantional to Linux and if it needs driver core help it shouldn't
> be any different from PM, pinctl, or other subsystems that have
> inserted themselves into the driver core.
>=20
> Something kind of like the below.
>=20
> If I recall, once it is done like this then the entire iommu notifier
> infrastructure can be ripped out which is a lot of code.

thanks for the guidance. will think more along this direction...

>=20
>=20
> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
> index 68ea1f949daa90..e39612c99c6123 100644
> --- a/drivers/base/dd.c
> +++ b/drivers/base/dd.c
> @@ -566,6 +566,10 @@ static int really_probe(struct device *dev, struct
> device_driver *drv)
>                 goto done;
>         }
>=20
> +       ret =3D iommu_set_kernel_ownership(dev);
> +       if (ret)
> +               return ret;
> +
>  re_probe:
>         dev->driver =3D drv;
>=20
> @@ -673,6 +677,7 @@ static int really_probe(struct device *dev, struct
> device_driver *drv)
>                 dev->pm_domain->dismiss(dev);
>         pm_runtime_reinit(dev);
>         dev_pm_set_driver_flags(dev, 0);
> +       iommu_release_kernel_ownership(dev);
>  done:
>         return ret;
>  }
> @@ -1214,6 +1219,7 @@ static void __device_release_driver(struct device
> *dev, struct device *parent)
>                         dev->pm_domain->dismiss(dev);
>                 pm_runtime_reinit(dev);
>                 dev_pm_set_driver_flags(dev, 0);
> +               iommu_release_kernel_ownership(dev);
>=20
>                 klist_remove(&dev->p->knode_driver);
>                 device_pm_check_callbacks(dev);