Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
MIME-Version: 1.0
References: <20230923170540.1447301-1-lulu@redhat.com> <20230923170540.1447301-2-lulu@redhat.com>
In-Reply-To: <20230923170540.1447301-2-lulu@redhat.com>
From:   Jason Wang <jasowang@redhat.com>
Date:   Tue, 26 Sep 2023 10:47:25 +0800
Message-ID: <CACGkMEsL-jWiNeWG5pEA2jp50=BqYmGWM1HsMiCigxic+S0_5A@mail.gmail.com>
Subject: Re: [RFC 1/7] vhost/iommufd: Add the functions support iommufd
To:     Cindy Lu <lulu@redhat.com>
Cc:     mst@redhat.com, yi.l.liu@intel.com, jgg@nvidia.com,
        linux-kernel@vger.kernel.org,
        virtualization@lists.linux-foundation.org, netdev@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Sun, Sep 24, 2023 at 1:05=E2=80=AFAM Cindy Lu <lulu@redhat.com> wrote:
>
> Add a new file vhost/iommufd.c to support the function of
> iommufd, This file contains iommufd function of emulated device and
> the physical device.
>
> Signed-off-by: Cindy Lu <lulu@redhat.com>
> ---
>  drivers/vhost/iommufd.c | 151 ++++++++++++++++++++++++++++++++++++++++
>  drivers/vhost/vhost.h   |  21 ++++++
>  2 files changed, 172 insertions(+)
>  create mode 100644 drivers/vhost/iommufd.c
>
> diff --git a/drivers/vhost/iommufd.c b/drivers/vhost/iommufd.c
> new file mode 100644
> index 000000000000..080858f76fd5
> --- /dev/null
> +++ b/drivers/vhost/iommufd.c
> @@ -0,0 +1,151 @@
> +#include <linux/vdpa.h>
> +#include <linux/iommufd.h>
> +
> +#include "vhost.h"
> +
> +MODULE_IMPORT_NS(IOMMUFD);
> +
> +int vdpa_iommufd_bind(struct vdpa_device *vdpa, struct iommufd_ctx *ictx=
,
> +                     u32 *ioas_id, u32 *device_id)
> +{
> +       int ret;
> +
> +       vhost_vdpa_lockdep_assert_held(vdpa);
> +
> +       /*
> +        * If the driver doesn't provide this op then it means the device=
 does
> +        * not do DMA at all. So nothing to do.
> +        */
> +       if (!vdpa->config->bind_iommufd)
> +               return 0;
> +
> +       ret =3D vdpa->config->bind_iommufd(vdpa, ictx, device_id);
> +       if (ret)
> +               return ret;
> +
> +       ret =3D vdpa->config->attach_ioas(vdpa, ioas_id);
> +       if (ret)
> +               goto err_unbind;
> +       vdpa->iommufd_attached =3D true;
> +
> +       return 0;
> +
> +err_unbind:
> +       if (vdpa->config->unbind_iommufd)
> +               vdpa->config->unbind_iommufd(vdpa);
> +       return ret;
> +}
> +
> +void vdpa_iommufd_unbind(struct vdpa_device *vdpa)
> +{
> +       vhost_vdpa_lockdep_assert_held(vdpa);
> +
> +       if (vdpa->config->unbind_iommufd)
> +               vdpa->config->unbind_iommufd(vdpa);
> +}
> +
> +int vdpa_iommufd_physical_bind(struct vdpa_device *vdpa,
> +                              struct iommufd_ctx *ictx, u32 *out_device_=
id)
> +{
> +       struct device *dma_dev =3D vdpa_get_dma_dev(vdpa);
> +       struct iommufd_device *idev;
> +
> +       idev =3D iommufd_device_bind(ictx, dma_dev, out_device_id);
> +       if (IS_ERR(idev))
> +               return PTR_ERR(idev);
> +       vdpa->iommufd_device =3D idev;
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(vdpa_iommufd_physical_bind);
> +
> +void vdpa_iommufd_physical_unbind(struct vdpa_device *vdpa)
> +{
> +       vhost_vdpa_lockdep_assert_held(vdpa);
> +
> +       if (vdpa->iommufd_attached) {
> +               iommufd_device_detach(vdpa->iommufd_device);
> +               vdpa->iommufd_attached =3D false;
> +       }
> +       iommufd_device_unbind(vdpa->iommufd_device);
> +       vdpa->iommufd_device =3D NULL;
> +}
> +EXPORT_SYMBOL_GPL(vdpa_iommufd_physical_unbind);
> +
> +int vdpa_iommufd_physical_attach_ioas(struct vdpa_device *vdpa, u32 *pt_=
id)
> +{
> +       unsigned int flags =3D 0;
> +
> +       return iommufd_device_attach(vdpa->iommufd_device, pt_id);
> +}
> +EXPORT_SYMBOL_GPL(vdpa_iommufd_physical_attach_ioas);
> +
> +static void vdpa_emulated_unmap(void *data, unsigned long iova,
> +                               unsigned long length)
> +{
> +       struct vdpa_device *vdpa =3D data;
> +
> +       vdpa->config->dma_unmap(vdpa, 0, iova, length);
> +}
> +
> +static const struct iommufd_access_ops vdpa_user_ops =3D {
> +       .needs_pin_pages =3D 1,

Note that simulators support VA, so no page pinning in that case while reba=
sing.

static bool use_va =3D true;
module_param(use_va, bool, 0444);
MODULE_PARM_DESC(use_va, "Enable/disable the device's ability to use VA");

So we need to handle that case as well.

(Note that it looks like VA mode is broken, I may need some time to fix tha=
t).

> +       .unmap =3D vdpa_emulated_unmap,
> +};
> +
> +int vdpa_iommufd_emulated_bind(struct vdpa_device *vdpa,
> +                              struct iommufd_ctx *ictx, u32 *out_device_=
id)
> +{
> +       vhost_vdpa_lockdep_assert_held(vdpa);
> +
> +       vdpa->iommufd_ictx =3D ictx;
> +       iommufd_ctx_get(ictx);
> +       struct iommufd_device *idev;
> +
> +       idev =3D iommufd_device_bind(ictx, vdpa->dma_dev, out_device_id);

This seems not appropriate for emulated devices as it deals with the
concepts that only exist in physical devices like the IOMMU domain
etc.

If possible, please refer how VFIO handles this (I guess it should
have something)

Thanks