Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4;
MIME-Version: 1.0
References: <20230928164550.980832-2-dtatulea@nvidia.com> <20230928164550.980832-18-dtatulea@nvidia.com>
In-Reply-To: <20230928164550.980832-18-dtatulea@nvidia.com>
From:   Eugenio Perez Martin <eperezma@redhat.com>
Date:   Thu, 5 Oct 2023 12:41:48 +0200
Message-ID: <CAJaqyWe9iYnh_DwZbU4986bSAZ0C1=Y9Th6-vv_ZRTTYwQqhYA@mail.gmail.com>
Subject: Re: [PATCH vhost 16/16] vdpa/mlx5: Update cvq iotlb mapping on ASID change
To:     Dragos Tatulea <dtatulea@nvidia.com>
Cc:     gal@nvidia.com, "Michael S . Tsirkin" <mst@redhat.com>,
        Jason Wang <jasowang@redhat.com>,
        Xuan Zhuo <xuanzhuo@linux.alibaba.com>,
        virtualization@lists.linux-foundation.org,
        linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Thu, Sep 28, 2023 at 6:50=E2=80=AFPM Dragos Tatulea <dtatulea@nvidia.com=
> wrote:
>
> For the following sequence:
> - cvq group is in ASID 0
> - .set_map(1, cvq_iotlb)
> - .set_group_asid(cvq_group, 1)
>
> ... the cvq mapping from ASID 0 will be used. This is not always correct
> behaviour.
>
> This patch adds support for the above mentioned flow by saving the iotlb
> on each .set_map and updating the cvq iotlb with it on a cvq group change=
.
>
> Signed-off-by: Dragos Tatulea <dtatulea@nvidia.com>
> ---
>  drivers/vdpa/mlx5/core/mlx5_vdpa.h |  2 ++
>  drivers/vdpa/mlx5/core/mr.c        | 26 ++++++++++++++++++++++++++
>  drivers/vdpa/mlx5/net/mlx5_vnet.c  |  9 ++++++++-
>  3 files changed, 36 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/=
mlx5_vdpa.h
> index ae09296f4270..db988ced5a5d 100644
> --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h
> +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
> @@ -32,6 +32,8 @@ struct mlx5_vdpa_mr {
>         unsigned long num_directs;
>         unsigned long num_klms;
>
> +       struct vhost_iotlb *iotlb;
> +
>         bool user_mr;
>  };
>
> diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c
> index a4135c16b5bf..403c08271489 100644
> --- a/drivers/vdpa/mlx5/core/mr.c
> +++ b/drivers/vdpa/mlx5/core/mr.c
> @@ -499,6 +499,8 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_de=
v *mvdev, struct mlx5_vdpa_
>                 destroy_user_mr(mvdev, mr);
>         else
>                 destroy_dma_mr(mvdev, mr);
> +
> +       vhost_iotlb_free(mr->iotlb);
>  }
>
>  void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev,
> @@ -558,6 +560,30 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev=
 *mvdev,
>         else
>                 err =3D create_dma_mr(mvdev, mr);
>
> +       if (err)
> +               return err;
> +
> +       mr->iotlb =3D vhost_iotlb_alloc(0, 0);
> +       if (!mr->iotlb) {
> +               err =3D -ENOMEM;
> +               goto err_mr;
> +       }
> +
> +       err =3D dup_iotlb(mr->iotlb, iotlb);
> +       if (err)
> +               goto err_iotlb;
> +
> +       return 0;
> +
> +err_iotlb:
> +       vhost_iotlb_free(mr->iotlb);
> +
> +err_mr:
> +       if (iotlb)
> +               destroy_user_mr(mvdev, mr);
> +       else
> +               destroy_dma_mr(mvdev, mr);
> +
>         return err;
>  }
>
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/ml=
x5_vnet.c
> index 46441e41892c..fc5d6b989a5a 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -3154,12 +3154,19 @@ static int mlx5_set_group_asid(struct vdpa_device=
 *vdev, u32 group,
>                                unsigned int asid)
>  {
>         struct mlx5_vdpa_dev *mvdev =3D to_mvdev(vdev);
> +       int err =3D 0;
>
>         if (group >=3D MLX5_VDPA_NUMVQ_GROUPS || asid >=3D MLX5_VDPA_NUM_=
AS)
>                 return -EINVAL;
>
>         mvdev->group2asid[group] =3D asid;
> -       return 0;
> +
> +       mutex_lock(&mvdev->mr_mtx);
> +       if (group =3D=3D MLX5_VDPA_CVQ_GROUP && mvdev->mr[asid])
> +               err =3D mlx5_vdpa_update_cvq_iotlb(mvdev, mvdev->mr[asid]=
->iotlb, asid);

Do we need to protect here in case userspace sets the same ASID twice?
mlx5_vdpa_update_cvq_iotlb shouldn't call dup_iotlb with the same src
and dst.

Apart from that:

Acked-by: Eugenio P=C3=A9rez <eperezma@redhat.com>

> +       mutex_unlock(&mvdev->mr_mtx);
> +
> +       return err;
>  }
>
>  static const struct vdpa_config_ops mlx5_vdpa_ops =3D {
> --
> 2.41.0
>