Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230111062809.25020-1-jasowang@redhat.com> <20230111062809.25020-6-jasowang@redhat.com>
 <DM8PR12MB5400D84460E084D3459B49B1ABC09@DM8PR12MB5400.namprd12.prod.outlook.com>
In-Reply-To: <DM8PR12MB5400D84460E084D3459B49B1ABC09@DM8PR12MB5400.namprd12.prod.outlook.com>
From:   Jason Wang <jasowang@redhat.com>
Date:   Mon, 16 Jan 2023 10:49:10 +0800
Message-ID: <CACGkMEu4ZWgh8xb5xF26AEWafpX=CfUh_CcV0Nf97+iN9+E2ZA@mail.gmail.com>
Subject: Re: [PATCH 5/5] vdpa: mlx5: support per virtqueue dma device
To:     Eli Cohen <elic@nvidia.com>
Cc:     "mst@redhat.com" <mst@redhat.com>,
        "gdawar@amd.com" <gdawar@amd.com>,
        "virtualization@lists.linux-foundation.org" 
        <virtualization@lists.linux-foundation.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "tanuj.kamde@amd.com" <tanuj.kamde@amd.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Sun, Jan 15, 2023 at 7:08 PM Eli Cohen <elic@nvidia.com> wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Wednesday, 11 January 2023 8:28
> > To: mst@redhat.com; jasowang@redhat.com
> > Cc: Eli Cohen <elic@nvidia.com>; gdawar@amd.com;
> > virtualization@lists.linux-foundation.org; linux-kernel@vger.kernel.org;
> > tanuj.kamde@amd.com
> > Subject: [PATCH 5/5] vdpa: mlx5: support per virtqueue dma device
> >
> > This patch implements per virtqueue dma device for mlx5_vdpa. This is
> > needed for virtio_vdpa to work for CVQ which is backed by vringh but
> > not DMA. We simply advertise the vDPA device itself as the DMA device
> > for CVQ then DMA API can simply use PA so the identical mapping for
> > CVQ can still be used. Otherwise the identical (1:1) mapping won't
> > work when platform IOMMU is enabled since the IOVA is allocated on
> > demand which is not necessarily the PA.
> >
> > This fixes the following crash when mlx5 vDPA device is bound to
> > virtio-vdpa with platform IOMMU enabled but not in passthrough mode:
> >
> > BUG: unable to handle page fault for address: ff2fb3063deb1002
> > #PF: supervisor read access in kernel mode
> > #PF: error_code(0x0000) - not-present page
> > PGD 1393001067 P4D 1393002067 PUD 0
> > Oops: 0000 [#1] PREEMPT SMP NOPTI
> > CPU: 55 PID: 8923 Comm: kworker/u112:3 Kdump: loaded Not tainted 6.1.0+
> > #7
> > Hardware name: Dell Inc. PowerEdge R750/0PJ80M, BIOS 1.5.4 12/17/2021
> > Workqueue: mlx5_vdpa_wq mlx5_cvq_kick_handler [mlx5_vdpa]
> > RIP: 0010:vringh_getdesc_iotlb+0x93/0x1d0 [vringh]
> > Code: 14 25 40 ef 01 00 83 82 c0 0a 00 00 01 48 2b 05 93 5a 1b ea 8b 4c 24
> > 14 48 c1 f8 06 48 c1 e0 0c 48 03 05 90 5a 1b ea 48 01 c8 <0f> b7 00 83 aa c0
> > 0a 00 00 01 65 ff 0d bc e4 41 3f 0f 84 05 01 00
> > RSP: 0018:ff46821ba664fdf8 EFLAGS: 00010282
> > RAX: ff2fb3063deb1002 RBX: 0000000000000a20 RCX: 0000000000000002
> > RDX: ff2fb318d2f94380 RSI: 0000000000000002 RDI: 0000000000000001
> > RBP: ff2fb3065e832410 R08: ff46821ba664fe00 R09: 0000000000000001
> > R10: 0000000000000000 R11: 000000000000000d R12: ff2fb3065e832488
> > R13: ff2fb3065e8324a8 R14: ff2fb3065e8324c8 R15: ff2fb3065e8324a8
> > FS:  0000000000000000(0000) GS:ff2fb3257fac0000(0000)
> > knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: ff2fb3063deb1002 CR3: 0000001392010006 CR4: 0000000000771ee0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > PKRU: 55555554
> > Call Trace:
> > <TASK>
> >   mlx5_cvq_kick_handler+0x89/0x2b0 [mlx5_vdpa]
> >   process_one_work+0x1e2/0x3b0
> >   ? rescuer_thread+0x390/0x390
> >   worker_thread+0x50/0x3a0
> >   ? rescuer_thread+0x390/0x390
> >   kthread+0xd6/0x100
> >   ? kthread_complete_and_exit+0x20/0x20
> >   ret_from_fork+0x1f/0x30
> >   </TASK>
> >
> > Signed-off-by: Jason Wang <jasowang@redhat.com>
>
> mlx5_get_vq_dma_dev needs to made static.

Right, I will fix it in v2.

> Other than that:
> Reviewed-by: Eli Cohen <elic@nvidia.com>
> Tested-by: <elic@nvidia.com>

Thanks

>
> Reviewed-by: Eli Cohen <elic@nvidia.com>
> Tested-by: <elic@nvidia.com>
> > ---
> >  drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> >
> > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > index 6632651b1e54..b0e74c25bf48 100644
> > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > @@ -2682,6 +2682,16 @@ static int mlx5_vdpa_set_map(struct vdpa_device
> > *vdev, unsigned int asid,
> >       return err;
> >  }
> >
> > +struct device *mlx5_get_vq_dma_dev(struct vdpa_device *vdev, u16 idx)
> > +{
> > +     struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev);
> > +
> > +     if (is_ctrl_vq_idx(mvdev, idx))
> > +             return &vdev->dev;
> > +
> > +     return mvdev->vdev.dma_dev;
> > +}
> > +
> >  static void mlx5_vdpa_free(struct vdpa_device *vdev)
> >  {
> >       struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev);
> > @@ -2897,6 +2907,7 @@ static const struct vdpa_config_ops mlx5_vdpa_ops
> > = {
> >       .get_generation = mlx5_vdpa_get_generation,
> >       .set_map = mlx5_vdpa_set_map,
> >       .set_group_asid = mlx5_set_group_asid,
> > +     .get_vq_dma_dev = mlx5_get_vq_dma_dev,
> >       .free = mlx5_vdpa_free,
> >       .suspend = mlx5_vdpa_suspend,
> >  };
> > --
> > 2.25.1
>