2020-10-28 21:46:12

by Andy Duan

[permalink] [raw]
Subject: RE: [EXT] Re: [PATCH V5 0/2] Change vring space from nomal memory to dma coherent memory

From: Greg KH <[email protected]> Sent: Wednesday, October 28, 2020 7:14 PM
> On Wed, Oct 28, 2020 at 10:17:39AM +0000, Andy Duan wrote:
> > From: Greg KH <[email protected]> Sent: Wednesday, October
> > 28, 2020 3:07 PM
> > > On Wed, Oct 28, 2020 at 06:05:28AM +0000, Sherry Sun wrote:
> > > > Hi Greg,
> > > >
> > > > > Subject: Re: [PATCH V5 0/2] Change vring space from nomal memory
> > > > > to dma coherent memory
> > > > >
> > > > > On Wed, Oct 28, 2020 at 10:03:03AM +0800, Sherry Sun wrote:
> > > > > > Changes in V5:
> > > > > > 1. Reorganize the vop_mmap function code in patch 1, which is
> > > > > > done by
> > > > > Christoph.
> > > > > > 2. Completely remove the unnecessary code related to reassign
> > > > > > the used ring for card in patch 2.
> > > > > >
> > > > > > The original vop driver only supports dma coherent device, as
> > > > > > it allocates and maps vring by _get_free_pages and
> > > > > > dma_map_single, but not use dma_sync_single_for_cpu/device to
> > > > > > sync the updates of device_page/vring between EP and RC, which
> > > > > > will cause memory synchronization problem for device don't support
> hardware dma coherent.
> > > > > >
> > > > > > And allocate vrings use dma_alloc_coherent is a common way in
> > > > > > kernel, as the memory interacted between two systems should
> > > > > > use consistent memory to avoid caching effects. So here add
> > > > > > noncoherent platform
> > > > > support for vop driver.
> > > > > > Also add some related dma changes to make sure noncoherent
> > > > > > platform works well.
> > > > > >
> > > > > > Sherry Sun (2):
> > > > > > misc: vop: change the way of allocating vrings and device page
> > > > > > misc: vop: do not allocate and reassign the used ring
> > > > > >
> > > > > > drivers/misc/mic/bus/vop_bus.h | 2 +
> > > > > > drivers/misc/mic/host/mic_boot.c | 9 ++
> > > > > > drivers/misc/mic/host/mic_main.c | 43 ++------
> > > > > > drivers/misc/mic/vop/vop_debugfs.c | 4 -
> > > > > > drivers/misc/mic/vop/vop_main.c | 70 +-----------
> > > > > > drivers/misc/mic/vop/vop_vringh.c | 166 ++++++++++-------------------
> > > > > > include/uapi/linux/mic_common.h | 9 +-
> > > > > > 7 files changed, 85 insertions(+), 218 deletions(-)
> > > > >
> > > > > Have you all seen:
> > > > >
> > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > 25
> > > > >
> > >
> 2Flore.kernel.org%2Fr%2F8c1443136563de34699d2c084df478181c205db4.16
> > > > >
> > >
> 03854416.git.sudeep.dutt%40intel.com&amp;data=04%7C01%7Csherry.sun%
> > > > >
> > >
> 40nxp.com%7Cc19c987667434969847e08d87b0685e8%7C686ea1d3bc2b4c6f
> > > > >
> > >
> a92cd99c5c301635%7C0%7C0%7C637394615238940323%7CUnknown%7CTW
> > > > >
> > >
> FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> > > > >
> > >
> VCI6Mn0%3D%7C1000&amp;sdata=Zq%2FtHWTq%2BuIVBYXFGoeBmq0JJzYd
> > > > > 9zDyv4NVN4TpC%2FU%3D&amp;reserved=0
> > > > >
> > > > > Looks like this code is asking to just be deleted, is that ok with you?
> > > >
> > > > Yes, I saw that patch. I'm ok with it.
> > >
> > > Great, can you please provide a "Reviewed-by:" or "Acked-by:" for it?
> > >
> > > thanks,
> > >
> > > greg k-h
> >
> > Sherry took much effort on the features support on i.MX series like
> i.MX8QM/i.MX8QXP/i.MX8MM.
> >
> > Now it is a pity to delete the vop code.
> >
> > One question,
> > can we resubmit vop code by clean up, now only for i.MX series as Dutt's
> suggestion ?
> > Or we have to drop the design and switch to select other solutions ?
>

Okay, we plan to switch to NTB solution.

> If this whole subsystem is being deleted because it is not used and never shipped,
> yes, please use a different solution.
>
> I don't understand why you were trying to piggy-back on this codebase if the
> hardware was totally different, for some reason I thought this was the same
> hardware. What exactly is this?

Not the whole codebase, just the vop framework.

>
> thanks,
>
> greg k-h



2020-10-29 00:59:38

by Greg KH

[permalink] [raw]
Subject: Re: [EXT] Re: [PATCH V5 0/2] Change vring space from nomal memory to dma coherent memory

On Wed, Oct 28, 2020 at 03:11:15PM +0000, Andy Duan wrote:
> From: Greg KH <[email protected]> Sent: Wednesday, October 28, 2020 7:14 PM
> > On Wed, Oct 28, 2020 at 10:17:39AM +0000, Andy Duan wrote:
> > > From: Greg KH <[email protected]> Sent: Wednesday, October
> > > 28, 2020 3:07 PM
> > > > On Wed, Oct 28, 2020 at 06:05:28AM +0000, Sherry Sun wrote:
> > > > > Hi Greg,
> > > > >
> > > > > > Subject: Re: [PATCH V5 0/2] Change vring space from nomal memory
> > > > > > to dma coherent memory
> > > > > >
> > > > > > On Wed, Oct 28, 2020 at 10:03:03AM +0800, Sherry Sun wrote:
> > > > > > > Changes in V5:
> > > > > > > 1. Reorganize the vop_mmap function code in patch 1, which is
> > > > > > > done by
> > > > > > Christoph.
> > > > > > > 2. Completely remove the unnecessary code related to reassign
> > > > > > > the used ring for card in patch 2.
> > > > > > >
> > > > > > > The original vop driver only supports dma coherent device, as
> > > > > > > it allocates and maps vring by _get_free_pages and
> > > > > > > dma_map_single, but not use dma_sync_single_for_cpu/device to
> > > > > > > sync the updates of device_page/vring between EP and RC, which
> > > > > > > will cause memory synchronization problem for device don't support
> > hardware dma coherent.
> > > > > > >
> > > > > > > And allocate vrings use dma_alloc_coherent is a common way in
> > > > > > > kernel, as the memory interacted between two systems should
> > > > > > > use consistent memory to avoid caching effects. So here add
> > > > > > > noncoherent platform
> > > > > > support for vop driver.
> > > > > > > Also add some related dma changes to make sure noncoherent
> > > > > > > platform works well.
> > > > > > >
> > > > > > > Sherry Sun (2):
> > > > > > > misc: vop: change the way of allocating vrings and device page
> > > > > > > misc: vop: do not allocate and reassign the used ring
> > > > > > >
> > > > > > > drivers/misc/mic/bus/vop_bus.h | 2 +
> > > > > > > drivers/misc/mic/host/mic_boot.c | 9 ++
> > > > > > > drivers/misc/mic/host/mic_main.c | 43 ++------
> > > > > > > drivers/misc/mic/vop/vop_debugfs.c | 4 -
> > > > > > > drivers/misc/mic/vop/vop_main.c | 70 +-----------
> > > > > > > drivers/misc/mic/vop/vop_vringh.c | 166 ++++++++++-------------------
> > > > > > > include/uapi/linux/mic_common.h | 9 +-
> > > > > > > 7 files changed, 85 insertions(+), 218 deletions(-)
> > > > > >
> > > > > > Have you all seen:
> > > > > >
> > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > 25
> > > > > >
> > > >
> > 2Flore.kernel.org%2Fr%2F8c1443136563de34699d2c084df478181c205db4.16
> > > > > >
> > > >
> > 03854416.git.sudeep.dutt%40intel.com&amp;data=04%7C01%7Csherry.sun%
> > > > > >
> > > >
> > 40nxp.com%7Cc19c987667434969847e08d87b0685e8%7C686ea1d3bc2b4c6f
> > > > > >
> > > >
> > a92cd99c5c301635%7C0%7C0%7C637394615238940323%7CUnknown%7CTW
> > > > > >
> > > >
> > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> > > > > >
> > > >
> > VCI6Mn0%3D%7C1000&amp;sdata=Zq%2FtHWTq%2BuIVBYXFGoeBmq0JJzYd
> > > > > > 9zDyv4NVN4TpC%2FU%3D&amp;reserved=0
> > > > > >
> > > > > > Looks like this code is asking to just be deleted, is that ok with you?
> > > > >
> > > > > Yes, I saw that patch. I'm ok with it.
> > > >
> > > > Great, can you please provide a "Reviewed-by:" or "Acked-by:" for it?
> > > >
> > > > thanks,
> > > >
> > > > greg k-h
> > >
> > > Sherry took much effort on the features support on i.MX series like
> > i.MX8QM/i.MX8QXP/i.MX8MM.
> > >
> > > Now it is a pity to delete the vop code.
> > >
> > > One question,
> > > can we resubmit vop code by clean up, now only for i.MX series as Dutt's
> > suggestion ?
> > > Or we have to drop the design and switch to select other solutions ?
> >
>
> Okay, we plan to switch to NTB solution.

What is a "NTB solution" exactly?

>
> > If this whole subsystem is being deleted because it is not used and never shipped,
> > yes, please use a different solution.
> >
> > I don't understand why you were trying to piggy-back on this codebase if the
> > hardware was totally different, for some reason I thought this was the same
> > hardware. What exactly is this?
>
> Not the whole codebase, just the vop framework.

That didn't answer the question at all, what are you all trying to do
here, with what hardware, that the VOP code seemed like a good fit?

thanks,

greg k-h

2020-10-29 08:48:16

by Sherry Sun

[permalink] [raw]
Subject: RE: [EXT] Re: [PATCH V5 0/2] Change vring space from nomal memory to dma coherent memory

Hi Greg,

> Subject: Re: [EXT] Re: [PATCH V5 0/2] Change vring space from nomal
> memory to dma coherent memory
>
> On Wed, Oct 28, 2020 at 03:11:15PM +0000, Andy Duan wrote:
> > From: Greg KH <[email protected]> Sent: Wednesday, October
> > 28, 2020 7:14 PM
> > > On Wed, Oct 28, 2020 at 10:17:39AM +0000, Andy Duan wrote:
> > > > From: Greg KH <[email protected]> Sent: Wednesday,
> > > > October 28, 2020 3:07 PM
> > > > > On Wed, Oct 28, 2020 at 06:05:28AM +0000, Sherry Sun wrote:
> > > > > > Hi Greg,
> > > > > >
> > > > > > > Subject: Re: [PATCH V5 0/2] Change vring space from nomal
> > > > > > > memory to dma coherent memory
> > > > > > >
> > > > > > > On Wed, Oct 28, 2020 at 10:03:03AM +0800, Sherry Sun wrote:
> > > > > > > > Changes in V5:
> > > > > > > > 1. Reorganize the vop_mmap function code in patch 1, which
> > > > > > > > is done by
> > > > > > > Christoph.
> > > > > > > > 2. Completely remove the unnecessary code related to
> > > > > > > > reassign the used ring for card in patch 2.
> > > > > > > >
> > > > > > > > The original vop driver only supports dma coherent device,
> > > > > > > > as it allocates and maps vring by _get_free_pages and
> > > > > > > > dma_map_single, but not use dma_sync_single_for_cpu/device
> > > > > > > > to sync the updates of device_page/vring between EP and
> > > > > > > > RC, which will cause memory synchronization problem for
> > > > > > > > device don't support
> > > hardware dma coherent.
> > > > > > > >
> > > > > > > > And allocate vrings use dma_alloc_coherent is a common way
> > > > > > > > in kernel, as the memory interacted between two systems
> > > > > > > > should use consistent memory to avoid caching effects. So
> > > > > > > > here add noncoherent platform
> > > > > > > support for vop driver.
> > > > > > > > Also add some related dma changes to make sure noncoherent
> > > > > > > > platform works well.
> > > > > > > >
> > > > > > > > Sherry Sun (2):
> > > > > > > > misc: vop: change the way of allocating vrings and device page
> > > > > > > > misc: vop: do not allocate and reassign the used ring
> > > > > > > >
> > > > > > > > drivers/misc/mic/bus/vop_bus.h | 2 +
> > > > > > > > drivers/misc/mic/host/mic_boot.c | 9 ++
> > > > > > > > drivers/misc/mic/host/mic_main.c | 43 ++------
> > > > > > > > drivers/misc/mic/vop/vop_debugfs.c | 4 -
> > > > > > > > drivers/misc/mic/vop/vop_main.c | 70 +-----------
> > > > > > > > drivers/misc/mic/vop/vop_vringh.c | 166 ++++++++++-------------
> ------
> > > > > > > > include/uapi/linux/mic_common.h | 9 +-
> > > > > > > > 7 files changed, 85 insertions(+), 218 deletions(-)
> > > > > > >
> > > > > > > Have you all seen:
> > > > > > >
> > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A
> > > > > > > %2F%25
> > > > > > > 25
> > > > > > >
> > > > >
> > >
> 2Flore.kernel.org%2Fr%2F8c1443136563de34699d2c084df478181c205db4.16
> > > > > > >
> > > > >
> > >
> 03854416.git.sudeep.dutt%40intel.com&amp;data=04%7C01%7Csherry.sun%
> > > > > > >
> > > > >
> > >
> 40nxp.com%7Cc19c987667434969847e08d87b0685e8%7C686ea1d3bc2b4c6f
> > > > > > >
> > > > >
> > >
> a92cd99c5c301635%7C0%7C0%7C637394615238940323%7CUnknown%7CTW
> > > > > > >
> > > > >
> > >
> FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> > > > > > >
> > > > >
> > >
> VCI6Mn0%3D%7C1000&amp;sdata=Zq%2FtHWTq%2BuIVBYXFGoeBmq0JJzYd
> > > > > > > 9zDyv4NVN4TpC%2FU%3D&amp;reserved=0
> > > > > > >
> > > > > > > Looks like this code is asking to just be deleted, is that ok with you?
> > > > > >
> > > > > > Yes, I saw that patch. I'm ok with it.
> > > > >
> > > > > Great, can you please provide a "Reviewed-by:" or "Acked-by:" for it?
> > > > >
> > > > > thanks,
> > > > >
> > > > > greg k-h
> > > >
> > > > Sherry took much effort on the features support on i.MX series
> > > > like
> > > i.MX8QM/i.MX8QXP/i.MX8MM.
> > > >
> > > > Now it is a pity to delete the vop code.
> > > >
> > > > One question,
> > > > can we resubmit vop code by clean up, now only for i.MX series as
> > > > Dutt's
> > > suggestion ?
> > > > Or we have to drop the design and switch to select other solutions ?
> > >
> >
> > Okay, we plan to switch to NTB solution.
>
> What is a "NTB solution" exactly?

The driver located at drivers/ntb/, it also can setup a point-to-point PCI-E bus connecting between two systems.
But we haven't got a deep look of this driver yet, so we are not sure whether it can replace the vop framework.

>
> >
> > > If this whole subsystem is being deleted because it is not used and
> > > never shipped, yes, please use a different solution.
> > >
> > > I don't understand why you were trying to piggy-back on this
> > > codebase if the hardware was totally different, for some reason I
> > > thought this was the same hardware. What exactly is this?
> >
> > Not the whole codebase, just the vop framework.
>
> That didn't answer the question at all, what are you all trying to do here, with
> what hardware, that the VOP code seemed like a good fit?

Vop is a common framework which is independent of the Intel MIC hardware.
We planed to reuse vop framework on two arm64 architecture devices, to setup the connection between two systems based on virtio over PCIE.

Best regards
Sherry

>
> thanks,
>
> greg k-h

2020-10-29 08:51:23

by Dutt, Sudeep

[permalink] [raw]
Subject: Re: [EXT] Re: [PATCH V5 0/2] Change vring space from nomal memory to dma coherent memory

On Thu, 2020-10-29 at 01:51 +0000, Sherry Sun wrote:
> Hi Greg,
>
> > Subject: Re: [EXT] Re: [PATCH V5 0/2] Change vring space from nomal
> > memory to dma coherent memory
> >
> > On Wed, Oct 28, 2020 at 03:11:15PM +0000, Andy Duan wrote:
> > > From: Greg KH <[email protected]> Sent: Wednesday,
> > > October
> > > 28, 2020 7:14 PM
> > > > On Wed, Oct 28, 2020 at 10:17:39AM +0000, Andy Duan wrote:
> > > > > From: Greg KH <[email protected]> Sent: Wednesday,
> > > > > October 28, 2020 3:07 PM
> > > > > > On Wed, Oct 28, 2020 at 06:05:28AM +0000, Sherry Sun wrote:
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > > Subject: Re: [PATCH V5 0/2] Change vring space from
> > > > > > > > nomal
> > > > > > > > memory to dma coherent memory
> > > > > > > >
> > > > > > > > On Wed, Oct 28, 2020 at 10:03:03AM +0800, Sherry Sun
> > > > > > > > wrote:
> > > > > > > > > Changes in V5:
> > > > > > > > > 1. Reorganize the vop_mmap function code in patch 1,
> > > > > > > > > which
> > > > > > > > > is done by
> > > > > > > >
> > > > > > > > Christoph.
> > > > > > > > > 2. Completely remove the unnecessary code related to
> > > > > > > > > reassign the used ring for card in patch 2.
> > > > > > > > >
> > > > > > > > > The original vop driver only supports dma coherent
> > > > > > > > > device,
> > > > > > > > > as it allocates and maps vring by _get_free_pages and
> > > > > > > > > dma_map_single, but not use
> > > > > > > > > dma_sync_single_for_cpu/device
> > > > > > > > > to sync the updates of device_page/vring between EP
> > > > > > > > > and
> > > > > > > > > RC, which will cause memory synchronization problem
> > > > > > > > > for
> > > > > > > > > device don't support
> > > >
> > > > hardware dma coherent.
> > > > > > > > >
> > > > > > > > > And allocate vrings use dma_alloc_coherent is a
> > > > > > > > > common way
> > > > > > > > > in kernel, as the memory interacted between two
> > > > > > > > > systems
> > > > > > > > > should use consistent memory to avoid caching
> > > > > > > > > effects. So
> > > > > > > > > here add noncoherent platform
> > > > > > > >
> > > > > > > > support for vop driver.
> > > > > > > > > Also add some related dma changes to make sure
> > > > > > > > > noncoherent
> > > > > > > > > platform works well.
> > > > > > > > >
> > > > > > > > > Sherry Sun (2):
> > > > > > > > > misc: vop: change the way of allocating vrings and
> > > > > > > > > device page
> > > > > > > > > misc: vop: do not allocate and reassign the used
> > > > > > > > > ring
> > > > > > > > >
> > > > > > > > > drivers/misc/mic/bus/vop_bus.h | 2 +
> > > > > > > > > drivers/misc/mic/host/mic_boot.c | 9 ++
> > > > > > > > > drivers/misc/mic/host/mic_main.c | 43 ++------
> > > > > > > > > drivers/misc/mic/vop/vop_debugfs.c | 4 -
> > > > > > > > > drivers/misc/mic/vop/vop_main.c | 70 +--------
> > > > > > > > > ---
> > > > > > > > > drivers/misc/mic/vop/vop_vringh.c | 166 ++++++++++-
> > > > > > > > > ------------
> >
> > ------
> > > > > > > > > include/uapi/linux/mic_common.h | 9 +-
> > > > > > > > > 7 files changed, 85 insertions(+), 218 deletions(-)
> > > > > > > >
> > > > > > > > Have you all seen:
> > > > > > > >
> > > > > > > >
https://eur01.safelinks.protection.outlook.com/?url=https%3A
> > > > > > > > %2F%25
> > > > > > > > 25
> > > > > > > >
> >
> > 2Flore.kernel.org%2Fr%2F8c1443136563de34699d2c084df478181c205db4.16
> > > > > > > >
> >
> > 03854416.git.sudeep.dutt%40intel.com&amp;data=04%7C01%7Csherry.sun%
> > > > > > > >
> >
> > 40nxp.com%7Cc19c987667434969847e08d87b0685e8%7C686ea1d3bc2b4c6f
> > > > > > > >
> >
> > a92cd99c5c301635%7C0%7C0%7C637394615238940323%7CUnknown%7CTW
> > > > > > > >
> >
> > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> > > > > > > >
> >
> > VCI6Mn0%3D%7C1000&amp;sdata=Zq%2FtHWTq%2BuIVBYXFGoeBmq0JJzYd
> > > > > > > > 9zDyv4NVN4TpC%2FU%3D&amp;reserved=0
> > > > > > > >
> > > > > > > > Looks like this code is asking to just be deleted, is
> > > > > > > > that ok with you?
> > > > > > >
> > > > > > > Yes, I saw that patch. I'm ok with it.
> > > > > >
> > > > > > Great, can you please provide a "Reviewed-by:" or "Acked-
> > > > > > by:" for it?
> > > > > >
> > > > > > thanks,
> > > > > >
> > > > > > greg k-h
> > > > >
> > > > > Sherry took much effort on the features support on i.MX
> > > > > series
> > > > > like
> > > >
> > > > i.MX8QM/i.MX8QXP/i.MX8MM.
> > > > >
> > > > > Now it is a pity to delete the vop code.
> > > > >
> > > > > One question,
> > > > > can we resubmit vop code by clean up, now only for i.MX
> > > > > series as
> > > > > Dutt's
> > > >
> > > > suggestion ?

Resubmitting the VOP code with cleanups tailored for i.MX makes sense
to me.

> > > > > Or we have to drop the design and switch to select other
> > > > > solutions ?
> > >
> > > Okay, we plan to switch to NTB solution.
> >
> > What is a "NTB solution" exactly?
>
> The driver located at drivers/ntb/, it also can setup a point-to-
> point PCI-E bus connecting between two systems.
> But we haven't got a deep look of this driver yet, so we are not sure
> whether it can replace the vop framework.
>
> >
> > >
> > > > If this whole subsystem is being deleted because it is not used
> > > > and
> > > > never shipped, yes, please use a different solution.
> > > >
> > > > I don't understand why you were trying to piggy-back on this
> > > > codebase if the hardware was totally different, for some reason
> > > > I
> > > > thought this was the same hardware. What exactly is this?
> > >
> > > Not the whole codebase, just the vop framework.
> >
> > That didn't answer the question at all, what are you all trying to
> > do here, with
> > what hardware, that the VOP code seemed like a good fit?
>
> Vop is a common framework which is independent of the Intel MIC
> hardware.
> We planed to reuse vop framework on two arm64 architecture devices,
> to setup the connection between two systems based on virtio over
> PCIE.

Yes, we wanted Virtio Over PCIe (VOP) to be independent of the hardware
as much as possible. It did end up under the mic/ driver subsystem
though so it would be good to attempt placing it in a generic folder
which is not tied to a specific hardware layer this time around.

Regards,
Sudeep Dutt