During bnx2i iSCSI testing we ran into page refcounting issues in the
uio mmaps exported from cnic to the iscsiuio process, and bisected back
to the removal of the __GFP_COMP flag from dma_alloc_coherent calls.
The cnic uio interface also has issues running with an iommu enabled,
which these changes correct.
In order to fix these drivers to be able to mmap dma coherent memory via
a uio device, introduce a new uio mmap type backed by dma_mmap_coherent.
While I understand some complaints about how these drivers have been
structured, I also don't like letting support bitrot when there's a
reasonable alternative to re-architecting an existing driver. I believe
this to be the most sane way to restore these drivers to functioning
properly.
There are two other uio drivers which are mmaping dma_alloc_coherent
memory as UIO_MEM_PHYS, uio_dmem_genirq and uio_pruss.
These drivers are converted in the later patches of this series.
v5:
- convert uio_pruss and uio_dmem_genirq
- added dev_warn and comment about not adding more users
- put some PAGE_ALIGNs back in cnic to keep checks in
uio_mmap_dma_coherent matched with uio_mmap_physical.
- dropped the Fixes trailer
v4:
- re-introduce the dma_device member to uio_map,
it needs to be passed to dma_mmap_coherent somehow
- drop patch 3 to focus only on the uio interface,
explicit page alignment isn't needed
- re-add the v1 mail recipients,
this isn't something to be handled through linux-scsi
v3 (Nilesh Javali <[email protected]>):
- fix warnings reported by kernel test robot
and added base commit
v2 (Nilesh Javali <[email protected]>):
- expose only the dma_addr within uio and cnic.
- Cleanup newly added unions comprising virtual_addr
and struct device
previous threads:
v1: https://lore.kernel.org/all/[email protected]/
attempt at an alternative change: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
Chris Leech (4):
uio: introduce UIO_MEM_DMA_COHERENT type
cnic,bnx2,bnx2x: use UIO_MEM_DMA_COHERENT
uio_pruss: UIO_MEM_DMA_COHERENT conversion
uio_dmem_genirq: UIO_MEM_DMA_COHERENT conversion
drivers/net/ethernet/broadcom/bnx2.c | 1 +
.../net/ethernet/broadcom/bnx2x/bnx2x_main.c | 2 +
drivers/net/ethernet/broadcom/cnic.c | 25 ++++++----
drivers/net/ethernet/broadcom/cnic.h | 1 +
drivers/net/ethernet/broadcom/cnic_if.h | 1 +
drivers/uio/uio.c | 47 +++++++++++++++++++
drivers/uio/uio_dmem_genirq.c | 22 ++++-----
drivers/uio/uio_pruss.c | 6 ++-
include/linux/uio_driver.h | 8 ++++
9 files changed, 89 insertions(+), 24 deletions(-)
base-commit: 861c0981648f5b64c86fd028ee622096eb7af05a
--
2.43.0
Conversion of this driver to use UIO_MEM_DMA_COHERENT for
dma_alloc_coherent memory instead of UIO_MEM_PHYS.
Signed-off-by: Chris Leech <[email protected]>
---
drivers/uio/uio_dmem_genirq.c | 22 ++++++++--------------
1 file changed, 8 insertions(+), 14 deletions(-)
diff --git a/drivers/uio/uio_dmem_genirq.c b/drivers/uio/uio_dmem_genirq.c
index 5313307c2754a..8634eba0ef2ab 100644
--- a/drivers/uio/uio_dmem_genirq.c
+++ b/drivers/uio/uio_dmem_genirq.c
@@ -36,7 +36,6 @@ struct uio_dmem_genirq_platdata {
struct platform_device *pdev;
unsigned int dmem_region_start;
unsigned int num_dmem_regions;
- void *dmem_region_vaddr[MAX_UIO_MAPS];
struct mutex alloc_lock;
unsigned int refcnt;
};
@@ -50,7 +49,6 @@ static int uio_dmem_genirq_open(struct uio_info *info, struct inode *inode)
{
struct uio_dmem_genirq_platdata *priv = info->priv;
struct uio_mem *uiomem;
- int dmem_region = priv->dmem_region_start;
uiomem = &priv->uioinfo->mem[priv->dmem_region_start];
@@ -61,11 +59,8 @@ static int uio_dmem_genirq_open(struct uio_info *info, struct inode *inode)
break;
addr = dma_alloc_coherent(&priv->pdev->dev, uiomem->size,
- (dma_addr_t *)&uiomem->addr, GFP_KERNEL);
- if (!addr) {
- uiomem->addr = DMEM_MAP_ERROR;
- }
- priv->dmem_region_vaddr[dmem_region++] = addr;
+ &uiomem->dma_addr, GFP_KERNEL);
+ uiomem->addr = addr ? (phys_addr_t) addr : DMEM_MAP_ERROR;
++uiomem;
}
priv->refcnt++;
@@ -80,7 +75,6 @@ static int uio_dmem_genirq_release(struct uio_info *info, struct inode *inode)
{
struct uio_dmem_genirq_platdata *priv = info->priv;
struct uio_mem *uiomem;
- int dmem_region = priv->dmem_region_start;
/* Tell the Runtime PM code that the device has become idle */
pm_runtime_put_sync(&priv->pdev->dev);
@@ -93,13 +87,12 @@ static int uio_dmem_genirq_release(struct uio_info *info, struct inode *inode)
while (!priv->refcnt && uiomem < &priv->uioinfo->mem[MAX_UIO_MAPS]) {
if (!uiomem->size)
break;
- if (priv->dmem_region_vaddr[dmem_region]) {
- dma_free_coherent(&priv->pdev->dev, uiomem->size,
- priv->dmem_region_vaddr[dmem_region],
- uiomem->addr);
+ if (uiomem->addr) {
+ dma_free_coherent(uiomem->dma_device, uiomem->size,
+ (void *) uiomem->addr,
+ uiomem->dma_addr);
}
uiomem->addr = DMEM_MAP_ERROR;
- ++dmem_region;
++uiomem;
}
@@ -264,7 +257,8 @@ static int uio_dmem_genirq_probe(struct platform_device *pdev)
" dynamic and fixed memory regions.\n");
break;
}
- uiomem->memtype = UIO_MEM_PHYS;
+ uiomem->memtype = UIO_MEM_DMA_COHERENT;
+ uiomem->dma_device = &pdev->dev,
uiomem->addr = DMEM_MAP_ERROR;
uiomem->size = pdata->dynamic_region_sizes[i];
++uiomem;
--
2.43.0
On Thu, Feb 01, 2024 at 03:34:00PM -0800, Chris Leech wrote:
> Conversion of this driver to use UIO_MEM_DMA_COHERENT for
> dma_alloc_coherent memory instead of UIO_MEM_PHYS.
>
> Signed-off-by: Chris Leech <[email protected]>
> ---
> drivers/uio/uio_dmem_genirq.c | 22 ++++++++--------------
> 1 file changed, 8 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/uio/uio_dmem_genirq.c b/drivers/uio/uio_dmem_genirq.c
..
> @@ -264,7 +257,8 @@ static int uio_dmem_genirq_probe(struct platform_device *pdev)
> " dynamic and fixed memory regions.\n");
> break;
> }
> - uiomem->memtype = UIO_MEM_PHYS;
> + uiomem->memtype = UIO_MEM_DMA_COHERENT;
> + uiomem->dma_device = &pdev->dev,
Hi Chris,
a nit from my side.
Probably the ',' would be better written as a ';' here.
I don't think this is a bug, but using comma like this is
somewhat unexpected and confusing.
Flagged by clang-17 with -Wcomma
> uiomem->addr = DMEM_MAP_ERROR;
> uiomem->size = pdata->dynamic_region_sizes[i];
> ++uiomem;
> --
> 2.43.0
>
>
From: Chris Leech <[email protected]>
Date: Thu, 1 Feb 2024 15:33:56 -0800
> During bnx2i iSCSI testing we ran into page refcounting issues in the
> uio mmaps exported from cnic to the iscsiuio process, and bisected back
> to the removal of the __GFP_COMP flag from dma_alloc_coherent calls.
IIRC Jakub mentioned some time ago that he doesn't want to see
third-party userspace <-> kernel space communication in the networking
drivers, to me this looks exactly like that :z
Thanks,
Olek
On Mon, Feb 05, 2024 at 05:57:58PM +0100, Alexander Lobakin wrote:
> From: Chris Leech <[email protected]>
> Date: Thu, 1 Feb 2024 15:33:56 -0800
>
> > During bnx2i iSCSI testing we ran into page refcounting issues in the
> > uio mmaps exported from cnic to the iscsiuio process, and bisected back
> > to the removal of the __GFP_COMP flag from dma_alloc_coherent calls.
>
> IIRC Jakub mentioned some time ago that he doesn't want to see
> third-party userspace <-> kernel space communication in the networking
> drivers, to me this looks exactly like that :z
This isn't something anyone likes, but it's an interface that's been in
the kernel and in use since 2009. I'm trying to see if it can be fixed
"enough" to keep existing users functioning. If not, maybe the cnic
interface and the stacking protocol drivers (bnx2i/bnx2fc) should be
marked as broken.
- Chris
On Sun, Feb 04, 2024 at 10:19:03AM +0000, Simon Horman wrote:
> On Thu, Feb 01, 2024 at 03:34:00PM -0800, Chris Leech wrote:
..
> > @@ -264,7 +257,8 @@ static int uio_dmem_genirq_probe(struct platform_device *pdev)
> > " dynamic and fixed memory regions.\n");
> > break;
> > }
> > - uiomem->memtype = UIO_MEM_PHYS;
> > + uiomem->memtype = UIO_MEM_DMA_COHERENT;
> > + uiomem->dma_device = &pdev->dev,
>
> Hi Chris,
>
> a nit from my side.
>
> Probably the ',' would be better written as a ';' here.
> I don't think this is a bug, but using comma like this is
> somewhat unexpected and confusing.
>
> Flagged by clang-17 with -Wcomma
That's an embarrassing typo to slip through.
I'll fix this,and add the kdoc comments for the API additions.
Thanks,
- Chris
Conversion of this driver to use UIO_MEM_DMA_COHERENT for
dma_alloc_coherent memory instead of UIO_MEM_PHYS.
Signed-off-by: Chris Leech <[email protected]>
---
v6: fixed single char ',' -> ';' typo
drivers/uio/uio_dmem_genirq.c | 22 ++++++++--------------
1 file changed, 8 insertions(+), 14 deletions(-)
diff --git a/drivers/uio/uio_dmem_genirq.c b/drivers/uio/uio_dmem_genirq.c
index 5313307c2754a..d5f9384df1255 100644
--- a/drivers/uio/uio_dmem_genirq.c
+++ b/drivers/uio/uio_dmem_genirq.c
@@ -36,7 +36,6 @@ struct uio_dmem_genirq_platdata {
struct platform_device *pdev;
unsigned int dmem_region_start;
unsigned int num_dmem_regions;
- void *dmem_region_vaddr[MAX_UIO_MAPS];
struct mutex alloc_lock;
unsigned int refcnt;
};
@@ -50,7 +49,6 @@ static int uio_dmem_genirq_open(struct uio_info *info, struct inode *inode)
{
struct uio_dmem_genirq_platdata *priv = info->priv;
struct uio_mem *uiomem;
- int dmem_region = priv->dmem_region_start;
uiomem = &priv->uioinfo->mem[priv->dmem_region_start];
@@ -61,11 +59,8 @@ static int uio_dmem_genirq_open(struct uio_info *info, struct inode *inode)
break;
addr = dma_alloc_coherent(&priv->pdev->dev, uiomem->size,
- (dma_addr_t *)&uiomem->addr, GFP_KERNEL);
- if (!addr) {
- uiomem->addr = DMEM_MAP_ERROR;
- }
- priv->dmem_region_vaddr[dmem_region++] = addr;
+ &uiomem->dma_addr, GFP_KERNEL);
+ uiomem->addr = addr ? (phys_addr_t) addr : DMEM_MAP_ERROR;
++uiomem;
}
priv->refcnt++;
@@ -80,7 +75,6 @@ static int uio_dmem_genirq_release(struct uio_info *info, struct inode *inode)
{
struct uio_dmem_genirq_platdata *priv = info->priv;
struct uio_mem *uiomem;
- int dmem_region = priv->dmem_region_start;
/* Tell the Runtime PM code that the device has become idle */
pm_runtime_put_sync(&priv->pdev->dev);
@@ -93,13 +87,12 @@ static int uio_dmem_genirq_release(struct uio_info *info, struct inode *inode)
while (!priv->refcnt && uiomem < &priv->uioinfo->mem[MAX_UIO_MAPS]) {
if (!uiomem->size)
break;
- if (priv->dmem_region_vaddr[dmem_region]) {
- dma_free_coherent(&priv->pdev->dev, uiomem->size,
- priv->dmem_region_vaddr[dmem_region],
- uiomem->addr);
+ if (uiomem->addr) {
+ dma_free_coherent(uiomem->dma_device, uiomem->size,
+ (void *) uiomem->addr,
+ uiomem->dma_addr);
}
uiomem->addr = DMEM_MAP_ERROR;
- ++dmem_region;
++uiomem;
}
@@ -264,7 +257,8 @@ static int uio_dmem_genirq_probe(struct platform_device *pdev)
" dynamic and fixed memory regions.\n");
break;
}
- uiomem->memtype = UIO_MEM_PHYS;
+ uiomem->memtype = UIO_MEM_DMA_COHERENT;
+ uiomem->dma_device = &pdev->dev;
uiomem->addr = DMEM_MAP_ERROR;
uiomem->size = pdata->dynamic_region_sizes[i];
++uiomem;
--
2.43.0
On Mon, 5 Feb 2024 11:51:00 -0800 Chris Leech wrote:
> > IIRC Jakub mentioned some time ago that he doesn't want to see
> > third-party userspace <-> kernel space communication in the networking
> > drivers, to me this looks exactly like that :z
>
> This isn't something anyone likes, but it's an interface that's been in
> the kernel and in use since 2009. I'm trying to see if it can be fixed
> "enough" to keep existing users functioning. If not, maybe the cnic
> interface and the stacking protocol drivers (bnx2i/bnx2fc) should be
> marked as broken.
Yeah, is this one of the "converged Ethernet" monstrosities from
the 2000s. All the companies which went deep into this stuff are
now defunct AFAIK, and we're left holding the bag.
Yay.
On Tue, Feb 6, 2024 at 7:54 AM Jakub Kicinski <[email protected]> wrote:
>
> On Mon, 5 Feb 2024 11:51:00 -0800 Chris Leech wrote:
> > > IIRC Jakub mentioned some time ago that he doesn't want to see
> > > third-party userspace <-> kernel space communication in the networking
> > > drivers, to me this looks exactly like that :z
> >
> > This isn't something anyone likes, but it's an interface that's been in
> > the kernel and in use since 2009. I'm trying to see if it can be fixed
> > "enough" to keep existing users functioning. If not, maybe the cnic
> > interface and the stacking protocol drivers (bnx2i/bnx2fc) should be
> > marked as broken.
>
> Yeah, is this one of the "converged Ethernet" monstrosities from
> the 2000s. All the companies which went deep into this stuff are
> now defunct AFAIK, and we're left holding the bag.
>
> Yay.
Actually, Marvel is around but seems loath to invest in re-architecting this
driver since it's so long in the tooth.
I think all the feedback on these has been addressed, so I'm asking
once more if these UIO additions can be considered for inclusion.
Thanks,
- Chris
On Thu, Feb 1, 2024 at 3:34 PM Chris Leech <[email protected]> wrote:
>
> During bnx2i iSCSI testing we ran into page refcounting issues in the
> uio mmaps exported from cnic to the iscsiuio process, and bisected back
> to the removal of the __GFP_COMP flag from dma_alloc_coherent calls.
>
> The cnic uio interface also has issues running with an iommu enabled,
> which these changes correct.
>
> In order to fix these drivers to be able to mmap dma coherent memory via
> a uio device, introduce a new uio mmap type backed by dma_mmap_coherent.
>
> While I understand some complaints about how these drivers have been
> structured, I also don't like letting support bitrot when there's a
> reasonable alternative to re-architecting an existing driver. I believe
> this to be the most sane way to restore these drivers to functioning
> properly.
>
> There are two other uio drivers which are mmaping dma_alloc_coherent
> memory as UIO_MEM_PHYS, uio_dmem_genirq and uio_pruss.
> These drivers are converted in the later patches of this series.
>
> v5:
> - convert uio_pruss and uio_dmem_genirq
> - added dev_warn and comment about not adding more users
> - put some PAGE_ALIGNs back in cnic to keep checks in
> uio_mmap_dma_coherent matched with uio_mmap_physical.
> - dropped the Fixes trailer
> v4:
> - re-introduce the dma_device member to uio_map,
> it needs to be passed to dma_mmap_coherent somehow
> - drop patch 3 to focus only on the uio interface,
> explicit page alignment isn't needed
> - re-add the v1 mail recipients,
> this isn't something to be handled through linux-scsi
> v3 (Nilesh Javali <[email protected]>):
> - fix warnings reported by kernel test robot
> and added base commit
> v2 (Nilesh Javali <[email protected]>):
> - expose only the dma_addr within uio and cnic.
> - Cleanup newly added unions comprising virtual_addr
> and struct device
>
> previous threads:
> v1: https://lore.kernel.org/all/[email protected]/
> attempt at an alternative change: https://lore.kernel.org/all/[email protected]/
> v2: https://lore.kernel.org/all/[email protected]/
> v3: https://lore.kernel.org/all/[email protected]/
> v4: https://lore.kernel.org/all/[email protected]/
>
> Chris Leech (4):
> uio: introduce UIO_MEM_DMA_COHERENT type
> cnic,bnx2,bnx2x: use UIO_MEM_DMA_COHERENT
> uio_pruss: UIO_MEM_DMA_COHERENT conversion
> uio_dmem_genirq: UIO_MEM_DMA_COHERENT conversion
>
> drivers/net/ethernet/broadcom/bnx2.c | 1 +
> .../net/ethernet/broadcom/bnx2x/bnx2x_main.c | 2 +
> drivers/net/ethernet/broadcom/cnic.c | 25 ++++++----
> drivers/net/ethernet/broadcom/cnic.h | 1 +
> drivers/net/ethernet/broadcom/cnic_if.h | 1 +
> drivers/uio/uio.c | 47 +++++++++++++++++++
> drivers/uio/uio_dmem_genirq.c | 22 ++++-----
> drivers/uio/uio_pruss.c | 6 ++-
> include/linux/uio_driver.h | 8 ++++
> 9 files changed, 89 insertions(+), 24 deletions(-)
>
>
> base-commit: 861c0981648f5b64c86fd028ee622096eb7af05a
> --
> 2.43.0
>
Is this series stalled?
I believe the main objections came from Greg earlier in the series,
but I'd gotten the impression Greg accepted the latest version.
On Thu, Feb 1, 2024 at 3:34 PM Chris Leech <[email protected]> wrote:
>
> During bnx2i iSCSI testing we ran into page refcounting issues in the
> uio mmaps exported from cnic to the iscsiuio process, and bisected back
> to the removal of the __GFP_COMP flag from dma_alloc_coherent calls.
>
> The cnic uio interface also has issues running with an iommu enabled,
> which these changes correct.
>
> In order to fix these drivers to be able to mmap dma coherent memory via
> a uio device, introduce a new uio mmap type backed by dma_mmap_coherent.
>
> While I understand some complaints about how these drivers have been
> structured, I also don't like letting support bitrot when there's a
> reasonable alternative to re-architecting an existing driver. I believe
> this to be the most sane way to restore these drivers to functioning
> properly.
>
> There are two other uio drivers which are mmaping dma_alloc_coherent
> memory as UIO_MEM_PHYS, uio_dmem_genirq and uio_pruss.
> These drivers are converted in the later patches of this series.
>
> v5:
> - convert uio_pruss and uio_dmem_genirq
> - added dev_warn and comment about not adding more users
> - put some PAGE_ALIGNs back in cnic to keep checks in
> uio_mmap_dma_coherent matched with uio_mmap_physical.
> - dropped the Fixes trailer
> v4:
> - re-introduce the dma_device member to uio_map,
> it needs to be passed to dma_mmap_coherent somehow
> - drop patch 3 to focus only on the uio interface,
> explicit page alignment isn't needed
> - re-add the v1 mail recipients,
> this isn't something to be handled through linux-scsi
> v3 (Nilesh Javali <[email protected]>):
> - fix warnings reported by kernel test robot
> and added base commit
> v2 (Nilesh Javali <[email protected]>):
> - expose only the dma_addr within uio and cnic.
> - Cleanup newly added unions comprising virtual_addr
> and struct device
>
> previous threads:
> v1: https://lore.kernel.org/all/[email protected]/
> attempt at an alternative change: https://lore.kernel.org/all/[email protected]/
> v2: https://lore.kernel.org/all/[email protected]/
> v3: https://lore.kernel.org/all/[email protected]/
> v4: https://lore.kernel.org/all/[email protected]/
>
> Chris Leech (4):
> uio: introduce UIO_MEM_DMA_COHERENT type
> cnic,bnx2,bnx2x: use UIO_MEM_DMA_COHERENT
> uio_pruss: UIO_MEM_DMA_COHERENT conversion
> uio_dmem_genirq: UIO_MEM_DMA_COHERENT conversion
>
> drivers/net/ethernet/broadcom/bnx2.c | 1 +
> .../net/ethernet/broadcom/bnx2x/bnx2x_main.c | 2 +
> drivers/net/ethernet/broadcom/cnic.c | 25 ++++++----
> drivers/net/ethernet/broadcom/cnic.h | 1 +
> drivers/net/ethernet/broadcom/cnic_if.h | 1 +
> drivers/uio/uio.c | 47 +++++++++++++++++++
> drivers/uio/uio_dmem_genirq.c | 22 ++++-----
> drivers/uio/uio_pruss.c | 6 ++-
> include/linux/uio_driver.h | 8 ++++
> 9 files changed, 89 insertions(+), 24 deletions(-)
>
>
> base-commit: 861c0981648f5b64c86fd028ee622096eb7af05a
> --
> 2.43.0
>
On Wed, Feb 28, 2024 at 10:20:58AM -0800, Lee Duncan wrote:
> Is this series stalled?
>
> I believe the main objections came from Greg earlier in the series,
> but I'd gotten the impression Greg accepted the latest version.
I'm behind in patch review, should get to it in the next few days...
thanks,
greg k-h