Hi,
This series is the first 13 patches from my previous Userspace P2P
with the CMB series[1]. These patches just do the first part which is to
allow passing P2PDMA pages into dma_map_sg() and support it with both
dma-direct and dma-iommu. The series ends with dropping the old
pci_p2pdma_[un]map_sg() interface.
This will be followed up later with the rest of the series which has
more issues to work out.
This series is based on v5.19-rc5. A git branch is available here:
https://github.com/sbates130272/linux-p2pmem/ p2pdma_map_v8
Logan
[1] https://lkml.kernel.org/r/[email protected]
--
Changes since v7 (first half of the series only):
- Rebase onto v5.19-rc5
- Collect reviewed-by tags from Christoph
- Change the first patch to use a dma_flags field in the scatterlist
instead of a bit in the page link, per an idea from Robin
- Rework the iommu-dma implementation to store the bus addresses
in iommu_dma_map_sg() and only copy them in __finalise_sg. This
allows the code to use the same pci_p2pdma_map_segment() helper
that the dma-direct code uses and we can drop the two specialized
helpers. (Per Robin)
- Instead of using a zero length segment which __iommu_map_sg()
ignores, explicitly skip segments marked with as a dma bus address.
(per Robin)
- Rework the iommu_dma_unmap_sg() function per a suggestion from
Robin.
Changes since v6:
- Rebase onto v5.19-rc1
- Rework how the pages are stored in the VMA per Jason's suggestion
Changes since v5:
- Rebased onto v5.18-rc1 which includes Christophs cleanup to
free_zone_device_page() (similar to Ralph's patch).
- Fix bug with concurrent first calls to pci_p2pdma_vma_fault()
that caused a double allocation and lost p2p memory. Noticed
by Andrew Maier.
- Collected a Reviewed-by tag from Chaitanya.
- Numerous minor fixes to commit messages
Changes since v4:
- Rebase onto v5.17-rc1.
- Included Ralph Cambell's patches which removes the ZONE_DEVICE page
reference count offset. This is just to demonstrate that this
series is compatible with that direction.
- Added a comment in pci_p2pdma_map_sg_attrs(), per Chaitanya and
included his Reviewed-by tags.
- Patch 1 in the last series which cleaned up scatterlist.h
has been upstreamed.
- Dropped NEED_SG_DMA_BUS_ADDR_FLAG seeing depends on doesn't
work with selected symbols, per Christoph.
- Switched iov_iter_get_pages_[alloc_]flags to be exported with
EXPORT_SYMBOL_GPL, per Christoph.
- Renamed zone_device_pages_are_mergeable() to
zone_device_pages_have_same_pgmap(), per Christoph.
- Renamed .mmap_file_open operation in nvme_ctrl_ops to
cdev_file_open(), per Christoph.
Changes since v3:
- Add some comment and commit message cleanups I had missed for v3,
also moved the prototypes for some of the p2pdma helpers to
dma-map-ops.h (which I missed in v3 and was suggested in v2).
- Add separate cleanup patch for scatterlist.h and change the macros
to functions. (Suggested by Chaitanya and Jason, respectively)
- Rename sg_dma_mark_pci_p2pdma() and sg_is_dma_pci_p2pdma() to
sg_dma_mark_bus_address() and sg_is_dma_bus_address() which
is a more generic name (As requested by Jason)
- Fixes to some comments and commit messages as suggested by Bjorn
and Jason.
- Ensure swiotlb is not used with P2PDMA pages. (Per Jason)
- The sgtable coversion in RDMA was split out and sent upstream
separately, the new patch is only the removal. (Per Jason)
- Moved the FOLL_PCI_P2PDMA check outside of get_dev_pagemap() as
Jason suggested this will be removed in the near term.
- Add two patches to ensure that zone device pages with different
pgmaps are never merged in the block layer or
sg_alloc_append_table_from_pages() (Per Jason)
- Ensure synchronize_rcu() or call_rcu() is used before returning
pages to the genalloc. (Jason pointed out that pages are not
gauranteed to be unused in all architectures until at least
after an RCU grace period, and that synchronize_rcu() was likely
too slow to use in the vma close operation.
- Collected Acks and Reviews by Bjorn, Jason and Max.
--
Logan Gunthorpe (13):
lib/scatterlist: add flag for indicating P2PDMA segments in an SGL
PCI/P2PDMA: Attempt to set map_type if it has not been set
PCI/P2PDMA: Introduce helpers for dma_map_sg implementations
dma-mapping: allow EREMOTEIO return code for P2PDMA transfers
dma-direct: support PCI P2PDMA pages in dma-direct map_sg
dma-mapping: add flags to dma_map_ops to indicate PCI P2PDMA support
iommu: Explicitly skip bus address marked segments in
__iommu_map_sg()
iommu/dma: support PCI P2PDMA pages in dma-iommu map_sg
nvme-pci: check DMA ops when indicating support for PCI P2PDMA
nvme-pci: convert to using dma_map_sgtable()
RDMA/core: introduce ib_dma_pci_p2p_dma_supported()
RDMA/rw: drop pci_p2pdma_[un]map_sg()
PCI/P2PDMA: Remove pci_p2pdma_[un]map_sg()
drivers/infiniband/core/rw.c | 45 ++++------------
drivers/iommu/dma-iommu.c | 99 +++++++++++++++++++++++++++++++-----
drivers/iommu/iommu.c | 4 ++
drivers/nvme/host/core.c | 3 +-
drivers/nvme/host/nvme.h | 2 +-
drivers/nvme/host/pci.c | 81 ++++++++++++++---------------
drivers/nvme/target/rdma.c | 2 +-
drivers/pci/Kconfig | 5 ++
drivers/pci/p2pdma.c | 93 +++++++++++----------------------
include/linux/dma-map-ops.h | 63 +++++++++++++++++++++++
include/linux/dma-mapping.h | 5 ++
include/linux/pci-p2pdma.h | 27 ----------
include/linux/scatterlist.h | 69 +++++++++++++++++++++++++
include/rdma/ib_verbs.h | 11 ++++
kernel/dma/direct.c | 43 +++++++++++++---
kernel/dma/direct.h | 8 ++-
kernel/dma/mapping.c | 23 ++++++++-
17 files changed, 390 insertions(+), 193 deletions(-)
base-commit: 88084a3df1672e131ddc1b4e39eeacfd39864acf
--
2.30.2
Introduce a supports_pci_p2pdma() operation in nvme_ctrl_ops to
replace the fixed NVME_F_PCI_P2PDMA flag such that the dma_map_ops
flags can be checked for PCI P2PDMA support.
Signed-off-by: Logan Gunthorpe <[email protected]>
Reviewed-by: Chaitanya Kulkarni <[email protected]>
Reviewed-by: Christoph Hellwig <[email protected]>
---
drivers/nvme/host/core.c | 3 ++-
drivers/nvme/host/nvme.h | 2 +-
drivers/nvme/host/pci.c | 12 +++++++++---
3 files changed, 12 insertions(+), 5 deletions(-)
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index ec6ac298d8de..2831e248dd71 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3996,7 +3996,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid,
blk_queue_flag_set(QUEUE_FLAG_STABLE_WRITES, ns->queue);
blk_queue_flag_set(QUEUE_FLAG_NONROT, ns->queue);
- if (ctrl->ops->flags & NVME_F_PCI_P2PDMA)
+ if (ctrl->ops->supports_pci_p2pdma &&
+ ctrl->ops->supports_pci_p2pdma(ctrl))
blk_queue_flag_set(QUEUE_FLAG_PCI_P2PDMA, ns->queue);
ns->ctrl = ctrl;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 5558f8812157..ea097bb1edde 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -495,7 +495,6 @@ struct nvme_ctrl_ops {
unsigned int flags;
#define NVME_F_FABRICS (1 << 0)
#define NVME_F_METADATA_SUPPORTED (1 << 1)
-#define NVME_F_PCI_P2PDMA (1 << 2)
int (*reg_read32)(struct nvme_ctrl *ctrl, u32 off, u32 *val);
int (*reg_write32)(struct nvme_ctrl *ctrl, u32 off, u32 val);
int (*reg_read64)(struct nvme_ctrl *ctrl, u32 off, u64 *val);
@@ -505,6 +504,7 @@ struct nvme_ctrl_ops {
void (*stop_ctrl)(struct nvme_ctrl *ctrl);
int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size);
void (*print_device_info)(struct nvme_ctrl *ctrl);
+ bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl);
};
/*
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index e7af2234e53b..0f556e954ffc 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2984,7 +2984,6 @@ static int nvme_pci_get_address(struct nvme_ctrl *ctrl, char *buf, int size)
return snprintf(buf, size, "%s\n", dev_name(&pdev->dev));
}
-
static void nvme_pci_print_device_info(struct nvme_ctrl *ctrl)
{
struct pci_dev *pdev = to_pci_dev(to_nvme_dev(ctrl)->dev);
@@ -2999,11 +2998,17 @@ static void nvme_pci_print_device_info(struct nvme_ctrl *ctrl)
subsys->firmware_rev);
}
+static bool nvme_pci_supports_pci_p2pdma(struct nvme_ctrl *ctrl)
+{
+ struct nvme_dev *dev = to_nvme_dev(ctrl);
+
+ return dma_pci_p2pdma_supported(dev->dev);
+}
+
static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = {
.name = "pcie",
.module = THIS_MODULE,
- .flags = NVME_F_METADATA_SUPPORTED |
- NVME_F_PCI_P2PDMA,
+ .flags = NVME_F_METADATA_SUPPORTED,
.reg_read32 = nvme_pci_reg_read32,
.reg_write32 = nvme_pci_reg_write32,
.reg_read64 = nvme_pci_reg_read64,
@@ -3011,6 +3016,7 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = {
.submit_async_event = nvme_pci_submit_async_event,
.get_address = nvme_pci_get_address,
.print_device_info = nvme_pci_print_device_info,
+ .supports_pci_p2pdma = nvme_pci_supports_pci_p2pdma,
};
static int nvme_dev_map(struct nvme_dev *dev)
--
2.30.2
I'd like to hear from Robin on the dma-iommu parts, but otherwise
this looks fine to me modulo a little filed naming change I could
handle when applying this.
I've applied this to the dma-mapping for-next branch.
I'd have liked a review from Robin, but as he seems to be out and the
rework was to fit his comments I'd rather have us unblocked before
the end of the merge window.
On 2022-07-26 05:29, Christoph Hellwig wrote:
> I've applied this to the dma-mapping for-next branch.
>
> I'd have liked a review from Robin, but as he seems to be out and the
> rework was to fit his comments I'd rather have us unblocked before
> the end of the merge window.
Great thanks! If Robin has some feedback when he gets back I'd be happy
to make a couple more cleanup patches.
Logan
On 2022-07-26 16:55, Logan Gunthorpe wrote:
>
>
> On 2022-07-26 05:29, Christoph Hellwig wrote:
>> I've applied this to the dma-mapping for-next branch.
>>
>> I'd have liked a review from Robin, but as he seems to be out and the
>> rework was to fit his comments I'd rather have us unblocked before
>> the end of the merge window.
>
> Great thanks! If Robin has some feedback when he gets back I'd be happy
> to make a couple more cleanup patches.
Indeed I've been taking some (rather overdue) holiday the last couple of
weeks, sorry I forgot to take a look at this version beforehand. From a
quick skim, nothing stands out as objectionable; thanks for taking my
suggestions on board!
At some point once this lands I'll have a go at using the new flags to
improve iommu-dma bounce buffering, so if I do spot anything else while
I'm in there I'm sure could clean it up myself.
Cheers,
Robin.