Received-SPF: pass (google.com: domain of linux-kernel+bounces-203893-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: Zhu Yanjun <zyjzyj2000@gmail.com>
Message-ID: <285e6cdb-f765-4969-ab16-abc53ad66a95@linux.dev>
Date: Thu, 6 Jun 2024 10:53:52 +0200
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 0/6] Enable P2PDMA in Userspace RDMA
To: Martin Oliveira <martin.oliveira@eideticom.com>,
 linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org,
 linux-pci@vger.kernel.org, linux-mm@kvack.org
Cc: Jason Gunthorpe <jgg@ziepe.ca>, Leon Romanovsky <leon@kernel.org>,
 Bjorn Helgaas <bhelgaas@google.com>, Logan Gunthorpe <logang@deltatee.com>,
 Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Tejun Heo <tj@kernel.org>,
 "Rafael J. Wysocki" <rafael@kernel.org>,
 Andrew Morton <akpm@linux-foundation.org>,
 Mike Marciniszyn <mike.marciniszyn@intel.com>,
 Michael Guralnik <michaelgur@nvidia.com>,
 Dan Williams <dan.j.williams@intel.com>, Ard Biesheuvel <ardb@kernel.org>,
 Valentine Sinitsyn <valesini@yandex-team.ru>, Lukas Wunner <lukas@wunner.de>
References: <20240605192934.742369-1-martin.oliveira@eideticom.com>
Content-Language: en-US
In-Reply-To: <20240605192934.742369-1-martin.oliveira@eideticom.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 05.06.24 21:29, Martin Oliveira wrote:
> This patch series enables P2PDMA memory to be used in userspace RDMA
> transfers. With this series, P2PDMA memory mmaped into userspace (ie.
> only NVMe CMBs, at the moment) can then be used with ibv_reg_mr() (or
> similar) interfaces. This can be tested by passing a sysfs p2pmem
> allocator to the --mmap flag of the perftest tools.

Do you mean the following --mmap flag?
"
--mmap=file  Use an mmap'd file as the buffer for testing P2P transfers.
"
I am interested in this. Can you provide the full steps to make tests 
with this patch series?

Thanks a lot.
Zhu Yanjun

> 
> This requires addressing three issues:
> 
> * Stop exporting the P2PDMA VMAs with page_mkwrite which is incompatible
> with FOLL_LONGTERM
> 
> * Fix folio_fast_pin_allowed() path to take into account ZONE_DEVICE pages.
> 
> * Remove the restriction on FOLL_LONGTREM with FOLL_PCI_P2PDMA which was
> initially put in place due to excessive caution with assuming P2PDMA
> would have similar problems to fsdax with unmap_mapping_range(). Seeing
> P2PDMA only uses unmap_mapping_range() on device unbind and immediately
> waits for all page reference counts to go to zero after calling it, it
> is actually believed to be safe from reuse and user access faults. See
> [1] for more discussion.
> 
> This was tested using a Mellanox ConnectX-6 SmartNIC (MT28908 Family),
> using the mlx5_core driver, as well as an NVMe CMB.
> 
> Thanks,
> Martin
> 
> [1]: https://lore.kernel.org/linux-mm/87cypuvh2i.fsf@nvdebian.thelocal/T/
> 
> Martin Oliveira (6):
>    kernfs: create vm_operations_struct without page_mkwrite()
>    sysfs: add mmap_allocates parameter to struct bin_attribute
>    PCI/P2PDMA: create VMA without page_mkwrite() operator
>    mm/gup: handle ZONE_DEVICE pages in folio_fast_pin_allowed()
>    mm/gup: allow FOLL_LONGTERM & FOLL_PCI_P2PDMA
>    RDMA/umem: add support for P2P RDMA
> 
>   drivers/infiniband/core/umem.c |  3 +++
>   drivers/pci/p2pdma.c           |  1 +
>   fs/kernfs/file.c               | 15 ++++++++++++++-
>   fs/sysfs/file.c                | 25 +++++++++++++++++++------
>   include/linux/kernfs.h         |  7 +++++++
>   include/linux/sysfs.h          |  1 +
>   mm/gup.c                       |  9 ++++-----
>   7 files changed, 49 insertions(+), 12 deletions(-)
> 
> 
> base-commit: c3f38fa61af77b49866b006939479069cd451173