This patch series use ZONE_DEVICE and mhp to add Direct I/O support to reserved memory
when rmem is as dio's src buffer.
Our use case is when isp generates frame and writes to given memory region, arm(kernel) will
try to read frames from the reserved memory region. If throughput is low, frame loss
will be serious.
Before this patch series, we can only use bufferd I/O and the throughput is very low even
with the help of AIO/io_uring.
This patch is tested on v5.15.35 + no-map rmem region, and can be git am into
5.19-rc5 without conflicts.
Li Chen (4):
of: add struct page support to rmem
mm/sparse: skip no-map memblock check when fill_subsection_map
arm64: mm: move memblock_clear_nomap after __add_pages
sample/reserved_mem: Introduce a sample of struct page and dio support
to no-map rmem
arch/arm64/mm/mmu.c | 2 +-
drivers/of/Kconfig | 9 ++
drivers/of/of_reserved_mem.c | 218 +++++++++++++++++++++++++++++++-
include/linux/of_reserved_mem.h | 11 ++
mm/sparse.c | 4 +-
samples/Kconfig | 7 +
samples/Makefile | 1 +
samples/reserved_mem/Makefile | 2 +
samples/reserved_mem/rmem_dio.c | 116 +++++++++++++++++
9 files changed, 367 insertions(+), 3 deletions(-)
create mode 100755 samples/reserved_mem/Makefile
create mode 100755 samples/reserved_mem/rmem_dio.c
--
2.25.1
Who is going to use it and how? Because normally the reserved memory
is used through the DMA allocator, and you can't just do direct I/O
to that.
Hi Christoph,
---- On Mon, 11 Jul 2022 23:01:32 +0800 Christoph Hellwig <[email protected]> wrote ---
> Who is going to use it and how? Because normally the reserved memory
> is used through the DMA allocator, and you can't just do direct I/O
> to that.
>
My use case has been stated in the cover letter, but our driver is not ready for upstream yet.
With DMA allocator, we can access buffer in kernel space, not userspace, however, this patch
series servers for userspace direct I/O, so that you can mmap device file as src buffer from userspace,
and dio to file on disk.
There are some mmap + rmem cases in the kernel source tree which don't use the DMA allocator already.
I also found some users have asked for a solution of supporting direct I/O on file_operation->mmap like:
https://stackoverflow.com/questions/44820740/using-o-direct-with-io-memory
https://www.mail-archive.com/[email protected]/msg02314.html
I believe there are some other potential users who didn't post questions on the internet.
If the upstream kernel has this feature, these users can mmap their
reserved memory to userspace and get direct i/o support.
Regards,
Li
On Tue, Jul 12, 2022 at 12:05:06AM +0800, Li Chen wrote:
> My use case has been stated in the cover letter, but our driver is not ready for upstream yet.
Which means we can't review the use case. I'd suggest you come back
when you submit your driver.
> With DMA allocator, we can access buffer in kernel space, not userspace, however, this patch
Take a look at dma_mmap_* on how to map DMA coherent allocations to
usersapce. This is of course easily possible.
Hi Christoph,
---- On Tue, 12 Jul 2022 00:09:55 +0800 Christoph Hellwig <[email protected]> wrote ---
> On Tue, Jul 12, 2022 at 12:05:06AM +0800, Li Chen wrote:
> > My use case has been stated in the cover letter, but our driver is not ready for upstream yet.
>
> Which means we can't review the use case. I'd suggest you come back
> when you submit your driver.
Totally agree, but we plan to start rewriting the code of our video driver in a long time, it has many legacy codes and I need to rewrite a lot of codes to migrate to v4l2.
That's why I also submit a sample driver here: to make the review progress easier and don't need reviewers to read video driver codes.
>
> > With DMA allocator, we can access buffer in kernel space, not userspace, however, this patch
>
> Take a look at dma_mmap_* on how to map DMA coherent allocations to
> usersapce. This is of course easily possible.
>
Yes, I know them. They take use of remap_pfn_range, which set VM_IO and VM_PFNMAP on vma, so dio is not possible with them.
IIUC, if you want to expose the kernel's reserved memory to userspace, it doesn't matter whether the memory came from dma allocator or something else.
The point is if they want to get better throughput, they can consider replacing their dma_mmap_* with my reserved_mem_memremap_pages + reserved_mem_dio_mmap.
Regards,
Li