Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley
        Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United
        Kingdom.
        Registered in England and Wales under Company Registration No. 3798903
From:   David Howells <dhowells@redhat.com>
To:     Christoph Hellwig <hch@infradead.org>,
        Al Viro <viro@zeniv.linux.org.uk>
Cc:     dhowells@redhat.com, willy@infradead.org, dchinner@redhat.com,
        Steve French <smfrench@gmail.com>,
        Shyam Prasad N <nspmangalore@gmail.com>,
        Rohith Surabattula <rohiths.msft@gmail.com>,
        Jeff Layton <jlayton@kernel.org>,
        torvalds@linux-foundation.org, linux-cifs@vger.kernel.org,
        linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: How to convert I/O iterators to iterators, sglists and RDMA lists
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <1762412.1665761217.1@warthog.procyon.org.uk>
Content-Transfer-Encoding: quoted-printable
Date:   Fri, 14 Oct 2022 16:26:57 +0100
Message-ID: <1762414.1665761217@warthog.procyon.org.uk>
Precedence: bulk

Hi Christoph, Al,

One of the aims I have for netfslib is to hide the involvement of pages/fo=
lios
entirely from the filesystem.  That way the filesystem need not concern it=
self
with changes such as multipage folios appearing in the VM.

To this end, I'm trying to make it such that each netfs_io_subrequest cont=
ains
an iterator that describes the segment of buffer that a subrequest is deal=
ing
with.  The filesystem interprets the buffer appropriately, and can even pa=
ss
the iterator directly to kernel_sendmsg() or kernel_recvmsg() if this is
convenient.

In netfslib and in the network filesystems using it, however, there are a
number of situations where we need to "convert" an iterator:

 (1) Async direct I/O.

     In the async case direct I/O, we cannot hold on to the iterator when =
we
     return, even if the operation is still in progress (ie. we return
     EIOCBQUEUED), as it is likely to be on the caller's stack.

     Also, simply copying the iterator isn't sufficient as virtual userspa=
ce
     addresses cannot be trusted and we may have to pin the pages that
     comprise the buffer.

 (2) Crypto.

     The crypto interface takes scatterlists, not iterators, so we need to=
 be
     able to convert an iterator into a scatterlist in order to do content
     encryption within netfslib.  Doing this in netfslib makes it easier t=
o
     store content-encrypted files encrypted in fscache.

 (3) RDMA.

     To perform RDMA, a buffer list needs to be presented as a QPE array.
     Currently, cifs converts the iterator it is given to lists of pages, =
then
     each list to a scatterlist and thence to a QPE array.  I have code to
     pass the iterator down to the bottom, using an intermediate BVEC iter=
ator
     instead of a page list if I can't pass down the original directly (eg=
. an
     XARRAY iterator on the pagecache), but I still end up converting it t=
o a
     scatterlist, which is then converted to a QPE.  I'm trying to go dire=
ctly
     from an iterator to a QPE array, thus avoiding the need to allocate a=
n
     sglist.

Constraints:

 (A) Userspace gives us a list (IOVEC/UBUF) of buffers that may not be pag=
e
     aligned and may not be contiguous; further, within a particular buffe=
r
     span, the pages may not be contiguous and may be part of multipage
     folios.

     Converting to a BVEC iterator allows a whole buffer to be described, =
and
     extracting a subset of a BVEC iterator is straightforward.

 (B) Kernel buffers may not be pinnable.  If we get a KVEC iterator, say, =
we
     can't assume that we can pin the pages (say the buffer is part of the
     kernel rodata or belongs to a device - say a flash).

     This may also apply to mmap'd devices in userspace iovecs.

 (C) We don't want to pin pages if we can avoid it.

 (D) PIPE iterators.


So, my first attempt at dealing with (1) involved creating a function that
extracted part of an iterator into another iterator[2].  Just copying and
shaping if possible (assuming, say, that an XARRAY iterator doesn't need t=
o
pin the pages), but otherwise using repeated application of
iov_iter_get_pages() to build up a BVEC iterator (which is basically just =
a
list of {page,offset,len} tuples).

Al objected on the basis that it was pinning pages that it didn't need to =
(say
extracting BVEC->BVEC) and that it didn't deal correctly with PIPE (becaus=
e
the underlying pipe would get advanced too early) or KVEC/BVEC (because it
might refer to a page that was un-get_pages-able).

Christoph objected that it shouldn't be available as a general purpose hel=
per
and that it should be kept inside cifs - but I'm wanting to use it inside =
of
netfslib also.

My first attempt at dealing with (2) involved creating a function to scan =
an
iterator[2] and call a function on each segment of it.  This could be used=
 to
perform checksumming or to build up a scatterlist.  However, as Al pointed
out, I didn't get the IOBUF or KVEC handling right.  Mostly, though, I wan=
t to
convert to an sglist and work from that.

I then had a go at implementing a common framework[3] to extract an iterat=
or
into another iterator, an sglist, a RDMA QPE array or any other type of li=
st
we might envision.  Al's not keen on that for a number of reasons (see his
reply) including that it loses type safety and that I should be using
iov_iter_get_pages2() - which he already objected to me doing in[1]:-/


So any thoughts on what the right way to do this is?  What is the right AP=
I?

I have three things I need to make from a source iterator: a copy and/or a
subset iterator, a scatterlist and an RDMA QPE array, and several differen=
t
types of iterator to extract from.  I shouldn't pin pages unless I need to=
,
sometimes pages cannot be pinned and sometimes I may have to add the physi=
cal
address to the entry.

If I can share part of the infrastructure, that would seem to be a good th=
ing.

David

https://lore.kernel.org/r/165364824259.3334034.5837838050291740324.stgit@w=
arthog.procyon.org.uk/ [1]
https://lore.kernel.org/r/165364824973.3334034.10715738699511650662.stgit@=
warthog.procyon.org.uk/ [2]
https://lore.kernel.org/r/3750754.1662765490@warthog.procyon.org.uk/ [3]