Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley
        Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United
        Kingdom.
        Registered in England and Wales under Company Registration No. 3798903
From:   David Howells <dhowells@redhat.com>
In-Reply-To: <Y01VjOE2RrLVA2T6@infradead.org>
References: <Y01VjOE2RrLVA2T6@infradead.org> <1762414.1665761217@warthog.procyon.org.uk>
To:     Christoph Hellwig <hch@infradead.org>
Cc:     dhowells@redhat.com, Al Viro <viro@zeniv.linux.org.uk>,
        willy@infradead.org, dchinner@redhat.com,
        Steve French <smfrench@gmail.com>,
        Shyam Prasad N <nspmangalore@gmail.com>,
        Rohith Surabattula <rohiths.msft@gmail.com>,
        Jeff Layton <jlayton@kernel.org>,
        Ira Weiny <ira.weiny@intel.com>, torvalds@linux-foundation.org,
        linux-cifs@vger.kernel.org, linux-fsdevel@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: How to convert I/O iterators to iterators, sglists and RDMA lists
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <1415914.1666274636.1@warthog.procyon.org.uk>
Content-Transfer-Encoding: quoted-printable
Date:   Thu, 20 Oct 2022 15:03:56 +0100
Message-ID: <1415915.1666274636@warthog.procyon.org.uk>
Precedence: bulk

Christoph Hellwig <hch@infradead.org> wrote:

> >  (1) Async direct I/O.
> > =

> >      In the async case direct I/O, we cannot hold on to the iterator w=
hen we
> >      return, even if the operation is still in progress (ie. we return
> >      EIOCBQUEUED), as it is likely to be on the caller's stack.
> > =

> >      Also, simply copying the iterator isn't sufficient as virtual use=
rspace
> >      addresses cannot be trusted and we may have to pin the pages that
> >      comprise the buffer.
> =

> This is very related to the discussion we are having related to pinning
> for O_DIRECT with Ira and Al.

Do you have a link to that discussion?  I don't see anything obvious on
fsdevel including Ira.

I do see a discussion involving iov_iter_pin_pages, but I don't see Ira
involved in that.

> What block file systems do is to take the pages from the iter and some f=
lags
> on what is pinned.  We can generalize this to store all extra state in a
> flags word, or byte the bullet and allow cloning of the iter in one form=
 or
> another.

Yeah, I know.  A list of pages is not an ideal solution.  It can only hand=
le
contiguous runs of pages, possibly with a partial page at either end.  A b=
vec
iterator would be of more use as it can handle a series of partial pages.

Note also that I would need to turn the pages *back* into an iterator in o=
rder
to commune with sendmsg() in the nether reaches of some network filesystem=
s.

> >  (2) Crypto.
> > =

> >      The crypto interface takes scatterlists, not iterators, so we nee=
d to
> >      be able to convert an iterator into a scatterlist in order to do
> >      content encryption within netfslib.  Doing this in netfslib makes=
 it
> >      easier to store content-encrypted files encrypted in fscache.
> =

> Note that the scatterlist is generally a pretty bad interface.  We've
> been talking for a while to have an interface that takes a page array
> as an input and return an array of { dma_addr, len } tuples.  Thinking
> about it taking in an iter might actually be an even better idea.

It would be nice to be able to pass an iterator to the crypto layer.  I'm =
not
sure what the crypto people think of that.

> >  (3) RDMA.
> > =

> >      To perform RDMA, a buffer list needs to be presented as a QPE arr=
ay.
> >      Currently, cifs converts the iterator it is given to lists of pag=
es,
> >      then each list to a scatterlist and thence to a QPE array.  I hav=
e
> >      code to pass the iterator down to the bottom, using an intermedia=
te
> >      BVEC iterator instead of a page list if I can't pass down the
> >      original directly (eg. an XARRAY iterator on the pagecache), but =
I
> >      still end up converting it to a scatterlist, which is then conver=
ted
> >      to a QPE.  I'm trying to go directly from an iterator to a QPE ar=
ray,
> >      thus avoiding the need to allocate an sglist.
> =

> I'm not sure what you mean with QPE.  The fundamental low-level
> interface in RDMA is the ib_sge.

Sorry, yes. ib_sge array.  I think it appears as QPs on the wire.

> If you feed it to RDMA READ/WRITE requests the interface for that is the
> RDMA R/W API in drivers/infiniband/core/rw.c, which currently takes a
> scatterlist but to which all of the above remarks on DMA interface apply=
.
> For RDMA SEND that ULP has to do a dma_map_single/page to fill it, which=
 is
> a quite horrible layering violation and should move into the driver, but
> that is going to a massive change to the whole RDMA subsystem, so unlike=
ly
> to happen anytime soon.

In cifs, as it is upstream, in RDMA transmission, the iterator is converte=
d
into a clutch of pages in the top, which is converted back into iterators
(smbd_send()) and those into scatterlists (smbd_post_send_data()), thence =
into
sge lists (see smbd_post_send_sgl()).

I have patches that pass an iterator (which it decants to a bvec if async)=
 all
the way down to the bottom layer.  Snippets are then converted to scatterl=
ists
and those to sge lists.  I would like to skip the scatterlist intermediate=
 and
convert directly to sge lists.

On the other hand, if you think the RDMA API should be taking scatterlists
rather than sge lists, that would be fine.  Even better if I can just pass=
 an
iterator in directly - though neither scatterlist nor iterator has a place=
 to
put the RDMA local_dma_key - though I wonder if that's actually necessary =
for
each sge element, or whether it could be handed through as part of the req=
uest
as a hole.

> Neither case has anything to do with what should be in common iov_iter
> code, all this needs to live in the RDMA subsystem as a consumer.

That's fine in principle.  However, I have some extraction code that can
convert an iterator to another iterator, an sglist or an rdma sge list, us=
ing
a common core of code to do all three.

I can split it up if that is preferable.

Do you have code that's ready to be used?  I can make immediate use of it.

David