Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 20 Oct 2022 20:30:34 -0700
From:   Ira Weiny <ira.weiny@intel.com>
To:     David Howells <dhowells@redhat.com>
CC:     Christoph Hellwig <hch@infradead.org>,
        Al Viro <viro@zeniv.linux.org.uk>, <willy@infradead.org>,
        <dchinner@redhat.com>, Steve French <smfrench@gmail.com>,
        Shyam Prasad N <nspmangalore@gmail.com>,
        "Rohith Surabattula" <rohiths.msft@gmail.com>,
        Jeff Layton <jlayton@kernel.org>,
        <torvalds@linux-foundation.org>, <linux-cifs@vger.kernel.org>,
        <linux-fsdevel@vger.kernel.org>, <linux-kernel@vger.kernel.org>
Subject: Re: How to convert I/O iterators to iterators, sglists and RDMA lists
Message-ID: <Y1ISWla50g5gHax6@iweiny-desk3>
References: <Y01VjOE2RrLVA2T6@infradead.org>
 <1762414.1665761217@warthog.procyon.org.uk>
 <1415915.1666274636@warthog.procyon.org.uk>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <1415915.1666274636@warthog.procyon.org.uk>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?BcMvvY0cPQE+ZNHUHWy0YQOqQOGp8Iy7bBH+fQZnMtodUadba4908maPP5lM?=
 =?us-ascii?Q?Y7OGkeh/h4OnFLOaxb1vHGtZghJTiyKq0xfYCyaQtxV2PgmmAra3EyevwKeB?=
 =?us-ascii?Q?NO06O3Wr5n32PGjRQdffGAQw0GitXVbpqcURo/GNSyDvyScjdpDc221DKQ4H?=
 =?us-ascii?Q?o0jnAay2fnAVCWJHgSaX+3Pxd4OzyAs/QxvchPpg73fvFddqnX0YqcJf87Dw?=
 =?us-ascii?Q?/fUihhnslkJnrySIen3fAHs35rTrwIXOAI9cAz21Q9tSGa/7hNHD7YjvK1pv?=
 =?us-ascii?Q?pslGTIyauIKXTUuRllM4OzGyUVtMRssiQhR4WkRaBSSdDFRp0qR6ZrXJSVlj?=
 =?us-ascii?Q?4GTGJAFp/GbU4itABVeWayQHq//lfVr6BqbEzYFl5i7rAbckrXI3Gb3EzOSl?=
 =?us-ascii?Q?GtllQ6dJq5PmuS39smIsrccWKVfDu06ae58yZQK1fo3CaiM5++DsHDqWgioQ?=
 =?us-ascii?Q?GnZaMJvCCbPBG3rLei7hf6I5jfRngrQ2HR+9AitquOtit/tkfOg9saz20XOK?=
 =?us-ascii?Q?lpo7vG4HvOvPtUjiO30GoCbW2TmHgyO4A7FLsayRiG1ldByhm2SNK5C+Ek1m?=
 =?us-ascii?Q?/klZZe7B/EoFOOBa+CajOK4NTmAeg58Yr1HP0CesasSI83eoVy4yzxWv6peT?=
 =?us-ascii?Q?01lWfr1oBykrUmPFVl0zhUcXLgj875Yo9GD5oqGzShqmlRQZl2rmtSL3iV+Y?=
 =?us-ascii?Q?Him4QwBbGi81NqbwMSN+gRxuv13aQckBQ1yIL/ARKZqM8UNB6xqNd87lD8UP?=
 =?us-ascii?Q?b4Gt1belGhcN0rgKjBQKOoZbdQbkiaf+xyLBdDKV0W77dIy8o+fIRrF9BeAK?=
 =?us-ascii?Q?sCWYUGpw55BZsbe2BLq1Dvgv+DMlKqblOE/VMKC5O6jzCFenV0pglv7CIGdO?=
 =?us-ascii?Q?WqYSI81mlo6TsUzBsBEpllbiE7zukZBuX/T2CcRvotOodVk/vtMldpc/0iGe?=
 =?us-ascii?Q?fUBkn8qYDthxhSZzJDWF+5r+VBCvGbR0u/8W3Y/99ggoCMxtMGGNDfi7FEOe?=
 =?us-ascii?Q?eeRSZE32oCvNGH2f6UdAItd8a6R7vWtRNamUsB2XfGSb9UJFRHrHutZ8pTMs?=
 =?us-ascii?Q?/5BvFSjFaNGI7mn6+P+li9vE9W9KKg0R6IL+ENvAi2ww2WgXqnPOMMcjH5sg?=
 =?us-ascii?Q?XIOM+P1It7yieOLJG3fCr6uk+xyMI2Vk+x53XMj5g7VZ/AgG7Wl2ZeY7lCIw?=
 =?us-ascii?Q?C8z0ryI33wbwcTila6G93jQdNdLEO9JpjsV7fCOBP2qZJpjlgda8Lt2YCgpk?=
 =?us-ascii?Q?LkwKdxZLdSnuID5c1TkIYf6A2NxveCSXgW+z2kFz99uUdDSe7Aeoh0IFR394?=
 =?us-ascii?Q?e6jLbQ3vlIJHv8Eu7spJdIsGSYqq0VPjo/9Skl4mKo26Mzymwa9gf63co14L?=
 =?us-ascii?Q?aV+ENMmfMyUNvY0KZ8FEqquSK2YMQqJ63spbyFxfSuCEQ5WwII46bWAaNUiI?=
 =?us-ascii?Q?GPOxBQmhPsNLohsVOBJc5nW3JQ/LQWU3aw26Alfx/C8/3iVj9TclUkt2DJSz?=
 =?us-ascii?Q?i/1gbC0St4UBVjq+S0WRd83p5HPYmDlxZyxUjE4kfCvXDjEDvjVrPzfK17Kp?=
 =?us-ascii?Q?0885lpQYQcNjRSMDcNnqi4syl32LOuC2z2UpVb8g?=
X-MS-Exchange-CrossTenant-Network-Message-Id: afc5e1fb-18b3-44ad-a85e-08dab3149fb7
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6733.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Oct 2022 03:30:38.7771
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gA9mWG3E6zL2MKMGADKOAPWRBrF8cqemdyfHnuXkb/5U1DFaXiA6GNiMxpFn0bRCwVS5pFIj7VCxwYviI3a/wg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR11MB6295
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Oct 20, 2022 at 03:03:56PM +0100, David Howells wrote:
> Christoph Hellwig <hch@infradead.org> wrote:
> 
> > >  (1) Async direct I/O.
> > > 
> > >      In the async case direct I/O, we cannot hold on to the iterator when we
> > >      return, even if the operation is still in progress (ie. we return
> > >      EIOCBQUEUED), as it is likely to be on the caller's stack.
> > > 
> > >      Also, simply copying the iterator isn't sufficient as virtual userspace
> > >      addresses cannot be trusted and we may have to pin the pages that
> > >      comprise the buffer.
> > 
> > This is very related to the discussion we are having related to pinning
> > for O_DIRECT with Ira and Al.
> 
> Do you have a link to that discussion?  I don't see anything obvious on
> fsdevel including Ira.

I think Christoph meant to say John Hubbard.

> 
> I do see a discussion involving iov_iter_pin_pages, but I don't see Ira
> involved in that.

This one?

https://lore.kernel.org/all/20220831041843.973026-5-jhubbard@nvidia.com/

I've been casually reading it but not directly involved.

Ira

> 
> > What block file systems do is to take the pages from the iter and some flags
> > on what is pinned.  We can generalize this to store all extra state in a
> > flags word, or byte the bullet and allow cloning of the iter in one form or
> > another.
> 
> Yeah, I know.  A list of pages is not an ideal solution.  It can only handle
> contiguous runs of pages, possibly with a partial page at either end.  A bvec
> iterator would be of more use as it can handle a series of partial pages.
> 
> Note also that I would need to turn the pages *back* into an iterator in order
> to commune with sendmsg() in the nether reaches of some network filesystems.
> 
> > >  (2) Crypto.
> > > 
> > >      The crypto interface takes scatterlists, not iterators, so we need to
> > >      be able to convert an iterator into a scatterlist in order to do
> > >      content encryption within netfslib.  Doing this in netfslib makes it
> > >      easier to store content-encrypted files encrypted in fscache.
> > 
> > Note that the scatterlist is generally a pretty bad interface.  We've
> > been talking for a while to have an interface that takes a page array
> > as an input and return an array of { dma_addr, len } tuples.  Thinking
> > about it taking in an iter might actually be an even better idea.
> 
> It would be nice to be able to pass an iterator to the crypto layer.  I'm not
> sure what the crypto people think of that.
> 
> > >  (3) RDMA.
> > > 
> > >      To perform RDMA, a buffer list needs to be presented as a QPE array.
> > >      Currently, cifs converts the iterator it is given to lists of pages,
> > >      then each list to a scatterlist and thence to a QPE array.  I have
> > >      code to pass the iterator down to the bottom, using an intermediate
> > >      BVEC iterator instead of a page list if I can't pass down the
> > >      original directly (eg. an XARRAY iterator on the pagecache), but I
> > >      still end up converting it to a scatterlist, which is then converted
> > >      to a QPE.  I'm trying to go directly from an iterator to a QPE array,
> > >      thus avoiding the need to allocate an sglist.
> > 
> > I'm not sure what you mean with QPE.  The fundamental low-level
> > interface in RDMA is the ib_sge.
> 
> Sorry, yes. ib_sge array.  I think it appears as QPs on the wire.
> 
> > If you feed it to RDMA READ/WRITE requests the interface for that is the
> > RDMA R/W API in drivers/infiniband/core/rw.c, which currently takes a
> > scatterlist but to which all of the above remarks on DMA interface apply.
> > For RDMA SEND that ULP has to do a dma_map_single/page to fill it, which is
> > a quite horrible layering violation and should move into the driver, but
> > that is going to a massive change to the whole RDMA subsystem, so unlikely
> > to happen anytime soon.
> 
> In cifs, as it is upstream, in RDMA transmission, the iterator is converted
> into a clutch of pages in the top, which is converted back into iterators
> (smbd_send()) and those into scatterlists (smbd_post_send_data()), thence into
> sge lists (see smbd_post_send_sgl()).
> 
> I have patches that pass an iterator (which it decants to a bvec if async) all
> the way down to the bottom layer.  Snippets are then converted to scatterlists
> and those to sge lists.  I would like to skip the scatterlist intermediate and
> convert directly to sge lists.
> 
> On the other hand, if you think the RDMA API should be taking scatterlists
> rather than sge lists, that would be fine.  Even better if I can just pass an
> iterator in directly - though neither scatterlist nor iterator has a place to
> put the RDMA local_dma_key - though I wonder if that's actually necessary for
> each sge element, or whether it could be handed through as part of the request
> as a hole.
> 
> > Neither case has anything to do with what should be in common iov_iter
> > code, all this needs to live in the RDMA subsystem as a consumer.
> 
> That's fine in principle.  However, I have some extraction code that can
> convert an iterator to another iterator, an sglist or an rdma sge list, using
> a common core of code to do all three.
> 
> I can split it up if that is preferable.
> 
> Do you have code that's ready to be used?  I can make immediate use of it.
> 
> David
>