Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Mingkai Dong <mingkaidong@gmail.com>
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.40.0.2.32\))
Subject: Re: Expense of read_iter
In-Reply-To: <20210107151125.GB5270@casper.infradead.org>
Date:   Fri, 8 Jan 2021 00:43:20 +0800
Cc:     Mikulas Patocka <mpatocka@redhat.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Jan Kara <jack@suse.cz>,
        Steven Whitehouse <swhiteho@redhat.com>,
        Eric Sandeen <esandeen@redhat.com>,
        Dave Chinner <dchinner@redhat.com>,
        Theodore Ts'o <tytso@mit.edu>,
        Wang Jianchao <jianchao.wan9@gmail.com>,
        "Tadakamadla, Rajesh" <rajesh.tadakamadla@hpe.com>,
        linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org,
        linux-nvdimm@lists.01.org, sunrise_l@sjtu.edu.cn
Content-Transfer-Encoding: quoted-printable
Message-Id: <17045315-CC1F-4165-B8E3-BA55DD16D46B@gmail.com>
References: <alpine.LRH.2.02.2101061245100.30542@file01.intranet.prod.int.rdu2.redhat.com>
 <20210107151125.GB5270@casper.infradead.org>
To:     Matthew Wilcox <willy@infradead.org>
Precedence: bulk

Hi Matthew,

We have also discovered the expense of `->read_iter` in our study on =
Ext4-DAX.
In single-thread 4K-reads, the `->read` version could outperform =
`->read_iter`
by 41.6% in terms of throughput.

According to our observation and evaluation, at least for Ext4-DAX, the =
cost
also comes from the invocation of `->iomap_begin` (`ext4_iomap_begin`),
which might not be simply avoided by adding a new iter_type.
The slowdown is more significant when multiple threads reading different =
files
concurrently, due to the scalability issue (grabbing a read lock to =
check the
status of the journal) in `ext4_iomap_begin`.

In our solution, we implemented the `->read` and `->write` interfaces =
for
Ext4-DAX. Thus, we also think it would be good if both `->read` and =
`->read_iter`
could exist.

By the way, besides the implementation of `->read` and `->write`, we =
have
some other optimizations for Ext4-DAX and would like to share them once =
our
patches are prepared.

Thanks,
Mingkai


> On Jan 7, 2021, at 23:11, Matthew Wilcox <willy@infradead.org> wrote:
>=20
> On Thu, Jan 07, 2021 at 08:15:41AM -0500, Mikulas Patocka wrote:
>> I'd like to ask about this piece of code in __kernel_read:
>> 	if (unlikely(!file->f_op->read_iter || file->f_op->read))
>> 		return warn_unsupported...
>> and __kernel_write:
>> 	if (unlikely(!file->f_op->write_iter || file->f_op->write))
>> 		return warn_unsupported...
>>=20
>> - It exits with an error if both read_iter and read or write_iter and=20=

>> write are present.
>>=20
>> I found out that on NVFS, reading a file with the read method has 10%=20=

>> better performance than the read_iter method. The benchmark just =
reads the=20
>> same 4k page over and over again - and the cost of creating and =
parsing=20
>> the kiocb and iov_iter structures is just that high.
>=20
> Which part of it is so expensive?  Is it worth, eg adding an iov_iter
> type that points to a single buffer instead of a single-member iov?
>=20
> +++ b/include/linux/uio.h
> @@ -19,6 +19,7 @@ struct kvec {
>=20
> enum iter_type {
>        /* iter types */
> +       ITER_UBUF =3D 2,
>        ITER_IOVEC =3D 4,
>        ITER_KVEC =3D 8,
>        ITER_BVEC =3D 16,
> @@ -36,6 +36,7 @@ struct iov_iter {
>        size_t iov_offset;
>        size_t count;
>        union {
> +               void __user *buf;
>                const struct iovec *iov;
>                const struct kvec *kvec;
>                const struct bio_vec *bvec;
>=20
> and then doing all the appropriate changes to make that work.
> _______________________________________________
> Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
> To unsubscribe send an email to linux-nvdimm-leave@lists.01.org