Received-SPF: pass (google.com: best guess record for domain of linux-ext4-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (1.0)
Subject: Re: [RFC] performance regression with "ext4: Allow parallel DIO reads"
From:   Andreas Dilger <adilger@dilger.ca>
In-Reply-To: <15112e38-94fe-39d6-a8e2-064ff47187d5@linux.alibaba.com>
Date:   Fri, 26 Jul 2019 19:57:29 -0600
Cc:     Joseph Qi <jiangqi903@gmail.com>, Theodore Ts'o <tytso@mit.edu>,
        Jan Kara <jack@suse.cz>,
        Ext4 Developers List <linux-ext4@vger.kernel.org>,
        Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>,
        Liu Bo <bo.liu@linux.alibaba.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <A7FF6ED7-D480-4B01-A812-E100D595C515@dilger.ca>
References: <ab7cf51b-6b52-d151-e22c-6f4400a14589@linux.alibaba.com> <29d50d24-f8e7-5ef4-d4d8-3ea6fb1c6ed3@gmail.com> <6DADA28C-542F-45F6-ADB0-870A81ABED23@dilger.ca> <15112e38-94fe-39d6-a8e2-064ff47187d5@linux.alibaba.com>
To:     Joseph Qi <joseph.qi@linux.alibaba.com>
Sender: linux-ext4-owner@vger.kernel.org
Precedence: bulk

It would be useful to post some details about your test hardware
(eg. HDD vs. SSD, CPU cores+speed, RAM), so that it is possible to make
a good comparison of someone sees different results.=20

Cheers, Andreas

> On Jul 25, 2019, at 19:12, Joseph Qi <joseph.qi@linux.alibaba.com> wrote:
>=20
>=20
>=20
>> On 19/7/26 05:20, Andreas Dilger wrote:
>>=20
>>> On Jul 23, 2019, at 5:17 AM, Joseph Qi <jiangqi903@gmail.com> wrote:
>>>=20
>>> Hi Ted & Jan,
>>> Could you please give your valuable comments?
>>=20
>> It seems like the original patches should be reverted?  There is no data
>=20
> =46rom my test result, yes.
> I've also tested libaio with iodepth 16, it behaves the same. Here is the t=
est
> data for libaio 4k randrw:
>=20
> --------------------------------------------------------------------------=
-----------------
> w/ parallel dio reads | READ 78313KB/s, 19578, 1698.70us  | WRITE 78313KB/=
s, 19578, 4837.60us
> --------------------------------------------------------------------------=
-----------------
> w/o parallel dio reads| READ 387774KB/s, 96943, 1009.73us | WRITE 387656KB=
/s=EF=BC=8C96914, 308.87us
> --------------------------------------------------------------------------=
-----------------
>=20
> Since this commit went into upstream long time ago=EF=BC=8Cto be precise, L=
inux
> 4.9, I wonder if someone else has also observed this regression, or
> anything I missed?
>=20
> Thanks,
> Joseph
>=20
>> in the original commit message that indicates there is an actual performa=
nce
>> improvement from that patch, but there is data here showing it hurts both=

>> read and write performance quite significantly.
>>> Cheers, Andreas
>>=20
>>>> On 19/7/19 17:22, Joseph Qi wrote:
>>>> Hi Ted & Jan,
>>>> I've observed an significant performance regression with the following
>>>> commit in my Intel P3600 NVMe SSD.
>>>> 16c54688592c ext4: Allow parallel DIO reads
>>>>=20
>>>> =46rom my initial investigation, it may be because of the
>>>> inode_lock_shared (down_read) consumes more than inode_lock (down_write=
)
>>>> in mixed random read write workload.
>>>>=20
>>>> Here is my test result.
>>>>=20
>>>> ioengine=3Dpsync
>>>> direct=3D1
>>>> rw=3Drandrw
>>>> iodepth=3D1
>>>> numjobs=3D8
>>>> size=3D20G
>>>> runtime=3D600
>>>>=20
>>>> w/ parallel dio reads : kernel 5.2.0
>>>> w/o parallel dio reads: kernel 5.2.0, then revert the following commits=
:
>>>> 1d39834fba99 ext4: remove EXT4_STATE_DIOREAD_LOCK flag (related)
>>>> e5465795cac4 ext4: fix off-by-one error when writing back pages before d=
io read (related)
>>>> 16c54688592c ext4: Allow parallel DIO reads
>>>>=20
>>>> bs=3D4k:
>>>> -----------------------------------------------------------------------=
--------------------
>>>> w/ parallel dio reads | READ 30898KB/s, 7724, 555.00us   | WRITE 30875K=
B/s, 7718, 479.70us
>>>> -----------------------------------------------------------------------=
--------------------
>>>> w/o parallel dio reads| READ 117915KB/s, 29478, 248.18us | WRITE 117854=
KB/s=EF=BC=8C29463, 21.91us
>>>> -----------------------------------------------------------------------=
--------------------
>>>>=20
>>>> bs=3D16k:
>>>> -----------------------------------------------------------------------=
--------------------
>>>> w/ parallel dio reads | READ 58961KB/s, 3685, 835.28us   | WRITE 58877K=
B/s, 3679, 1335.98us
>>>> -----------------------------------------------------------------------=
--------------------
>>>> w/o parallel dio reads| READ 218409KB/s, 13650, 554.46us | WRITE 218257=
KB/s=EF=BC=8C13641, 29.22us
>>>> -----------------------------------------------------------------------=
--------------------
>>>>=20
>>>> bs=3D64k:
>>>> -----------------------------------------------------------------------=
--------------------
>>>> w/ parallel dio reads | READ 119396KB/s, 1865, 1759.38us | WRITE 119159=
KB/s, 1861, 2532.26us
>>>> -----------------------------------------------------------------------=
--------------------
>>>> w/o parallel dio reads| READ 422815KB/s, 6606, 1146.05us | WRITE 421619=
KB/s, 6587, 60.72us
>>>> -----------------------------------------------------------------------=
--------------------
>>>>=20
>>>> bs=3D512k:
>>>> -----------------------------------------------------------------------=
--------------------
>>>> w/ parallel dio reads | READ 392973KB/s, 767, 5046.35us  | WRITE 393165=
KB/s, 767, 5359.86us
>>>> -----------------------------------------------------------------------=
--------------------
>>>> w/o parallel dio reads| READ 590266KB/s, 1152, 4312.01us | WRITE 590554=
KB/s, 1153, 2606.82us
>>>> -----------------------------------------------------------------------=
--------------------
>>>>=20
>>>> bs=3D1M:
>>>> -----------------------------------------------------------------------=
--------------------
>>>> w/ parallel dio reads | READ 487779KB/s, 476, 8058.55us  | WRITE 485592=
KB/s, 474, 8630.51us
>>>> -----------------------------------------------------------------------=
--------------------
>>>> w/o parallel dio reads| READ 593927KB/s, 580, 7623.63us  | WRITE 591265=
KB/s, 577, 6163.42us
>>>> -----------------------------------------------------------------------=
--------------------
>>>>=20
>>>> Thanks,
>>>> Joseph
>>>>=20
>>=20
>>=20
>> Cheers, Andreas
>>=20
>>=20
>>=20
>>=20
>>=20