Received-SPF: pass (google.com: best guess record for domain of linux-ext4-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: Discussion: is it time to remove dioread_nolock?
To:     "Theodore Y. Ts'o" <tytso@mit.edu>
Cc:     Jan Kara <jack@suse.cz>,
        Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>,
        Ext4 Developers List <linux-ext4@vger.kernel.org>,
        joseph.qi@linux.alibaba.com, Liu Bo <bo.liu@linux.alibaba.com>
References: <20200109163802.GA33929@mit.edu>
 <20200114233054.890D7A4040@d06av23.portsmouth.uk.ibm.com>
 <20200115164829.GB165687@mit.edu>
From:   Ritesh Harjani <riteshh@linux.ibm.com>
Date:   Thu, 16 Jan 2020 15:16:52 +0530
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.7.2
MIME-Version: 1.0
In-Reply-To: <20200115164829.GB165687@mit.edu>
Content-Type: multipart/mixed;
 boundary="------------D66429D529101157068B4F61"
Content-Language: en-US
Message-Id: <20200116094653.BF97FA405B@b06wcsmtp001.portsmouth.uk.ibm.com>
Sender: linux-ext4-owner@vger.kernel.org
Precedence: bulk

This is a multi-part message in MIME format.
--------------D66429D529101157068B4F61
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hello Ted,

On 1/15/20 10:18 PM, Theodore Y. Ts'o wrote:
> On Wed, Jan 15, 2020 at 05:00:53AM +0530, Ritesh Harjani wrote:
>>
>> I too collected some performance numbers on my x86 box with
>> --direct=1, bs=4K/1M & ioengine=libaio, with default opt v/s dioread_nolock
>> opt on latest ext4 git tree.
>>
>> I found the delta to be within +/- 6% in all of the runs which includes, seq
>> read, mixed rw & mixed randrw.
> 
> Thanks for taking the performance measurements!
> 
> Are you able to release more detail of what the deltas were for those
> tests?  And how stable were those results across repeated runs?
> 

I have collected these numbers using fio. All of the column values
shown in the attachment & pasted below are average of 3 runs.
I checked all those individual runs too and saw that even in the run-to-
run variations the delta was within +/- 5% only.

The naming of those individual runs *.json files are a bit weird and 
will take sometime if I have to publish the individual runs. But do let
me know if that as well is required. I can make some changes in the 
scripts to print individual run's numbers too.

I have pasted the individual fio files too which I used for my testing.
I have used libaio as ioengine (with iodepth=16) in the SeqRead case and
psync in others.


Performance numbers:
===================

(seq) read (4K) - libaio
============
threads     default_opt_read(MB/s)   dioread_nolock_opt_read(MB/s)
----------  ---------------------    ----------------------------
1           138.928059895833             138.824869791667 

8           129.345052083333             124.472005208333 

24          71.6555989583333             72.275390625


(Not sure why 24 thread case has lower perf numbers, but on increasing
the iodepth=32, I could see an increase in bw with 24 threads case
too.)


(seq) read (1M) - libaio
===========
threads     default_opt_read(MB/s)  dioread_nolock_opt_read(MB/s)
----------  ---------------------   ---------------
1           138.905598958333             138.832682291667 

8           111.263997395833             109.301106770833 

24          121.895182291667             127.75390625


randrw(4K)  - read (psync) 
 

========================== 

threads     default_opt_read [KB/s]  dioread_opt_read [KB/s] 

----------  ----------------    ---------------- 

1           414.666666666667    410.0 

8           780.0               792.333333333333 

24          967.333333333333    991.666666666667 


randrw(4K)  - write (psync) 

=========================== 

threads     default_opt_write [KB/s]  dioread_opt_write [KB/s] 

----------  -----------------    ----------------- 

1           418.0                413.666666666667 

8           796.666666666667     809.666666666667 

24          981.333333333333     1007.66666666667


randrw(1M) - read (psync)
=================
threads     default_opt_read(MB/s)   dioread_nolock_opt_read(MB/s)
----------  -----------------        -----------------------
1           39.5693359375                39.7288411458333 

8           44.5179036458333             47.9098307291667 

24          50.2861328125                51.720703125


randrw(1M) - write (psync)
==================
threads     default_opt_write(MB/s)  dioread_nolock_opt_write(MB/s)
----------  ----------------------   -----------------------
1           41.4583333333333              41.5068359375 

8           46.0768229166667              49.568359375 

24          49.5947265625                 50.7083333333333


rw(1M) - read (psync)
=============
threads     default_opt_read(MB/s)  dioread_nolock_opt_read(MB/s)
----------  ----------------------  -------------------
1           43.6458333333333             43.6770833333333 

8           48.1178385416667             49.2718098958333 

24          50.5703125                   53.7890625


rw(1M) - write (psync)
==============
threads     default_opt_write(MB/s)   dioread_nolock_opt_write(MB/s)
----------  -----------------------   ------------------
1           45.5065104166667               45.5654296875 

8           49.7431640625                  51.0690104166667 

24          50.2493489583333               53.3463541666667


FIO FILES
=========

dio_read.fio
============
[global]
         ioengine=libaio
         rw=read
         runtime=60
         iodepth=16
         direct=1
         size=10G
         filename=./testfile
         group_reporting=1
         thread=1
         ;bs=$bs
         ;numjobs=24

[fio-run]


dio_randrw.fio
==============
[global]
         ioengine=psync
         rw=randrw
         runtime=60
         iodepth=1
         direct=1
         size=10G
         filename=./testfile
         group_reporting=1
         thread=1
         ;bs=$bs
         ;numjobs=24

[fio-run]

dio_rw.fio
==========
[global]
         ioengine=psync
         rw=rw
         runtime=60
         iodepth=1
         direct=1
         size=10G
         filename=./testfile
         group_reporting=1
         thread=1
         ;bs=$bs
         ;numjobs=24

[fio-run]


Thanks
-ritesh


--------------D66429D529101157068B4F61
Content-Type: text/plain; charset=UTF-8;
 name="datafile.txt"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="datafile.txt"

U2VxUmVhZCAoNEspIC0gbGliYWlvCj09PT09PT09PT09PQp0aHJlYWRzICAgICBkZWZhdWx0
X29wdFtyZWFkXzRLKE1CL3MpXSAgIGRpb3JlYWRfbm9sb2NrX29wdCBbcmVhZF80SyhNQi9z
KV0KLS0tLS0tLS0tLSAgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tICAtLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0KMSAgICAgICAgICAgMTM4LjkyODA1OTg5NTgzMyAgICAg
ICAgICAgICAxMzguODI0ODY5NzkxNjY3ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgCjggICAgICAgICAgIDEyOS4zNDUwNTIwODMzMzMgICAgICAgICAgICAgMTI0LjQ3
MjAwNTIwODMzMyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAoyNCAgICAg
ICAgICA3MS42NTU1OTg5NTgzMzMzICAgICAgICAgICAgIDcyLjI3NTM5MDYyNSAgIAoKClNl
cVJlYWQoMU0pIC0gbGliYWlvCj09PT09PT09PT09CnRocmVhZHMgICAgIGRlZmF1bHRfb3B0
IFtyZWFkXzFNKE1CL3MpXSAgZGlvcmVhZF9ub2xvY2tfb3B0IFtyZWFkXzFNKE1CL3MpXQot
LS0tLS0tLS0tICAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0gIC0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tCjEgICAgICAgICAgIDEzOC45MDU1OTg5NTgzMzMgICAgICAg
ICAgICAgMTM4LjgzMjY4MjI5MTY2NyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIAo4ICAgICAgICAgICAxMTEuMjYzOTk3Mzk1ODMzICAgICAgICAgICAgIDEwOS4zMDEx
MDY3NzA4MzMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKMjQgICAgICAg
ICAgMTIxLjg5NTE4MjI5MTY2NyAgICAgICAgICAgICAxMjcuNzUzOTA2MjUgICAgICAKCgpy
YW5kcncoNEspICAtIHJlYWQgKHBzeW5jKQo9PT09PT09PT09PT09PT09PT09PT09PT09PQp0
aHJlYWRzICAgICBkZWZhdWx0X29wdF9yZWFkIFtLQi9zXSAgZGlvcmVhZF9vcHRfcmVhZCBb
S0Ivc10KLS0tLS0tLS0tLSAgLS0tLS0tLS0tLS0tLS0tLSAgCS0tLS0tLS0tLS0tLS0tLS0K
MSAgICAgICAgICAgNDE0LjY2NjY2NjY2NjY2NyAgCTQxMC4wCjggICAgICAgICAgIDc4MC4w
ICAgICAgICAgICAgIAk3OTIuMzMzMzMzMzMzMzMzCjI0ICAgICAgICAgIDk2Ny4zMzMzMzMz
MzMzMzMgIAk5OTEuNjY2NjY2NjY2NjY3CgoKcmFuZHJ3KDRLKSAgLSB3cml0ZSAocHN5bmMp
Cj09PT09PT09PT09PT09PT09PT09PT09PT09PQp0aHJlYWRzICAgICBkZWZhdWx0X29wdF93
cml0ZSBbS0Ivc10gIGRpb3JlYWRfb3B0X3dyaXRlIFtLQi9zXQotLS0tLS0tLS0tICAtLS0t
LS0tLS0tLS0tLS0tLSAJIC0tLS0tLS0tLS0tLS0tLS0tICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAKMSAgICAgICAgICAgNDE4LjAgICAgICAgICAgICAgCSA0MTMu
NjY2NjY2NjY2NjY3CjggICAgICAgICAgIDc5Ni42NjY2NjY2NjY2NjcgIAkgODA5LjY2NjY2
NjY2NjY2NwoyNCAgICAgICAgICA5ODEuMzMzMzMzMzMzMzMzICAJIDEwMDcuNjY2NjY2NjY2
NjcKCnJhbmRydygxTSkgLSByZWFkIChwc3luYykKPT09PT09PT09PT09PT09PT0KdGhyZWFk
cyAgICAgZGVmYXVsdF9vcHQgW3JlYWRfMU0oTUIvcykgICBkaW9yZWFkX25vbG9ja19vcHQg
W3JlYWRfMU0oTUIvcyldCi0tLS0tLS0tLS0gIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LSAgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCjEgICAgICAgICAgIDM5LjU2
OTMzNTkzNzUgICAgICAgICAgICAgICAgMzkuNzI4ODQxMTQ1ODMzMyAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgCjggICAgICAgICAgIDQ0LjUxNzkwMzY0NTgzMzMg
ICAgICAgICAgICAgNDcuOTA5ODMwNzI5MTY2NyAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgCjI0ICAgICAgICAgIDUwLjI4NjEzMjgxMjUgICAgICAgICAgICAgICAg
NTEuNzIwNzAzMTI1ICAgICAgIAoKCnJhbmRydygxTSkgLSB3cml0ZSAocHN5bmMpCj09PT09
PT09PT09PT09PT09PQp0aHJlYWRzICAgICBkZWZhdWx0X29wdCBbd3JpdGVfMU0oTUIvcyld
ICBkaW9yZWFkX25vbG9ja19vcHQgW3dyaXRlXzFNKE1CL3MpXQotLS0tLS0tLS0tICAtLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tICAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tCjEgICAgICAgICAgIDQxLjQ1ODMzMzMzMzMzMzMgICAgICAgICAgICAgIDQxLjUw
NjgzNTkzNzUgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKOCAg
ICAgICAgICAgNDYuMDc2ODIyOTE2NjY2NyAgICAgICAgICAgICAgNDkuNTY4MzU5Mzc1ICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAoyNCAgICAgICAgICA0
OS41OTQ3MjY1NjI1ICAgICAgICAgICAgICAgICA1MC43MDgzMzMzMzMzMzMzICAgICAgICAg
CgoKcncoMU0pIC0gcmVhZCAocHN5bmMpCj09PT09PT09PT09PT0KdGhyZWFkcyAgICAgZGVm
YXVsdF9vcHQgW3JlYWRfMU0oTUIvcyldICBkaW9yZWFkX25vbG9ja19vcHQgW3JlYWRfMU0o
TUIvcyldCi0tLS0tLS0tLS0gIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSAgLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCjEgICAgICAgICAgIDQzLjY0NTgzMzMzMzMz
MzMgICAgICAgICAgICAgNDMuNjc3MDgzMzMzMzMzMyAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAKOCAgICAgICAgICAgNDguMTE3ODM4NTQxNjY2NyAgICAgICAgICAgICA0
OS4yNzE4MDk4OTU4MzMzICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAoyNCAg
ICAgICAgICA1MC41NzAzMTI1ICAgICAgICAgICAgICAgICAgIDUzLjc4OTA2MjUKCgpydygx
TSkgLSB3cml0ZSAocHN5bmMpCj09PT09PT09PT09PT09CnRocmVhZHMgICAgIGRlZmF1bHRf
b3B0IFt3cml0ZV8xTShNQi9zKV0gICBkaW9yZWFkX25vbG9ja19vcHQgW3dyaXRlXzFNKE1C
L3MpCi0tLS0tLS0tLS0gIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tICAtLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KMSAgICAgICAgICAgNDUuNTA2NTEwNDE2NjY2
NyAgICAgICAgICAgICAgIDQ1LjU2NTQyOTY4NzUgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgIAo4ICAgICAgICAgICA0OS43NDMxNjQwNjI1ICAgICAgICAgICAgICAg
ICAgNTEuMDY5MDEwNDE2NjY2NyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
CjI0ICAgICAgICAgIDUwLjI0OTM0ODk1ODMzMzMgICAgICAgICAgICAgICA1My4zNDYzNTQx
NjY2NjY3ICAgICAgICAgCgo=
--------------D66429D529101157068B4F61--