LinuxLists.cc - Slow file transfer speeds with CFQ IO scheduler in some cases

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Alexey Dobriyan wrote:
> On Sun, Nov 09, 2008 at 08:04:45PM +0200, Vitaly V. Bursov wrote:
>> I'm building small server system with openvz kernel and have ran into
>> some IO performance problems. Reading a single file via NFS delivers
>> around 9 MB/s over gigabit network, but while reading, say, 2 different
>> or same file 2 times at the same time I get >60MB/s.
>
> openvz kernels have very changed CFQ and, IIRC, this one-reader problem is
> fixed somewhere. Ask [email protected].

That's why I've checked vanilla kernel also. It performs somewhat better,
but definitely shares same issues...

2008-11-10 10:46:50

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Sun, Nov 09 2008, Vitaly V. Bursov wrote:
> Hello,
>
> I'm building small server system with openvz kernel and have ran into
> some IO performance problems. Reading a single file via NFS delivers
> around 9 MB/s over gigabit network, but while reading, say, 2 different
> or same file 2 times at the same time I get >60MB/s.
>
> Changing IO scheduler to deadline or anticipatory fixes problem.
>
> Tested kernels:
> OpenVZ RHEL5 028stab059.3 (9 MB/s with HZ=100, 20MB/s with HZ=1000
> fast local reads)
> Vanilla 2.6.27.5 (40 MB/s with HZ=100, slow local reads)
>
> Vanilla performs better in worst case but I believe 40 is still low
> concerning test results below.

Can you check with this patch applied?

http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view

--
Jens Axboe

2008-11-10 13:53:39

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jens Axboe <[email protected]> writes:

> On Sun, Nov 09 2008, Vitaly V. Bursov wrote:
>> Hello,
>>
>> I'm building small server system with openvz kernel and have ran into
>> some IO performance problems. Reading a single file via NFS delivers
>> around 9 MB/s over gigabit network, but while reading, say, 2 different
>> or same file 2 times at the same time I get >60MB/s.
>>
>> Changing IO scheduler to deadline or anticipatory fixes problem.
>>
>> Tested kernels:
>> OpenVZ RHEL5 028stab059.3 (9 MB/s with HZ=100, 20MB/s with HZ=1000
>> fast local reads)
>> Vanilla 2.6.27.5 (40 MB/s with HZ=100, slow local reads)
>>
>> Vanilla performs better in worst case but I believe 40 is still low
>> concerning test results below.
>
> Can you check with this patch applied?
>
> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view

Funny, I was going to ask the same question. ;) The reason Jens wants
you to try this patch is that nfsd may be farming off the I/O requests
to different threads which are then performing interleaved I/O. The
above patch tries to detect this and allow cooperating processes to get
disk time instead of waiting for the idle timeout.

Cheers,

Jeff

2008-11-10 13:57:58

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Mon, Nov 10 2008, Jeff Moyer wrote:
> Jens Axboe <[email protected]> writes:
>
> > On Sun, Nov 09 2008, Vitaly V. Bursov wrote:
> >> Hello,
> >>
> >> I'm building small server system with openvz kernel and have ran into
> >> some IO performance problems. Reading a single file via NFS delivers
> >> around 9 MB/s over gigabit network, but while reading, say, 2 different
> >> or same file 2 times at the same time I get >60MB/s.
> >>
> >> Changing IO scheduler to deadline or anticipatory fixes problem.
> >>
> >> Tested kernels:
> >> OpenVZ RHEL5 028stab059.3 (9 MB/s with HZ=100, 20MB/s with HZ=1000
> >> fast local reads)
> >> Vanilla 2.6.27.5 (40 MB/s with HZ=100, slow local reads)
> >>
> >> Vanilla performs better in worst case but I believe 40 is still low
> >> concerning test results below.
> >
> > Can you check with this patch applied?
> >
> > http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
>
> Funny, I was going to ask the same question. ;) The reason Jens wants
> you to try this patch is that nfsd may be farming off the I/O requests
> to different threads which are then performing interleaved I/O. The
> above patch tries to detect this and allow cooperating processes to get
> disk time instead of waiting for the idle timeout.

Precisely :-)

The only reason I haven't merged it yet is because of worry of extra
cost, but I'll throw some SSD love at it and see how it turns out.

--
Jens Axboe

2008-11-10 17:16:25

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jens Axboe wrote:
> On Mon, Nov 10 2008, Jeff Moyer wrote:
>> Jens Axboe <[email protected]> writes:
>>
>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
>> Funny, I was going to ask the same question. ;) The reason Jens wants
>> you to try this patch is that nfsd may be farming off the I/O requests
>> to different threads which are then performing interleaved I/O. The
>> above patch tries to detect this and allow cooperating processes to get
>> disk time instead of waiting for the idle timeout.
>
> Precisely :-)
>
> The only reason I haven't merged it yet is because of worry of extra
> cost, but I'll throw some SSD love at it and see how it turns out.
>

Sorry, but I get "oops" same moment nfs read transfer starts.
I can get directory list via nfs, read files locally (not
carefully tested, though)

Dumps captured via netconsole, so these may not be completely accurate
but hopefully will give a hint.

Linux 2.6.27.5

======================================
[ 618.526302] BUG: unable to handle kernel
NULL pointer dereference
at 0000000000000000
[ 618.526676] IP:
[<ffffffff81111f16>] rb_erase+0x124/0x290
[ 618.526738] PGD 11b96c067
PUD 11b96d067
PMD 0

[ 618.526856] Oops: 0000 [1]
SMP

[ 618.527020] CPU 1

[ 618.527285] Modules linked in:
nfsd
auth_rpcgss
exportfs
netconsole
ipv6
nfs
lockd
nfs_acl
sunrpc
cypress_m8
usbserial
it87
hwmon_vid
hwmon
ppdev
ehci_hcd
ohci_hcd
sg
r8169
thermal
parport_pc
processor
parport
thermal_sys

[ 618.528868] Pid: 2184, comm: md1_raid5 Not tainted 2.6.27.5 #2
[ 618.528918] RIP: 0010:[<ffffffff81111f16>]
[<ffffffff81111f16>] rb_erase+0x124/0x290
[ 618.529087] RSP: 0018:ffff88011d9c9a58 EFLAGS: 00010046
[ 618.529136] RAX: ffff88011d3d7c01 RBX: ffff88011d3d7c00 RCX: 0000000000000000
[ 618.529187] RDX: 000000002d59fabb RSI: ffff88011e44d430 RDI: 0000000000000000
[ 618.529238] RBP: ffff88011e44d430 R08: 0000000000000000 R09: ffff88011c08d8b8
[ 618.529289] R10: 000000002d59fdbb R11: 000000002d59fbbb R12: ffff88011e44d400
[ 618.529339] R13: ffff88011e44d430 R14: ffff88011e44d400 R15: 0000000000800000
[ 618.529390] FS: 00007f6c86182750(0000) GS:ffff88011f86abc0(0000) knlGS:0000000000000000
[ 618.529480] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[ 618.529530] CR2: 0000000000000000 CR3: 000000011b9c6000 CR4: 00000000000006e0
[ 618.529580] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 618.529631] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 618.529682] Process md1_raid5 (pid: 2184, threadinfo ffff88011d9c8000, task ffff88011fa5d8e0)
[ 618.529773] Stack:
ffff88011d3d7660
ffff88011d3d7660
ffffffff8110a0b9
ffff88011d3d7630

[ 618.529895] ffffffff8110a0ff
ffff88011d3d7630
ffff88011e44d400
ffff880119a1cc48

[ 618.530085] ffff88011d3d7630
ffff88011e44d400
ffff880119a1cc48
ffff88011d3d7678

[ 618.530154] Call Trace:
[ 618.530252] [<ffffffff8110a0b9>] ? rb_erase_init+0x9/0x17
[ 618.530303] [<ffffffff8110a0ff>] ? cfq_prio_tree_add+0x38/0xa8
[ 618.530354] [<ffffffff8110b12b>] ? cfq_add_rq_rb+0xb5/0xc8
[ 618.530404] [<ffffffff8110b198>] ? cfq_insert_request+0x5a/0x356
[ 618.530457] [<ffffffff811000a1>] ? elv_insert+0x14b/0x218
[ 618.530510] [<ffffffff810ab757>] ? bio_phys_segments+0xf/0x15
[ 618.530561] [<ffffffff811028dc>] ? __make_request+0x3b9/0x3eb
[ 618.530612] [<ffffffff8110120c>] ? generic_make_request+0x30b/0x346
[ 618.530665] [<ffffffff811baae4>] ? raid5_end_write_request+0x0/0xb8
[ 618.530716] [<ffffffff811b8ace>] ? ops_run_io+0x16a/0x1c1
[ 618.530767] [<ffffffff811ba524>] ? handle_stripe5+0x9b5/0x9d6
[ 618.530818] [<ffffffff811bbef8>] ? handle_stripe+0xc3a/0xc6a
[ 618.530870] [<ffffffff81253782>] ? thread_return+0x3a/0xaa
[ 618.530921] [<ffffffff811bc2be>] ? raid5d+0x396/0x3cd
[ 618.530972] [<ffffffff81253bc8>] ? schedule_timeout+0x1e/0xad
[ 618.531023] [<ffffffff811c715f>] ? md_thread+0xdd/0xf9
[ 618.531075] [<ffffffff81044f9c>] ? autoremove_wake_function+0x0/0x2e
[ 618.531127] [<ffffffff811c7082>] ? md_thread+0x0/0xf9
[ 618.531177] [<ffffffff81044e80>] ? kthread+0x47/0x73
[ 618.531228] [<ffffffff8102f867>] ? schedule_tail+0x28/0x60
[ 618.531279] [<ffffffff8100cda9>] ? child_rip+0xa/0x11
[ 618.531330] [<ffffffff81044e39>] ? kthread+0x0/0x73
[ 618.531380] [<ffffffff8100cd9f>] ? child_rip+0x0/0x11
[ 618.531428]
[ 618.531474]
[ 618.531521] Code:
eb
0a
48
89
4b
08
eb
04
48
89
4d
00
41
ff
c8
0f
85
7f
01
00
00
e9
5d
01
00
00
48
8b
7b
10
48
39
cf
0f
85
a5
00
00
00
48
8b
7b
08

8b
07
a8
01
75
1a
48
83
c8
01
48
89
ee
48
89
07
48
83
23
fe

[ 618.532199] RIP
[<ffffffff81111f16>] rb_erase+0x124/0x290
[ 618.532510] RSP <ffff88011d9c9a58>
[ 618.532510] CR2: 0000000000000000
[ 618.532510] ---[ end trace db64b81474e1505e ]---

======================================

[ 334.271949] BUG: unable to handle kernel
NULL pointer dereference
at 0000000000000010
[ 334.272213] IP:
[<ffffffff81111bbf>] __rb_rotate_left+0x7/0x5b
[ 334.27 PGD 11c15d067
PUD 11c140067
PMD 0

[ 334.272508] Oops: 0000 [1]
SMP

[ 334.272508] CPU 1

[ 334.272508] Modules linked in:
nfsd
auth_rpcgss
exportfs
netconsole
ipv6
nfs
lockd
nfs_acl
sunrpc
cypress_m8
usbserial
it87
hwmon_vid
hwmon
ppdev
parport_pc
ehci_hcd
ohci_hcd
thermal
parport
sg
processor
r8169
thermal_sys

[ 334.272508] Pid: 2185, comm: md1_raid5 Not tainted 2.6.27.5 #2
[ 334.272508] RIP: 0010:[<ffffffff81111bbf>]
[<ffffffff81111bbf>] __rb_rotate_left+0x7/0x5b
[ 334.272508] RSP: 0018:ffff88011d591a48 EFLAGS: 00010082
[ 334.272508] RAX: ffff88011cc20a20 RBX: ffff88011cc20a20 RCX: 0000000000000000
[ 334.272508] RDX: 0000000000000000 RSI: ffff88011f8bae30 RDI: ffff88011cc20a20
[ 334.272508] RBP: ffff88011cc20a20 R08: ffff88011cc20a20 R09: ffff88011ac6b3f8
[ 334.272508] R10: 000000002d5a237b R11: 000000002d5a30bb R12: ffff88011cc20a20
[ 334.272508] R13: 0000000000000000 R14: ffff88011f8bae30 R15: 0000000000800010
[ 334.272508] FS: 00007fb5925496f0(0000) GS:ffff88011f86abc0(0000) knlGS:0000000000000000
[ 334.272508] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[ 334.272508] CR2: 0000000000000010 CR3: 000000011c18c000 CR4: 00000000000006e0
[ 334.272508] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 334.272508] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 334.272508] Process md1_raid5 (pid: 2185, threadinfo ffff88011d590000, task ffff88011fb12860)
[ 334.272508] Stack:
ffffffff81111d20
ffff88011cc209f0
ffff88011cc20a20
ffff88011f8bae00

[ 334.272508] ffff88011f8bae30
ffff88011f8bae00
ffffffff8110a164
ffff88011cc209f0

[ 334.272508] ffff88011cc20a20
ffff88011f8bae30
ffff88011cc209f0
ffff88011f8bae00

[ 334.272508] Call Trace:
[ 334.272508] [<ffffffff81111d20>] ? rb_insert_color+0xb2/0xda
[ 334.272508] [<ffffffff8110a164>] ? cfq_prio_tree_add+0x9d/0xa8
[ 334.272508] [<ffffffff8110b12b>] ? cfq_add_rq_rb+0xb5/0xc8
[ 334.272508] [<ffffffff8110b198>] ? cfq_insert_request+0x5a/0x356
[ 334.272508] [<ffffffff811000a1>] ? elv_insert+0x14b/0x218
[ 334.272508] [<ffffffff810ab757>] ? bio_phys_segments+0xf/0x15
[ 334.272508] [<ffffffff811028dc>] ? __make_request+0x3b9/0x3eb
[ 334.272508] [<ffffffff8110120c>] ? generic_make_request+0x30b/0x346
[ 334.272508] [<ffffffff811baae4>] ? raid5_end_write_request+0x0/0xb8
[ 334.272508] [<ffffffff811b8ace>] ? ops_run_io+0x16a/0x1c1
[ 334.272508] [<ffffffff811ba524>] ? handle_stripe5+0x9b5/0x9d6
[ 334.272508] [<ffffffff811bbef8>] ? handle_stripe+0xc3a/0xc6a
[ 334.272508] [<ffffffff810296e5>] ? pick_next_task_fair+0x8d/0x9c
[ 334.272508] [<ffffffff81253782>] ? thread_return+0x3a/0xaa
[ 334.272508] [<ffffffff811bc2be>] ? raid5d+0x396/0x3cd
[ 334.272508] [<ffffffff81253bc8>] ? schedule_timeout+0x1e/0xad
[ 334.272508] [<ffffffff811c715f>] ? md_thread+0xdd/0xf9
[ 334.272508] [<ffffffff81044f9c>] ? autoremove_wake_function+0x0/0x2e
[ 334.272508] [<ffffffff811c7082>] ? md_thread+0x0/0xf9
[ 334.272508] [<ffffffff81044e80>] ? kthread+0x47/0x73
[ 334.272508] [<ffffffff8102f867>] ? schedule_tail+0x28/0x60
[ 334.272508] [<ffffffff8100cda9>] ? child_rip+0xa/0x11
[ 334.272508] [<ffffffff81044e39>] ? kthread+0x0/0x73
[ 334.272508] [<ffffffff8100cd9f>] ? child_rip+0x0/0x11
[ 334.272508]
[ 334.272508]
[ 334.272508] Code:
00
31
c0
eb
19
ff
c0
48
89
ee
48
c7
c7
e8
39
44
81
89
43
08
e8
3a
33
14
00
b8
01
00
00
00
5a
5b
5d
c3
90
90
48
8b
4f
08
4c
8b
07

8b
51
10
49
83
e0
fc
48
85
d2
48
89
57
08
74
0c
48
8b
02
83

[ 334.272508] RIP
[<ffffffff81111bbf>] __rb_rotate_left+0x7/0x5b
[ 334.272508] RSP <ffff88011d591a48>
[ 334.272508] CR2: 0000000000000010
[ 334.272508] ---[ end trace f1bde8b7b9263879 ]---
======================================

--
Thanks,
Vitaly

2008-11-10 17:36:45

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
> Jens Axboe wrote:
> > On Mon, Nov 10 2008, Jeff Moyer wrote:
> >> Jens Axboe <[email protected]> writes:
> >>
> >>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
> >> Funny, I was going to ask the same question. ;) The reason Jens wants
> >> you to try this patch is that nfsd may be farming off the I/O requests
> >> to different threads which are then performing interleaved I/O. The
> >> above patch tries to detect this and allow cooperating processes to get
> >> disk time instead of waiting for the idle timeout.
> >
> > Precisely :-)
> >
> > The only reason I haven't merged it yet is because of worry of extra
> > cost, but I'll throw some SSD love at it and see how it turns out.
> >
>
> Sorry, but I get "oops" same moment nfs read transfer starts.
> I can get directory list via nfs, read files locally (not
> carefully tested, though)
>
> Dumps captured via netconsole, so these may not be completely accurate
> but hopefully will give a hint.

Interesting, strange how that hasn't triggered here. Or perhaps the
version that Jeff posted isn't the one I tried. Anyway, search for:

RB_CLEAR_NODE(&cfqq->rb_node);

and add a

RB_CLEAR_NODE(&cfqq->prio_node);

just below that. It's in cfq_find_alloc_queue(). I think that should fix
it.

--
Jens Axboe

2008-11-10 18:27:34

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jens Axboe wrote:
> On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
>> Jens Axboe wrote:
>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
>>>> Jens Axboe <[email protected]> writes:
>>>>
>>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
>>>> Funny, I was going to ask the same question. ;) The reason Jens wants
>>>> you to try this patch is that nfsd may be farming off the I/O requests
>>>> to different threads which are then performing interleaved I/O. The
>>>> above patch tries to detect this and allow cooperating processes to get
>>>> disk time instead of waiting for the idle timeout.
>>> Precisely :-)
>>>
>>> The only reason I haven't merged it yet is because of worry of extra
>>> cost, but I'll throw some SSD love at it and see how it turns out.
>>>
>> Sorry, but I get "oops" same moment nfs read transfer starts.
>> I can get directory list via nfs, read files locally (not
>> carefully tested, though)
>>
>> Dumps captured via netconsole, so these may not be completely accurate
>> but hopefully will give a hint.
>
> Interesting, strange how that hasn't triggered here. Or perhaps the
> version that Jeff posted isn't the one I tried. Anyway, search for:
>
> RB_CLEAR_NODE(&cfqq->rb_node);
>
> and add a
>
> RB_CLEAR_NODE(&cfqq->prio_node);
>
> just below that. It's in cfq_find_alloc_queue(). I think that should fix
> it.
>

Same problem.

I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
but It won't boot anymore with cfq with same error...

Switching cfq io scheduler at runtime (booting with "as") appears to work with
two parallel local dd reads.

But when NFS server starts up:

[ 469.000105] BUG: unable to handle kernel
NULL pointer dereference
at 0000000000000000
[ 469.000305] IP:
[<ffffffff81111f2a>] rb_erase+0x124/0x290
...

[ 469.001905] Pid: 2296, comm: md1_raid5 Not tainted 2.6.27.5 #4
[ 469.001982] RIP: 0010:[<ffffffff81111f2a>]
[<ffffffff81111f2a>] rb_erase+0x124/0x290
...
[ 469.002509] Call Trace:
[ 469.002509] [<ffffffff8110a0b9>] ? rb_erase_init+0x9/0x17
[ 469.002509] [<ffffffff8110a0ff>] ? cfq_prio_tree_add+0x38/0xa8
[ 469.002509] [<ffffffff8110b13d>] ? cfq_add_rq_rb+0xb5/0xc8
[ 469.002509] [<ffffffff8110b1aa>] ? cfq_insert_request+0x5a/0x356
[ 469.002509] [<ffffffff811000a1>] ? elv_insert+0x14b/0x218
[ 469.002509] [<ffffffff810ab757>] ? bio_phys_segments+0xf/0x15
[ 469.002509] [<ffffffff811028dc>] ? __make_request+0x3b9/0x3eb
[ 469.002509] [<ffffffff8110120c>] ? generic_make_request+0x30b/0x346
[ 469.002509] [<ffffffff811baaf4>] ? raid5_end_write_request+0x0/0xb8
[ 469.002509] [<ffffffff811b8ade>] ? ops_run_io+0x16a/0x1c1
[ 469.002509] [<ffffffff811ba534>] ? handle_stripe5+0x9b5/0x9d6
[ 469.002509] [<ffffffff811bbf08>] ? handle_stripe+0xc3a/0xc6a
[ 469.002509] [<ffffffff810296e5>] ? pick_next_task_fair+0x8d/0x9c
[ 469.002509] [<ffffffff81253792>] ? thread_return+0x3a/0xaa
[ 469.002509] [<ffffffff811bc2ce>] ? raid5d+0x396/0x3cd
[ 469.002509] [<ffffffff81253bd8>] ? schedule_timeout+0x1e/0xad
[ 469.002509] [<ffffffff811c716f>] ? md_thread+0xdd/0xf9
[ 469.002509] [<ffffffff81044f9c>] ? autoremove_wake_function+0x0/0x2e
[ 469.002509] [<ffffffff811c7092>] ? md_thread+0x0/0xf9
[ 469.002509] [<ffffffff81044e80>] ? kthread+0x47/0x73
[ 469.002509] [<ffffffff8102f867>] ? schedule_tail+0x28/0x60
[ 469.002509] [<ffffffff8100cda9>] ? child_rip+0xa/0x11
[ 469.002509] [<ffffffff81044e39>] ? kthread+0x0/0x73
[ 469.002509] [<ffffffff8100cd9f>] ? child_rip+0x0/0x11
...
[ 469.002509] RIP
[<ffffffff81111f2a>] rb_erase+0x124/0x290
[ 469.002509] RSP <ffff88011d4c7a58>
[ 469.002509] CR2: 0000000000000000
[ 469.002509] ---[ end trace acdef779aeb56048 ]---

"Best" result I got with NFS was
avg-cpu: %user %nice %system %iowait %steal %idle
0,00 0,00 0,20 0,65 0,00 99,15

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 11,30 0,00 7,60 0,00 245,60 0,00 32,32 0,01 1,18 0,79 0,60
sdb 12,10 0,00 8,00 0,00 246,40 0,00 30,80 0,01 1,62 0,62 0,50

and it lasted around 30 seconds.

--
Thanks,
Vitaly

2008-11-10 18:31:17

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
> Jens Axboe wrote:
> > On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
> >> Jens Axboe wrote:
> >>> On Mon, Nov 10 2008, Jeff Moyer wrote:
> >>>> Jens Axboe <[email protected]> writes:
> >>>>
> >>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
> >>>> Funny, I was going to ask the same question. ;) The reason Jens wants
> >>>> you to try this patch is that nfsd may be farming off the I/O requests
> >>>> to different threads which are then performing interleaved I/O. The
> >>>> above patch tries to detect this and allow cooperating processes to get
> >>>> disk time instead of waiting for the idle timeout.
> >>> Precisely :-)
> >>>
> >>> The only reason I haven't merged it yet is because of worry of extra
> >>> cost, but I'll throw some SSD love at it and see how it turns out.
> >>>
> >> Sorry, but I get "oops" same moment nfs read transfer starts.
> >> I can get directory list via nfs, read files locally (not
> >> carefully tested, though)
> >>
> >> Dumps captured via netconsole, so these may not be completely accurate
> >> but hopefully will give a hint.
> >
> > Interesting, strange how that hasn't triggered here. Or perhaps the
> > version that Jeff posted isn't the one I tried. Anyway, search for:
> >
> > RB_CLEAR_NODE(&cfqq->rb_node);
> >
> > and add a
> >
> > RB_CLEAR_NODE(&cfqq->prio_node);
> >
> > just below that. It's in cfq_find_alloc_queue(). I think that should fix
> > it.
> >
>
> Same problem.
>
> I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
> but It won't boot anymore with cfq with same error...
>
> Switching cfq io scheduler at runtime (booting with "as") appears to work with
> two parallel local dd reads.
>
> But when NFS server starts up:
>
> [ 469.000105] BUG: unable to handle kernel
> NULL pointer dereference
> at 0000000000000000
> [ 469.000305] IP:
> [<ffffffff81111f2a>] rb_erase+0x124/0x290
> ...
>
> [ 469.001905] Pid: 2296, comm: md1_raid5 Not tainted 2.6.27.5 #4
> [ 469.001982] RIP: 0010:[<ffffffff81111f2a>]
> [<ffffffff81111f2a>] rb_erase+0x124/0x290
> ...
> [ 469.002509] Call Trace:
> [ 469.002509] [<ffffffff8110a0b9>] ? rb_erase_init+0x9/0x17
> [ 469.002509] [<ffffffff8110a0ff>] ? cfq_prio_tree_add+0x38/0xa8
> [ 469.002509] [<ffffffff8110b13d>] ? cfq_add_rq_rb+0xb5/0xc8
> [ 469.002509] [<ffffffff8110b1aa>] ? cfq_insert_request+0x5a/0x356
> [ 469.002509] [<ffffffff811000a1>] ? elv_insert+0x14b/0x218
> [ 469.002509] [<ffffffff810ab757>] ? bio_phys_segments+0xf/0x15
> [ 469.002509] [<ffffffff811028dc>] ? __make_request+0x3b9/0x3eb
> [ 469.002509] [<ffffffff8110120c>] ? generic_make_request+0x30b/0x346
> [ 469.002509] [<ffffffff811baaf4>] ? raid5_end_write_request+0x0/0xb8
> [ 469.002509] [<ffffffff811b8ade>] ? ops_run_io+0x16a/0x1c1
> [ 469.002509] [<ffffffff811ba534>] ? handle_stripe5+0x9b5/0x9d6
> [ 469.002509] [<ffffffff811bbf08>] ? handle_stripe+0xc3a/0xc6a
> [ 469.002509] [<ffffffff810296e5>] ? pick_next_task_fair+0x8d/0x9c
> [ 469.002509] [<ffffffff81253792>] ? thread_return+0x3a/0xaa
> [ 469.002509] [<ffffffff811bc2ce>] ? raid5d+0x396/0x3cd
> [ 469.002509] [<ffffffff81253bd8>] ? schedule_timeout+0x1e/0xad
> [ 469.002509] [<ffffffff811c716f>] ? md_thread+0xdd/0xf9
> [ 469.002509] [<ffffffff81044f9c>] ? autoremove_wake_function+0x0/0x2e
> [ 469.002509] [<ffffffff811c7092>] ? md_thread+0x0/0xf9
> [ 469.002509] [<ffffffff81044e80>] ? kthread+0x47/0x73
> [ 469.002509] [<ffffffff8102f867>] ? schedule_tail+0x28/0x60
> [ 469.002509] [<ffffffff8100cda9>] ? child_rip+0xa/0x11
> [ 469.002509] [<ffffffff81044e39>] ? kthread+0x0/0x73
> [ 469.002509] [<ffffffff8100cd9f>] ? child_rip+0x0/0x11
> ...
> [ 469.002509] RIP
> [<ffffffff81111f2a>] rb_erase+0x124/0x290
> [ 469.002509] RSP <ffff88011d4c7a58>
> [ 469.002509] CR2: 0000000000000000
> [ 469.002509] ---[ end trace acdef779aeb56048 ]---
>
>
> "Best" result I got with NFS was
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0,00 0,00 0,20 0,65 0,00 99,15
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
> sda 11,30 0,00 7,60 0,00 245,60 0,00 32,32 0,01 1,18 0,79 0,60
> sdb 12,10 0,00 8,00 0,00 246,40 0,00 30,80 0,01 1,62 0,62 0,50
>
> and it lasted around 30 seconds.

OK, I'll throw some NFS at this patch in the morning and do some
measurements as well, so it can get queued up.

--
Jens Axboe

2008-11-10 18:39:29

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jens Axboe <[email protected]> writes:

> OK, I'll throw some NFS at this patch in the morning and do some
> measurements as well, so it can get queued up.

I'll take a look at this this afternoon.

Cheers,

Jeff

2008-11-10 18:44:24

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Mon, Nov 10 2008, Jens Axboe wrote:
> On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
> > Jens Axboe wrote:
> > > On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
> > >> Jens Axboe wrote:
> > >>> On Mon, Nov 10 2008, Jeff Moyer wrote:
> > >>>> Jens Axboe <[email protected]> writes:
> > >>>>
> > >>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
> > >>>> Funny, I was going to ask the same question. ;) The reason Jens wants
> > >>>> you to try this patch is that nfsd may be farming off the I/O requests
> > >>>> to different threads which are then performing interleaved I/O. The
> > >>>> above patch tries to detect this and allow cooperating processes to get
> > >>>> disk time instead of waiting for the idle timeout.
> > >>> Precisely :-)
> > >>>
> > >>> The only reason I haven't merged it yet is because of worry of extra
> > >>> cost, but I'll throw some SSD love at it and see how it turns out.
> > >>>
> > >> Sorry, but I get "oops" same moment nfs read transfer starts.
> > >> I can get directory list via nfs, read files locally (not
> > >> carefully tested, though)
> > >>
> > >> Dumps captured via netconsole, so these may not be completely accurate
> > >> but hopefully will give a hint.
> > >
> > > Interesting, strange how that hasn't triggered here. Or perhaps the
> > > version that Jeff posted isn't the one I tried. Anyway, search for:
> > >
> > > RB_CLEAR_NODE(&cfqq->rb_node);
> > >
> > > and add a
> > >
> > > RB_CLEAR_NODE(&cfqq->prio_node);
> > >
> > > just below that. It's in cfq_find_alloc_queue(). I think that should fix
> > > it.
> > >
> >
> > Same problem.
> >
> > I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
> > but It won't boot anymore with cfq with same error...
> >
> > Switching cfq io scheduler at runtime (booting with "as") appears to work with
> > two parallel local dd reads.
> >
> > But when NFS server starts up:
> >
> > [ 469.000105] BUG: unable to handle kernel
> > NULL pointer dereference
> > at 0000000000000000
> > [ 469.000305] IP:
> > [<ffffffff81111f2a>] rb_erase+0x124/0x290
> > ...
> >
> > [ 469.001905] Pid: 2296, comm: md1_raid5 Not tainted 2.6.27.5 #4
> > [ 469.001982] RIP: 0010:[<ffffffff81111f2a>]
> > [<ffffffff81111f2a>] rb_erase+0x124/0x290
> > ...
> > [ 469.002509] Call Trace:
> > [ 469.002509] [<ffffffff8110a0b9>] ? rb_erase_init+0x9/0x17
> > [ 469.002509] [<ffffffff8110a0ff>] ? cfq_prio_tree_add+0x38/0xa8
> > [ 469.002509] [<ffffffff8110b13d>] ? cfq_add_rq_rb+0xb5/0xc8
> > [ 469.002509] [<ffffffff8110b1aa>] ? cfq_insert_request+0x5a/0x356
> > [ 469.002509] [<ffffffff811000a1>] ? elv_insert+0x14b/0x218
> > [ 469.002509] [<ffffffff810ab757>] ? bio_phys_segments+0xf/0x15
> > [ 469.002509] [<ffffffff811028dc>] ? __make_request+0x3b9/0x3eb
> > [ 469.002509] [<ffffffff8110120c>] ? generic_make_request+0x30b/0x346
> > [ 469.002509] [<ffffffff811baaf4>] ? raid5_end_write_request+0x0/0xb8
> > [ 469.002509] [<ffffffff811b8ade>] ? ops_run_io+0x16a/0x1c1
> > [ 469.002509] [<ffffffff811ba534>] ? handle_stripe5+0x9b5/0x9d6
> > [ 469.002509] [<ffffffff811bbf08>] ? handle_stripe+0xc3a/0xc6a
> > [ 469.002509] [<ffffffff810296e5>] ? pick_next_task_fair+0x8d/0x9c
> > [ 469.002509] [<ffffffff81253792>] ? thread_return+0x3a/0xaa
> > [ 469.002509] [<ffffffff811bc2ce>] ? raid5d+0x396/0x3cd
> > [ 469.002509] [<ffffffff81253bd8>] ? schedule_timeout+0x1e/0xad
> > [ 469.002509] [<ffffffff811c716f>] ? md_thread+0xdd/0xf9
> > [ 469.002509] [<ffffffff81044f9c>] ? autoremove_wake_function+0x0/0x2e
> > [ 469.002509] [<ffffffff811c7092>] ? md_thread+0x0/0xf9
> > [ 469.002509] [<ffffffff81044e80>] ? kthread+0x47/0x73
> > [ 469.002509] [<ffffffff8102f867>] ? schedule_tail+0x28/0x60
> > [ 469.002509] [<ffffffff8100cda9>] ? child_rip+0xa/0x11
> > [ 469.002509] [<ffffffff81044e39>] ? kthread+0x0/0x73
> > [ 469.002509] [<ffffffff8100cd9f>] ? child_rip+0x0/0x11
> > ...
> > [ 469.002509] RIP
> > [<ffffffff81111f2a>] rb_erase+0x124/0x290
> > [ 469.002509] RSP <ffff88011d4c7a58>
> > [ 469.002509] CR2: 0000000000000000
> > [ 469.002509] ---[ end trace acdef779aeb56048 ]---
> >
> >
> > "Best" result I got with NFS was
> > avg-cpu: %user %nice %system %iowait %steal %idle
> > 0,00 0,00 0,20 0,65 0,00 99,15
> >
> > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
> > sda 11,30 0,00 7,60 0,00 245,60 0,00 32,32 0,01 1,18 0,79 0,60
> > sdb 12,10 0,00 8,00 0,00 246,40 0,00 30,80 0,01 1,62 0,62 0,50
> >
> > and it lasted around 30 seconds.
>
> OK, I'll throw some NFS at this patch in the morning and do some
> measurements as well, so it can get queued up.

I spotted a bug - if the ioprio of a process gets changed, it needs to
be repositioned in the cooperator tree or we'll end up doing erase on a
wrong root. Perhaps that is what is biting you here.

--
Jens Axboe

2008-11-10 21:54:18

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

"Vitaly V. Bursov" <[email protected]> writes:

> Jens Axboe wrote:
>> On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
>>> Jens Axboe wrote:
>>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
>>>>> Jens Axboe <[email protected]> writes:
>>>>>
>>>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
>>>>> Funny, I was going to ask the same question. ;) The reason Jens wants
>>>>> you to try this patch is that nfsd may be farming off the I/O requests
>>>>> to different threads which are then performing interleaved I/O. The
>>>>> above patch tries to detect this and allow cooperating processes to get
>>>>> disk time instead of waiting for the idle timeout.
>>>> Precisely :-)
>>>>
>>>> The only reason I haven't merged it yet is because of worry of extra
>>>> cost, but I'll throw some SSD love at it and see how it turns out.
>>>>
>>> Sorry, but I get "oops" same moment nfs read transfer starts.
>>> I can get directory list via nfs, read files locally (not
>>> carefully tested, though)
>>>
>>> Dumps captured via netconsole, so these may not be completely accurate
>>> but hopefully will give a hint.
>>
>> Interesting, strange how that hasn't triggered here. Or perhaps the
>> version that Jeff posted isn't the one I tried. Anyway, search for:
>>
>> RB_CLEAR_NODE(&cfqq->rb_node);
>>
>> and add a
>>
>> RB_CLEAR_NODE(&cfqq->prio_node);
>>
>> just below that. It's in cfq_find_alloc_queue(). I think that should fix
>> it.
>>
>
> Same problem.
>
> I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
> but It won't boot anymore with cfq with same error...
>
> Switching cfq io scheduler at runtime (booting with "as") appears to work with
> two parallel local dd reads.

Strange, I can't reproduce a failure. I'll keep trying. For now, these
are the results I see:

[root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
[root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 26.8128 s, 40.0 MB/s
[root@maiden ~]# umount /mnt/megadeth/
[root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
[root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 23.7025 s, 45.3 MB/s
[root@maiden ~]# umount /mnt/megadeth/

Here is the patch, with the suggestion from Jens to switch the cfqq to
the right priority tree when the priority is changed.

Cheers,

Jeff

diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 6a062ee..dd26cba 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -83,6 +83,12 @@ struct cfq_data {
* rr list of queues with requests and the count of them
*/
struct cfq_rb_root service_tree;
+ /*
+ * Each priority tree is sorted by next_request position. These
+ * trees are used when determining if two or more queues are
+ * interleaving requests (see cfq_close_cooperator).
+ */
+ struct rb_root prio_trees[CFQ_PRIO_LISTS];
unsigned int busy_queues;

int rq_in_driver;
@@ -142,6 +148,8 @@ struct cfq_queue {
struct rb_node rb_node;
/* service_tree key */
unsigned long rb_key;
+ /* prio tree member */
+ struct rb_node prio_node;
/* sorted list of pending requests */
struct rb_root sort_list;
/* if fifo isn't expired, next request to serve */
@@ -415,13 +423,17 @@ static struct cfq_queue *cfq_rb_first(struct cfq_rb_root *root)
return NULL;
}

+static void rb_erase_init(struct rb_node *n, struct rb_root *root)
+{
+ rb_erase(n, root);
+ RB_CLEAR_NODE(n);
+}
+
static void cfq_rb_erase(struct rb_node *n, struct cfq_rb_root *root)
{
if (root->left == n)
root->left = NULL;
-
- rb_erase(n, &root->rb);
- RB_CLEAR_NODE(n);
+ rb_erase_init(n, &root->rb);
}

/*
@@ -540,6 +552,62 @@ static void cfq_service_tree_add(struct cfq_data *cfqd,
rb_insert_color(&cfqq->rb_node, &cfqd->service_tree.rb);
}

+static struct cfq_queue *cfq_prio_tree_lookup(struct cfq_data *cfqd,
+ int ioprio, sector_t sector, struct rb_node **ret_parent,
+ struct rb_node ***rb_link)
+{
+ struct rb_root *root = &cfqd->prio_trees[ioprio];
+ struct rb_node **p, *parent;
+ struct cfq_queue *cfqq = NULL;
+
+ parent = NULL;
+ p = &root->rb_node;
+ while (*p) {
+ struct rb_node **n;
+
+ parent = *p;
+ cfqq = rb_entry(parent, struct cfq_queue, prio_node);
+
+ /*
+ * Sort strictly based on sector. Smallest to the left,
+ * largest to the right.
+ */
+ if (sector > cfqq->next_rq->sector)
+ n = &(*p)->rb_right;
+ else if (sector < cfqq->next_rq->sector)
+ n = &(*p)->rb_left;
+ else
+ break;
+ p = n;
+ }
+
+ *ret_parent = parent;
+ if (rb_link)
+ *rb_link = p;
+ return NULL;
+}
+
+static void cfq_prio_tree_add(struct cfq_data *cfqd, struct cfq_queue *cfqq)
+{
+ struct rb_root *root = &cfqd->prio_trees[cfqq->ioprio];
+ struct rb_node **p, *parent;
+ struct cfq_queue *__cfqq;
+
+ if (!RB_EMPTY_NODE(&cfqq->prio_node))
+ rb_erase_init(&cfqq->prio_node, root);
+
+ if (cfq_class_idle(cfqq))
+ return;
+ if (!cfqq->next_rq)
+ return;
+
+ __cfqq = cfq_prio_tree_lookup(cfqd, cfqq->ioprio, cfqq->next_rq->sector, &parent, &p);
+ BUG_ON(__cfqq);
+
+ rb_link_node(&cfqq->prio_node, parent, p);
+ rb_insert_color(&cfqq->prio_node, root);
+}
+
/*
* Update cfqq's position in the service tree.
*/
@@ -548,8 +616,10 @@ static void cfq_resort_rr_list(struct cfq_data *cfqd, struct cfq_queue *cfqq)
/*
* Resorting requires the cfqq to be on the RR list already.
*/
- if (cfq_cfqq_on_rr(cfqq))
+ if (cfq_cfqq_on_rr(cfqq)) {
cfq_service_tree_add(cfqd, cfqq, 0);
+ cfq_prio_tree_add(cfqd, cfqq);
+ }
}

/*
@@ -578,6 +648,8 @@ static void cfq_del_cfqq_rr(struct cfq_data *cfqd, struct cfq_queue *cfqq)

if (!RB_EMPTY_NODE(&cfqq->rb_node))
cfq_rb_erase(&cfqq->rb_node, &cfqd->service_tree);
+ if (!RB_EMPTY_NODE(&cfqq->prio_node))
+ rb_erase_init(&cfqq->prio_node, &cfqd->prio_trees[cfqq->ioprio]);

BUG_ON(!cfqd->busy_queues);
cfqd->busy_queues--;
@@ -623,6 +695,9 @@ static void cfq_add_rq_rb(struct request *rq)
* check if this request is a better next-serve candidate
*/
cfqq->next_rq = cfq_choose_req(cfqd, cfqq->next_rq, rq);
+ /* this changes the position in the priority tree */
+ cfq_prio_tree_add(cfqd, cfqq);
+
BUG_ON(!cfqq->next_rq);
}

@@ -831,11 +906,11 @@ static struct cfq_queue *cfq_get_next_queue(struct cfq_data *cfqd)
/*
* Get and set a new active queue for service.
*/
-static struct cfq_queue *cfq_set_active_queue(struct cfq_data *cfqd)
+static struct cfq_queue *cfq_set_active_queue(struct cfq_data *cfqd,
+ struct cfq_queue *cfqq)
{
- struct cfq_queue *cfqq;
-
- cfqq = cfq_get_next_queue(cfqd);
+ if (!cfqq)
+ cfqq = cfq_get_next_queue(cfqd);
__cfq_set_active_queue(cfqd, cfqq);
return cfqq;
}
@@ -859,15 +934,77 @@ static inline int cfq_rq_close(struct cfq_data *cfqd, struct request *rq)
return cfq_dist_from_last(cfqd, rq) <= cic->seek_mean;
}

-static int cfq_close_cooperator(struct cfq_data *cfq_data,
- struct cfq_queue *cfqq)
+static struct cfq_queue *cfqq_close(struct cfq_data *cfqd,
+ struct cfq_queue *cur_cfqq)
{
+ struct rb_root *root = &cfqd->prio_trees[cur_cfqq->ioprio];
+ struct rb_node *parent, *node;
+ struct cfq_queue *__cfqq;
+ sector_t sector = cfqd->last_position;
+
+ if (RB_EMPTY_ROOT(root))
+ return NULL;
+
+ /*
+ * First, if we find a request starting at the end of the last
+ * request, choose it.
+ */
+ __cfqq = cfq_prio_tree_lookup(cfqd, cur_cfqq->ioprio,
+ sector, &parent, NULL);
+ if (__cfqq)
+ return __cfqq;
+
+ /*
+ * If the exact sector wasn't found, the parent of the NULL leaf
+ * will contain the closest sector.
+ */
+ __cfqq = rb_entry(parent, struct cfq_queue, prio_node);
+ if (cfq_rq_close(cfqd, __cfqq->next_rq))
+ return __cfqq;
+ if (__cfqq->next_rq->sector < sector)
+ node = rb_next(&__cfqq->prio_node);
+ else
+ node = rb_prev(&__cfqq->prio_node);
+ if (!node)
+ return NULL;
+ __cfqq = rb_entry(node, struct cfq_queue, prio_node);
+ if (cfq_rq_close(cfqd, __cfqq->next_rq))
+ return __cfqq;
+
+ return NULL;
+}
+
+/*
+ * cfqd - obvious
+ * cur_cfqq - passed in so that we don't decide that the current queue is
+ * closely cooperating with itself.
+ *
+ * So, basically we're assuming that that cur_cfqq has dispatched at least
+ * one request, and that cfqd->last_position reflects a position on the disk
+ * associated with the I/O issued by cur_cfqq. I'm not sure this is a valid
+ * assumption.
+ */
+static struct cfq_queue *cfq_close_cooperator(struct cfq_data *cfqd,
+ struct cfq_queue *cur_cfqq)
+{
+ struct cfq_queue *cfqq;
+
+ /*
+ * A valid cfq_io_context is necessary to compare requests against
+ * the seek_mean of the current cfqq.
+ */
+ if (!cfqd->active_cic)
+ return NULL;
+
/*
* We should notice if some of the queues are cooperating, eg
* working closely on the same area of the disk. In that case,
* we can group them together and don't waste time idling.
*/
- return 0;
+ if ((cfqq = cfqq_close(cfqd, cur_cfqq)))
+ return cfqq;
+
+ return NULL;
}

#define CIC_SEEKY(cic) ((cic)->seek_mean > (8 * 1024))
@@ -908,13 +1045,6 @@ static void cfq_arm_slice_timer(struct cfq_data *cfqd)
if (!cic || !atomic_read(&cic->ioc->nr_tasks))
return;

- /*
- * See if this prio level has a good candidate
- */
- if (cfq_close_cooperator(cfqd, cfqq) &&
- (sample_valid(cic->ttime_samples) && cic->ttime_mean > 2))
- return;
-
cfq_mark_cfqq_must_dispatch(cfqq);
cfq_mark_cfqq_wait_request(cfqq);

@@ -992,7 +1122,7 @@ cfq_prio_to_maxrq(struct cfq_data *cfqd, struct cfq_queue *cfqq)
*/
static struct cfq_queue *cfq_select_queue(struct cfq_data *cfqd)
{
- struct cfq_queue *cfqq;
+ struct cfq_queue *cfqq, *new_cfqq = NULL;

cfqq = cfqd->active_queue;
if (!cfqq)
@@ -1012,6 +1142,15 @@ static struct cfq_queue *cfq_select_queue(struct cfq_data *cfqd)
goto keep_queue;

/*
+ * If another queue has a request waiting within our mean seek
+ * distance, let it run. The expire code will check for close
+ * cooperators and put the close queue at the front of the service
+ * tree.
+ */
+ if ((new_cfqq = cfq_close_cooperator(cfqd, cfqq)))
+ goto expire;
+
+ /*
* No requests pending. If the active queue still has requests in
* flight or is idling for a new request, allow either of these
* conditions to happen (or time out) before selecting a new queue.
@@ -1025,7 +1164,7 @@ static struct cfq_queue *cfq_select_queue(struct cfq_data *cfqd)
expire:
cfq_slice_expired(cfqd, 0);
new_queue:
- cfqq = cfq_set_active_queue(cfqd);
+ cfqq = cfq_set_active_queue(cfqd, new_cfqq);
keep_queue:
return cfqq;
}
@@ -1466,6 +1605,7 @@ retry:
}

RB_CLEAR_NODE(&cfqq->rb_node);
+ RB_CLEAR_NODE(&cfqq->prio_node);
INIT_LIST_HEAD(&cfqq->fifo);

atomic_set(&cfqq->ref, 0);
@@ -1949,7 +2089,18 @@ static void cfq_completed_request(struct request_queue *q, struct request *rq)
cfq_set_prio_slice(cfqd, cfqq);
cfq_clear_cfqq_slice_new(cfqq);
}
- if (cfq_slice_used(cfqq) || cfq_class_idle(cfqq))
+ /*
+ * If there are no requests waiting in this queue, and
+ * there are other queues ready to issue requests, AND
+ * those other queues are issuing requests within our
+ * mean seek distance, give them a chance to run instead
+ * of idling.
+ */
+ if (RB_EMPTY_ROOT(&cfqq->sort_list) &&
+ !RB_EMPTY_ROOT(&cfqd->service_tree.rb)) {
+ if (cfq_close_cooperator(cfqd, cfqq))
+ cfq_slice_expired(cfqd, 1);
+ } else if (cfq_slice_used(cfqq) || cfq_class_idle(cfqq))
cfq_slice_expired(cfqd, 1);
else if (sync && RB_EMPTY_ROOT(&cfqq->sort_list))
cfq_arm_slice_timer(cfqd);
@@ -1983,6 +2134,7 @@ static void cfq_prio_boost(struct cfq_queue *cfqq)
if (cfqq->ioprio != cfqq->org_ioprio)
cfqq->ioprio = cfqq->org_ioprio;
}
+ cfq_prio_tree_add(cfqq->cfqd, cfqq);
}

static inline int __cfq_may_queue(struct cfq_queue *cfqq)

2008-11-11 09:36:17

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Mon, Nov 10 2008, Jeff Moyer wrote:
> "Vitaly V. Bursov" <[email protected]> writes:
>
> > Jens Axboe wrote:
> >> On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
> >>> Jens Axboe wrote:
> >>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
> >>>>> Jens Axboe <[email protected]> writes:
> >>>>>
> >>>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
> >>>>> Funny, I was going to ask the same question. ;) The reason Jens wants
> >>>>> you to try this patch is that nfsd may be farming off the I/O requests
> >>>>> to different threads which are then performing interleaved I/O. The
> >>>>> above patch tries to detect this and allow cooperating processes to get
> >>>>> disk time instead of waiting for the idle timeout.
> >>>> Precisely :-)
> >>>>
> >>>> The only reason I haven't merged it yet is because of worry of extra
> >>>> cost, but I'll throw some SSD love at it and see how it turns out.
> >>>>
> >>> Sorry, but I get "oops" same moment nfs read transfer starts.
> >>> I can get directory list via nfs, read files locally (not
> >>> carefully tested, though)
> >>>
> >>> Dumps captured via netconsole, so these may not be completely accurate
> >>> but hopefully will give a hint.
> >>
> >> Interesting, strange how that hasn't triggered here. Or perhaps the
> >> version that Jeff posted isn't the one I tried. Anyway, search for:
> >>
> >> RB_CLEAR_NODE(&cfqq->rb_node);
> >>
> >> and add a
> >>
> >> RB_CLEAR_NODE(&cfqq->prio_node);
> >>
> >> just below that. It's in cfq_find_alloc_queue(). I think that should fix
> >> it.
> >>
> >
> > Same problem.
> >
> > I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
> > but It won't boot anymore with cfq with same error...
> >
> > Switching cfq io scheduler at runtime (booting with "as") appears to work with
> > two parallel local dd reads.
>
> Strange, I can't reproduce a failure. I'll keep trying. For now, these
> are the results I see:
>
> [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
> [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 26.8128 s, 40.0 MB/s
> [root@maiden ~]# umount /mnt/megadeth/
> [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
> [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 23.7025 s, 45.3 MB/s
> [root@maiden ~]# umount /mnt/megadeth/
>
> Here is the patch, with the suggestion from Jens to switch the cfqq to
> the right priority tree when the priority is changed.

I don't see the issue here either. Vitaly, are you using any openvz
kernel patches? IIRC, they patch cfq so it could just be that your cfq
version is incompatible with Jeff's patch.

--
Jens Axboe

2008-11-11 09:37:28

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Tue, Nov 11 2008, Jens Axboe wrote:
> On Mon, Nov 10 2008, Jeff Moyer wrote:
> > "Vitaly V. Bursov" <[email protected]> writes:
> >
> > > Jens Axboe wrote:
> > >> On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
> > >>> Jens Axboe wrote:
> > >>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
> > >>>>> Jens Axboe <[email protected]> writes:
> > >>>>>
> > >>>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
> > >>>>> Funny, I was going to ask the same question. ;) The reason Jens wants
> > >>>>> you to try this patch is that nfsd may be farming off the I/O requests
> > >>>>> to different threads which are then performing interleaved I/O. The
> > >>>>> above patch tries to detect this and allow cooperating processes to get
> > >>>>> disk time instead of waiting for the idle timeout.
> > >>>> Precisely :-)
> > >>>>
> > >>>> The only reason I haven't merged it yet is because of worry of extra
> > >>>> cost, but I'll throw some SSD love at it and see how it turns out.
> > >>>>
> > >>> Sorry, but I get "oops" same moment nfs read transfer starts.
> > >>> I can get directory list via nfs, read files locally (not
> > >>> carefully tested, though)
> > >>>
> > >>> Dumps captured via netconsole, so these may not be completely accurate
> > >>> but hopefully will give a hint.
> > >>
> > >> Interesting, strange how that hasn't triggered here. Or perhaps the
> > >> version that Jeff posted isn't the one I tried. Anyway, search for:
> > >>
> > >> RB_CLEAR_NODE(&cfqq->rb_node);
> > >>
> > >> and add a
> > >>
> > >> RB_CLEAR_NODE(&cfqq->prio_node);
> > >>
> > >> just below that. It's in cfq_find_alloc_queue(). I think that should fix
> > >> it.
> > >>
> > >
> > > Same problem.
> > >
> > > I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
> > > but It won't boot anymore with cfq with same error...
> > >
> > > Switching cfq io scheduler at runtime (booting with "as") appears to work with
> > > two parallel local dd reads.
> >
> > Strange, I can't reproduce a failure. I'll keep trying. For now, these
> > are the results I see:
> >
> > [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
> > [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
> > 1024+0 records in
> > 1024+0 records out
> > 1073741824 bytes (1.1 GB) copied, 26.8128 s, 40.0 MB/s
> > [root@maiden ~]# umount /mnt/megadeth/
> > [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
> > [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
> > 1024+0 records in
> > 1024+0 records out
> > 1073741824 bytes (1.1 GB) copied, 23.7025 s, 45.3 MB/s
> > [root@maiden ~]# umount /mnt/megadeth/
> >
> > Here is the patch, with the suggestion from Jens to switch the cfqq to
> > the right priority tree when the priority is changed.
>
> I don't see the issue here either. Vitaly, are you using any openvz
> kernel patches? IIRC, they patch cfq so it could just be that your cfq
> version is incompatible with Jeff's patch.

Heh, got it to trigger about 3 seconds after sending that email! I'll
look more into it.

--
Jens Axboe

2008-11-11 11:54:21

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Tue, Nov 11 2008, Jens Axboe wrote:
> On Tue, Nov 11 2008, Jens Axboe wrote:
> > On Mon, Nov 10 2008, Jeff Moyer wrote:
> > > "Vitaly V. Bursov" <[email protected]> writes:
> > >
> > > > Jens Axboe wrote:
> > > >> On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
> > > >>> Jens Axboe wrote:
> > > >>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
> > > >>>>> Jens Axboe <[email protected]> writes:
> > > >>>>>
> > > >>>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
> > > >>>>> Funny, I was going to ask the same question. ;) The reason Jens wants
> > > >>>>> you to try this patch is that nfsd may be farming off the I/O requests
> > > >>>>> to different threads which are then performing interleaved I/O. The
> > > >>>>> above patch tries to detect this and allow cooperating processes to get
> > > >>>>> disk time instead of waiting for the idle timeout.
> > > >>>> Precisely :-)
> > > >>>>
> > > >>>> The only reason I haven't merged it yet is because of worry of extra
> > > >>>> cost, but I'll throw some SSD love at it and see how it turns out.
> > > >>>>
> > > >>> Sorry, but I get "oops" same moment nfs read transfer starts.
> > > >>> I can get directory list via nfs, read files locally (not
> > > >>> carefully tested, though)
> > > >>>
> > > >>> Dumps captured via netconsole, so these may not be completely accurate
> > > >>> but hopefully will give a hint.
> > > >>
> > > >> Interesting, strange how that hasn't triggered here. Or perhaps the
> > > >> version that Jeff posted isn't the one I tried. Anyway, search for:
> > > >>
> > > >> RB_CLEAR_NODE(&cfqq->rb_node);
> > > >>
> > > >> and add a
> > > >>
> > > >> RB_CLEAR_NODE(&cfqq->prio_node);
> > > >>
> > > >> just below that. It's in cfq_find_alloc_queue(). I think that should fix
> > > >> it.
> > > >>
> > > >
> > > > Same problem.
> > > >
> > > > I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
> > > > but It won't boot anymore with cfq with same error...
> > > >
> > > > Switching cfq io scheduler at runtime (booting with "as") appears to work with
> > > > two parallel local dd reads.
> > >
> > > Strange, I can't reproduce a failure. I'll keep trying. For now, these
> > > are the results I see:
> > >
> > > [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
> > > [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
> > > 1024+0 records in
> > > 1024+0 records out
> > > 1073741824 bytes (1.1 GB) copied, 26.8128 s, 40.0 MB/s
> > > [root@maiden ~]# umount /mnt/megadeth/
> > > [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
> > > [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
> > > 1024+0 records in
> > > 1024+0 records out
> > > 1073741824 bytes (1.1 GB) copied, 23.7025 s, 45.3 MB/s
> > > [root@maiden ~]# umount /mnt/megadeth/
> > >
> > > Here is the patch, with the suggestion from Jens to switch the cfqq to
> > > the right priority tree when the priority is changed.
> >
> > I don't see the issue here either. Vitaly, are you using any openvz
> > kernel patches? IIRC, they patch cfq so it could just be that your cfq
> > version is incompatible with Jeff's patch.
>
> Heh, got it to trigger about 3 seconds after sending that email! I'll
> look more into it.

OK, found the issue. A few bugs there... cfq_prio_tree_lookup() doesn't
even return a hit, since it just breaks and returns NULL always. That
can cause cfq_prio_tree_add() to screw up the rbtree. The code to
correct on ioprio change wasn't correct either, I changed that as well.
New patch below, Vitaly can you give it a spin?

diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 6a062ee..b1f1b47 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -83,6 +83,12 @@ struct cfq_data {
* rr list of queues with requests and the count of them
*/
struct cfq_rb_root service_tree;
+ /*
+ * Each priority tree is sorted by next_request position. These
+ * trees are used when determining if two or more queues are
+ * interleaving requests (see cfq_close_cooperator).
+ */
+ struct rb_root prio_trees[CFQ_PRIO_LISTS];
unsigned int busy_queues;

int rq_in_driver;
@@ -142,6 +148,9 @@ struct cfq_queue {
struct rb_node rb_node;
/* service_tree key */
unsigned long rb_key;
+ /* prio tree member */
+ struct rb_node prio_node;
+ struct rb_root *prio_root;
/* sorted list of pending requests */
struct rb_root sort_list;
/* if fifo isn't expired, next request to serve */
@@ -415,13 +424,17 @@ static struct cfq_queue *cfq_rb_first(struct cfq_rb_root *root)
return NULL;
}

+static void rb_erase_init(struct rb_node *n, struct rb_root *root)
+{
+ rb_erase(n, root);
+ RB_CLEAR_NODE(n);
+}
+
static void cfq_rb_erase(struct rb_node *n, struct cfq_rb_root *root)
{
if (root->left == n)
root->left = NULL;
-
- rb_erase(n, &root->rb);
- RB_CLEAR_NODE(n);
+ rb_erase_init(n, &root->rb);
}

/*
@@ -540,6 +553,67 @@ static void cfq_service_tree_add(struct cfq_data *cfqd,
rb_insert_color(&cfqq->rb_node, &cfqd->service_tree.rb);
}

+static struct cfq_queue *cfq_prio_tree_lookup(struct cfq_data *cfqd,
+ int ioprio, sector_t sector, struct rb_node **ret_parent,
+ struct rb_node ***rb_link)
+{
+ struct rb_root *root = &cfqd->prio_trees[ioprio];
+ struct rb_node **p, *parent;
+ struct cfq_queue *cfqq = NULL;
+
+ parent = NULL;
+ p = &root->rb_node;
+ while (*p) {
+ struct rb_node **n;
+
+ parent = *p;
+ cfqq = rb_entry(parent, struct cfq_queue, prio_node);
+
+ /*
+ * Sort strictly based on sector. Smallest to the left,
+ * largest to the right.
+ */
+ if (sector > cfqq->next_rq->sector)
+ n = &(*p)->rb_right;
+ else if (sector < cfqq->next_rq->sector)
+ n = &(*p)->rb_left;
+ else
+ return cfqq;
+ p = n;
+ }
+
+ *ret_parent = parent;
+ if (rb_link)
+ *rb_link = p;
+ return NULL;
+}
+
+static void cfq_prio_tree_add(struct cfq_data *cfqd, struct cfq_queue *cfqq)
+{
+ struct rb_node **p, *parent;
+ struct cfq_queue *__cfqq;
+
+ if (cfqq->prio_root) {
+ rb_erase_init(&cfqq->prio_node, cfqq->prio_root);
+ cfqq->prio_root = NULL;
+ }
+
+ if (cfq_class_idle(cfqq))
+ return;
+ if (!cfqq->next_rq)
+ return;
+
+ /*
+ * If an alias is found, don't bother adding this one
+ */
+ __cfqq = cfq_prio_tree_lookup(cfqd, cfqq->ioprio, cfqq->next_rq->sector, &parent, &p);
+ if (!__cfqq) {
+ cfqq->prio_root = &cfqd->prio_trees[cfqq->ioprio];
+ rb_link_node(&cfqq->prio_node, parent, p);
+ rb_insert_color(&cfqq->prio_node, cfqq->prio_root);
+ }
+}
+
/*
* Update cfqq's position in the service tree.
*/
@@ -548,8 +622,10 @@ static void cfq_resort_rr_list(struct cfq_data *cfqd, struct cfq_queue *cfqq)
/*
* Resorting requires the cfqq to be on the RR list already.
*/
- if (cfq_cfqq_on_rr(cfqq))
+ if (cfq_cfqq_on_rr(cfqq)) {
cfq_service_tree_add(cfqd, cfqq, 0);
+ cfq_prio_tree_add(cfqd, cfqq);
+ }
}

/*
@@ -578,6 +654,10 @@ static void cfq_del_cfqq_rr(struct cfq_data *cfqd, struct cfq_queue *cfqq)

if (!RB_EMPTY_NODE(&cfqq->rb_node))
cfq_rb_erase(&cfqq->rb_node, &cfqd->service_tree);
+ if (cfqq->prio_root) {
+ rb_erase_init(&cfqq->prio_node, cfqq->prio_root);
+ cfqq->prio_root = NULL;
+ }

BUG_ON(!cfqd->busy_queues);
cfqd->busy_queues--;
@@ -623,6 +703,9 @@ static void cfq_add_rq_rb(struct request *rq)
* check if this request is a better next-serve candidate
*/
cfqq->next_rq = cfq_choose_req(cfqd, cfqq->next_rq, rq);
+ /* this changes the position in the priority tree */
+ cfq_prio_tree_add(cfqd, cfqq);
+
BUG_ON(!cfqq->next_rq);
}

@@ -831,11 +914,11 @@ static struct cfq_queue *cfq_get_next_queue(struct cfq_data *cfqd)
/*
* Get and set a new active queue for service.
*/
-static struct cfq_queue *cfq_set_active_queue(struct cfq_data *cfqd)
+static struct cfq_queue *cfq_set_active_queue(struct cfq_data *cfqd,
+ struct cfq_queue *cfqq)
{
- struct cfq_queue *cfqq;
-
- cfqq = cfq_get_next_queue(cfqd);
+ if (!cfqq)
+ cfqq = cfq_get_next_queue(cfqd);
__cfq_set_active_queue(cfqd, cfqq);
return cfqq;
}
@@ -859,15 +942,78 @@ static inline int cfq_rq_close(struct cfq_data *cfqd, struct request *rq)
return cfq_dist_from_last(cfqd, rq) <= cic->seek_mean;
}

-static int cfq_close_cooperator(struct cfq_data *cfq_data,
- struct cfq_queue *cfqq)
+static struct cfq_queue *cfqq_close(struct cfq_data *cfqd,
+ struct cfq_queue *cur_cfqq)
+{
+ struct rb_root *root = &cfqd->prio_trees[cur_cfqq->ioprio];
+ struct rb_node *parent, *node;
+ struct cfq_queue *__cfqq;
+ sector_t sector = cfqd->last_position;
+
+ if (RB_EMPTY_ROOT(root))
+ return NULL;
+
+ /*
+ * First, if we find a request starting at the end of the last
+ * request, choose it.
+ */
+ __cfqq = cfq_prio_tree_lookup(cfqd, cur_cfqq->ioprio,
+ sector, &parent, NULL);
+ if (__cfqq)
+ return __cfqq;
+
+ /*
+ * If the exact sector wasn't found, the parent of the NULL leaf
+ * will contain the closest sector.
+ */
+ __cfqq = rb_entry(parent, struct cfq_queue, prio_node);
+ if (cfq_rq_close(cfqd, __cfqq->next_rq))
+ return __cfqq;
+ if (__cfqq->next_rq->sector < sector)
+ node = rb_next(&__cfqq->prio_node);
+ else
+ node = rb_prev(&__cfqq->prio_node);
+ if (!node)
+ return NULL;
+ __cfqq = rb_entry(node, struct cfq_queue, prio_node);
+ if (cfq_rq_close(cfqd, __cfqq->next_rq))
+ return __cfqq;
+
+ return NULL;
+}
+
+/*
+ * cfqd - obvious
+ * cur_cfqq - passed in so that we don't decide that the current queue is
+ * closely cooperating with itself.
+ *
+ * So, basically we're assuming that that cur_cfqq has dispatched at least
+ * one request, and that cfqd->last_position reflects a position on the disk
+ * associated with the I/O issued by cur_cfqq. I'm not sure this is a valid
+ * assumption.
+ */
+static struct cfq_queue *cfq_close_cooperator(struct cfq_data *cfqd,
+ struct cfq_queue *cur_cfqq)
{
+ struct cfq_queue *cfqq;
+
+ /*
+ * A valid cfq_io_context is necessary to compare requests against
+ * the seek_mean of the current cfqq.
+ */
+ if (!cfqd->active_cic)
+ return NULL;
+
/*
* We should notice if some of the queues are cooperating, eg
* working closely on the same area of the disk. In that case,
* we can group them together and don't waste time idling.
*/
- return 0;
+ cfqq = cfqq_close(cfqd, cur_cfqq);
+ if (cfqq)
+ return cfqq;
+
+ return NULL;
}

#define CIC_SEEKY(cic) ((cic)->seek_mean > (8 * 1024))
@@ -908,13 +1054,6 @@ static void cfq_arm_slice_timer(struct cfq_data *cfqd)
if (!cic || !atomic_read(&cic->ioc->nr_tasks))
return;

- /*
- * See if this prio level has a good candidate
- */
- if (cfq_close_cooperator(cfqd, cfqq) &&
- (sample_valid(cic->ttime_samples) && cic->ttime_mean > 2))
- return;
-
cfq_mark_cfqq_must_dispatch(cfqq);
cfq_mark_cfqq_wait_request(cfqq);

@@ -992,7 +1131,7 @@ cfq_prio_to_maxrq(struct cfq_data *cfqd, struct cfq_queue *cfqq)
*/
static struct cfq_queue *cfq_select_queue(struct cfq_data *cfqd)
{
- struct cfq_queue *cfqq;
+ struct cfq_queue *cfqq, *new_cfqq = NULL;

cfqq = cfqd->active_queue;
if (!cfqq)
@@ -1012,6 +1151,15 @@ static struct cfq_queue *cfq_select_queue(struct cfq_data *cfqd)
goto keep_queue;

/*
+ * If another queue has a request waiting within our mean seek
+ * distance, let it run. The expire code will check for close
+ * cooperators and put the close queue at the front of the service
+ * tree.
+ */
+ if ((new_cfqq = cfq_close_cooperator(cfqd, cfqq)))
+ goto expire;
+
+ /*
* No requests pending. If the active queue still has requests in
* flight or is idling for a new request, allow either of these
* conditions to happen (or time out) before selecting a new queue.
@@ -1025,7 +1173,7 @@ static struct cfq_queue *cfq_select_queue(struct cfq_data *cfqd)
expire:
cfq_slice_expired(cfqd, 0);
new_queue:
- cfqq = cfq_set_active_queue(cfqd);
+ cfqq = cfq_set_active_queue(cfqd, new_cfqq);
keep_queue:
return cfqq;
}
@@ -1174,6 +1322,7 @@ static void cfq_put_queue(struct cfq_queue *cfqq)

cfq_log_cfqq(cfqd, cfqq, "put_queue");
BUG_ON(rb_first(&cfqq->sort_list));
+ BUG_ON(!RB_EMPTY_NODE(&cfqq->prio_node));
BUG_ON(cfqq->allocated[READ] + cfqq->allocated[WRITE]);
BUG_ON(cfq_cfqq_on_rr(cfqq));

@@ -1386,6 +1535,15 @@ static void cfq_init_prio_data(struct cfq_queue *cfqq, struct io_context *ioc)
}

/*
+ * Reposition in prio_tree array, if we are already there
+ */
+ if (cfqq->prio_root) {
+ rb_erase_init(&cfqq->prio_node, cfqq->prio_root);
+ cfqq->prio_root = NULL;
+ cfq_prio_tree_add(cfqq->cfqd, cfqq);
+ }
+
+ /*
* keep track of original prio settings in case we have to temporarily
* elevate the priority of this queue
*/
@@ -1466,6 +1624,7 @@ retry:
}

RB_CLEAR_NODE(&cfqq->rb_node);
+ RB_CLEAR_NODE(&cfqq->prio_node);
INIT_LIST_HEAD(&cfqq->fifo);

atomic_set(&cfqq->ref, 0);
@@ -1949,7 +2108,18 @@ static void cfq_completed_request(struct request_queue *q, struct request *rq)
cfq_set_prio_slice(cfqd, cfqq);
cfq_clear_cfqq_slice_new(cfqq);
}
- if (cfq_slice_used(cfqq) || cfq_class_idle(cfqq))
+ /*
+ * If there are no requests waiting in this queue, and
+ * there are other queues ready to issue requests, AND
+ * those other queues are issuing requests within our
+ * mean seek distance, give them a chance to run instead
+ * of idling.
+ */
+ if (RB_EMPTY_ROOT(&cfqq->sort_list) &&
+ !RB_EMPTY_ROOT(&cfqd->service_tree.rb)) {
+ if (cfq_close_cooperator(cfqd, cfqq))
+ cfq_slice_expired(cfqd, 1);
+ } else if (cfq_slice_used(cfqq) || cfq_class_idle(cfqq))
cfq_slice_expired(cfqd, 1);
else if (sync && RB_EMPTY_ROOT(&cfqq->sort_list))
cfq_arm_slice_timer(cfqd);
@@ -2210,12 +2380,17 @@ static void cfq_exit_queue(elevator_t *e)
static void *cfq_init_queue(struct request_queue *q)
{
struct cfq_data *cfqd;
+ int i;

cfqd = kmalloc_node(sizeof(*cfqd), GFP_KERNEL | __GFP_ZERO, q->node);
if (!cfqd)
return NULL;

cfqd->service_tree = CFQ_RB_ROOT;
+
+ for (i = 0; i < CFQ_PRIO_LISTS; i++)
+ cfqd->prio_trees[i] = RB_ROOT;
+
INIT_LIST_HEAD(&cfqd->cic_list);

cfqd->queue = q;

--
Jens Axboe

2008-11-11 16:50:31

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jens Axboe <[email protected]> writes:

> On Tue, Nov 11 2008, Jens Axboe wrote:
>> On Tue, Nov 11 2008, Jens Axboe wrote:
>> > On Mon, Nov 10 2008, Jeff Moyer wrote:
>> > > "Vitaly V. Bursov" <[email protected]> writes:
>> > >
>> > > > Jens Axboe wrote:
>> > > >> On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
>> > > >>> Jens Axboe wrote:
>> > > >>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
>> > > >>>>> Jens Axboe <[email protected]> writes:
>> > > >>>>>
>> > > >>>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
>> > > >>>>> Funny, I was going to ask the same question. ;) The reason Jens wants
>> > > >>>>> you to try this patch is that nfsd may be farming off the I/O requests
>> > > >>>>> to different threads which are then performing interleaved I/O. The
>> > > >>>>> above patch tries to detect this and allow cooperating processes to get
>> > > >>>>> disk time instead of waiting for the idle timeout.
>> > > >>>> Precisely :-)
>> > > >>>>
>> > > >>>> The only reason I haven't merged it yet is because of worry of extra
>> > > >>>> cost, but I'll throw some SSD love at it and see how it turns out.
>> > > >>>>
>> > > >>> Sorry, but I get "oops" same moment nfs read transfer starts.
>> > > >>> I can get directory list via nfs, read files locally (not
>> > > >>> carefully tested, though)
>> > > >>>
>> > > >>> Dumps captured via netconsole, so these may not be completely accurate
>> > > >>> but hopefully will give a hint.
>> > > >>
>> > > >> Interesting, strange how that hasn't triggered here. Or perhaps the
>> > > >> version that Jeff posted isn't the one I tried. Anyway, search for:
>> > > >>
>> > > >> RB_CLEAR_NODE(&cfqq->rb_node);
>> > > >>
>> > > >> and add a
>> > > >>
>> > > >> RB_CLEAR_NODE(&cfqq->prio_node);
>> > > >>
>> > > >> just below that. It's in cfq_find_alloc_queue(). I think that should fix
>> > > >> it.
>> > > >>
>> > > >
>> > > > Same problem.
>> > > >
>> > > > I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
>> > > > but It won't boot anymore with cfq with same error...
>> > > >
>> > > > Switching cfq io scheduler at runtime (booting with "as") appears to work with
>> > > > two parallel local dd reads.
>> > >
>> > > Strange, I can't reproduce a failure. I'll keep trying. For now, these
>> > > are the results I see:
>> > >
>> > > [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
>> > > [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
>> > > 1024+0 records in
>> > > 1024+0 records out
>> > > 1073741824 bytes (1.1 GB) copied, 26.8128 s, 40.0 MB/s
>> > > [root@maiden ~]# umount /mnt/megadeth/
>> > > [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
>> > > [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
>> > > 1024+0 records in
>> > > 1024+0 records out
>> > > 1073741824 bytes (1.1 GB) copied, 23.7025 s, 45.3 MB/s
>> > > [root@maiden ~]# umount /mnt/megadeth/
>> > >
>> > > Here is the patch, with the suggestion from Jens to switch the cfqq to
>> > > the right priority tree when the priority is changed.
>> >
>> > I don't see the issue here either. Vitaly, are you using any openvz
>> > kernel patches? IIRC, they patch cfq so it could just be that your cfq
>> > version is incompatible with Jeff's patch.
>>
>> Heh, got it to trigger about 3 seconds after sending that email! I'll
>> look more into it.
>
> OK, found the issue. A few bugs there... cfq_prio_tree_lookup() doesn't
> even return a hit, since it just breaks and returns NULL always. That
> can cause cfq_prio_tree_add() to screw up the rbtree. The code to
> correct on ioprio change wasn't correct either, I changed that as well.
> New patch below, Vitaly can you give it a spin?

Thanks for doing that! Yeah, that was a stupid bug with the lookup
routine. I don't know that I agree with you that the ioprio change code
was wrong. I looked at all of the callers and that seemed the code path
that was used for I/O priority *changes*. The initial creation was
already okay, wasn't it?

Anwyay, I'll test this new version.

Thanks!

Jeff

2008-11-11 16:53:38

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jens Axboe wrote:
> On Tue, Nov 11 2008, Jens Axboe wrote:
>> On Tue, Nov 11 2008, Jens Axboe wrote:
>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
>>>> "Vitaly V. Bursov" <[email protected]> writes:
>>>>
>>>>> Jens Axboe wrote:
>>>>>> On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
>>>>>>> Jens Axboe wrote:
>>>>>>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
>>>>>>>>> Jens Axboe <[email protected]> writes:
>>>>>>>>>
>>>>>>>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
>>>>>>>>> Funny, I was going to ask the same question. ;) The reason Jens wants
>>>>>>>>> you to try this patch is that nfsd may be farming off the I/O requests
>>>>>>>>> to different threads which are then performing interleaved I/O. The
>>>>>>>>> above patch tries to detect this and allow cooperating processes to get
>>>>>>>>> disk time instead of waiting for the idle timeout.
>>>>>>>> Precisely :-)
>>>>>>>>
>>>>>>>> The only reason I haven't merged it yet is because of worry of extra
>>>>>>>> cost, but I'll throw some SSD love at it and see how it turns out.
>>>>>>>>
>>>>>>> Sorry, but I get "oops" same moment nfs read transfer starts.
>>>>>>> I can get directory list via nfs, read files locally (not
>>>>>>> carefully tested, though)
>>>>>>>
>>>>>>> Dumps captured via netconsole, so these may not be completely accurate
>>>>>>> but hopefully will give a hint.
>>>>>> Interesting, strange how that hasn't triggered here. Or perhaps the
>>>>>> version that Jeff posted isn't the one I tried. Anyway, search for:
>>>>>>
>>>>>> RB_CLEAR_NODE(&cfqq->rb_node);
>>>>>>
>>>>>> and add a
>>>>>>
>>>>>> RB_CLEAR_NODE(&cfqq->prio_node);
>>>>>>
>>>>>> just below that. It's in cfq_find_alloc_queue(). I think that should fix
>>>>>> it.
>>>>>>
>>>>> Same problem.
>>>>>
>>>>> I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
>>>>> but It won't boot anymore with cfq with same error...
>>>>>
>>>>> Switching cfq io scheduler at runtime (booting with "as") appears to work with
>>>>> two parallel local dd reads.
>>>> Strange, I can't reproduce a failure. I'll keep trying. For now, these
>>>> are the results I see:
>>>>
>>>> [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
>>>> [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
>>>> 1024+0 records in
>>>> 1024+0 records out
>>>> 1073741824 bytes (1.1 GB) copied, 26.8128 s, 40.0 MB/s
>>>> [root@maiden ~]# umount /mnt/megadeth/
>>>> [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
>>>> [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
>>>> 1024+0 records in
>>>> 1024+0 records out
>>>> 1073741824 bytes (1.1 GB) copied, 23.7025 s, 45.3 MB/s
>>>> [root@maiden ~]# umount /mnt/megadeth/
>>>>
>>>> Here is the patch, with the suggestion from Jens to switch the cfqq to
>>>> the right priority tree when the priority is changed.
>>> I don't see the issue here either. Vitaly, are you using any openvz
>>> kernel patches? IIRC, they patch cfq so it could just be that your cfq
>>> version is incompatible with Jeff's patch.
>> Heh, got it to trigger about 3 seconds after sending that email! I'll
>> look more into it.
>
> OK, found the issue. A few bugs there... cfq_prio_tree_lookup() doesn't
> even return a hit, since it just breaks and returns NULL always. That
> can cause cfq_prio_tree_add() to screw up the rbtree. The code to
> correct on ioprio change wasn't correct either, I changed that as well.
> New patch below, Vitaly can you give it a spin?
>

No crashes so far. Transfer speed is quiet good also.

NFS+deadline, file not cached:

avg-cpu: %user %nice %system %iowait %steal %idle
0,00 0,00 25,50 19,40 0,00 55,10

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 6648,80 0,00 1281,70 0,00 115179,20 0,00 89,86 5,35 4,18 0,35 45,20
sdb 6672,30 0,00 1257,00 0,00 115292,80 0,00 91,72 5,09 4,06 0,35 44,60

NFS+cfq, file not cached:

avg-cpu: %user %nice %system %iowait %steal %idle
0,05 0,00 25,30 23,95 0,00 50,70

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 6403,00 0,00 1089,90 0,00 108655,20 0,00 99,69 4,50 4,13 0,41 44,50
sdb 6394,90 0,00 1099,60 0,00 108639,20 0,00 98,80 4,53 4,12 0,39 42,50

Just for reference: 10 sec interval average, gigabit network,
no tcp/udp hardware checksumming may lead to high system cpu load.

Also, a few more test (server has 4G RAM):

NFS+cfq, file not cached:
$ dd if=test of=/dev/null bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 24.9147 s, 84.2 MB/s

NFS+deadline, file not cached:
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 23.2999 s, 90.0 MB/s

file cached on server:
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 21.9784 s, 95.4 MB/s

Local single dd read leads to 193 MB/s for deadline and
167 MB/s for cfq.

--
Thanks,
Vitaly

2008-11-11 18:08:44

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Tue, Nov 11 2008, Vitaly V. Bursov wrote:
> Jens Axboe wrote:
> > On Tue, Nov 11 2008, Jens Axboe wrote:
> >> On Tue, Nov 11 2008, Jens Axboe wrote:
> >>> On Mon, Nov 10 2008, Jeff Moyer wrote:
> >>>> "Vitaly V. Bursov" <[email protected]> writes:
> >>>>
> >>>>> Jens Axboe wrote:
> >>>>>> On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
> >>>>>>> Jens Axboe wrote:
> >>>>>>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
> >>>>>>>>> Jens Axboe <[email protected]> writes:
> >>>>>>>>>
> >>>>>>>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
> >>>>>>>>> Funny, I was going to ask the same question. ;) The reason Jens wants
> >>>>>>>>> you to try this patch is that nfsd may be farming off the I/O requests
> >>>>>>>>> to different threads which are then performing interleaved I/O. The
> >>>>>>>>> above patch tries to detect this and allow cooperating processes to get
> >>>>>>>>> disk time instead of waiting for the idle timeout.
> >>>>>>>> Precisely :-)
> >>>>>>>>
> >>>>>>>> The only reason I haven't merged it yet is because of worry of extra
> >>>>>>>> cost, but I'll throw some SSD love at it and see how it turns out.
> >>>>>>>>
> >>>>>>> Sorry, but I get "oops" same moment nfs read transfer starts.
> >>>>>>> I can get directory list via nfs, read files locally (not
> >>>>>>> carefully tested, though)
> >>>>>>>
> >>>>>>> Dumps captured via netconsole, so these may not be completely accurate
> >>>>>>> but hopefully will give a hint.
> >>>>>> Interesting, strange how that hasn't triggered here. Or perhaps the
> >>>>>> version that Jeff posted isn't the one I tried. Anyway, search for:
> >>>>>>
> >>>>>> RB_CLEAR_NODE(&cfqq->rb_node);
> >>>>>>
> >>>>>> and add a
> >>>>>>
> >>>>>> RB_CLEAR_NODE(&cfqq->prio_node);
> >>>>>>
> >>>>>> just below that. It's in cfq_find_alloc_queue(). I think that should fix
> >>>>>> it.
> >>>>>>
> >>>>> Same problem.
> >>>>>
> >>>>> I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
> >>>>> but It won't boot anymore with cfq with same error...
> >>>>>
> >>>>> Switching cfq io scheduler at runtime (booting with "as") appears to work with
> >>>>> two parallel local dd reads.
> >>>> Strange, I can't reproduce a failure. I'll keep trying. For now, these
> >>>> are the results I see:
> >>>>
> >>>> [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
> >>>> [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
> >>>> 1024+0 records in
> >>>> 1024+0 records out
> >>>> 1073741824 bytes (1.1 GB) copied, 26.8128 s, 40.0 MB/s
> >>>> [root@maiden ~]# umount /mnt/megadeth/
> >>>> [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
> >>>> [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
> >>>> 1024+0 records in
> >>>> 1024+0 records out
> >>>> 1073741824 bytes (1.1 GB) copied, 23.7025 s, 45.3 MB/s
> >>>> [root@maiden ~]# umount /mnt/megadeth/
> >>>>
> >>>> Here is the patch, with the suggestion from Jens to switch the cfqq to
> >>>> the right priority tree when the priority is changed.
> >>> I don't see the issue here either. Vitaly, are you using any openvz
> >>> kernel patches? IIRC, they patch cfq so it could just be that your cfq
> >>> version is incompatible with Jeff's patch.
> >> Heh, got it to trigger about 3 seconds after sending that email! I'll
> >> look more into it.
> >
> > OK, found the issue. A few bugs there... cfq_prio_tree_lookup() doesn't
> > even return a hit, since it just breaks and returns NULL always. That
> > can cause cfq_prio_tree_add() to screw up the rbtree. The code to
> > correct on ioprio change wasn't correct either, I changed that as well.
> > New patch below, Vitaly can you give it a spin?
> >
>
> No crashes so far. Transfer speed is quiet good also.
>
>
> NFS+deadline, file not cached:
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0,00 0,00 25,50 19,40 0,00 55,10
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
> sda 6648,80 0,00 1281,70 0,00 115179,20 0,00 89,86 5,35 4,18 0,35 45,20
> sdb 6672,30 0,00 1257,00 0,00 115292,80 0,00 91,72 5,09 4,06 0,35 44,60
>
>
>
> NFS+cfq, file not cached:
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0,05 0,00 25,30 23,95 0,00 50,70
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
> sda 6403,00 0,00 1089,90 0,00 108655,20 0,00 99,69 4,50 4,13 0,41 44,50
> sdb 6394,90 0,00 1099,60 0,00 108639,20 0,00 98,80 4,53 4,12 0,39 42,50
>
>
> Just for reference: 10 sec interval average, gigabit network,
> no tcp/udp hardware checksumming may lead to high system cpu load.
>
>
> Also, a few more test (server has 4G RAM):
>
> NFS+cfq, file not cached:
> $ dd if=test of=/dev/null bs=1M count=2000
> 2000+0 records in
> 2000+0 records out
> 2097152000 bytes (2.1 GB) copied, 24.9147 s, 84.2 MB/s
>
> NFS+deadline, file not cached:
> 2000+0 records in
> 2000+0 records out
> 2097152000 bytes (2.1 GB) copied, 23.2999 s, 90.0 MB/s
>
> file cached on server:
> 2000+0 records in
> 2000+0 records out
> 2097152000 bytes (2.1 GB) copied, 21.9784 s, 95.4 MB/s
>
>
> Local single dd read leads to 193 MB/s for deadline and
> 167 MB/s for cfq.

OK, that looks better. Can I talk you into just trying this little
patch, just to see what kind of performance that yields? Remove the cfq
patch first. I would have patched nfsd only, but this is just a quick'n
dirty.

diff --git a/kernel/kthread.c b/kernel/kthread.c
index 8e7a7ce..3aacf48 100644
--- a/kernel/kthread.c
+++ b/kernel/kthread.c
@@ -92,7 +92,7 @@ static void create_kthread(struct kthread_create_info *create)
int pid;

/* We want our own signal handler (we take no signals by default). */
- pid = kernel_thread(kthread, create, CLONE_FS | CLONE_FILES | SIGCHLD);
+ pid = kernel_thread(kthread, create, CLONE_FS | CLONE_FILES | CLONE_IO | SIGCHLD);
if (pid < 0) {
create->result = ERR_PTR(pid);
} else {

--
Jens Axboe

2008-11-11 18:10:35

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Tue, Nov 11 2008, Jeff Moyer wrote:
> Jens Axboe <[email protected]> writes:
>
> > On Tue, Nov 11 2008, Jens Axboe wrote:
> >> On Tue, Nov 11 2008, Jens Axboe wrote:
> >> > On Mon, Nov 10 2008, Jeff Moyer wrote:
> >> > > "Vitaly V. Bursov" <[email protected]> writes:
> >> > >
> >> > > > Jens Axboe wrote:
> >> > > >> On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
> >> > > >>> Jens Axboe wrote:
> >> > > >>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
> >> > > >>>>> Jens Axboe <[email protected]> writes:
> >> > > >>>>>
> >> > > >>>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
> >> > > >>>>> Funny, I was going to ask the same question. ;) The reason Jens wants
> >> > > >>>>> you to try this patch is that nfsd may be farming off the I/O requests
> >> > > >>>>> to different threads which are then performing interleaved I/O. The
> >> > > >>>>> above patch tries to detect this and allow cooperating processes to get
> >> > > >>>>> disk time instead of waiting for the idle timeout.
> >> > > >>>> Precisely :-)
> >> > > >>>>
> >> > > >>>> The only reason I haven't merged it yet is because of worry of extra
> >> > > >>>> cost, but I'll throw some SSD love at it and see how it turns out.
> >> > > >>>>
> >> > > >>> Sorry, but I get "oops" same moment nfs read transfer starts.
> >> > > >>> I can get directory list via nfs, read files locally (not
> >> > > >>> carefully tested, though)
> >> > > >>>
> >> > > >>> Dumps captured via netconsole, so these may not be completely accurate
> >> > > >>> but hopefully will give a hint.
> >> > > >>
> >> > > >> Interesting, strange how that hasn't triggered here. Or perhaps the
> >> > > >> version that Jeff posted isn't the one I tried. Anyway, search for:
> >> > > >>
> >> > > >> RB_CLEAR_NODE(&cfqq->rb_node);
> >> > > >>
> >> > > >> and add a
> >> > > >>
> >> > > >> RB_CLEAR_NODE(&cfqq->prio_node);
> >> > > >>
> >> > > >> just below that. It's in cfq_find_alloc_queue(). I think that should fix
> >> > > >> it.
> >> > > >>
> >> > > >
> >> > > > Same problem.
> >> > > >
> >> > > > I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
> >> > > > but It won't boot anymore with cfq with same error...
> >> > > >
> >> > > > Switching cfq io scheduler at runtime (booting with "as") appears to work with
> >> > > > two parallel local dd reads.
> >> > >
> >> > > Strange, I can't reproduce a failure. I'll keep trying. For now, these
> >> > > are the results I see:
> >> > >
> >> > > [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
> >> > > [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
> >> > > 1024+0 records in
> >> > > 1024+0 records out
> >> > > 1073741824 bytes (1.1 GB) copied, 26.8128 s, 40.0 MB/s
> >> > > [root@maiden ~]# umount /mnt/megadeth/
> >> > > [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
> >> > > [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
> >> > > 1024+0 records in
> >> > > 1024+0 records out
> >> > > 1073741824 bytes (1.1 GB) copied, 23.7025 s, 45.3 MB/s
> >> > > [root@maiden ~]# umount /mnt/megadeth/
> >> > >
> >> > > Here is the patch, with the suggestion from Jens to switch the cfqq to
> >> > > the right priority tree when the priority is changed.
> >> >
> >> > I don't see the issue here either. Vitaly, are you using any openvz
> >> > kernel patches? IIRC, they patch cfq so it could just be that your cfq
> >> > version is incompatible with Jeff's patch.
> >>
> >> Heh, got it to trigger about 3 seconds after sending that email! I'll
> >> look more into it.
> >
> > OK, found the issue. A few bugs there... cfq_prio_tree_lookup() doesn't
> > even return a hit, since it just breaks and returns NULL always. That
> > can cause cfq_prio_tree_add() to screw up the rbtree. The code to
> > correct on ioprio change wasn't correct either, I changed that as well.
> > New patch below, Vitaly can you give it a spin?
>
> Thanks for doing that! Yeah, that was a stupid bug with the lookup
> routine. I don't know that I agree with you that the ioprio change code
> was wrong. I looked at all of the callers and that seemed the code path
> that was used for I/O priority *changes*. The initial creation was
> already okay, wasn't it?

You only did it in cfq_prio_boost(), you should go one down and do it
for all prio changes. cfq_init_prio_data() gets called to fix state up
lazily when it notices a prio change, either due to prio boost or
because someone ran ionice.

--
Jens Axboe

2008-11-11 19:36:27

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jens Axboe <[email protected]> writes:

> OK, that looks better. Can I talk you into just trying this little
> patch, just to see what kind of performance that yields? Remove the cfq
> patch first. I would have patched nfsd only, but this is just a quick'n
> dirty.

I went ahead and gave it a shot. The updated CFQ patch with no I/O
context sharing does about 40MB/s reading a 1GB file. Backing that
patch out, and then adding the patch to share io_context's between
kthreads yields 45MB/s.

By the way, in looking at the copy_io function, I noticed what appears
to be a (minor) bug:

if (clone_flags & CLONE_IO) {
tsk->io_context = ioc_task_link(ioc);
if (unlikely(!tsk->io_context))
return -ENOMEM;

According to comments in ioc_task_link, tsk->io_context == NULL means:
/*
* if ref count is zero, don't allow sharing (ioc is going away, it's
* a race).
*/

It seems more appropriate to just create a new I/O context at this
point, don't you think? (Sorry, I know it's off-topic!)

Cheers,

Jeff

diff --git a/kernel/fork.c b/kernel/fork.c
index f608356..483d95c 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -723,10 +723,17 @@ static int copy_io(unsigned long clone_flags, struct task_struct *tsk)
* Share io context with parent, if CLONE_IO is set
*/
if (clone_flags & CLONE_IO) {
+ /*
+ * If ioc_task_link fails, it just means that we raced
+ * with io context cleanup. Continue on to allocate
+ * a new context in this case.
+ */
tsk->io_context = ioc_task_link(ioc);
- if (unlikely(!tsk->io_context))
- return -ENOMEM;
- } else if (ioprio_valid(ioc->ioprio)) {
+ if (likely(tsk->io_context))
+ return 0;
+ }
+
+ if (ioprio_valid(ioc->ioprio)) {
tsk->io_context = alloc_io_context(GFP_KERNEL, -1);
if (unlikely(!tsk->io_context))
return -ENOMEM;

2008-11-11 19:42:22

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jens Axboe wrote:
> On Tue, Nov 11 2008, Vitaly V. Bursov wrote:
>> Jens Axboe wrote:
>>> On Tue, Nov 11 2008, Jens Axboe wrote:
>>>> On Tue, Nov 11 2008, Jens Axboe wrote:
>>>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
>>>>>> "Vitaly V. Bursov" <[email protected]> writes:
>>>>>>
>>>>>>> Jens Axboe wrote:
>>>>>>>> On Mon, Nov 10 2008, Vitaly V. Bursov wrote:
>>>>>>>>> Jens Axboe wrote:
>>>>>>>>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
>>>>>>>>>>> Jens Axboe <[email protected]> writes:
>>>>>>>>>>>
>>>>>>>>>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
>>>>>>>>>>> Funny, I was going to ask the same question. ;) The reason Jens wants
>>>>>>>>>>> you to try this patch is that nfsd may be farming off the I/O requests
>>>>>>>>>>> to different threads which are then performing interleaved I/O. The
>>>>>>>>>>> above patch tries to detect this and allow cooperating processes to get
>>>>>>>>>>> disk time instead of waiting for the idle timeout.
>>>>>>>>>> Precisely :-)
>>>>>>>>>>
>>>>>>>>>> The only reason I haven't merged it yet is because of worry of extra
>>>>>>>>>> cost, but I'll throw some SSD love at it and see how it turns out.
>>>>>>>>>>
>>>>>>>>> Sorry, but I get "oops" same moment nfs read transfer starts.
>>>>>>>>> I can get directory list via nfs, read files locally (not
>>>>>>>>> carefully tested, though)
>>>>>>>>>
>>>>>>>>> Dumps captured via netconsole, so these may not be completely accurate
>>>>>>>>> but hopefully will give a hint.
>>>>>>>> Interesting, strange how that hasn't triggered here. Or perhaps the
>>>>>>>> version that Jeff posted isn't the one I tried. Anyway, search for:
>>>>>>>>
>>>>>>>> RB_CLEAR_NODE(&cfqq->rb_node);
>>>>>>>>
>>>>>>>> and add a
>>>>>>>>
>>>>>>>> RB_CLEAR_NODE(&cfqq->prio_node);
>>>>>>>>
>>>>>>>> just below that. It's in cfq_find_alloc_queue(). I think that should fix
>>>>>>>> it.
>>>>>>>>
>>>>>>> Same problem.
>>>>>>>
>>>>>>> I did make clean; make -j3; sync; on (2 times) patched kernel and it went OK
>>>>>>> but It won't boot anymore with cfq with same error...
>>>>>>>
>>>>>>> Switching cfq io scheduler at runtime (booting with "as") appears to work with
>>>>>>> two parallel local dd reads.
>>>>>> Strange, I can't reproduce a failure. I'll keep trying. For now, these
>>>>>> are the results I see:
>>>>>>
>>>>>> [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
>>>>>> [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
>>>>>> 1024+0 records in
>>>>>> 1024+0 records out
>>>>>> 1073741824 bytes (1.1 GB) copied, 26.8128 s, 40.0 MB/s
>>>>>> [root@maiden ~]# umount /mnt/megadeth/
>>>>>> [root@maiden ~]# mount megadeth:/export/cciss /mnt/megadeth/
>>>>>> [root@maiden ~]# dd if=/mnt/megadeth/file1 of=/dev/null bs=1M
>>>>>> 1024+0 records in
>>>>>> 1024+0 records out
>>>>>> 1073741824 bytes (1.1 GB) copied, 23.7025 s, 45.3 MB/s
>>>>>> [root@maiden ~]# umount /mnt/megadeth/
>>>>>>
>>>>>> Here is the patch, with the suggestion from Jens to switch the cfqq to
>>>>>> the right priority tree when the priority is changed.
>>>>> I don't see the issue here either. Vitaly, are you using any openvz
>>>>> kernel patches? IIRC, they patch cfq so it could just be that your cfq
>>>>> version is incompatible with Jeff's patch.
>>>> Heh, got it to trigger about 3 seconds after sending that email! I'll
>>>> look more into it.
>>> OK, found the issue. A few bugs there... cfq_prio_tree_lookup() doesn't
>>> even return a hit, since it just breaks and returns NULL always. That
>>> can cause cfq_prio_tree_add() to screw up the rbtree. The code to
>>> correct on ioprio change wasn't correct either, I changed that as well.
>>> New patch below, Vitaly can you give it a spin?
>>>
>> No crashes so far. Transfer speed is quiet good also.
>>
>>
>> NFS+deadline, file not cached:
>>
>> avg-cpu: %user %nice %system %iowait %steal %idle
>> 0,00 0,00 25,50 19,40 0,00 55,10
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
>> sda 6648,80 0,00 1281,70 0,00 115179,20 0,00 89,86 5,35 4,18 0,35 45,20
>> sdb 6672,30 0,00 1257,00 0,00 115292,80 0,00 91,72 5,09 4,06 0,35 44,60
>>
>>
>>
>> NFS+cfq, file not cached:
>>
>> avg-cpu: %user %nice %system %iowait %steal %idle
>> 0,05 0,00 25,30 23,95 0,00 50,70
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
>> sda 6403,00 0,00 1089,90 0,00 108655,20 0,00 99,69 4,50 4,13 0,41 44,50
>> sdb 6394,90 0,00 1099,60 0,00 108639,20 0,00 98,80 4,53 4,12 0,39 42,50
>>
>>
>> Just for reference: 10 sec interval average, gigabit network,
>> no tcp/udp hardware checksumming may lead to high system cpu load.
>>
>>
>> Also, a few more test (server has 4G RAM):
>>
>> NFS+cfq, file not cached:
>> $ dd if=test of=/dev/null bs=1M count=2000
>> 2000+0 records in
>> 2000+0 records out
>> 2097152000 bytes (2.1 GB) copied, 24.9147 s, 84.2 MB/s
>>
>> NFS+deadline, file not cached:
>> 2000+0 records in
>> 2000+0 records out
>> 2097152000 bytes (2.1 GB) copied, 23.2999 s, 90.0 MB/s
>>
>> file cached on server:
>> 2000+0 records in
>> 2000+0 records out
>> 2097152000 bytes (2.1 GB) copied, 21.9784 s, 95.4 MB/s
>>
>>
>> Local single dd read leads to 193 MB/s for deadline and
>> 167 MB/s for cfq.
>
> OK, that looks better. Can I talk you into just trying this little
> patch, just to see what kind of performance that yields? Remove the cfq
> patch first. I would have patched nfsd only, but this is just a quick'n
> dirty.
>
> diff --git a/kernel/kthread.c b/kernel/kthread.c
> index 8e7a7ce..3aacf48 100644
> --- a/kernel/kthread.c
> +++ b/kernel/kthread.c
> @@ -92,7 +92,7 @@ static void create_kthread(struct kthread_create_info *create)
> int pid;
>
> /* We want our own signal handler (we take no signals by default). */
> - pid = kernel_thread(kthread, create, CLONE_FS | CLONE_FILES | SIGCHLD);
> + pid = kernel_thread(kthread, create, CLONE_FS | CLONE_FILES | CLONE_IO | SIGCHLD);
> if (pid < 0) {
> create->result = ERR_PTR(pid);
> } else {
>

No patches:

iostat for nfs+cfq read
avg-cpu: %user %nice %system %iowait %steal %idle
0,00 0,00 3,25 52,20 0,00 44,55

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 2820,10 0,00 452,40 0,00 47648,80 0,00 105,32 7,54 16,70 1,96 88,60
sdb 2818,60 0,00 453,90 0,00 47391,20 0,00 104,41 4,13 9,02 1,33 60,30

NFS+cfq, file not cached:
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 57.5762 s, 36.4 MB/s

NFS+deadline, file not cached:
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 23.6672 s, 88.6 MB/s

======================
Above patch applied:

avg-cpu: %user %nice %system %iowait %steal %idle
0,00 0,00 3,60 51,10 0,00 45,30

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 2805,80 0,00 446,20 0,00 47267,20 0,00 105,93 5,61 12,62 1,71 76,50
sdb 2803,90 0,00 448,50 0,00 47246,40 0,00 105,34 5,56 12,46 1,68 75,40

NFS+cfq, file not cached:
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 57.5903 s, 36.4 MB/s

NFS+deadline, file not cached:
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 23.46 s, 89.4 MB/s

======================
Both patches applied:

avg-cpu: %user %nice %system %iowait %steal %idle
0,00 0,00 22,95 24,65 0,00 52,40

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 6504,60 0,00 1089,80 0,00 110359,20 0,00 101,27 4,67 4,29 0,40 43,50
sdb 6495,50 0,00 1097,50 0,00 110312,80 0,00 100,51 4,57 4,17 0,39 43,10

NFS+cfq, file not cached:
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 25.4477 s, 82.4 MB/s

NFS+deadline, file not cached:
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 23.1639 s, 90.5 MB/s

--
Thanks,
Vitaly

2008-11-11 21:41:28

by Jeff Layton

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Tue, 11 Nov 2008 14:36:07 -0500
Jeff Moyer <[email protected]> wrote:

> Jens Axboe <[email protected]> writes:
>
> > OK, that looks better. Can I talk you into just trying this little
> > patch, just to see what kind of performance that yields? Remove the cfq
> > patch first. I would have patched nfsd only, but this is just a quick'n
> > dirty.
>
> I went ahead and gave it a shot. The updated CFQ patch with no I/O
> context sharing does about 40MB/s reading a 1GB file. Backing that
> patch out, and then adding the patch to share io_context's between
> kthreads yields 45MB/s.
>

Here's a quick and dirty patch to make all of the nfsd's have the same
io_context. Comments appreciated -- I'm not that familiar with the IO
scheduling code. If this looks good, I'll clean it up, add some
comments and formally send it to Bruce.

----------------[snip]-------------------

>From dd15b19a0eab3e181a6f76f1421b97950e255b4b Mon Sep 17 00:00:00 2001
From: Jeff Layton <[email protected]>
Date: Tue, 11 Nov 2008 15:43:15 -0500
Subject: [PATCH] knfsd: make all nfsd threads share an io_context

This apparently makes the I/O scheduler treat the threads as a group
which helps throughput when sequential I/O is multiplexed over several
nfsd's.

Signed-off-by: Jeff Layton <[email protected]>
---
fs/nfsd/nfssvc.c | 27 +++++++++++++++++++++++++++
1 files changed, 27 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 07e4f5d..6d87f74 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -22,6 +22,7 @@
#include <linux/freezer.h>
#include <linux/fs_struct.h>
#include <linux/kthread.h>
+#include <linux/iocontext.h>

#include <linux/sunrpc/types.h>
#include <linux/sunrpc/stats.h>
@@ -42,6 +43,7 @@ static int nfsd(void *vrqstp);
struct timeval nfssvc_boot;
static atomic_t nfsd_busy;
static unsigned long nfsd_last_call;
+static struct io_context *nfsd_io_context;
static DEFINE_SPINLOCK(nfsd_call_lock);

/*
@@ -173,6 +175,7 @@ static void nfsd_last_thread(struct svc_serv *serv)
nfsd_serv = NULL;
nfsd_racache_shutdown();
nfs4_state_shutdown();
+ nfsd_io_context = NULL;

printk(KERN_WARNING "nfsd: last server has exited, flushing export "
"cache\n");
@@ -398,6 +401,28 @@ update_thread_usage(int busy_threads)
}

/*
+ * should be called while holding nfsd_mutex
+ */
+static void
+nfsd_set_io_context(void)
+{
+ int cpu, node;
+
+ if (!nfsd_io_context) {
+ cpu = get_cpu();
+ node = cpu_to_node(cpu);
+ put_cpu();
+
+ /*
+ * get_io_context can return NULL if the alloc_context fails.
+ * That's not technically fatal here, so we don't bother to
+ * check for it.
+ */
+ nfsd_io_context = get_io_context(GFP_KERNEL, node);
+ } else
+ copy_io_context(&current->io_context, &nfsd_io_context);
+}
+/*
* This is the NFS server kernel thread
*/
static int
@@ -410,6 +435,8 @@ nfsd(void *vrqstp)
/* Lock module and set up kernel thread */
mutex_lock(&nfsd_mutex);

+ nfsd_set_io_context();
+
/* At this point, the thread shares current->fs
* with the init process. We need to create files with a
* umask of 0 instead of init's umask. */
--
1.5.5.1

2008-11-11 21:59:56

by Jeff Layton

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Tue, 11 Nov 2008 16:41:04 -0500
Jeff Layton <[email protected]> wrote:

> On Tue, 11 Nov 2008 14:36:07 -0500
> Jeff Moyer <[email protected]> wrote:
>
> > Jens Axboe <[email protected]> writes:
> >
> > > OK, that looks better. Can I talk you into just trying this little
> > > patch, just to see what kind of performance that yields? Remove the cfq
> > > patch first. I would have patched nfsd only, but this is just a quick'n
> > > dirty.
> >
> > I went ahead and gave it a shot. The updated CFQ patch with no I/O
> > context sharing does about 40MB/s reading a 1GB file. Backing that
> > patch out, and then adding the patch to share io_context's between
> > kthreads yields 45MB/s.
> >
>
> Here's a quick and dirty patch to make all of the nfsd's have the same
> io_context. Comments appreciated -- I'm not that familiar with the IO
> scheduling code. If this looks good, I'll clean it up, add some
> comments and formally send it to Bruce.
>

No sooner than I send it out than I find a bug. We need to eventually
put the io_context reference we get. This should be more correct:

----------------[snip]-------------------

>From d0ee67045a12c677883f77791c6f260588c7b41f Mon Sep 17 00:00:00 2001
From: Jeff Layton <[email protected]>
Date: Tue, 11 Nov 2008 16:54:16 -0500
Subject: [PATCH] knfsd: make all nfsd threads share an io_context

This apparently makes the I/O scheduler treat the threads as a group
which helps throughput when sequential I/O is multiplexed over several
nfsd's.

Signed-off-by: Jeff Layton <[email protected]>
---
fs/nfsd/nfssvc.c | 30 ++++++++++++++++++++++++++++++
1 files changed, 30 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 07e4f5d..5cd99f9 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -22,6 +22,7 @@
#include <linux/freezer.h>
#include <linux/fs_struct.h>
#include <linux/kthread.h>
+#include <linux/iocontext.h>

#include <linux/sunrpc/types.h>
#include <linux/sunrpc/stats.h>
@@ -42,6 +43,7 @@ static int nfsd(void *vrqstp);
struct timeval nfssvc_boot;
static atomic_t nfsd_busy;
static unsigned long nfsd_last_call;
+static struct io_context *nfsd_io_context;
static DEFINE_SPINLOCK(nfsd_call_lock);

/*
@@ -173,6 +175,10 @@ static void nfsd_last_thread(struct svc_serv *serv)
nfsd_serv = NULL;
nfsd_racache_shutdown();
nfs4_state_shutdown();
+ if (nfsd_io_context) {
+ put_io_context(nfsd_io_context);
+ nfsd_io_context = NULL;
+ }

printk(KERN_WARNING "nfsd: last server has exited, flushing export "
"cache\n");
@@ -398,6 +404,28 @@ update_thread_usage(int busy_threads)
}

/*
+ * should be called while holding nfsd_mutex
+ */
+static void
+nfsd_set_io_context(void)
+{
+ int cpu, node;
+
+ if (!nfsd_io_context) {
+ cpu = get_cpu();
+ node = cpu_to_node(cpu);
+ put_cpu();
+
+ /*
+ * get_io_context can return NULL if the alloc_context fails.
+ * That's not technically fatal here, so we don't bother to
+ * check for it.
+ */
+ nfsd_io_context = get_io_context(GFP_KERNEL, node);
+ } else
+ copy_io_context(&current->io_context, &nfsd_io_context);
+}
+/*
* This is the NFS server kernel thread
*/
static int
@@ -410,6 +438,8 @@ nfsd(void *vrqstp)
/* Lock module and set up kernel thread */
mutex_lock(&nfsd_mutex);

+ nfsd_set_io_context();
+
/* At this point, the thread shares current->fs
* with the init process. We need to create files with a
* umask of 0 instead of init's umask. */
--
1.5.5.1

2008-11-12 12:22:21

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Tue, Nov 11 2008, Jeff Layton wrote:
> On Tue, 11 Nov 2008 16:41:04 -0500
> Jeff Layton <[email protected]> wrote:
>
> > On Tue, 11 Nov 2008 14:36:07 -0500
> > Jeff Moyer <[email protected]> wrote:
> >
> > > Jens Axboe <[email protected]> writes:
> > >
> > > > OK, that looks better. Can I talk you into just trying this little
> > > > patch, just to see what kind of performance that yields? Remove the cfq
> > > > patch first. I would have patched nfsd only, but this is just a quick'n
> > > > dirty.
> > >
> > > I went ahead and gave it a shot. The updated CFQ patch with no I/O
> > > context sharing does about 40MB/s reading a 1GB file. Backing that
> > > patch out, and then adding the patch to share io_context's between
> > > kthreads yields 45MB/s.
> > >
> >
> > Here's a quick and dirty patch to make all of the nfsd's have the same
> > io_context. Comments appreciated -- I'm not that familiar with the IO
> > scheduling code. If this looks good, I'll clean it up, add some
> > comments and formally send it to Bruce.
> >
>
> No sooner than I send it out than I find a bug. We need to eventually
> put the io_context reference we get. This should be more correct:

That sort of thing happens a lot, I can definitely sympathize with you
there :-)

> ----------------[snip]-------------------
>
> From d0ee67045a12c677883f77791c6f260588c7b41f Mon Sep 17 00:00:00 2001
> From: Jeff Layton <[email protected]>
> Date: Tue, 11 Nov 2008 16:54:16 -0500
> Subject: [PATCH] knfsd: make all nfsd threads share an io_context
>
> This apparently makes the I/O scheduler treat the threads as a group
> which helps throughput when sequential I/O is multiplexed over several
> nfsd's.

That's a lot more nifty than my stupid CLONE_IO flag addition. Both are
only good for test purposes though.

It's a bit difficult to make this really mergeable. I don't know
anything about how nfsd manages its thread pool, but something more
appropriate would be an io context per client mount. That's still not
perfect as you could easily have more than one process doing simultanous
IO on the client side, but it's a lot better.

--
Jens Axboe

2008-11-12 12:45:28

by Jeff Layton

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Wed, 12 Nov 2008 13:20:35 +0100
Jens Axboe <[email protected]> wrote:

> On Tue, Nov 11 2008, Jeff Layton wrote:
> > On Tue, 11 Nov 2008 16:41:04 -0500
> > Jeff Layton <[email protected]> wrote:
> >
> > > On Tue, 11 Nov 2008 14:36:07 -0500
> > > Jeff Moyer <[email protected]> wrote:
> > >
> > > > Jens Axboe <[email protected]> writes:
> > > >
> > > > > OK, that looks better. Can I talk you into just trying this little
> > > > > patch, just to see what kind of performance that yields? Remove the cfq
> > > > > patch first. I would have patched nfsd only, but this is just a quick'n
> > > > > dirty.
> > > >
> > > > I went ahead and gave it a shot. The updated CFQ patch with no I/O
> > > > context sharing does about 40MB/s reading a 1GB file. Backing that
> > > > patch out, and then adding the patch to share io_context's between
> > > > kthreads yields 45MB/s.
> > > >
> > >
> > > Here's a quick and dirty patch to make all of the nfsd's have the same
> > > io_context. Comments appreciated -- I'm not that familiar with the IO
> > > scheduling code. If this looks good, I'll clean it up, add some
> > > comments and formally send it to Bruce.
> > >
> >
> > No sooner than I send it out than I find a bug. We need to eventually
> > put the io_context reference we get. This should be more correct:
>
> That sort of thing happens a lot, I can definitely sympathize with you
> there :-)
>
> > ----------------[snip]-------------------
> >
> > From d0ee67045a12c677883f77791c6f260588c7b41f Mon Sep 17 00:00:00 2001
> > From: Jeff Layton <[email protected]>
> > Date: Tue, 11 Nov 2008 16:54:16 -0500
> > Subject: [PATCH] knfsd: make all nfsd threads share an io_context
> >
> > This apparently makes the I/O scheduler treat the threads as a group
> > which helps throughput when sequential I/O is multiplexed over several
> > nfsd's.
>
> That's a lot more nifty than my stupid CLONE_IO flag addition. Both are
> only good for test purposes though.
>
> It's a bit difficult to make this really mergeable. I don't know
> anything about how nfsd manages its thread pool, but something more
> appropriate would be an io context per client mount. That's still not
> perfect as you could easily have more than one process doing simultanous
> IO on the client side, but it's a lot better.
>

Ahh Good point...I wasn't thinking about it the right way. I guess what
we really need is some way to tell that a series of I/O requests
originated from the same client thread. NFS isn't really conducive to
this...

We might be able to do something like that with NFSv4 though. Maybe an
io_context per state owner or something. That's probably not going to
be trivial to implement however...

--
Jeff Layton <[email protected]>

2008-11-12 12:54:46

by Christoph Hellwig

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Wed, Nov 12, 2008 at 07:45:03AM -0500, Jeff Layton wrote:
> Ahh Good point...I wasn't thinking about it the right way. I guess what
> we really need is some way to tell that a series of I/O requests
> originated from the same client thread. NFS isn't really conducive to
> this...
>
> We might be able to do something like that with NFSv4 though. Maybe an
> io_context per state owner or something. That's probably not going to
> be trivial to implement however...

Talk to Greg Banks, he's been thinking about and working on an open
files cache for quite a while already.

2008-11-12 18:33:18

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jens Axboe <[email protected]> writes:

> On Mon, Nov 10 2008, Jeff Moyer wrote:
>> Jens Axboe <[email protected]> writes:
>>
>> > On Sun, Nov 09 2008, Vitaly V. Bursov wrote:
>> >> Hello,
>> >>
>> >> I'm building small server system with openvz kernel and have ran into
>> >> some IO performance problems. Reading a single file via NFS delivers
>> >> around 9 MB/s over gigabit network, but while reading, say, 2 different
>> >> or same file 2 times at the same time I get >60MB/s.
>> >>
>> >> Changing IO scheduler to deadline or anticipatory fixes problem.
>> >>
>> >> Tested kernels:
>> >> OpenVZ RHEL5 028stab059.3 (9 MB/s with HZ=100, 20MB/s with HZ=1000
>> >> fast local reads)
>> >> Vanilla 2.6.27.5 (40 MB/s with HZ=100, slow local reads)
>> >>
>> >> Vanilla performs better in worst case but I believe 40 is still low
>> >> concerning test results below.
>> >
>> > Can you check with this patch applied?
>> >
>> > http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
>>
>> Funny, I was going to ask the same question. ;) The reason Jens wants
>> you to try this patch is that nfsd may be farming off the I/O requests
>> to different threads which are then performing interleaved I/O. The
>> above patch tries to detect this and allow cooperating processes to get
>> disk time instead of waiting for the idle timeout.
>
> Precisely :-)
>
> The only reason I haven't merged it yet is because of worry of extra
> cost, but I'll throw some SSD love at it and see how it turns out.

OK, I'm not actually able to reproduce the horrible 9MB/s reported by
Vitaly. Here are the numbers I see.

Single dd performing a cold cache read of a 1GB file from an
nfs server. read_ahead_kb is 128 (the default) for all tests.
cfq-cc denotes that the cfq scheduler was patched with the close
cooperator patch. All numbers are in MB/s.

nfsd threads| 1 | 2 | 4 | 8
----------------------------------------
deadline | 65.3 | 52.2 | 46.7 | 46.1
cfq | 64.1 | 57.8 | 53.3 | 46.9
cfq-cc | 65.7 | 55.8 | 52.1 | 40.3

So, in my configuration, cfq and deadline both degrade in performance as
the number of nfsd threads is increased. The close cooperator patch
seems to hurt a bit more at 8 threads, instead of helping; I'm not sure
why that is.

Now, the workload that showed most slowdown for cfq with respect to
other I/O schedulers was using dump(8) to backup a file system. Here
are the numbers for that:

deadline 82241 kB/s
cfq 34143 kB/s
cfq-cc 82241 kB/s

And a customer actually went to the trouble to write a test to approximate
dump(8)'s I/O patterns. For that test, we also see a big speedup (as
expected):

deadline 87157 kB/s
cfq 20218 kB/s
cfq-cc 87056 kB/s

Jens, if you have any canned fio workloads that you use for regression
testing, please pass them my way and I'll give them a go on some SAN
storage.

Cheers,

Jeff

2008-11-12 19:04:18

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Wed, Nov 12 2008, Jeff Moyer wrote:
> Jens Axboe <[email protected]> writes:
>
> > On Mon, Nov 10 2008, Jeff Moyer wrote:
> >> Jens Axboe <[email protected]> writes:
> >>
> >> > On Sun, Nov 09 2008, Vitaly V. Bursov wrote:
> >> >> Hello,
> >> >>
> >> >> I'm building small server system with openvz kernel and have ran into
> >> >> some IO performance problems. Reading a single file via NFS delivers
> >> >> around 9 MB/s over gigabit network, but while reading, say, 2 different
> >> >> or same file 2 times at the same time I get >60MB/s.
> >> >>
> >> >> Changing IO scheduler to deadline or anticipatory fixes problem.
> >> >>
> >> >> Tested kernels:
> >> >> OpenVZ RHEL5 028stab059.3 (9 MB/s with HZ=100, 20MB/s with HZ=1000
> >> >> fast local reads)
> >> >> Vanilla 2.6.27.5 (40 MB/s with HZ=100, slow local reads)
> >> >>
> >> >> Vanilla performs better in worst case but I believe 40 is still low
> >> >> concerning test results below.
> >> >
> >> > Can you check with this patch applied?
> >> >
> >> > http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
> >>
> >> Funny, I was going to ask the same question. ;) The reason Jens wants
> >> you to try this patch is that nfsd may be farming off the I/O requests
> >> to different threads which are then performing interleaved I/O. The
> >> above patch tries to detect this and allow cooperating processes to get
> >> disk time instead of waiting for the idle timeout.
> >
> > Precisely :-)
> >
> > The only reason I haven't merged it yet is because of worry of extra
> > cost, but I'll throw some SSD love at it and see how it turns out.
>
> OK, I'm not actually able to reproduce the horrible 9MB/s reported by
> Vitaly. Here are the numbers I see.
>
> Single dd performing a cold cache read of a 1GB file from an
> nfs server. read_ahead_kb is 128 (the default) for all tests.
> cfq-cc denotes that the cfq scheduler was patched with the close
> cooperator patch. All numbers are in MB/s.
>
> nfsd threads| 1 | 2 | 4 | 8
> ----------------------------------------
> deadline | 65.3 | 52.2 | 46.7 | 46.1
> cfq | 64.1 | 57.8 | 53.3 | 46.9
> cfq-cc | 65.7 | 55.8 | 52.1 | 40.3
>
> So, in my configuration, cfq and deadline both degrade in performance as
> the number of nfsd threads is increased. The close cooperator patch
> seems to hurt a bit more at 8 threads, instead of helping; I'm not sure
> why that is.
>
> Now, the workload that showed most slowdown for cfq with respect to
> other I/O schedulers was using dump(8) to backup a file system. Here
> are the numbers for that:
>
> deadline 82241 kB/s
> cfq 34143 kB/s
> cfq-cc 82241 kB/s
>
> And a customer actually went to the trouble to write a test to approximate
> dump(8)'s I/O patterns. For that test, we also see a big speedup (as
> expected):
>
> deadline 87157 kB/s
> cfq 20218 kB/s
> cfq-cc 87056 kB/s
>
> Jens, if you have any canned fio workloads that you use for regression
> testing, please pass them my way and I'll give them a go on some SAN
> storage.

I already talked about this with Jeff on irc, but I guess should post it
here as well.

nfsd aside (which does seem to have some different behaviour skewing the
results), the original patch came about because dump(8) has a really
stupid design that offloads IO to a number of processes. This basically
makes fairly sequential IO more random with CFQ, since each process gets
its own io context. My feeling is that we should fix dump instead of
introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
aware of any other good programs out there that would do something
similar, so I don't think there's a lot of merrit to spending cycles on
detecting cooperating processes.

Jeff will take a look at fixing dump instead, and I may have promised
him that santa will bring him something nice this year if he does (since
I'm sure it'll be painful on the eyes).

--
Jens Axboe

2008-11-13 06:54:22

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jeff Moyer пишет:
> Jens Axboe <[email protected]> writes:
>
>> On Mon, Nov 10 2008, Jeff Moyer wrote:
>>> Jens Axboe <[email protected]> writes:
>>>
>>>> On Sun, Nov 09 2008, Vitaly V. Bursov wrote:
>>>>> Hello,
>>>>>
>>>>> I'm building small server system with openvz kernel and have ran into
>>>>> some IO performance problems. Reading a single file via NFS delivers
>>>>> around 9 MB/s over gigabit network, but while reading, say, 2 different
>>>>> or same file 2 times at the same time I get >60MB/s.
>>>>>
>>>>> Changing IO scheduler to deadline or anticipatory fixes problem.
>>>>>
>>>>> Tested kernels:
>>>>> OpenVZ RHEL5 028stab059.3 (9 MB/s with HZ=100, 20MB/s with HZ=1000
>>>>> fast local reads)
>>>>> Vanilla 2.6.27.5 (40 MB/s with HZ=100, slow local reads)
>>>>>
>>>>> Vanilla performs better in worst case but I believe 40 is still low
>>>>> concerning test results below.
>>>> Can you check with this patch applied?
>>>>
>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
>>> Funny, I was going to ask the same question. ;) The reason Jens wants
>>> you to try this patch is that nfsd may be farming off the I/O requests
>>> to different threads which are then performing interleaved I/O. The
>>> above patch tries to detect this and allow cooperating processes to get
>>> disk time instead of waiting for the idle timeout.
>> Precisely :-)
>>
>> The only reason I haven't merged it yet is because of worry of extra
>> cost, but I'll throw some SSD love at it and see how it turns out.
>
> OK, I'm not actually able to reproduce the horrible 9MB/s reported by
> Vitaly. Here are the numbers I see.

It's 2.6.18-openvz-rhel5 kernel gives me 9MB/s, and with 2.6.27 I get ~40-50MB/s
instead of 80-90 MB/s as there should be no bottlenecks except the network.

> Single dd performing a cold cache read of a 1GB file from an
> nfs server. read_ahead_kb is 128 (the default) for all tests.
> cfq-cc denotes that the cfq scheduler was patched with the close
> cooperator patch. All numbers are in MB/s.
>
> nfsd threads| 1 | 2 | 4 | 8
> ----------------------------------------
> deadline | 65.3 | 52.2 | 46.7 | 46.1
> cfq | 64.1 | 57.8 | 53.3 | 46.9
> cfq-cc | 65.7 | 55.8 | 52.1 | 40.3
>
> So, in my configuration, cfq and deadline both degrade in performance as
> the number of nfsd threads is increased. The close cooperator patch
> seems to hurt a bit more at 8 threads, instead of helping; I'm not sure
> why that is.

Interesting, I'll try to change nfsd threads number and see how it performs
on my setup. Setting it to 1 seems like a good idea for cfq and a non-high-end
hardware.

I'll look into it this evening.

--
Regards,
Vitaly

2008-11-13 08:52:13

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Hi all,

//Sorry for being late.

On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
[...]
> I already talked about this with Jeff on irc, but I guess should post it
> here as well.
>
> nfsd aside (which does seem to have some different behaviour skewing the
> results), the original patch came about because dump(8) has a really
> stupid design that offloads IO to a number of processes. This basically
> makes fairly sequential IO more random with CFQ, since each process gets
> its own io context. My feeling is that we should fix dump instead of
> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
> aware of any other good programs out there that would do something
> similar, so I don't think there's a lot of merrit to spending cycles on
> detecting cooperating processes.
>
> Jeff will take a look at fixing dump instead, and I may have promised
> him that santa will bring him something nice this year if he does (since
> I'm sure it'll be painful on the eyes).

This could also be fixed at the VFS readahead level.

In fact I've seen many kinds of interleaved accesses:
- concurrently reading 40 files that are in fact hard links of one single file
- a backup tool that splits a big file into 8k chunks, and serve the
{1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
chunks in another one
- a pool of NFSDs randomly serving some originally sequential read requests
- now dump(8) seems to have some similar problem.

In summary there have been all kinds of efforts on trying to
parallelize I/O tasks, but unfortunately they can easily screw up the
sequential pattern. It may not be easily fixable for many of them.

It is however possible to detect most of these patterns at the
readahead layer and restore sequential I/Os, before they propagate
into the block layer and hurt performance.

Vitaly, if that's what you need, I can try to prepare a patch for testing out.

Thank you,
Fengguang

2008-11-13 08:56:29

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Thu, Nov 13 2008, Wu Fengguang wrote:
> Hi all,
>
> //Sorry for being late.
>
> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
> [...]
> > I already talked about this with Jeff on irc, but I guess should post it
> > here as well.
> >
> > nfsd aside (which does seem to have some different behaviour skewing the
> > results), the original patch came about because dump(8) has a really
> > stupid design that offloads IO to a number of processes. This basically
> > makes fairly sequential IO more random with CFQ, since each process gets
> > its own io context. My feeling is that we should fix dump instead of
> > introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
> > aware of any other good programs out there that would do something
> > similar, so I don't think there's a lot of merrit to spending cycles on
> > detecting cooperating processes.
> >
> > Jeff will take a look at fixing dump instead, and I may have promised
> > him that santa will bring him something nice this year if he does (since
> > I'm sure it'll be painful on the eyes).
>
> This could also be fixed at the VFS readahead level.
>
> In fact I've seen many kinds of interleaved accesses:
> - concurrently reading 40 files that are in fact hard links of one single file
> - a backup tool that splits a big file into 8k chunks, and serve the
> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
> chunks in another one
> - a pool of NFSDs randomly serving some originally sequential read requests
> - now dump(8) seems to have some similar problem.
>
> In summary there have been all kinds of efforts on trying to
> parallelize I/O tasks, but unfortunately they can easily screw up the
> sequential pattern. It may not be easily fixable for many of them.
>
> It is however possible to detect most of these patterns at the
> readahead layer and restore sequential I/Os, before they propagate
> into the block layer and hurt performance.
>
> Vitaly, if that's what you need, I can try to prepare a patch for
> testing out.

It's not easy. To really fix it, you have to get that sequential RA
pattern from just the single process. As soon as you spread the IO
between processes (eg N-1 aren't just getting cache hits), then you may
run into trouble on the IO scheduler side.

--
Jens Axboe

2008-11-13 14:34:56

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

"Vitaly V. Bursov" <[email protected]> writes:

> Jeff Moyer пишет:
>> Jens Axboe <[email protected]> writes:
>>
>>> On Mon, Nov 10 2008, Jeff Moyer wrote:
>>>> Jens Axboe <[email protected]> writes:
>>>>
>>>>> On Sun, Nov 09 2008, Vitaly V. Bursov wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I'm building small server system with openvz kernel and have ran into
>>>>>> some IO performance problems. Reading a single file via NFS delivers
>>>>>> around 9 MB/s over gigabit network, but while reading, say, 2 different
>>>>>> or same file 2 times at the same time I get >60MB/s.
>>>>>>
>>>>>> Changing IO scheduler to deadline or anticipatory fixes problem.
>>>>>>
>>>>>> Tested kernels:
>>>>>> OpenVZ RHEL5 028stab059.3 (9 MB/s with HZ=100, 20MB/s with HZ=1000
>>>>>> fast local reads)
>>>>>> Vanilla 2.6.27.5 (40 MB/s with HZ=100, slow local reads)
>>>>>>
>>>>>> Vanilla performs better in worst case but I believe 40 is still low
>>>>>> concerning test results below.
>>>>> Can you check with this patch applied?
>>>>>
>>>>> http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
>>>> Funny, I was going to ask the same question. ;) The reason Jens wants
>>>> you to try this patch is that nfsd may be farming off the I/O requests
>>>> to different threads which are then performing interleaved I/O. The
>>>> above patch tries to detect this and allow cooperating processes to get
>>>> disk time instead of waiting for the idle timeout.
>>> Precisely :-)
>>>
>>> The only reason I haven't merged it yet is because of worry of extra
>>> cost, but I'll throw some SSD love at it and see how it turns out.
>>
>> OK, I'm not actually able to reproduce the horrible 9MB/s reported by
>> Vitaly. Here are the numbers I see.
>
> It's 2.6.18-openvz-rhel5 kernel gives me 9MB/s, and with 2.6.27 I get ~40-50MB/s
> instead of 80-90 MB/s as there should be no bottlenecks except the network.

Reading back through your original problem report, I'm not seeing what
your numbers were with deadline; you simply mentioned that it "fixed"
the problem. Are you sure you'll get 80-90MB/s for this? The local
disks in my configuration, when performing a dd on the server system,
can produce numbers around 85 MB/s, yet the NFS performance is around 65
MB/s (and this is a gigabit network).

>> Single dd performing a cold cache read of a 1GB file from an
>> nfs server. read_ahead_kb is 128 (the default) for all tests.
>> cfq-cc denotes that the cfq scheduler was patched with the close
>> cooperator patch. All numbers are in MB/s.
>>
>> nfsd threads| 1 | 2 | 4 | 8
>> ----------------------------------------
>> deadline | 65.3 | 52.2 | 46.7 | 46.1
>> cfq | 64.1 | 57.8 | 53.3 | 46.9
>> cfq-cc | 65.7 | 55.8 | 52.1 | 40.3
>>
>> So, in my configuration, cfq and deadline both degrade in performance as
>> the number of nfsd threads is increased. The close cooperator patch
>> seems to hurt a bit more at 8 threads, instead of helping; I'm not sure
>> why that is.
>
> Interesting, I'll try to change nfsd threads number and see how it performs
> on my setup. Setting it to 1 seems like a good idea for cfq and a non-high-end
> hardware.

I think you're looking at this backwards. I'm no nfsd tuning expert,
but I'm pretty sure that you would tune the numbe,r of threads based on
the number of active clients and the amount of memory on the server
(since each thread has to reserve memory for incoming requests).

> I'll look into it this evening.

The real reason I tried varying the number of nfsd threads was to show,
at least for CFQ, that spreading a sequential I/O load across multiple
threads would result in suboptimal performance. What I found, instead,
was that it hurt performance for cfq *and* deadline (and that the close
cooperator patches did not help in this regard). This tells me that
there is something else which is affecting the performance. What that
something is I don't know, I think we'd have to take a closer look at
what's going on on the server to figure it out.

Cheers,

Jeff

2008-11-13 18:33:43

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jeff Moyer wrote:

>> It's 2.6.18-openvz-rhel5 kernel gives me 9MB/s, and with 2.6.27 I get ~40-50MB/s
>> instead of 80-90 MB/s as there should be no bottlenecks except the network.
>
> Reading back through your original problem report, I'm not seeing what
> your numbers were with deadline; you simply mentioned that it "fixed"
> the problem. Are you sure you'll get 80-90MB/s for this? The local
> disks in my configuration, when performing a dd on the server system,
> can produce numbers around 85 MB/s, yet the NFS performance is around 65
> MB/s (and this is a gigabit network).

I have pair of 1TB HDDs and each of them is able to deliver around
100MB/s for sequental reads and PCI-E Ethernet adapter not sharing
a bus with SATA controller.

2.6.18-openvz-rhel5, loopback mounted nfs with deadline gives
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 7.12014 s, 147 MB/s

and same system via network:
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 14.3197 s, 73.2 MB/s

and network+cfq:
dd if=samefile of=/dev/null bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 59.3116 s, 17.7 MB/s

and network+file cached on server side:
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 12.4204 s, 84.4 MB/s

Well, 73 is still not 80, but way much better than 17 (or,
even worse - 9)

I have 8 NFS threads by default here.

I got 17 MB here because of HZ=1000, also 2.6.27 performed
better in every heavy-nfs-transfer test so far

Changed network parateters:

net.core.wmem_default = 1048576
net.core.wmem_max = 1048576
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576

net.ipv4.tcp_mem = 1048576 1048576 4194304
net.ipv4.tcp_rmem = 1048576 1048576 4194304
net.ipv4.tcp_rmem = 1048576 1048576 4194304
net.ipv4.tcp_wmem = 1048576 1048576 4194304
net.ipv4.tcp_wmem = 1048576 1048576 4194304

MTU: 6000

Sorry, I didn't mention these in original post.

>>> Single dd performing a cold cache read of a 1GB file from an
>>> nfs server. read_ahead_kb is 128 (the default) for all tests.
>>> cfq-cc denotes that the cfq scheduler was patched with the close
>>> cooperator patch. All numbers are in MB/s.
>>>
>>> nfsd threads| 1 | 2 | 4 | 8
>>> ----------------------------------------
>>> deadline | 65.3 | 52.2 | 46.7 | 46.1
>>> cfq | 64.1 | 57.8 | 53.3 | 46.9
>>> cfq-cc | 65.7 | 55.8 | 52.1 | 40.3
>>>
>>> So, in my configuration, cfq and deadline both degrade in performance as
>>> the number of nfsd threads is increased. The close cooperator patch
>>> seems to hurt a bit more at 8 threads, instead of helping; I'm not sure
>>> why that is.
>> Interesting, I'll try to change nfsd threads number and see how it performs
>> on my setup. Setting it to 1 seems like a good idea for cfq and a non-high-end
>> hardware.
>
> I think you're looking at this backwards. I'm no nfsd tuning expert,
> but I'm pretty sure that you would tune the numbe,r of threads based on
> the number of active clients and the amount of memory on the server
> (since each thread has to reserve memory for incoming requests).

I understand this. It's just one of the parameters I completely missed
out of my sight :)

>> I'll look into it this evening.
>
> The real reason I tried varying the number of nfsd threads was to show,
> at least for CFQ, that spreading a sequential I/O load across multiple
> threads would result in suboptimal performance. What I found, instead,
> was that it hurt performance for cfq *and* deadline (and that the close
> cooperator patches did not help in this regard). This tells me that
> there is something else which is affecting the performance. What that
> something is I don't know, I think we'd have to take a closer look at
> what's going on on the server to figure it out.
>

I've tested it also...

loopback:
nfsd threads | 1 | 2 | 4 | 8 | 16
----------------------------------------
deadline-vz | 97| 92| 128| 145| 148
deadline | 145| 160| 173| 170| 150
cfq-cc | 137| 150| 167| 157| 133
cfq | 26| 28| 34| 38| 38

network:
nfsd threads | 1 | 2 | 4| 8| 16
----------------------------------------
deadline-vz | 68| 69| 75| 73| 72
deadline | 91| 89| 87| 88| 84
cfq-cc | 91| 89| 88| 82| 74
cfq | 25| 28| 32| 36| 34

deadline-vz - deadline with 2.6.18-openvz-rhel5 kernel
deadline, cfq, cfq-cc - linux-2.6.27.5

Yep, it's not that simple as I thought...

--
Regards,
Vitaly

2008-11-13 18:46:35

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

2008-11-14 01:36:48

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Thu, Nov 13, 2008 at 09:54:39AM +0100, Jens Axboe wrote:
> On Thu, Nov 13 2008, Wu Fengguang wrote:
> > Hi all,
> >
> > //Sorry for being late.
> >
> > On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
> > [...]
> > > I already talked about this with Jeff on irc, but I guess should post it
> > > here as well.
> > >
> > > nfsd aside (which does seem to have some different behaviour skewing the
> > > results), the original patch came about because dump(8) has a really
> > > stupid design that offloads IO to a number of processes. This basically
> > > makes fairly sequential IO more random with CFQ, since each process gets
> > > its own io context. My feeling is that we should fix dump instead of
> > > introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
> > > aware of any other good programs out there that would do something
> > > similar, so I don't think there's a lot of merrit to spending cycles on
> > > detecting cooperating processes.
> > >
> > > Jeff will take a look at fixing dump instead, and I may have promised
> > > him that santa will bring him something nice this year if he does (since
> > > I'm sure it'll be painful on the eyes).
> >
> > This could also be fixed at the VFS readahead level.
> >
> > In fact I've seen many kinds of interleaved accesses:
> > - concurrently reading 40 files that are in fact hard links of one single file
> > - a backup tool that splits a big file into 8k chunks, and serve the
> > {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
> > chunks in another one
> > - a pool of NFSDs randomly serving some originally sequential read requests
> > - now dump(8) seems to have some similar problem.
> >
> > In summary there have been all kinds of efforts on trying to
> > parallelize I/O tasks, but unfortunately they can easily screw up the
> > sequential pattern. It may not be easily fixable for many of them.
> >
> > It is however possible to detect most of these patterns at the
> > readahead layer and restore sequential I/Os, before they propagate
> > into the block layer and hurt performance.
> >
> > Vitaly, if that's what you need, I can try to prepare a patch for
> > testing out.
>
> It's not easy. To really fix it, you have to get that sequential RA
> pattern from just the single process. As soon as you spread the IO
> between processes (eg N-1 aren't just getting cache hits), then you may
> run into trouble on the IO scheduler side.

Yes, it's not easy(or possible) to tell from file->f_ra all those
cooperative processes working on the same sequential stream, since
they will have different file->f_ra instances. In the case of NFSD,
the file->f_ra may well be all zeros.

Another scheme is to detect the sequential pattern via looking up
the page cache, which provides one single and consistent view of the
pages recently accessed. That makes sequential detection possible.

The cost will be one extra page cache lookup per random read.
If it's not acceptable, the corresponding code could be disabled
by default.

Thanks,
Fengguang

2008-11-24 15:33:27

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jens Axboe <[email protected]> writes:

> nfsd aside (which does seem to have some different behaviour skewing the
> results), the original patch came about because dump(8) has a really
> stupid design that offloads IO to a number of processes. This basically
> makes fairly sequential IO more random with CFQ, since each process gets
> its own io context. My feeling is that we should fix dump instead of
> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
> aware of any other good programs out there that would do something
> similar, so I don't think there's a lot of merrit to spending cycles on
> detecting cooperating processes.
>
> Jeff will take a look at fixing dump instead, and I may have promised
> him that santa will bring him something nice this year if he does (since
> I'm sure it'll be painful on the eyes).

Sorry to drum up this topic once again, but we've recently run into
another instance where the close cooperator patch helps significantly.
The case is KVM using the virtio disk driver. The host-side uses
posix_aio calls to issue I/O on behalf of the guest. It's worth noting
that pthread_create does not pass CLONE_IO (at least that was my reading
of the code). It is questionable whether it really should as that will
change the I/O scheduling dynamics.

So, Jens, what do you think? Should we collect some performance numbers
to make sure that the close cooperator patch doesn't hurt the common
case?

Cheers,
Jeff

2008-11-24 18:15:50

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Mon, Nov 24 2008, Jeff Moyer wrote:
> Jens Axboe <[email protected]> writes:
>
> > nfsd aside (which does seem to have some different behaviour skewing the
> > results), the original patch came about because dump(8) has a really
> > stupid design that offloads IO to a number of processes. This basically
> > makes fairly sequential IO more random with CFQ, since each process gets
> > its own io context. My feeling is that we should fix dump instead of
> > introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
> > aware of any other good programs out there that would do something
> > similar, so I don't think there's a lot of merrit to spending cycles on
> > detecting cooperating processes.
> >
> > Jeff will take a look at fixing dump instead, and I may have promised
> > him that santa will bring him something nice this year if he does (since
> > I'm sure it'll be painful on the eyes).
>
> Sorry to drum up this topic once again, but we've recently run into
> another instance where the close cooperator patch helps significantly.
> The case is KVM using the virtio disk driver. The host-side uses
> posix_aio calls to issue I/O on behalf of the guest. It's worth noting
> that pthread_create does not pass CLONE_IO (at least that was my reading
> of the code). It is questionable whether it really should as that will
> change the I/O scheduling dynamics.
>
> So, Jens, what do you think? Should we collect some performance numbers
> to make sure that the close cooperator patch doesn't hurt the common
> case?

No, posix aio is a piece of crap on Linux/glibc so we want to be fixing
that instead. A quick fix is again to use CLONE_IO, though posix aio
needs more work than that. I told the qemu guys not to use posix aio a
long time ago since it does stink and doesn't perform well under any
circumstance... So I don't consider that a valid use case, there's a
reason that basically nobody is using posix aio.

--
Jens Axboe

2008-11-24 18:50:57

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jens Axboe <[email protected]> writes:

> On Mon, Nov 24 2008, Jeff Moyer wrote:
>> Jens Axboe <[email protected]> writes:
>>
>> > nfsd aside (which does seem to have some different behaviour skewing the
>> > results), the original patch came about because dump(8) has a really
>> > stupid design that offloads IO to a number of processes. This basically
>> > makes fairly sequential IO more random with CFQ, since each process gets
>> > its own io context. My feeling is that we should fix dump instead of
>> > introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>> > aware of any other good programs out there that would do something
>> > similar, so I don't think there's a lot of merrit to spending cycles on
>> > detecting cooperating processes.
>> >
>> > Jeff will take a look at fixing dump instead, and I may have promised
>> > him that santa will bring him something nice this year if he does (since
>> > I'm sure it'll be painful on the eyes).
>>
>> Sorry to drum up this topic once again, but we've recently run into
>> another instance where the close cooperator patch helps significantly.
>> The case is KVM using the virtio disk driver. The host-side uses
>> posix_aio calls to issue I/O on behalf of the guest. It's worth noting
>> that pthread_create does not pass CLONE_IO (at least that was my reading
>> of the code). It is questionable whether it really should as that will
>> change the I/O scheduling dynamics.
>>
>> So, Jens, what do you think? Should we collect some performance numbers
>> to make sure that the close cooperator patch doesn't hurt the common
>> case?
>
> No, posix aio is a piece of crap on Linux/glibc so we want to be fixing
> that instead. A quick fix is again to use CLONE_IO, though posix aio
> needs more work than that. I told the qemu guys not to use posix aio a
> long time ago since it does stink and doesn't perform well under any
> circumstance... So I don't consider that a valid use case, there's a
> reason that basically nobody is using posix aio.

It doesn't help that we never took in patches to the kernel that would
allow for a usable posix aio implementation, but I digress.

My question to you is how many use cases do we dismiss as broken before
recognizing that people actually do this, and that we should at least
try to detect and gracefully deal with it? Is this too much to expect
from the default I/O scheduler? Sorry to beat a dead horse, but folks
do view this as a regression, and they won't be changing their
applications, they'll be switching I/O schedulers to fix this.

Cheers,
Jeff

2008-11-24 18:53:32

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Mon, Nov 24 2008, Jeff Moyer wrote:
> Jens Axboe <[email protected]> writes:
>
> > On Mon, Nov 24 2008, Jeff Moyer wrote:
> >> Jens Axboe <[email protected]> writes:
> >>
> >> > nfsd aside (which does seem to have some different behaviour skewing the
> >> > results), the original patch came about because dump(8) has a really
> >> > stupid design that offloads IO to a number of processes. This basically
> >> > makes fairly sequential IO more random with CFQ, since each process gets
> >> > its own io context. My feeling is that we should fix dump instead of
> >> > introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
> >> > aware of any other good programs out there that would do something
> >> > similar, so I don't think there's a lot of merrit to spending cycles on
> >> > detecting cooperating processes.
> >> >
> >> > Jeff will take a look at fixing dump instead, and I may have promised
> >> > him that santa will bring him something nice this year if he does (since
> >> > I'm sure it'll be painful on the eyes).
> >>
> >> Sorry to drum up this topic once again, but we've recently run into
> >> another instance where the close cooperator patch helps significantly.
> >> The case is KVM using the virtio disk driver. The host-side uses
> >> posix_aio calls to issue I/O on behalf of the guest. It's worth noting
> >> that pthread_create does not pass CLONE_IO (at least that was my reading
> >> of the code). It is questionable whether it really should as that will
> >> change the I/O scheduling dynamics.
> >>
> >> So, Jens, what do you think? Should we collect some performance numbers
> >> to make sure that the close cooperator patch doesn't hurt the common
> >> case?
> >
> > No, posix aio is a piece of crap on Linux/glibc so we want to be fixing
> > that instead. A quick fix is again to use CLONE_IO, though posix aio
> > needs more work than that. I told the qemu guys not to use posix aio a
> > long time ago since it does stink and doesn't perform well under any
> > circumstance... So I don't consider that a valid use case, there's a
> > reason that basically nobody is using posix aio.
>
> It doesn't help that we never took in patches to the kernel that would
> allow for a usable posix aio implementation, but I digress.
>
> My question to you is how many use cases do we dismiss as broken before
> recognizing that people actually do this, and that we should at least
> try to detect and gracefully deal with it? Is this too much to expect
> from the default I/O scheduler? Sorry to beat a dead horse, but folks
> do view this as a regression, and they won't be changing their
> applications, they'll be switching I/O schedulers to fix this.

Yes, I'm aware of that. If posix aio was in wide spread use it would be
an issue, and it's really a shame that it sucks as much as it does. A
single case like dump is worth changing, if there was 1 or 2 other real
cases I'd say we'd have a real case for doing the coop checking.

--
Jens Axboe

2008-11-25 11:00:16

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Wu Fengguang wrote:
> Hi all,
>
> //Sorry for being late.
>
> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
> [...]
>> I already talked about this with Jeff on irc, but I guess should post it
>> here as well.
>>
>> nfsd aside (which does seem to have some different behaviour skewing the
>> results), the original patch came about because dump(8) has a really
>> stupid design that offloads IO to a number of processes. This basically
>> makes fairly sequential IO more random with CFQ, since each process gets
>> its own io context. My feeling is that we should fix dump instead of
>> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>> aware of any other good programs out there that would do something
>> similar, so I don't think there's a lot of merrit to spending cycles on
>> detecting cooperating processes.
>>
>> Jeff will take a look at fixing dump instead, and I may have promised
>> him that santa will bring him something nice this year if he does (since
>> I'm sure it'll be painful on the eyes).
>
> This could also be fixed at the VFS readahead level.
>
> In fact I've seen many kinds of interleaved accesses:
> - concurrently reading 40 files that are in fact hard links of one single file
> - a backup tool that splits a big file into 8k chunks, and serve the
> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
> chunks in another one
> - a pool of NFSDs randomly serving some originally sequential read requests
> - now dump(8) seems to have some similar problem.
>
> In summary there have been all kinds of efforts on trying to
> parallelize I/O tasks, but unfortunately they can easily screw up the
> sequential pattern. It may not be easily fixable for many of them.
>
> It is however possible to detect most of these patterns at the
> readahead layer and restore sequential I/Os, before they propagate
> into the block layer and hurt performance.

I believe this would be the most effective way to go, especially in case
if data delivery path to the original client has its own latency
depended from the amount of transferred data as it is in the case of
remote NFS mount, which does synchronous sequential reads. In this case
it is essential for performance to make both links (local to the storage
and network to the client) be always busy and transfer data
simultaneously. Since the reads are synchronous, the only way to achieve
that is perform read ahead on the server sufficient to cover the network
link latency. Otherwise you would end up with only half of possible
throughput.

However, from one side, server has to have a pool of threads/processes
to perform well, but, from other side, current read ahead code doesn't
detect too well that those threads/processes are doing joint sequential
read, so the read ahead window gets smaller, hence the overall read
performance gets considerably smaller too.

> Vitaly, if that's what you need, I can try to prepare a patch for testing out.

I can test it with SCST SCSI target sybsystem (http://scst.sf.net). SCST
needs such feature very much, otherwise it can't get full backstorage
read speed. The maximum I can see is about ~80MB/s from ~130MB/s 15K RPM
disk over 1Gbps iSCSI link (maximum possible is ~110MB/s).

> Thank you,
> Fengguang
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2008-11-25 11:08:17

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Wu Fengguang wrote:
> On Thu, Nov 13, 2008 at 09:54:39AM +0100, Jens Axboe wrote:
>> On Thu, Nov 13 2008, Wu Fengguang wrote:
>>> Hi all,
>>>
>>> //Sorry for being late.
>>>
>>> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
>>> [...]
>>>> I already talked about this with Jeff on irc, but I guess should post it
>>>> here as well.
>>>>
>>>> nfsd aside (which does seem to have some different behaviour skewing the
>>>> results), the original patch came about because dump(8) has a really
>>>> stupid design that offloads IO to a number of processes. This basically
>>>> makes fairly sequential IO more random with CFQ, since each process gets
>>>> its own io context. My feeling is that we should fix dump instead of
>>>> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>>>> aware of any other good programs out there that would do something
>>>> similar, so I don't think there's a lot of merrit to spending cycles on
>>>> detecting cooperating processes.
>>>>
>>>> Jeff will take a look at fixing dump instead, and I may have promised
>>>> him that santa will bring him something nice this year if he does (since
>>>> I'm sure it'll be painful on the eyes).
>>> This could also be fixed at the VFS readahead level.
>>>
>>> In fact I've seen many kinds of interleaved accesses:
>>> - concurrently reading 40 files that are in fact hard links of one single file
>>> - a backup tool that splits a big file into 8k chunks, and serve the
>>> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
>>> chunks in another one
>>> - a pool of NFSDs randomly serving some originally sequential read requests
>>> - now dump(8) seems to have some similar problem.
>>>
>>> In summary there have been all kinds of efforts on trying to
>>> parallelize I/O tasks, but unfortunately they can easily screw up the
>>> sequential pattern. It may not be easily fixable for many of them.
>>>
>>> It is however possible to detect most of these patterns at the
>>> readahead layer and restore sequential I/Os, before they propagate
>>> into the block layer and hurt performance.
>>>
>>> Vitaly, if that's what you need, I can try to prepare a patch for
>>> testing out.
>> It's not easy. To really fix it, you have to get that sequential RA
>> pattern from just the single process. As soon as you spread the IO
>> between processes (eg N-1 aren't just getting cache hits), then you may
>> run into trouble on the IO scheduler side.
>
> Yes, it's not easy(or possible) to tell from file->f_ra all those
> cooperative processes working on the same sequential stream, since
> they will have different file->f_ra instances. In the case of NFSD,
> the file->f_ra may well be all zeros.
>
> Another scheme is to detect the sequential pattern via looking up
> the page cache, which provides one single and consistent view of the
> pages recently accessed. That makes sequential detection possible.
>
> The cost will be one extra page cache lookup per random read.
> If it's not acceptable, the corresponding code could be disabled
> by default.

I think, this should be the best and the simplest way to go. Since in
most case data from the cache should be later copied to user, one more
page cache lookup should be negligible.

> Thanks,
> Fengguang
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2008-11-25 11:26:26

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Tue, Nov 25, 2008 at 02:02:45PM +0300, Vladislav Bolkhovitin wrote:
>
>
> Wu Fengguang wrote:
>> On Thu, Nov 13, 2008 at 09:54:39AM +0100, Jens Axboe wrote:
>>> On Thu, Nov 13 2008, Wu Fengguang wrote:
>>>> Hi all,
>>>>
>>>> //Sorry for being late.
>>>>
>>>> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
>>>> [...]
>>>>> I already talked about this with Jeff on irc, but I guess should post it
>>>>> here as well.
>>>>>
>>>>> nfsd aside (which does seem to have some different behaviour skewing the
>>>>> results), the original patch came about because dump(8) has a really
>>>>> stupid design that offloads IO to a number of processes. This basically
>>>>> makes fairly sequential IO more random with CFQ, since each process gets
>>>>> its own io context. My feeling is that we should fix dump instead of
>>>>> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>>>>> aware of any other good programs out there that would do something
>>>>> similar, so I don't think there's a lot of merrit to spending cycles on
>>>>> detecting cooperating processes.
>>>>>
>>>>> Jeff will take a look at fixing dump instead, and I may have promised
>>>>> him that santa will bring him something nice this year if he does (since
>>>>> I'm sure it'll be painful on the eyes).
>>>> This could also be fixed at the VFS readahead level.
>>>>
>>>> In fact I've seen many kinds of interleaved accesses:
>>>> - concurrently reading 40 files that are in fact hard links of one single file
>>>> - a backup tool that splits a big file into 8k chunks, and serve the
>>>> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
>>>> chunks in another one
>>>> - a pool of NFSDs randomly serving some originally sequential read
>>>> requests - now dump(8) seems to have some similar problem.
>>>>
>>>> In summary there have been all kinds of efforts on trying to
>>>> parallelize I/O tasks, but unfortunately they can easily screw up the
>>>> sequential pattern. It may not be easily fixable for many of them.
>>>>
>>>> It is however possible to detect most of these patterns at the
>>>> readahead layer and restore sequential I/Os, before they propagate
>>>> into the block layer and hurt performance.
>>>>
>>>> Vitaly, if that's what you need, I can try to prepare a patch for
>>>> testing out.
>>> It's not easy. To really fix it, you have to get that sequential RA
>>> pattern from just the single process. As soon as you spread the IO
>>> between processes (eg N-1 aren't just getting cache hits), then you may
>>> run into trouble on the IO scheduler side.
>>
>> Yes, it's not easy(or possible) to tell from file->f_ra all those
>> cooperative processes working on the same sequential stream, since
>> they will have different file->f_ra instances. In the case of NFSD,
>> the file->f_ra may well be all zeros.
>>
>> Another scheme is to detect the sequential pattern via looking up
>> the page cache, which provides one single and consistent view of the
>> pages recently accessed. That makes sequential detection possible.
>>
>> The cost will be one extra page cache lookup per random read.
>> If it's not acceptable, the corresponding code could be disabled
>> by default.
>
> I think, this should be the best and the simplest way to go. Since in
> most case data from the cache should be later copied to user, one more
> page cache lookup should be negligible.

After the initial proposal, two merits come to my mind to implement
cooperative sequential I/O detection in the readahead code:

1) readahead can make larger(hence more efficient) I/O requests
2) the page cache lookup trick eliminates the overheads of extra rbtrees.

So I would definitely like to try it out :-)

Thank you,
Fengguang

2008-11-25 11:31:20

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Tue, Nov 25, 2008 at 01:59:53PM +0300, Vladislav Bolkhovitin wrote:
> Wu Fengguang wrote:
>> Hi all,
>>
>> //Sorry for being late.
>>
>> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
>> [...]
>>> I already talked about this with Jeff on irc, but I guess should post it
>>> here as well.
>>>
>>> nfsd aside (which does seem to have some different behaviour skewing the
>>> results), the original patch came about because dump(8) has a really
>>> stupid design that offloads IO to a number of processes. This basically
>>> makes fairly sequential IO more random with CFQ, since each process gets
>>> its own io context. My feeling is that we should fix dump instead of
>>> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>>> aware of any other good programs out there that would do something
>>> similar, so I don't think there's a lot of merrit to spending cycles on
>>> detecting cooperating processes.
>>>
>>> Jeff will take a look at fixing dump instead, and I may have promised
>>> him that santa will bring him something nice this year if he does (since
>>> I'm sure it'll be painful on the eyes).
>>
>> This could also be fixed at the VFS readahead level.
>>
>> In fact I've seen many kinds of interleaved accesses:
>> - concurrently reading 40 files that are in fact hard links of one single file
>> - a backup tool that splits a big file into 8k chunks, and serve the
>> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
>> chunks in another one
>> - a pool of NFSDs randomly serving some originally sequential read
>> requests - now dump(8) seems to have some similar problem.
>>
>> In summary there have been all kinds of efforts on trying to
>> parallelize I/O tasks, but unfortunately they can easily screw up the
>> sequential pattern. It may not be easily fixable for many of them.
>>
>> It is however possible to detect most of these patterns at the
>> readahead layer and restore sequential I/Os, before they propagate
>> into the block layer and hurt performance.
>
> I believe this would be the most effective way to go, especially in case
> if data delivery path to the original client has its own latency
> depended from the amount of transferred data as it is in the case of
> remote NFS mount, which does synchronous sequential reads. In this case
> it is essential for performance to make both links (local to the storage
> and network to the client) be always busy and transfer data
> simultaneously. Since the reads are synchronous, the only way to achieve
> that is perform read ahead on the server sufficient to cover the network
> link latency. Otherwise you would end up with only half of possible
> throughput.
>
> However, from one side, server has to have a pool of threads/processes
> to perform well, but, from other side, current read ahead code doesn't
> detect too well that those threads/processes are doing joint sequential
> read, so the read ahead window gets smaller, hence the overall read
> performance gets considerably smaller too.
>
>> Vitaly, if that's what you need, I can try to prepare a patch for testing out.
>
> I can test it with SCST SCSI target sybsystem (http://scst.sf.net). SCST
> needs such feature very much, otherwise it can't get full backstorage
> read speed. The maximum I can see is about ~80MB/s from ~130MB/s 15K RPM
> disk over 1Gbps iSCSI link (maximum possible is ~110MB/s).

Thank you very much!

BTW, do you implicate that the SCSI system (or its applications) has
similar behaviors that the current readahead code cannot handle well?

Thanks,
Fengguang

2008-11-25 11:42:25

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Wu Fengguang wrote:
> On Tue, Nov 25, 2008 at 01:59:53PM +0300, Vladislav Bolkhovitin wrote:
>> Wu Fengguang wrote:
>>> Hi all,
>>>
>>> //Sorry for being late.
>>>
>>> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
>>> [...]
>>>> I already talked about this with Jeff on irc, but I guess should post it
>>>> here as well.
>>>>
>>>> nfsd aside (which does seem to have some different behaviour skewing the
>>>> results), the original patch came about because dump(8) has a really
>>>> stupid design that offloads IO to a number of processes. This basically
>>>> makes fairly sequential IO more random with CFQ, since each process gets
>>>> its own io context. My feeling is that we should fix dump instead of
>>>> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>>>> aware of any other good programs out there that would do something
>>>> similar, so I don't think there's a lot of merrit to spending cycles on
>>>> detecting cooperating processes.
>>>>
>>>> Jeff will take a look at fixing dump instead, and I may have promised
>>>> him that santa will bring him something nice this year if he does (since
>>>> I'm sure it'll be painful on the eyes).
>>> This could also be fixed at the VFS readahead level.
>>>
>>> In fact I've seen many kinds of interleaved accesses:
>>> - concurrently reading 40 files that are in fact hard links of one single file
>>> - a backup tool that splits a big file into 8k chunks, and serve the
>>> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
>>> chunks in another one
>>> - a pool of NFSDs randomly serving some originally sequential read
>>> requests - now dump(8) seems to have some similar problem.
>>>
>>> In summary there have been all kinds of efforts on trying to
>>> parallelize I/O tasks, but unfortunately they can easily screw up the
>>> sequential pattern. It may not be easily fixable for many of them.
>>>
>>> It is however possible to detect most of these patterns at the
>>> readahead layer and restore sequential I/Os, before they propagate
>>> into the block layer and hurt performance.
>> I believe this would be the most effective way to go, especially in case
>> if data delivery path to the original client has its own latency
>> depended from the amount of transferred data as it is in the case of
>> remote NFS mount, which does synchronous sequential reads. In this case
>> it is essential for performance to make both links (local to the storage
>> and network to the client) be always busy and transfer data
>> simultaneously. Since the reads are synchronous, the only way to achieve
>> that is perform read ahead on the server sufficient to cover the network
>> link latency. Otherwise you would end up with only half of possible
>> throughput.
>>
>> However, from one side, server has to have a pool of threads/processes
>> to perform well, but, from other side, current read ahead code doesn't
>> detect too well that those threads/processes are doing joint sequential
>> read, so the read ahead window gets smaller, hence the overall read
>> performance gets considerably smaller too.
>>
>>> Vitaly, if that's what you need, I can try to prepare a patch for testing out.
>> I can test it with SCST SCSI target sybsystem (http://scst.sf.net). SCST
>> needs such feature very much, otherwise it can't get full backstorage
>> read speed. The maximum I can see is about ~80MB/s from ~130MB/s 15K RPM
>> disk over 1Gbps iSCSI link (maximum possible is ~110MB/s).
>
> Thank you very much!
>
> BTW, do you implicate that the SCSI system (or its applications) has
> similar behaviors that the current readahead code cannot handle well?

No. SCSI target subsystem is not the same as SCSI initiator subsystem,
which usually called simply SCSI (sub)system. SCSI target is a SCSI
server. It has the same amount of common with SCSI initiator as there
is, e.g., between Apache (HTTP server) and Firefox (HTTP client).

> Thanks,
> Fengguang
>

2008-11-25 11:49:39

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Tue, Nov 25, 2008 at 02:41:47PM +0300, Vladislav Bolkhovitin wrote:
> Wu Fengguang wrote:
>> On Tue, Nov 25, 2008 at 01:59:53PM +0300, Vladislav Bolkhovitin wrote:
>>> Wu Fengguang wrote:
>>>> Hi all,
>>>>
>>>> //Sorry for being late.
>>>>
>>>> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
>>>> [...]
>>>>> I already talked about this with Jeff on irc, but I guess should post it
>>>>> here as well.
>>>>>
>>>>> nfsd aside (which does seem to have some different behaviour skewing the
>>>>> results), the original patch came about because dump(8) has a really
>>>>> stupid design that offloads IO to a number of processes. This basically
>>>>> makes fairly sequential IO more random with CFQ, since each process gets
>>>>> its own io context. My feeling is that we should fix dump instead of
>>>>> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>>>>> aware of any other good programs out there that would do something
>>>>> similar, so I don't think there's a lot of merrit to spending cycles on
>>>>> detecting cooperating processes.
>>>>>
>>>>> Jeff will take a look at fixing dump instead, and I may have promised
>>>>> him that santa will bring him something nice this year if he does (since
>>>>> I'm sure it'll be painful on the eyes).
>>>> This could also be fixed at the VFS readahead level.
>>>>
>>>> In fact I've seen many kinds of interleaved accesses:
>>>> - concurrently reading 40 files that are in fact hard links of one single file
>>>> - a backup tool that splits a big file into 8k chunks, and serve the
>>>> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
>>>> chunks in another one
>>>> - a pool of NFSDs randomly serving some originally sequential read
>>>> requests - now dump(8) seems to have some similar problem.
>>>>
>>>> In summary there have been all kinds of efforts on trying to
>>>> parallelize I/O tasks, but unfortunately they can easily screw up the
>>>> sequential pattern. It may not be easily fixable for many of them.
>>>>
>>>> It is however possible to detect most of these patterns at the
>>>> readahead layer and restore sequential I/Os, before they propagate
>>>> into the block layer and hurt performance.
>>> I believe this would be the most effective way to go, especially in
>>> case if data delivery path to the original client has its own
>>> latency depended from the amount of transferred data as it is in the
>>> case of remote NFS mount, which does synchronous sequential reads.
>>> In this case it is essential for performance to make both links
>>> (local to the storage and network to the client) be always busy and
>>> transfer data simultaneously. Since the reads are synchronous, the
>>> only way to achieve that is perform read ahead on the server
>>> sufficient to cover the network link latency. Otherwise you would
>>> end up with only half of possible throughput.
>>>
>>> However, from one side, server has to have a pool of
>>> threads/processes to perform well, but, from other side, current
>>> read ahead code doesn't detect too well that those threads/processes
>>> are doing joint sequential read, so the read ahead window gets
>>> smaller, hence the overall read performance gets considerably
>>> smaller too.
>>>
>>>> Vitaly, if that's what you need, I can try to prepare a patch for testing out.
>>> I can test it with SCST SCSI target sybsystem (http://scst.sf.net).
>>> SCST needs such feature very much, otherwise it can't get full
>>> backstorage read speed. The maximum I can see is about ~80MB/s from
>>> ~130MB/s 15K RPM disk over 1Gbps iSCSI link (maximum possible is
>>> ~110MB/s).
>>
>> Thank you very much!
>>
>> BTW, do you implicate that the SCSI system (or its applications) has
>> similar behaviors that the current readahead code cannot handle well?
>
> No. SCSI target subsystem is not the same as SCSI initiator subsystem,
> which usually called simply SCSI (sub)system. SCSI target is a SCSI
> server. It has the same amount of common with SCSI initiator as there
> is, e.g., between Apache (HTTP server) and Firefox (HTTP client).

Got it. So the SCSI server will split&spread sequential IO of one
single file to cooperative threads? I'm trying to understand why the
proposed page cache context based readahead would help a SCSI server.

Thanks,
Fengguang

2008-11-25 12:03:18

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Wu Fengguang wrote:
> On Tue, Nov 25, 2008 at 02:41:47PM +0300, Vladislav Bolkhovitin wrote:
>> Wu Fengguang wrote:
>>> On Tue, Nov 25, 2008 at 01:59:53PM +0300, Vladislav Bolkhovitin wrote:
>>>> Wu Fengguang wrote:
>>>>> Hi all,
>>>>>
>>>>> //Sorry for being late.
>>>>>
>>>>> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
>>>>> [...]
>>>>>> I already talked about this with Jeff on irc, but I guess should post it
>>>>>> here as well.
>>>>>>
>>>>>> nfsd aside (which does seem to have some different behaviour skewing the
>>>>>> results), the original patch came about because dump(8) has a really
>>>>>> stupid design that offloads IO to a number of processes. This basically
>>>>>> makes fairly sequential IO more random with CFQ, since each process gets
>>>>>> its own io context. My feeling is that we should fix dump instead of
>>>>>> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>>>>>> aware of any other good programs out there that would do something
>>>>>> similar, so I don't think there's a lot of merrit to spending cycles on
>>>>>> detecting cooperating processes.
>>>>>>
>>>>>> Jeff will take a look at fixing dump instead, and I may have promised
>>>>>> him that santa will bring him something nice this year if he does (since
>>>>>> I'm sure it'll be painful on the eyes).
>>>>> This could also be fixed at the VFS readahead level.
>>>>>
>>>>> In fact I've seen many kinds of interleaved accesses:
>>>>> - concurrently reading 40 files that are in fact hard links of one single file
>>>>> - a backup tool that splits a big file into 8k chunks, and serve the
>>>>> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
>>>>> chunks in another one
>>>>> - a pool of NFSDs randomly serving some originally sequential read
>>>>> requests - now dump(8) seems to have some similar problem.
>>>>>
>>>>> In summary there have been all kinds of efforts on trying to
>>>>> parallelize I/O tasks, but unfortunately they can easily screw up the
>>>>> sequential pattern. It may not be easily fixable for many of them.
>>>>>
>>>>> It is however possible to detect most of these patterns at the
>>>>> readahead layer and restore sequential I/Os, before they propagate
>>>>> into the block layer and hurt performance.
>>>> I believe this would be the most effective way to go, especially in
>>>> case if data delivery path to the original client has its own
>>>> latency depended from the amount of transferred data as it is in the
>>>> case of remote NFS mount, which does synchronous sequential reads.
>>>> In this case it is essential for performance to make both links
>>>> (local to the storage and network to the client) be always busy and
>>>> transfer data simultaneously. Since the reads are synchronous, the
>>>> only way to achieve that is perform read ahead on the server
>>>> sufficient to cover the network link latency. Otherwise you would
>>>> end up with only half of possible throughput.
>>>>
>>>> However, from one side, server has to have a pool of
>>>> threads/processes to perform well, but, from other side, current
>>>> read ahead code doesn't detect too well that those threads/processes
>>>> are doing joint sequential read, so the read ahead window gets
>>>> smaller, hence the overall read performance gets considerably
>>>> smaller too.
>>>>
>>>>> Vitaly, if that's what you need, I can try to prepare a patch for testing out.
>>>> I can test it with SCST SCSI target sybsystem (http://scst.sf.net).
>>>> SCST needs such feature very much, otherwise it can't get full
>>>> backstorage read speed. The maximum I can see is about ~80MB/s from
>>>> ~130MB/s 15K RPM disk over 1Gbps iSCSI link (maximum possible is
>>>> ~110MB/s).
>>> Thank you very much!
>>>
>>> BTW, do you implicate that the SCSI system (or its applications) has
>>> similar behaviors that the current readahead code cannot handle well?
>> No. SCSI target subsystem is not the same as SCSI initiator subsystem,
>> which usually called simply SCSI (sub)system. SCSI target is a SCSI
>> server. It has the same amount of common with SCSI initiator as there
>> is, e.g., between Apache (HTTP server) and Firefox (HTTP client).
>
> Got it. So the SCSI server will split&spread sequential IO of one
> single file to cooperative threads?

Yes. It has to do so, because Linux doesn't have async. cached IO and a
client can queue several tens of commands at time. Then, on the
sequential IO with 1 command at time, CPU scheduler comes to play and
spreads those commands over those threads, so read ahead gets too small
to cover the external link latency and fill both links with data, so
that uncovered latency kills throughput.

> I'm trying to understand why the
> proposed page cache context based readahead would help a SCSI server.
>
> Thanks,
> Fengguang
>

2008-11-25 12:09:22

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Vladislav Bolkhovitin wrote:
> Wu Fengguang wrote:
>> On Tue, Nov 25, 2008 at 02:41:47PM +0300, Vladislav Bolkhovitin wrote:
>>> Wu Fengguang wrote:
>>>> On Tue, Nov 25, 2008 at 01:59:53PM +0300, Vladislav Bolkhovitin wrote:
>>>>> Wu Fengguang wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> //Sorry for being late.
>>>>>>
>>>>>> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
>>>>>> [...]
>>>>>>> I already talked about this with Jeff on irc, but I guess should post it
>>>>>>> here as well.
>>>>>>>
>>>>>>> nfsd aside (which does seem to have some different behaviour skewing the
>>>>>>> results), the original patch came about because dump(8) has a really
>>>>>>> stupid design that offloads IO to a number of processes. This basically
>>>>>>> makes fairly sequential IO more random with CFQ, since each process gets
>>>>>>> its own io context. My feeling is that we should fix dump instead of
>>>>>>> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>>>>>>> aware of any other good programs out there that would do something
>>>>>>> similar, so I don't think there's a lot of merrit to spending cycles on
>>>>>>> detecting cooperating processes.
>>>>>>>
>>>>>>> Jeff will take a look at fixing dump instead, and I may have promised
>>>>>>> him that santa will bring him something nice this year if he does (since
>>>>>>> I'm sure it'll be painful on the eyes).
>>>>>> This could also be fixed at the VFS readahead level.
>>>>>>
>>>>>> In fact I've seen many kinds of interleaved accesses:
>>>>>> - concurrently reading 40 files that are in fact hard links of one single file
>>>>>> - a backup tool that splits a big file into 8k chunks, and serve the
>>>>>> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
>>>>>> chunks in another one
>>>>>> - a pool of NFSDs randomly serving some originally sequential read
>>>>>> requests - now dump(8) seems to have some similar problem.
>>>>>>
>>>>>> In summary there have been all kinds of efforts on trying to
>>>>>> parallelize I/O tasks, but unfortunately they can easily screw up the
>>>>>> sequential pattern. It may not be easily fixable for many of them.
>>>>>>
>>>>>> It is however possible to detect most of these patterns at the
>>>>>> readahead layer and restore sequential I/Os, before they propagate
>>>>>> into the block layer and hurt performance.
>>>>> I believe this would be the most effective way to go, especially in
>>>>> case if data delivery path to the original client has its own
>>>>> latency depended from the amount of transferred data as it is in the
>>>>> case of remote NFS mount, which does synchronous sequential reads.
>>>>> In this case it is essential for performance to make both links
>>>>> (local to the storage and network to the client) be always busy and
>>>>> transfer data simultaneously. Since the reads are synchronous, the
>>>>> only way to achieve that is perform read ahead on the server
>>>>> sufficient to cover the network link latency. Otherwise you would
>>>>> end up with only half of possible throughput.
>>>>>
>>>>> However, from one side, server has to have a pool of
>>>>> threads/processes to perform well, but, from other side, current
>>>>> read ahead code doesn't detect too well that those threads/processes
>>>>> are doing joint sequential read, so the read ahead window gets
>>>>> smaller, hence the overall read performance gets considerably
>>>>> smaller too.
>>>>>
>>>>>> Vitaly, if that's what you need, I can try to prepare a patch for testing out.
>>>>> I can test it with SCST SCSI target sybsystem (http://scst.sf.net).
>>>>> SCST needs such feature very much, otherwise it can't get full
>>>>> backstorage read speed. The maximum I can see is about ~80MB/s from
>>>>> ~130MB/s 15K RPM disk over 1Gbps iSCSI link (maximum possible is
>>>>> ~110MB/s).
>>>> Thank you very much!
>>>>
>>>> BTW, do you implicate that the SCSI system (or its applications) has
>>>> similar behaviors that the current readahead code cannot handle well?
>>> No. SCSI target subsystem is not the same as SCSI initiator subsystem,
>>> which usually called simply SCSI (sub)system. SCSI target is a SCSI
>>> server. It has the same amount of common with SCSI initiator as there
>>> is, e.g., between Apache (HTTP server) and Firefox (HTTP client).
>> Got it. So the SCSI server will split&spread sequential IO of one
>> single file to cooperative threads?
>
> Yes. It has to do so, because Linux doesn't have async. cached IO and a
> client can queue several tens of commands at time. Then, on the
> sequential IO with 1 command at time, CPU scheduler comes to play and
> spreads those commands over those threads, so read ahead gets too small
> to cover the external link latency and fill both links with data, so
> that uncovered latency kills throughput.

Additionally, if the uncovered external link latency is too large, one
more factor is getting noticeable: storage rotation latency. If the next
unread sector is missed to be read at time, server has to wait a full
rotation to start receiving data for the next block, which even more
decreases the resulting throughput.

>> I'm trying to understand why the
>> proposed page cache context based readahead would help a SCSI server.
>>
>> Thanks,
>> Fengguang
>>
>
>

2008-11-25 12:16:27

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Tue, Nov 25, 2008 at 03:09:12PM +0300, Vladislav Bolkhovitin wrote:
> Vladislav Bolkhovitin wrote:
>> Wu Fengguang wrote:
>>> On Tue, Nov 25, 2008 at 02:41:47PM +0300, Vladislav Bolkhovitin wrote:
>>>> Wu Fengguang wrote:
>>>>> On Tue, Nov 25, 2008 at 01:59:53PM +0300, Vladislav Bolkhovitin wrote:
>>>>>> Wu Fengguang wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> //Sorry for being late.
>>>>>>>
>>>>>>> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
>>>>>>> [...]
>>>>>>>> I already talked about this with Jeff on irc, but I guess should post it
>>>>>>>> here as well.
>>>>>>>>
>>>>>>>> nfsd aside (which does seem to have some different behaviour skewing the
>>>>>>>> results), the original patch came about because dump(8) has a really
>>>>>>>> stupid design that offloads IO to a number of processes. This basically
>>>>>>>> makes fairly sequential IO more random with CFQ, since each process gets
>>>>>>>> its own io context. My feeling is that we should fix dump instead of
>>>>>>>> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>>>>>>>> aware of any other good programs out there that would do something
>>>>>>>> similar, so I don't think there's a lot of merrit to spending cycles on
>>>>>>>> detecting cooperating processes.
>>>>>>>>
>>>>>>>> Jeff will take a look at fixing dump instead, and I may have promised
>>>>>>>> him that santa will bring him something nice this year if he does (since
>>>>>>>> I'm sure it'll be painful on the eyes).
>>>>>>> This could also be fixed at the VFS readahead level.
>>>>>>>
>>>>>>> In fact I've seen many kinds of interleaved accesses:
>>>>>>> - concurrently reading 40 files that are in fact hard links of one single file
>>>>>>> - a backup tool that splits a big file into 8k chunks, and serve the
>>>>>>> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
>>>>>>> chunks in another one
>>>>>>> - a pool of NFSDs randomly serving some originally sequential
>>>>>>> read requests - now dump(8) seems to have some similar
>>>>>>> problem.
>>>>>>>
>>>>>>> In summary there have been all kinds of efforts on trying to
>>>>>>> parallelize I/O tasks, but unfortunately they can easily screw up the
>>>>>>> sequential pattern. It may not be easily fixable for many of them.
>>>>>>>
>>>>>>> It is however possible to detect most of these patterns at the
>>>>>>> readahead layer and restore sequential I/Os, before they propagate
>>>>>>> into the block layer and hurt performance.
>>>>>> I believe this would be the most effective way to go,
>>>>>> especially in case if data delivery path to the original
>>>>>> client has its own latency depended from the amount of
>>>>>> transferred data as it is in the case of remote NFS mount,
>>>>>> which does synchronous sequential reads. In this case it is
>>>>>> essential for performance to make both links (local to the
>>>>>> storage and network to the client) be always busy and
>>>>>> transfer data simultaneously. Since the reads are synchronous,
>>>>>> the only way to achieve that is perform read ahead on the
>>>>>> server sufficient to cover the network link latency. Otherwise
>>>>>> you would end up with only half of possible throughput.
>>>>>>
>>>>>> However, from one side, server has to have a pool of
>>>>>> threads/processes to perform well, but, from other side,
>>>>>> current read ahead code doesn't detect too well that those
>>>>>> threads/processes are doing joint sequential read, so the read
>>>>>> ahead window gets smaller, hence the overall read performance
>>>>>> gets considerably smaller too.
>>>>>>
>>>>>>> Vitaly, if that's what you need, I can try to prepare a patch for testing out.
>>>>>> I can test it with SCST SCSI target sybsystem
>>>>>> (http://scst.sf.net). SCST needs such feature very much,
>>>>>> otherwise it can't get full backstorage read speed. The
>>>>>> maximum I can see is about ~80MB/s from ~130MB/s 15K RPM disk
>>>>>> over 1Gbps iSCSI link (maximum possible is ~110MB/s).
>>>>> Thank you very much!
>>>>>
>>>>> BTW, do you implicate that the SCSI system (or its applications) has
>>>>> similar behaviors that the current readahead code cannot handle well?
>>>> No. SCSI target subsystem is not the same as SCSI initiator
>>>> subsystem, which usually called simply SCSI (sub)system. SCSI
>>>> target is a SCSI server. It has the same amount of common with
>>>> SCSI initiator as there is, e.g., between Apache (HTTP server) and
>>>> Firefox (HTTP client).
>>> Got it. So the SCSI server will split&spread sequential IO of one
>>> single file to cooperative threads?
>>
>> Yes. It has to do so, because Linux doesn't have async. cached IO and a
>> client can queue several tens of commands at time. Then, on the
>> sequential IO with 1 command at time, CPU scheduler comes to play and
>> spreads those commands over those threads, so read ahead gets too small
>> to cover the external link latency and fill both links with data, so
>> that uncovered latency kills throughput.
>
> Additionally, if the uncovered external link latency is too large, one
> more factor is getting noticeable: storage rotation latency. If the next
> unread sector is missed to be read at time, server has to wait a full
> rotation to start receiving data for the next block, which even more
> decreases the resulting throughput.

Thank you for the details. I've been working slowly on the idea, and
should be able to send you a patch in the next one or two days.

Thanks,
Fengguang

2008-11-25 15:21:37

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Vladislav Bolkhovitin <[email protected]> writes:

> Wu Fengguang wrote:
>> Another scheme is to detect the sequential pattern via looking up
>> the page cache, which provides one single and consistent view of the
>> pages recently accessed. That makes sequential detection possible.
>>
>> The cost will be one extra page cache lookup per random read.
>> If it's not acceptable, the corresponding code could be disabled
>> by default.
>
> I think, this should be the best and the simplest way to go. Since in
> most case data from the cache should be later copied to user, one more
> page cache lookup should be negligible.

I haven't thought about your suggestion in any detail, but it seems to
me that you will not handle O_DIRECT I/O with this scheme.

Cheers,
Jeff

2008-11-25 16:17:19

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Jeff Moyer wrote:
> Vladislav Bolkhovitin <[email protected]> writes:
>
>> Wu Fengguang wrote:
>>> Another scheme is to detect the sequential pattern via looking up
>>> the page cache, which provides one single and consistent view of the
>>> pages recently accessed. That makes sequential detection possible.
>>>
>>> The cost will be one extra page cache lookup per random read.
>>> If it's not acceptable, the corresponding code could be disabled
>>> by default.
>> I think, this should be the best and the simplest way to go. Since in
>> most case data from the cache should be later copied to user, one more
>> page cache lookup should be negligible.
>
> I haven't thought about your suggestion in any detail, but it seems to
> me that you will not handle O_DIRECT I/O with this scheme.

And shouldn't. One using O_DIRECT is supposed to understand and accept
that no features provided by page cache, including read ahead, will be
available to him. Hence, if necessary, he must implement own read ahead
procedures.

> Cheers,
> Jeff
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2008-11-27 17:47:32

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Wu Fengguang wrote:
> On Tue, Nov 25, 2008 at 03:09:12PM +0300, Vladislav Bolkhovitin wrote:
>> Vladislav Bolkhovitin wrote:
>>> Wu Fengguang wrote:
>>>> On Tue, Nov 25, 2008 at 02:41:47PM +0300, Vladislav Bolkhovitin wrote:
>>>>> Wu Fengguang wrote:
>>>>>> On Tue, Nov 25, 2008 at 01:59:53PM +0300, Vladislav Bolkhovitin wrote:
>>>>>>> Wu Fengguang wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> //Sorry for being late.
>>>>>>>>
>>>>>>>> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
>>>>>>>> [...]
>>>>>>>>> I already talked about this with Jeff on irc, but I guess should post it
>>>>>>>>> here as well.
>>>>>>>>>
>>>>>>>>> nfsd aside (which does seem to have some different behaviour skewing the
>>>>>>>>> results), the original patch came about because dump(8) has a really
>>>>>>>>> stupid design that offloads IO to a number of processes. This basically
>>>>>>>>> makes fairly sequential IO more random with CFQ, since each process gets
>>>>>>>>> its own io context. My feeling is that we should fix dump instead of
>>>>>>>>> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>>>>>>>>> aware of any other good programs out there that would do something
>>>>>>>>> similar, so I don't think there's a lot of merrit to spending cycles on
>>>>>>>>> detecting cooperating processes.
>>>>>>>>>
>>>>>>>>> Jeff will take a look at fixing dump instead, and I may have promised
>>>>>>>>> him that santa will bring him something nice this year if he does (since
>>>>>>>>> I'm sure it'll be painful on the eyes).
>>>>>>>> This could also be fixed at the VFS readahead level.
>>>>>>>>
>>>>>>>> In fact I've seen many kinds of interleaved accesses:
>>>>>>>> - concurrently reading 40 files that are in fact hard links of one single file
>>>>>>>> - a backup tool that splits a big file into 8k chunks, and serve the
>>>>>>>> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
>>>>>>>> chunks in another one
>>>>>>>> - a pool of NFSDs randomly serving some originally sequential
>>>>>>>> read requests - now dump(8) seems to have some similar
>>>>>>>> problem.
>>>>>>>>
>>>>>>>> In summary there have been all kinds of efforts on trying to
>>>>>>>> parallelize I/O tasks, but unfortunately they can easily screw up the
>>>>>>>> sequential pattern. It may not be easily fixable for many of them.
>>>>>>>>
>>>>>>>> It is however possible to detect most of these patterns at the
>>>>>>>> readahead layer and restore sequential I/Os, before they propagate
>>>>>>>> into the block layer and hurt performance.
>>>>>>> I believe this would be the most effective way to go,
>>>>>>> especially in case if data delivery path to the original
>>>>>>> client has its own latency depended from the amount of
>>>>>>> transferred data as it is in the case of remote NFS mount,
>>>>>>> which does synchronous sequential reads. In this case it is
>>>>>>> essential for performance to make both links (local to the
>>>>>>> storage and network to the client) be always busy and
>>>>>>> transfer data simultaneously. Since the reads are synchronous,
>>>>>>> the only way to achieve that is perform read ahead on the
>>>>>>> server sufficient to cover the network link latency. Otherwise
>>>>>>> you would end up with only half of possible throughput.
>>>>>>>
>>>>>>> However, from one side, server has to have a pool of
>>>>>>> threads/processes to perform well, but, from other side,
>>>>>>> current read ahead code doesn't detect too well that those
>>>>>>> threads/processes are doing joint sequential read, so the read
>>>>>>> ahead window gets smaller, hence the overall read performance
>>>>>>> gets considerably smaller too.
>>>>>>>
>>>>>>>> Vitaly, if that's what you need, I can try to prepare a patch for testing out.
>>>>>>> I can test it with SCST SCSI target sybsystem
>>>>>>> (http://scst.sf.net). SCST needs such feature very much,
>>>>>>> otherwise it can't get full backstorage read speed. The
>>>>>>> maximum I can see is about ~80MB/s from ~130MB/s 15K RPM disk
>>>>>>> over 1Gbps iSCSI link (maximum possible is ~110MB/s).
>>>>>> Thank you very much!
>>>>>>
>>>>>> BTW, do you implicate that the SCSI system (or its applications) has
>>>>>> similar behaviors that the current readahead code cannot handle well?
>>>>> No. SCSI target subsystem is not the same as SCSI initiator
>>>>> subsystem, which usually called simply SCSI (sub)system. SCSI
>>>>> target is a SCSI server. It has the same amount of common with
>>>>> SCSI initiator as there is, e.g., between Apache (HTTP server) and
>>>>> Firefox (HTTP client).
>>>> Got it. So the SCSI server will split&spread sequential IO of one
>>>> single file to cooperative threads?
>>> Yes. It has to do so, because Linux doesn't have async. cached IO and a
>>> client can queue several tens of commands at time. Then, on the
>>> sequential IO with 1 command at time, CPU scheduler comes to play and
>>> spreads those commands over those threads, so read ahead gets too small
>>> to cover the external link latency and fill both links with data, so
>>> that uncovered latency kills throughput.
>> Additionally, if the uncovered external link latency is too large, one
>> more factor is getting noticeable: storage rotation latency. If the next
>> unread sector is missed to be read at time, server has to wait a full
>> rotation to start receiving data for the next block, which even more
>> decreases the resulting throughput.
>
> Thank you for the details. I've been working slowly on the idea, and
> should be able to send you a patch in the next one or two days.

Actually, there's one more thing, which should have been mentioned. It
is possible that remote clients have several sequential read streams at
time together with some "noise" of random requests. A good read-ahead
subsystem should handle such case by keeping big read-ahead windows for
the sequential streams and don't do any read ahead for the random
requests. And all at the same time.

Currently on such workloads read ahead will be completely disabled for
all the requests. Hence, there is a possibility here to improve
performance in 3-5 times or even more by making the workload more linear.

I guess, such functionality can be done by keeping several read-ahead
contexts not in file handle as now, but in inode, or ever in task_struct
similarly to io_context. Or even as a part of struct io_context. Then
those contexts would be rotated in, e.g., round robin manner. I have
some basic thoughts in this area and would be glad to share them, if you
are interested.

Going further, ultimately, it would be great to provide somehow a
capability to allow to assign for each read-ahead stream own IO context,
because on the remote side in most cases such streams would correspond
to different programs reading data in parallel. This should allow to
increase performance even more.

> Thanks,
> Fengguang
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2008-11-28 00:49:34

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Thu, Nov 27, 2008 at 08:46:35PM +0300, Vladislav Bolkhovitin wrote:
>
> Wu Fengguang wrote:
>> On Tue, Nov 25, 2008 at 03:09:12PM +0300, Vladislav Bolkhovitin wrote:
>>> Vladislav Bolkhovitin wrote:
>>>> Wu Fengguang wrote:
>>>>> On Tue, Nov 25, 2008 at 02:41:47PM +0300, Vladislav Bolkhovitin wrote:
>>>>>> Wu Fengguang wrote:
>>>>>>> On Tue, Nov 25, 2008 at 01:59:53PM +0300, Vladislav Bolkhovitin wrote:
>>>>>>>> Wu Fengguang wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> //Sorry for being late.
>>>>>>>>>
>>>>>>>>> On Wed, Nov 12, 2008 at 08:02:28PM +0100, Jens Axboe wrote:
>>>>>>>>> [...]
>>>>>>>>>> I already talked about this with Jeff on irc, but I guess should post it
>>>>>>>>>> here as well.
>>>>>>>>>>
>>>>>>>>>> nfsd aside (which does seem to have some different behaviour skewing the
>>>>>>>>>> results), the original patch came about because dump(8) has a really
>>>>>>>>>> stupid design that offloads IO to a number of processes. This basically
>>>>>>>>>> makes fairly sequential IO more random with CFQ, since each process gets
>>>>>>>>>> its own io context. My feeling is that we should fix dump instead of
>>>>>>>>>> introducing a fair bit of complexity (and slowdown) in CFQ. I'm not
>>>>>>>>>> aware of any other good programs out there that would do something
>>>>>>>>>> similar, so I don't think there's a lot of merrit to spending cycles on
>>>>>>>>>> detecting cooperating processes.
>>>>>>>>>>
>>>>>>>>>> Jeff will take a look at fixing dump instead, and I may have promised
>>>>>>>>>> him that santa will bring him something nice this year if he does (since
>>>>>>>>>> I'm sure it'll be painful on the eyes).
>>>>>>>>> This could also be fixed at the VFS readahead level.
>>>>>>>>>
>>>>>>>>> In fact I've seen many kinds of interleaved accesses:
>>>>>>>>> - concurrently reading 40 files that are in fact hard links of one single file
>>>>>>>>> - a backup tool that splits a big file into 8k chunks, and serve the
>>>>>>>>> {1, 3, 5, 7, ...} chunks in one process and the {0, 2, 4, 6, ...}
>>>>>>>>> chunks in another one
>>>>>>>>> - a pool of NFSDs randomly serving some originally
>>>>>>>>> sequential read requests - now dump(8) seems to have
>>>>>>>>> some similar problem.
>>>>>>>>>
>>>>>>>>> In summary there have been all kinds of efforts on trying to
>>>>>>>>> parallelize I/O tasks, but unfortunately they can easily screw up the
>>>>>>>>> sequential pattern. It may not be easily fixable for many of them.
>>>>>>>>>
>>>>>>>>> It is however possible to detect most of these patterns at the
>>>>>>>>> readahead layer and restore sequential I/Os, before they propagate
>>>>>>>>> into the block layer and hurt performance.
>>>>>>>> I believe this would be the most effective way to go,
>>>>>>>> especially in case if data delivery path to the original
>>>>>>>> client has its own latency depended from the amount of
>>>>>>>> transferred data as it is in the case of remote NFS mount,
>>>>>>>> which does synchronous sequential reads. In this case it
>>>>>>>> is essential for performance to make both links (local to
>>>>>>>> the storage and network to the client) be always busy and
>>>>>>>> transfer data simultaneously. Since the reads are
>>>>>>>> synchronous, the only way to achieve that is perform read
>>>>>>>> ahead on the server sufficient to cover the network link
>>>>>>>> latency. Otherwise you would end up with only half of
>>>>>>>> possible throughput.
>>>>>>>>
>>>>>>>> However, from one side, server has to have a pool of
>>>>>>>> threads/processes to perform well, but, from other side,
>>>>>>>> current read ahead code doesn't detect too well that those
>>>>>>>> threads/processes are doing joint sequential read, so the
>>>>>>>> read ahead window gets smaller, hence the overall read
>>>>>>>> performance gets considerably smaller too.
>>>>>>>>
>>>>>>>>> Vitaly, if that's what you need, I can try to prepare a patch for testing out.
>>>>>>>> I can test it with SCST SCSI target sybsystem
>>>>>>>> (http://scst.sf.net). SCST needs such feature very much,
>>>>>>>> otherwise it can't get full backstorage read speed. The
>>>>>>>> maximum I can see is about ~80MB/s from ~130MB/s 15K RPM
>>>>>>>> disk over 1Gbps iSCSI link (maximum possible is ~110MB/s).
>>>>>>> Thank you very much!
>>>>>>>
>>>>>>> BTW, do you implicate that the SCSI system (or its applications) has
>>>>>>> similar behaviors that the current readahead code cannot handle well?
>>>>>> No. SCSI target subsystem is not the same as SCSI initiator
>>>>>> subsystem, which usually called simply SCSI (sub)system. SCSI
>>>>>> target is a SCSI server. It has the same amount of common with
>>>>>> SCSI initiator as there is, e.g., between Apache (HTTP server)
>>>>>> and Firefox (HTTP client).
>>>>> Got it. So the SCSI server will split&spread sequential IO of one
>>>>> single file to cooperative threads?
>>>> Yes. It has to do so, because Linux doesn't have async. cached IO
>>>> and a client can queue several tens of commands at time. Then, on
>>>> the sequential IO with 1 command at time, CPU scheduler comes to
>>>> play and spreads those commands over those threads, so read ahead
>>>> gets too small to cover the external link latency and fill both
>>>> links with data, so that uncovered latency kills throughput.
>>> Additionally, if the uncovered external link latency is too large,
>>> one more factor is getting noticeable: storage rotation latency. If
>>> the next unread sector is missed to be read at time, server has to
>>> wait a full rotation to start receiving data for the next block,
>>> which even more decreases the resulting throughput.
>
>> Thank you for the details. I've been working slowly on the idea, and
>> should be able to send you a patch in the next one or two days.
>
> Actually, there's one more thing, which should have been mentioned. It
> is possible that remote clients have several sequential read streams at
> time together with some "noise" of random requests. A good read-ahead
> subsystem should handle such case by keeping big read-ahead windows for
> the sequential streams and don't do any read ahead for the random
> requests. And all at the same time.
>
> Currently on such workloads read ahead will be completely disabled for
> all the requests. Hence, there is a possibility here to improve
> performance in 3-5 times or even more by making the workload more linear.

Are you sure? I'd expect such mixed-sequential-random pattern to be
handled by the current readahead code pretty well: sequential ones
will get large readahead and random ones won't get readahead at all.

Attached is the context readahead patch plus a kernel module for
readahead tracing and accounting, which will hopefully help clarify the
read patterns and readahead behaviors on production workloads. It is
based on 2.6.27 for your convenience, but also applies to 2.6.28.

The patch is not targeted for code review, but if anyone are interested,
you can take a look at try_context_readahead(). This is the only newly
introduced readahead policy, the other majorities are code refactor
and tracing facilities.

The newly introduced context readahead policy is disabled by default.
To enable it:
echo 1 > /sys/block/sda/queue/context_readahead
I'm not sure for now whether this parameter will be a long term one, or
whether the context readahead policy should be enabled unconditionally.

The readahead accounting stats can be viewed by
mount -t debugfs none /sys/kernel/debug
cat /sys/kernel/debug/readahead/stats
The numbers can be reset by
echo > /sys/kernel/debug/readahead/stats

Here is a sample output from my desktop:

% cat /sys/kernel/debug/readahead/stats
pattern count sync_count eof_count size async_size actual
none 0 0 0 0 0 0
initial0 3009 3009 2033 5 4 2
initial 35 35 0 5 4 3
subsequent 1294 240 827 52 49 26
marker 220 0 109 54 53 29
trail 0 0 0 0 0 0
oversize 0 0 0 0 0 0
reverse 0 0 0 0 0 0
stride 0 0 0 0 0 0
thrash 0 0 0 0 0 0
mmap 2833 2833 1379 142 0 47
fadvise 7 7 7 0 0 40
random 7621 7621 69 1 0 1
all 15019 13745 4424 33 5 12

The readahead/read tracing messages are disabled by default.
To enable them:
echo 1 > /sys/kernel/debug/readahead/trace_enable
echo 1 > /sys/kernel/debug/readahead/read_jprobes
They(especially the latter one) will generate a lot of printk messages like:

[ 828.151013] readahead-initial0(pid=4644(zsh), dev=00:10(0:10), ino=351452(whoami), req=0+1, ra=0+4-3, async=0) = 4
[ 828.167853] readahead-mmap(pid=4644(whoami), dev=00:10(0:10), ino=351452(whoami), req=0+0, ra=0+60-0, async=0) = 3
[ 828.195652] readahead-initial0(pid=4629(zsh), dev=00:10(0:10), ino=115569(zsh_prompt), req=0+128, ra=0+120-60, async=0) = 3
[ 828.225081] readahead-initial0(pid=4629(zsh), dev=00:10(0:10), ino=342086(.zsh_history), req=0+128, ra=0+120-60, async=0) = 4

[ 964.471450] do_generic_file_read(pid=4685(zsh), dev=00:10(0:10), ino=351445(wc), pos=0, count=128)
[ 964.471544] do_generic_file_read(pid=4685(zsh), dev=00:10(0:10), ino=351445(wc), pos=64, count=448)
[ 964.471575] do_generic_file_read(pid=4685(zsh), dev=00:10(0:10), ino=351445(wc), pos=512, count=28)
[ 964.472659] do_generic_file_read(pid=4685(zsh), dev=00:10(0:10), ino=383002(ld-2.7.so), pos=0, count=128)
[ 964.473431] do_generic_file_read(pid=4685(wc), dev=00:10(0:10), ino=383002(ld-2.7.so), pos=64, count=336)
[ 964.475639] do_generic_file_read(pid=4685(wc), dev=00:10(0:10), ino=383010(libc-2.7.so), pos=0, count=832)
[ 964.479037] do_generic_file_read(pid=4685(wc), dev=00:10(0:10), ino=196085(locale.alias), pos=0, count=524288)
[ 964.479166] do_generic_file_read(pid=4685(wc), dev=00:10(0:10), ino=196085(locale.alias), pos=2586, count=524288)

So please enable them only when necessary.

My recommendation for the double readahead in NFS client and NFS servers,
is to keep client side readahead size small and the server side one large.
For example, 128K-512K/1M-2M(more for RAID). The NFS client side readahead size
is not directly tunable, but setting rsize to a small value does the trick.
Currently the NFS magic is readahead_size=N*rsize. The default numbers in my
2.6.28 kernel are rsize=512k, N=15, readahead_size=7680k. The latter is
obviously way too large.

Thanks,
Fengguang

> I guess, such functionality can be done by keeping several read-ahead
> contexts not in file handle as now, but in inode, or ever in task_struct
> similarly to io_context. Or even as a part of struct io_context. Then
> those contexts would be rotated in, e.g., round robin manner. I have
> some basic thoughts in this area and would be glad to share them, if you
> are interested.
>
> Going further, ultimately, it would be great to provide somehow a
> capability to allow to assign for each read-ahead stream own IO context,
> because on the remote side in most cases such streams would correspond
> to different programs reading data in parallel. This should allow to
> increase performance even more.
>
>> Thanks,
>> Fengguang
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at http://www.tux.org/lkml/
>>

Attachments:

(No filename) (11.77 kB)
readahead-context-plus-tracing-module-2.6.27.patch (25.01 kB)
Download all attachments

2009-03-19 17:43:45

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Wu Fengguang, on 02/19/2009 05:05 AM wrote:
> On Tue, Feb 17, 2009 at 10:01:40PM +0300, Vladislav Bolkhovitin wrote:
>> Wu Fengguang, on 02/13/2009 04:57 AM wrote:
>>> On Thu, Feb 12, 2009 at 09:35:18PM +0300, Vladislav Bolkhovitin wrote:
>>>> Sorry for such a huge delay. There were many other activities I had
>>>> to do before + I had to be sure I didn't miss anything.
>>>>
>>>> We didn't use NFS, we used SCST (http://scst.sourceforge.net) with
>>>> iSCSI-SCST target driver. It has similar to NFS architecture, where N
>>>> threads (N=5 in this case) handle IO from remote initiators
>>>> (clients) coming from wire using iSCSI protocol. In addition, SCST
>>>> has patch called export_alloc_io_context (see
>>>> http://lkml.org/lkml/2008/12/10/282), which allows for the IO threads
>>>> queue IO using single IO context, so we can see if context RA can
>>>> replace grouping IO threads in single IO context.
>>>>
>>>> Unfortunately, the results are negative. We find neither any
>>>> advantages of context RA over current RA implementation, nor
>>>> possibility for context RA to replace grouping IO threads in single
>>>> IO context.
>>>>
>>>> Setup on the target (server) was the following. 2 SATA drives grouped
>>>> in md RAID-0 with average local read throughput ~120MB/s ("dd
>>>> if=/dev/zero of=/dev/md0 bs=1M count=20000" outputs "20971520000
>>>> bytes (21 GB) copied, 177,742 s, 118 MB/s"). The md device was
>>>> partitioned on 3 partitions. The first partition was 10% of space in
>>>> the beginning of the device, the last partition was 10% of space in
>>>> the end of the device, the middle one was the rest in the middle of
>>>> the space them. Then the first and the last partitions were exported
>>>> to the initiator (client). They were /dev/sdb and /dev/sdc on it
>>>> correspondingly.
>>> Vladislav, Thank you for the benchmarks! I'm very interested in
>>> optimizing your workload and figuring out what happens underneath.
>>>
>>> Are the client and server two standalone boxes connected by GBE?
>>>
>>> When you set readahead sizes in the benchmarks, you are setting them
>>> in the server side? I.e. "linux-4dtq" is the SCST server? What's the
>>> client side readahead size?
>>>
>>> It would help a lot to debug readahead if you can provide the
>>> server side readahead stats and trace log for the worst case.
>>> This will automatically answer the above questions as well as disclose
>>> the micro-behavior of readahead:
>>>
>>> mount -t debugfs none /sys/kernel/debug
>>>
>>> echo > /sys/kernel/debug/readahead/stats # reset counters
>>> # do benchmark
>>> cat /sys/kernel/debug/readahead/stats
>>>
>>> echo 1 > /sys/kernel/debug/readahead/trace_enable
>>> # do micro-benchmark, i.e. run the same benchmark for a short time
>>> echo 0 > /sys/kernel/debug/readahead/trace_enable
>>> dmesg
>>>
>>> The above readahead trace should help find out how the client side
>>> sequential reads convert into server side random reads, and how we can
>>> prevent that.
>> See attached. Could you comment the logs, please, so I will also be able
>> to read them in the future?
>
> Vladislav, thank you for the logs!
>
> The printk format for the following lines is:
>
> + printk(KERN_DEBUG "readahead-%s(pid=%d(%s), dev=%02x:%02x(%s), "
> + "ino=%lu(%s), req=%lu+%lu, ra=%lu+%d-%d, async=%d) = %d\n",
> + ra_pattern_names[pattern],
> + current->pid, current->comm,
> + MAJOR(mapping->host->i_sb->s_dev),
> + MINOR(mapping->host->i_sb->s_dev),
> + mapping->host->i_sb->s_id,
> + mapping->host->i_ino,
> + filp->f_path.dentry->d_name.name,
> + offset, req_size,
> + ra->start, ra->size, ra->async_size,
> + async,
> + actual);
>
> readahead-marker(pid=3838(vdiskd3_3), dev=00:02(bdev), ino=0(raid-1st), req=10596+1, ra=10628+32-32, async=1) = 32
> readahead-marker(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=10628+1, ra=10660+32-32, async=1) = 32
> readahead-marker(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=10660+1, ra=10692+32-32, async=1) = 32
> readahead-marker(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=10692+1, ra=10724+32-32, async=1) = 32
> readahead-marker(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=10724+1, ra=10756+32-32, async=1) = 32
> readahead-marker(pid=3838(vdiskd3_3), dev=00:02(bdev), ino=0(raid-1st), req=10756+1, ra=10788+32-32, async=1) = 32
> readahead-marker(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=10788+1, ra=10820+32-32, async=1) = 32
> readahead-marker(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=10820+1, ra=10852+32-32, async=1) = 32
> readahead-marker(pid=3838(vdiskd3_3), dev=00:02(bdev), ino=0(raid-1st), req=10852+1, ra=10884+32-32, async=1) = 32
> readahead-marker(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=10884+1, ra=10916+32-32, async=1) = 32
> readahead-marker(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=10916+1, ra=10948+32-32, async=1) = 32
> readahead-marker(pid=3836(vdiskd3_1), dev=00:02(bdev), ino=0(raid-1st), req=10948+1, ra=10980+32-32, async=1) = 32
> readahead-marker(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=10980+1, ra=11012+32-32, async=1) = 32
> readahead-marker(pid=3838(vdiskd3_3), dev=00:02(bdev), ino=0(raid-1st), req=11012+1, ra=11044+32-32, async=1) = 32
> readahead-marker(pid=3836(vdiskd3_1), dev=00:02(bdev), ino=0(raid-1st), req=11044+1, ra=11076+32-32, async=1) = 32
> readahead-subsequent(pid=3836(vdiskd3_1), dev=00:02(bdev), ino=0(raid-1st), req=11076+1, ra=11108+32-32, async=1) = 32
> readahead-marker(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=11108+1, ra=11140+32-32, async=1) = 32
> readahead-subsequent(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=11140+1, ra=11172+32-32, async=1) = 32
> readahead-marker(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=11172+1, ra=11204+32-32, async=1) = 32
> readahead-subsequent(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=11204+1, ra=11236+32-32, async=1) = 32
> readahead-marker(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=11236+1, ra=11268+32-32, async=1) = 32
> readahead-subsequent(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=11268+1, ra=11300+32-32, async=1) = 32
> readahead-marker(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=11300+1, ra=11332+32-32, async=1) = 32
> readahead-subsequent(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=11332+1, ra=11364+32-32, async=1) = 32
> readahead-marker(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=11364+1, ra=11396+32-32, async=1) = 32
> readahead-subsequent(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=11396+1, ra=11428+32-32, async=1) = 32
>
> The above trace shows that the readahead logic is working pretty well,
> however the max readahead size(32 pages) is way too small. This can
> also be confirmed in the following table, where the average readahead
> request size/async_size and actual readahead I/O size are all 30.
>
> linux-4dtq:/ # cat /sys/kernel/debug/readahead/stats
> pattern count sync_count eof_count size async_size actual
> none 0 0 0 0 0 0
> initial0 71 71 41 4 3 2
> initial 23 23 0 4 3 4
> subsequent 3845 4 21 31 31 31
> marker 4222 0 1 31 31 31
> trail 0 0 0 0 0 0
> oversize 0 0 0 0 0 0
> reverse 0 0 0 0 0 0
> stride 0 0 0 0 0 0
> thrash 0 0 0 0 0 0
> mmap 135 135 15 32 0 17
> fadvise 180 180 180 0 0 1
> random 23 23 2 1 0 1
> all 8499 436 260 30 30 30
> ^^^^^^^^^^^^^^^^^^^^^^^^
>
> I suspect that your main performance problem comes from the small read/readahead size.
> If larger values are used, even the vanilla 2.6.27 kernel will perform well.

Yes, it was misconfiguration on our side: readahead size was not set
correctly on all devices. In the correct configuration context based RA
shows constant advantage over the current vanilla algorithm, but not as
much as I would expect. It still performs considerably worse, than in
case when all the IO threads work in the same IO context. To remind, our
setup and tests described in http://lkml.org/lkml/2009/2/12/277.

Here are the conclusions from tests:

1. Making all IO threads work in the same IO context with CFQ (vanilla
RA and default RA size) brings near 100% link utilization on single
stream reads (100MB/s) and with deadline about 50% (50MB/s). I.e. there
is 100% improvement of CFQ over deadline. With 2 read streams CFQ has
ever more advantage: >400% (23MB/s vs 5MB/s).

2. All IO threads work in different IO contexts. With vanilla RA and
default RA size CFQ performs 50% worse (48MB/s), even worse than deadline.

3. All IO threads work in different IO contexts. With default RA size
context RA brings on single stream 40% improvement with deadline (71MB/s
vs 51MB/s), no improvement with cfq (48MB/s).

4. All IO threads work in different IO contexts. With higher RA sizes
there is stable 6% improvement with context RA over vanilla RA with CFQ
starting from 20%. Deadline performs similarly. In parallel reads
improvement is bigger: 30% on 4M RA size with deadline (39MB/s vs 27MB/s)

5. All IO threads work in different IO contexts. The best performance
achieved with RA maximum size 4M on both RA algorithms, but starting
from size 1M it starts growing very slowly.

6. Unexpected result. In case, when ll IO threads work in the same IO
context with CFQ increasing RA size *decreases* throughput. I think this
is, because RA requests performed as single big READ requests, while
requests coming from remote clients are much smaller in size (up to
256K), so, when the read by RA data transferred to the remote client on
100MB/s speed, the backstorage media gets rotated a bit, so the next
read request must wait the rotation latency (~0.1ms on 7200RPM). This is
well conforms with (3) above, when context RA has 40% advantage over
vanilla RA with default RA, but much smaller with higher RA.

Bottom line IMHO conclusions:

1. Context RA should be considered after additional examination to
replace current RA algorithm in the kernel

2. It would be better to increase default RA size to 1024K

*AND* one of the following:

3.1. All RA requests should be split in smaller requests with size up to
256K, which should not be merged with any other request

OR

3.2. New RA requests should be sent before the previous one completed to
don't let the storage device rotate too far to need full rotation to
serve the next request.

I like suggestion 3.1 a lot more, since it should be simple to implement
and has the following 2 positive side effects:

1. It would allow to minimize negative effect of higher RA size on the
I/O delay latency by allowing CFQ to switch to too long waiting
requests, when necessary.

2. It would allow better requests pipelining, which is very important to
minimize uplink latency for synchronous requests (i.e. with only one IO
request at time, next request issued, when the previous one completed).
You can see in http://www.3ware.com/kb/article.aspx?id=11050 that 3ware
recommends for maximum performance set max_sectors_kb as low as *64K*
with 16MB RA. It allows to maximize serving commands pipelining. And
this suggestion really works allowing to improve throughput in 50-100%!

Here are the raw numbers. I also attached context RA debug output for
2MB RA size case for your viewing pleasure.

--------------------------------------------------------------------

Performance baseline: all IO threads work in the same IO context,
current vanilla RA, default RA size:

CFQ scheduler:

#dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 102 MB/s
b) 102 MB/s
c) 102 MB/s

Run at the same time:
#while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
#dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 21,6 MB/s
b) 22,8 MB/s
c) 24,1 MB/s
d) 23,1 MB/s

Deadline scheduler:

#dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 51,1 MB/s
b) 51,4 MB/s
c) 51,1 MB/s

Run at the same time:
#while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
#dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 4,7 MB/s
b) 4,6 MB/s
c) 4,8 MB/s

--------------------------------------------------------------------

RA performance baseline: all IO threads work in different IO contexts,
current vanilla RA, default RA size:

CFQ:

#dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 48,6 MB/s
b) 49,2 MB/s
c) 48,9 MB/s

Run at the same time:
#while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
#dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 4,2 MB/s
b) 3,9 MB/s
c) 4,1 MB/s

Deadline:

1) dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 53,2 MB/s
b) 51,8 MB/s
c) 51,6 MB/s

Run at the same time:
#while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
#dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 5,1 MB/s
b) 4,6 MB/s
c) 4,8 MB/s

--------------------------------------------------------------------

Context RA, all IO threads work in different IO contexts, default RA size:

CFQ:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 47,9 MB/s
b) 48,2 MB/s
c) 48,1 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 3,5 MB/s
b) 3,6 MB/s
c) 3,8 MB/s

Deadline:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 72,4 MB/s
b) 68,3 MB/s
c) 71,3 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 4,3 MB/s
b) 5,0 MB/s
c) 4,8 MB/s

--------------------------------------------------------------------

Vanilla RA, all IO threads work in different IO contexts, various RA sizes:

CFQ:

RA 512K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 60,5 MB/s
b) 59,3 MB/s
c) 59,7 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 9,4 MB/s
b) 9,4 MB/s
c) 9,1 MB/s

--- RA 1024K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 74,7 MB/s
b) 73,2 MB/s
c) 74,1 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 13,7 MB/s
b) 13,6 MB/s
c) 13,1 MB/s

--- RA 2048K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 76,7 MB/s
b) 76,8 MB/s
c) 76,6 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 21,8 MB/s
b) 22,1 MB/s
c) 20,3 MB/s

--- RA 4096K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 80,8 MB/s
b) 80.8 MB/s
c) 80,3 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 29,6 MB/s
b) 29,4 MB/s
c) 27,2 MB/s

=== Deadline:

RA 512K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 68,4 MB/s
b) 67,0 MB/s
c) 67,6 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 8,8 MB/s
b) 8,9 MB/s
c) 8,7 MB/s

--- RA 1024K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 81,0 MB/s
b) 82,4 MB/s
c) 81,7 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 13,5 MB/s
b) 13,1 MB/s
c) 12,9 MB/s

--- RA 2048K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 81,1 MB/s
b) 80,1 MB/s
c) 81,8 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 21,9 MB/s
b) 20,7 MB/s
c) 21,3 MB/s

--- RA 4096K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 83,1 MB/s
b) 82,7 MB/s
c) 82,9 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 27,9 MB/s
b) 23,5 MB/s
c) 27,6 MB/s

--------------------------------------------------------------------

Context RA, all IO threads work in different IO contexts, various RA sizes:

CFQ:

RA 512K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 63,7 MB/s
b) 63,5 MB/s
c) 62,8 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 7,1 MB/s
b) 6,7 MB/s
c) 7,0 MB/s
d) 6,9 MB/s

--- RA 1024K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 81,1 MB/s
b) 81,8 MB/s
c) MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 14,1 MB/s
b) 14,0 MB/s
c) 14,1 MB/s

--- RA 2048K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 81,6 MB/s
b) 81,4 MB/s
c) 86,0 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 22,3 MB/s
b) 21,5 MB/s
c) 21,7 MB/s

--- RA 4096K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 83,1 MB/s
b) 83,5 MB/s
c) 82,9 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 32,8 MB/s
b) 32,7 MB/s
c) 30,2 MB/s

=== Deadline:

RA 512K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 68,8 MB/s
b) 68,9 MB/s
c) 69,0 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 8,7 MB/s
b) 9,0 MB/s
c) 8,9 MB/s

--- RA 1024K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 83,5 MB/s
b) 83,1 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 14,0 MB/s
b) 13.9 MB/s
c) 13,8 MB/s

--- RA 2048K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 82,6 MB/s
b) 82,4 MB/s
c) 81,9 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 21,9 MB/s
b) 23,1 MB/s
c) 17,8 MB/s
d) 21,1 MB/s

--- RA 4096K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 84,5 MB/s
b) 83,7 MB/s
c) 83,8 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 39,9 MB/s
b) 39,5 MB/s
c) 38,4 MB/s

--------------------------------------------------------------------

all IO threads work in the same IO context, context RA, various RA sizes:

=== CFQ:

--- RA 512K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 86,4 MB/s
b) 87,9 MB/s
c) 86,7 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 17,8 MB/s
b) 18,3 MB/s
c) 17,7 MB/s

--- RA 1024K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 83,3 MB/s
b) 81,6 MB/s
c) 81,9 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 22,1 MB/s
b) 21,5 MB/s
c) 21,2 MB/s

--- RA 2048K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 81,1 MB/s
b) 81,0 MB/s
c) 81,6 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 22,2 MB/s
b) 20,2 MB/s
c) 20,9 MB/s

--- RA 4096K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 83,4 MB/s
b) 82,8 MB/s
c) 83,3 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 22,6 MB/s
b) 23,4 MB/s
c) 21,8 MB/s

=== Deadline:

--- RA 512K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 70,0 MB/s
b) 70,7 MB/s
c) 69,7 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 9,1 MB/s
b) 8,3 MB/s
c) 8,4 MB/s

--- RA 1024K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 84,3 MB/s
b) 83,2 MB/s
c) 83,3 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 13,9 MB/s
b) 13,1 MB/s
c) 13,4 MB/s

--- RA 2048K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 82,6 MB/s
b) 82,1 MB/s
c) 82,3 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 21,6 MB/s
b) 22,4 MB/s
c) 21,3 MB/s

--- RA 4096K:

dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 83,8 MB/s
b) 83,8 MB/s
c) 83,1 MB/s

Run at the same time:
linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 39,5 MB/s
b) 39,6 MB/s
c) 37,0 MB/s

Thanks,
Vlad

Attachments:

ra.tar.bz2 (25.70 kB)

2009-03-20 08:53:32

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Vladislav Bolkhovitin, on 03/19/2009 08:44 PM wrote:
> 6. Unexpected result. In case, when all IO threads work in the same IO
> context with CFQ increasing RA size *decreases* throughput. I think this
> is, because RA requests performed as single big READ requests, while
> requests coming from remote clients are much smaller in size (up to
> 256K), so, when the read by RA data transferred to the remote client on
> 100MB/s speed, the backstorage media gets rotated a bit, so the next
> read request must wait the rotation latency (~0.1ms on 7200RPM).
^^^^^^^^^^^^^^^^^
Oops, it's RP*M*, hence the full rotation latency is in 60 times more,
i.e. 8.3 ms.

Sorry,
Vlad

2009-03-23 01:43:49

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

On Thu, Mar 19, 2009 at 08:44:13PM +0300, Vladislav Bolkhovitin wrote:
> Wu Fengguang, on 02/19/2009 05:05 AM wrote:
>> On Tue, Feb 17, 2009 at 10:01:40PM +0300, Vladislav Bolkhovitin wrote:
>>> Wu Fengguang, on 02/13/2009 04:57 AM wrote:
>>>> On Thu, Feb 12, 2009 at 09:35:18PM +0300, Vladislav Bolkhovitin wrote:
>>>>> Sorry for such a huge delay. There were many other activities I
>>>>> had to do before + I had to be sure I didn't miss anything.
>>>>>
>>>>> We didn't use NFS, we used SCST (http://scst.sourceforge.net)
>>>>> with iSCSI-SCST target driver. It has similar to NFS
>>>>> architecture, where N threads (N=5 in this case) handle IO from
>>>>> remote initiators (clients) coming from wire using iSCSI
>>>>> protocol. In addition, SCST has patch called
>>>>> export_alloc_io_context (see
>>>>> http://lkml.org/lkml/2008/12/10/282), which allows for the IO
>>>>> threads queue IO using single IO context, so we can see if
>>>>> context RA can replace grouping IO threads in single IO
>>>>> context.
>>>>>
>>>>> Unfortunately, the results are negative. We find neither any
>>>>> advantages of context RA over current RA implementation, nor
>>>>> possibility for context RA to replace grouping IO threads in
>>>>> single IO context.
>>>>>
>>>>> Setup on the target (server) was the following. 2 SATA drives
>>>>> grouped in md RAID-0 with average local read throughput ~120MB/s
>>>>> ("dd if=/dev/zero of=/dev/md0 bs=1M count=20000" outputs
>>>>> "20971520000 bytes (21 GB) copied, 177,742 s, 118 MB/s"). The md
>>>>> device was partitioned on 3 partitions. The first partition was
>>>>> 10% of space in the beginning of the device, the last partition
>>>>> was 10% of space in the end of the device, the middle one was
>>>>> the rest in the middle of the space them. Then the first and the
>>>>> last partitions were exported to the initiator (client). They
>>>>> were /dev/sdb and /dev/sdc on it correspondingly.
>>>> Vladislav, Thank you for the benchmarks! I'm very interested in
>>>> optimizing your workload and figuring out what happens underneath.
>>>>
>>>> Are the client and server two standalone boxes connected by GBE?
>>>>
>>>> When you set readahead sizes in the benchmarks, you are setting them
>>>> in the server side? I.e. "linux-4dtq" is the SCST server? What's the
>>>> client side readahead size?
>>>>
>>>> It would help a lot to debug readahead if you can provide the
>>>> server side readahead stats and trace log for the worst case.
>>>> This will automatically answer the above questions as well as disclose
>>>> the micro-behavior of readahead:
>>>>
>>>> mount -t debugfs none /sys/kernel/debug
>>>>
>>>> echo > /sys/kernel/debug/readahead/stats # reset counters
>>>> # do benchmark
>>>> cat /sys/kernel/debug/readahead/stats
>>>>
>>>> echo 1 > /sys/kernel/debug/readahead/trace_enable
>>>> # do micro-benchmark, i.e. run the same benchmark for a short time
>>>> echo 0 > /sys/kernel/debug/readahead/trace_enable
>>>> dmesg
>>>>
>>>> The above readahead trace should help find out how the client side
>>>> sequential reads convert into server side random reads, and how we can
>>>> prevent that.
>>> See attached. Could you comment the logs, please, so I will also be
>>> able to read them in the future?
>>
>> Vladislav, thank you for the logs!
>>
>> The printk format for the following lines is:
>>
>> + printk(KERN_DEBUG "readahead-%s(pid=%d(%s), dev=%02x:%02x(%s), "
>> + "ino=%lu(%s), req=%lu+%lu, ra=%lu+%d-%d, async=%d) = %d\n",
>> + ra_pattern_names[pattern],
>> + current->pid, current->comm,
>> + MAJOR(mapping->host->i_sb->s_dev),
>> + MINOR(mapping->host->i_sb->s_dev),
>> + mapping->host->i_sb->s_id,
>> + mapping->host->i_ino,
>> + filp->f_path.dentry->d_name.name,
>> + offset, req_size,
>> + ra->start, ra->size, ra->async_size,
>> + async,
>> + actual);
>>
>> readahead-marker(pid=3838(vdiskd3_3), dev=00:02(bdev), ino=0(raid-1st), req=10596+1, ra=10628+32-32, async=1) = 32
>> readahead-marker(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=10628+1, ra=10660+32-32, async=1) = 32
>> readahead-marker(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=10660+1, ra=10692+32-32, async=1) = 32
>> readahead-marker(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=10692+1, ra=10724+32-32, async=1) = 32
>> readahead-marker(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=10724+1, ra=10756+32-32, async=1) = 32
>> readahead-marker(pid=3838(vdiskd3_3), dev=00:02(bdev), ino=0(raid-1st), req=10756+1, ra=10788+32-32, async=1) = 32
>> readahead-marker(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=10788+1, ra=10820+32-32, async=1) = 32
>> readahead-marker(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=10820+1, ra=10852+32-32, async=1) = 32
>> readahead-marker(pid=3838(vdiskd3_3), dev=00:02(bdev), ino=0(raid-1st), req=10852+1, ra=10884+32-32, async=1) = 32
>> readahead-marker(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=10884+1, ra=10916+32-32, async=1) = 32
>> readahead-marker(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=10916+1, ra=10948+32-32, async=1) = 32
>> readahead-marker(pid=3836(vdiskd3_1), dev=00:02(bdev), ino=0(raid-1st), req=10948+1, ra=10980+32-32, async=1) = 32
>> readahead-marker(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=10980+1, ra=11012+32-32, async=1) = 32
>> readahead-marker(pid=3838(vdiskd3_3), dev=00:02(bdev), ino=0(raid-1st), req=11012+1, ra=11044+32-32, async=1) = 32
>> readahead-marker(pid=3836(vdiskd3_1), dev=00:02(bdev), ino=0(raid-1st), req=11044+1, ra=11076+32-32, async=1) = 32
>> readahead-subsequent(pid=3836(vdiskd3_1), dev=00:02(bdev), ino=0(raid-1st), req=11076+1, ra=11108+32-32, async=1) = 32
>> readahead-marker(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=11108+1, ra=11140+32-32, async=1) = 32
>> readahead-subsequent(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=11140+1, ra=11172+32-32, async=1) = 32
>> readahead-marker(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=11172+1, ra=11204+32-32, async=1) = 32
>> readahead-subsequent(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=11204+1, ra=11236+32-32, async=1) = 32
>> readahead-marker(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=11236+1, ra=11268+32-32, async=1) = 32
>> readahead-subsequent(pid=3837(vdiskd3_2), dev=00:02(bdev), ino=0(raid-1st), req=11268+1, ra=11300+32-32, async=1) = 32
>> readahead-marker(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=11300+1, ra=11332+32-32, async=1) = 32
>> readahead-subsequent(pid=3835(vdiskd3_0), dev=00:02(bdev), ino=0(raid-1st), req=11332+1, ra=11364+32-32, async=1) = 32
>> readahead-marker(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=11364+1, ra=11396+32-32, async=1) = 32
>> readahead-subsequent(pid=3839(vdiskd3_4), dev=00:02(bdev), ino=0(raid-1st), req=11396+1, ra=11428+32-32, async=1) = 32
>>
>> The above trace shows that the readahead logic is working pretty well,
>> however the max readahead size(32 pages) is way too small. This can
>> also be confirmed in the following table, where the average readahead
>> request size/async_size and actual readahead I/O size are all 30.
>>
>> linux-4dtq:/ # cat /sys/kernel/debug/readahead/stats
>> pattern count sync_count eof_count size async_size actual
>> none 0 0 0 0 0 0
>> initial0 71 71 41 4 3 2
>> initial 23 23 0 4 3 4
>> subsequent 3845 4 21 31 31 31
>> marker 4222 0 1 31 31 31
>> trail 0 0 0 0 0 0
>> oversize 0 0 0 0 0 0
>> reverse 0 0 0 0 0 0
>> stride 0 0 0 0 0 0
>> thrash 0 0 0 0 0 0
>> mmap 135 135 15 32 0 17
>> fadvise 180 180 180 0 0 1
>> random 23 23 2 1 0 1
>> all 8499 436 260 30 30 30
>> ^^^^^^^^^^^^^^^^^^^^^^^^
>>
>> I suspect that your main performance problem comes from the small read/readahead size.
>> If larger values are used, even the vanilla 2.6.27 kernel will perform well.
>
> Yes, it was misconfiguration on our side: readahead size was not set
> correctly on all devices. In the correct configuration context based RA
> shows constant advantage over the current vanilla algorithm, but not as
> much as I would expect. It still performs considerably worse, than in
> case when all the IO threads work in the same IO context. To remind, our
> setup and tests described in http://lkml.org/lkml/2009/2/12/277.

Vladislav, thank you very much for the great efforts and details!

> Here are the conclusions from tests:
>
> 1. Making all IO threads work in the same IO context with CFQ (vanilla
> RA and default RA size) brings near 100% link utilization on single
> stream reads (100MB/s) and with deadline about 50% (50MB/s). I.e. there
> is 100% improvement of CFQ over deadline. With 2 read streams CFQ has
> ever more advantage: >400% (23MB/s vs 5MB/s).

The ideal 2-stream throughput should be >60MB/s, so I guess there are
still room of improvements for the CFQ's 23MB/s?

> 2. All IO threads work in different IO contexts. With vanilla RA and
> default RA size CFQ performs 50% worse (48MB/s), even worse than
> deadline.
>
> 3. All IO threads work in different IO contexts. With default RA size
> context RA brings on single stream 40% improvement with deadline (71MB/s
> vs 51MB/s), no improvement with cfq (48MB/s).
>
> 4. All IO threads work in different IO contexts. With higher RA sizes
> there is stable 6% improvement with context RA over vanilla RA with CFQ
> starting from 20%. Deadline performs similarly. In parallel reads
> improvement is bigger: 30% on 4M RA size with deadline (39MB/s vs 27MB/s)

Your attached readahead trace shows that context RA is submitting
perfect 256-page async readahead IOs. (However the readahead-subsequent
cache hits are puzzling.)

The vanilla RA detects concurrent streams in a passive/opportunistic way.
The context RA works in an active/guaranteed way. It's also better at
serving the NFS style cooperative io processes. And your SCST workload
looks very like NFS.

The one fact I cannot understand is that SCST seems to breaking up the
client side 64K reads into server side 4K reads(above readahead layer).
But I remember you told me that SCST don't do NFS rsize style split-ups.
Is this a bug? The 4K read size is too small to be CPU/network friendly...
Where are the split-up and re-assemble done? On the client side or
internal to the server?

> 5. All IO threads work in different IO contexts. The best performance
> achieved with RA maximum size 4M on both RA algorithms, but starting
> from size 1M it starts growing very slowly.

True.

> 6. Unexpected result. In case, when ll IO threads work in the same IO
> context with CFQ increasing RA size *decreases* throughput. I think this
> is, because RA requests performed as single big READ requests, while
> requests coming from remote clients are much smaller in size (up to
> 256K), so, when the read by RA data transferred to the remote client on
> 100MB/s speed, the backstorage media gets rotated a bit, so the next
> read request must wait the rotation latency (~0.1ms on 7200RPM). This is
> well conforms with (3) above, when context RA has 40% advantage over
> vanilla RA with default RA, but much smaller with higher RA.

Maybe. But the readahead IOs (as showed by the trace) are _async_ ones...

> Bottom line IMHO conclusions:
>
> 1. Context RA should be considered after additional examination to
> replace current RA algorithm in the kernel

That's my plan to push context RA to mainline. And thank you very much
for providing and testing out a real world application for it!

> 2. It would be better to increase default RA size to 1024K

That's a long wish to increase the default RA size. However I have a
vague feeling that it would be better to first make the lower layers
more smart on max_sectors_kb granularity request splitting and batching.

> *AND* one of the following:
>
> 3.1. All RA requests should be split in smaller requests with size up to
> 256K, which should not be merged with any other request

Are you referring to max_sectors_kb?

What's your max_sectors_kb and nr_requests? Something like

grep -r . /sys/block/sda/queue/

> OR
>
> 3.2. New RA requests should be sent before the previous one completed to
> don't let the storage device rotate too far to need full rotation to
> serve the next request.

Linus has a mmap readahead cleanup patch that can do this. It
basically replaces a {find_lock_page(); readahead();} sequence into
{find_get_page(); readahead(); lock_page();}.

I'll try to push that patch into mainline.

> I like suggestion 3.1 a lot more, since it should be simple to implement
> and has the following 2 positive side effects:
>
> 1. It would allow to minimize negative effect of higher RA size on the
> I/O delay latency by allowing CFQ to switch to too long waiting
> requests, when necessary.
>
> 2. It would allow better requests pipelining, which is very important to
> minimize uplink latency for synchronous requests (i.e. with only one IO
> request at time, next request issued, when the previous one completed).
> You can see in http://www.3ware.com/kb/article.aspx?id=11050 that 3ware
> recommends for maximum performance set max_sectors_kb as low as *64K*
> with 16MB RA. It allows to maximize serving commands pipelining. And
> this suggestion really works allowing to improve throughput in 50-100%!
>
> Here are the raw numbers. I also attached context RA debug output for
> 2MB RA size case for your viewing pleasure.

Thank you, it helped a lot!

(Can I wish a CONFIG_PRINTK_TIME=y next time? :-)

Thanks,
Fengguang

> --------------------------------------------------------------------
>
> Performance baseline: all IO threads work in the same IO context,
> current vanilla RA, default RA size:
>
> CFQ scheduler:
>
> #dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 102 MB/s
> b) 102 MB/s
> c) 102 MB/s
>
> Run at the same time:
> #while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> #dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 21,6 MB/s
> b) 22,8 MB/s
> c) 24,1 MB/s
> d) 23,1 MB/s
>
> Deadline scheduler:
>
> #dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 51,1 MB/s
> b) 51,4 MB/s
> c) 51,1 MB/s
>
> Run at the same time:
> #while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> #dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 4,7 MB/s
> b) 4,6 MB/s
> c) 4,8 MB/s
>
> --------------------------------------------------------------------
>
> RA performance baseline: all IO threads work in different IO contexts,
> current vanilla RA, default RA size:
>
> CFQ:
>
> #dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 48,6 MB/s
> b) 49,2 MB/s
> c) 48,9 MB/s
>
> Run at the same time:
> #while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> #dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 4,2 MB/s
> b) 3,9 MB/s
> c) 4,1 MB/s
>
> Deadline:
>
> 1) dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 53,2 MB/s
> b) 51,8 MB/s
> c) 51,6 MB/s
>
> Run at the same time:
> #while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> #dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 5,1 MB/s
> b) 4,6 MB/s
> c) 4,8 MB/s
>
> --------------------------------------------------------------------
>
> Context RA, all IO threads work in different IO contexts, default RA size:
>
> CFQ:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 47,9 MB/s
> b) 48,2 MB/s
> c) 48,1 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 3,5 MB/s
> b) 3,6 MB/s
> c) 3,8 MB/s
>
> Deadline:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 72,4 MB/s
> b) 68,3 MB/s
> c) 71,3 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 4,3 MB/s
> b) 5,0 MB/s
> c) 4,8 MB/s
>
> --------------------------------------------------------------------
>
> Vanilla RA, all IO threads work in different IO contexts, various RA sizes:
>
> CFQ:
>
> RA 512K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 60,5 MB/s
> b) 59,3 MB/s
> c) 59,7 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 9,4 MB/s
> b) 9,4 MB/s
> c) 9,1 MB/s
>
> --- RA 1024K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 74,7 MB/s
> b) 73,2 MB/s
> c) 74,1 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 13,7 MB/s
> b) 13,6 MB/s
> c) 13,1 MB/s
>
> --- RA 2048K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 76,7 MB/s
> b) 76,8 MB/s
> c) 76,6 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 21,8 MB/s
> b) 22,1 MB/s
> c) 20,3 MB/s
>
> --- RA 4096K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 80,8 MB/s
> b) 80.8 MB/s
> c) 80,3 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 29,6 MB/s
> b) 29,4 MB/s
> c) 27,2 MB/s
>
> === Deadline:
>
> RA 512K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 68,4 MB/s
> b) 67,0 MB/s
> c) 67,6 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 8,8 MB/s
> b) 8,9 MB/s
> c) 8,7 MB/s
>
> --- RA 1024K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 81,0 MB/s
> b) 82,4 MB/s
> c) 81,7 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 13,5 MB/s
> b) 13,1 MB/s
> c) 12,9 MB/s
>
> --- RA 2048K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 81,1 MB/s
> b) 80,1 MB/s
> c) 81,8 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 21,9 MB/s
> b) 20,7 MB/s
> c) 21,3 MB/s
>
> --- RA 4096K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 83,1 MB/s
> b) 82,7 MB/s
> c) 82,9 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 27,9 MB/s
> b) 23,5 MB/s
> c) 27,6 MB/s
>
> --------------------------------------------------------------------
>
> Context RA, all IO threads work in different IO contexts, various RA sizes:
>
> CFQ:
>
> RA 512K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 63,7 MB/s
> b) 63,5 MB/s
> c) 62,8 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 7,1 MB/s
> b) 6,7 MB/s
> c) 7,0 MB/s
> d) 6,9 MB/s
>
> --- RA 1024K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 81,1 MB/s
> b) 81,8 MB/s
> c) MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 14,1 MB/s
> b) 14,0 MB/s
> c) 14,1 MB/s
>
> --- RA 2048K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 81,6 MB/s
> b) 81,4 MB/s
> c) 86,0 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 22,3 MB/s
> b) 21,5 MB/s
> c) 21,7 MB/s
>
> --- RA 4096K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 83,1 MB/s
> b) 83,5 MB/s
> c) 82,9 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 32,8 MB/s
> b) 32,7 MB/s
> c) 30,2 MB/s
>
> === Deadline:
>
> RA 512K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 68,8 MB/s
> b) 68,9 MB/s
> c) 69,0 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 8,7 MB/s
> b) 9,0 MB/s
> c) 8,9 MB/s
>
> --- RA 1024K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 83,5 MB/s
> b) 83,1 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 14,0 MB/s
> b) 13.9 MB/s
> c) 13,8 MB/s
>
> --- RA 2048K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 82,6 MB/s
> b) 82,4 MB/s
> c) 81,9 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 21,9 MB/s
> b) 23,1 MB/s
> c) 17,8 MB/s
> d) 21,1 MB/s
>
> --- RA 4096K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 84,5 MB/s
> b) 83,7 MB/s
> c) 83,8 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 39,9 MB/s
> b) 39,5 MB/s
> c) 38,4 MB/s
>
> --------------------------------------------------------------------
>
> all IO threads work in the same IO context, context RA, various RA sizes:
>
> === CFQ:
>
> --- RA 512K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 86,4 MB/s
> b) 87,9 MB/s
> c) 86,7 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 17,8 MB/s
> b) 18,3 MB/s
> c) 17,7 MB/s
>
> --- RA 1024K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 83,3 MB/s
> b) 81,6 MB/s
> c) 81,9 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 22,1 MB/s
> b) 21,5 MB/s
> c) 21,2 MB/s
>
> --- RA 2048K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 81,1 MB/s
> b) 81,0 MB/s
> c) 81,6 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 22,2 MB/s
> b) 20,2 MB/s
> c) 20,9 MB/s
>
> --- RA 4096K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 83,4 MB/s
> b) 82,8 MB/s
> c) 83,3 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 22,6 MB/s
> b) 23,4 MB/s
> c) 21,8 MB/s
>
> === Deadline:
>
> --- RA 512K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 70,0 MB/s
> b) 70,7 MB/s
> c) 69,7 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 9,1 MB/s
> b) 8,3 MB/s
> c) 8,4 MB/s
>
> --- RA 1024K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 84,3 MB/s
> b) 83,2 MB/s
> c) 83,3 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 13,9 MB/s
> b) 13,1 MB/s
> c) 13,4 MB/s
>
> --- RA 2048K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 82,6 MB/s
> b) 82,1 MB/s
> c) 82,3 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 21,6 MB/s
> b) 22,4 MB/s
> c) 21,3 MB/s
>
> --- RA 4096K:
>
> dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 83,8 MB/s
> b) 83,8 MB/s
> c) 83,1 MB/s
>
> Run at the same time:
> linux-4dtq:~ # while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
> linux-4dtq:~ # dd if=/dev/sdb of=/dev/null bs=64K count=80000
> a) 39,5 MB/s
> b) 39,6 MB/s
> c) 37,0 MB/s
>
> Thanks,
> Vlad

2009-04-21 18:19:55

[permalink] [raw]

Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases

Wu Fengguang, on 03/23/2009 04:42 AM wrote:
>> Here are the conclusions from tests:
>>
>> 1. Making all IO threads work in the same IO context with CFQ (vanilla
>> RA and default RA size) brings near 100% link utilization on single
>> stream reads (100MB/s) and with deadline about 50% (50MB/s). I.e. there
>> is 100% improvement of CFQ over deadline. With 2 read streams CFQ has
>> ever more advantage: >400% (23MB/s vs 5MB/s).
>
> The ideal 2-stream throughput should be >60MB/s, so I guess there are
> still room of improvements for the CFQ's 23MB/s?

Yes, plenty. But, I think, not in CFQ, but in readahead. With RA 4096K
we were able to get ~40MB/s, see the previous e-mail and below.

> The one fact I cannot understand is that SCST seems to breaking up the
> client side 64K reads into server side 4K reads(above readahead layer).
> But I remember you told me that SCST don't do NFS rsize style split-ups.
> Is this a bug? The 4K read size is too small to be CPU/network friendly...
> Where are the split-up and re-assemble done? On the client side or
> internal to the server?

This is on the client's side. See the target's log in the attachment.
Here is the summary of commands data sizes, came to the server, for "dd
if=/dev/sdb of=/dev/null bs=64K count=200" ran on the client:

4K 11
8K 0
16K 0
32K 0
64K 0
128K 81
256K 8
512K 0
1024K 0
2048K 0
4096K 0

There's a way too many 4K requests. Apparently, the requests submission
path isn't optimal.

Actually, this is another question I wanted to rise from the very
beginning.

>> 6. Unexpected result. In case, when ll IO threads work in the same IO
>> context with CFQ increasing RA size *decreases* throughput. I think this
>> is, because RA requests performed as single big READ requests, while
>> requests coming from remote clients are much smaller in size (up to
>> 256K), so, when the read by RA data transferred to the remote client on
>> 100MB/s speed, the backstorage media gets rotated a bit, so the next
>> read request must wait the rotation latency (~0.1ms on 7200RPM). This is
>> well conforms with (3) above, when context RA has 40% advantage over
>> vanilla RA with default RA, but much smaller with higher RA.
>
> Maybe. But the readahead IOs (as showed by the trace) are _async_ ones...

That doesn't matter, because new request from the client won't come
until all data for the previous one transferred to it. And that transfer
is done on a very *finite* speed.

>> Bottom line IMHO conclusions:
>>
>> 1. Context RA should be considered after additional examination to
>> replace current RA algorithm in the kernel
>
> That's my plan to push context RA to mainline. And thank you very much
> for providing and testing out a real world application for it!

You're welcome!

>> 2. It would be better to increase default RA size to 1024K
>
> That's a long wish to increase the default RA size. However I have a
> vague feeling that it would be better to first make the lower layers
> more smart on max_sectors_kb granularity request splitting and batching.

Can you elaborate more on that, please?

>> *AND* one of the following:
>>
>> 3.1. All RA requests should be split in smaller requests with size up to
>> 256K, which should not be merged with any other request
>
> Are you referring to max_sectors_kb?

Yes

> What's your max_sectors_kb and nr_requests? Something like
>
> grep -r . /sys/block/sda/queue/

Default: 512 and 128 correspondingly.

>> OR
>>
>> 3.2. New RA requests should be sent before the previous one completed to
>> don't let the storage device rotate too far to need full rotation to
>> serve the next request.
>
> Linus has a mmap readahead cleanup patch that can do this. It
> basically replaces a {find_lock_page(); readahead();} sequence into
> {find_get_page(); readahead(); lock_page();}.
>
> I'll try to push that patch into mainline.

Good!

>> I like suggestion 3.1 a lot more, since it should be simple to implement
>> and has the following 2 positive side effects:
>>
>> 1. It would allow to minimize negative effect of higher RA size on the
>> I/O delay latency by allowing CFQ to switch to too long waiting
>> requests, when necessary.
>>
>> 2. It would allow better requests pipelining, which is very important to
>> minimize uplink latency for synchronous requests (i.e. with only one IO
>> request at time, next request issued, when the previous one completed).
>> You can see in http://www.3ware.com/kb/article.aspx?id=11050 that 3ware
>> recommends for maximum performance set max_sectors_kb as low as *64K*
>> with 16MB RA. It allows to maximize serving commands pipelining. And
>> this suggestion really works allowing to improve throughput in 50-100%!

Seems I should elaborate more on this. Case, when client is remote has a
fundamental difference from the case, when client is local, for which
Linux currently optimized. When client is local data delivered to it
from the page cache with a virtually infinite speed. But when client is
remote data delivered to it from the server's cache on a *finite* speed.
In our case this speed is about the same as speed of reading data to the
cache from the storage. It has the following consequences:

1. Data for any READ request at first transferred from the storage to
the cache, then from the cache to the client. If those transfers are
done purely sequentially without overlapping, i.e. without any
readahead, resulting throughput T can be found from equation: 1/T =
1/Tlocal + 1/Tremote, where Tlocal and Tremote are throughputs of the
local (i.e. from the storage) and remote links. In case, when Tlocal ~=
Tremote, T ~= Tremote/2. Quite unexpected result, right? ;)

2. If data transfers on the local and remote links aren't coordinated,
it is possible that only one link transfers data at some time. From the
(1) above you can calculate that % of this "idle" time is % of the lost
throughput. I.e. to get the maximum throughput both links should
transfer data as simultaneous as possible. For our case, when Tlocal ~=
Tremote, both links should be all the time busy. Moreover, it is
possible that the local transfer finished, but during the remote
transfer the storage media rotated too far, so for the next request it
will be needed to wait the full rotation to finish (i.e. several ms of
lost bandwidth).

Thus, to get the maximum possible throughput, we need to maximize
simultaneous load of both local and remote links. It can be done by
using well known pipelining technique. For that client should read the
same amount of data at once, but those read should be split on smaller
chunks, like 64K at time. This approach looks being against the
"conventional wisdom", saying that bigger request means bigger
throughput, but, in fact, it doesn't, because the same (big) amount of
data are read at time. Bigger count of smaller requests will make more
simultaneous load on both participating in the data transfers links. In
fact, even if client is local, in most cases there is a second data
transfer link. It's in the storage. This is especially true for RAID
controllers. Guess, why 3ware recommends to set max_sectors_kb to 64K
and increase RA in the above link? ;)

Of course, max_sectors_kb should be decreased only for smart devices,
which allow >1 outstanding requests at time, i.e. for all modern
SCSI/SAS/SATA/iSCSI/FC/etc. drives.

There is an objection against having too many outstanding requests at
time. This is latency. But, since overall size of all requests remains
unchanged, this objection isn't relevant in this proposal. There is the
same latency-related objection against increasing RA. But many small
individual RA requests it isn't relevant as well.

We did some measurements to support the this proposal. They were done
only with deadline scheduler to make the picture clearer. They were done
with context RA. The tests were the same as before.

--- Baseline, all default:

# dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 51,1 MB/s
b) 51,4 MB/s
c) 51,1 MB/s

Run at the same time:
# while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
# dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 4,7 MB/s
b) 4,6 MB/s
c) 4,8 MB/s

--- Client - all default, on the server max_sectors_kb set to 64K:

# dd if=/dev/sdb of=/dev/null bs=64K count=80000
- 100 MB/s
- 100 MB/s
- 102 MB/s

# while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
# dd if=/dev/sdb of=/dev/null bs=64K count=80000
- 5,2 MB/s
- 5,3 MB/s
- 4,2 MB/s

100% and 8% improvement comparing to the baseline.

From the previous e-mail you can see that with 4096K RA

# while true; do dd if=/dev/sdc of=/dev/null bs=64K; done
# dd if=/dev/sdb of=/dev/null bs=64K count=80000
a) 39,9 MB/s
b) 39,5 MB/s
c) 38,4 MB/s

I.e. there is 760% improvement over the baseline.

Thus, I believe, that for all devices, supporting queue depths >1,
max_sectors_kb should be set by default to 64K (or to 128K, maybe, but
not more), and default RA increased to at least 1M, better 2-4M.

> (Can I wish a CONFIG_PRINTK_TIME=y next time? :-)

Sure

Thanks,
Vlad

Attachments:

req_split.log.bz2 (5.55 kB)

2009-04-24 08:44:23