Hi,
we are facing below 2 performance issue on NFS:
1. Read speed is low for small files
==========================
[ Log on NFS Client]
$ echo 3 > /proc/sys/vm/drop_caches
$ dd if=200KBfile.txt of=/dev/null
400+0 records in
400+0 records out
204800 bytes (200.0KB) copied, 0.027074 seconds, 7.2MB/s
Read speed for 200KB file is 7.2 MB
[ Log on NFS Client]
$ echo 3 > /proc/sys/vm/drop_caches
$ dd if=100MBfile.txt of=/dev/null
204800+0 records in
204800+0 records out
104857600 bytes (100.0MB) copied, 9.351221 seconds, 10.7MB/s
Read speed for 100MB file is 10.7 MB
As you see read speed for 200KB file is only 7.2MB/sec while it is
10.7 MB/sec when we read 100MB file.
Why there is so much difference in read performance ?
Is there any way to achieve high read speed for small files ?
2. Read/stat for a directory tree is slow on NFS than local
==========================================
we have lot of *.jpg files in a directory. If we try to "stat" and
"read" from this directory,
performannce is very slow on NFS Client compared to Local(NFS server)
"stat" and "read"
[ Log on Local (NFS Server) ]
$ echo 3 > /proc/sys/vm/drop_caches
$ ./stat_read_files_test ./lot_of_jpg_files/
Time Taken : 9288 msec
[ Log on NFS Client]
$ echo 3 > /proc/sys/vm/drop_caches
$ ./stat_read_files_test ./lot_of_jpg_files/
Time Taken : 19966 msec
As you see, on NFS client time taken is almost *double* than that of
local(NFS server)
We are using UDP with rsize,wsize=32k on 100MB ethernet link.
I am attaching read/stat testcase.
Is there any way to improve this performance ?
Thanks,
Vivek
T24gRnJpLCAyMDEyLTAzLTIzIGF0IDA3OjQ5IC0wNDAwLCBKaW0gUmVlcyB3cm90ZToNCj4gVml2
ZWsgVHJpdmVkaSB3cm90ZToNCj4gDQo+ICAgMjA0ODAwIGJ5dGVzICgyMDAuMEtCKSBjb3BpZWQs
IDAuMDI3MDc0IHNlY29uZHMsIDcuMk1CL3MNCj4gICBSZWFkIHNwZWVkIGZvciAyMDBLQiBmaWxl
IGlzIDcuMiBNQg0KPiANCj4gICAxMDQ4NTc2MDAgYnl0ZXMgKDEwMC4wTUIpIGNvcGllZCwgOS4z
NTEyMjEgc2Vjb25kcywgMTAuN01CL3MNCj4gICBSZWFkIHNwZWVkIGZvciAxMDBNQiBmaWxlIGlz
IDEwLjcgTUINCj4gICANCj4gICBBcyB5b3Ugc2VlIHJlYWQgc3BlZWQgZm9yIDIwMEtCIGZpbGUg
aXMgb25seSA3LjJNQi9zZWMgd2hpbGUgaXQgaXMNCj4gICAxMC43IE1CL3NlYyB3aGVuIHdlIHJl
YWQgMTAwTUIgZmlsZS4NCj4gICBXaHkgdGhlcmUgaXMgc28gbXVjaCBkaWZmZXJlbmNlIGluIHJl
YWQgcGVyZm9ybWFuY2UgPw0KPiAgIElzIHRoZXJlIGFueSB3YXkgdG8gYWNoaWV2ZSBoaWdoIHJl
YWQgc3BlZWQgZm9yIHNtYWxsIGZpbGVzID8NCj4gDQo+IFRoYXQgc2VlbXMgZXhjZWxsZW50IHRv
IG1lLiAgMjA0ODAwIGJ5dGVzIGF0IDExMjEzMjUyIHBlciBzZWMgd291bGQgYmUgMTguMg0KPiBt
c2VjLCBzbyB5b3VyIHBlci1maWxlIG92ZXJoZWFkIGlzIGFyb3VuZCA5IG1zZWMuICBUaGUgZGlz
ayBsYXRlbmN5IGFsb25lDQo+IHdvdWxkIG5vcm1hbGx5IGJlIG1vcmUgdGhhbiB0aGF0Lg0KDQou
Li5hbmQgdGhlIHJlYXNvbiB3aHkgdGhlIHBlcmZvcm1hbmNlIGlzIHdvcnNlIGZvciB0aGUgMjAw
SyBmaWxlDQpjb21wYXJlZCB0byB0aGUgMTAwTSBvbmUgaXMgZWFzaWx5IGV4cGxhaW5lZC4NCg0K
V2hlbiBvcGVuaW5nIHRoZSBmaWxlIGZvciByZWFkaW5nLCB0aGUgY2xpZW50IGhhcyBhIG51bWJl
ciBvZg0Kc3luY2hyb25vdXMgUlBDIGNhbGxzIHRvIG1ha2U6IGl0IG5lZWRzIHRvIGxvb2sgdXAg
dGhlIGZpbGUsIGNoZWNrDQphY2Nlc3MgcGVybWlzc2lvbnMgYW5kIHBvc3NpYmx5IHJldmFsaWRh
dGUgaXRzIGNhY2hlLiBBbGwgdGhlc2UgdGFza3MNCmhhdmUgdG8gYmUgZG9uZSBpbiBzZXJpZXMg
KHlvdSBjYW5ub3QgZG8gdGhlbSBpbiBwYXJhbGxlbCksIGFuZCBzbyB0aGUNCmxhdGVuY3kgb2Yg
ZWFjaCB0YXNrIGlzIGxpbWl0ZWQgYnkgdGhlIHJvdW5kLXRyaXAgdGltZSB0byB0aGUgc2VydmVy
Lg0KDQpPbiB0aGUgb3RoZXIgaGFuZCwgb25jZSB5b3UgZ2V0IHRvIGRvaW5nIFJFQURzLCB0aGUg
Y2xpZW50IGNhbiBzZW5kIGENCmJ1bmNoIG9mIHJlYWRhaGVhZCByZXF1ZXN0cyBpbiBwYXJhbGxl
bCwgdGh1cyBlbnN1cmluZyB0aGF0IHRoZSBzZXJ2ZXINCmNhbiB1c2UgYWxsIHRoZSBiYW5kd2lk
dGggYXZhaWxhYmxlIHRvIHRoZSBUQ1AgY29ubmVjdGlvbi4NCg0KU28geW91ciByZXN1bHQgaXMg
YmFzaWNhbGx5IHNob3dpbmcgdGhhdCBmb3Igc21hbGwgZmlsZXMsIHRoZSBwcm9wb3J0aW9uDQpv
ZiAocmVhZGFoZWFkKSB0YXNrcyB0aGF0IGNhbiBiZSBkb25lIGluIHBhcmFsbGVsIGlzIHNtYWxs
ZXIuIFRoaXMgaXMgYXMNCmV4cGVjdGVkLg0KDQotLSANClRyb25kIE15a2xlYnVzdA0KTGludXgg
TkZTIGNsaWVudCBtYWludGFpbmVyDQoNCk5ldEFwcA0KVHJvbmQuTXlrbGVidXN0QG5ldGFwcC5j
b20NCnd3dy5uZXRhcHAuY29tDQoNCg==
Vivek Trivedi wrote:
204800 bytes (200.0KB) copied, 0.027074 seconds, 7.2MB/s
Read speed for 200KB file is 7.2 MB
104857600 bytes (100.0MB) copied, 9.351221 seconds, 10.7MB/s
Read speed for 100MB file is 10.7 MB
As you see read speed for 200KB file is only 7.2MB/sec while it is
10.7 MB/sec when we read 100MB file.
Why there is so much difference in read performance ?
Is there any way to achieve high read speed for small files ?
That seems excellent to me. 204800 bytes at 11213252 per sec would be 18.2
msec, so your per-file overhead is around 9 msec. The disk latency alone
would normally be more than that.
On Mar 23, 2012, at 5:16 AM, Myklebust, Trond wrote:
> All these tasks have to be done in series (you cannot do them in parallel), and so the
> latency of each task is limited by the round-trip time to the server.
Will the v4 client at least collapse them into a single compound op?
T24gVGh1LCAyMDEyLTAzLTI5IGF0IDA4OjUyIC0wNzAwLCBLZXZpbiBHcmFoYW0gd3JvdGU6DQo+
IE9uIE1hciAyMywgMjAxMiwgYXQgNToxNiBBTSwgTXlrbGVidXN0LCBUcm9uZCB3cm90ZToNCj4g
DQo+ID4gQWxsIHRoZXNlIHRhc2tzIGhhdmUgdG8gYmUgZG9uZSBpbiBzZXJpZXMgKHlvdSBjYW5u
b3QgZG8gdGhlbSBpbiBwYXJhbGxlbCksIGFuZCBzbyB0aGUNCj4gPiBsYXRlbmN5IG9mIGVhY2gg
dGFzayBpcyBsaW1pdGVkIGJ5IHRoZSByb3VuZC10cmlwIHRpbWUgdG8gdGhlIHNlcnZlci4NCj4g
DQo+IA0KPiBXaWxsIHRoZSB2NCBjbGllbnQgYXQgbGVhc3QgY29sbGFwc2UgdGhlbSBpbnRvIGEg
c2luZ2xlIGNvbXBvdW5kIG9wPw0KPiANCg0KV2hlcmUgcG9zc2libGUsIHllcywgYnV0IHRoZXJl
IGlzIHBsZW50eSBvZiBzdHVmZiB3aGVyZSB0aGUgTkZTIGNsaWVudA0Kc2ltcGx5IGRvZXNuJ3Qg
aGF2ZSBlbm91Z2ggaW5mb3JtYXRpb24gdG8gZG8gdGhhdC4gRm9yIGluc3RhbmNlLCB3ZQ0KZG9u
J3QgeWV0IGhhdmUgdGhlIGNhcGFiaWxpdHkgaW4gdGhlIFZGUyB0byBhc2sgTkZTIHRvIGxvb2sg
dXAgc2V2ZXJhbA0KcGF0aCBjb21wb25lbnRzIGF0IHRoZSBzYW1lIHRpbWUuDQoNCkZ1cnRoZXJt
b3JlLCBzb21lIG9mIHRoZXNlIHRoaW5ncyBhcmUgc2VwYXJhdGUgc3lzdGVtIGNhbGxzLiBUaGUN
CmZpbGVzeXN0ZW0gY2FuJ3QganVzdCBhc3N1bWUgdGhhdCBvcGVuKE9fUkRPTkxZKSBtZWFucyB0
aGF0IHdlIHNob3VsZA0Kc3RhcnQgcmVhZCBhaGVhZCwgYW5kIHRoZW4gY2xvc2UgdGhlIGZpbGUs
IHNvIEkgZG9uJ3QgZXZlciBzZWUgdXMgYmVpbmcNCmFibGUgdG8gY29sbGFwc2UgT1BFTitSRUFE
K0NMT1NFIGludG8gYSBzaW5nbGUgY29tcG91bmQuDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpM
aW51eCBORlMgY2xpZW50IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWtsZWJ1c3RAbmV0
YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5jb20NCg0K
2012/3/23 Myklebust, Trond <[email protected]>:
> On Fri, 2012-03-23 at 07:49 -0400, Jim Rees wrote:
>> Vivek Trivedi wrote:
>>
>> 204800 bytes (200.0KB) copied, 0.027074 seconds, 7.2MB/s
>> Read speed for 200KB file is 7.2 MB
>>
>> 104857600 bytes (100.0MB) copied, 9.351221 seconds, 10.7MB/s
>> Read speed for 100MB file is 10.7 MB
>>
>> As you see read speed for 200KB file is only 7.2MB/sec while it is
>> 10.7 MB/sec when we read 100MB file.
>> Why there is so much difference in read performance ?
>> Is there any way to achieve high read speed for small files ?
>>
>> That seems excellent to me. 204800 bytes at 11213252 per sec would be 18.2
>> msec, so your per-file overhead is around 9 msec. The disk latency alone
>> would normally be more than that.
>
> ...and the reason why the performance is worse for the 200K file
> compared to the 100M one is easily explained.
>
> When opening the file for reading, the client has a number of
> synchronous RPC calls to make: it needs to look up the file, check
> access permissions and possibly revalidate its cache. All these tasks
> have to be done in series (you cannot do them in parallel), and so the
> latency of each task is limited by the round-trip time to the server.
>
> On the other hand, once you get to doing READs, the client can send a
> bunch of readahead requests in parallel, thus ensuring that the server
> can use all the bandwidth available to the TCP connection.
>
> So your result is basically showing that for small files, the proportion
> of (readahead) tasks that can be done in parallel is smaller. This is as
> expected.
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> [email protected]
> http://www.netapp.com
>
Dear Trond.
I agree your answer.
Thanks a lot for your specific explaination.