I am using tiobench to test performance of an NFS mounted volume, and
notice that Sequential Reads are much slower than Random Reads. This
isn't the behavior when I run the same test on the disk mounted
locally.
For random reads I'm getting:
50 MB/s over NFS
v.s
384 MB/s when mounted locally
This is in comparison to the benchmark for _Random Reads_, in which I get:
288 MB/s both over NFS _and_ when directly mounted
The other benchmarks seem to be in line with what I would expect, but
I'm fairly new to NFS. Why would sequential reads over NFS be sooo
much slower than random reads over NFS?
I am exporting the volume on the server like this
/export *.internal(no_subtree_check,rw,no_root_squash)
and mounting with this:
mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/export /nfs
Additionally I am doing all this in amazon EC2, exporting an EBS
volume with the XFS file system (redundant, I know).
I have tried using jumbo frames and various other mount options, but
none seem to have much effect.
Thanks for any clues.
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
I am using tiobench to test performance of an NFS mounted volume, and
notice that Sequential Reads are much slower than Random Reads. This
isn't the behavior when I run the same test on the disk mounted
locally.
For random reads I'm getting:
? ?50 MB/s ?over NFS
v.s
? ?384 MB/s ?when mounted locally
This is in comparison to the benchmark for _Random Reads_, in which I get:
? ?288 MB/s both over NFS _and_ when directly mounted
The other benchmarks seem to be in line with what I would expect, but
I'm fairly new to NFS. Why would sequential reads over NFS be sooo
much slower than random reads over NFS?
I am exporting the volume on the server like this
/export *.internal(no_subtree_check,rw,no_root_squash)
and mounting with this:
mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/export /nfs
Additionally I am doing all this in amazon EC2, exporting an EBS
volume with the XFS file system (redundant, I know).
I have tried using jumbo frames and various other mount options, but
none seem to have much effect.
Thanks for any clues.
On 03/12/2010 02:09 PM, Brandon Simmons wrote:
> On Fri, Mar 12, 2010 at 1:30 PM, Ric Wheeler<[email protected]> wrote:
>
>> On 03/12/2010 01:22 PM, Brandon Simmons wrote:
>>
>>> I am using tiobench to test performance of an NFS mounted volume, and
>>> notice that Sequential Reads are much slower than Random Reads. This
>>> isn't the behavior when I run the same test on the disk mounted
>>> locally.
>>>
>>> For random reads I'm getting:
>>>
>>> 50 MB/s over NFS
>>>
>>> v.s
>>>
>>> 384 MB/s when mounted locally
>>>
>>> This is in comparison to the benchmark for _Random Reads_, in which I get:
>>>
>>> 288 MB/s both over NFS _and_ when directly mounted
>>>
>>> The other benchmarks seem to be in line with what I would expect, but
>>> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
>>> much slower than random reads over NFS?
>>>
>>> I am exporting the volume on the server like this
>>>
>>> /export *.internal(no_subtree_check,rw,no_root_squash)
>>>
>>> and mounting with this:
>>>
>>> mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/export /nfs
>>>
>>> Additionally I am doing all this in amazon EC2, exporting an EBS
>>> volume with the XFS file system (redundant, I know).
>>>
>>> I have tried using jumbo frames and various other mount options, but
>>> none seem to have much effect.
>>>
>>> Thanks for any clues.
>>>
>>>
>> Not sure what kind of network you are running the NFS test over so it is
>> quite hard to figure out why your performance varies so wildly.
>>
>> Normal NFS testing with a gigabit network between the client and server
>> would be much closer to 50MB/sec than your 288MB/sec.
>>
>> Can you try to reproduce this locally with known client and server hardware?
>>
>> ric
>>
>>
>>
> I'm not sure. My servers are EC2 instances in Amazon's cloud computing
> service.I am doing the test from an EBS which is a virtual disk
> mounted locally on an instance and exported via NFS.
>
> So I don't think I can do any relevant tests locally.
>
> Thanks,
> Brandon
>
The joys of working in the cloud :-)
One possible reason could be that one test is actually going to an NFS
server that is remote, one might be going to one locally (not leaving
the box). I think that you will have to escalate with the Amazon
support/technical people to try and peek under the covers a bit.
ric
On Fri, Mar 12, 2010 at 1:33 PM, Trond Myklebust
<[email protected]> wrote:
> On Fri, 2010-03-12 at 13:22 -0500, Brandon Simmons wrote:
>> I am using tiobench to test performance of an NFS mounted volume, an=
d
>> notice that Sequential Reads are much slower than Random Reads. This
>> isn't the behavior when I run the same test on the disk mounted
>> locally.
>>
>> For random reads I'm getting:
>>
>> =A0 =A050 MB/s =A0over NFS
>>
>> v.s
>>
>> =A0 =A0384 MB/s =A0when mounted locally
>>
>> This is in comparison to the benchmark for _Random Reads_, in which =
I get:
>>
>> =A0 =A0288 MB/s both over NFS _and_ when directly mounted
>>
>> The other benchmarks seem to be in line with what I would expect, bu=
t
>> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
>> much slower than random reads over NFS?
>
> They're not usually. My guess is that this is an artifact of your tes=
t.
> What is tiobench doing prior to the sequential read?
>
> Trond
>
>
I'm wondering if this is caused by caching. my benchmark does
Sequential Reads first, then it does Random Reads. Is it possible that
the random reads could be working on cached data?
That would mean my sequential reads aren't too slow, rather the random
reads are extraordinarily fast because of caching. Looking at running
some different benchmarks.
Brandon Simmons wrote:
> On Fri, Mar 12, 2010 at 1:33 PM, Trond Myklebust
> <[email protected]> wrote:
>
>> On Fri, 2010-03-12 at 13:22 -0500, Brandon Simmons wrote:
>>
>>> I am using tiobench to test performance of an NFS mounted volume, and
>>> notice that Sequential Reads are much slower than Random Reads. This
>>> isn't the behavior when I run the same test on the disk mounted
>>> locally.
>>>
>>> For random reads I'm getting:
>>>
>>> 50 MB/s over NFS
>>>
>>> v.s
>>>
>>> 384 MB/s when mounted locally
>>>
>>> This is in comparison to the benchmark for _Random Reads_, in which I get:
>>>
>>> 288 MB/s both over NFS _and_ when directly mounted
>>>
>>> The other benchmarks seem to be in line with what I would expect, but
>>> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
>>> much slower than random reads over NFS?
>>>
>> They're not usually. My guess is that this is an artifact of your test.
>> What is tiobench doing prior to the sequential read?
>>
>> Trond
>>
>
> I'm wondering if this is caused by caching. my benchmark does
> Sequential Reads first, then it does Random Reads. Is it possible that
> the random reads could be working on cached data?
>
> That would mean my sequential reads aren't too slow, rather the random
> reads are extraordinarily fast because of caching. Looking at running
> some different benchmarks.
>
>
I think so too - since your throughput are way too good (when compared
with other I/O-bound numbers). Using simple benchmark(s) such as
tiobench can generate very misleading results if you don't know how to
steer away from cache effects. For NFS workload, SPECsfs is a much
better benchmark but it is cumbersome to run (and you have to pay for
it, I think).
In general, however, I find sequential read performance hard to predict
in a virtual environment. This is because the data can be (physically)
scattered around. It takes non-trivial efforts for system software to
get its pre-fetch logic right.
-- Wendy
On 03/12/2010 01:22 PM, Brandon Simmons wrote:
> I am using tiobench to test performance of an NFS mounted volume, and
> notice that Sequential Reads are much slower than Random Reads. This
> isn't the behavior when I run the same test on the disk mounted
> locally.
>
> For random reads I'm getting:
>
> 50 MB/s over NFS
>
> v.s
>
> 384 MB/s when mounted locally
>
> This is in comparison to the benchmark for _Random Reads_, in which I get:
>
> 288 MB/s both over NFS _and_ when directly mounted
>
> The other benchmarks seem to be in line with what I would expect, but
> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
> much slower than random reads over NFS?
>
> I am exporting the volume on the server like this
>
> /export *.internal(no_subtree_check,rw,no_root_squash)
>
> and mounting with this:
>
> mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/export /nfs
>
> Additionally I am doing all this in amazon EC2, exporting an EBS
> volume with the XFS file system (redundant, I know).
>
> I have tried using jumbo frames and various other mount options, but
> none seem to have much effect.
>
> Thanks for any clues.
>
Not sure what kind of network you are running the NFS test over so it is
quite hard to figure out why your performance varies so wildly.
Normal NFS testing with a gigabit network between the client and server
would be much closer to 50MB/sec than your 288MB/sec.
Can you try to reproduce this locally with known client and server hardware?
ric
On Fri, 2010-03-12 at 13:22 -0500, Brandon Simmons wrote:
> I am using tiobench to test performance of an NFS mounted volume, and
> notice that Sequential Reads are much slower than Random Reads. This
> isn't the behavior when I run the same test on the disk mounted
> locally.
>
> For random reads I'm getting:
>
> 50 MB/s over NFS
>
> v.s
>
> 384 MB/s when mounted locally
>
> This is in comparison to the benchmark for _Random Reads_, in which I get:
>
> 288 MB/s both over NFS _and_ when directly mounted
>
> The other benchmarks seem to be in line with what I would expect, but
> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
> much slower than random reads over NFS?
They're not usually. My guess is that this is an artifact of your test.
What is tiobench doing prior to the sequential read?
Trond
On Fri, Mar 12, 2010 at 1:30 PM, Ric Wheeler <[email protected]> wro=
te:
> On 03/12/2010 01:22 PM, Brandon Simmons wrote:
>>
>> I am using tiobench to test performance of an NFS mounted volume, an=
d
>> notice that Sequential Reads are much slower than Random Reads. This
>> isn't the behavior when I run the same test on the disk mounted
>> locally.
>>
>> For random reads I'm getting:
>>
>> =A0 =A050 MB/s =A0over NFS
>>
>> v.s
>>
>> =A0 =A0384 MB/s =A0when mounted locally
>>
>> This is in comparison to the benchmark for _Random Reads_, in which =
I get:
>>
>> =A0 =A0288 MB/s both over NFS _and_ when directly mounted
>>
>> The other benchmarks seem to be in line with what I would expect, bu=
t
>> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
>> much slower than random reads over NFS?
>>
>> I am exporting the volume on the server like this
>>
>> /export *.internal(no_subtree_check,rw,no_root_squash)
>>
>> and mounting with this:
>>
>> mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/expor=
t /nfs
>>
>> Additionally I am doing all this in amazon EC2, exporting an EBS
>> volume with the XFS file system (redundant, I know).
>>
>> I have tried using jumbo frames and various other mount options, but
>> none seem to have much effect.
>>
>> Thanks for any clues.
>>
>
> Not sure what kind of network you are running the NFS test over so it=
is
> quite hard to figure out why your performance varies so wildly.
>
> Normal NFS testing with a gigabit network between the client and serv=
er
> would be much closer to 50MB/sec than your 288MB/sec.
>
> Can you try to reproduce this locally with known client and server ha=
rdware?
>
> ric
>
>
I'm not sure. My servers are EC2 instances in Amazon's cloud computing
service.I am doing the test from an EBS which is a virtual disk
mounted locally on an instance and exported via NFS.
So I don't think I can do any relevant tests locally.
Thanks,
Brandon