Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: rsize,wsize=1M causes severe lags in 10/100 Mbps
From:   Alkis Georgopoulos <alkisg@gmail.com>
To:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
References: <ee758eaf-c02d-f669-bc31-f30e6b17d92a@gmail.com>
 <3d00928cd3244697442a75b36b75cf47ef872657.camel@hammerspace.com>
 <7afc5770-abfa-99bb-dae9-7d11680875fd@gmail.com>
 <e51876b8c2540521c8141ba11b11556d22bde20b.camel@hammerspace.com>
 <915fa536-c992-3b77-505e-829c4d049b02@gmail.com>
 <1d5f6643330afd2c04350006ad2a60e83aebb59d.camel@hammerspace.com>
 <5601db40-ee2f-262d-7d01-5c589c9a07eb@gmail.com>
 <d7ea48b4cd665eced45783bf94d6b1ff1f211960.camel@hammerspace.com>
 <20190919211912.GA21865@cosmos.ssec.wisc.edu>
 <503c22ad34b3f3a15015b7384bcad469b2899cb4.camel@hammerspace.com>
 <20190919221601.GA30751@cosmos.ssec.wisc.edu>
 <0213704b-3930-5be6-bd3d-dbaabc24a270@gmail.com>
 <1e7c9896-eb1b-9d7c-fff0-6df2b3d96392@gmail.com>
 <6fa596e9-b154-310e-9685-7663731618ba@gmail.com>
Message-ID: <ddf47f3c-d27e-972b-fd38-1fdfeaa105a3@gmail.com>
Date:   Sat, 21 Sep 2019 10:52:05 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.8.0
MIME-Version: 1.0
In-Reply-To: <6fa596e9-b154-310e-9685-7663731618ba@gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

I think it's caused by the kernel readahead, not glibc readahead.
TL;DR: This solves the problem:
echo 4 > /sys/devices/virtual/bdi/0:58/read_ahead_kb

Question: how to configure NFS/kernel to automatically set that?

Long version:
Doing step (4) below results in tremendous speedup:

1) mount -t nfs -o tcp,timeo=600,rsize=1048576,wsize=1048576 
10.161.254.11:/srv/ltsp /mnt

2) cat /proc/fs/nfsfs/volumes
We see the DEV number from there, e.g. 0:58

3) cat /sys/devices/virtual/bdi/0:58/read_ahead_kb
15360
I assume that this means the kernel will try to read ahead up to 15 MB 
for each accessed file. *THIS IS THE PROBLEM*. For non-NFS devices, this 
value is 128 (KB).

4) echo 4 > /sys/devices/virtual/bdi/0:58/read_ahead_kb

5) Test. Traffic now should be a *lot* less, and speed a *lot* more.
E.g. my NFS booting tests:
  - read_ahead_kb=15360 (the default) => 1160 MB traffic to boot
  - read_ahead_kb=128 => 324MB traffic
  - read_ahead_kb=4 => 223MB traffic

So the question that remains, is how to properly configure either NFS or 
the kernel, to use small readahead values for NFS.

I'm currently doing it with this workaround:
for f in $(awk '/^v[0-9]/ { print $4 }' < /proc/fs/nfsfs/volumes); do 
echo 4 > /sys/devices/virtual/bdi/$f/read_ahead_kb; done

Thanks,
Alkis