Hi,
We are deploying a number of Linux desktop clients running NFS against
our Linux NFS server. We are seeing occasional odd spikes in traffic
that are causing performance troubles for all users. We are trying to
isolate the cause of these spikes, but to date haven't been able to.
The best we can do is use iostat and verify that yes, the disk we
expected is seeing a lot of traffic.
nfsd threads seem to be impervious to lsof. top also doesn't show much
about them, and of course you can't strace them.
Is there any tool out there that could give us any of this sort of info:
* What IP addresses are generating high volumes of read or write
traffic
* What files on disk are being accessed frequently via NFS
* Anything else that could help us pinpoint the trouble
nfsstat does not seem to provide fine enough detail for this.
Thanks,
-- John
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Try lighting up an instance of ntop on the network you've got your NFS
server(s) on. Either span/mirror the switch ports so it sees all the
traffic in promisc mode, or turn on sflow forwarding to the box. As for
determining what files are being accessed, you can use ethereal to
capture traffic on the wire and analyze file ops occuring, but it's not
fun to try and analyze #ops/file... you'd have to filter quite well,
dump as XML/text and do some post-processing to generate your stats.
You could also start up inotify on the linux NFS server, watching the
entire directory tree you're exporting via NFS, log the output, and
later analyze the file ops as reported. Maybe you should turn on some
trending/monitoring of the clients themselves, I'd suggest ganglia for
ease-of-deployment, but you could also use net-snmp and cacti, or
monami... there are a number of ways to get data on traffic occuring at
both ends of the transaction.
There's also an ncurses-based traffic tool that's sometimes handy for
looking at NFS, but I don't recall its name offhand.
/eli
John Goerzen wrote:
> Hi,
>
> We are deploying a number of Linux desktop clients running NFS against
> our Linux NFS server. We are seeing occasional odd spikes in traffic
> that are causing performance troubles for all users. We are trying to
> isolate the cause of these spikes, but to date haven't been able to.
> The best we can do is use iostat and verify that yes, the disk we
> expected is seeing a lot of traffic.
>
> nfsd threads seem to be impervious to lsof. top also doesn't show much
> about them, and of course you can't strace them.
>
> Is there any tool out there that could give us any of this sort of info:
>
> * What IP addresses are generating high volumes of read or write
> traffic
>
> * What files on disk are being accessed frequently via NFS
>
> * Anything else that could help us pinpoint the trouble
>
> nfsstat does not seem to provide fine enough detail for this.
>
> Thanks,
>
> -- John
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tue August 14 2007 12:39:52 pm Eli Stair wrote:
> Try lighting up an instance of ntop on the network you've got your NFS
> server(s) on. Either span/mirror the switch ports so it sees all the
> traffic in promisc mode, or turn on sflow forwarding to the box. As for
I was using iftop, but the traffic was too "spikey" to be able to identify
long-term trends with it. I'll take a look at ntop.
> determining what files are being accessed, you can use ethereal to
> capture traffic on the wire and analyze file ops occuring, but it's not
> fun to try and analyze #ops/file... you'd have to filter quite well,
Yes, that's exactly the problem I was fearing.
> You could also start up inotify on the linux NFS server, watching the
> entire directory tree you're exporting via NFS, log the output, and
Oh, EXCELLENT idea. I don't know why I didn't think of that. That could be
very useful indeed.
I'm not familiar with ganglia, cacti, or monami, but will investigate.
Thanks very much for the suggestions. This will be helpful.
-- John
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tue, Aug 14, 2007 at 10:39:52AM -0700, Eli Stair wrote:
> You could also start up inotify on the linux NFS server, watching the
> entire directory tree you're exporting via NFS, log the output, and
> later analyze the file ops as reported.
blktrace might also be useful here.
/* Steinar */
--
Homepage: http://www.sesse.net/
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs