I just wanted to let people know that as a result of a discussion on
linux-mm I've added support for the new slab allocator to my collectl
utility, now making it real easy to dynamically monitor allocations
along with all the other types of monitoring collectl does. I've also
put together a webpage at http://collectl.sourceforge.net/SlabInfo.html
to give a taste of how this all works as well as to show a few different
types of output.
-mark
In keeping with the spirit of adding new useful features when they help
solve real world problems, I've added this ability to collectl as a
direct result of a problem we were recently having when doing some
network performance testing. It turned out all the interrupts were
being processed by cpu0 (this was on an 8-core system) but all collectl
told us was the aggregate number! Once we moved to a NIC/driver that
supported multiple queues that could distribute interrupts to multiple
CPUs it only made sense to enhance collectl to let us verify that this
was indeed happening - I personally find examining /proc/interrupts for
changes more trouble than it's worth.
In any event, the following is an example of how collectl can present
this data, first summarized by CPU:
#Time Cpu0 Cpu1 Cpu2 Cpu3 Cpu4 Cpu5 Cpu6 Cpu7
12:49:55 4828 16K 1000 16K 18 16K 16K 0
12:49:56 4804 16K 1000 16K 0 16K 16K 0
12:49:57 4811 16K 1000 16K 18 16K 16K 0
12:49:58 4789 16K 1000 16K 0 17K 16K 44
and here by the type of the interrupt itself:
# INTERRUPT DETAILS
# Int Cpu0 Cpu1 Cpu2 Cpu3 Cpu4 Cpu5 Cpu6 Cpu7
Type Device(s)
12:48:50 082 0 0 0 7731 0 0 0
0 PCI-MSI-X eth2 (queue 0)
12:48:50 098 0 0 0 0 2037 0 0
0 PCI-MSI-X eth2 (queue 2)
12:48:50 122 0 0 2240 0 0 0 0
0 PCI-MSI-X eth2 (queue 5)
12:48:50 138 0 7084 0 0 0 0 0
0 PCI-MSI-X eth2 (queue 7)
12:48:50 154 0 0 0 0 0 7723 0
0 PCI-MSI-X eth3 (queue 0)
12:48:50 162 9082 0 0 0 0 0 0
0 PCI-MSI-X eth3 (queue 1)
12:48:50 178 0 0 0 0 0 0 8253
0 PCI-MSI-X eth3 (queue 3)
12:48:50 210 0 0 0 0 0 0 0
6417 PCI-MSI-X eth3 (queue 7)
12:48:50 218 1 0 0 0 0 0 0
0 PCI-MSI eth0
You can also look at all CPU loads and interrupts together like this:
# SINGLE CPU STATISTICS
# CPU USER NICE SYS WAIT IRQ SOFT STEAL IDLE INTRPT
07:09:28 0 0 0 0 0 0 0 0 100 16
07:09:28 1 0 0 0 0 0 0 0 100 0
07:09:28 2 0 0 0 0 0 0 0 100 999
07:09:28 3 0 0 0 0 0 0 0 100 16
For those not familiar with collectl, you can collect virtually
everything all the existing linux 'stat' utilitie provide plus a lot
more such as processes (including I/O stats), Infininband, Quadrics,
Slab, Lustre and more. Plus a lot more feature too numerous to list but
there's a pretty good summary here -
http://collectl.sourceforge.net/Features.html
But don't take my word for it, check out the website at
http://collectl.sourceforge.net/ where you can see a pretty good set of
examples in the documentation and even follow the tutorial.
-mark
Just a quick note on this. If you've been staying away from collectl because it didn't support V4 of nfs, I finally got around to adding it. One thing that's kind of cool is if you run the command "collectl -sF --home", you'll see a top-like command that displays most of the nfs stats on all types of the systems' nfs client/servers for versions 2,3 and 4. You can read a little more about it at: http://collectl.sourceforge.net/NfsInfo.html
While I was at it I also added buddyinfo monitoring, so you can literally watch your fragment distribution change in real-time using the command "collectl -sB --home" as described here - http://collectl.sourceforge.net/BuddyInfo.html. I'm still not sure how one can make the most out of this information or if I'm displaying it in the most useful manner, but as always all feedback is welcome, preferably on the collectl mailing list [email protected].
-mark