2016-03-07 15:36:26

by Jesper Dangaard Brouer

[permalink] [raw]
Subject: Tool for sampling /proc/net/softnet_stat statistics

Hi Google,

While playing with RPS, I needed to read stats from
/proc/net/softnet_stat and the tools I could find [1] and [2] was not
very good.

I lack of better, I coded up my own tool softnet_stat.pl here:
https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl

The output format/columns in /proc/net/softnet_stat is undocumented,
plus values are printed in hex. E.g. to decode the columns you need to
read kernel function kernel softnet_seq_show() in
kernel/net/core/net-procfs.c.

To make things easier I wrote this small perl script for get
so human readable statistics from /proc/net/softnet_stat.

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer


[1] https://www.redhat.com/archives/rhl-list/2007-September/msg03735.html

[2] https://gist.github.com/SaveTheRbtz/172b2e2eb3cbd96b598d


2016-03-07 17:55:02

by Willem de Bruijn

[permalink] [raw]
Subject: Re: Tool for sampling /proc/net/softnet_stat statistics

On Mon, Mar 7, 2016 at 10:36 AM, Jesper Dangaard Brouer
<[email protected]> wrote:
> Hi Google,
>
> While playing with RPS, I needed to read stats from
> /proc/net/softnet_stat and the tools I could find [1] and [2] was not
> very good.
>
> I lack of better, I coded up my own tool softnet_stat.pl here:
> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
>
> The output format/columns in /proc/net/softnet_stat is undocumented,
> plus values are printed in hex. E.g. to decode the columns you need to
> read kernel function kernel softnet_seq_show() in
> kernel/net/core/net-procfs.c.
>
> To make things easier I wrote this small perl script for get
> so human readable statistics from /proc/net/softnet_stat.

Very nice. Thanks for sharing, Jesper. I maintained something similar,
but never got around to clean up and upstream it. Will start using
yours, instead.

A few points, from using my earlier tool:

A minimum cut-off value is helpful, especially on beefy servers, to
suppress the many 0 rows. Preferably configurable, to also be able to
suppress low-rate background traffic when analyzing a few large
streams. My default was 500.

The number of columns has grown with kernel versions. The latest
column is flow_limit, added in 3.11 at99bbc7074190. It is helpful for
the script to be robust against both older and future kernels. On
which note, to be able to support these kinds of tools, any new
columns to such procfs files should be appended, not inserted as for
instance in https://patchwork.ozlabs.org/patch/574171/

Time squeeze is an exception, in that it is number of squeeze events
per second, not number of packets squeezed. This is often
misunderstood if not explained clearly.

2016-03-08 09:51:11

by Jesper Dangaard Brouer

[permalink] [raw]
Subject: Re: Tool for sampling /proc/net/softnet_stat statistics

On Mon, 7 Mar 2016 12:54:13 -0500
Willem de Bruijn <[email protected]> wrote:

> On Mon, Mar 7, 2016 at 10:36 AM, Jesper Dangaard Brouer
> <[email protected]> wrote:
> > Hi Google,
> >
> > While playing with RPS, I needed to read stats from
> > /proc/net/softnet_stat and the tools I could find [1] and [2] was not
> > very good.
> >
> > I lack of better, I coded up my own tool softnet_stat.pl here:
> > https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
> >
> > The output format/columns in /proc/net/softnet_stat is undocumented,
> > plus values are printed in hex. E.g. to decode the columns you need to
> > read kernel function kernel softnet_seq_show() in
> > kernel/net/core/net-procfs.c.
> >
> > To make things easier I wrote this small perl script for get
> > so human readable statistics from /proc/net/softnet_stat.
>
> Very nice. Thanks for sharing, Jesper. I maintained something similar,
> but never got around to clean up and upstream it. Will start using
> yours, instead.
>
> A few points, from using my earlier tool:
>
> A minimum cut-off value is helpful, especially on beefy servers, to
> suppress the many 0 rows. Preferably configurable, to also be able to
> suppress low-rate background traffic when analyzing a few large
> streams. My default was 500.
>
> The number of columns has grown with kernel versions. The latest
> column is flow_limit, added in 3.11 at99bbc7074190. It is helpful for
> the script to be robust against both older and future kernels. On
> which note, to be able to support these kinds of tools, any new
> columns to such procfs files should be appended, not inserted as for
> instance in https://patchwork.ozlabs.org/patch/574171/
>
> Time squeeze is an exception, in that it is number of squeeze events
> per second, not number of packets squeezed. This is often
> misunderstood if not explained clearly.

Thanks a lot for your feedback. I don't have time to address it right
away, so I've instead added a section with future development todo's.
So, I don't forget this valuable feedback :-)

https://github.com/netoptimizer/network-testing/commit/c464676e456aab

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer

2016-03-08 19:50:34

by David Miller

[permalink] [raw]
Subject: Re: Tool for sampling /proc/net/softnet_stat statistics

From: Jesper Dangaard Brouer <[email protected]>
Date: Mon, 7 Mar 2016 16:36:07 +0100

> I lack of better, I coded up my own tool softnet_stat.pl here:
> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
>
> The output format/columns in /proc/net/softnet_stat is undocumented,
> plus values are printed in hex. E.g. to decode the columns you need to
> read kernel function kernel softnet_seq_show() in
> kernel/net/core/net-procfs.c.
>
> To make things easier I wrote this small perl script for get
> so human readable statistics from /proc/net/softnet_stat.

Good stuff!