I am using the kernel IP Accounting in Linux to record the amount of data
transfered via my Linux internet gateway from individual IP addresses. This
currently requires me to set up an accounting rule for each IP address that I
want to record accounting info for. If I had 200 machines to individually log
this would require me to set 200 rules.
In the 2.5 series of kernels, working towards 2.6, could you please make the
IP Accounting so that I can set a single rule that will make it watch all IP
traffic going from the local network, through the masquerading service to the
internet, and log local IP Addresses using it? This would allow me to set 1
rule, but have the information I want on a per IP address system.
One other person I have talked to would like to see this too, and he
basically says we need a software version of the Cisco IP Accounting
server/router.
Could you please add this to the next kernel? Please CC me your responses as
I am not a member of the kernel mailing list. Thanks,
David
> In the 2.5 series of kernels, working towards 2.6, could you please make the
> IP Accounting so that I can set a single rule that will make it watch all IP
> traffic going from the local network, through the masquerading service to the
> internet, and log local IP Addresses using it? This would allow me to set 1
> rule, but have the information I want on a per IP address system.
You could try using a mature userspace traffic meter like 'NeTraMet' (uses
libpcap).
ftp://ftp.auckland.ac.nz/pub/iawg/NeTraMet/
> One other person I have talked to would like to see this too, and he
> basically says we need a software version of the Cisco IP Accounting
> server/router.
NeTraMet can also account using Cisco Netflow accounting records.
> Could you please add this to the next kernel? Please CC me your responses as
> I am not a member of the kernel mailing list. Thanks,
>
> David
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
>
On Tue, 17 Apr 2001, David Findlay wrote:
>I am using the kernel IP Accounting in Linux to record the amount of data
>transfered via my Linux internet gateway from individual IP addresses. This
>currently requires me to set up an accounting rule for each IP address that I
>want to record accounting info for. If I had 200 machines to individually log
>this would require me to set 200 rules.
>
>In the 2.5 series of kernels, working towards 2.6, could you please make the
>IP Accounting so that I can set a single rule that will make it watch all IP
>traffic going from the local network, through the masquerading service to the
>internet, and log local IP Addresses using it? This would allow me to set 1
>rule, but have the information I want on a per IP address system.
>
>One other person I have talked to would like to see this too, and he
>basically says we need a software version of the Cisco IP Accounting
>server/router.
>
>Could you please add this to the next kernel? Please CC me your responses as
>I am not a member of the kernel mailing list. Thanks,
Perhaps I misunderstand what it is exactly you are trying to do,
but I would think that this could be done entirely in userland by
software that just adds rules for you instead of you having to do
it manually.
Just a thought.
----------------------------------------------------------------------
Mike A. Harris - Linux advocate - Free Software advocate
This message is copyright 2001, all rights reserved.
Views expressed are my own, not necessarily shared by my employer.
----------------------------------------------------------------------
There are two major products that come out of Berkeley: LSD and BSD.
We don't believe this to be a coincidence. -- Jeremy S. Anderson
On Monday 16 April 2001 10:40, you wrote:
> Perhaps I misunderstand what it is exactly you are trying to do,
> but I would think that this could be done entirely in userland by
> software that just adds rules for you instead of you having to do
> it manually.
I suppose, but it would be so much easier if the kernel did it automatically.
Having a rule to go through for each IP address to be logged would be slower
than implementing one rule that would log all of them. Doing this in the
kernel would improve preformance.
David
On Tue, 17 Apr 2001, David Findlay wrote:
>> Perhaps I misunderstand what it is exactly you are trying to do,
>> but I would think that this could be done entirely in userland by
>> software that just adds rules for you instead of you having to do
>> it manually.
>
>I suppose, but it would be so much easier if the kernel did it automatically.
>Having a rule to go through for each IP address to be logged would be slower
>than implementing one rule that would log all of them. Doing this in the
>kernel would improve preformance.
I don't think it would, but then only benchmarking it both ways
would know for sure. Even with incredibly large rulesets,
ipchains &&/|| netfilter works admirably well. Rusty?
----------------------------------------------------------------------
Mike A. Harris - Linux advocate - Free Software advocate
This message is copyright 2001, all rights reserved.
Views expressed are my own, not necessarily shared by my employer.
----------------------------------------------------------------------
On Tue, Apr 17, 2001 at 08:46:12AM +1000, David Findlay wrote:
> On Monday 16 April 2001 10:40, you wrote:
> > Perhaps I misunderstand what it is exactly you are trying to do,
> > but I would think that this could be done entirely in userland by
> > software that just adds rules for you instead of you having to do
> > it manually.
>
> I suppose, but it would be so much easier if the kernel did it automatically.
> Having a rule to go through for each IP address to be logged would be slower
> than implementing one rule that would log all of them. Doing this in the
> kernel would improve preformance.
Perhaps "netflow" would be an answer ?
It is cisco idea for collecting accounting data for
network flows. System has a cache of flows for
which it collects data, and once the cache overflows
(or flow times out), data is sent to designated flow
collection server(s).
http://www.cisco.com/warp/public/cc/pd/iosw/ioft/neflct/tech/napps_wp.htm
> David
/Matti Aarnio
No, one rule would be MUCH faster. What's do you think would be faster of the two:
if ((ipaddr>=3232235521)&&(ipaddr<=3232235774))
return 1;
else
return 0;
or
for (a=0;a<(3232235774-3232235521);a++)
if (ipaddr==a)
return 1;
return 0;
Obviously it compares the 192.168.0.1 - 192.168.0.255 range, but I think we both
know which one will be faster. Not to mention countless other redundant checks
will be added in on both, but on the second one the time of the checks is
multiplied by the number of times you have too loop.
But, I'm just a newbie so I don't really know :P Just taking up bandwith :(
L8ers,
Matt
"Mike A. Harris" wrote:
> On Tue, 17 Apr 2001, David Findlay wrote:
>
> >> Perhaps I misunderstand what it is exactly you are trying to do,
> >> but I would think that this could be done entirely in userland by
> >> software that just adds rules for you instead of you having to do
> >> it manually.
> >
> >I suppose, but it would be so much easier if the kernel did it automatically.
> >Having a rule to go through for each IP address to be logged would be slower
> >than implementing one rule that would log all of them. Doing this in the
> >kernel would improve preformance.
>
> I don't think it would, but then only benchmarking it both ways
> would know for sure. Even with incredibly large rulesets,
> ipchains &&/|| netfilter works admirably well. Rusty?
>
> ----------------------------------------------------------------------
> Mike A. Harris - Linux advocate - Free Software advocate
> This message is copyright 2001, all rights reserved.
> Views expressed are my own, not necessarily shared by my employer.
> ----------------------------------------------------------------------
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
David Findlay <[email protected]> writes:
> On Monday 16 April 2001 10:40, you wrote:
> > Perhaps I misunderstand what it is exactly you are trying to do,
> > but I would think that this could be done entirely in userland by
> > software that just adds rules for you instead of you having to do
> > it manually.
>
> I suppose, but it would be so much easier if the kernel did it
> automatically.
Yes, but by what criteria? You want to log on a per host basis.
Someone else wants to log by service (www, ftp, mail, news), or
any number of other criteria.
> Having a rule to go through for each IP address to be logged would
> be slower than implementing one rule that would log all of
> them. Doing this in the kernel would improve preformance.
If there really is a performance issue with a few hundred rules, then
it can be overcome by grouping rules in separate custom chains. F.e.
if you have 1024 rules create 32 custom chains with 32 rules each.
Then have 32 rules in your main table which jump to the appropriate
custom chain --> maximum rules traversed by each packet = 64.
There is another issue with logging in general:
*COUNTERS MUST NOT BE RESETABLE!!!*
Resetable counters guarantee that no two programs can co-exists if
they happen to reset the same counters.
All logging counters should be implemented with 32bit or 64bit
unsigned integers. Any software using correct unsigned integer
arithmetic can then simply subtract a previous value from the current
value to get the difference. This works reliably across counter
wrap-arounds. There is absolutely *no need for reset* !
--
Manfred Bartz
---------------------------------------------------------------
ipchainsLogAnalyzer, NetCalc, whois at: <http://logi.cc/linux/>
NEW: <http://logi.cc/linux/NetfilterLogAnalyzer.php3>
David Findlay said once upon a time (Tue, 17 Apr 2001):
> I am using the kernel IP Accounting in Linux to record the amount of data
> transfered via my Linux internet gateway from individual IP addresses. This
> currently requires me to set up an accounting rule for each IP address that I
> want to record accounting info for. If I had 200 machines to individually log
> this would require me to set 200 rules.
In 1999 I setup a Linux kernel 2.2 box to monitor traffic for about 2500
hosts. I created two rules for each host (to monitor inbound and
outbound). I had over 5000 total rules on a Pentium II 300 Mhz box. I
wrote some perl glue so I could accounts on a per customer basis (each
customer had one or more IPs, not necessarily contiguous) and graph the
results in MRTG.
There was ZERO performance problems on the box. The box is still
runningly happily today. I don't there is a "problem" to fix.
Dax
On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
> There is another issue with logging in general:
>
> *COUNTERS MUST NOT BE RESETABLE!!!*
Umm, no. Counters can be resetable - you just specify that accounting
programs should not reset them, ever.
The ability to reset counters is extremely useful if you're a human
looking at the output of iptables -L -v. (I thus far know of no one
who can memorise the counter values for around 40 rules).
--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html
Hi,
On Tue, Apr 17, 2001 at 08:46:12AM +1000, David Findlay wrote:
>
> I suppose, but it would be so much easier if the kernel did it automatically.
> Having a rule to go through for each IP address to be logged would be slower
> than implementing one rule that would log all of them. Doing this in the
> kernel would improve preformance.
You can use the iptables ULOG facility (see netfilter CVS for the
patches). With this you can copy packets to userspace programs which
can do any further processing, like the accounting stuff. You can even
choose to only copy part of a packet (say only the IP header) to
userspace, to reduce the amount of data your application has to
handle.
Simply copy the headers of all IP packets going through your router to
userspace, and write a small application (possibly using libipulog
which comes with the netfilter userspace code) which does the actual
accounting. This has the additional benefit that further processing of
the packet in kernel (i.e. outputting it on the destination interface)
can continue while your application is processing the accounting data.
Andreas
--
You are in a maze of little twisting passages, all alike.
Russell King <[email protected]> writes:
> On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
> > There is another issue with logging in general:
> >
> > *COUNTERS MUST NOT BE RESETABLE!!!*
>
> Umm, no. Counters can be resetable - you just specify that accounting
> programs should not reset them, ever.
>
> The ability to reset counters is extremely useful if you're a human
> looking at the output of iptables -L -v. (I thus far know of no one
> who can memorise the counter values for around 40 rules).
You just illustrated my point. While there is a reset capability
people will use it and accounting/logging programs will get wrong
data. Resetable counters might be a minor convenience when debugging
but the price is unreliable programs and the loss of the ability of
several programs to use the same counters.
Also, do you always reset the date and time everytime you make time
measurements?
--
Manfred Bartz
Manfred Bartz responded to
> Russell King <[email protected]> who writes:
>
> > On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
> > > There is another issue with logging in general:
> > >
> > > *COUNTERS MUST NOT BE RESETABLE!!!*
> >
> > Umm, no. Counters can be resetable - you just specify that
> > accounting programs should not reset them, ever.
> >
> > The ability to reset counters is extremely useful if you're a human
> > looking at the output of iptables -L -v. (I thus far know of no one
> > who can memorise the counter values for around 40 rules).
>
> You just illustrated my point. While there is a reset capability
> people will use it and accounting/logging programs will get wrong
> data. Resetable counters might be a minor convenience when debugging
> but the price is unreliable programs and the loss of the ability of
> several programs to use the same counters.
You of course, are commenting from the fact that your applications are
stupid, written poorly, and cannot handle 'wrapped' data. Take MRTG
as one example of a good application, for it can handle counters that
have been cleared between reads.
> Also, do you always reset the date and time everytime you make time
> measurements?
No, but i realize that if my time has incremented, and my date has not
changed, then it's later. If my time has _decremented_ and my date has not,
then I've time-warped. If my time has decremented and my date has
incremented,
then it's simply the next-date-increment.
Similarly, if my InPackets are at 102345 at one read, and 2345 the next
read,
and I know that my counter is 32 bits, then I know i've wrapped and can do
my
own math.
Applications need to get smarter. Don't place arbitrary rules where they
don't
belong.
>
> Manfred Bartz responded to
> > Russell King <[email protected]> who writes:
<snip>
> > You just illustrated my point. While there is a reset capability
> > people will use it and accounting/logging programs will get wrong
> > data. Resetable counters might be a minor convenience when debugging
> > but the price is unreliable programs and the loss of the ability of
> > several programs to use the same counters.
>
> You of course, are commenting from the fact that your applications are
> stupid, written poorly, and cannot handle 'wrapped' data. Take MRTG
<snip>
> Similarly, if my InPackets are at 102345 at one read, and 2345 the next
> read,
> and I know that my counter is 32 bits, then I know i've wrapped and can do
I think the point being made is that if InPackets are at 102345 at one read,
and 2345 the next, and you know it's a 32 bit counter, it's completely
unreliable to assume that you have in fact recieved 4294867295
packets, if the counter can be zeroed.
You can say nothing other than at least 2345 packets, at most
2345+n*2^32 have been got since you last checked.
> From: Ian Stirling [mailto:[email protected]]
> > Manfred Bartz responded to
> > > Russell King <[email protected]> who writes:
> <snip>
> > > You just illustrated my point. While there is a reset capability
> > > people will use it and accounting/logging programs will get wrong
> > > data. Resetable counters might be a minor convenience
> when debugging
> > > but the price is unreliable programs and the loss of the
> ability of
> > > several programs to use the same counters.
> >
> > You of course, are commenting from the fact that your
> applications are
> > stupid, written poorly, and cannot handle 'wrapped' data. Take MRTG
> <snip>
> > Similarly, if my InPackets are at 102345 at one read, and
> 2345 the next
> > read,
> > and I know that my counter is 32 bits, then I know i've
> wrapped and can do
>
> I think the point being made is that if InPackets are at
> 102345 at one read,
> and 2345 the next, and you know it's a 32 bit counter, it's completely
> unreliable to assume that you have in fact recieved 4294867295
> packets, if the counter can be zeroed.
> You can say nothing other than at least 2345 packets, at most
> 2345+n*2^32 have been got since you last checked.
Ah, yes.. I seem to have misplaced a bit of text in my reply.
The continuation of thought:
How the application derives the status of a wrapped counter or
a zero'ed counter is dependant on the device being monitored.
Yes, you have to know what your interface is capable of (maxbytes/sec)
so that you can do a simple calculation where:
maximum_throughput = maxbytes_sec * (time_now - time_last_read)
and if your previous good counter + the maximum throughput wraps the
counter, you have a good chance that you've simply wrapped.
If not, then you can assume that your counters were cleared at some point,
log the data you've got, and keep moving forward.
Leif Sawyer <[email protected]> writes:
> Manfred Bartz responded to
> > Russell King <[email protected]> who writes:
> >
> > > On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
> > > > There is another issue with logging in general:
> > > >
> > > > *COUNTERS MUST NOT BE RESETABLE!!!*
> > >
> > > Umm, no. Counters can be resetable - you just specify that
> > > accounting programs should not reset them, ever.
> > >
> > > The ability to reset counters is extremely useful if you're a human
> > > looking at the output of iptables -L -v. (I thus far know of no one
> > > who can memorise the counter values for around 40 rules).
> >
> > You just illustrated my point. While there is a reset capability
> > people will use it and accounting/logging programs will get wrong
> > data. Resetable counters might be a minor convenience when debugging
> > but the price is unreliable programs and the loss of the ability of
> > several programs to use the same counters.
>
> You of course, are commenting from the fact that your applications are
> stupid, written poorly, and cannot handle 'wrapped' data.
Well, apps written by me *can handle* wrapped data; it is actually far
easier that resetting at each read.
> Take MRTG as one example of a good application, for it can handle
> counters that have been cleared between reads.
Regardless of how well a logging application is written, there is no
way that it does not loose data when someone or something resets its
counter. F.e. if an app reads a counter every 10 minutes and a
counter reset occurs from elsewhere, then anywhere between 0 and 10
minutes worth of data is lost.
I had a brief look at MRTG. It seems to be a well written app and
while it can handle counter reset (with potential loss of an unknown
amount of data), it does not actively reset counters. It also doesn't
use iptables.
> Applications need to get smarter.
Agreed.
> Don't place arbitrary rules where they don't belong.
Agreed too. Counters should not arbitrarily be equipped with a reset
capability, there is hardly any benefit in that and it causes nothing
but problems.
As far as I can see, the counters in /proc/net/snmp don't have a
reset, same with /proc/net/dev and possibly other counters elsewhere.
Ideally iptables would fall in line with that. Rules can still be
unloaded and reloaded, also causing counter reset and loss of data,
but since that is a lot more involved, application authors would have
an incentive to handle counters properly.
--
Manfred Bartz
On Tue, Apr 17, 2001 at 07:53:28AM +1000, David Findlay wrote:
> In the 2.5 series of kernels, working towards 2.6, could you please make the
> IP Accounting so that I can set a single rule that will make it watch all IP
> traffic going from the local network, through the masquerading service to the
> internet, and log local IP Addresses using it? This would allow me to set 1
> rule, but have the information I want on a per IP address system.
:) Well... there was a discussion about this on the netfilter-devel
list some weeks ago.
This is definitely not a 2.5 / 2.6 issue, as it could be easily implemented
by using the already existing flexibility of netfilter/iptables.
It's just a matter of somebody who needs it getting around actually
implementing it. Could be either implemented as a 'raw' netfilter
hook-attaching module or (more convenient) as a new iptables target,
which allocates some internal tables for storing the accounting data.
Doesn't make sense to have a discussion about it on linux-kernel, and
it certainly doesn't belong into a 2.5 featurelist.
Please refer to the netfilter-devel mailinglist (subscription information
available at http://netfilter.samba.org/)
> David
--
Live long and prosper
- Harald Welte / [email protected] http://www.gnumonks.org
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M-
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
>
> If there really is a performance issue with a few hundred rules, then
> it can be overcome by grouping rules in separate custom chains. F.e.
> if you have 1024 rules create 32 custom chains with 32 rules each.
> Then have 32 rules in your main table which jump to the appropriate
> custom chain --> maximum rules traversed by each packet = 64.
still.. if you want to log multiple class B's it's a PITA, especially
if your ruleset changes and you have to atomically replace all chains
in one table to update the ruleset from userspace->kernel.
> There is another issue with logging in general:
>
> *COUNTERS MUST NOT BE RESETABLE!!!*
>
> Resetable counters guarantee that no two programs can co-exists if
> they happen to reset the same counters.
That sounds like crap (sorry). Counters are resettable, and will be.
If you run two applications resetting counters individually, you have
a problem with your applications.
> All logging counters should be implemented with 32bit or 64bit
> unsigned integers. Any software using correct unsigned integer
they are uint32 right now.
> arithmetic can then simply subtract a previous value from the current
> value to get the difference. This works reliably across counter
> wrap-arounds. There is absolutely *no need for reset* !
so why are your applications resetting counters then? Nobody forces you
to reset them right now.
> Manfred Bartz
> ---------------------------------------------------------------
--
Live long and prosper
- Harald Welte / [email protected] http://www.gnumonks.org
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M-
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
Harald Welte <[email protected]> writes:
> On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
> > Resetable counters guarantee that no two programs can co-exists if
> > they happen to reset the same counters.
>
> That sounds like crap (sorry).
Care to explain how two independent apps can co-exists if they happen
to reset the same counters?
> so why are your applications resetting counters then?
Ask the authors, not me. I wiped those apps from my system.
Its just a pity about the time wasted.
> Nobody forces you to reset them right now.
Why do authors then write apps (f.e. ipac) which reset counters?
Perhaps only because they can, not because it makes sense.
--
Manfred Bartz
Harald Welte <[email protected]> writes:
>On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
>> *COUNTERS MUST NOT BE RESETABLE!!!*
>>
>> Resetable counters guarantee that no two programs can co-exists if
>> they happen to reset the same counters.
>That sounds like crap (sorry). Counters are resettable, and will be.
>If you run two applications resetting counters individually, you have
>a problem with your applications.
Resettable counters in a security sensitive environment are just a
call for trouble. That's why you can't reset the SNMP counters on any
Cisco device I've encountered today. They learned their lesson. Maybe
you will, too.
Regards
Henning
--
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH [email protected]
Am Schwabachgrund 22 Fon.: 09131 / 50654-0 [email protected]
D-91054 Buckenhof Fax.: 09131 / 50654-20
> Umm, no. Counters can be resetable - you just specify that accounting
> programs should not reset them, ever.
>
> The ability to reset counters is extremely useful if you're a human
> looking at the output of iptables -L -v. (I thus far know of no one
> who can memorise the counter values for around 40 rules).
You'll get bogus accounting results unless you stop/restart the
accounting programs every time you manually deal with the counters.
This sounds dangerously easy to make mistakes to me.
Olaf
> Similarly, if my InPackets are at 102345 at one read, and 2345 the
> next read, and I know that my counter is 32 bits, then I know i've
> wrapped and can do my own math.
No. When you have resettable counters, you don't know if the counter
has wrapped or been reset. Either you have received 2345 packets in
between, or 2^32-102345. The difference is not negligible. ;-)
Unless you have a second counter which is incremented with every wrap
(_or_, perhaps better, with every reset) and can _not_ be reset
manually. This would be the date in your date/time example.
Olaf
Leif Sawyer <[email protected]>:
> > From: Ian Stirling [mailto:[email protected]]
> > > Manfred Bartz responded to
> > > > Russell King <[email protected]> who writes:
> > <snip>
> > > > You just illustrated my point. While there is a reset capability
> > > > people will use it and accounting/logging programs will get wrong
> > > > data. Resetable counters might be a minor convenience
> > when debugging
> > > > but the price is unreliable programs and the loss of the
> > ability of
> > > > several programs to use the same counters.
> > >
> > > You of course, are commenting from the fact that your
> > applications are
> > > stupid, written poorly, and cannot handle 'wrapped' data. Take MRTG
> > <snip>
> > > Similarly, if my InPackets are at 102345 at one read, and
> > 2345 the next
> > > read,
> > > and I know that my counter is 32 bits, then I know i've
> > wrapped and can do
> >
> > I think the point being made is that if InPackets are at
> > 102345 at one read,
> > and 2345 the next, and you know it's a 32 bit counter, it's completely
> > unreliable to assume that you have in fact recieved 4294867295
> > packets, if the counter can be zeroed.
> > You can say nothing other than at least 2345 packets, at most
> > 2345+n*2^32 have been got since you last checked.
>
> Ah, yes.. I seem to have misplaced a bit of text in my reply.
>
> The continuation of thought:
>
> How the application derives the status of a wrapped counter or
> a zero'ed counter is dependant on the device being monitored.
>
> Yes, you have to know what your interface is capable of (maxbytes/sec)
> so that you can do a simple calculation where:
>
> maximum_throughput = maxbytes_sec * (time_now - time_last_read)
>
> and if your previous good counter + the maximum throughput wraps the
> counter, you have a good chance that you've simply wrapped.
>
> If not, then you can assume that your counters were cleared at some point,
> log the data you've got, and keep moving forward.
And that introduces errors in measurement. It also depends on how frequently
an uncontroled process is clearing the counters. You may never be able to
get a valid measurement.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
Jesse Pollard continues with:
> Leif Sawyer <[email protected]>:
>>> Ian Stirling [mailto:[email protected]]
>>>> Manfred Bartz responded to
>>>>> Russell King <[email protected]> who writes:
>>>>>
>>>>> You just illustrated my point. While there is a
>>>>> reset capability people will use it and accounting/
>>>>> logging programs will get wrong data. Resetable
>>>>> counters might be a minor convenience when debugging
>>>>> but the price is unreliable programs and the loss of the
>>>>> ability of several programs to use the same counters.
>>>>
>>>> You of course, are commenting from the fact that your
>>>> applications are stupid, written poorly, and cannot handle
>>>> 'wrapped' data. Take MRTG
>>>> <snip>
>>>> Similarly, if my InPackets are at 102345 at one read, and
>>>> 2345 the next read, and I know that my counter is 32 bits,
>>>> then I know i've wrapped and can do
>>>
>>> I think the point being made is that if InPackets are at
>>> 102345 at one read, and 2345 the next, and you know it's
>>> a 32 bit counter, it's completely unreliable to assume that
>>> you have in fact recieved 4294867295 packets, if the counter
>>> can be zeroed. You can say nothing other than at least 2345
>>> packets, at most 2345+n*2^32 have been got since you last
>>> checked.
>>
>> Ah, yes.. I seem to have misplaced a bit of text in my reply.
>>
>> The continuation of thought:
>>
>> How the application derives the status of a wrapped counter or
>> a zero'ed counter is dependant on the device being monitored.
>>
>> Yes, you have to know what your interface is capable of
>> (maxbytes/sec) so that you can do a simple calculation where:
>>
>> maximum_throughput = maxbytes_sec * (time_now - time_last_read)
>>
>> and if your previous good counter + the maximum throughput wraps
>> the counter, you have a good chance that you've simply wrapped.
>>
>> If not, then you can assume that your counters were cleared
>> at some point, log the data you've got, and keep moving forward.
>
> And that introduces errors in measurement. It also depends on
> how frequently an uncontroled process is clearing the counters.
> You may never be able to get a valid measurement.
This is true. Which is why application programmers need to write
code as if they are not the only [ab]users of data.
Which brings me back to my point.
Don't force the kernel to uphold your local application requirements
of stable counters.
Enforce it in the userspace portion of the code.
<subtopic>
Yes, you could extend the proc filesystem (ugh) with a flag that could
be read by the ip[chains|tables] user app to determine if clearing flags
were allowed. Then a simple
echo 1 > /proc/sys/net/ipv4/counters_locked
or some such cruft. But I don't see this extension making into the
standard kernel at this time. It just seems to be wasteful.
</subtopic>
If you (at your site) really need this type of functionality, it's
pretty darn simple to write a wrapper to ip[tables|chains] which
silently (or not so) drops the option to clear the counters before
calling the real version.
Besides, what would be gained in making the counters RO, if they were
cleared every time the module was loaded/unloaded?
Leif Sawyer <[email protected]>:
> > And that introduces errors in measurement. It also depends on
> > how frequently an uncontroled process is clearing the counters.
> > You may never be able to get a valid measurement.
>
> This is true. Which is why application programmers need to write
> code as if they are not the only [ab]users of data.
>
> Which brings me back to my point.
>
> Don't force the kernel to uphold your local application requirements
> of stable counters.
>
> Enforce it in the userspace portion of the code.
>
> <subtopic>
> Yes, you could extend the proc filesystem (ugh) with a flag that could
> be read by the ip[chains|tables] user app to determine if clearing flags
> were allowed. Then a simple
>
> echo 1 > /proc/sys/net/ipv4/counters_locked
>
> or some such cruft. But I don't see this extension making into the
> standard kernel at this time. It just seems to be wasteful.
> </subtopic>
>
> If you (at your site) really need this type of functionality, it's
> pretty darn simple to write a wrapper to ip[tables|chains] which
> silently (or not so) drops the option to clear the counters before
> calling the real version.
>
> Besides, what would be gained in making the counters RO, if they were
> cleared every time the module was loaded/unloaded?
1. Knowlege that the module was reloaded.
2. Knowlege that the data being measured is correct
3. Having reliable measures
4. being able to derive valid statistics
....
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
Jesse Pollard replies:
to Leif Sawyer who wrote:
>> Besides, what would be gained in making the counters RO, if
>> they were cleared every time the module was loaded/unloaded?
>
> 1. Knowlege that the module was reloaded.
> 2. Knowlege that the data being measured is correct
> 3. Having reliable measures
> 4. being able to derive valid statistics
> ....
Good. Now that we have valid objectives to reach, which of these
are NOT met by making the fixes entirely in userspace, say by
incorporating a wrapper script to ensure that no external applications
can flush the table counters?
They're still all met, right?
And we haven't had to fill the kernel with more cruft.
I repeat myself, fighting is apparently so pleasant that you are stuck on
fighting over dead-end technology:
I seriously suggest that for the primary (subject given) topic
you are SERIOUSLY OFF TARGET. Look around, counting hits on
some fw rules is waste of time! (And mightly inaccurate!)
You absolutely don't want to do any sort of counting aggeration policy
control within kernel ( = FW rules ). You want to collect accounting
per flow, and send those data records to offline analysis.
No more fighting of when to clear counters, and when not.
Having used (with own custom analyzers) cisco netflow, I can say
that any sort of "count hits on access-list elements" things are
from stone-age:
http://www.cisco.com/warp/public/cc/pd/iosw/ioft/neflct/tech/napps_wp.htm
Yet another nice thing to cook up -- if I had time ...
/Matti Aarnio
--------- Received message begins Here ---------
>
> Jesse Pollard replies:
> to Leif Sawyer who wrote:
> >> Besides, what would be gained in making the counters RO, if
> >> they were cleared every time the module was loaded/unloaded?
> >
> > 1. Knowlege that the module was reloaded.
> > 2. Knowlege that the data being measured is correct
> > 3. Having reliable measures
> > 4. being able to derive valid statistics
> > ....
>
> Good. Now that we have valid objectives to reach, which of these
> are NOT met by making the fixes entirely in userspace, say by
> incorporating a wrapper script to ensure that no external applications
> can flush the table counters?
>
> They're still all met, right?
Nope - some of the applications that may be purchased do not have
to go through the scripts to reset the counters. They just need access
to the counters, and reset is built into the applications. This violates
all 4 objectives.
> And we haven't had to fill the kernel with more cruft.
Removing/no-oping the reset code would make the module
SMALLER, and simpler.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
> > Jesse Pollard replies:
> > to Leif Sawyer who wrote:
> > >> Besides, what would be gained in making the counters RO, if
> > >> they were cleared every time the module was loaded/unloaded?
> > >
> > > 1. Knowlege that the module was reloaded.
> > > 2. Knowlege that the data being measured is correct
> > > 3. Having reliable measures
> > > 4. being able to derive valid statistics
> > > ....
> >
> > Good. Now that we have valid objectives to reach, which of these
> > are NOT met by making the fixes entirely in userspace, say by
> > incorporating a wrapper script to ensure that no external
> applications
> > can flush the table counters?
> >
> > They're still all met, right?
>
> Nope - some of the applications that may be purchased do not have
> to go through the scripts to reset the counters. They just need access
> to the counters, and reset is built into the applications.
> This violates
> all 4 objectives.
>
> > And we haven't had to fill the kernel with more cruft.
>
> Removing/no-oping the reset code would make the module
> SMALLER, and simpler.
>
NO. Don't remove the functionality that is required. Fix your
userspace applications to behave correctly. If _you_ require your
userspace applications to not clear counters, then fix the application.
The counters are not gospel. The modules are not omnipresent.
Time and again, people beg and plead the kernel developers to change
the kernel to suit their personal biases and applications. Historically
they have been vehemently against it. Of what purpose does breaking EVERY
application out there by performing this kernel mod have?
As Matti Aarnio points out, this is nearly beaten to death. Since none
of the 'Big Boys' have commented, my guess is that this is a moot argument,
and no changes to the kernel will take place.
/me steps off of soapbox
/me retires this thread
Leif Sawyer <[email protected]> writes:
> Jesse Pollard replies:
> to Leif Sawyer who wrote:
> > > Besides, what would be gained in making the counters RO, if
> > > they were cleared every time the module was loaded/unloaded?
> >
> > 1. Knowlege that the module was reloaded.
> > 2. Knowlege that the data being measured is correct
> > 3. Having reliable measures
> > 4. being able to derive valid statistics
> Good. Now that we have valid objectives to reach, which of these
> are NOT met by making the fixes entirely in userspace, ...
By this logic you could ask why we need separate user spaces,
just fix all applications so they cooperate nicely.
> They're still all met, right?
No. Because they require well behaved, cooperative user programs.
> And we haven't had to fill the kernel with more cruft.
Nonsense. Removing counter reset capabilities reduces kernel cruft.
--
Manfred Bartz
Leif Sawyer <[email protected]> writes:
> > > Jesse Pollard replies:
> > Removing/no-oping the reset code would make the module
> > SMALLER, and simpler.
> NO. Don't remove the functionality that is required.
Please explain where counter reset capability provides any
functionality that is not already available without it.
You might want to read RFC2724 ``Traffic Flow Measurement''.
Search for ``reset''.
Counter resets have always caused problems, that's why mission
critical counters never have a reset. Have you ever seen an
electricity, gas or water meter with a reset? The same reasoning
applies when you have accounting rules that are used to charge
customers for traffic volume.
> Fix your userspace applications to behave correctly. If _you_
> require your userspace applications to not clear counters, then fix
> the application.
You are confused. What would you say if a close() by another,
unrelated application closed all open descriptors for that file,
including the one you just opened? Just fix your applications?
--
Manfred Bartz
Manfred Bartz writes:
> You are confused. What would you say if a close() by another,
> unrelated application closed all open descriptors for that file,
> including the one you just opened? Just fix your applications?
Of course I would fix the application.
And I certainly wouldn't rip the close() out of libc, which is
the tactic that you are suggesting.
Sheesh.
> > Fix your userspace applications to behave correctly. If _you_
> > require your userspace applications to not clear counters, then fix
> > the application.
>
> You are confused. What would you say if a close() by another,
No he isnt confused, you are trying to dictate policy.
> unrelated application closed all open descriptors for that file,
> including the one you just opened? Just fix your applications?
That would be a bug. The standards and common sense say so, but your argument
is more akin to 'if I delete a file in one app it hurts when I try and use
it in another'.
Alan Cox <[email protected]> writes:
> > > Fix your userspace applications to behave correctly. If _you_
> > > require your userspace applications to not clear counters, then fix
> > > the application.
> >
> > You are confused. What would you say if a close() by another,
>
> No he isnt confused, you are trying to dictate policy.
What then *is* the policy?
AFAIK, no counters in the Linux kernel have resets, except in netfilter.
--
Manfred Bartz
> > No he isnt confused, you are trying to dictate policy.
>
> What then *is* the policy?
The policy is not to have policy. It works as well in kernel design as politics.
Alan
Alan Cox wrote:
> > > Fix your userspace applications to behave correctly. If _you_
> > > require your userspace applications to not clear counters, then fix
> > > the application.
> >
> > You are confused. What would you say if a close() by another,
>
> No he isnt confused, you are trying to dictate policy.
Well it's not actually possible to do a fix in userspace for a odometer type of counter that can be reset. I don't know what you mean about policy but this reset "feature" is a shure way to get bad values. I have not seen one good reason to have a reset other than it easier to read and that is something that can be fixed in userspace.
> I repeat myself, fighting is apparently so pleasant that
> you are stuck on
> fighting over dead-end technology:
>
> I seriously suggest that for the primary (subject given) topic
> you are SERIOUSLY OFF TARGET. Look around, counting hits on
> some fw rules is waste of time! (And mightly inaccurate!)
I agree. We could all stop re-inventing the wheel and use a
RFC2724/RFC2722/RFC2720 compliant traffic meter such as NeTraMet -
which has already solved most of the mentioned problems - has a
flexible rule language for matching flows and managing counters -
support for multiple protocols; not just IP - a distributed
architecture - SNMP accessable meter and remote Manager and Controller
(NeMaC) which can concurrently read from multiiple meters (including
NetFlow meters).
> You absolutely don't want to do any sort of counting
> aggeration policy
> control within kernel ( = FW rules ). You want to
> collect accounting
> per flow, and send those data records to offline analysis.
Yes, the IP accounting effort could do well by creating a fastpath for
feeding packet headers (can't say I know how optimal libpcap is
currently on Linux) to a userspace meter (like NeTraMet) letting it
deal with all of the policy.
I remember the DOS version of NeTraMet performed much better than the
Linux version (some years ago) due to custom ethernet drivers (for
some cards) that only generate interupts when a ring buffer is full of
packet headers - maybe the same sort of infrastructure (some of the
Linux GigE drivers also avoid the interupt per packet performance hit)
could be added to Linux and integrated with libpcap and leave the rest
up to a userspace meter application.
I'm sure Neville (traffic metering god - Hi Neville) would be pleased
to have optimized support for NeTraMet in the Linux kernel.
> No more fighting of when to clear counters, and when not.
>
> Having used (with own custom analyzers) cisco netflow, I can say
> that any sort of "count hits on access-list elements" things are
> from stone-age:
NetFlow really sucks alot doesn't - I remeber having bad aliasing
problems (trying to generate 5min averages) due to its minumum flow
export interval of 1 minute. Is this still the case?
Michael Clark.
At 10:16 AM +0200 2001-04-18, Kenneth Johansson wrote:
>Alan Cox wrote:
>
>> > > Fix your userspace applications to behave correctly. If _you_
>> > > require your userspace applications to not clear counters, then fix
>> > > the application.
>> >
>> > You are confused. What would you say if a close() by another,
>>
>> No he isnt confused, you are trying to dictate policy.
>
>Well it's not actually possible to do a fix in userspace for a odometer type of counter that can be reset. I don't know what you mean about policy but this reset "feature" is a shure way to get bad values. I have not seen one good reason to have a reset other than it easier to read and that is something that can be fixed in userspace.
Moreover, in a non-resettable-counter environment, any client of the counter can trivially achieve the effect of a reset by (locally) storing a snapshot and doing a subtract. Conversely, if the counter is truly reset, information is lost permanently.
--
/Jonathan Lundell.
In article <[email protected]>,
Alan Cox <[email protected]> writes:
>> > No he isnt confused, you are trying to dictate policy.
>>
>> What then *is* the policy?
>
> The policy is not to have policy. It works as well in kernel design as politics.
>
> Alan
>
Since my job is in fact mainly this kind of apps, i really feel strongly
about this.
Resettable counters are evil.
Having resettable counters may not sound like it, but it is in fact policy.
It forces all apps to add code to detect these resets (and then give
warnings/errors, since there is just no way to do anything sensible with
them), since ignoring them will seemingly cause up to 2**32 counts suddenly.
It is also doing something in kernelspace which can be done in userspace,
which is normally considered a big nono.
Proposal: have a snapshot command, that remembers the current value of a
counter. Then have two interfaces: one that shows the continuous counter
and one that shows the subtraction of the current value from the snapshot.
The first can be used by used by serious applications (don't have to
add code to give warnings about dataloss), and the second can be used by
users who want to watch the counters a bit to get a feel for what a rule
is doing.
I really think cisco got this right: from the commandline interface
you can reset counters, and watch them, the SNMP counters however just
keep going and going and going independently from this.
(I think this snapshotting belongs in the apps reading the counters
really, but if you really insist on a kernel based reset, this might be
reasonable).
On Tue, Apr 17, 2001 at 11:13:19AM +1000, Manfred Bartz wrote:
> I had a brief look at MRTG. It seems to be a well written app and
> while it can handle counter reset (with potential loss of an unknown
> amount of data), it does not actively reset counters. It also doesn't
> use iptables.
Yes, that's the whole point. If you want to have some kind of per-ip /
per-network accounting, I'd rather recommend using something else
(i.e. an accounting module attaching to the netfilter hook itself,
something using the ULOG target to do the logging in userspace, ...)
Just reading out the per-rule counters always introduces problems as soon
as your ruleset changes. And when do you have a really static ruleset?
There are always new services/... to configure.
> Agreed too. Counters should not arbitrarily be equipped with a reset
> capability, there is hardly any benefit in that and it causes nothing
> but problems.
so what about iptables-save at shutdown time and iptables-restore at
bootup time? Then you can have your counters persist even after kernel
upgrades / reboots / crashes / ...
> As far as I can see, the counters in /proc/net/snmp don't have a
> reset, same with /proc/net/dev and possibly other counters elsewhere.
Yes, because it is per network device, not per some arbitrarily inserted
rule which can be changed all the time. As I've stated more than once
in this thread, and as you just continue to say: just delete and re-insert
the rule, and you have your counter reset.
> Ideally iptables would fall in line with that. Rules can still be
> unloaded and reloaded, also causing counter reset and loss of data,
> but since that is a lot more involved, application authors would have
> an incentive to handle counters properly.
I don't think that the iptables kernel part should remove some feature
just because there are application programmers wrongly designing their
applications.
> Manfred Bartz
--
Live long and prosper
- Harald Welte / [email protected] http://www.gnumonks.org
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M-
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
On Tue, Apr 17, 2001 at 12:29:30PM +0200, Olaf Titz wrote:
> > Umm, no. Counters can be resetable - you just specify that accounting
> > programs should not reset them, ever.
> >
> > The ability to reset counters is extremely useful if you're a human
> > looking at the output of iptables -L -v. (I thus far know of no one
> > who can memorise the counter values for around 40 rules).
>
> You'll get bogus accounting results unless you stop/restart the
> accounting programs every time you manually deal with the counters.
> This sounds dangerously easy to make mistakes to me.
mmh. But isn't that obvious? Every time you mess with anything (counters
or rules) of the chains/tables, your accounting program will get into
trouble, if it relies on those counters.
I don't think that it makes sense to do iptables-based accounting if you have
changes to the ruleset (counters and/or rules) at runtime. You'd have
to be very cautious what you are doing.
> Olaf
--
Live long and prosper
- Harald Welte / [email protected] http://www.gnumonks.org
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M-
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
On Tue, Apr 17, 2001 at 06:56:42AM +0000, Henning P. Schmiedehausen wrote:
>
> Resettable counters in a security sensitive environment are just a
> call for trouble. That's why you can't reset the SNMP counters on any
> Cisco device I've encountered today. They learned their lesson. Maybe
> you will, too.
Well, I'm not sure about which SNTP counters you are talking, but I suppose
it is not about per-filtering-rule counters, but something like per-interface
counters, etc.
There's always a way for somebody with root access to reset the counters of
a rule:
just delete and re-insert the rule.
If somebody wants to reset the counter, he can. If we remove the functionality
from iptables, people still can - but it's more difficult.
> Regards
> Henning
--
Live long and prosper
- Harald Welte / [email protected] http://www.gnumonks.org
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M-
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
Harald Welte <[email protected]> writes:
>On Tue, Apr 17, 2001 at 06:56:42AM +0000, Henning P. Schmiedehausen wrote:
>>
>> Resettable counters in a security sensitive environment are just a
>> call for trouble. That's why you can't reset the SNMP counters on any
>> Cisco device I've encountered today. They learned their lesson. Maybe
>> you will, too.
>Well, I'm not sure about which SNTP counters you are talking, but I suppose
>it is not about per-filtering-rule counters, but something like per-interface
>counters, etc.
You don't want your counters going backward. Full stop. If a program
can reset your counter, your application will never know, if it was a
legal, correct, valid reason or just a hacker trying to hide his
traces. At least provide some sort of lock-down.
>There's always a way for somebody with root access to reset the counters of
>a rule:
>just delete and re-insert the rule.
Bad thing. If you want to use the rules in a security sensitive
environment, don't allow removal. If you need to, reset the whole
module and notify the user. Better, shut down the filter and yell for
help.
ipfilter is about security, isn't it?
>If somebody wants to reset the counter, he can. If we remove the functionality
>from iptables, people still can - but it's more difficult.
There is no "more difficult", just "different ways". If you bother to
use a filtering environment where the filter counters tell e.g. "we
rejected xxx attacks", you don't want anyone to mess with these
counters. If this is a counter for a filtering rule, don't allow the
rule (and its counters) to be removed. Inactivate the rule but keep
the evidence (counter settings) around till module removal or reboot.
Sorry, I may be anal about security but as more and more people start
thinking "why should I bother with FW-1 or PIX when I can get a $99
Linux box and hack some filters", at least I want the $99 software
trying not to be sloppy about security.
Regards
Henning
--
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH [email protected]
Am Schwabachgrund 22 Fon.: 09131 / 50654-0 [email protected]
D-91054 Buckenhof Fax.: 09131 / 50654-20