Hello,
I want to analyze how much time is spent on forwarding process in linux
kernel.
This is second post but the matter is very important to me. I am dealing
with this for few months now.
To accomplish this I looked at networking code linux_src/net/ipv4/* and did
simple (I started with C programming for the purpose of this analyzys so
apologize my possibly simple problems) modifications in files.
To learn packet trace trough the kernel, I read some docs on forwarding
fucntions executed. Later on I found some tips on measuring time in Linux
(O'Reilly Device drivers). The most precise measurements (as far as I know)
is rdtscl() that gives cycle count.
I put code snippets like:
*********************************************************************
rdtscl(start);
/* some important networking code .. like irq, rcv, fwd, dev_xmit */
rdtscl(end);
printk("[irq] serving irq took %li cycles", end-start);
******************************************************************
..all over vital forwarding components. Compiled it (kernel 2.4.18, x386)
ane generated some traffic with IP Traffic trial software.
As a result please see example dmesg underneath... ([x,y] means
gettimeofday() usec's):
>>dmesg
[1055450168,605398][irq] (numer cyklu->-1665244472)
[1055450168,605464][rx_irq] numer cyklu->-1665229089
[1055450168,605515][rx_irq] (eth_copy_and_sum) trwalo : 1125
[1055450168,605585]call netif_rx(pakiet nr[126]) (numer cyklu->-1665200955)
[1055450168,605623]started netif_rx() (numer cyklu->-1665191843)
[1055450168,605661][netif_rx()] packet is put to backlog: (numer
cyklu->-1665183171)
[1055450168,605747][netif_rx]jump to ip_rcv1(). (numer cyklu->-1665163077)
[1055450168,605803][netif_rx]jump to ip_rcv2(). (numer cyklu->-1665150012)
[1055450168,605845][netif_rx]jump to ip_rcv3(). (numer cyklu->-1665140257)
[1055450168,605877][ip_rcv()] pakiet_seq[0] rozpoczeto ip_rcv() (numer
cyklu -1665132875)
[1055450168,605919][ip_rcv()] pakiet_seq[26112] fast_csum() took=311
[1055450168,606000]determining: locally/forward ...
[1055450168,606025][ip_route_input]rt_hash_code() took= -176
[1055450168,606056][ip_route_input]started lookup
>>
But I am aware that this code (i mean - my modifications) kills performance,
if not disturbs outcomes completely ;(
Problem is that dmesg buffer is TOO SMALL (although i am aware i can modify
its size in print.c) and every packets brings lots of data on the screen.
What I would like to do is to run some 2mbps (2 flows; EF PHB and BE PHB)
and see how packet size influences processing time of forwarding steps
(including classification, lookup, fwd, queueing etc.). THIS IS IMPOSSIBLE
with such a small dmesg-buffer and using file system writes for every packet
(this harms outcomes - i thinnk).
Now I am thinking about two things that might help me in my analyzys:
1) creating some mmap'ed memory object and writing timing info to shared
buffer, so that I can postprocess it after experimental packet number have
been sent (forwarded)
2) totally changing the way of measurements
My question is:
A) could anyone COMMENT on my solution to forwarding performance tracing
B) could anyone SUGGEST other (better) way to do this efficiently and to get
reliable data (times)
C) does it have anything in common with kernel profiling??? if yes please
explain me how should I do this
D) ..anything helpfull welcomed here, maybye someone has tried something
like this and would like to share her/his experiences???
i would be most grateful for any comments on my EMAIL.
desperately looking for answer
adam
Adam Flizikowski writes:
>
> Hello,
>
> I want to analyze how much time is spent on forwarding process in linux
> kernel.
>
> This is second post but the matter is very important to me. I am dealing
> with this for few months now.
Hello!
"time spent on forwarding" is very vague. Raw forwarding capacity use
to be measured in pps (packets per second) and it depends on many things
beside hardware as packet size, routing table size, new flows/s etc.
You can look at (o)profiles during forwarding.
Cheers.
--ro
I agree the term forwarding is vague - this is problably my english
imprecission...
I mean timing the whole packet path:
strating from irq handing through putting into backlog, picking from
backlog, handling packet to higher layers (3), dealing with packet header
processing (CRC, TTL etc), lookup, classification (for output queueing),
forwarding, puting packet to output queue, tx_queue, and hard_dev_xmit et
least....
this is what I would like to measure/benchmark (sorry my imprecise english
usage).
I would like to see how packet size influences certain operations (as
mentioned earlier) etc....
let's say having as the output the functions called with its timings ...
i would like to start two flows through linux router: PHB EF and BE PHB..
like voip and ftp or so...
than i want to analyse what exactly happens ... since my theoretical
analysys show delays (or latencies - from packet entering the NIC to going
out of the outgoing interface) of hundereds of usec (~200us) while
experiments shows 5-10ms !!!!! with CBQ (configured like CBWFQ and LLQ)
this is it
any help welcomed
regards
adam
-----Original Message-----
From: [email protected]
[mailto:[email protected]]On Behalf Of Robert Olsson
Sent: Friday, June 27, 2003 1:41 PM
To: Adam Flizikowski
Cc: [email protected]; [email protected]
Subject: How to do kernel packet forwarding performance analysys - please
comment on my method
Adam Flizikowski writes:
>
> Hello,
>
> I want to analyze how much time is spent on forwarding process in linux
> kernel.
>
> This is second post but the matter is very important to me. I am dealing
> with this for few months now.
Hello!
"time spent on forwarding" is very vague. Raw forwarding capacity use
to be measured in pps (packets per second) and it depends on many things
beside hardware as packet size, routing table size, new flows/s etc.
You can look at (o)profiles during forwarding.
Cheers.
--ro
adamski writes:
> i would like to start two flows through linux router: PHB EF and BE PHB..
> like voip and ftp or so...
>
> than i want to analyse what exactly happens ... since my theoretical
> analysys show delays (or latencies - from packet entering the NIC to going
> out of the outgoing interface) of hundereds of usec (~200us) while
> experiments shows 5-10ms !!!!! with CBQ (configured like CBWFQ and LLQ)
Seems you have to add your own hooks for this and I guess you've seen
CONFIG_NET_PROFILE which does accurate measurements for network related
operations. It was long time since I used though...
Cheers.
--ro
> i would like to start two flows through linux router: PHB EF and BE PHB..
> like voip and ftp or so...
>
> than i want to analyse what exactly happens ... since my theoretical
> analysys show delays (or latencies - from packet entering the NIC to
going
> out of the outgoing interface) of hundereds of usec (~200us) while
> experiments shows 5-10ms !!!!! with CBQ (configured like CBWFQ and LLQ)
Seems you have to add your own hooks for this and I guess you've seen
CONFIG_NET_PROFILE which does accurate measurements for network related
operations. It was long time since I used though...
.... could you please say something about CONFIG_NET_PROFILE and hooks you
have mentioned????
are there any documents ... related with hooks and CONFIG_NET_PROFILE
best regards
adam