2003-01-10 14:38:17

by Mihnea Balta

[permalink] [raw]
Subject: Kernel hooks just to get rid of copy_[to/from]_user() and syscall overhead?

Hi,

I have to implement a system which grabs udp packets off a gigabit connection,
take some basic action based on what they contain, repack their data with a
custom protocol header and send them through a gigabit ethernet interface on
broadcast.

I know how to do this in userspace, but I need to know if doing everyting in
the kernel would show a considerable speed improvement due to removing
syscall and memory copy overhead. The system will be quite stressed, having
to deal with around 15-20000 packets/second.

I didn't want to start this e-mail with an excuse, so I delayed it until here
:). I appologise if this isn't the right place to ask, it seemed that way to
me. I wasn't able to find sufficient and coherent information about this
issue on the internet or on this mailing list's archives, so I decided to ask
you people directly.

Thanks for your time,
Mihnea Balta


2003-01-10 14:54:19

by Richard B. Johnson

[permalink] [raw]
Subject: Re: Kernel hooks just to get rid of copy_[to/from]_user() and syscall overhead?

On Fri, 10 Jan 2003, Mihnea Balta wrote:

> Hi,
>
> I have to implement a system which grabs udp packets off a gigabit connection,
> take some basic action based on what they contain, repack their data with a
> custom protocol header and send them through a gigabit ethernet interface on
> broadcast.
>
> I know how to do this in userspace, but I need to know if doing everyting in
> the kernel would show a considerable speed improvement due to removing
> syscall and memory copy overhead. The system will be quite stressed, having
> to deal with around 15-20000 packets/second.
>
> I didn't want to start this e-mail with an excuse, so I delayed it until here
> :). I appologise if this isn't the right place to ask, it seemed that way to
> me. I wasn't able to find sufficient and coherent information about this
> issue on the internet or on this mailing list's archives, so I decided to ask
> you people directly.
>
> Thanks for your time,
> Mihnea Balta
>

I think you should do everything in kernel space, with a user-mode
interface for non-realtime control, i.e., what characteristics
of UDP packets are being "filtered". You just make a module that
contains what you need, with an ioctl() hook to control it.

That way, data from your gigabit interface(s) never has to get to
"user-space" at all.


Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Why is the government concerned about the lunatic fringe? Think about it.


2003-01-10 15:20:32

by Gianni Tedesco

[permalink] [raw]
Subject: Re: Kernel hooks just to get rid of copy_[to/from]_user() and syscall overhead?

On Fri, 2003-01-10 at 14:45, Mihnea Balta wrote:
> Hi,
>
> I have to implement a system which grabs udp packets off a gigabit connection,
> take some basic action based on what they contain, repack their data with a
> custom protocol header and send them through a gigabit ethernet interface on
> broadcast.
>
> I know how to do this in userspace, but I need to know if doing everyting in
> the kernel would show a considerable speed improvement due to removing
> syscall and memory copy overhead. The system will be quite stressed, having
> to deal with around 15-20000 packets/second.

mmap() packet socket interface eliminates the need for system calls when
traffic is coming in at a high rate. The kernel -> user copy is also
eliminated, but its just replaced with a kernel -> kernel copy :P

You could perhaps also use linux socket filters to minimize the number
of packets you need to evaluate...

Check out this sample code: http://www.scaramanga.co.uk/code-fu/lincap.c

HTH

--
// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source http://www.scaramanga.co.uk/gianni-at-ecsc.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D


Attachments:
signature.asc (189.00 B)
This is a digitally signed message part

2003-01-13 08:30:22

by Mihnea Balta

[permalink] [raw]
Subject: Re: Kernel hooks just to get rid of copy_[to/from]_user() and syscall overhead?

On Friday 10 January 2003 16:45, Mihnea Balta wrote:
> Hi,
>
> I have to implement a system which grabs udp packets off a gigabit
> connection, take some basic action based on what they contain, repack their
> data with a custom protocol header and send them through a gigabit ethernet
> interface on broadcast.

Thank you to everybody who answered, I'll try the mmap()ed descriptor
userspace solution, as it seems good enough.

2003-01-14 11:39:57

by Mihnea Balta

[permalink] [raw]
Subject: Re: Kernel hooks just to get rid of copy_[to/from]_user() and syscall overhead?

On Friday 10 January 2003 16:45, Mihnea Balta wrote:
> Hi,
>
> I have to implement a system which grabs udp packets off a gigabit
> connection, take some basic action based on what they contain, repack their
> data with a custom protocol header and send them through a gigabit ethernet
> interface on broadcast.

Following the indications I got here, I did the packet recieving stuff using a
mmaped packet socket. From what I understand, that is a recieve-only
interface, so it seems that I'm stuck with old sendto() for putting the
packets back on the wire. I'd like to know if sendto() can do 20000 (small)
packets/second on a fast x86 MP machine (dual or maybe quad) which doesn't do
much besides this routing process. If it can't, please tell me if there's any
feasible way of implementing a kind of buffer, i.e. keeping recieved packets
in a local buffer and sending them alltoghether when the buffer fills or
after a timer expires (I'm thinking DMA to the NIC or something simmilar, as
the buffer will contain the complete packets, with all the required packets).

Thanks,
Mihnea

PS: feasible == not spending 1 month development time for a 1
microsecond/packet improvement.