Hi:
I have a very puzzled question. When I test the limits of my network
adapters, I send many small UDP packets and compute the average
packets sent per second. Use those codes, I get the result: 75000
packets per second.
(All my sockets has set to unblock)
//////
socket = socket(AF_INET, SOCK_DGRAM, .....)
bind....
time = time(NULL);
while (1)
{
sendto(socket, "", 1, 0, dstaddr, addrlen);
count++;
}
time = time(NULL) - time;
avg = count / time; // here, I get avg = 75000 //////
This result can't satisfy me, so I add another network adapter, and
try the code blow:
//////
socket1 = socket(AF_INET, SOCK_DGRAM, .....)
socket2 = socket(AF_INET, SOCK_DGRAM, .....)
bind socket1.network adapter1...
bind socket2 network adapter2
time = time(NULL);
while (1)
{
sendto(socket1, "", 1, 0, dstaddr1, addrlen);
sendto(socket2, "", 1, 0, dstaddr2, addrlen);
count += 2;
}
time = time(NULL) - time;
avg = count / time;
///////////////////
But I get the result is also 75000 packet per second, WHY?
Then I find a technique name bond, and then I configure my server's
two network adapters into a dev bond0. And then I test again with the
first paragraph of the code, then I get result 150000 packets per
second. So I want to know how bond can increate the speed of network
adapters (use my own code to send with the two network adapters at the
same time is not helpful), then I open the kernel code, and then find
such code:
bond_main.c L3861:
if (IS_UP(slave->dev) &&
(slave->link == BOND_LINK_UP) &&
(slave->state == BOND_STATE_ACTIVE)) {
res = bond_dev_queue_xmit(bond, skb, slave->dev);
write_lock(&bond->curr_slave_lock);
bond->curr_active_slave = slave->next;
write_unlock(&bond->curr_slave_lock);
break;
}
It look like that when send a packet to bond dev, bond use current
slave and send packet, then change current slave to next. What is the
essence different between the bond and my code (use two network
adapters)?
Any suggestions?
xixi
On Fri, 15 Sep 2006, xixi lii wrote:
> This result can't satisfy me, so I add another network adapter, and
> try the code blow:
>
> //////
> socket1 = socket(AF_INET, SOCK_DGRAM, .....)
> socket2 = socket(AF_INET, SOCK_DGRAM, .....)
> bind socket1.network adapter1...
> bind socket2 network adapter2
> time = time(NULL);
> while (1)
> {
> sendto(socket1, "", 1, 0, dstaddr1, addrlen);
> sendto(socket2, "", 1, 0, dstaddr2, addrlen);
> count += 2;
> }
> time = time(NULL) - time;
> avg = count / time;
> ///////////////////
>
> But I get the result is also 75000 packet per second, WHY?
> It look like that when send a packet to bond dev, bond use current
> slave and send packet, then change current slave to next. What is the
> essence different between the bond and my code (use two network
> adapters)?
>
> Any suggestions?
I am not really sure, but I think the bind to an adapter under linux only
chooses the source ip, not really the adapter used to send the packets.
Did you check that the two destination ips have routes through different
interfaces, and not go out through the same one?
(You should even be able to verify this with tcpdump, if you get packets
on one interface and nothing on the second)
c'ya
sven
--
The Internet treats censorship as a routing problem, and routes around it.
(John Gilmore on http://www.cygnus.com/~gnu/)
>> bind socket1.network adapter1...
>> bind socket2 network adapter2
> I am not really sure, but I think the bind to an adapter under linux only
> chooses the source ip, not really the adapter used to send the packets.
To explicitly send things through a specific interface, you need to use
some magic, like PF_RAW. ping for example is one program that will do
this (-I option).
> Did you check that the two destination ips have routes through different
> interfaces, and not go out through the same one?
One cannot have the same subnet on multiple interfaces, because ARP
queries will only be sent through the first one. You need br0 (or bond0
- depending on how you plan to plan your network) to make them one
interface.
Jan Engelhardt
--
2006/9/15, Jan Engelhardt <[email protected]>:
>
> >> bind socket1.network adapter1...
> >> bind socket2 network adapter2
>
> > I am not really sure, but I think the bind to an adapter under linux only
> > chooses the source ip, not really the adapter used to send the packets.
>
> To explicitly send things through a specific interface, you need to use
> some magic, like PF_RAW. ping for example is one program that will do
> this (-I option).
>
> > Did you check that the two destination ips have routes through different
> > interfaces, and not go out through the same one?
>
> One cannot have the same subnet on multiple interfaces, because ARP
> queries will only be sent through the first one. You need br0 (or bond0
> - depending on how you plan to plan your network) to make them one
> interface.
>
>
> Jan Engelhardt
> --
My two adapters has two different IP address, and I bind one IP on one socket,
do you mean that I alloc two socket and bind different IP is not
helpful? In fact, all the packet sent from two socket is go out by one
network adapter?
xixi
It is a question of where you are sending the packets TO, not from. It
is the destination address that is used to route the packet, not the
source. If you were sending the packets to the same destination
address, then they will go out the same interface since the routing
table identifies that interface as the way to reach the destination
address.
xixi lii wrote:
> My two adapters has two different IP address, and I bind one IP on one
> socket,
> do you mean that I alloc two socket and bind different IP is not
> helpful? In fact, all the packet sent from two socket is go out by one
> network adapter?
>
> xixi
> My two adapters has two different IP address, and I bind one IP
> on one socket,
> do you mean that I alloc two socket and bind different IP is not
> helpful?
Correct. You are still sending all the packets *to* the same place.
> In fact, all the packet sent from two socket is go out by one
> network adapter?
Yes, of course. Why would the kernel send traffic to a destination out an
interface that doesn't go to that destination?
Suppose you have two interfaces, 1.2.3.4/8 and 10.2.3.4/8, if you are
sending a packet *to* 1.2.4.5, it will go out the first interface. This
applies whether the source address is 1.2.3.4 or 10.2.3.4.
By default, the kernel routes traffic based on where it is going, not which
interface address it came from.
DS
2006/9/16, David Schwartz <[email protected]>:
>
> > My two adapters has two different IP address, and I bind one IP
> > on one socket,
> > do you mean that I alloc two socket and bind different IP is not
> > helpful?
>
> Correct. You are still sending all the packets *to* the same place.
>
> > In fact, all the packet sent from two socket is go out by one
> > network adapter?
>
> Yes, of course. Why would the kernel send traffic to a destination out an
> interface that doesn't go to that destination?
>
> Suppose you have two interfaces, 1.2.3.4/8 and 10.2.3.4/8, if you are
> sending a packet *to* 1.2.4.5, it will go out the first interface. This
> applies whether the source address is 1.2.3.4 or 10.2.3.4.
>
> By default, the kernel routes traffic based on where it is going, not which
> interface address it came from.
>
> DS
>
Let me explain my network environment, My program is running on a two
adapters machine, whose IP is 192.168.0.1/8 and 192.168.0.2/8, then,
my destination is two machine, whose IP is 192.168.0.3/8 and
192.168.0.4/8. I use four 100M exchange and a 1000M exchange cennected
them to ensure the choke point is not at network equipment.
when I use two socket without bonding, one socket is bind
192.168.0.1/8 and sendto 192.168.0.3/8, the other is bind
192.168.0.2/8 and sendto 192.168.0.4/8, but, as you see, I get a
result that the speed of send by two adapters is equal to the only one
adapter's.
yesterday. I got an uncertain idea, is the problem that IP layer is
separate with Eth layer ? when I bind src IP, it just do helpful to IP
layer, not real bind the adapter? when I send, the real ethreal
adapter is select by IP route? If the two interface can go
destinnation both, IP layer will choose the frist, not use both? Am I
right?
If so, when I use bonding, the adapter's physical address is the frist
one, Do this means that all of the packet come to my machine will go
through in the frist one adapter?
Thx all.
Best Regard.
xixi
> Let me explain my network environment, My program is running on a two
> adapters machine, whose IP is 192.168.0.1/8 and 192.168.0.2/8, then,
> my destination is two machine, whose IP is 192.168.0.3/8 and
> 192.168.0.4/8. I use four 100M exchange and a 1000M exchange cennected
> them to ensure the choke point is not at network equipment.
So both interfaces are part of the same network, and the machines are not
connected to the Internet? (The host ns4.bbn.com is 192.1.122.13, for
example.)
> when I use two socket without bonding, one socket is bind
> 192.168.0.1/8 and sendto 192.168.0.3/8, the other is bind
> 192.168.0.2/8 and sendto 192.168.0.4/8, but, as you see, I get a
> result that the speed of send by two adapters is equal to the only one
> adapter's.
None of your code gives the kernel any reason to prefer one interface over
the other. Why would an interface bound to 192.168.0.1 be better than one
for 192.168.0.2 if you're sending to 192.168.0.3?
> yesterday. I got an uncertain idea, is the problem that IP layer is
> separate with Eth layer ? when I bind src IP, it just do helpful to IP
> layer, not real bind the adapter? when I send, the real ethreal
> adapter is select by IP route? If the two interface can go
> destinnation both, IP layer will choose the frist, not use both? Am I
> right?
Correct, you are binding to the adapter's address, not to the adapter. The
IP routing layer still determines which interface a packet is transmitted
on.
> If so, when I use bonding, the adapter's physical address is the frist
> one, Do this means that all of the packet come to my machine will go
> through in the frist one adapter?
It depends how you have the IP routing layer configured. You can configure
it to select the adapter based on the source address if you want to.
DS
On 9/16/06, xixi lii <[email protected]> wrote:
> 2006/9/16, David Schwartz <[email protected]>:
> >
> > > My two adapters has two different IP address, and I bind one IP
> > > on one socket,
> > > do you mean that I alloc two socket and bind different IP is not
> > > helpful?
> >
> > Correct. You are still sending all the packets *to* the same place.
> >
> > > In fact, all the packet sent from two socket is go out by one
> > > network adapter?
> >
> > Yes, of course. Why would the kernel send traffic to a destination out an
> > interface that doesn't go to that destination?
> >
> > Suppose you have two interfaces, 1.2.3.4/8 and 10.2.3.4/8, if you are
> > sending a packet *to* 1.2.4.5, it will go out the first interface. This
> > applies whether the source address is 1.2.3.4 or 10.2.3.4.
> >
> > By default, the kernel routes traffic based on where it is going, not which
> > interface address it came from.
> >
> > DS
> >
>
> Let me explain my network environment, My program is running on a two
> adapters machine, whose IP is 192.168.0.1/8 and 192.168.0.2/8, then,
> my destination is two machine, whose IP is 192.168.0.3/8 and
> 192.168.0.4/8. I use four 100M exchange and a 1000M exchange cennected
> them to ensure the choke point is not at network equipment.
>
> when I use two socket without bonding, one socket is bind
> 192.168.0.1/8 and sendto 192.168.0.3/8, the other is bind
> 192.168.0.2/8 and sendto 192.168.0.4/8, but, as you see, I get a
> result that the speed of send by two adapters is equal to the only one
> adapter's.
why dont you try a different subnet for the other adapter & its dest.
something like 192.168.1.2/8 & dest 192.168.1.4/8 ?
Because whenever a packet needs to be sent out, the routing table is
consulted. (check out the output of route / netstat -nr ) So for a
given subnet, only the first interface encountered in the table is
used. Remaining interfaces are used, only if the first fails fails.
>
> yesterday. I got an uncertain idea, is the problem that IP layer is
> separate with Eth layer ? when I bind src IP, it just do helpful to IP
> layer, not real bind the adapter? when I send, the real ethreal
> adapter is select by IP route? If the two interface can go
> destinnation both, IP layer will choose the frist, not use both? Am I
> right?
yes, that is what the routing table is about.
> If so, when I use bonding, the adapter's physical address is the frist
> one, Do this means that all of the packet come to my machine will go
> through in the frist one adapter?
No. It just means that the MAC address of the 2 nd card is overridden
with the 1st one's MAC. i.e. all packets sent through 2nd NIC will
bear the 1st MAC. The reason this is done is not to confuse network
equipment which sends out ARP messages etc to resolve IP addresses.
(Switches should be intelligent here, else you will have issues with
recving packets on both interfaces).
>
> Thx all.
> Best Regard.
>
> xixi
Bhanu.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
2006/9/16, David Schwartz <[email protected]>:
>
> > Let me explain my network environment, My program is running on a two
> > adapters machine, whose IP is 192.168.0.1/8 and 192.168.0.2/8, then,
> > my destination is two machine, whose IP is 192.168.0.3/8 and
> > 192.168.0.4/8. I use four 100M exchange and a 1000M exchange cennected
> > them to ensure the choke point is not at network equipment.
>
> So both interfaces are part of the same network, and the machines are not
> connected to the Internet? (The host ns4.bbn.com is 192.1.122.13, for
> example.)
>
> > when I use two socket without bonding, one socket is bind
> > 192.168.0.1/8 and sendto 192.168.0.3/8, the other is bind
> > 192.168.0.2/8 and sendto 192.168.0.4/8, but, as you see, I get a
> > result that the speed of send by two adapters is equal to the only one
> > adapter's.
>
> None of your code gives the kernel any reason to prefer one interface over
> the other. Why would an interface bound to 192.168.0.1 be better than one
> for 192.168.0.2 if you're sending to 192.168.0.3?
>
> > yesterday. I got an uncertain idea, is the problem that IP layer is
> > separate with Eth layer ? when I bind src IP, it just do helpful to IP
> > layer, not real bind the adapter? when I send, the real ethreal
> > adapter is select by IP route? If the two interface can go
> > destinnation both, IP layer will choose the frist, not use both? Am I
> > right?
>
> Correct, you are binding to the adapter's address, not to the adapter. The
> IP routing layer still determines which interface a packet is transmitted
> on.
>
> > If so, when I use bonding, the adapter's physical address is the frist
> > one, Do this means that all of the packet come to my machine will go
> > through in the frist one adapter?
>
> It depends how you have the IP routing layer configured. You can configure
> it to select the adapter based on the source address if you want to.
then how do I configure this?
>
> DS
>
>
>
2006/9/16, Holy Aavu <[email protected]>:
> On 9/16/06, xixi lii <[email protected]> wrote:
> > 2006/9/16, David Schwartz <[email protected]>:
> > >
> > > > My two adapters has two different IP address, and I bind one IP
> > > > on one socket,
> > > > do you mean that I alloc two socket and bind different IP is not
> > > > helpful?
> > >
> > > Correct. You are still sending all the packets *to* the same place.
> > >
> > > > In fact, all the packet sent from two socket is go out by one
> > > > network adapter?
> > >
> > > Yes, of course. Why would the kernel send traffic to a destination out an
> > > interface that doesn't go to that destination?
> > >
> > > Suppose you have two interfaces, 1.2.3.4/8 and 10.2.3.4/8, if you are
> > > sending a packet *to* 1.2.4.5, it will go out the first interface. This
> > > applies whether the source address is 1.2.3.4 or 10.2.3.4.
> > >
> > > By default, the kernel routes traffic based on where it is going, not which
> > > interface address it came from.
> > >
> > > DS
> > >
> >
> > Let me explain my network environment, My program is running on a two
> > adapters machine, whose IP is 192.168.0.1/8 and 192.168.0.2/8, then,
> > my destination is two machine, whose IP is 192.168.0.3/8 and
> > 192.168.0.4/8. I use four 100M exchange and a 1000M exchange cennected
> > them to ensure the choke point is not at network equipment.
> >
> > when I use two socket without bonding, one socket is bind
> > 192.168.0.1/8 and sendto 192.168.0.3/8, the other is bind
> > 192.168.0.2/8 and sendto 192.168.0.4/8, but, as you see, I get a
> > result that the speed of send by two adapters is equal to the only one
> > adapter's.
>
> why dont you try a different subnet for the other adapter & its dest.
> something like 192.168.1.2/8 & dest 192.168.1.4/8 ?
>
> Because whenever a packet needs to be sent out, the routing table is
> consulted. (check out the output of route / netstat -nr ) So for a
> given subnet, only the first interface encountered in the table is
> used. Remaining interfaces are used, only if the first fails fails.
>
> >
> > yesterday. I got an uncertain idea, is the problem that IP layer is
> > separate with Eth layer ? when I bind src IP, it just do helpful to IP
> > layer, not real bind the adapter? when I send, the real ethreal
> > adapter is select by IP route? If the two interface can go
> > destinnation both, IP layer will choose the frist, not use both? Am I
> > right?
>
> yes, that is what the routing table is about.
>
> > If so, when I use bonding, the adapter's physical address is the frist
> > one, Do this means that all of the packet come to my machine will go
> > through in the frist one adapter?
>
> No. It just means that the MAC address of the 2 nd card is overridden
> with the 1st one's MAC. i.e. all packets sent through 2nd NIC will
> bear the 1st MAC. The reason this is done is not to confuse network
> equipment which sends out ARP messages etc to resolve IP addresses.
> (Switches should be intelligent here, else you will have issues with
> recving packets on both interfaces).
>
Sorry, I still don't understand how can I recv packets on both
interfaces, Since the two adapters have the same MAC address, the
packet will be the same in the field of ethreal protocol, Do you means
that the switch will choose the two adapters with a balance policy, or
I must do configure on switch?
xixi