I observe that UDP sockets listening on ANY
send response packets with ip addr derived from
ip address of interface which is used to send 'em
instead of using dst ip address of client's packet.
I was bitten by this with DNS and NTP.
This is a test setup:
{net 172.16/16}----- 172.16.22.2 [multihomed] 195.66.192.167 ----....
This is a UDP server on multihomed box:
# echo test | nc -vvvnu -l -p 222
listening on [any] 222 ...
connect to [172.16.22.2] from (UNKNOWN) [172.16.42.177] 333
ping
sent 5, rcvd 5 : Connection refused
#
This is client on 172.16.42.177:
# echo ping | nc -vvvnu -p 333 195.66.192.167 222
(UNKNOWN) [195.66.192.167] 222 (?) open
it waits forever, never getting 'test' string from server.
This is a tcpdump of this event:
13:58:00.555207 172.16.42.177.333 > 195.66.192.167.222: udp 5 (DF)
13:58:00.559849 172.16.22.2.222 > 172.16.42.177.333: udp 5 (DF)
^^^^^^^^^^^
13:58:00.560457 172.16.42.177 > 172.16.22.2: icmp: 172.16.42.177 udp port 333 unreachable [tos 0xc0]
This is why 172.16.42.177 thinks that port 333 is unreachable:
# lsof -nP | grep 333
ntpd 418 root mem REG 3,4 133316 68184 /.share/usr/lib/all-lib/libreadline.so.3.0
most 7559 root txt REG 3,4 46828 24333 /.share/usr/app/most-4.9.2/bin/most
nc 15085 root 3u IPv4 29381 UDP 172.16.42.177:333->195.66.192.167:222
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kernel does not think that 172.16.22.2.222 > 172.16.42.177.333 packet
belong to this socket, and it is right imo.
Test works ok if I remove -u from both netcats (i.e. TCP works).
client:
# uname -a
Linux firebird 2.6.6-rc1-bk1 #1 Wed May 26 11:03:11 EEST 2004 i686 unknown
server:
# uname -a
Linux oldie 2.4.22-rc2_QoS #1 Tue Nov 18 15:55:00 GMT-2 2003 i586 unknown
root@oldie:/.share/home/vda#
I saw this behavior on 2.6 based machines as well.
--
vda
Hello,
On Wed, 9 Jun 2004, Denis Vlasenko wrote:
> I observe that UDP sockets listening on ANY
> send response packets with ip addr derived from
> ip address of interface which is used to send 'em
> instead of using dst ip address of client's packet.
>
> I was bitten by this with DNS and NTP.
The problem is in the apps. You have some options:
- use sockets listening on ANY and using sendmsg with IP_PKTINFO
and providing the desired src IP in ipi_spec_dst
- you can also create many listener sockets listening on
particular src IP and to reply using the socket where the
request is received, because the kernel does not know any
association between request and reply packets if the listener
is bound to ANY.
Regards
--
Julian Anastasov <[email protected]>
In article <[email protected]> (at Wed, 9 Jun 2004 14:25:39 +0300), Denis Vlasenko <[email protected]> says:
> I observe that UDP sockets listening on ANY
> send response packets with ip addr derived from
> ip address of interface which is used to send 'em
> instead of using dst ip address of client's packet.
use IP_PKTINFO when responding the client.
--yoshfuji
On Wed, Jun 09, 2004 at 02:25:39PM +0300, Denis Vlasenko wrote:
> I observe that UDP sockets listening on ANY
> send response packets with ip addr derived from
> ip address of interface which is used to send 'em
> instead of using dst ip address of client's packet.
>
> I was bitten by this with DNS and NTP.
This is the responsibility of the program. Unless the UDP packet is
bound to a particular address, or the program specifies an IP, the
kernel will pick one. For this reason both the BIND and NTP daemons
open a socket for each interface so they can control this. netstat on
my machine shows:
udp 0 0 192.168.1.225:123 0.0.0.0:*
udp 0 0 127.0.0.1:123 0.0.0.0:*
udp 0 0 0.0.0.0:123 0.0.0.0:*
for the NTP server. The DNS has similar machanism. Remember, UDP
doesn't involve connections, so there is no concept of "replying" to a
packet, the program has to manage that itself.
In your example, if you tell netcat which address to bind to, it will
work.
Hope this helps,
--
Martijn van Oosterhout <[email protected]> http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.
On Wednesday 09 June 2004 15:18, Julian Anastasov wrote:
> Hello,
>
> On Wed, 9 Jun 2004, Denis Vlasenko wrote:
> > I observe that UDP sockets listening on ANY
> > send response packets with ip addr derived from
> > ip address of interface which is used to send 'em
> > instead of using dst ip address of client's packet.
> >
> > I was bitten by this with DNS and NTP.
>
> The problem is in the apps. You have some options:
>
> - use sockets listening on ANY and using sendmsg with IP_PKTINFO
> and providing the desired src IP in ipi_spec_dst
>
> - you can also create many listener sockets listening on
> particular src IP and to reply using the socket where the
> request is received, because the kernel does not know any
> association between request and reply packets if the listener
> is bound to ANY.
Aha! That's why ntpd is so eager to bind to everything imaginable!
(Which does not help BTW with non-permanent interfaces like
ppp, tun etc).
/me googling for that IP_PKTINFO thing...
--
vda
On Wednesday 09 June 2004 15:24, YOSHIFUJI Hideaki wrote:
> Denis Vlasenko <[email protected]> says:
> > I observe that UDP sockets listening on ANY
> > send response packets with ip addr derived from
> > ip address of interface which is used to send 'em
> > instead of using dst ip address of client's packet.
>
> use IP_PKTINFO when responding the client.
Thanks!
With your help and some googling I've found and adapted
code to get dst ip of UDP packet.
Small test program successfully ran and reported correct
dst addresses of incoming UDP packets.
Now, I am trying to fix (or shall I say 'improve'?) dnscache.
You may find some code below my sig. It's a start.
The problem is, how to _send replies_ with correct src ip?
I can bind a temporary socket to needed src address,
do a sendto(), then close socket. This will work,
but this can introduce a race - any incoming
packet to this (ip,port) will inadvertently
be classified as belonging to temp socket!
This is going to be a nasty bug, manifesting
itself only under load.
I looked into sendmsg(). Looks like ther is no way to
indicate source ip.
Shall I use some other technique?
--
vda
#if defined IP_RECVDSTADDR
# define DSTADDR_SOCKOPT IP_RECVDSTADDR
# define DSTADDR_DATASIZE (CMSG_SPACE(sizeof(struct in_addr)))
# define dstaddr(x) (CMSG_DATA(x))
#elif defined IP_PKTINFO
# define DSTADDR_SOCKOPT IP_PKTINFO
# define DSTADDR_DATASIZE (CMSG_SPACE(sizeof(struct in_pktinfo)))
# define dstaddr(x) (&(((struct in_pktinfo *)(CMSG_DATA(x)))->ipi_addr))
#else
# error "can't determine socket option"
#endif
int socket_recv4_dst(int s,char *buf,int len,char ip[4],uint16 *port, char ipdst[4])
{
int r;
struct iovec iov[1];
struct sockaddr_in sa;
union control_data cmsg;
struct cmsghdr *cmsgptr;
struct msghdr msg;
iov[0].iov_base = buf;
iov[0].iov_len = len;
memset(&msg, 0, sizeof msg);
msg.msg_name = &sa;
msg.msg_namelen = sizeof sa;
msg.msg_iov = iov;
msg.msg_iovlen = 1;
msg.msg_control = &cmsg;
msg.msg_controllen = sizeof cmsg;
{ // FIXME: we need to do it ONCE! move it into socket_bind4_dstaddropt()
int sockopt;
sockopt = 1;
if (setsockopt(s, IPPROTO_IP, DSTADDR_SOCKOPT, &sockopt, sizeof sockopt) == -1)
return -1;
}
//r = recvfrom(s,buf,len,0,(struct sockaddr *) &sa,&dummy);
r = recvmsg(s, &msg, 0);
if (r == -1) return -1;
// Here we retrieve destination IP and memorize it
for (cmsgptr = CMSG_FIRSTHDR(&msg);
cmsgptr != NULL;
cmsgptr = CMSG_NXTHDR(&msg, cmsgptr)) {
if (cmsgptr->cmsg_level == IPPROTO_IP
&& cmsgptr->cmsg_type == DSTADDR_SOCKOPT) {
byte_copy(ipdst,4,(char *) dstaddr(cmsgptr));
}
}
return r;
}
Denis Vlasenko <[email protected]> wrote:
>
> I looked into sendmsg(). Looks like ther is no way to
> indicate source ip.
>
> Shall I use some other technique?
IP_PKTINFO works just as well there. Look at the ip_cmsg_send call
in udp_sendmsg.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Friday 11 June 2004 13:34, Herbert Xu wrote:
> Denis Vlasenko <[email protected]> wrote:
> > I looked into sendmsg(). Looks like ther is no way to
> > indicate source ip.
> >
> > Shall I use some other technique?
>
> IP_PKTINFO works just as well there. Look at the ip_cmsg_send call
> in udp_sendmsg.
int udp_sendmsg(...)
{
...
ipc.addr = inet->saddr;
ipc.oif = sk->sk_bound_dev_if;
if (msg->msg_controllen) {
err = ip_cmsg_send(msg, &ipc);
if (err)
return err;
if (ipc.opt)
free = 1;
connected = 0;
}
if (!ipc.opt)
ipc.opt = inet->opt;
saddr = ipc.addr;
...
int ip_cmsg_send(struct msghdr *msg, struct ipcm_cookie *ipc)
{
...
case IP_PKTINFO:
{
struct in_pktinfo *info;
if (cmsg->cmsg_len != CMSG_LEN(sizeof(struct in_pktinfo)))
return -EINVAL;
info = (struct in_pktinfo *)CMSG_DATA(cmsg);
ipc->oif = info->ipi_ifindex;
ipc->addr = info->ipi_spec_dst.s_addr;
manpage:
IP_PKTINFO
Pass an IP_PKTINFO ancillary message that contains a pktinfo structure that supplies some information about the
incoming packet. This only works for datagram oriented sockets.
struct in_pktinfo
{
unsigned int ipi_ifindex; /* Interface index */
struct in_addr ipi_spec_dst; /* Routing destination address */
struct in_addr ipi_addr; /* Header Destination address */
};
Hmmm... do I have to set a *routing dest address* field
to set src ip address of my UDP packet?
--
vda
Hello,
On Fri, 11 Jun 2004, Denis Vlasenko wrote:
> Hmmm... do I have to set a *routing dest address* field
> to set src ip address of my UDP packet?
Try such function:
static int my_send(int fd, unsigned srcip, (struct sockaddr *) remote,
char *data, int len)
{
struct iovec iov = { data, len };
struct {
struct cmsghdr cm;
struct in_pktinfo ipi;
} cmsg = {
.cm = {
.cmsg_len = sizeof(struct cmsghdr) +
sizeof(struct in_pktinfo),
.cmsg_level = SOL_IP,
.cmsg_type = IP_PKTINFO,
},
.ipi = {
.ipi_ifindex = 0,
.ipi_spec_dst = srcip,
},
};
struct msghdr m = {
.msg_name = remote,
.msg_namelen = sizeof(struct sockaddr_in),
.msg_iov = &iov,
.msg_iovlen = 1,
.msg_control = &cmsg,
.msg_controllen = sizeof(cmsg),
.msg_flags = 0,
};
return sendmsg(fd, &m, MSG_NOSIGNAL|MSG_DONTWAIT);
}
Regards
--
Julian Anastasov <[email protected]>