* Summary:
While traversing packets through Netfilter, changing dest address from a foreign to a local address causes the packet to drop (and show up at ip_rt_bug(), along a syslog entry).
* Description:
I'm using libipq/ip_tables and ip_queue to trap packets to a userspace VPN product, using nothing but standard kernel modules (and my own VPN proxy app).
The packets flowing into or out of the machine get diverted to a userspace application, src/dest addresses are modified, and injected back into the IP stack.
For example, an outgoing packet (that has a foreign dest addr) is overridden with a local dest address, hoping it would end up at the local VPN listener.
Under kernel 2.4 this works fine.
In 2.6 it breaks.
* Details:
An outgoing packet (has a non-local dest addr) is queued and recognized at the ip_queue userspace app. Its dest addr+port are set to that of the local machine (to get to my userspace VPN app).
The modified packet is marked NF_ACCEPT and sent back into the kernel, but ends up at the ip_rt_bug function (with a syslog entry).
* Assumed bug analysis:
Due to the destination address change, the packet needed to go through routing once again, since it's no longer an outgoing packet.
This does happen in the ip_route_me_harder function, which sets the dst->output to point at ip_rt_bug.
Since this was an outgoing packet (in the NF_IP_LOCAL_OUT chain), the final operation done on the packet is calling the *okfn function, which points to dst->output which is ip_rt_bug.
I would have expected the routing function to realize it needs to re-evaluate the route, and set the *okfn to dst->input instead.
* Kernel version:
2.6.9-prep, (Red Hat 3.4.2-6.fc3) compiled locally with no modifications.
Please advise (and please CC "YAIR at ARX.COM")
A similar problem has been reported a while back but never replied (http://groups-beta.google.com/group/linux.kernel/msg/455c04e17e354d04?dmode=source&hl=en)
Yair
[NETFILTER]: Don't use ip_route_input() for local addresses
Local input routes have ->output set to ip_rt_bug().
Signed-off-by: Patrick McHardy <[email protected]>
---
commit bef30866b7440f4c8aff99dc025ea99b8d396390
tree a469360c577fdf6919b9a771521eca120103db45
parent a5c2178d8f07f6180a2daf8df4524cf3b45e62ed
author Patrick McHardy <[email protected]> 1114419959 +0200
committer Patrick McHardy <[email protected]> 1114419959 +0200
Index: net/core/netfilter.c
===================================================================
--- 70652aa8f30bea3ea83594cc4a47a11f7a8db89d/net/core/netfilter.c (mode:100644 sha1:e51cfa46950cf8f1f4dea42be94e71d76d8c3c5b)
+++ a469360c577fdf6919b9a771521eca120103db45/net/core/netfilter.c (mode:100644 sha1:85936a0b23d9ea42e2cd9d45e8254c2f780eb786)
@@ -611,7 +611,8 @@
/* some non-standard hacks like ipt_REJECT.c:send_reset() can cause
* packets with foreign saddr to appear on the NF_IP_LOCAL_OUT hook.
*/
- if (inet_addr_type(iph->saddr) == RTN_LOCAL) {
+ if (inet_addr_type(iph->saddr) == RTN_LOCAL ||
+ inet_addr_type(iph->daddr) == RTN_LOCAL) {
fl.nl_u.ip4_u.daddr = iph->daddr;
fl.nl_u.ip4_u.saddr = iph->saddr;
fl.nl_u.ip4_u.tos = RT_TOS(iph->tos);
Patrick McHardy <[email protected]> wrote:
> --- 70652aa8f30bea3ea83594cc4a47a11f7a8db89d/net/core/netfilter.c (mode:100644 sha1:e51cfa46950cf8f1f4dea42be94e71d76d8c3c5b)
> +++ a469360c577fdf6919b9a771521eca120103db45/net/core/netfilter.c (mode:100644 sha1:85936a0b23d9ea42e2cd9d45e8254c2f780eb786)
> @@ -611,7 +611,8 @@
> /* some non-standard hacks like ipt_REJECT.c:send_reset() can cause
> * packets with foreign saddr to appear on the NF_IP_LOCAL_OUT hook.
> */
> - if (inet_addr_type(iph->saddr) == RTN_LOCAL) {
> + if (inet_addr_type(iph->saddr) == RTN_LOCAL ||
> + inet_addr_type(iph->daddr) == RTN_LOCAL) {
> fl.nl_u.ip4_u.daddr = iph->daddr;
> fl.nl_u.ip4_u.saddr = iph->saddr;
> fl.nl_u.ip4_u.tos = RT_TOS(iph->tos);
You'll still BUG if the destination is multicast/broadcast. I'm also
unsure whether ipt_REJECT isn't susceptible to the same problem.
Luckily ipt_MIRROR is no more :)
What are the issues with getting rid of the ip_route_input path
altogether?
The only thing we gain over calling ip_route_output is the ability
to set the input device. But even that is just a fake derived from
the source address in a deterministic way. Therefore any effects
achievable through using ip_route_input can also be done by simply
reconfiguring the policy routing table to look at the local source
addresses instead of (or in conjunction with) the input device.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert Xu wrote:
> You'll still BUG if the destination is multicast/broadcast. I'm also
> unsure whether ipt_REJECT isn't susceptible to the same problem.
> Luckily ipt_MIRROR is no more :)
ipt_REJECT is fine, ip_route_input() is only used in NF_IP_FORWARD.
But you're right about multicast/broadcast still causing problems,
I'll have another look.
> What are the issues with getting rid of the ip_route_input path
> altogether?
>
> The only thing we gain over calling ip_route_output is the ability
> to set the input device. But even that is just a fake derived from
> the source address in a deterministic way. Therefore any effects
> achievable through using ip_route_input can also be done by simply
> reconfiguring the policy routing table to look at the local source
> addresses instead of (or in conjunction with) the input device.
No, ip_route_me_harder() can be called for packets with non-local
source. ip_route_output_slow() rejects non-local source addresses,
so the only way to use them for policy routing is by using
ip_route_input().
Regards
Patrick
No, this does not help.
The failure is now inside "ip_route_output_slow", since the reversed address packet has the source address of the remote machine, and the call to "ip_dev_find" fails to find a device with matching address.
In 2.4 the okfn pointer used to point to "ip_queue_xmit2" which evaluated the new route from scratch.
It was passed in as the completion function when calling the NF_HOOK chain.
In 2.6 this function is gone (replaced with a reference to "dst_output).
Was removing it a mistake?
-----Original Message-----
From: Patrick McHardy [mailto:[email protected]]
Sent: Monday, April 25, 2005 11:07
To: Yair Itzhaki
Cc: [email protected]; Netfilter Development Mailinglist
Subject: Re: Re-routing packets via netfilter (ip_rt_bug)
Yair Itzhaki wrote:
> While traversing packets through Netfilter, changing dest address from a foreign to a local address causes the packet to drop (and show up at ip_rt_bug(), along a syslog entry).
Does this patch fix your problem?
On Mon, Apr 25, 2005 at 05:28:57PM +0200, Patrick McHardy wrote:
>
> No, ip_route_me_harder() can be called for packets with non-local
> source. ip_route_output_slow() rejects non-local source addresses,
> so the only way to use them for policy routing is by using
> ip_route_input().
You're right. But then we can't call ip_route_output in the case
where saddr is foreign but daddr is local. Nor can we call
ip_route_input since the output will be ip_rt_bug.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert Xu wrote:
> You're right. But then we can't call ip_route_output in the case
> where saddr is foreign but daddr is local. Nor can we call
> ip_route_input since the output will be ip_rt_bug.
In that case we need to use saddr=0, which shouldn't make any difference
with sane routing.
Regards
Patrick
On Tue, Apr 26, 2005 at 02:08:18AM +0200, Patrick McHardy wrote:
> Herbert Xu wrote:
> >You're right. But then we can't call ip_route_output in the case
> >where saddr is foreign but daddr is local. Nor can we call
> >ip_route_input since the output will be ip_rt_bug.
>
> In that case we need to use saddr=0, which shouldn't make any difference
> with sane routing.
Makes sense. But what about the case where saddr is foreign but
daddr is broadcast/multicast?
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert Xu wrote:
> Makes sense. But what about the case where saddr is foreign but
> daddr is broadcast/multicast?
Looks like we have no choice but to also use saddr=0 and
ip_route_output() in this case.
Regards
Patrick
I'm afraid I'm not following you.
Where did you want to set saddr=0 ?
Yair
> -----Original Message-----
> From: Herbert Xu [mailto:[email protected]]
> Sent: Tuesday, April 26, 2005 02:39
> To: Patrick McHardy
> Cc: Yair Itzhaki; [email protected];
> [email protected]; [email protected]
> Subject: Re: Re-routing packets via netfilter (ip_rt_bug)
>
>
> On Tue, Apr 26, 2005 at 02:08:18AM +0200, Patrick McHardy wrote:
> > Herbert Xu wrote:
> > >You're right. But then we can't call ip_route_output in the case
> > >where saddr is foreign but daddr is local. Nor can we call
> > >ip_route_input since the output will be ip_rt_bug.
> >
> > In that case we need to use saddr=0, which shouldn't make
> any difference
> > with sane routing.
>
> Makes sense. But what about the case where saddr is foreign but
> daddr is broadcast/multicast?
>
> Cheers,
> --
> Visit Openswan at http://www.openswan.org/
> Email: Herbert Xu ~{PmV>HI~} <[email protected]>
> Home Page: http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
>
On Tue, Apr 26, 2005 at 03:17:27PM +0200, Patrick McHardy wrote:
>
> Looks like we have no choice but to also use saddr=0 and
> ip_route_output() in this case.
Let's look at the bigger picture. There are three users of
ip_route_me_harder: nat, mangle and queue. They're all done
in LOCAL_OUT.
For nat/mangle, the source address cannot change so it's
guaranteed to be a local IP address. On the face of it,
queue could be changing the source address. However, the
more I think about it the more I reckon that it should
be disallowed.
If the user is changing the source address in LOCAL_OUT/queue
then he's doing SNAT in LOCAL_OUT. This violates some fundamental
assumptions in netfilter. The user also isn't going to have
an easy time setting up the reverse DNAT since the corresponding
location on the reverse side does not have a ip_route_me_harder
call.
Even if there is really a demand for SNAT in LOCAL_OUT, we
should probably be implementing it properly rather than having
the user craft their own in user-space.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert Xu wrote:
> Let's look at the bigger picture. There are three users of
> ip_route_me_harder: nat, mangle and queue. They're all done
> in LOCAL_OUT.
>
> For nat/mangle, the source address cannot change so it's
> guaranteed to be a local IP address. On the face of it,
> queue could be changing the source address. However, the
> more I think about it the more I reckon that it should
> be disallowed.
The ipt_REJECT target can send TCP RSTs with foreign source which
go through LOCAL_OUT. Restricting it to this case and adding proper
checks to ipt_REJECT would relieve us of having to handle the last
case you pointed out (foreign saddr, broadcast/multicast daddr), but
it shouldn't be hard to also handle this case.
> If the user is changing the source address in LOCAL_OUT/queue
> then he's doing SNAT in LOCAL_OUT. This violates some fundamental
> assumptions in netfilter. The user also isn't going to have
> an easy time setting up the reverse DNAT since the corresponding
> location on the reverse side does not have a ip_route_me_harder
> call.
These assumptions are only for stateful NAT, the mangle table seems
to try to deal with stateless NAT by rerouting in LOCAL_OUT when
saddr/daddr changed. But it could also just be some left-over
cut-n-pasted from ip_nat_standalone.c, I don't think anyone is doing
stateless NAT with netfilter.
Regards
Patrick
On Wed, Apr 27, 2005 at 02:56:48AM +0200, Patrick McHardy wrote:
>
> The ipt_REJECT target can send TCP RSTs with foreign source which
> go through LOCAL_OUT. Restricting it to this case and adding proper
Couldn't we feed the TCP RST packets with foreign sources through
the FORWARD table? We're lying to the routing system already by
telling it that the packet is forwarded. So I don't see anything
wrong with lying to netfilter as well :)
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert Xu wrote:
> Couldn't we feed the TCP RST packets with foreign sources through
> the FORWARD table? We're lying to the routing system already by
> telling it that the packet is forwarded. So I don't see anything
> wrong with lying to netfilter as well :)
Forwarded packets can't have any NAT manips in LOCAL_OUT, so it
should work. I'm not sure about it though because it would be
the only place where packets just appear in FORWARD, usually
all packets enters through PRE_ROUTING or LOCAL_OUT.
Regards
Patrick
On Wed, Apr 27, 2005 at 12:26:13PM +0200, Patrick McHardy wrote:
>
> Forwarded packets can't have any NAT manips in LOCAL_OUT, so it
> should work. I'm not sure about it though because it would be
> the only place where packets just appear in FORWARD, usually
> all packets enters through PRE_ROUTING or LOCAL_OUT.
It's also the only place where we generate a packet with a non-local
source address :)
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Wed, 27 Apr 2005, Herbert Xu wrote:
> On Wed, Apr 27, 2005 at 12:26:13PM +0200, Patrick McHardy wrote:
> >
> > Forwarded packets can't have any NAT manips in LOCAL_OUT, so it
> > should work. I'm not sure about it though because it would be
> > the only place where packets just appear in FORWARD, usually
> > all packets enters through PRE_ROUTING or LOCAL_OUT.
>
> It's also the only place where we generate a packet with a non-local
> source address :)
Besides the REJECT target, TARPIT in patch-o-matic-ng also generates
packets with non-local source addresses. We cannot assume that REJECT is
the only one which can create such packets.
Best regards,
Jozsef
-
E-mail : [email protected], [email protected]
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
On Wed, Apr 27, 2005 at 12:41:01PM +0200, Jozsef Kadlecsik wrote:
> > >
> > > Forwarded packets can't have any NAT manips in LOCAL_OUT, so it
> > > should work. I'm not sure about it though because it would be
> > > the only place where packets just appear in FORWARD, usually
> > > all packets enters through PRE_ROUTING or LOCAL_OUT.
> >
> > It's also the only place where we generate a packet with a non-local
> > source address :)
>
> Besides the REJECT target, TARPIT in patch-o-matic-ng also generates
> packets with non-local source addresses. We cannot assume that REJECT is
> the only one which can create such packets.
Any reason why it can't be fed through the FORWARD chain as opposed
to LOCAL_OUT? In general, is there anything that's generating packets
with foreign addresses that can't be fed through FORWARD?
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Wed, Apr 27, 2005 at 09:35:42PM +1000, herbert wrote:
>
> > Besides the REJECT target, TARPIT in patch-o-matic-ng also generates
> > packets with non-local source addresses. We cannot assume that REJECT is
> > the only one which can create such packets.
>
> Any reason why it can't be fed through the FORWARD chain as opposed
> to LOCAL_OUT? In general, is there anything that's generating packets
> with foreign addresses that can't be fed through FORWARD?
Here is another reason why these packets should go through FORWARD.
They were generated in response to packets in INPUT/FORWARD/OUTPUT.
The original packet has not undergone SNAT in any of these cases.
However, if we feed the response packet through LOCAL_OUT it will
be subject to DNAT. This creates a NAT asymmetry and we may end
up with the wrong destination address.
By pushing it through FORWARD it will only undergo SNAT which is
correct since the original packet would have undergone DNAT.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert Xu wrote:
> Here is another reason why these packets should go through FORWARD.
> They were generated in response to packets in INPUT/FORWARD/OUTPUT.
> The original packet has not undergone SNAT in any of these cases.
>
> However, if we feed the response packet through LOCAL_OUT it will
> be subject to DNAT. This creates a NAT asymmetry and we may end
> up with the wrong destination address.
>
> By pushing it through FORWARD it will only undergo SNAT which is
> correct since the original packet would have undergone DNAT.
This is only a problem since the recent NAT changes, but I agree
that we should fix it by moving these packets to FORWARD.
Regards
Patrick
Can anyone propose a patch that I can start checking?
I have come up with the following:
--- net/core/netfilter.c.orig 2005-04-18 21:55:30.000000000 +0300
+++ net/core/netfilter.c 2005-05-02 17:35:20.000000000 +0300
@@ -622,9 +622,10 @@
/* some non-standard hacks like ipt_REJECT.c:send_reset() can cause
* packets with foreign saddr to appear on the NF_IP_LOCAL_OUT hook.
*/
- if (inet_addr_type(iph->saddr) == RTN_LOCAL) {
+ if ((inet_addr_type(iph->saddr) == RTN_LOCAL) ||
+ (inet_addr_type(iph->daddr) == RTN_LOCAL)) {
fl.nl_u.ip4_u.daddr = iph->daddr;
- fl.nl_u.ip4_u.saddr = iph->saddr;
+ fl.nl_u.ip4_u.saddr = 0;
fl.nl_u.ip4_u.tos = RT_TOS(iph->tos);
fl.oif = (*pskb)->sk ? (*pskb)->sk->sk_bound_dev_if : 0;
#ifdef CONFIG_IP_ROUTE_FWMARK
Please advise,
Yair
> -----Original Message-----
> From: Patrick McHardy [mailto:[email protected]]
> Sent: Wednesday, April 27, 2005 14:05
> To: Herbert Xu
> Cc: Jozsef Kadlecsik; [email protected];
> [email protected]; Yair Itzhaki;
> [email protected]
> Subject: Re: Re-routing packets via netfilter (ip_rt_bug)
>
>
> Herbert Xu wrote:
> > Here is another reason why these packets should go through FORWARD.
> > They were generated in response to packets in INPUT/FORWARD/OUTPUT.
> > The original packet has not undergone SNAT in any of these cases.
> >
> > However, if we feed the response packet through LOCAL_OUT it will
> > be subject to DNAT. This creates a NAT asymmetry and we may end
> > up with the wrong destination address.
> >
> > By pushing it through FORWARD it will only undergo SNAT which is
> > correct since the original packet would have undergone DNAT.
>
> This is only a problem since the recent NAT changes, but I agree
> that we should fix it by moving these packets to FORWARD.
>
> Regards
> Patrick
>
I have been using free VPN but it is not that good because network is always
congested which leads to slower connection. I just finished reading
expressvpn review <https://www.reviewsdir.com/expressvpn-review/> and it
seems like a good option. Does anyone here have experience with this VPN? If
yes, please share.
--
View this message in context: http://linux-kernel.2935.n7.nabble.com/Re-routing-packets-via-netfilter-ip-rt-bug-tp435p1381848.html
Sent from the Linux Kernel mailing list archive at Nabble.com.