Hi kernel developers:
[I'm am not subscribed to netfilter-devel, so please Cc: linux-kernel]
I am experiencing some problems with FTP connection tracking in recent
2.4.x kernels. In short, it seems ip_conntrack_ftp misses some
legitimate packets belonging to already stablished connections, and
sometimes FTP sessions don't work. Of course, this is NATing outgoing
traffic, including FTP. A more complete explanation follows.
A Linux machine (tried 2.4.17 and 2.4.18-pre9) acting as a router/NAT
inconditionally does SNAT on all traffic leaving some interface:
iptables -t nat -A POSTROUTING --out-interface eth2 --jump SNAT \
--to-source $PUBLIC_ADDRESS
As we don't want to leave the box open to external port scan, but we do
want to let the machine connect to the Internet to grab security updates
from time to time, we allow all outgoing traffic through "eth2" but just
allow related traffic in the opposite direction:
iptables -t filter -A OUTPUT --jump ACCEPT
iptables -t filter -A INPUT --in-interface eth2 --match state \
--state RELATED,ESTABLISHED --jump ACCEPT
iptables -t filter -A INPUT --in-interface eth2 --match state \
--state NEW,INVALID --jump DROP
So, as we manually (well, our script :) install all required netfilter
modules into memory, we expected most protocols to work, including FTP
(ip_conntrack, ip_conntrack_ftp, ip_nat_ftp among others are in memory).
And in fact, most of the time every "supported" protocol work fine.
However, some combinations of FTP client and remote FTP server fail.
Starting with iptables counters reset, we try to connect to some remote
FTP server, and the connection isn't fully established. With no other
network traffic going on we see packets matching the DROP rule. Logging
those packets to syslog (via --jump LOG) shows they are packets from the
FTP connection. This happens both with active and passive transfer modes.
We tried to ACCEPT INVALID traffic, with no luck: packets continue to be
dropped, so to FTP connection tracking they appear as NEW instead of
somewhat related to the ongoing FTP session.
The strange part is that this will show up with just some FTP clients
and/or remote FTP servers. For example, text-mode web browser "links"
doesn't work in any case. On the other hand, text-mode browser "lynx"
works nice in the same places where "links" fails. "wget", "ftp" and
"ncftp" show various degrees of success with several combinations of
active/passive transfer mode and remote FTP servers (tried with
ftp.kernel.org, ftp.at.kernel.org, ftp.mozilla.org, ftp.rediris.es,
ftp.sourceforge.net).
For example, "ftp.rediris.es" won't even show the greeting message with
"links", but will with for example "ncftp". However, trying to download
some file from the server, stalls as soon as some kilobytes of data have
been received (in the order of 8 to 10 KB).
If we configure INPUT to not discard packets, and just ACCEPT all that
comes in solves the proble. ECN is not compiled in nor activated. The rest
of network-related kernel tunnable parameters have default values.
Searching the Internet for information took me to a messages on
netfilter-devel mailing list, namely:
http://lists.samba.org/pipermail/netfilter-devel/2001-October/002433.html
It contains a patch for a problem very similar to what I am
experiencing, and seems to be related to the fact that some FTP clients
end their FTP commands with just \r instead of \r\n, and viceversa. A
"ethereal" packet capture in a "clean" (no firewall rules) state with
the FTP clients described above show that all of them follow RFC959 and
terminate FTP commands with "\r\n".
I compiled the same kernel (2.4.18-pre9 in this case) with the patch
applied, and repeated some of the test described above with some degree
of success. Some combinations of FTP client and remote FTP servers now
work OK, but some other combinations continue failing. As previously
said, in absence of (DROP) firewall rules, everything works fine. We
checked the possibility of our ISP messing the packets up, but some
tests limited to our LAN show the same problems.
NOTE: "links" author says in the manual page:
BUGS
Can't connect to some FTP servers (Novell, NT).
Connection stays in "Request sent" state.
So in this particular case, the application could also have problems.
I am willing to perform as much tests as you need to help discover what
is going on, so if I missed something important, please ask.
Keep up the good work !
--
Jos? Luis Domingo L?pez
Linux Registered User #189436 Debian Linux Woody (P166 64 MB RAM)
jdomingo AT internautas DOT org => Spam at your own risk
On Tue, Feb 12, 2002 at 11:12:39PM +0100, Jose Luis Domingo Lopez wrote:
> The strange part is that this will show up with just some FTP clients
> and/or remote FTP servers. For example, text-mode web browser "links"
> doesn't work in any case. On the other hand, text-mode browser "lynx"
> works nice in the same places where "links" fails. "wget", "ftp" and
> "ncftp" show various degrees of success with several combinations of
> active/passive transfer mode and remote FTP servers (tried with
> ftp.kernel.org, ftp.at.kernel.org, ftp.mozilla.org, ftp.rediris.es,
> ftp.sourceforge.net).
FTP connection tracking generally works very well. It's widely deployed
and in most scenarios it is working fine.
The "links" browser is violating the RFC's by not waiting for server
responses after sending PORT commands to the server. This is a problem
with the browser, and it should be fixed on the browser side. I doubt
it works well with other ftp masquerading/nat solutions.
I'm a heavy user of wget / ftp / mirror / ncftp, and I've never experienced
any of the problems you describe. Please give me exact ftp server hostnames
where I can try to reproduce your problem.
> For example, "ftp.rediris.es" won't even show the greeting message with
> "links", but will with for example "ncftp". However, trying to download
> some file from the server, stalls as soon as some kilobytes of data have
> been received (in the order of 8 to 10 KB).
As I said, forget about links. Use ftp or ncftp and you can be sure they
comply to the RFC's.
Some other notes:
- greeting messages are part of the control connection which is not
really touched by conntrack.
- I can not imagine how the ftp conntrack helper should cause stalls in
file transfers. Are you 100% sure this is caused by netfilter? Do
you see packet drops/ delays when comparing incoming/outgoing packets
on your firewall? What happens without ip_conntrack_ftp? Stalls could
be network problems in general or bandwith throttling on the server.
- I have just successfully downloaded about 50MB from ftp.rediris.es
through a netfilter conntrack+nat box, I get constant transfer rates of
about 700kbits/sec on a 1mbit internet uplink.
> If we configure INPUT to not discard packets, and just ACCEPT all that
> comes in solves the proble. ECN is not compiled in nor activated. The rest
> of network-related kernel tunnable parameters have default values.
how are your input rules related to forwarded/NAT'ed traffic through
the firewall? INPUT rules are for the local machine only. Or are you
running some ftp/http proxy like squid on the local box?
> I am willing to perform as much tests as you need to help discover what
> is going on, so if I missed something important, please ask.
- tcpdump/ethereral on incoming+outgoing interface. see if there are
really packet drops / ack storms / ...
- enable ftp conntrack debugging (in ip_conntrack_ftp.c and ip_nat_ftp.c
there is something like
#if 0
#define DEBUGP(format, args...) printk(KERN_DEBUG __FILE__ ":" __FUNCTION__ \
":" format, ## args)
#else
#define DEBUGP(format, args...)
#endif
just change the "if 0" to "if 1", recompile and reinstall your modules.
you should get detailed debug logs in your syslog for every ftp connection
- what does /proc/net/ip_conntrack say about expectations
- what if you only use ip_conntrack_ftp without nat
> Jos? Luis Domingo L?pez
> Linux Registered User #189436 Debian Linux Woody (P166 64 MB RAM)
>
> jdomingo AT internautas DOT org => Spam at your own risk
>
>
>
--
Live long and prosper
- Harald Welte / [email protected] http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M+
V-- PS++ PE-- Y++ PGP++ t+ 5-- !X !R tv-- b+++ !DI !D G+ e* h--- r++ y+(*)
On Wednesday, 13 February 2002, at 14:33:59 +0100,
Harald Welte wrote:
> On Tue, Feb 12, 2002 at 11:12:39PM +0100, Jose Luis Domingo Lopez wrote:
>
> > The strange part is that this will show up with just some FTP clients
> > and/or remote FTP servers. For example, text-mode web browser "links"
> > doesn't work in any case. On the other hand, text-mode browser "lynx"
> > works nice in the same places where "links" fails. "wget", "ftp" and
> > "ncftp" show various degrees of success with several combinations of
> > active/passive transfer mode and remote FTP servers (tried with
> > ftp.kernel.org, ftp.at.kernel.org, ftp.mozilla.org, ftp.rediris.es,
> > ftp.sourceforge.net).
>
The NAT/router box that seemed to have problems with FTP connection
tracking went into production a week ago. Since them it has been working
OK, at least there are no reports froms people on the internal network
complaining about their file downloads stalling or failing.
However, most of the users don't use the net heavily. I use the net a
lot every day, and have had no big problems so far, except from some
(random) combinations of browser and remote FTP server that make
directory listing not complete (although file downloads work OK).
So it seems the problem I was having was somewhere outside the linux
kernel inside the router. Sorry for any inconvenience :-)
--
Jos? Luis Domingo L?pez
Linux Registered User #189436 Debian Linux Woody (P166 64 MB RAM)
jdomingo AT internautas DOT org => Spam at your own risk