2018-07-25 07:14:32

by Tan Hu

Subject: [PATCH v3] ipvs: fix race between ip_vs_conn_new() and ip_vs_del_dest()

We came across infinite loop in ipvs when using ipvs in docker
env.

When ipvs receives new packets and cannot find an ipvs connection,
it will create a new connection, then if the dest is unavailable
(i.e. IP_VS_DEST_F_AVAILABLE), the packet will be dropped sliently.

But if the dropped packet is the first packet of this connection,
the connection control timer never has a chance to start and the
ipvs connection cannot be released. This will lead to memory leak, or
infinite loop in cleanup_net() when net namespace is released like
this:

ip_vs_conn_net_cleanup at ffffffffa0a9f31a [ip_vs]
__ip_vs_cleanup at ffffffffa0a9f60a [ip_vs]
ops_exit_list at ffffffff81567a49
cleanup_net at ffffffff81568b40
process_one_work at ffffffff810a851b
worker_thread at ffffffff810a9356
kthread at ffffffff810b0b6f
ret_from_fork at ffffffff81697a18

race condition:
CPU1 CPU2
ip_vs_in()
ip_vs_conn_new()
ip_vs_del_dest()
__ip_vs_unlink_dest()
~IP_VS_DEST_F_AVAILABLE
cp->dest && !IP_VS_DEST_F_AVAILABLE
__ip_vs_conn_put
...
cleanup_net ---> infinite looping

Fix this by checking whether the timer already started.

Signed-off-by: Tan Hu <[email protected]>
Reviewed-by: Jiang Biao <[email protected]>
---
v2: fix use-after-free in CONN_ONE_PACKET case suggested by Julian Anastasov
v3: remove trailing whitespace for patch checking

net/netfilter/ipvs/ip_vs_core.c | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
index 0679dd1..a17104f 100644
--- a/net/netfilter/ipvs/ip_vs_core.c
+++ b/net/netfilter/ipvs/ip_vs_core.c
@@ -1972,13 +1972,20 @@ static int ip_vs_in_icmp_v6(struct netns_ipvs *ipvs, struct sk_buff *skb,
if (cp->dest && !(cp->dest->flags & IP_VS_DEST_F_AVAILABLE)) {
/* the destination server is not available */

- if (sysctl_expire_nodest_conn(ipvs)) {
+ __u32 flags = cp->flags;
+
+ /* when timer already started, silently drop the packet.*/
+ if (timer_pending(&cp->timer))
+ __ip_vs_conn_put(cp);
+ else
+ ip_vs_conn_put(cp);
+
+ if (sysctl_expire_nodest_conn(ipvs) &&
+ !(flags & IP_VS_CONN_F_ONE_PACKET)) {
/* try to expire the connection immediately */
ip_vs_conn_expire_now(cp);
}
- /* don't restart its timer, and silently
- drop the packet. */
- __ip_vs_conn_put(cp);
+
return NF_DROP;
}

--
1.8.3.1

2018-07-25 19:15:10

by Julian Anastasov

Subject: Re: [PATCH v3] ipvs: fix race between ip_vs_conn_new() and ip_vs_del_dest()

Hello,

On Wed, 25 Jul 2018, Tan Hu wrote:

> We came across infinite loop in ipvs when using ipvs in docker
> env.
>
> When ipvs receives new packets and cannot find an ipvs connection,
> it will create a new connection, then if the dest is unavailable
> (i.e. IP_VS_DEST_F_AVAILABLE), the packet will be dropped sliently.
>
> But if the dropped packet is the first packet of this connection,
> the connection control timer never has a chance to start and the
> ipvs connection cannot be released. This will lead to memory leak, or
> infinite loop in cleanup_net() when net namespace is released like
> this:
>
> ip_vs_conn_net_cleanup at ffffffffa0a9f31a [ip_vs]
> __ip_vs_cleanup at ffffffffa0a9f60a [ip_vs]
> ops_exit_list at ffffffff81567a49
> cleanup_net at ffffffff81568b40
> process_one_work at ffffffff810a851b
> worker_thread at ffffffff810a9356
> kthread at ffffffff810b0b6f
> ret_from_fork at ffffffff81697a18
>
> race condition:
> CPU1 CPU2
> ip_vs_in()
> ip_vs_conn_new()
> ip_vs_del_dest()
> __ip_vs_unlink_dest()
> ~IP_VS_DEST_F_AVAILABLE
> cp->dest && !IP_VS_DEST_F_AVAILABLE
> __ip_vs_conn_put
> ...
> cleanup_net ---> infinite looping
>
> Fix this by checking whether the timer already started.
>
> Signed-off-by: Tan Hu <[email protected]>
> Reviewed-by: Jiang Biao <[email protected]>

v3 looks good to me,

Acked-by: Julian Anastasov <[email protected]>

Simon and Pablo, this can be applied to ipvs/nf tree...

> ---
> v2: fix use-after-free in CONN_ONE_PACKET case suggested by Julian Anastasov
> v3: remove trailing whitespace for patch checking
>
> net/netfilter/ipvs/ip_vs_core.c | 15 +++++++++++----
> 1 file changed, 11 insertions(+), 4 deletions(-)
>
> diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
> index 0679dd1..a17104f 100644
> --- a/net/netfilter/ipvs/ip_vs_core.c
> +++ b/net/netfilter/ipvs/ip_vs_core.c
> @@ -1972,13 +1972,20 @@ static int ip_vs_in_icmp_v6(struct netns_ipvs *ipvs, struct sk_buff *skb,
> if (cp->dest && !(cp->dest->flags & IP_VS_DEST_F_AVAILABLE)) {
> /* the destination server is not available */
>
> - if (sysctl_expire_nodest_conn(ipvs)) {
> + __u32 flags = cp->flags;
> +
> + /* when timer already started, silently drop the packet.*/
> + if (timer_pending(&cp->timer))
> + __ip_vs_conn_put(cp);
> + else
> + ip_vs_conn_put(cp);
> +
> + if (sysctl_expire_nodest_conn(ipvs) &&
> + !(flags & IP_VS_CONN_F_ONE_PACKET)) {
> /* try to expire the connection immediately */
> ip_vs_conn_expire_now(cp);
> }
> - /* don't restart its timer, and silently
> - drop the packet. */
> - __ip_vs_conn_put(cp);
> +
> return NF_DROP;
> }
>
> --
> 1.8.3.1

Regards

--
Julian Anastasov <[email protected]>

2018-07-31 17:28:23

by Simon Horman

Subject: Re: [PATCH v3] ipvs: fix race between ip_vs_conn_new() and ip_vs_del_dest()

On Wed, Jul 25, 2018 at 10:12:48PM +0300, Julian Anastasov wrote:
>
> Hello,
>
> On Wed, 25 Jul 2018, Tan Hu wrote:
>
> > We came across infinite loop in ipvs when using ipvs in docker
> > env.
> >
> > When ipvs receives new packets and cannot find an ipvs connection,
> > it will create a new connection, then if the dest is unavailable
> > (i.e. IP_VS_DEST_F_AVAILABLE), the packet will be dropped sliently.
> >
> > But if the dropped packet is the first packet of this connection,
> > the connection control timer never has a chance to start and the
> > ipvs connection cannot be released. This will lead to memory leak, or
> > infinite loop in cleanup_net() when net namespace is released like
> > this:
> >
> > ip_vs_conn_net_cleanup at ffffffffa0a9f31a [ip_vs]
> > __ip_vs_cleanup at ffffffffa0a9f60a [ip_vs]
> > ops_exit_list at ffffffff81567a49
> > cleanup_net at ffffffff81568b40
> > process_one_work at ffffffff810a851b
> > worker_thread at ffffffff810a9356
> > kthread at ffffffff810b0b6f
> > ret_from_fork at ffffffff81697a18
> >
> > race condition:
> > CPU1 CPU2
> > ip_vs_in()
> > ip_vs_conn_new()
> > ip_vs_del_dest()
> > __ip_vs_unlink_dest()
> > ~IP_VS_DEST_F_AVAILABLE
> > cp->dest && !IP_VS_DEST_F_AVAILABLE
> > __ip_vs_conn_put
> > ...
> > cleanup_net ---> infinite looping
> >
> > Fix this by checking whether the timer already started.
> >
> > Signed-off-by: Tan Hu <[email protected]>
> > Reviewed-by: Jiang Biao <[email protected]>
>
> v3 looks good to me,
>
> Acked-by: Julian Anastasov <[email protected]>
>
> Simon and Pablo, this can be applied to ipvs/nf tree...

Acked-by: Simon Horman <[email protected]>

Pablo, please consider this for the nf tree.

>
> > ---
> > v2: fix use-after-free in CONN_ONE_PACKET case suggested by Julian Anastasov
> > v3: remove trailing whitespace for patch checking
> >
> > net/netfilter/ipvs/ip_vs_core.c | 15 +++++++++++----
> > 1 file changed, 11 insertions(+), 4 deletions(-)
> >
> > diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
> > index 0679dd1..a17104f 100644
> > --- a/net/netfilter/ipvs/ip_vs_core.c
> > +++ b/net/netfilter/ipvs/ip_vs_core.c
> > @@ -1972,13 +1972,20 @@ static int ip_vs_in_icmp_v6(struct netns_ipvs *ipvs, struct sk_buff *skb,
> > if (cp->dest && !(cp->dest->flags & IP_VS_DEST_F_AVAILABLE)) {
> > /* the destination server is not available */
> >
> > - if (sysctl_expire_nodest_conn(ipvs)) {
> > + __u32 flags = cp->flags;
> > +
> > + /* when timer already started, silently drop the packet.*/
> > + if (timer_pending(&cp->timer))
> > + __ip_vs_conn_put(cp);
> > + else
> > + ip_vs_conn_put(cp);
> > +
> > + if (sysctl_expire_nodest_conn(ipvs) &&
> > + !(flags & IP_VS_CONN_F_ONE_PACKET)) {
> > /* try to expire the connection immediately */
> > ip_vs_conn_expire_now(cp);
> > }
> > - /* don't restart its timer, and silently
> > - drop the packet. */
> > - __ip_vs_conn_put(cp);
> > +
> > return NF_DROP;
> > }
> >
> > --
> > 1.8.3.1
>
> Regards
>
> --
> Julian Anastasov <[email protected]>
>

2018-08-03 11:29:18

by Pablo Neira Ayuso

Subject: Re: [PATCH v3] ipvs: fix race between ip_vs_conn_new() and ip_vs_del_dest()

On Tue, Jul 31, 2018 at 06:54:21PM +0200, Simon Horman wrote:
> On Wed, Jul 25, 2018 at 10:12:48PM +0300, Julian Anastasov wrote:
[...]
> > v3 looks good to me,
> >
> > Acked-by: Julian Anastasov <[email protected]>
> >
> > Simon and Pablo, this can be applied to ipvs/nf tree...
>
> Acked-by: Simon Horman <[email protected]>
>
> Pablo, please consider this for the nf tree.

Applied, thanks!