Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <8dff3e46-6dac-af6a-1a3b-e6a8b93fdc60@collabora.com>
Date:   Fri, 30 Sep 2022 18:24:00 +0500
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.13.0
Cc:     usama.anjum@collabora.com, LKML <linux-kernel@vger.kernel.org>,
        "David S. Miller" <davem@davemloft.net>,
        Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
        David Ahern <dsahern@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paul Gofman <pgofman@codeweavers.com>
Subject: Re: [RFC] EADDRINUSE from bind() on application restart after killing
Content-Language: en-US
To:     Eric Dumazet <edumazet@google.com>,
        "open list:NETWORKING [TCP]" <netdev@vger.kernel.org>
References: <5099dc39-c6d9-115a-855b-6aa98d17eb4b@collabora.com>
From:   Muhammad Usama Anjum <usama.anjum@collabora.com>
In-Reply-To: <5099dc39-c6d9-115a-855b-6aa98d17eb4b@collabora.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi Eric,

RFC 1337 describes the TIME-WAIT Assassination Hazards in TCP. Because
of this hazard we have 60 seconds timeout in TIME_WAIT state if
connection isn't closed properly. From RFC 1337:
> The TIME-WAIT delay allows all old duplicate segments time
enough to die in the Internet before the connection is reopened.

As on localhost there is virtually no delay. I think the TIME-WAIT delay
must be zero for localhost connections. I'm no expert here. On localhost
there is no delay. So why should we wait for 60 seconds to mitigate a
hazard which isn't there?

Zapping the sockets in TIME_WAIT and FIN_WAIT_2 does removes them. But
zap is required from privileged (CAP_NET_ADMIN) process. We are having
hard time finding a privileged process to do this.

Thanks,
Usama


On 5/24/22 1:18 PM, Muhammad Usama Anjum wrote:
> Hello,
> 
> We have a set of processes which talk with each other through a local
> TCP socket. If the process(es) are killed (through SIGKILL) and
> restarted at once, the bind() fails with EADDRINUSE error. This error
> only appears if application is restarted at once without waiting for 60
> seconds or more. It seems that there is some timeout of 60 seconds for
> which the previous TCP connection remains alive waiting to get closed
> completely. In that duration if we try to connect again, we get the error.
> 
> We are able to avoid this error by adding SO_REUSEADDR attribute to the
> socket in a hack. But this hack cannot be added to the application
> process as we don't own it.
> 
> I've looked at the TCP connection states after killing processes in
> different ways. The TCP connection ends up in 2 different states with
> timeouts:
> 
> (1) Timeout associated with FIN_WAIT_1 state which is set through
> `tcp_fin_timeout` in procfs (60 seconds by default)
> 
> (2) Timeout associated with TIME_WAIT state which cannot be changed. It
> seems like this timeout has come from RFC 1337.
> 
> The timeout in (1) can be changed. Timeout in (2) cannot be changed. It
> also doesn't seem feasible to change the timeout of TIME_WAIT state as
> the RFC mentions several hazards. But we are talking about a local TCP
> connection where maybe those hazards aren't applicable directly? Is it
> possible to change timeout for TIME_WAIT state for only local
> connections without any hazards?
> 
> We have tested a hack where we replace timeout of TIME_WAIT state from a
> value in procfs for local connections. This solves our problem and
> application starts to work without any modifications to it.
> 
> The question is that what can be the best possible solution here? Any
> thoughts will be very helpful.
> 
> Regards,
> 

-- 
Muhammad Usama Anjum