Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 21 Jun 2022 15:43:10 +0000
To:     Bernd Petrovitsch <bernd@petrovitsch.priv.at>
From:   "Ryan P. Nicholl" <rnicholl@protonmail.com>
Cc:     linux-kernel <linux-kernel@vger.kernel.org>
Reply-To: "Ryan P. Nicholl" <rnicholl@protonmail.com>
Subject: Re: Networking Question
Message-ID: <yaXzQ-VgbQxXxbv_b8c4kQRbIG5MGXsnQHf8-yqT___iT2bMmYpXlqi_L5ml14y1KcFhYSkc22-L1nOeJRECv355it-XgjLwjg0DGvUQJ64=@protonmail.com>
In-Reply-To: <b6d8611a-714c-4fb0-5027-584884bb66a1@petrovitsch.priv.at>
References: <6_nVwntZEyEq2Gn0PYDz5w_yuD20E9Je_Bk2rJHsOersGq3v2x3_6hXnyjNm-X83-OTl1KKo4eLoHzyBVM_nzCcDrpPtz4dz3_71A3QezfI=@protonmail.com> <b6d8611a-714c-4fb0-5027-584884bb66a1@petrovitsch.priv.at>
Feedback-ID: 15386781:user:proton
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Thanks for taking time to respond to this question, but unfortunately Linux=
's shutdown(2) cannot do what I need. This isn't a general question since P=
OSIX has no standard async API other than AIO which isn't implemented effic=
iently in GNU/Linux, and I don't think the kernel supports any aio calls di=
rectly, instead exposing clone and epoll_* facilities to solve the concurre=
ncy issues. However; none of them do what I need efficiently.

In theory, shutdown(2) could be used, the problem is that it's just not eff=
iciently scalable to 100s of thousands or millions of connections. I'm not =
speaking on behalf of my employeer, but I work for a financial company that=
 processes a lot of network traffic.

What I want essentially is similar to TIME_WAIT but instead of sending an R=
ST packet I want to be notified when the tcp connection is actually drained=
.

If you call shutdown, my understanding is that you get one of two behaviors=
:
1. You send RST and immediately discard the send buffer.
2. The call blocks for the SO_LINGER timeout.

Option 1 isn't acceptable because it gives the wrong behavior. Option 2 can=
 be made to work but the downside is that Linux only exposes a synchronous =
API for it which requires me to have at least 1 thread per shutdown operati=
on. So it could be quite bad if the network is very congested. Unfortunatel=
y threads are many times more expensive than sockets/tcp connections. So id=
eally having few threads and many sockets gives the best use of resources.


--
Ryan P. Nicholl


------- Original Message -------
On Tuesday, June 21st, 2022 at 3:02 AM, Bernd Petrovitsch <bernd@petrovitsc=
h.priv.at> wrote:


> Hi all!
>
> On 21/06/2022 02:29, Ryan P. Nicholl wrote:
>
> > I've been unable to find any Linux API for asynchronously waiting for t=
he TCP send buffer to drain.
> >
> > The problem I have, in a nutshell, is noted in this part of the documen=
tation:
> >
> > If fildes refers to a socket, close() shall cause the socket to
> > be destroyed. If the socket is in connection-mode, and the
>
>
> That's not really a Linux kernel question as such (because that should
> work that way on all TCP connections anywhere) but the shutdown()
> syscall is probably what you need:
> - your side shuts down the sending part of the socket.
> - the other side reads data and gets eventually EOF
> - the other side call shutdown() for it's sending side when it's done.
> - your side gets EOF.
> And then your side knows that no data is in flight.
> - finally, you clean up with close(). You can shutdown() the receiving
> side too but doesn't change anything.
>
> [ deleted SO_LINGER stuff - that's for something completelly different ..=
. ]
>
> Kind regards,
> Bernd
> --
> Bernd Petrovitsch Email : bernd@petrovitsch.priv.at
> There is NO CLOUD, just other people's computers. - FSFE
> LUGA : http://www.luga.at