When a Linux machine has reached the tcp_mem limit, what will happen to
'write's on non-blocking sockets? Will they block until more TCP memory is
available? Will they return an error code? ENOMEM?
If it varies by kernel version, details about different versions would be
extremely helpful. I'm most interested in late 2.4 kernels.
Thanks in advance.
DS
--
David Schwartz
<[email protected]>
> When a Linux machine has reached the tcp_mem limit, what will happen to
> 'write's on non-blocking sockets? Will they block until more TCP memory is
> available? Will they return an error code? ENOMEM?
>
> If it varies by kernel version, details about different versions would be
> extremely helpful. I'm most interested in late 2.4 kernels.
>
> Thanks in advance.
>
Returns EAGAIN.
Fairly static ~late 2.4.
thanks,
Nivedita
On Thu, Nov 21, 2002 at 05:27:45PM -0800, Nivedita Singhvi wrote:
> > When a Linux machine has reached the tcp_mem limit, what will happen to
> > 'write's on non-blocking sockets? Will they block until more TCP memory is
> > available? Will they return an error code? ENOMEM?
> >
> > If it varies by kernel version, details about different versions would be
> > extremely helpful. I'm most interested in late 2.4 kernels.
> >
> > Thanks in advance.
> Returns EAGAIN.
> Fairly static ~late 2.4.
returns number of bytes sent and sets errno to EAGAIN.
-alex
On Thu, 2002-11-21 at 21:34, David Schwartz wrote:
> When a Linux machine has reached the tcp_mem limit, what will happen to
> 'write's on non-blocking sockets? Will they block until more TCP memory is
> available? Will they return an error code? ENOMEM?
from write(2) man page.
EAGAIN Non-blocking I/O has been selected using O_NONBLOCK and the write
would block.
--
// Gianni Tedesco (gianni at ecsc dot co dot uk)
lynx --source http://www.scaramanga.co.uk/gianni-at-ecsc.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D
On 22 Nov 2002 12:06:29 +0000, Gianni Tedesco wrote:
>On Thu, 2002-11-21 at 21:34, David Schwartz wrote:
>> When a Linux machine has reached the tcp_mem limit, what will happen to
>>'write's on non-blocking sockets? Will they block until more TCP memory is
>>available? Will they return an error code? ENOMEM?
>from write(2) man page.
>EAGAIN Non-blocking I/O has been selected using O_NONBLOCK and the write
>would block.
So this would be a case where 'poll' or 'select' would return a write hit
for a socket but 'write' would return -1 and set errno to EAGAIN.
DS
David Schwartz wrote:
> So this would be a case where 'poll' or 'select' would return
> a write hit for a socket but 'write' would return -1 and set errno
> to EAGAIN.
Is this really true? It would livelock several servers I've worked on...
-- Jamie
On Sat, 23 Nov 2002 00:06:17 +0000, Jamie Lokier wrote:
>David Schwartz wrote:
>>So this would be a case where 'poll' or 'select' would return
>>a write hit for a socket but 'write' would return -1 and set errno
>>to EAGAIN.
>Is this really true? It would livelock several servers I've worked on...
>
>-- Jamie
We're now getting close to my motivation for asking this question.
If it does in fact return EAGAIN, then a poll/select loop program that is
trying to write could get into trouble if another process was responsible for
large receive queues. It would spin uselessly burning its timeslice as more
data comes in and it delays the execution of the other process that might be
able to drain receive queues.
On the other hand, if send blocked, there would be a disaster if a
select/poll loop process were the only TCP user on the box. In this case, it
would deadlock. The process is blocked waiting for memory but the only way to
get that memory is for the process to get around to reading from other
connections, which it can't do because it's blocked.
My motivation in assessing what happens is to develop a sane application
strategy to detect and handle this condition. If we could detect it, we could
try not to do TCP writes and try to do reads to relieve the memory problem.
Some strategies I was considering were this:
1) For applications that probably aren't using most of the TCP memory: In
the select/poll loop, we keep track of whether we were able to do any work.
If we never successfully wrote or read, but did get a hit on at least one
socket, we sleep for a few milliseconds. (We hope that while we sleep other
applications can free up more memory for us by draining their receive
queues.)
2) For applications that probably are using all or a significant fraction of
the TCP memory: If a 'write' for a socket that we got a write hit on returns
EAGAIN, for the rest of this poll/select cycle, we do not attempt any writes,
only reads. (We hope that this will relieve the memory pressure scenario and
allow the system to function normally as quickly as possible.)
3) For applications that have control over how much load they take: If
writes begin returning EAGAIN under suspicious circumstances, treat this as
an overload condition and trigger your handling logic. Refuse expensive
commands or refuse to handle new connections.
Any comments or suggestions are appreciated. I've found that when we hit TCP
memory pressure, many applications become very badly behaved.
DS
> Any comments or suggestions are appreciated. I've found that when we hit
TCP
> memory pressure, many applications become very badly behaved.
What about:
int WRITE(int handle, char *whereto, int len)
{
int cnt=len;
while(len>0)
{
int rc;
rc = write(handle, whereto, len);
if (rc == -1)
{
if (errno == EINTR)
{
/* just try again */
}
else if (errno == EAGAIN)
{
/* give up time-slice */
if (sched_yield() == -1)
{
/* BIG troubles */
syslog(LOG_DEBUG, "WRITE(), during EAGAIN
handling: sched_yield failed! [%d - %s]", errno, strerror(errno));
return -1;
}
}
else
{
syslog(LOG_DEBUG, "WRITE(): io-error [%d -
%s]", errno, strerror(errno));
return -1;
}
}
else if (rc == 0)
{
return 0;
}
else
{
whereto += rc;
len -= rc;
}
}
return cnt;
}
On Sat, 23 Nov 2002 23:34:15 +0100, Folkert van Heusden wrote:
>What about:
>int WRITE(int handle, char *whereto, int len)
>{
>int cnt=len;
>
>while(len>0)
>{
>int rc;
>
>rc = write(handle, whereto, len);
>
>if (rc == -1)
>{
>if (errno == EINTR)
> {
> /* just try again */
> }
> else if (errno == EAGAIN)
> {
> /* give up time-slice */
> if (sched_yield() == -1)
> {
By yielding your time slice, you prevent yourself from performing any
'read's. So what's going to fix the memory pressure?
There are two cases where this could work:
1) For an application that uses very little TCP memory. By yielding, perhaps
some other process will come along, a more heavy TCP memory user, and fix the
problem.
2) For an application that uses a thread-per-connection architecture. By
yielding, we give other threads a chance to run and hope that they will
relieve the memory pressure.
But for a poll/select loop application that is heavily using TCP memory,
this makes things worse. By stopping our own execution, we delay the time
when we'll do 'read's. So we extend the TCP memory pressure problem for even
longer.
DS