Hello,
I've been having a problem with a high volume Linux web server. This
particular web server used to be a FreeBSD machine and I've been trying to
successfully make the switch for some time now. I've been trying the 2.4
development kernels as they come out and I've been tweaking the /proc
filesystem variables but so far nothing seems to have fixed the
problem. The problem is that I get "Out of socket memory" errors and the
networking locks up. Sometimes the server will go for weeks without
running into the problem and other times it'll last 30 minutes. The
hardware in question is an 1Ghz Athalon system with 256Mb of ram and an IDE
hard disk. I've tried every 2.4 test kernel to date. The web server is a
specialized web server running about 10 million hits a day. Of the 256Mb
of ram the web server uses 40Mb and there are no other significant memory
consuming processes on the system. Currently I am using the following
/proc modifications in the rc.local file.
echo "7168 11776 16384" > /proc/sys/net/ipv4/tcp_mem
echo 32768 > /proc/sys/net/ipv4/tcp_max_orphans
What am I doing wrong? Is this a kernel problem or a configuration
problem? Is there any way I can get runtime information from the kernel on
things like amount of socket memory used and amount available? Am I using
the right variables to increase available socket memory and just not giving
it enough yet?
I appreciate any help provided.
Thank you,
Daniel Walton
I'm not quite clear how the settings under /proc/sys/vm/* would effect the
problem. I neglected to mention in my previous post that all web content
is served directly from the memory of the web server (no file
accesses). The only file accesses that happen are from a MySQL server
which gets queried about once a second.
Here's the output of /proc/meminfo. I'm not sure how helpful it is. I was
kinda hoping for something that would allow me to see how much memory had
been allocated for sockets and what the max was.
[root@s4 /proc]# cat meminfo
total: used: free: shared: buffers: cached:
Mem: 261742592 122847232 138895360 0 1757184 88633344
Swap: 271392768 0 271392768
MemTotal: 255608 kB
MemFree: 135640 kB
MemShared: 0 kB
Buffers: 1716 kB
Cached: 86556 kB
Active: 15684 kB
Inact_dirty: 72588 kB
Inact_clean: 0 kB
Inact_target: 68 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 255608 kB
LowFree: 135640 kB
SwapTotal: 265032 kB
SwapFree: 265032 kB
-Dan
At 12:30 AM 12/7/2000 -0500, you wrote:
>backlog queue? tuning /proc/sys/vm/*?
>
> > problem? Is there any way I can get runtime information from the
> kernel on
> > things like amount of socket memory used and amount available? Am I using
>
>/proc/meminfo?
Daniel Walton ([email protected]) wrote:
> I've been having a problem with a high volume Linux web server. This
> particular web server used to be a FreeBSD machine and I've been trying to
> successfully make the switch for some time now. I've been trying the 2.4
> development kernels as they come out and I've been tweaking the /proc
> filesystem variables but so far nothing seems to have fixed the
> problem. The problem is that I get "Out of socket memory" errors and the
> networking locks up...
FWIW, the code that prints this is in ipv4/tcp_timer.c
and it looks like it's a DoS attack countermeasure.
Looks like you can tune it with sysctl_tcp_max_orphans, among
other things. Here's the code from test11-pre4:
/* Do not allow orphaned sockets to eat all our resources.
* This is direct violation of TCP specs, but it is required
* to prevent DoS attacks. It is called when a retransmission timeout
* or zero probe timeout occurs on orphaned socket.
*
* Criterium is still not confirmed experimentally and may change.
* We kill the socket, if:
* 1. If number of orphaned sockets exceeds an administratively configured
* limit.
* 2. If we have strong memory pressure.
*/
static int tcp_out_of_resources(struct sock *sk, int do_reset)
{
struct tcp_opt *tp = &(sk->tp_pinfo.af_tcp);
int orphans = atomic_read(&tcp_orphan_count);
/* If peer does not open window for long time, or did not transmit
* anything for long time, penalize it. */
if ((s32)(tcp_time_stamp - tp->lsndtime) > 2*TCP_RTO_MAX || !do_reset)
orphans <<= 1;
/* If some dubious ICMP arrived, penalize even more. */
if (sk->err_soft)
orphans <<= 1;
if (orphans >= sysctl_tcp_max_orphans ||
(sk->wmem_queued > SOCK_MIN_SNDBUF &&
atomic_read(&tcp_memory_allocated) > sysctl_tcp_mem[2])) {
if (net_ratelimit())
printk(KERN_INFO "Out of socket memory\n");
On Wed, Dec 06, 2000 at 10:56:32PM -0600, Daniel Walton wrote:
>
> Hello,
>
> I've been having a problem with a high volume Linux web server. This
> particular web server used to be a FreeBSD machine and I've been trying to
> successfully make the switch for some time now. I've been trying the 2.4
> development kernels as they come out and I've been tweaking the /proc
> filesystem variables but so far nothing seems to have fixed the
> problem. The problem is that I get "Out of socket memory" errors and the
> networking locks up. Sometimes the server will go for weeks without
You should probably first find out what and why is generating these messages.
It isn't the kernel. When the web server is generating when a send blocks
then it is badly broken.
-Andi
Hello!
> What am I doing wrong?
You change parameters without investigating why failure happened.
This approach cannot succeed, of course.
> problem? Is there any way I can get runtime information from the kernel on
> things like amount of socket memory used and amount available?
cat /proc/net/sockstat
cat /proc/net/netstat
cat /proc/net/snmp
cat /proc/slabinfo
cat /proc/net/tcp
Next time when you will see these messages, do this, pack the result
and send to me pointopoint.
Alexey
Dan Kegel wrote:
> Daniel Walton ([email protected]) wrote:
> > I've been having a problem with a high volume Linux web server. This
> > particular web server used to be a FreeBSD machine and I've been trying to
> > successfully make the switch for some time now. I've been trying the 2.4
> > development kernels as they come out and I've been tweaking the /proc
> > filesystem variables but so far nothing seems to have fixed the
> > problem. The problem is that I get "Out of socket memory" errors and the
> > networking locks up...
>
> FWIW, the code that prints this is in ipv4/tcp_timer.c
> and it looks like it's a DoS attack countermeasure.
>
> /* Do not allow orphaned sockets to eat all our resources.
> * This is direct violation of TCP specs, but it is required
> * to prevent DoS attacks. It is called when a retransmission timeout
> * or zero probe timeout occurs on orphaned socket.
> ...
> if (orphans >= sysctl_tcp_max_orphans ||
> (sk->wmem_queued > SOCK_MIN_SNDBUF &&
> atomic_read(&tcp_memory_allocated) > sysctl_tcp_mem[2])) {
> if (net_ratelimit())
> printk(KERN_INFO "Out of socket memory\n");
See Documentation/networking/ip-sysctl.txt for more info and tuning
suggestions, esp. tcp_orphan_retries and tcp_max_orphans.
An orphaned socket is one that has at least partially
opened, but has not been fully accepted. You might try reducing
tcp_orphan_retries.
It looks like you can view the current number of orphans at
/proc/net/sockstat or something like that.
/*
* Report socket allocation statistics [[email protected]]
*/
int afinet_get_info(char *buffer, char **start, off_t offset, int length)
{
/* From net/socket.c */
extern int socket_get_info(char *, char **, off_t, int);
int len = socket_get_info(buffer,start,offset,length);
len += sprintf(buffer+len,"TCP: inuse %d orphan %d tw %d alloc %d mem %d\n",
fold_prot_inuse(&tcp_prot),
atomic_read(&tcp_orphan_count), tcp_tw_count,
atomic_read(&tcp_sockets_allocated),
atomic_read(&tcp_memory_allocated));
- Dan