Hi all,
I'm expiriencing a problem with thttpd web server
(http://www.acme.com/software/thttpd) on recent linux 2.2 kernels with Andrea's
VM-global patches. Without the patch server runs normally with its usual
dose of complaints on the linux platform (it's being developed on BSD
afaik), but with the patch it runs ok for about three days (depends on
traffic i guess), then enters into some state where it reports 'out of
memory' for every larger file (>1Mb) it starts serving and dies. When it
comes back up it dies again whitin 10 seconds. As this is not happening
on a stock kernel and the restart of the server itself has no efect, i
conclude it has to be something there in the VM-global that thttpd
doesnt really like. As the VM-global seems to be the only cure for the
VM_do_try_to_free_pages problem, which is an issue for me too, i'd
really like to hear some official words on this before 2.2.19 comes out
with VM-global ... and while i'm at it, can we expect ide patch in
2.2.19 too?
--
Pegasus
On Fri, Dec 29, 2000 at 02:32:32AM +0100, Jure Pecar wrote:
> Hi all,
>
> I'm expiriencing a problem with thttpd web server
> (http://www.acme.com/software/thttpd) on recent linux 2.2 kernels with Andrea's
> VM-global patches. Without the patch server runs normally with its usual
Before the -7 revision the VM-global was sharing a memory corruption bug with
vanilla 2.2.x. VM-global was incidentally hiding such bug extremely well though
(but with heavy load and using LVM snapshotting I triggered it even under
VM-global-6, so then I spotted such last leftover too and the strict fix for
such MM corruption bug got merged into 2.2.18pre2x too, and at the same time I
released VM-global-7)
I need to know exactly which kernel sources you were using (and also the
compiler of course ;)
> dose of complaints on the linux platform (it's being developed on BSD
> afaik), but with the patch it runs ok for about three days (depends on
> traffic i guess), then enters into some state where it reports 'out of
> memory' for every larger file (>1Mb) it starts serving and dies. When it
out of memory looks an userspace message, so it looks like malloc request was
too large (it could happen because of an userspace corruption in the 'size'
parameter for example).
> comes back up it dies again whitin 10 seconds. As this is not happening
> on a stock kernel and the restart of the server itself has no efect, i
> conclude it has to be something there in the VM-global that thttpd
> doesnt really like. As the VM-global seems to be the only cure for the
> VM_do_try_to_free_pages problem, which is an issue for me too, i'd
> really like to hear some official words on this before 2.2.19 comes out
> with VM-global ... and while i'm at it, can we expect ide patch in
> 2.2.19 too?
Even if you are using VM-global-7 with only this information it's non obvious
that VM-global-7 (the one included 19pre3) is the source of your problem.
So if you were using the -7 revision please check the logs and the console for
Oopses and try to strace the daemon to see how it dies. It's also not clear
what it means that restarting the server itself has no effect, as far I can
tell it could even be an userspace issue (some rc script that doesn't cope with
an unclean exit of the server?). And besides the webserver, how does the rest
of the system behaves when the problem appears? There are too many
possibilities, we need to restrict our search.
You were using -6 revision as first thing you can try to reproduce with
2.2.19pre3 after checking the compiler. If you send me your .config I can send
you a binary image to test.
Thanks for the feedback,
Andrea
On Fri, Dec 29, 2000 at 03:29:53AM +0100, Andrea Arcangeli wrote:
> On Fri, Dec 29, 2000 at 02:32:32AM +0100, Jure Pecar wrote:
> > Hi all,
> >
> > I'm expiriencing a problem with thttpd web server
> > (http://www.acme.com/software/thttpd) on recent linux 2.2 kernels with Andrea's
I downloaded the sources right now to see what 'out of memory' means. I assume
you were using version 2.20b.
> out of memory looks an userspace message, so it looks like malloc request was
My guess was right, it's not possible to know where it came from though:
andrea@athlon:~/devel/thttpd-2.20b > grep 'out of' *.c
fdwatch.c:** or 0 if the timeout expired, or -1 on errors. A timeout of INFTIM
means
libhttpd.c: syslog( LOG_CRIT, "out of memory" );
libhttpd.c: syslog( LOG_CRIT, "out of memory" );
libhttpd.c: syslog( LOG_CRIT, "out of memory" );
libhttpd.c: syslog( LOG_CRIT, "out of memory" );
libhttpd.c: syslog( LOG_CRIT, "out of memory" );
libhttpd.c: syslog( LOG_CRIT, "out of memory" );
libhttpd.c: syslog( LOG_ERR, "out of memory" );
libhttpd.c: ** since it's impossible to get out of the tree. However, we still
libhttpd.c: syslog( LOG_ERR, "out of memory" );
libhttpd.c: syslog( LOG_ERR, "out of memory" );
thttpd.c: syslog( LOG_CRIT, "out of memory" );
thttpd.c: syslog( LOG_CRIT, "out of memory" );
thttpd.c: (void) fprintf( stderr, "%s: out of memory\n", argv0 );
thttpd.c: syslog( LOG_CRIT, "out of memory" );
thttpd.c: (void) fprintf( stderr, "out of memory\n" );
thttpd.c: syslog( LOG_CRIT, "out of memory" );
thttpd.c: (void) fprintf( stderr, "out of memory\n" );
thttpd.c: syslog( LOG_CRIT, "out of memory" );
andrea@athlon:~/devel/thttpd-2.20b >
But luckily it always happens after some kind of memory allocation, so it's
going to be the overcommit check of mmap that is complaining as I guessed
(assuming glibc isn't buggy in your system).
I was thinking I have a minor fix (in aa patchkit) that can help if the problem
happens while you are quite near to use all the memory (swap included) and you
have very low level of buffercache and pagecache. But I don't think you were
near the out of memory condition, right? (can you monitor via `vmstat 1 >log`
to be sure?) It addresses more a correctness issue than a real life problem.
And if it was a real life problem then you can workaround it with `echo 1
>/proc/sys/vm/overcommit_memory' without need to apply the real fix and
recompile the kernel (but again: I don't think this is the problem, setting
overcommit_memory to 1 would probably only temporarly hide the problem, you
would probably get the task killed by the kernel some time later anyways)
BTW, after checking the sources we at least know it wasn't due memory
fragmentation (fork complains with a different message).
Andrea
PS. I'm very suprised thttpd isn't threaded, it should really be threaded at
least on the x86 family to run fast.
On Fri, Dec 29, 2000 at 03:47:12AM +0100, Andrea Arcangeli wrote:
> PS. I'm very suprised thttpd isn't threaded, it should really be threaded at
> least on the x86 family to run fast.
This is one of the main thttpd design points: run in a select() loop. Since
it is intended for mainly static workloads, it performs quite well...
--
Petru Paler, mailto:[email protected]
http://www.ppetru.net - ICQ: 41817235
Andrea Arcangeli wrote:
> I need to know exactly which kernel sources you were using (and also the
> compiler of course ;)
At the moment it is 2.2.18 + all pathces that can be found at
http://www.linuxraid.org and ide patch 2.2.18.1221, but i first spotted this
problem on a similary configured 2.2.17 with VM-global patch 3. gcc
--version returns egcs-2.91.66. The box is an old p200mmx with 128mb ram
running redhat 6.0 with lots of patches and customisations ... glibc rpm
packet is glibc-2.1.3-21.
Sample vmstat output:
procs memory swap io
system cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 5804 2716 2624 92188 3 1 50 9 446 441
7 3 90
Straceing the server shows that it indeed quits after a munmap:
[snip]
select(9, [5], [6 7 8], NULL, {72, 164000}) = 1 (out [7], left {71,
960000})
gettimeofday({978077877, 462325}, NULL) = 0
write(7, "\364\362\222\210\311\336\323\206"..., 243498437) = 20272
select(9, [5], [6 7 8], NULL, {71, 944000}) = 1 (in [5], left {71,
770000})
gettimeofday({978077877, 645302}, NULL) = 0
accept(5, {sin_family=AF_INET, sin_port=htons(1444),
sin_addr=inet_addr("24.11.189.33")}, [16]) = 9
fcntl(9, F_SETFD, FD_CLOEXEC) = 0
fcntl(9, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(9, F_SETFL, O_RDWR|O_NONBLOCK) = 0
brk(0x8079000) = 0x8078000
brk(0x8079000) = 0x8078000
time([978077877]) = 978077877
getpid() = 22024
send(3, "<26>Dec 29 09:17:57 thttpd[22024"..., 68, 0) = 68
munmap(0x125000, 4096) = 0
_exit(1) = ?
Hmm ... this looks like it dies when another request comes in when one
download is already in progress ... let me try again:
[snip]
select(7, [5], [6], NULL, {98, 652000}) = 1 (out [6], left {98, 560000})
gettimeofday({978078108, 982991}, NULL) = 0
write(6, "O\327\372R\220\354r\317[\231T\0\340"..., 246365477) = 15928
select(7, [5], [6], NULL, {98, 547000}) = 1 (in [5], left {98, 210000})
gettimeofday({978078109, 324744}, NULL) = 0
accept(5, {sin_family=AF_INET, sin_port=htons(3687),
sin_addr=inet_addr("62.76.36.242")}, [16]) = 7
fcntl(7, F_SETFD, FD_CLOEXEC) = 0
fcntl(7, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(7, F_SETFL, O_RDWR|O_NONBLOCK) = 0
brk(0x8078000) = 0x8077000
brk(0x8078000) = 0x8077000
time([978078109]) = 978078109
getpid() = 22061
send(3, "<27>Dec 29 09:21:49 thttpd[22061"..., 69, 0) = 69
munmap(0x125000, 4096) = 0
_exit(1) = ?
Am i right? My IP was 213.143.68.212 in both runs.
I just tried the 'echo 1 >/proc/sys/vm/overcommit_memory' trick ... and
no, it's still dieing at about the same rate.
My .config on 2.2.18-cdhs :
#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
#
# Processor type and features
#
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
CONFIG_M586TSC=y
# CONFIG_M686 is not set
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_TSC=y
# CONFIG_MICROCODE is not set
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_1GB=y
# CONFIG_2GB is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_SMP is not set
#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODVERSIONS=y
CONFIG_KMOD=y
#
# General setup
#
CONFIG_NET=y
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_QUIRKS=y
# CONFIG_PCI_OPTIMIZE is not set
CONFIG_PCI_OLD_PROC=y
# CONFIG_MCA is not set
# CONFIG_VISWS is not set
CONFIG_SYSVIPC=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_SYSCTL=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_MISC=m
# CONFIG_BINFMT_JAVA is not set
# CONFIG_PARPORT is not set
# CONFIG_APM is not set
# CONFIG_TOSHIBA is not set
#
# Plug and Play support
#
# CONFIG_PNP is not set
#
# Block devices
#
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_DEV_IDE=y
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
# CONFIG_IDEDISK_STROKE is not set
CONFIG_BLK_DEV_IDECD=m
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set
# CONFIG_PKT_TASK_IOCTL is not set
# CONFIG_IDE_TASK_IOCTL_DEBUG is not set
# CONFIG_BLK_DEV_CMD640 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_AUTO=y
# CONFIG_IDEDMA_NEW_DRIVE_LISTINGS is not set
CONFIG_IDEDMA_PCI_EXPERIMENTAL=y
# CONFIG_IDEDMA_PCI_WIP is not set
CONFIG_BLK_DEV_OFFBOARD=y
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD7409 is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
CONFIG_BLK_DEV_HPT366=y
CONFIG_BLK_DEV_PIIX=y
CONFIG_PIIX_TUNING=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_PDC202XX is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_OSB4 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_CHIPSETS is not set
# CONFIG_IDEDMA_IVB is not set
CONFIG_BLK_DEV_LOOP=m
# CONFIG_BLK_DEV_NBD is not set
CONFIG_BLK_DEV_MD=y
CONFIG_AUTODETECT_RAID=y
CONFIG_MD_LINEAR=y
CONFIG_MD_STRIPED=y
CONFIG_MD_MIRRORING=y
CONFIG_MD_RAID5=y
# CONFIG_MD_TRANSLUCENT is not set
# CONFIG_MD_HSM is not set
CONFIG_MD_BOOT=y
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_SIZE=4096
# CONFIG_BLK_DEV_XD is not set
# CONFIG_BLK_DEV_DAC960 is not set
CONFIG_PARIDE_PARPORT=y
# CONFIG_PARIDE is not set
CONFIG_BLK_DEV_IDE_MODES=y
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_HD is not set
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_NETLINK=y
CONFIG_RTNETLINK=y
CONFIG_NETLINK_DEV=y
CONFIG_FIREWALL=y
CONFIG_FILTER=y
CONFIG_UNIX=y
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
# CONFIG_IP_PNP is not set
CONFIG_IP_FIREWALL=y
CONFIG_IP_FIREWALL_NETLINK=y
CONFIG_NETLINK_DEV=y
# CONFIG_IP_TRANSPARENT_PROXY is not set
# CONFIG_IP_MASQUERADE is not set
# CONFIG_IP_ROUTER is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_IP_ALIAS=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_RARP is not set
CONFIG_SKB_LARGE=y
# CONFIG_IPV6 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_BRIDGE is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_LLC is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_FASTROUTE is not set
# CONFIG_NET_HW_FLOWCONTROL is not set
# CONFIG_CPU_IS_SLOW is not set
#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set
#
# Telephony Support
#
# CONFIG_PHONE is not set
# CONFIG_PHONE_IXJ is not set
#
# SCSI support
#
# CONFIG_SCSI is not set
#
# I2O device support
#
# CONFIG_I2O is not set
# CONFIG_I2O_PCI is not set
# CONFIG_I2O_BLOCK is not set
# CONFIG_I2O_SCSI is not set
#
# Network device support
#
CONFIG_NETDEVICES=y
#
# ARCnet devices
#
# CONFIG_ARCNET is not set
CONFIG_DUMMY=m
CONFIG_BONDING=m
# CONFIG_EQUALIZER is not set
# CONFIG_ETHERTAP is not set
# CONFIG_NET_SB1000 is not set
#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_NET_VENDOR_3COM=y
# CONFIG_EL1 is not set
# CONFIG_EL2 is not set
# CONFIG_ELPLUS is not set
# CONFIG_EL16 is not set
CONFIG_EL3=m
# CONFIG_3C515 is not set
CONFIG_VORTEX=m
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_NET_VENDOR_RACAL is not set
# CONFIG_RTL8139 is not set
# CONFIG_RTL8139TOO is not set
# CONFIG_NET_ISA is not set
CONFIG_NET_EISA=y
# CONFIG_PCNET32 is not set
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_CS89x0 is not set
# CONFIG_DM9102 is not set
# CONFIG_DE4X5 is not set
# CONFIG_DEC_ELCP is not set
# CONFIG_DEC_ELCP_OLD is not set
# CONFIG_DGRS is not set
CONFIG_EEXPRESS_PRO100=m
# CONFIG_LNE390 is not set
# CONFIG_NE3210 is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_SIS900 is not set
# CONFIG_ES3210 is not set
# CONFIG_EPIC100 is not set
# CONFIG_ZNET is not set
# CONFIG_NET_POCKET is not set
#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_SK98LIN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_RADIO is not set
#
# Token ring devices
#
# CONFIG_TR is not set
# CONFIG_NET_FC is not set
# CONFIG_RCPCI is not set
# CONFIG_SHAPER is not set
#
# Wan interfaces
#
# CONFIG_HOSTESS_SV11 is not set
# CONFIG_COSA is not set
# CONFIG_SEALEVEL_4021 is not set
# CONFIG_SYNCLINK_SYNCPPP is not set
# CONFIG_LANMEDIA is not set
# CONFIG_COMX is not set
# CONFIG_HDLC is not set
# CONFIG_DLCI is not set
# CONFIG_XPEED is not set
# CONFIG_SBNI is not set
#
# Amateur Radio support
#
# CONFIG_HAMRADIO is not set
#
# IrDA (infrared) support
#
# CONFIG_IRDA is not set
#
# ISDN subsystem
#
# CONFIG_ISDN is not set
#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set
#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_SERIAL=y
# CONFIG_SERIAL_CONSOLE is not set
# CONFIG_SERIAL_EXTENDED is not set
# CONFIG_SERIAL_NONSTANDARD is not set
CONFIG_UNIX98_PTYS=y
CONFIG_UNIX98_PTY_COUNT=2048
CONFIG_MOUSE=y
#
# Mice
#
# CONFIG_ATIXL_BUSMOUSE is not set
# CONFIG_BUSMOUSE is not set
# CONFIG_MS_BUSMOUSE is not set
CONFIG_PSMOUSE=y
# CONFIG_82C710_MOUSE is not set
# CONFIG_PC110_PAD is not set
#
# Joysticks
#
# CONFIG_JOYSTICK is not set
# CONFIG_QIC02_TAPE is not set
# CONFIG_WATCHDOG is not set
CONFIG_NVRAM=m
CONFIG_RTC=y
# CONFIG_INTEL_RNG is not set
# CONFIG_AGP is not set
# CONFIG_DRM is not set
#
# Video For Linux
#
# CONFIG_VIDEO_DEV is not set
# CONFIG_DTLK is not set
#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set
#
# USB support
#
# CONFIG_USB is not set
#
# Filesystems
#
CONFIG_QUOTA=y
# CONFIG_AUTOFS_FS is not set
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
CONFIG_FAT_FS=m
# CONFIG_MSDOS_FS is not set
# CONFIG_UMSDOS_FS is not set
CONFIG_VFAT_FS=m
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
# CONFIG_MINIX_FS is not set
# CONFIG_NTFS_FS is not set
# CONFIG_HPFS_FS is not set
CONFIG_PROC_FS=y
CONFIG_DEVPTS_FS=y
# CONFIG_QNX4FS_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_EXT2_FS=y
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_EFS_FS is not set
#
# Network File Systems
#
# CONFIG_CODA_FS is not set
CONFIG_NFS_FS=m
# CONFIG_NFS_V3 is not set
CONFIG_NFSD=m
# CONFIG_NFSD_V3 is not set
# CONFIG_NFSD_TCP is not set
CONFIG_SUNRPC=m
CONFIG_LOCKD=m
# CONFIG_SMB_FS is not set
# CONFIG_NCP_FS is not set
#
# Partition Types
#
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MAC_PARTITION is not set
# CONFIG_SMD_DISKLABEL is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
CONFIG_NLS=y
#
# Native Language Support
#
CONFIG_NLS_DEFAULT="cp437"
# CONFIG_NLS_CODEPAGE_437 is not set
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_ISO8859_1 is not set
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
#
# Console drivers
#
CONFIG_VGA_CONSOLE=y
CONFIG_VIDEO_SELECT=y
# CONFIG_MDA_CONSOLE is not set
# CONFIG_FB is not set
#
# Sound
#
# CONFIG_SOUND is not set
#
# Security options
#
CONFIG_SECURE_STACK=y
CONFIG_SECURE_STACK_SMART=y
CONFIG_SECURE_LINK=y
CONFIG_SECURE_FIFO=y
CONFIG_SECURE_PROC=y
CONFIG_SECURE_FD_0_1_2=y
CONFIG_SECURE_RLIMIT_NPROC=y
# CONFIG_SECURE_SHM is not set
#
# Kernel hacking
#
CONFIG_MAGIC_SYSRQ=y
--
Pegasus
> At the moment it is 2.2.18 + all pathces that can be found at
> http://www.linuxraid.org and ide patch 2.2.18.1221, but i first spotted this
If you applied raid-0.90 and VM_global did you sort out the fact both of
them try to use the same free bit for extra buffer flags ?
On Fri, Dec 29, 2000 at 09:40:54AM +0100, Jure Pecar wrote:
> problem on a similary configured 2.2.17 with VM-global patch 3. gcc
Good. Can you try to reproduce with 2.2.19pre3? (if you absolutely need raid
0.90 you can try again with 2.2.19pre3aa3 after backing out 04_wake-one-3 that
introduces a deadlocks spotted by Chris and that I'm debugging right now)
> select(7, [5], [6], NULL, {98, 547000}) = 1 (in [5], left {98, 210000})
num_ready = fdwatch( tmr_mstimeout( &tv ) );
if ( num_ready < 0 )
{
if ( errno == EINTR )
continue; /* try again */
syslog( LOG_ERR, "fdwatch - %m" );
exit( 1 );
}
> gettimeofday({978078109, 324744}, NULL) = 0
(void) gettimeofday( &tv, (struct timezone*) 0 );
> accept(5, {sin_family=AF_INET, sin_port=htons(3687),
> sin_addr=inet_addr("62.76.36.242")}, [16]) = 7
httpd_get_conn( httpd_server* hs, int listen_fd, httpd_conn* hc )
[..]
hc->conn_fd = accept( listen_fd, &sa.sa, &sz );
> fcntl(7, F_SETFD, FD_CLOEXEC) = 0
(void) fcntl( hc->conn_fd, F_SETFD, 1 );
> fcntl(7, F_GETFL) = 0x2 (flags O_RDWR)
flags = fcntl( c->hc->conn_fd, F_GETFL, 0 );
> fcntl(7, F_SETFL, O_RDWR|O_NONBLOCK) = 0
else if ( fcntl( c->hc->conn_fd, F_SETFL, flags | O_NDELAY ) < 0 )
> brk(0x8078000) = 0x8077000
> brk(0x8078000) = 0x8077000
for some unknown reason it doesn't notice it has to handle the new connection.
> time([978078109]) = 978078109
> getpid() = 22061
> send(3, "<27>Dec 29 09:21:49 thttpd[22061"..., 69, 0) = 69
shut_down();
syslog( LOG_NOTICE, "exiting" );
> munmap(0x125000, 4096) = 0
> _exit(1) = ?
However according to the latest sources (2.20b) shut_down() should at least
call gettimeofday at once and that's not the case for you:
shut_down( void )
{
int cnum;
struct timeval tv;
#ifdef STATS_TIME
show_stats( JunkClientData, (struct timeval*) 0 );
#endif /* STATS_TIME */
(void) gettimeofday( &tv, (struct timezone*) 0 );
So you aren't using thttpd version 2.20b (or it's not the vanilla source).
Well, this would look like an userspace bug, but I understand it's strange that
it works well for three days.... Can you try to upgrade thttpd to the latest
version compiled from vanilla sources?
But regardless of the userspace upgrade, I still recommend to upgrade to
2.2.19pre3 or 2.2.19pre3aa3 minus 04_wake-one-3 (once I'll fix wake-one-3 I'll
release -4 revision and pre3aa4).
Thanks,
Andrea
On Fri, Dec 29, 2000 at 09:38:40AM +0200, Petru Paler wrote:
> This is one of the main thttpd design points: run in a select() loop. Since
> it is intended for mainly static workloads, it performs quite well...
It can't scale in SMP.
Andrea
Andrea Arcangeli wrote:
> Petru Paler wrote:
> > This is one of the main thttpd design points: run in a select() loop. Since
> > it is intended for mainly static workloads, it performs quite well...
>
> It can't scale in SMP.
thttpd is i/o bound, not CPU bound, so there's no need for SMP scaling.
It's an effective little server as long as the active document
tree fits in RAM.
If it ain't broke, don't tell people how to fix it :-)
(If for some reason SMP scaling was desirable, the thttpd
way to do it would be to introduce one thread for each
CPU, and have each thread run the same select() loop.)
- Dan
On Fri, Dec 29, 2000 at 04:53:40PM +0100, Andrea Arcangeli wrote:
> On Fri, Dec 29, 2000 at 09:38:40AM +0200, Petru Paler wrote:
> > This is one of the main thttpd design points: run in a select() loop. Since
> > it is intended for mainly static workloads, it performs quite well...
>
> It can't scale in SMP.
No one said it does, but it works nicely on UP.
--
Petru Paler, mailto:[email protected]
http://www.ppetru.net - ICQ: 41817235
On Fri, Dec 29, 2000 at 08:04:21PM +0200, Petru Paler wrote:
> On Fri, Dec 29, 2000 at 04:53:40PM +0100, Andrea Arcangeli wrote:
> > On Fri, Dec 29, 2000 at 09:38:40AM +0200, Petru Paler wrote:
> > > This is one of the main thttpd design points: run in a select() loop. Since
> > > it is intended for mainly static workloads, it performs quite well...
> >
> > It can't scale in SMP.
>
> No one said it does, but it works nicely on UP.
What ?
The TCP stack is threaded, so things like checksum calculation will
take advantage of multiple processors - right ?
Thes rest of the work is roughly copying data that isn't already
cached from the disk into memory. Well, you have one disk so threads
will buy you zero there.
Unless you do blocking I/O on the files or on the sockets, I fail to
see how threads could possibly boost the performance on a web server
that serves *static*only* pages.
99% of the real work is done by the kernel, so whether you have your
user-space application threaded or not should not be an issue - the
way that I see it...
Andrea, or someone else, would you care to enlighten me on that ?
(The reason I'm curious is because I'm about a month away from implementing
something that would run high-bandwidth TCP transfers and I'm planning
on keeping it single-threaded - unless someone can tell me that's a bad
idea)
Thanks,
--
................................................................
: [email protected] : And I see the elder races, :
:.........................: putrid forms of man :
: Jakob ?stergaard : See him rise and claim the earth, :
: OZ9ABN : his downfall is at hand. :
:.........................:............{Konkhra}...............:
On Fri, Dec 29, 2000 at 07:13:28PM +0100, Jakob ?stergaard wrote:
> > > It can't scale in SMP.
> >
> > No one said it does, but it works nicely on UP.
>
> What ?
Maybe you got me wrong (my english isnt that good): I said that it does
not scale on SMP, but it works just fine on UP.
> The TCP stack is threaded, so things like checksum calculation will
> take advantage of multiple processors - right ?
Wrong. "Threaded" TCP/IP stack -> fine grained locking, not "multiple
threads".
> Thes rest of the work is roughly copying data that isn't already
> cached from the disk into memory. Well, you have one disk so threads
> will buy you zero there.
>
> Unless you do blocking I/O on the files or on the sockets, I fail to
> see how threads could possibly boost the performance on a web server
> that serves *static*only* pages.
They do boost performance on SMP (because you can have N (N=nr. of CPUs)
threads serving data).
> (The reason I'm curious is because I'm about a month away from implementing
> something that would run high-bandwidth TCP transfers and I'm planning
> on keeping it single-threaded - unless someone can tell me that's a bad
> idea)
Keep it single threaded if you run on UP only...
--
Petru Paler, mailto:[email protected]
http://www.ppetru.net - ICQ: 41817235
> On Fri, Dec 29, 2000 at 09:38:40AM +0200, Petru Paler wrote:
> > This is one of the main thttpd design points: run in a select() loop. Since
> > it is intended for mainly static workloads, it performs quite well...
>
> It can't scale in SMP.
Your cgi will keep the other CPU occupied, or run two of them. thttpd has
superb scaling properties compared to say apache.
> They do boost performance on SMP (because you can have N (N=3Dnr. of CP=
> Us)
> threads serving data).
Depends on memory bandwidth, caches, locking overhead and a million other
issues.
> > on keeping it single-threaded - unless someone can tell me that's a =
> bad
> > idea)
>
> Keep it single threaded if you run on UP only...
Theory - SMP needs threading
Practice is generally a little different
On Fri, Dec 29, 2000 at 06:50:18PM +0000, Alan Cox wrote:
> Your cgi will keep the other CPU occupied, or run two of them. thttpd has
> superb scaling properties compared to say apache.
I think with 8 CPUs and 8 NICs (usual benchmark setup) you want more than 1 cpu
serving static data and it should be more efficient if it's threaded and
sleeping in accept() instead of running eight of them (starting from sharing
tlb entries and avoiding flushes probably without the need of CPU binding).
Andrea
On Fri, Dec 29, 2000 at 06:56:09PM +0000, Alan Cox wrote:
> Depends on memory bandwidth, [..]
BTW, it could as well use TCP_CORK + sendfile that will become truly zero copy
eventually.
Andrea
On Fri, Dec 29, 2000 at 08:06:57PM +0100, Andrea Arcangeli wrote:
> On Fri, Dec 29, 2000 at 06:50:18PM +0000, Alan Cox wrote:
> > Your cgi will keep the other CPU occupied, or run two of them. thttpd has
> > superb scaling properties compared to say apache.
>
> I think with 8 CPUs and 8 NICs (usual benchmark setup) you want more than 1 cpu
That's a good benchmark setup when the benchmark requires a single machine.
In the real world it often makes a lot of sense though to use a cluster
of cheap single CPU machines behind a load balancer (gives you better fault
tolerance and is likely cheaper)
For these thttpd is a nice web server.
-Andi
I wouldn't use cheap single CPU/NIC machines in the real world. Certainly a
cluster of smaller machines does better then one big machine, but that
doesn't mean you should use too small of a machine. Remember that as the
number of cluster nodes goes up the cluster becomes a management problem. I
would use a cluster of dual CPU/NIC machines instead. This gives you the
benefits of SMP machines along with the benefits of clusters. It should
also allow your cluster to handle more load with less nodes then single
CPU/NIC machines.
With that being said Linux is proving me wrong in the real world. Our
application server is dependant on pthreads (really just cloned process) and
as such doesn't scale well with Linux 2.2 on a dual CPU machine. Our
benchmarks show that we can handle more load on a single CPU machine then a
dual CPU one with Linux 2.2. However, it is encouraging to see that the
situation is reversed when using Linux 2.4.
-Matt
> -----Original Message-----
> From: Andi Kleen [mailto:[email protected]]
> Sent: Friday, December 29, 2000 11:14 AM
> To: Andrea Arcangeli
> Cc: Alan Cox; Petru Paler; Jure Pecar; [email protected];
> [email protected]
> Subject: Re: linux 2.2.19pre and thttpd (VM-global problem?)
>
>
> On Fri, Dec 29, 2000 at 08:06:57PM +0100, Andrea Arcangeli wrote:
> > On Fri, Dec 29, 2000 at 06:50:18PM +0000, Alan Cox wrote:
> > > Your cgi will keep the other CPU occupied, or run two of
> them. thttpd has
> > > superb scaling properties compared to say apache.
> >
> > I think with 8 CPUs and 8 NICs (usual benchmark setup) you
> want more than 1 cpu
>
> That's a good benchmark setup when the benchmark requires a
> single machine.
>
> In the real world it often makes a lot of sense though to use
> a cluster
> of cheap single CPU machines behind a load balancer (gives
> you better fault
> tolerance and is likely cheaper)
>
> For these thttpd is a nice web server.
>
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in
> the body of a message to [email protected]
> Please read the FAQ at http://www.tux.org/lkml/
>
On Fri, Dec 29, 2000 at 11:29:12AM -0800, Matt Liotta wrote:
> as such doesn't scale well with Linux 2.2 on a dual CPU machine. Our
> benchmarks show that we can handle more load on a single CPU machine then a
> dual CPU one with Linux 2.2. However, it is encouraging to see that the
If for whatever reason you can't use 2.4.x on it right now, in 2.2.19pre3aa4
there's an hack to do tcp_sendmsg checksum and all the copy_user (copies
between pagecache and userspace) with the big kernel lock released (so that it
can scale better in SMP). That's ugly but people on this list asked for this
feature and since it was very fast to implement it I added it. But don't expect
anything like 2.4.x SMP scalability!
Andrea
On Fri, 29 Dec 2000, Daniel R. Kegel wrote:
> Andrea Arcangeli wrote:
> > Petru Paler wrote:
> > > This is one of the main thttpd design points: run in a select() loop. Since
> > > it is intended for mainly static workloads, it performs quite well...
> >
> > It can't scale in SMP.
>
> thttpd is i/o bound, not CPU bound, so there's no need for SMP scaling.
> It's an effective little server as long as the active document
> tree fits in RAM.
>
> If it ain't broke, don't tell people how to fix it :-)
>
> (If for some reason SMP scaling was desirable, the thttpd
> way to do it would be to introduce one thread for each
> CPU, and have each thread run the same select() loop.)
One could possibly cheat and use fork() instead of using threads.
Do your listen() before forking..then both get the listener socket..
Threads aren't always the answer, in this case it wouldn't probably gain
you anything over just doing a fork() on SMP. Especially if you got not
good reason to be sharing data that closely, which I don't think thttpd
would.
Aaron
On Fri, Dec 29, 2000 at 08:06:57PM +0100, Andrea Arcangeli wrote:
> On Fri, Dec 29, 2000 at 06:50:18PM +0000, Alan Cox wrote:
> > Your cgi will keep the other CPU occupied, or run two of them. thttpd has
> > superb scaling properties compared to say apache.
>
> I think with 8 CPUs and 8 NICs (usual benchmark setup) you want more than 1 cpu
> serving static data and it should be more efficient if it's threaded and
> sleeping in accept() instead of running eight of them (starting from sharing
> tlb entries and avoiding flushes probably without the need of CPU binding).
Ok - my point was (and maybe I'm just misunderstanding something):
Multiple threads do send() etc. -> this gets serialized by the kernel
anyway, as TCP checksums are done in the kernel as well as the NIC
driving etc. If the TCP layer cannot parallelize checksum calculation.
One thread does many send()'s -> this also gets serialized by the
kernel.
Or: one thread vs. many threads reading from the disk -> makes no difference
since there is *one* buffer layer, one driver, one disk etc. etc.
My question is: Does it change anything in the way the TCP stack
can crank out packets, if I change the number of threads in user-space
making the system calls ?
Does the fine-grained locking allow multiple user-space threads to
actually have TCP checksums etc. done on multiple CPUs concurrently,
and is it true that this will not happen if I have one user-space
thread doing non-blocking send() calls ?
Thanks,
--
................................................................
: [email protected] : And I see the elder races, :
:.........................: putrid forms of man :
: Jakob ?stergaard : See him rise and claim the earth, :
: OZ9ABN : his downfall is at hand. :
:.........................:............{Konkhra}...............:
On Fri, 29 Dec 2000, Andrea Arcangeli wrote:
> On Fri, Dec 29, 2000 at 06:50:18PM +0000, Alan Cox wrote:
> > Your cgi will keep the other CPU occupied, or run two of them. thttpd has
> > superb scaling properties compared to say apache.
>
> I think with 8 CPUs and 8 NICs (usual benchmark setup) you want more than 1 cpu
> serving static data and it should be more efficient if it's threaded and
> sleeping in accept() instead of running eight of them (starting from sharing
> tlb entries and avoiding flushes probably without the need of CPU binding).
hey, nobody sane runs an 8 CPU box with 8 NICs for a production webserver.
8 single CPU boxes, or 4 dual boxes behind a load balancer. now that's
more common, more scalable, more robust. :)
oh yeah they all run perl, java, or php too :) i've seen sites with more
than 100 dynamic front-ends and a pair of 350Mhz x86 boxes in the corner
handling all the static needs (running apache even!). a pair only 'cause
of redundancy reasons, not because of load reasons.
100Mbits/s of *transit* internet bandwidth costs US$75000 per month in
typical datacenters btw. obviously there's cheaper bandwidth if you push
out to the edges of the net, into the central offices.
this isn't to say that nobody will ever want phat bandwidth on a single
webserver... but i'd say linux and most everyone else is at least 3 years
ahead of the long-haul networks in terms of ability to pump data. the
true value of zero-copy TCP will be more apparent in the upcoming age of 1
Mbit/s video thanks to MPEG4. i'd expect to see the cable/dsl
conglomerates start doing this in their central offices soon.
the RAM and CPU hungry dynamic content languages tend to dictate the sheer
quantity of machines required to handle even small web volumes -- folks
quickly exceed the reasonably priced SMP systems available. the cost per
CPU, and per Mb RAM are the limiting factors (these go up quickly when you
put all CPUs and RAM on the bus). that plus the desire for reliability /
lack of a single failure point mean that web development tools have grown
up expecting to be on a distributed network of nodes rather than on a
large SMP system.
the 8 CPU monsters run stuff like NFS/filesystems and oracle. some day
someone will build a horizontally scalable database which works, with a
reasonable licensing policy and then we'll see even more value in
commodity hardware/networking. this is an area i'm currently quite
interested in, pointers to research / interesting projects appreciated.
i know about globalfilesystem.org, veritas clustered fs, oracle parallel
server, and IBM DB2 EEE already -- none are quite to the point where i'd
use them to scale something across 100 nodes yet though.
-dean
On Fri, Dec 29, 2000 at 08:21:12PM -0800, dean gaudet wrote:
> On Fri, 29 Dec 2000, Andrea Arcangeli wrote:
>
> > On Fri, Dec 29, 2000 at 06:50:18PM +0000, Alan Cox wrote:
> > > Your cgi will keep the other CPU occupied, or run two of them. thttpd has
> > > superb scaling properties compared to say apache.
> >
> > I think with 8 CPUs and 8 NICs (usual benchmark setup) you want more than 1 cpu
^^^^^^^^^^^^^^^^^^^^^
> > serving static data and it should be more efficient if it's threaded and
> > sleeping in accept() instead of running eight of them (starting from sharing
> > tlb entries and avoiding flushes probably without the need of CPU binding).
>
> hey, nobody sane runs an 8 CPU box with 8 NICs for a production webserver.
> 8 single CPU boxes, or 4 dual boxes behind a load balancer. now that's
> more common, more scalable, more robust. :)
and it also provides high avaibility that is mandatory in a setup that
needs high performance anyways.
> oh yeah they all run perl, java, or php too :) i've seen sites with more
and zope+python as well indeed.
> than 100 dynamic front-ends and a pair of 350Mhz x86 boxes in the corner
> handling all the static needs (running apache even!). a pair only 'cause
> of redundancy reasons, not because of load reasons.
Exactly. I totally agree with you (everybody who talked with me about those
issues or attended my last two talks can confirm I agree ;).
But you're talking about real world. I was only talking about benchmarks.
Don't complain me if usual benchmark setup is done with one server N-way SMP +
N-Gbit-NICs (infact I can't run any usual webserving benchmark at home).
Andrea
Hi again,
I can't manage to reproduce the problem on my home box, based on redhat7
... thttpd runs ok on 2.2.18 with raid patch, 2.2.18-cdhs
(http://www.linuxraid.org) and 2.2.19pre3aa4 ... I tought it might be some
compiler/glibc problem, but even if i get a kernel and a statically
compiled thttpd from the box that is making problems(rh6.0) and run it
here, it runs ok ...
What more can i try? I'd really like to find out what's going on ...
I checked those bits Alan Cox mentioned and cdhs patch puts them like
this (include/linux/fs.h btw):
#define BH_LowPrio 7 /* 1 if the buffer is lowprio */
#define BH_Wait_IO 8 /* 1 if we should throttle on this
buffer */
Andrea, in your pre3aa4 patch you put them vice versa:
#define BH_Wait_IO 7 /* 1 if we should throttle on this
buffer */
#define BH_LowPrio 8 /* 1 if the buffer is lowprio */
I dont think this really matters, but which way should be official? :)
and btw, happy new year to all of you.
--
Pegasus