LinuxLists.cc - possible leak in kernel 2.6.10-ac12

2005-02-16 23:27:07

Subject: possible leak in kernel 2.6.10-ac12

hi everyone,

I maintain a couple of linux servers in an academic environment.
Having upgraded most of them to 2.6.10-ac12, one of them showed a linear
growth of used memory over the last 7 days, after the first 2.6.10-ac12
boot. It came to a point that it started swapping and the swap usage too
started to grow linearly.

Most of the userspace memory was occupied by postfix and spamassassin
processes, which after killed didn't stop the memory growth, so I'm left
with the feeling of a kernel memory leak.

This is mostly an e-mail server:
* smtp for about 300 hosts (postfix)
* 6 local e-mail accounts for in-house administrators
** spamassassin for the local accounts
** courier-imap-ssl for the local accounts
Running on a 2.0GHz P4 with 512Mb of ram.
kernel version is 2.6.10-ac12 [1]
kernel config: [2]

Since this is a production machine, I can't test it very thouroughly
(lots of reboots, etc), but I'd like to know more about this.

I had to reboot it, so it's all normal now.

If the behavior maintains, how can I infer if there is really a memory
leak? What kind of tests could I perform?

best regards,
pedro venda.

----------------------------------------------------------------------

[1]: ver_linux output
Linux admin 2.6.10-ac12 #1 Wed Feb 9 01:53:22 WET 2005 i686 Intel(R)
Pentium(R)
4 CPU 2.00GHz GenuineIntel GNU/Linux

Gnu C 3.3.5
Gnu make 3.80
binutils 2.15.92.0.2
util-linux 2.12i
mount 2.12i
module-init-tools 3.0
e2fsprogs 1.35
reiserfsprogs 3.6.19
reiser4progs line
Linux C Library 2.3.4
Dynamic linker (ldd) 2.3.4
Procps 3.2.4
Net-tools 1.60
Kbd 1.12
Sh-utils 5.2.1
Modules Loaded ipt_state ip_conntrack iptable_filter ip_tables
eepro100
i2c_i801 i2c_core 8139too crc32

[2]: kernel config
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_SYSCTL=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_LOG_BUF_SHIFT=14
CONFIG_KOBJECT_UEVENT=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_KALLSYMS=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_OBSOLETE_MODPARM=y
CONFIG_KMOD=y
CONFIG_X86_PC=y
CONFIG_MPENTIUM4=y
CONFIG_X86_HZ=1000
CONFIG_X86_GENERIC=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_HPET_TIMER=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=m
CONFIG_MICROCODE=m
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_NOHIGHMEM=y
CONFIG_MTRR=y
CONFIG_PM=y
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=m
CONFIG_ACPI_PROCESSOR=m
CONFIG_ACPI_THERMAL=m
CONFIG_ACPI_IBM=m
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y
CONFIG_APM=m
CONFIG_APM_DO_ENABLE=y
CONFIG_APM_CPU_IDLE=y
CONFIG_APM_DISPLAY_BLANK=y
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=m
CONFIG_CPU_FREQ_TABLE=y
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_P4_CLOCKMOD=m
CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
CONFIG_ISA=y
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
CONFIG_PARPORT_PC_CML1=y
CONFIG_PARPORT_PC_FIFO=y
CONFIG_PARPORT_PC_SUPERIO=y
CONFIG_PARPORT_OTHER=y
CONFIG_PARPORT_1284=y
CONFIG_PNP=y
CONFIG_ISAPNP=y
CONFIG_PNPBIOS=y
CONFIG_PNPBIOS_PROC_FS=y
CONFIG_PNPACPI=y
CONFIG_BLK_DEV_FD=y
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_INITRAMFS_SOURCE=""
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_BLK_DEV_IDECD=y
CONFIG_IDE_TASK_IOCTL=y
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_GENERIC=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_IDEDMA_PCI_AUTO=y
CONFIG_BLK_DEV_PIIX=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_AUTO=y
CONFIG_SCSI=y
CONFIG_SCSI_QLA2XXX=y
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_RAID1=y
CONFIG_IEEE1394=m
CONFIG_IEEE1394_EXTRA_CONFIG_ROMS=y
CONFIG_IEEE1394_CONFIG_ROM_IP1394=y
CONFIG_IEEE1394_OHCI1394=m
CONFIG_IEEE1394_VIDEO1394=m
CONFIG_IEEE1394_SBP2=m
CONFIG_IEEE1394_ETH1394=m
CONFIG_IEEE1394_DV1394=m
CONFIG_IEEE1394_RAWIO=m
CONFIG_IEEE1394_CMP=m
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_NET_KEY=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_TUNNEL=m
CONFIG_IP_TCPDIAG=y
CONFIG_IP_TCPDIAG_IPV6=y
CONFIG_IPV6=y
CONFIG_IPV6_PRIVACY=y
CONFIG_INET6_AH=y
CONFIG_INET6_ESP=y
CONFIG_INET6_IPCOMP=y
CONFIG_INET6_TUNNEL=y
CONFIG_IPV6_TUNNEL=y
CONFIG_NETFILTER=y
CONFIG_IP_NF_CONNTRACK=m
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_TFTP=m
CONFIG_IP_NF_AMANDA=m
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_REALM=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
CONFIG_IP_NF_NAT_LOCAL=y
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_NAT_TFTP=m
CONFIG_IP_NF_NAT_AMANDA=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_TARGET_NOTRACK=m
CONFIG_IP_NF_ARPTABLES=y
CONFIG_IP_NF_ARPFILTER=y
CONFIG_IP_NF_ARP_MANGLE=y
CONFIG_IP6_NF_QUEUE=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_LIMIT=m
CONFIG_IP6_NF_MATCH_MAC=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_MULTIPORT=m
CONFIG_IP6_NF_MATCH_OWNER=m
CONFIG_IP6_NF_MATCH_MARK=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_AHESP=m
CONFIG_IP6_NF_MATCH_LENGTH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_TARGET_MARK=m
CONFIG_IP6_NF_RAW=m
CONFIG_XFRM=y
CONFIG_NET_CLS_ROUTE=y
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_NET_VENDOR_3COM=y
CONFIG_EL3=m
CONFIG_VORTEX=m
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_TULIP=m
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_NET_PCI=y
CONFIG_EEPRO100=m
CONFIG_8139CP=m
CONFIG_8139TOO=m
CONFIG_8139TOO_PIO=y
CONFIG_8139TOO_TUNE_TWISTER=y
CONFIG_8139TOO_8129=y
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_ACPI=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_PRINTER=y
CONFIG_HW_RANDOM=y
CONFIG_NVRAM=y
CONFIG_RTC=y
CONFIG_HPET=y
CONFIG_HPET_RTC_IRQ=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=y
CONFIG_I2C=m
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCF=m
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_ELEKTOR=m
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_ISA=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_PROSAVAGE=m
CONFIG_I2C_SAVAGE4=m
CONFIG_SCx200_ACB=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
CONFIG_I2C_SENSOR=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_FSCHER=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83627HF=m
CONFIG_FB=y
CONFIG_FB_VESA=y
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_MONO=y
CONFIG_LOGO_LINUX_VGA16=y
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
CONFIG_REISERFS_PROC_INFO=y
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_FS_POSIX_ACL=y
CONFIG_QUOTA=y
CONFIG_QFMT_V1=m
CONFIG_QFMT_V2=m
CONFIG_QUOTACTL=y
CONFIG_DNOTIFY=y
CONFIG_AUTOFS4_FS=y
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=860
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-15"
CONFIG_NTFS_FS=y
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
CONFIG_DEVFS_FS=y
CONFIG_DEVFS_MOUNT=y
CONFIG_DEVPTS_FS_XATTR=y
CONFIG_DEVPTS_FS_SECURITY=y
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_SMB_FS=m
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-15"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_850=y
CONFIG_NLS_CODEPAGE_860=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_15=y
CONFIG_EARLY_PRINTK=y
CONFIG_4KSTACKS=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_AES_586=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_TEST=m
CONFIG_CRC_CCITT=m
CONFIG_CRC32=m
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y

--

Pedro Jo?o Lopes Venda
email: pjvenda < at > arrakis.dhis.org
http://arrakis.dhis.org

2005-02-16 23:36:02

by Parag Warudkar

[permalink] [raw]

Subject: Re: possible leak in kernel 2.6.10-ac12

On Wednesday 16 February 2005 06:28 pm, Pedro Venda wrote:
> Having upgraded most of them to 2.6.10-ac12, one of them showed a linear
> growth of used memory over the last 7 days, after the first 2.6.10-ac12
> boot. It came to a point that it started swapping and the swap usage too
> started to grow linearly.

cat /proc/slabinfo please. I am also seeing similar symptoms (although that is
with 2.6.11-rc4 there is a possibility of a common bug) here and I seem to
have linearly growing size-64 in slabinfo.

Parag

2005-02-17 00:00:43

by Pedro Venda

[permalink] [raw]

Subject: Re: possible leak in kernel 2.6.10-ac12

Parag Warudkar wrote:
> On Wednesday >
>>Having >>growth >>boot. It >>started to grow linearly.
>
>
> cat /proc/slabinfo > with 2.6.11-rc4 > have linearly

there isn't much

admin proc # uptime
23:56:08 up 1:50,

but here goes slabinfo anyway:

admin proc # cat slabinfo
slabinfo - version: 2.1
# name ip_conntrack_expect ip_conntrack xfrm6_tunnel_spi fib6_nodes ip6_dst_cache ndisc_cache rawv6_sock udpv6_sock tcpv6_sock unix_sock tcp_tw_bucket tcp_bind_bucket tcp_open_request inet_peer_cache secpath_cache xfrm_dst_cache ip_fib_alias ip_fib_hash ip_dst_cache arp_cache raw_sock udp_sock tcp_sock flow_cache cfq_ioc_pool cfq_pool crq_pool deadline_drq as_arq mqueue_inode_cache udf_inode_cache ntfs_big_inode_cache ntfs_inode_cache ntfs_name_cache ntfs_attr_ctx_cache ntfs_index_ctx_cache devfsd_event isofs_inode_cache fat_inode_cache fat_cache ext2_inode_cache ext2_xattr journal_handle journal_head revoke_table revoke_record ext3_inode_cache ext3_xattr reiser_inode_cache 14640 14652 dnotify_cache dquot eventpoll_pwq eventpoll_epi kioctx kiocb fasync_cache shmem_inode_cache posix_timers_cache uid_cache sgpool-128 sgpool-64 sgpool-32 sgpool-16 sgpool-8 blkdev_ioc blkdev_queue blkdev_requests biovec-(256) biovec-128 biovec-64 biovec-16 biovec-4 biovec-1 bio file_lock_cache sock_inode_cache skbuff_head_cache sock proc_inode_cache sigqueue radix_tree_node bdev_cache mnt_cache inode_cache dentry_cache filp names_cache idr_layer_cache buffer_head mm_struct vm_area_struct fs_cache files_cache signal_cache sighand_cache task_struct anon_vma pgd size-131072(DMA) size-131072 size-65536(DMA) size-65536 size-32768(DMA) size-32768 size-16384(DMA) size-16384 size-8192(DMA) size-8192 size-4096(DMA) size-4096 size-2048(DMA) size-2048 size-1024(DMA) size-1024 size-512(DMA) size-512 size-256(DMA) size-256 size-128(DMA) size-128 size-64(DMA) size-64 size-32(DMA) size-32 kmem_cache admin proc #

most likely it'll
regards,
--

Pedro Jo?o Lopes Venda
email: pjvenda http://arrakis.dhis.org

2005-02-17 00:38:55

by Noel Maddy

[permalink] [raw]

Subject: Re: possible leak in kernel 2.6.10-ac12

On Thu, Feb 17, 2005 at 12:02:33AM +0000, Pedro Venda wrote:

> admin proc # cat slabinfo
...
> biovec-1 74224 74354 16 226 1 : tunables 120 60 0 : slabdata 329 329 0
> bio 74212 74237 64 61 1 : tunables 120 60 0 : slabdata 1217 1217 0

If you're using md, you need this patch to fix a bio leak:

http://linux.bkbits.net:8080/linux-2.6/diffs/drivers/md/[email protected]

Index: ac-dev/drivers/md/md.c
===================================================================
--- ac-dev.orig/drivers/md/md.c 2005-02-07 17:50:37.000000000 -0500
+++ ac-dev/drivers/md/md.c 2005-02-08 17:49:57.000000000 -0500
@@ -336,8 +336,6 @@
struct completion event;
int ret;

- bio_get(bio);
-
rw |= (1 << BIO_RW_SYNC);

bio->bi_bdev = bdev;
--
It's a big galaxy, Mr. Scott.
-- Lieutenant Uhura
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
Noel Maddy <[email protected]>

2005-02-17 00:51:51

by Pedro Venda

[permalink] [raw]

Subject: Re: possible leak in kernel 2.6.10-ac12

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Noel Maddy wrote:
| On Thu, Feb 17, 2005 at 12:02:33AM +0000, Pedro Venda wrote:
|
|
|>admin proc # cat slabinfo
|
| ...
|
|>biovec-1 74224 74354 16 226 1 : tunables 120 60 0 :
slabdata 329 329 0
|>bio 74212 74237 64 61 1 : tunables 120 60 0 :
slabdata 1217 1217 0
|
|
| If you're using md, you need this patch to fix a bio leak:
|
| http://linux.bkbits.net:8080/linux-2.6/diffs/drivers/md/[email protected]

thanks.

has this patch been included in the recent rc's?

and howcome the other servers work normally [all with md]? at least so far the
memory usages aren't concerning, although the linear increase is starting to
show. perhaps the [different] apps running on them don't expose the leak as well
the one that broke today... is that reasonable?

regards,

- --

Pedro Jo?o Lopes Venda
email: [email protected]
http://arrakis.dhis.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCE+p/eRy7HWZxjWERArOTAKDmZ0fG1DpfN7pW9UNaVpLWK3LX2gCg0/Kr
u4kzp1PaId8tmo61oHFISuk=
=jGB3
-----END PGP SIGNATURE-----

2005-02-17 02:44:15

by Pedro Venda

[permalink] [raw]

Subject: Re: possible leak in kernel 2.6.10-ac12

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pedro Venda wrote:
| Noel Maddy wrote:
| | On Thu, Feb 17, 2005 at 12:02:33AM +0000, Pedro Venda wrote:
| |
| |
| |>admin proc # cat slabinfo
| |
| | ...
| |
| |>biovec-1 74224 74354 16 226 1 : tunables 120
| 60 0 :
| slabdata 329 329 0
| |>bio 74212 74237 64 61 1 : tunables 120
| 60 0 :
| slabdata 1217 1217 0
| |
| |
| | If you're using md, you need this patch to fix a bio leak:
| |
| | http://linux.bkbits.net:8080/linux-2.6/diffs/drivers/md/[email protected]
|
| thanks.
|
| has this patch been included in the recent rc's?
|
| and howcome the other servers work normally [all with md]? at least so
| far the
| memory usages aren't concerning, although the linear increase is
| starting to
| show. perhaps the [different] apps running on them don't expose the leak
| as well
| the one that broke today... is that reasonable?

correction: after looking at /proc/slabinfo, the counters around the bio lines
are 10-100 times bigger than all the other, so their also leaking.

I read some discussion on this list around this issue and people were not
positive the leak came from md...

any comments?

regards.
- --

Pedro Jo?o Lopes Venda
email: [email protected]
http://arrakis.dhis.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCFATzeRy7HWZxjWERApHtAKDsX1UVnKP1JICe1/RrzuzF00P84QCfac4+
Mtxq0Y31MFhdsIgBf5S0n/Y=
=u/li
-----END PGP SIGNATURE-----

2005-02-17 09:29:49

by Maciej Sołtysiak

[permalink] [raw]

Subject: Re: possible leak in kernel 2.6.10-ac12

Hello Pedro,

Thursday, February 17, 2005, 12:28:15 AM, you wrote:

> boot. It came to a point that it started swapping and the swap usage too
> started to grow linearly.
I had the same with swap being eaten especially by perl apps like qmail-scanner

I think this helps:
--- a/mm/vmscan.c 2004-12-24 13:36:18 -08:00
+++ b/mm/vmscan.c 2004-12-24 13:36:18 -08:00
@@ -675,6 +674,7 @@
}
pgscanned++;
}
+ zone->pages_scanned += pgscanned;
zone->nr_active -= pgmoved;
spin_unlock_irq(&zone->lru_lock);

This patchlet is at:
http://www.kernel.org/diff/diffview.cgi?file=%2Fpub%2Flinux%2Fkernel%2Fv2.6%2Fpatch-2.6.10.bz2;z=4918
This changeset contains other patches, you need only one.

2.6.11 will have it fixed.

Regards,
Maciej Soltysiak

2005-02-17 23:13:59

by Bill Davidsen

[permalink] [raw]

Subject: Re: possible leak in kernel 2.6.10-ac12

Parag Warudkar wrote:
> On Wednesday 16 February 2005 06:28 pm, Pedro Venda wrote:
>
>>Having upgraded most of them to 2.6.10-ac12, one of them showed a linear
>>growth of used memory over the last 7 days, after the first 2.6.10-ac12
>>boot. It came to a point that it started swapping and the swap usage too
>>started to grow linearly.
>
>
> cat /proc/slabinfo please. I am also seeing similar symptoms (although that is
> with 2.6.11-rc4 there is a possibility of a common bug) here and I seem to
> have linearly growing size-64 in slabinfo.

There has been discussion in another thread about a leak related to
network activity. You might find more information there, subject
contains "NOT BIO" if it helps.

--
-bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
last possible moment - but no longer" -me

2005-02-19 12:38:36

by Pedro Venda

[permalink] [raw]

Subject: Re: possible leak in kernel 2.6.10-ac12

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Parag Warudkar wrote:
| On Wednesday 16 February 2005 06:28 pm, Pedro Venda wrote:
|
|>Having upgraded most of them to 2.6.10-ac12, one of them showed a linear
|>growth of used memory over the last 7 days, after the first 2.6.10-ac12
|>boot. It came to a point that it started swapping and the swap usage too
|>started to grow linearly.
|
|
| cat /proc/slabinfo please. I am also seeing similar symptoms (although that is
| with 2.6.11-rc4 there is a possibility of a common bug) here and I seem to
| have linearly growing size-64 in slabinfo.
|

hi

I've read the leak thread above possibly about this bug... AFAI read, there isn't
really a conclusion about this leak, right?

here goes the /proc/slabinfo for 2 days uptime:

admin proc # uptime
~ 12:29:18 up 2 days, 14:23, 2 users, load average: 0.29, 0.38, 0.24
admin proc # cat slabinfo
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs>
<num_slabs> <sharedavail>
ip_conntrack_expect 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
ip_conntrack 230 820 384 10 1 : tunables 54 27 0 : slabdata 82 82 0
xfrm6_tunnel_spi 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
fib6_nodes 8 119 32 119 1 : tunables 120 60 0 : slabdata 1 1 0
ip6_dst_cache 6 15 256 15 1 : tunables 120 60 0 : slabdata 1 1 0
ndisc_cache 1 15 256 15 1 : tunables 120 60 0 : slabdata 1 1 0
rawv6_sock 3 6 640 6 1 : tunables 54 27 0 : slabdata 1 1 0
udpv6_sock 6 6 640 6 1 : tunables 54 27 0 : slabdata 1 1 0
tcpv6_sock 10 14 1152 7 2 : tunables 24 12 0 : slabdata 2 2 0
unix_sock 371 620 384 10 1 : tunables 54 27 0 : slabdata 62 62 0
tcp_tw_bucket 31 31 128 31 1 : tunables 120 60 0 : slabdata 1 1 0
tcp_bind_bucket 81 226 16 226 1 : tunables 120 60 0 : slabdata 1 1 0
tcp_open_request 16 31 128 31 1 : tunables 120 60 0 : slabdata 1 1 0
inet_peer_cache 3 61 64 61 1 : tunables 120 60 0 : slabdata 1 1 0
secpath_cache 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
xfrm_dst_cache 0 0 256 15 1 : tunables 120 60 0 : slabdata 0 0 0
ip_fib_alias 10 226 16 226 1 : tunables 120 60 0 : slabdata 1 1 0
ip_fib_hash 10 119 32 119 1 : tunables 120 60 0 : slabdata 1 1 0
ip_dst_cache 193 285 256 15 1 : tunables 120 60 0 : slabdata 19 19 0
arp_cache 3 31 128 31 1 : tunables 120 60 0 : slabdata 1 1 0
raw_sock 2 7 512 7 1 : tunables 54 27 0 : slabdata 1 1 0
udp_sock 14 14 512 7 1 : tunables 54 27 0 : slabdata 2 2 0
tcp_sock 21 52 1024 4 1 : tunables 54 27 0 : slabdata 13 13 0
flow_cache 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
cfq_ioc_pool 0 0 24 156 1 : tunables 120 60 0 : slabdata 0 0 0
cfq_pool 0 0 104 38 1 : tunables 120 60 0 : slabdata 0 0 0
crq_pool 0 0 52 75 1 : tunables 120 60 0 : slabdata 0 0 0
deadline_drq 0 0 48 81 1 : tunables 120 60 0 : slabdata 0 0 0
as_arq 135 260 60 65 1 : tunables 120 60 0 : slabdata 4 4 0
mqueue_inode_cache 1 7 512 7 1 : tunables 54 27 0 : slabdata 1 1 0
udf_inode_cache 0 0 348 11 1 : tunables 54 27 0 : slabdata 0 0 0
ntfs_big_inode_cache 0 0 512 7 1 : tunables 54 27 0 : slabdata 0 0 0
ntfs_inode_cache 0 0 156 25 1 : tunables 120 60 0 : slabdata 0 0 0
ntfs_name_cache 0 0 512 8 1 : tunables 54 27 0 : slabdata 0 0 0
ntfs_attr_ctx_cache 0 0 32 119 1 : tunables 120 60 0 : slabdata 0 0 0
ntfs_index_ctx_cache 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
devfsd_event 0 0 20 185 1 : tunables 120 60 0 : slabdata 0 0 0
isofs_inode_cache 0 0 320 12 1 : tunables 54 27 0 : slabdata 0 0 0
fat_inode_cache 0 0 348 11 1 : tunables 54 27 0 : slabdata 0 0 0
fat_cache 0 0 20 185 1 : tunables 120 60 0 : slabdata 0 0 0
ext2_inode_cache 0 0 420 9 1 : tunables 54 27 0 : slabdata 0 0 0
ext2_xattr 0 0 44 88 1 : tunables 120 60 0 : slabdata 0 0 0
journal_handle 0 0 20 185 1 : tunables 120 60 0 : slabdata 0 0 0
journal_head 0 0 48 81 1 : tunables 120 60 0 : slabdata 0 0 0
revoke_table 0 0 12 290 1 : tunables 120 60 0 : slabdata 0 0 0
revoke_record 0 0 16 226 1 : tunables 120 60 0 : slabdata 0 0 0
ext3_inode_cache 0 0 476 8 1 : tunables 54 27 0 : slabdata 0 0 0
ext3_xattr 0 0 44 88 1 : tunables 120 60 0 : slabdata 0 0 0
reiser_inode_cache 3620 5533 368 11 1 : tunables 54 27 0 : slabdata 503 503 0
dnotify_cache 0 0 20 185 1 : tunables 120 60 0 : slabdata 0 0 0
dquot 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
eventpoll_pwq 0 0 36 107 1 : tunables 120 60 0 : slabdata 0 0 0
eventpoll_epi 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
kioctx 0 0 256 15 1 : tunables 120 60 0 : slabdata 0 0 0
kiocb 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
fasync_cache 0 0 16 226 1 : tunables 120 60 0 : slabdata 0 0 0
shmem_inode_cache 4 10 384 10 1 : tunables 54 27 0 : slabdata 1 1 0
posix_timers_cache 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
uid_cache 5 61 64 61 1 : tunables 120 60 0 : slabdata 1 1 0
sgpool-128 32 32 2048 2 1 : tunables 24 12 0 : slabdata 16 16 0
sgpool-64 32 32 1024 4 1 : tunables 54 27 0 : slabdata 8 8 0
sgpool-32 32 32 512 8 1 : tunables 54 27 0 : slabdata 4 4 0
sgpool-16 32 45 256 15 1 : tunables 120 60 0 : slabdata 3 3 0
sgpool-8 32 62 128 31 1 : tunables 120 60 0 : slabdata 2 2 0
blkdev_ioc 74 312 24 156 1 : tunables 120 60 0 : slabdata 2 2 0
blkdev_queue 8 11 352 11 1 : tunables 54 27 0 : slabdata 1 1 0
blkdev_requests 136 196 140 28 1 : tunables 120 60 0 : slabdata 7 7 0
biovec-(256) 256 256 3072 2 2 : tunables 24 12 0 : slabdata 128 128 0
biovec-128 256 260 1536 5 2 : tunables 24 12 0 : slabdata 52 52 0
biovec-64 256 260 768 5 1 : tunables 54 27 0 : slabdata 52 52 0
biovec-16 256 270 256 15 1 : tunables 120 60 0 : slabdata 18 18 0
biovec-4 256 305 64 61 1 : tunables 120 60 0 : slabdata 5 5 0
biovec-1 1989252 1989478 16 226 1 : tunables 120 60 0 : slabdata 8803 8803 0
bio 1989270 1989271 64 61 1 : tunables 120 60 0 : slabdata 32611 32611 0
file_lock_cache 176 270 88 45 1 : tunables 120 60 0 : slabdata 6 6 0
sock_inode_cache 423 710 384 10 1 : tunables 54 27 0 : slabdata 71 71 0
skbuff_head_cache 165 165 256 15 1 : tunables 120 60 0 : slabdata 11 11 0
sock 4 10 384 10 1 : tunables 54 27 0 : slabdata 1 1 0
proc_inode_cache 686 923 308 13 1 : tunables 54 27 0 : slabdata 71 71 0
sigqueue 8 27 148 27 1 : tunables 120 60 0 : slabdata 1 1 0
radix_tree_node 4899 7854 276 14 1 : tunables 54 27 0 : slabdata 561 561 0
bdev_cache 14 14 512 7 1 : tunables 54 27 0 : slabdata 2 2 0
mnt_cache 18 31 128 31 1 : tunables 120 60 0 : slabdata 1 1 0
inode_cache 697 754 292 13 1 : tunables 54 27 0 : slabdata 58 58 0
dentry_cache 10003 20300 136 29 1 : tunables 120 60 0 : slabdata 700 700 0
filp 3840 5220 256 15 1 : tunables 120 60 0 : slabdata 348 348 0
names_cache 18 18 4096 1 1 : tunables 24 12 0 : slabdata 18 18 0
idr_layer_cache 81 87 136 29 1 : tunables 120 60 0 : slabdata 3 3 0
buffer_head 31554 45684 48 81 1 : tunables 120 60 0 : slabdata 564 564 0
mm_struct 187 210 640 6 1 : tunables 54 27 0 : slabdata 35 35 0
vm_area_struct 7446 9259 84 47 1 : tunables 120 60 0 : slabdata 197 197 0
fs_cache 169 476 32 119 1 : tunables 120 60 0 : slabdata 4 4 0
files_cache 187 210 512 7 1 : tunables 54 27 0 : slabdata 30 30 0
signal_cache 178 270 256 15 1 : tunables 120 60 0 : slabdata 18 18 0
sighand_cache 185 210 1408 5 2 : tunables 24 12 0 : slabdata 42 42 0
task_struct 189 207 1248 3 1 : tunables 24 12 0 : slabdata 69 69 0
anon_vma 3946 9361 8 407 1 : tunables 120 60 0 : slabdata 23 23 0
pgd 162 162 4096 1 1 : tunables 24 12 0 : slabdata 162 162 0
size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-131072 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
size-65536 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
size-32768 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
size-16384 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0
size-8192 1 1 8192 1 2 : tunables 8 4 0 : slabdata 1 1 0
size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 0 : slabdata 0 0 0
size-4096 414 414 4096 1 1 : tunables 24 12 0 : slabdata 414 414 0
size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 0 : slabdata 0 0 0
size-2048 100 122 2048 2 1 : tunables 24 12 0 : slabdata 61 61 0
size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 0 : slabdata 0 0 0
size-1024 244 260 1024 4 1 : tunables 54 27 0 : slabdata 65 65 0
size-512(DMA) 0 0 512 8 1 : tunables 54 27 0 : slabdata 0 0 0
size-512 165 192 512 8 1 : tunables 54 27 0 : slabdata 24 24 0
size-256(DMA) 0 0 256 15 1 : tunables 120 60 0 : slabdata 0 0 0
size-256 330 330 256 15 1 : tunables 120 60 0 : slabdata 22 22 0
size-128(DMA) 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
size-128 3863 4123 128 31 1 : tunables 120 60 0 : slabdata 133 133 0
size-64(DMA) 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
size-64 4767 5368 64 61 1 : tunables 120 60 0 : slabdata 88 88 0
size-32(DMA) 0 0 32 119 1 : tunables 120 60 0 : slabdata 0 0 0
size-32 3728 5593 32 119 1 : tunables 120 60 0 : slabdata 47 47 0
kmem_cache 124 124 128 31 1 : tunables 120 60 0 : slabdata 4 4 0
admin proc #

regards,
pedro venda.
- --

Pedro Jo?o Lopes Venda
email: pjvenda < at > arrakis.dhis.org
http://arrakis.dhis.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCFzMjeRy7HWZxjWERAqjKAJwILYhyoI/IfbKmWjfrkF2ZgSUmfQCg0jQJ
KwU6Z4/+hLa0ONF58kZ44GE=
=9shv
-----END PGP SIGNATURE-----

2005-02-19 14:48:44

by Parag Warudkar

[permalink] [raw]

Subject: Re: possible leak in kernel 2.6.10-ac12

On Saturday 19 February 2005 07:37 am, Pedro Venda wrote:
> biovec-1 ? ? ? ? ?1989252 1989478 ? ? 16 ?226 ? ?1 : tunables ?120 ? 60 ?
> ?0 : slabdata ? 8803 ? 8803 ? ? ?0 bio ? ? ? ? ? ? ? 1989270 1989271 ? ? 64
> ? 61 ? ?1 : tunables ?120 ? 60 ? ?0 : slabdata ?32611 ?32611 ? ? ?0

You have bio leak. Similar one was fixed in -rc4.
Did you already try using the md fix
http://linux.bkbits.net:8080/linux-2.6/diffs/drivers/md/[email protected]
And
http://www.kernel.org/diff/diffview.cgi?file=%2Fpub%2Flinux%2Fkernel%2Fv2.6%2Fpatch-2.6.10.bz2;z=4918
?

If not, is it possible for you to either port these fixes to -ac12 or use -rc4
and then report slabinfo after couple of days?

Parag