I'm reporting an oops. Details follow.
I have two of these machines. I will happily be anybody's guinea pig
to debug this. (more details, access to machine, try patches, kernels...)
Machines aren't in production.
- Brad
Kernel: 2.6.9-rc4 vanilla (.config below)
Hardware: IBM eServer 325, Dual Opteron 8GB ram (more info below)
Pre-crash and crash:
a1:~# mke2fs /dev/mapper/raid10-data
mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
25608192 inodes, 51200000 blocks
2560000 blocks (5.00%) reserved for the super user
First data block=0
1563 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Writing inode tables: 1091/1563
Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
localhost kernel: Oops: 0000 [1] SMP
Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
localhost kernel: CR2: 0000000000001770
Oct 13 11:46:01 localhost kernel: Unable to handle kernel paging request at 0000000000001770 RIP:
Oct 13 11:46:01 localhost kernel: <ffffffff8015efe4>{kmem_getpages+132}
Oct 13 11:46:01 localhost kernel: PML4 1f8958067 PGD 1f8ab1067 PMD 0
Oct 13 11:46:01 localhost kernel: Oops: 0000 [1] SMP
Oct 13 11:46:01 localhost kernel: CPU 1
Oct 13 11:46:01 localhost kernel: Modules linked in: af_packet mousedev tsdev joydev usbhid ohci_hcd hw_random amd74xx evdev tg3 dm_mod ide_generic ide_cd ide_core cdrom rtc isofs ext3 jbd mbcache sd_mod ips mptscsih mptbase scsi_mod unix
Oct 13 11:46:01 localhost kernel: Pid: 3176, comm: mke2fs Not tainted 2.6.9-rc4
Oct 13 11:46:01 localhost kernel: RIP: 0010:[kmem_getpages+132/432] <ffffffff8015efe4>{kmem_getpages+132}
Oct 13 11:46:01 localhost kernel: RSP: 0018:00000101f8b75aa8 EFLAGS: 00010213
Oct 13 11:46:01 localhost kernel: RAX: ffffffff7fffffff RBX: 00000101fffc9680 RCX: 0000000000000000
Oct 13 11:46:01 localhost kernel: RDX: 0000010000011700 RSI: 0000010000011a40 RDI: 0000010000012500
Oct 13 11:46:01 localhost kernel: RBP: 00000101fffc9680 R08: 000001016bc13000 R09: 00000101fffc96e8
Oct 13 11:46:01 localhost kernel: R10: 0000000000000000 R11: 00000000fffffffa R12: 00000101fffc9680
Oct 13 11:46:01 localhost kernel: R13: 0000000000000000 R14: 00000101fffc9728 R15: 0000000000000001
Oct 13 11:46:01 localhost kernel: FS: 0000002a95ddb4a0(0000) GS:ffffffff803df380(0000) knlGS:0000000000000000
Oct 13 11:46:01 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Oct 13 11:46:01 localhost kernel: CR2: 0000000000001770 CR3: 0000000008068000 CR4: 00000000000006e0
Oct 13 11:46:01 localhost kernel: Process mke2fs (pid: 3176, threadinfo 00000101f8b74000, task 00000101fec99070)
Oct 13 11:46:01 localhost kernel: Stack: 000001016c3e5000 0000000000000000 0000000000000050 ffffffff8015ff6e
Oct 13 11:46:01 localhost kernel: 0000005000000010 000000000000003c 000001001186c000 00000101fffc9680
Oct 13 11:46:01 localhost kernel: 00000101fffc96c8 00000101fffc9728
Oct 13 11:46:01 localhost kernel: Call Trace:<ffffffff8015ff6e>{cache_grow+190} <ffffffff801601c6>{cache_alloc_refill+422}
Oct 13 11:46:01 localhost kernel: <ffffffff801604b6>{kmem_cache_alloc+54} <ffffffff8017d761>{alloc_buffer_head+17}
Oct 13 11:46:01 localhost kernel: <ffffffff8017aeba>{create_buffers+42} <ffffffff8017b884>{create_empty_buffers+20}
Oct 13 11:46:01 localhost kernel: <ffffffff8017bcdf>{__block_prepare_write+175} <ffffffff80180190>{blkdev_get_block+0}
Oct 13 11:46:01 localhost kernel: <ffffffff8017c78a>{block_prepare_write+26} <ffffffff80158dc4>{generic_file_buffered_write+404}
Oct 13 11:46:01 localhost kernel: <ffffffff80193fae>{inode_update_time+158} <ffffffff801594dd>{generic_file_aio_write_nolock+765}
Oct 13 11:46:01 localhost kernel: <ffffffff801595b5>{generic_file_write_nolock+165} <ffffffff80134ef3>{__wake_up+67}
Oct 13 11:46:01 localhost kernel: <ffffffff802a46fe>{thread_return+41} <ffffffff80136890>{autoremove_wake_function+0}
Oct 13 11:46:01 localhost kernel: <ffffffff8018128a>{blkdev_file_write+26} <ffffffff801789e4>{vfs_write+228}
Oct 13 11:46:01 localhost kernel: <ffffffff80178b13>{sys_write+83} <ffffffff8011195a>{system_call+126}
Oct 13 11:46:01 localhost kernel:
Oct 13 11:46:01 localhost kernel:
Oct 13 11:46:01 localhost kernel: Code: 48 8b 91 70 17 00 00 76 07 b8 00 00 00 80 eb 0a 48 b8 00 00
Oct 13 11:46:01 localhost kernel: RIP <ffffffff8015efe4>{kmem_getpages+132} RSP <00000101f8b75aa8>
Oct 13 11:46:01 localhost kernel: CR2: 0000000000001770
a1:~# uname -a
Linux a1 2.6.9-rc4 #1 SMP Mon Oct 11 12:01:41 PDT 2004 x86_64 GNU/Linux
a1:~# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 5
model name : AMD Opteron(tm) Processor 246
stepping : 8
cpu MHz : 1994.060
cache size : 1024 KB
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxs r sse sse2 syscall nx mmxext lm 3dnowext 3dnow
bogomips : 3940.35
TLB size : 1088 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts ttp
processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 5
model name : AMD Opteron(tm) Processor 246
stepping : 8
cpu MHz : 1994.060
cache size : 1024 KB
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxs r sse sse2 syscall nx mmxext lm 3dnowext 3dnow
bogomips : 3981.31
TLB size : 1088 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts ttp
a1:~# free
total used free shared buffers cached
Mem: 8109124 128960 7980164 0 3884 19032
-/+ buffers/cache: 106044 8003080
Swap: 497964 0 497964
a1:~# cat /proc/swaps
Filename Type Size Used Priority
/dev/sda5 partition 497964 0 -1
a1:~# zcat /proc/config.gz | grep '='
CONFIG_X86_64=y
CONFIG_64BIT=y
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_X86_CMPXCHG=y
CONFIG_EARLY_PRINTK=y
CONFIG_HPET_TIMER=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_SYSCTL=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_HOTPLUG=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_EMBEDDED=y
CONFIG_KALLSYMS=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_SHMEM=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_OBSOLETE_MODPARM=y
CONFIG_MODVERSIONS=y
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y
CONFIG_MK8=y
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_TSC=y
CONFIG_X86_GOOD_APIC=y
CONFIG_MICROCODE=m
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_X86_IO_APIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_MTRR=y
CONFIG_SMP=y
CONFIG_SCHED_SMT=y
CONFIG_K8_NUMA=y
CONFIG_DISCONTIGMEM=y
CONFIG_NUMA=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_NR_CPUS=8
CONFIG_GART_IOMMU=y
CONFIG_SWIOTLB=y
CONFIG_X86_MCE=y
CONFIG_PM=y
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_SLEEP_PROC_FS=y
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_FAN=m
CONFIG_ACPI_PROCESSOR=m
CONFIG_ACPI_THERMAL=m
CONFIG_ACPI_ASUS=m
CONFIG_ACPI_TOSHIBA=m
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_PROC_INTF=m
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=m
CONFIG_CPU_FREQ_TABLE=m
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_POWERNOW_K8_ACPI=y
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_MSI=y
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
CONFIG_PCMCIA=m
CONFIG_YENTA=m
CONFIG_CARDBUS=y
CONFIG_PD6729=m
CONFIG_I82092=m
CONFIG_TCIC=m
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=y
CONFIG_COMPAT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_UID16=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_CPQ_DA=m
CONFIG_BLK_CPQ_CISS_DA=m
CONFIG_CISS_SCSI_TAPE=y
CONFIG_BLK_DEV_DAC960=m
CONFIG_BLK_DEV_UMEM=m
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_SX8=m
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=65536
CONFIG_BLK_DEV_INITRD=y
CONFIG_LBD=y
CONFIG_IDE=m
CONFIG_BLK_DEV_IDE=m
CONFIG_BLK_DEV_IDEDISK=m
CONFIG_BLK_DEV_IDECS=mbin/upgrading/migrate-userpics.pl --best-effort --verify --clusters=17
CONFIG_BLK_DEV_IDECD=m
CONFIG_BLK_DEV_IDETAPE=m
CONFIG_BLK_DEV_IDEFLOPPY=m
CONFIG_BLK_DEV_IDESCSI=m
CONFIG_IDE_GENERIC=m
CONFIG_BLK_DEV_CMD640=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_GENERIC=m
CONFIG_BLK_DEV_OPTI621=m
CONFIG_BLK_DEV_RZ1000=m
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_IDEDMA_PCI_AUTO=y
CONFIG_BLK_DEV_AEC62XX=m
CONFIG_BLK_DEV_ALI15X3=m
CONFIG_BLK_DEV_AMD74XX=m
CONFIG_BLK_DEV_ATIIXP=m
CONFIG_BLK_DEV_CMD64X=m
CONFIG_BLK_DEV_TRIFLEX=m
CONFIG_BLK_DEV_CY82C693=m
CONFIG_BLK_DEV_CS5520=m
CONFIG_BLK_DEV_CS5530=m
CONFIG_BLK_DEV_HPT34X=m
CONFIG_BLK_DEV_HPT366=m
CONFIG_BLK_DEV_SC1200=m
CONFIG_BLK_DEV_PIIX=m
CONFIG_BLK_DEV_NS87415=m
CONFIG_BLK_DEV_PDC202XX_OLD=m
CONFIG_PDC202XX_BURST=y
CONFIG_BLK_DEV_PDC202XX_NEW=m
CONFIG_PDC202XX_FORCE=y
CONFIG_BLK_DEV_SVWKS=m
CONFIG_BLK_DEV_SIIMAGE=m
CONFIG_BLK_DEV_SIS5513=m
CONFIG_BLK_DEV_SLC90E66=m
CONFIG_BLK_DEV_TRM290=m
CONFIG_BLK_DEV_VIA82CXXX=m
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_AUTO=y
CONFIG_SCSI=m
CONFIG_SCSI_PROC_FS=y
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
CONFIG_CHR_DEV_SG=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_BLK_DEV_3W_XXXX_RAID=m
CONFIG_SCSI_3W_9XXX=m
CONFIG_SCSI_ACARD=m
CONFIG_SCSI_AACRAID=m
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=8
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
CONFIG_AIC7XXX_DEBUG_ENABLE=y
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
CONFIG_SCSI_AIC7XXX_OLD=m
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=32
CONFIG_AIC79XX_RESET_DELAY_MS=15000
CONFIG_AIC79XX_ENABLE_RD_STRM=y
CONFIG_AIC79XX_DEBUG_ENABLE=y
CONFIG_AIC79XX_DEBUG_MASK=0
CONFIG_AIC79XX_REG_PRETTY_PRINT=y
CONFIG_SCSI_SATA=y
CONFIG_SCSI_SATA_SVW=m
CONFIG_SCSI_ATA_PIIX=m
CONFIG_SCSI_SATA_NV=m
CONFIG_SCSI_SATA_PROMISE=m
CONFIG_SCSI_SATA_SX4=m
CONFIG_SCSI_SATA_SIL=m
CONFIG_SCSI_SATA_SIS=m
CONFIG_SCSI_SATA_VIA=m
CONFIG_SCSI_SATA_VITESSE=m
CONFIG_SCSI_BUSLOGIC=m
CONFIG_SCSI_DMX3191D=m
CONFIG_SCSI_EATA=m
CONFIG_SCSI_EATA_TAGGED_QUEUE=y
CONFIG_SCSI_EATA_LINKED_COMMANDS=y
CONFIG_SCSI_EATA_MAX_TAGS=16
CONFIG_SCSI_EATA_PIO=m
CONFIG_SCSI_FUTURE_DOMAIN=m
CONFIG_SCSI_GDTH=m
CONFIG_SCSI_IPS=m
CONFIG_SCSI_INIA100=m
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
CONFIG_SCSI_IPR=m
CONFIG_SCSI_QLOGIC_ISP=m
CONFIG_SCSI_QLOGIC_FC=m
CONFIG_SCSI_QLOGIC_FC_FIRMWARE=y
CONFIG_SCSI_QLOGIC_1280=m
CONFIG_SCSI_QLA2XXX=m
CONFIG_SCSI_DC395x=m
CONFIG_SCSI_DC390T=m
CONFIG_SCSI_DEBUG=m
CONFIG_PCMCIA_FDOMAIN=m
CONFIG_PCMCIA_QLOGIC=m
CONFIG_PCMCIA_SYM53C500=m
CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID5=m
CONFIG_MD_RAID6=m
CONFIG_MD_MULTIPATH=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
CONFIG_DM_ZERO=m
CONFIG_FUSION=m
CONFIG_FUSION_MAX_SGE=40
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LAN=m
CONFIG_I2O=m
CONFIG_I2O_CONFIG=m
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m
CONFIG_NET=y
CONFIG_PACKET=m
CONFIG_PACKET_MMAP=y
CONFIG_NETLINK_DEV=m
CONFIG_UNIX=m
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_TUNNEL=m
CONFIG_IP_VS=m
CONFIG_IP_VS_TAB_BITS=12
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
CONFIG_IP_VS_FTP=m
CONFIG_NETFILTER=y
CONFIG_BRIDGE_NETFILTER=y
CONFIG_IP_NF_CONNTRACK=m
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_TFTP=m
CONFIG_IP_NF_AMANDA=m
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_PHYSDEV=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_REALM=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
CONFIG_IP_NF_NAT_LOCAL=y
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_NAT_TFTP=m
CONFIG_IP_NF_NAT_AMANDA=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_TARGET_NOTRACK=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
CONFIG_IP_NF_COMPAT_IPCHAINS=m
CONFIG_IP_NF_COMPAT_IPFWADM=m
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
CONFIG_IP_SCTP=m
CONFIG_SCTP_HMAC_MD5=y
CONFIG_BRIDGE=m
CONFIG_VLAN_8021Q=m
CONFIG_LLC=m
CONFIG_LLC2=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_PKTGEN=m
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
CONFIG_BONDING=m
CONFIG_EQUALIZER=m
CONFIG_TUN=m
CONFIG_ETHERTAP=m
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
CONFIG_HAPPYMEAL=m
CONFIG_SUNGEM=m
CONFIG_NET_VENDOR_3COM=y
CONFIG_VORTEX=m
CONFIG_TYPHOON=m
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_TULIP=m
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_PCMCIA_XIRCOM=m
CONFIG_HP100=m
CONFIG_NET_PCI=y
CONFIG_PCNET32=m
CONFIG_AMD8111_ETH=m
CONFIG_ADAPTEC_STARFIRE=m
CONFIG_B44=m
CONFIG_FORCEDETH=m
CONFIG_EEPRO100=m
CONFIG_E100=m
CONFIG_FEALNX=m
CONFIG_NATSEMI=m
CONFIG_NE2K_PCI=m
CONFIG_8139CP=m
CONFIG_8139TOO=m
CONFIG_8139TOO_TUNE_TWISTER=y
CONFIG_8139TOO_8129=y
CONFIG_SIS900=m
CONFIG_EPIC100=m
CONFIG_SUNDANCE=m
CONFIG_VIA_RHINE=m
CONFIG_VIA_VELOCITY=m
CONFIG_DL2K=m
CONFIG_E1000=m
CONFIG_NS83820=m
CONFIG_HAMACHI=m
CONFIG_YELLOWFIN=m
CONFIG_R8169=m
CONFIG_SK98LIN=m
CONFIG_TIGON3=m
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPPOE=m
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLIP_SMART=y
CONFIG_NET_FC=y
CONFIG_SHAPER=m
CONFIG_ISDN=m
CONFIG_ISDN_I4L=m
CONFIG_ISDN_PPP=y
CONFIG_ISDN_PPP_VJ=y
CONFIG_ISDN_MPP=y
CONFIG_IPPP_FILTER=y
CONFIG_ISDN_PPP_BSDCOMP=m
CONFIG_ISDN_AUDIO=y
CONFIG_ISDN_TTY_FAX=y
CONFIG_ISDN_DRV_HISAX=m
CONFIG_HISAX_EURO=y
CONFIG_DE_AOC=y
CONFIG_HISAX_1TR6=y
CONFIG_HISAX_NI1=y
CONFIG_HISAX_MAX_CARDS=8
CONFIG_HISAX_16_3=y
CONFIG_HISAX_TELESPCI=y
CONFIG_HISAX_S0BOX=y
CONFIG_HISAX_FRITZPCI=y
CONFIG_HISAX_AVM_A1_PCMCIA=y
CONFIG_HISAX_ELSA=y
CONFIG_HISAX_DIEHLDIVA=y
CONFIG_HISAX_SEDLBAUER=y
CONFIG_HISAX_NETJET=y
CONFIG_HISAX_NETJET_U=y
CONFIG_HISAX_NICCY=y
CONFIG_HISAX_BKM_A4T=y
CONFIG_HISAX_SCT_QUADRO=y
CONFIG_HISAX_GAZEL=y
CONFIG_HISAX_HFC_PCI=y
CONFIG_HISAX_W6692=y
CONFIG_HISAX_HFC_SX=y
CONFIG_HISAX_ENTERNOW_PCI=y
CONFIG_HISAX_SEDLBAUER_CS=m
CONFIG_HISAX_ELSA_CS=m
CONFIG_HISAX_AVM_A1_CS=m
CONFIG_HISAX_TELES_CS=m
CONFIG_HISAX_ST5481=m
CONFIG_HISAX_HFCUSB=m
CONFIG_HISAX_FRITZ_PCIPNP=m
CONFIG_HISAX_HDLC=y
CONFIG_ISDN_DRV_TPAM=m
CONFIG_ISDN_CAPI=m
CONFIG_ISDN_DRV_AVMB1_VERBOSE_REASON=y
CONFIG_ISDN_CAPI_MIDDLEWARE=y
CONFIG_ISDN_CAPI_CAPI20=m
CONFIG_ISDN_CAPI_CAPIFS_BOOL=y
CONFIG_ISDN_CAPI_CAPIFS=m
CONFIG_ISDN_CAPI_CAPIDRV=m
CONFIG_CAPI_AVM=y
CONFIG_CAPI_EICON=y
CONFIG_ISDN_DIVAS=m
CONFIG_ISDN_DIVAS_BRIPCI=y
CONFIG_ISDN_DIVAS_PRIPCI=y
CONFIG_ISDN_DIVAS_DIVACAPI=m
CONFIG_ISDN_DIVAS_USERIDI=m
CONFIG_ISDN_DIVAS_MAINT=m
CONFIG_PHONE=m
CONFIG_PHONE_IXJ=m
CONFIG_PHONE_IXJ_PCMCIA=m
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=m
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_TSDEV=m
CONFIG_INPUT_TSDEV_SCREEN_X=240
CONFIG_INPUT_TSDEV_SCREEN_Y=320
CONFIG_INPUT_EVDEV=m
CONFIG_INPUT_EVBUG=m
CONFIG_GAMEPORT=m
CONFIG_SOUND_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_VORTEX=m
CONFIG_GAMEPORT_FM801=m
CONFIG_GAMEPORT_CS461x=m
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_CT82C710=m
CONFIG_SERIO_PCIPS2=m
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_KEYBOARD_SUNKBD=m
CONFIG_KEYBOARD_XTKBD=m
CONFIG_KEYBOARD_NEWTON=m
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_SERIAL=m
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
CONFIG_JOYSTICK_A3D=m
CONFIG_JOYSTICK_ADI=m
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=m
CONFIG_JOYSTICK_GRIP=m
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=m
CONFIG_JOYSTICK_INTERACT=m
CONFIG_JOYSTICK_SIDEWINDER=m
CONFIG_JOYSTICK_TMDC=m
CONFIG_JOYSTICK_IFORCE=m
CONFIG_JOYSTICK_IFORCE_USB=y
CONFIG_JOYSTICK_IFORCE_232=y
CONFIG_JOYSTICK_WARRIOR=m
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
CONFIG_JOYSTICK_TWIDDLER=m
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_GUNZE=m
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
CONFIG_INPUT_UINPUT=m
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_ROCKETPORT=m
CONFIG_CYCLADES=m
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_N_HDLC=m
CONFIG_STALDRV=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_CS=m
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_MULTIPORT=y
CONFIG_SERIAL_8250_RSA=y
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_WATCHDOG=y
CONFIG_SOFT_WATCHDOG=m
CONFIG_ACQUIRE_WDT=m
CONFIG_ADVANTECH_WDT=m
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
CONFIG_SC520_WDT=m
CONFIG_EUROTECH_WDT=m
CONFIG_IB700_WDT=m
CONFIG_WAFER_WDT=m
CONFIG_I8XX_TCO=m
CONFIG_SC1200_WDT=m
CONFIG_SCx200_WDT=m
CONFIG_60XX_WDT=m
CONFIG_CPU5_WDT=m
CONFIG_W83627HF_WDT=m
CONFIG_W83877F_WDT=m
CONFIG_MACHZ_WDT=m
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m
CONFIG_WDT_501_PCI=y
CONFIG_USBPCWATCHDOG=m
CONFIG_HW_RANDOM=m
CONFIG_NVRAM=m
CONFIG_RTC=m
CONFIG_GEN_RTC=m
CONFIG_GEN_RTC_X=y
CONFIG_DTLK=m
CONFIG_R3964=m
CONFIG_APPLICOM=m
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_DRM=y
CONFIG_DRM_TDFX=m
CONFIG_DRM_R128=m
CONFIG_DRM_RADEON=m
CONFIG_DRM_SIS=m
CONFIG_SYNCLINK_CS=m
CONFIG_MWAVE=m
CONFIG_RAW_DRIVER=m
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_MAX_RAW_DEVS=256
CONFIG_HANGCHECK_TIMER=m
CONFIG_I2C=m
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCF=m
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_ISA=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_PROSAVAGE=m
CONFIG_I2C_SAVAGE4=m
CONFIG_SCx200_ACB=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
CONFIG_I2C_SENSOR=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_FSCHER=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_EEPROM=m
CONFIG_SENSORS_PCF8574=m
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_RTC8564=m
CONFIG_W1=m
CONFIG_W1_MATROX=m
CONFIG_W1_THERM=m
CONFIG_FB=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_CIRRUS=m
CONFIG_FB_PM2=m
CONFIG_FB_PM2_FIFO_DISCONNECT=y
CONFIG_FB_CYBER2000=m
CONFIG_FB_VGA16=m
CONFIG_VIDEO_SELECT=y
CONFIG_FB_HGA=m
CONFIG_FB_RIVA=m
CONFIG_FB_RIVA_I2C=y
CONFIG_FB_RIVA_DEBUG=y
CONFIG_FB_MATROX=m
CONFIG_FB_MATROX_MILLENIUM=y
CONFIG_FB_MATROX_MYSTIQUE=y
CONFIG_FB_MATROX_G450=y
CONFIG_FB_MATROX_G100=y
CONFIG_FB_MATROX_I2C=m
CONFIG_FB_MATROX_MAVEN=m
CONFIG_FB_MATROX_MULTIHEAD=y
CONFIG_FB_RADEON_OLD=m
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
CONFIG_FB_ATY128=m
CONFIG_FB_ATY=m
CONFIG_FB_ATY_CT=y
CONFIG_FB_ATY_GX=y
CONFIG_FB_ATY_XL_INIT=y
CONFIG_FB_SIS=m
CONFIG_FB_SIS_300=y
CONFIG_FB_SIS_315=y
CONFIG_FB_NEOMAGIC=m
CONFIG_FB_KYRO=m
CONFIG_FB_3DFX=m
CONFIG_FB_VOODOO1=m
CONFIG_FB_TRIDENT=m
CONFIG_FB_VIRTUAL=m
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=m
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_USB=y
CONFIG_USB_DEVICEFS=y
CONFIG_USB_BANDWIDTH=y
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_SPLIT_ISO=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_UHCI_HCD=m
CONFIG_USB_STORAGE=m
CONFIG_USB_STORAGE_RW_DETECT=y
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
CONFIG_USB_STORAGE_HP8200e=y
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT=y
CONFIG_USB_HIDDEV=y
CONFIG_USB_KBD=m
CONFIG_USB_MOUSE=m
CONFIG_USB_AIPTEK=m
CONFIG_USB_WACOM=m
CONFIG_USB_KBTAB=m
CONFIG_USB_POWERMATE=m
CONFIG_USB_MTOUCH=m
CONFIG_USB_EGALAX=m
CONFIG_USB_XPAD=m
CONFIG_USB_ATI_REMOTE=m
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_GENESYS=y
CONFIG_USB_NET1080=y
CONFIG_USB_PL2301=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_ZAURUS=y
CONFIG_USB_CDCETHER=y
CONFIG_USB_AX8817X=y
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_EZUSB=y
CONFIG_USB_TIGL=m
CONFIG_USB_AUERSWALD=m
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
CONFIG_USB_LED=m
CONFIG_USB_CYTHERM=m
CONFIG_USB_PHIDGETSERVO=m
CONFIG_USB_TEST=m
CONFIG_USB_GADGET=m
CONFIG_USB_GADGET_NET2280=y
CONFIG_USB_NET2280=m
CONFIG_USB_GADGET_DUALSPEED=y
CONFIG_USB_ZERO=m
CONFIG_USB_ETH=m
CONFIG_USB_ETH_RNDIS=y
CONFIG_USB_GADGETFS=m
CONFIG_USB_FILE_STORAGE=m
CONFIG_USB_G_SERIAL=m
CONFIG_EDD=m
CONFIG_EXT2_FS=m
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=m
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=m
CONFIG_FS_MBCACHE=m
CONFIG_REISERFS_FS=m
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_STATISTICS=y
CONFIG_FS_POSIX_ACL=y
CONFIG_XFS_FS=m
CONFIG_XFS_RT=y
CONFIG_XFS_QUOTA=y
CONFIG_XFS_SECURITY=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_MINIX_FS=m
CONFIG_ROMFS_FS=m
CONFIG_QUOTA=y
CONFIG_QFMT_V1=m
CONFIG_QFMT_V2=m
CONFIG_QUOTACTL=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=m
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
CONFIG_DEVFS_FS=y
CONFIG_DEVPTS_FS_XATTR=y
CONFIG_DEVPTS_FS_SECURITY=y
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_ADFS_FS=m
CONFIG_AFFS_FS=m
CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=m
CONFIG_BEFS_FS=m
CONFIG_BFS_FS=m
CONFIG_EFS_FS=m
CONFIG_CRAMFS=y
CONFIG_VXFS_FS=m
CONFIG_HPFS_FS=m
CONFIG_QNX4FS_FS=m
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=m
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V4=y
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_SMB_FS=m
CONFIG_CIFS=m
CONFIG_NCP_FS=m
CONFIG_NCPFS_NFS_NS=y
CONFIG_NCPFS_OS2_NS=y
CONFIG_NCPFS_NLS=y
CONFIG_NCPFS_EXTRAS=y
CONFIG_CODA_FS=m
CONFIG_AFS_FS=m
CONFIG_RXRPC=m
CONFIG_PARTITION_ADVANCED=y
CONFIG_ACORN_PARTITION=y
CONFIG_ACORN_PARTITION_CUMANA=y
CONFIG_ACORN_PARTITION_ICS=y
CONFIG_ACORN_PARTITION_RISCIX=y
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
CONFIG_ATARI_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
CONFIG_LDM_PARTITION=y
CONFIG_SGI_PARTITION=y
CONFIG_ULTRIX_PARTITION=y
CONFIG_SUN_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="cp437"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m
CONFIG_PROFILING=y
CONFIG_OPROFILE=m
CONFIG_SECURITY=y
CONFIG_SECURITY_CAPABILITIES=m
CONFIG_SECURITY_ROOTPLUG=m
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_TEST=m
CONFIG_CRC_CCITT=m
CONFIG_CRC32=m
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
Brad Fitzpatrick wrote:
> I'm reporting an oops. Details follow.
>
> I have two of these machines. I will happily be anybody's guinea pig
> to debug this. (more details, access to machine, try patches, kernels...)
> Machines aren't in production.
>
> - Brad
>
>
> Kernel: 2.6.9-rc4 vanilla (.config below)
>
> Hardware: IBM eServer 325, Dual Opteron 8GB ram (more info below)
>
> Pre-crash and crash:
>
> a1:~# mke2fs /dev/mapper/raid10-data
> mke2fs 1.35 (28-Feb-2004)
> Filesystem label=
> OS type: Linux
> Block size=4096 (log=2)
> Fragment size=4096 (log=2)
> 25608192 inodes, 51200000 blocks
> 2560000 blocks (5.00%) reserved for the super user
> First data block=0
> 1563 block groups
> 32768 blocks per group, 32768 fragments per group
> 16384 inodes per group
> Superblock backups stored on blocks:
> 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
> 4096000, 7962624, 11239424, 20480000, 23887872
>
> Writing inode tables: 1091/1563
> Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
> localhost kernel: Oops: 0000 [1] SMP
>
> Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
> localhost kernel: CR2: 0000000000001770
What's your block device configuration? What block devices are sitting
on top of what other block devices?
Jeff
On Wed, 13 Oct 2004, Jeff Garzik wrote:
> Brad Fitzpatrick wrote:
> > I'm reporting an oops. Details follow.
> >
> > I have two of these machines. I will happily be anybody's guinea pig
> > to debug this. (more details, access to machine, try patches, kernels...)
> > Machines aren't in production.
> >
> > - Brad
> >
> >
> > Kernel: 2.6.9-rc4 vanilla (.config below)
> >
> > Hardware: IBM eServer 325, Dual Opteron 8GB ram (more info below)
> >
> > Pre-crash and crash:
> >
> > a1:~# mke2fs /dev/mapper/raid10-data
> > mke2fs 1.35 (28-Feb-2004)
> > Filesystem label=
> > OS type: Linux
> > Block size=4096 (log=2)
> > Fragment size=4096 (log=2)
> > 25608192 inodes, 51200000 blocks
> > 2560000 blocks (5.00%) reserved for the super user
> > First data block=0
> > 1563 block groups
> > 32768 blocks per group, 32768 fragments per group
> > 16384 inodes per group
> > Superblock backups stored on blocks:
> > 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
> > 4096000, 7962624, 11239424, 20480000, 23887872
> >
> > Writing inode tables: 1091/1563
> > Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
> > localhost kernel: Oops: 0000 [1] SMP
> >
> > Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
> > localhost kernel: CR2: 0000000000001770
>
>
> What's your block device configuration? What block devices are sitting
> on top of what other block devices?
/dev/mapper/raid10-data is a LV taking 200GB of a 280GB VG ("raid10") with
a single PV in it: /dev/sdb1 -- ips driver, IBM ServeRAID 6M card,
representing a RAID 10 atop 8 SCSI disks.
I just made a new kernel without NUMA and made a filesystem on /dev/sdb1
directly instead of using LVM and it worked fine, if not a little slowly.
Now that I know it /can/ work, I'll try and narrow down whose fault it is:
NUMA or LVM.
- Brad
Brad Fitzpatrick wrote:
> On Wed, 13 Oct 2004, Jeff Garzik wrote:
>
>
>>Brad Fitzpatrick wrote:
>>
>>>I'm reporting an oops. Details follow.
>>>
>>>I have two of these machines. I will happily be anybody's guinea pig
>>>to debug this. (more details, access to machine, try patches, kernels...)
>>>Machines aren't in production.
>>>
>>>- Brad
>>>
>>>
>>>Kernel: 2.6.9-rc4 vanilla (.config below)
>>>
>>>Hardware: IBM eServer 325, Dual Opteron 8GB ram (more info below)
>>>
>>>Pre-crash and crash:
>>>
>>>a1:~# mke2fs /dev/mapper/raid10-data
>>>mke2fs 1.35 (28-Feb-2004)
>>>Filesystem label=
>>>OS type: Linux
>>>Block size=4096 (log=2)
>>>Fragment size=4096 (log=2)
>>>25608192 inodes, 51200000 blocks
>>>2560000 blocks (5.00%) reserved for the super user
>>>First data block=0
>>>1563 block groups
>>>32768 blocks per group, 32768 fragments per group
>>>16384 inodes per group
>>>Superblock backups stored on blocks:
>>> 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
>>> 4096000, 7962624, 11239424, 20480000, 23887872
>>>
>>>Writing inode tables: 1091/1563
>>>Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
>>>localhost kernel: Oops: 0000 [1] SMP
>>>
>>>Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
>>>localhost kernel: CR2: 0000000000001770
>>
>>
>>What's your block device configuration? What block devices are sitting
>>on top of what other block devices?
>
>
> /dev/mapper/raid10-data is a LV taking 200GB of a 280GB VG ("raid10") with
> a single PV in it: /dev/sdb1 -- ips driver, IBM ServeRAID 6M card,
> representing a RAID 10 atop 8 SCSI disks.
>
> I just made a new kernel without NUMA and made a filesystem on /dev/sdb1
> directly instead of using LVM and it worked fine, if not a little slowly.
>
> Now that I know it /can/ work, I'll try and narrow down whose fault it is:
> NUMA or LVM.
Very similar to
http://marc.theaimsgroup.com/?l=linux-kernel&m=109328505204081&w=2
and its follow-up:
http://marc.theaimsgroup.com/?l=linux-kernel&m=109330259511819&w=2
but no solutions there.
--
~Randy
MOTD: Always include version info.
(Again. Sometimes I think ln -s /usr/src/linux/.config .signature)
On Wed, 13 Oct 2004, Randy.Dunlap wrote:
> Brad Fitzpatrick wrote:
> > On Wed, 13 Oct 2004, Jeff Garzik wrote:
> >
> >
> >>Brad Fitzpatrick wrote:
> >>
> >>>I'm reporting an oops. Details follow.
> >>>
> >>>I have two of these machines. I will happily be anybody's guinea pig
> >>>to debug this. (more details, access to machine, try patches, kernels...)
> >>>Machines aren't in production.
> >>>
> >>>- Brad
> >>>
> >>>
> >>>Kernel: 2.6.9-rc4 vanilla (.config below)
> >>>
> >>>Hardware: IBM eServer 325, Dual Opteron 8GB ram (more info below)
> >>>
> >>>Pre-crash and crash:
> >>>
> >>>a1:~# mke2fs /dev/mapper/raid10-data
> >>>mke2fs 1.35 (28-Feb-2004)
> >>>Filesystem label=
> >>>OS type: Linux
> >>>Block size=4096 (log=2)
> >>>Fragment size=4096 (log=2)
> >>>25608192 inodes, 51200000 blocks
> >>>2560000 blocks (5.00%) reserved for the super user
> >>>First data block=0
> >>>1563 block groups
> >>>32768 blocks per group, 32768 fragments per group
> >>>16384 inodes per group
> >>>Superblock backups stored on blocks:
> >>> 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
> >>> 4096000, 7962624, 11239424, 20480000, 23887872
> >>>
> >>>Writing inode tables: 1091/1563
> >>>Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
> >>>localhost kernel: Oops: 0000 [1] SMP
> >>>
> >>>Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
> >>>localhost kernel: CR2: 0000000000001770
> >>
> >>
> >>What's your block device configuration? What block devices are sitting
> >>on top of what other block devices?
> >
> >
> > /dev/mapper/raid10-data is a LV taking 200GB of a 280GB VG ("raid10") with
> > a single PV in it: /dev/sdb1 -- ips driver, IBM ServeRAID 6M card,
> > representing a RAID 10 atop 8 SCSI disks.
> >
> > I just made a new kernel without NUMA and made a filesystem on /dev/sdb1
> > directly instead of using LVM and it worked fine, if not a little slowly.
> >
> > Now that I know it /can/ work, I'll try and narrow down whose fault it is:
> > NUMA or LVM.
>
> Very similar to
> http://marc.theaimsgroup.com/?l=linux-kernel&m=109328505204081&w=2
> and its follow-up:
> http://marc.theaimsgroup.com/?l=linux-kernel&m=109330259511819&w=2
>
> but no solutions there.
Well, good to know I'm not alone? :-)
I was just about to mail and report that disabling NUMA does help:
NUMA + mke2fs on LVM: OOPS (mailed earlier)
no NUMA + mke2fs on LVM: okay
NUMA + mke2fs on sdb1: OOPS (below)
no NUMA + mke2fs on sdb1: okay
no NUMA + mount e2fs on LVM: okay
no NUMA + mount e2fs on sb1: okay
NUMA + mount e2fs on LVM: okay
NUMA + mount e2fs on sb1: untested, assume okay
OOPs when doing mke2fs on /dev/sdb1, with NUMA enabled:
Oct 13 13:24:37 localhost kernel: Unable to handle kernel paging request at 0000000000001770 RIP:
Oct 13 13:24:37 localhost kernel: <ffffffff8015efe4>{kmem_getpages+132}
Oct 13 13:24:37 localhost kernel: PML4 1f8fe6067 PGD 1f8fef067 PMD 0
Oct 13 13:24:37 localhost kernel: Oops: 0000 [1] SMP
Oct 13 13:24:37 localhost kernel: CPU 0
Oct 13 13:24:37 localhost kernel: Modules linked in: af_packet tsdev mousedev joydev usbhid ohci_hcd hw_random amd74xx evdev tg3 dm_mod ide_generic ide_cd ide_core cdrom rtc ext3 jbd mbcache sd_mod ips mptscsih mptbase scsi_mod unix
Oct 13 13:24:37 localhost kernel: Pid: 3145, comm: mke2fs Not tainted 2.6.9-rc4
Oct 13 13:24:37 localhost kernel: RIP: 0010:[kmem_getpages+132/432] <ffffffff8015efe4>{kmem_getpages+132}
Oct 13 13:24:37 localhost kernel: RSP: 0018:00000101f81b7aa8 EFLAGS: 00010213
Oct 13 13:24:37 localhost kernel: RAX: ffffffff7fffffff RBX: 00000101fffc9680 RCX: 0000000000000000
Oct 13 13:24:37 localhost kernel: RDX: 0000010000011700 RSI: 00000100000119c0 RDI: 0000010000012500
Oct 13 13:24:37 localhost kernel: RBP: 00000101fffc9680 R08: 000001016bc01000 R09: 00000101fffc96e8
Oct 13 13:24:37 localhost kernel: R10: 0000000000000000 R11: 00000000fffffffa R12: 00000101fffc9680
Oct 13 13:24:37 localhost kernel: R13: 0000000000000000 R14: 00000101fffc9728 R15: 0000000000000001
Oct 13 13:24:37 localhost kernel: FS: 0000002a95ddb4a0(0000) GS:ffffffff803df300(0000) knlGS:0000000000000000
Oct 13 13:24:37 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Oct 13 13:24:37 localhost kernel: CR2: 0000000000001770 CR3: 0000000000101000 CR4: 00000000000006e0
Oct 13 13:24:37 localhost kernel: Process mke2fs (pid: 3145, threadinfo 00000101f81b6000, task 00000101fe9b4030)
Oct 13 13:24:37 localhost kernel: Stack: 000001016c3fa000 0000000000000000 0000000000000050 ffffffff8015ff6e
Oct 13 13:24:37 localhost kernel: 0000005000000010 000000000000003c 00000100fbf6b000 00000101fffc9680
Oct 13 13:24:37 localhost kernel: 00000101fffc96c8 00000101fffc9728
Oct 13 13:24:37 localhost kernel: Call Trace:<ffffffff8015ff6e>{cache_grow+190} <ffffffff801601c6>{cache_alloc_refill+422}
Oct 13 13:24:37 localhost kernel: <ffffffff801604b6>{kmem_cache_alloc+54} <ffffffff8017d761>{alloc_buffer_head+17}
Oct 13 13:24:37 localhost kernel: <ffffffff8017aeba>{create_buffers+42} <ffffffff8017b884>{create_empty_buffers+20}
Oct 13 13:24:37 localhost kernel: <ffffffff8017bcdf>{__block_prepare_write+175} <ffffffff80180190>{blkdev_get_block+0}
Oct 13 13:24:37 localhost kernel: <ffffffff8017c78a>{block_prepare_write+26} <ffffffff80158dc4>{generic_file_buffered_write+404}
Oct 13 13:24:37 localhost kernel: <ffffffff80193fae>{inode_update_time+158} <ffffffff801594dd>{generic_file_aio_write_nolock+765}
Oct 13 13:24:37 localhost kernel: <ffffffff801595b5>{generic_file_write_nolock+165} <ffffffff80134ef3>{__wake_up+67}
Oct 13 13:24:37 localhost kernel: <ffffffff802a46fe>{thread_return+41} <ffffffff80136890>{autoremove_wake_function+0}
Oct 13 13:24:37 localhost kernel: <ffffffff8018128a>{blkdev_file_write+26} <ffffffff801789e4>{vfs_write+228}
Oct 13 13:24:37 localhost kernel: <ffffffff80178b13>{sys_write+83} <ffffffff8011195a>{system_call+126}
Oct 13 13:24:37 localhost kernel:
Oct 13 13:24:37 localhost kernel:
Oct 13 13:24:37 localhost kernel: Code: 48 8b 91 70 17 00 00 76 07 b8 00 00 00 80 eb 0a 48 b8 00 00
Oct 13 13:24:37 localhost kernel: RIP <ffffffff8015efe4>{kmem_getpages+132} RSP <00000101f81b7aa8>
Oct 13 13:24:37 localhost kernel: CR2: 0000000000001770
Randy, if you're interested and you're actually at OSDL Beaverton, I'm
just across the street from you. I could carry this 1U server and 3U
drive cabinet over to you! :)
Who's responsible for the K8_NUMA stuff? I'd love to work with them to
narrow this down.
- Brad
>
> --
> ~Randy
> MOTD: Always include version info.
> (Again. Sometimes I think ln -s /usr/src/linux/.config .signature)
>
>
I'm trying to get kernel-2.6.9-rc4-mm1 to work with gnome-2.8. While
2.6.9-rc4 works fine with gnome-2.8 the mm1 version has an issue. Any
process that I'm trying to start that uses gnome libraries crashes
immediatly after startup. Mozilla, nautilus and gnome terminal to name a
few. The reason for using mm1 is that I'm using reiser4 for one of my
partitions.
The output that I get in bugbuddy is as following:
Backtrace was generated from '/usr/test/garnome2/lib/nautilus'
0x40d164ba in ?? ()
#0 0x40d164ba in ?? ()
#1 0x402b4860 in ?? ()
#2 0x40d14bc5 in ?? ()
#3 <signal handler called>
#4 0x40f33281 in ?? ()
#5 0x40f33347 in ?? ()
#6 0x40d11238 in ?? ()
#7 0x40d0dd28 in ?? ()
#8 0x40c6b19d in ?? ()
#9 0x40c6e3b9 in ?? ()
#10 0x40c6e5b1 in ?? ()
#11 0x40c7f4f2 in ?? ()
#12 0x40c3d10e in ?? ()
#13 0x40c42fe2 in ?? ()
#14 0x40c3e1ee in ?? ()
#15 0x40c40cbf in ?? ()
#16 0x40c40db4 in ?? ()
#17 0x40c42f90 in ?? ()
#18 0x40c3f2af in ?? ()
#19 0x40c3f59b in ?? ()
#20 0x4087127e in ?? ()
#21 0x408713fb in ?? ()
#22 0x408934e2 in ?? ()
#23 0x40cc373b in ?? ()
#24 0x40caa513 in ?? ()
#25 0x40caa8d1 in ?? ()
#26 0x40caa9f6 in ?? ()
#27 0x40893f4e in ?? ()
#28 0x40893fa6 in ?? ()
#29 0x80694ef in ?? ()
#30 0x40cc3fe5 in ?? ()
#31 0x40caaa22 in ?? ()
#32 0x408f9f57 in ?? ()
#33 0x40ca9cb0 in ?? ()
#34 0x40caa8d1 in ?? ()
#35 0x40caa9f6 in ?? ()
#36 0x8069600 in ?? ()
#37 0x8077205 in ?? ()
#38 0x40f1f916 in ?? ()
And on the terminal that I started X windows:
"/usr/test/garnome2/lib/nautilus": not in executable format: Is a directory
/home/stef/405: No such file or directory.
/usr/local/mozilla/run-mozilla.sh: line 423: 460 Segmentation
fault "$prog" ${1+"$@"}
"/usr/test/garnome2/lib/nautilus": not in executable format: Is a directory
/home/stef/448: No such file or directory.
"/usr/test/garnome2/lib/nautilus": not in executable format: Is a directory
/home/stef/448: No such file or directory.
ptrace: No such process.
/usr/test/garnome2/share/bug-buddy/gdb-cmd:1: Error in sourced command file:
No stack.
"/usr/test/garnome2/lib/nautilus": not in executable format: Is a directory
/home/stef/448: No such file or directory.
ptrace: No such process.
/usr/test/garnome2/share/bug-buddy/gdb-cmd:1: Error in sourced command file:
No stack.
/usr/local/mozilla/run-mozilla.sh: line 423: 483 Segmentation
fault "$prog" ${1+"$@"}
ldd for nautilus
bash-2.05b$ ldd /usr/test/garnome2/bin/nautilus
libnautilus.so.2 => /usr/test/garnome2/lib/libnautilus.so.2
(0x40016000)
libnautilus-adapter.so.2 =>
/usr/test/garnome2/lib/libnautilus-adapter.so.2 (0x40028000)
libnautilus-private.so.2 =>
/usr/test/garnome2/lib/libnautilus-private.so.2 (0x4002b000)
libnautilus-extension.so.1 =>
/usr/test/garnome2/lib/libnautilus-extension.so.1 (0x400d5000)
libeel-2.so.2 => /usr/test/garnome2/lib/libeel-2.so.2 (0x400de000)
libXrender.so.1 => /usr/local/lib/libXrender.so.1 (0x4017f000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x40187000)
libgailutil.so.17 => /usr/test/garnome2/lib/libgailutil.so.17
(0x40195000)
libglade-2.0.so.0 => /usr/test/garnome2/lib/libglade-2.0.so.0
(0x4019d000)
librsvg-2.so.2 => /usr/test/garnome2/lib/librsvg-2.so.2 (0x401b4000)
libgsf-1.so.1 => /usr/test/garnome2/lib/libgsf-1.so.1 (0x401e0000)
libbz2.so.1.0 => /lib/libbz2.so.1.0 (0x40201000)
libcroco-0.6.so.3 => /usr/test/garnome2/lib/libcroco-0.6.so.3
(0x40211000)
libgnome-desktop-2.so.2 =>
/usr/test/garnome2/lib/libgnome-desktop-2.so.2 (0x4024b000)
libgnomeui-2.so.0 => /usr/test/garnome2/lib/libgnomeui-2.so.0
(0x40260000)
libgnome-keyring.so.0 =>
/usr/test/garnome2/lib/libgnome-keyring.so.0 (0x402f1000)
libbonoboui-2.so.0 => /usr/test/garnome2/lib/libbonoboui-2.so.0
(0x402fc000)
libgnomecanvas-2.so.0 =>
/usr/test/garnome2/lib/libgnomecanvas-2.so.0 (0x4035c000)
libgnome-2.so.0 => /usr/test/garnome2/lib/libgnome-2.so.0
(0x40388000)
libesd.so.0 => /usr/test/garnome2/lib/libesd.so.0 (0x4039b000)
libasound.so.2 => /usr/lib/libasound.so.2 (0x403a4000)
libaudiofile.so.0 => /usr/test/garnome2/lib/libaudiofile.so.0
(0x4044e000)
libart_lgpl_2.so.2 => /usr/test/garnome2/lib/libart_lgpl_2.so.2
(0x40473000)
libpangoft2-1.0.so.0 =>
/usr/test/garnome2/lib/libpangoft2-1.0.so.0 (0x40488000)
libgtk-x11-2.0.so.0 =>
/usr/test/garnome2/lib/libgtk-x11-2.0.so.0 (0x404ad000)
libgdk-x11-2.0.so.0 =>
/usr/test/garnome2/lib/libgdk-x11-2.0.so.0 (0x40770000)
libatk-1.0.so.0 => /usr/test/garnome2/lib/libatk-1.0.so.0
(0x407dd000)
libgdk_pixbuf-2.0.so.0 =>
/usr/test/garnome2/lib/libgdk_pixbuf-2.0.so.0 (0x407f8000)
libpangoxft-1.0.so.0 =>
/usr/test/garnome2/lib/libpangoxft-1.0.so.0 (0x4080e000)
libpangox-1.0.so.0 => /usr/test/garnome2/lib/libpangox-1.0.so.0
(0x40813000)
libpango-1.0.so.0 => /usr/test/garnome2/lib/libpango-1.0.so.0
(0x4081d000)
libgnomevfs-2.so.0 => /usr/test/garnome2/lib/libgnomevfs-2.so.0
(0x4084b000)
libgconf-2.so.4 => /usr/test/garnome2/lib/libgconf-2.so.4
(0x408ac000)
libbonobo-2.so.0 => /usr/test/garnome2/lib/libbonobo-2.so.0
(0x408d5000)
libxml2.so.2 => /usr/test/garnome2/lib/libxml2.so.2 (0x4092f000)
libssl.so.0 => /usr/lib/libssl.so.0 (0x40a34000)
libhowl-0.9.6.so.1 => /usr/test/garnome2/lib/libhowl-0.9.6.so.1
(0x40aea000)
libresolv.so.2 => /lib/libresolv.so.2 (0x40c10000)
librt.so.1 => /lib/librt.so.1 (0x40c22000)
libbonobo-activation.so.4 =>
/usr/test/garnome2/lib/libbonobo-activation.so.4 (0x40c34000)
libORBitCosNaming-2.so.0 =>
/usr/test/garnome2/lib/libORBitCosNaming-2.so.0 (0x40c4a000)
libORBit-2.so.0 => /usr/test/garnome2/lib/libORBit-2.so.0
(0x40c4f000)
libgobject-2.0.so.0 =>
/usr/test/garnome2/lib/libgobject-2.0.so.0 (0x40c9d000)
libm.so.6 => /lib/libm.so.6 (0x40cda000)
libgmodule-2.0.so.0 =>
/usr/test/garnome2/lib/libgmodule-2.0.so.0 (0x40cfc000)
libdl.so.2 => /lib/libdl.so.2 (0x40d00000)
libgthread-2.0.so.0 =>
/usr/test/garnome2/lib/libgthread-2.0.so.0 (0x40d04000)
libpthread.so.0 => /lib/libpthread.so.0 (0x40d09000)
libglib-2.0.so.0 => /usr/test/garnome2/lib/libglib-2.0.so.0
(0x40d5a000)
libstartup-notification-1.so.0 =>
/usr/test/garnome2/lib/libstartup-notification-1.so.0 (0x40dd8000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x40de0000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x40de9000)
libcdda_paranoia.so.0 => /usr/local/lib/libcdda_paranoia.so.0
(0x40e02000)
libcdda_interface.so.0 => /usr/local/lib/libcdda_interface.so.0
(0x40e0a000)
libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0x40e1c000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x40e3b000)
libpopt.so.0 => /usr/lib/libpopt.so.0 (0x40f04000)
libc.so.6 => /lib/libc.so.6 (0x40f0a000)
libz.so.1 => /usr/lib/libz.so.1 (0x4103a000)
libfontconfig.so.1 => /usr/local/lib/libfontconfig.so.1 (0x4104a000)
libfreetype.so.6 => /usr/local/lib/libfreetype.so.6 (0x41071000)
libXrandr.so.2 => /usr/X11R6/lib/libXrandr.so.2 (0x410dd000)
libXi.so.6 => /usr/X11R6/lib/libXi.so.6 (0x410e1000)
libXinerama.so.1 => /usr/X11R6/lib/libXinerama.so.1 (0x410e9000)
libXft.so.2 => /usr/local/lib/libXft.so.2 (0x410ec000)
libXcursor.so.1 => /usr/local/lib/libXcursor.so.1 (0x410ff000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
libexpat.so.0 => /usr/local/lib/libexpat.so.0 (0x41108000)
libgcc_s.so.1 => /usr/local/lib/libgcc_s.so.1 (0x41127000)
I've compiled the kernel from source on a slackware-8.0 box. My compiler
is gcc-3.4.2 using glibc-2.3.2. Gnome has been compiled using the
garnome system version 2.8.0.
The information provided is probably not nearly enough to solve or find
the issue so a little help would be greatly appriciated.
Best regards,
Stef
Brad Fitzpatrick wrote:
> On Wed, 13 Oct 2004, Randy.Dunlap wrote:
>
>
>>Brad Fitzpatrick wrote:
>>
>>>On Wed, 13 Oct 2004, Jeff Garzik wrote:
>>>
>>>
>>>
>>>>Brad Fitzpatrick wrote:
>>>>
>>>>
>>>>>I'm reporting an oops. Details follow.
>>>>>
>>>>>I have two of these machines. I will happily be anybody's guinea pig
>>>>>to debug this. (more details, access to machine, try patches, kernels...)
>>>>>Machines aren't in production.
>>>>>
>>>>>- Brad
>>>>>
>>>>>
>>>>>Kernel: 2.6.9-rc4 vanilla (.config below)
>>>>>
>>>>>Hardware: IBM eServer 325, Dual Opteron 8GB ram (more info below)
>>>>>
>>>>>Pre-crash and crash:
>>>>>
>>>>>a1:~# mke2fs /dev/mapper/raid10-data
>>>>>mke2fs 1.35 (28-Feb-2004)
>>>>>Filesystem label=
>>>>>OS type: Linux
>>>>>Block size=4096 (log=2)
>>>>>Fragment size=4096 (log=2)
>>>>>25608192 inodes, 51200000 blocks
>>>>>2560000 blocks (5.00%) reserved for the super user
>>>>>First data block=0
>>>>>1563 block groups
>>>>>32768 blocks per group, 32768 fragments per group
>>>>>16384 inodes per group
>>>>>Superblock backups stored on blocks:
>>>>> 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
>>>>> 4096000, 7962624, 11239424, 20480000, 23887872
>>>>>
>>>>>Writing inode tables: 1091/1563
>>>>>Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
>>>>>localhost kernel: Oops: 0000 [1] SMP
>>>>>
>>>>>Message from syslogd@localhost at Wed Oct 13 11:46:01 2004 ...
>>>>>localhost kernel: CR2: 0000000000001770
>>>>
>>>>
>>>>What's your block device configuration? What block devices are sitting
>>>>on top of what other block devices?
>>>
>>>
>>>/dev/mapper/raid10-data is a LV taking 200GB of a 280GB VG ("raid10") with
>>>a single PV in it: /dev/sdb1 -- ips driver, IBM ServeRAID 6M card,
>>>representing a RAID 10 atop 8 SCSI disks.
>>>
>>>I just made a new kernel without NUMA and made a filesystem on /dev/sdb1
>>>directly instead of using LVM and it worked fine, if not a little slowly.
>>>
>>>Now that I know it /can/ work, I'll try and narrow down whose fault it is:
>>>NUMA or LVM.
>>
>>Very similar to
>>http://marc.theaimsgroup.com/?l=linux-kernel&m=109328505204081&w=2
>>and its follow-up:
>>http://marc.theaimsgroup.com/?l=linux-kernel&m=109330259511819&w=2
>>
>>but no solutions there.
>
>
> Well, good to know I'm not alone? :-)
>
> I was just about to mail and report that disabling NUMA does help:
>
> NUMA + mke2fs on LVM: OOPS (mailed earlier)
> no NUMA + mke2fs on LVM: okay
> NUMA + mke2fs on sdb1: OOPS (below)
> no NUMA + mke2fs on sdb1: okay
>
> no NUMA + mount e2fs on LVM: okay
> no NUMA + mount e2fs on sb1: okay
> NUMA + mount e2fs on LVM: okay
> NUMA + mount e2fs on sb1: untested, assume okay
>
>
> OOPs when doing mke2fs on /dev/sdb1, with NUMA enabled:
>
> Oct 13 13:24:37 localhost kernel: Unable to handle kernel paging request at 0000000000001770 RIP:
> Oct 13 13:24:37 localhost kernel: <ffffffff8015efe4>{kmem_getpages+132}
> Oct 13 13:24:37 localhost kernel: PML4 1f8fe6067 PGD 1f8fef067 PMD 0
> Oct 13 13:24:37 localhost kernel: Oops: 0000 [1] SMP
> Oct 13 13:24:37 localhost kernel: CPU 0
> Oct 13 13:24:37 localhost kernel: Modules linked in: af_packet tsdev mousedev joydev usbhid ohci_hcd hw_random amd74xx evdev tg3 dm_mod ide_generic ide_cd ide_core cdrom rtc ext3 jbd mbcache sd_mod ips mptscsih mptbase scsi_mod unix
> Oct 13 13:24:37 localhost kernel: Pid: 3145, comm: mke2fs Not tainted 2.6.9-rc4
> Oct 13 13:24:37 localhost kernel: RIP: 0010:[kmem_getpages+132/432] <ffffffff8015efe4>{kmem_getpages+132}
> Oct 13 13:24:37 localhost kernel: RSP: 0018:00000101f81b7aa8 EFLAGS: 00010213
> Oct 13 13:24:37 localhost kernel: RAX: ffffffff7fffffff RBX: 00000101fffc9680 RCX: 0000000000000000
> Oct 13 13:24:37 localhost kernel: RDX: 0000010000011700 RSI: 00000100000119c0 RDI: 0000010000012500
> Oct 13 13:24:37 localhost kernel: RBP: 00000101fffc9680 R08: 000001016bc01000 R09: 00000101fffc96e8
> Oct 13 13:24:37 localhost kernel: R10: 0000000000000000 R11: 00000000fffffffa R12: 00000101fffc9680
> Oct 13 13:24:37 localhost kernel: R13: 0000000000000000 R14: 00000101fffc9728 R15: 0000000000000001
> Oct 13 13:24:37 localhost kernel: FS: 0000002a95ddb4a0(0000) GS:ffffffff803df300(0000) knlGS:0000000000000000
> Oct 13 13:24:37 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Oct 13 13:24:37 localhost kernel: CR2: 0000000000001770 CR3: 0000000000101000 CR4: 00000000000006e0
> Oct 13 13:24:37 localhost kernel: Process mke2fs (pid: 3145, threadinfo 00000101f81b6000, task 00000101fe9b4030)
> Oct 13 13:24:37 localhost kernel: Stack: 000001016c3fa000 0000000000000000 0000000000000050 ffffffff8015ff6e
> Oct 13 13:24:37 localhost kernel: 0000005000000010 000000000000003c 00000100fbf6b000 00000101fffc9680
> Oct 13 13:24:37 localhost kernel: 00000101fffc96c8 00000101fffc9728
> Oct 13 13:24:37 localhost kernel: Call Trace:<ffffffff8015ff6e>{cache_grow+190} <ffffffff801601c6>{cache_alloc_refill+422}
> Oct 13 13:24:37 localhost kernel: <ffffffff801604b6>{kmem_cache_alloc+54} <ffffffff8017d761>{alloc_buffer_head+17}
> Oct 13 13:24:37 localhost kernel: <ffffffff8017aeba>{create_buffers+42} <ffffffff8017b884>{create_empty_buffers+20}
> Oct 13 13:24:37 localhost kernel: <ffffffff8017bcdf>{__block_prepare_write+175} <ffffffff80180190>{blkdev_get_block+0}
> Oct 13 13:24:37 localhost kernel: <ffffffff8017c78a>{block_prepare_write+26} <ffffffff80158dc4>{generic_file_buffered_write+404}
> Oct 13 13:24:37 localhost kernel: <ffffffff80193fae>{inode_update_time+158} <ffffffff801594dd>{generic_file_aio_write_nolock+765}
> Oct 13 13:24:37 localhost kernel: <ffffffff801595b5>{generic_file_write_nolock+165} <ffffffff80134ef3>{__wake_up+67}
> Oct 13 13:24:37 localhost kernel: <ffffffff802a46fe>{thread_return+41} <ffffffff80136890>{autoremove_wake_function+0}
> Oct 13 13:24:37 localhost kernel: <ffffffff8018128a>{blkdev_file_write+26} <ffffffff801789e4>{vfs_write+228}
> Oct 13 13:24:37 localhost kernel: <ffffffff80178b13>{sys_write+83} <ffffffff8011195a>{system_call+126}
> Oct 13 13:24:37 localhost kernel:
> Oct 13 13:24:37 localhost kernel:
> Oct 13 13:24:37 localhost kernel: Code: 48 8b 91 70 17 00 00 76 07 b8 00 00 00 80 eb 0a 48 b8 00 00
> Oct 13 13:24:37 localhost kernel: RIP <ffffffff8015efe4>{kmem_getpages+132} RSP <00000101f81b7aa8>
> Oct 13 13:24:37 localhost kernel: CR2: 0000000000001770
>
>
>
> Randy, if you're interested and you're actually at OSDL Beaverton, I'm
> just across the street from you. I could carry this 1U server and 3U
> drive cabinet over to you! :)
I am at the Round in Beaverton. It might be easier for me to go to
the server. :)
> Who's responsible for the K8_NUMA stuff? I'd love to work with them to
> narrow this down.
Andi Kleen (SUSE). Copied.
--
~Randy
On Wed, 13 Oct 2004 22:38:19 +0200, Stef van der Made <[email protected]> wrote:
>
> I'm trying to get kernel-2.6.9-rc4-mm1 to work with gnome-2.8. While
> 2.6.9-rc4 works fine with gnome-2.8 the mm1 version has an issue. Any
> process that I'm trying to start that uses gnome libraries crashes
> immediatly after startup. Mozilla, nautilus and gnome terminal to name a
> few. The reason for using mm1 is that I'm using reiser4 for one of my
> partitions.
>
> The output that I get in bugbuddy is as following:
>
> Backtrace was generated from '/usr/test/garnome2/lib/nautilus'
>
> <snip>
This was useless, w/o any information.
Next time please compile with debugging (-g) and without
-fomit-frame-pointer. (in case of gcc <3.5)
>
> And on the terminal that I started X windows:
>
> "/usr/test/garnome2/lib/nautilus": not in executable format: Is a directory
> /home/stef/405: No such file or directory.
> /usr/local/mozilla/run-mozilla.sh: line 423: 460 Segmentation
> fault "$prog" ${1+"$@"}
> <snip>
> /usr/local/mozilla/run-mozilla.sh: line 423: 483 Segmentation
> fault "$prog" ${1+"$@"}
>
Looks like you need to back out this patch, the root of all evil:
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/broken-out/optimize-profile-path-slightly.patch
like this:
cd linux-2.6.9-rc4-mm1
patch -p1 -R < /download-dir/optimize-profile-path-slightly.patch
Brad Fitzpatrick wrote:
> I was just about to mail and report that disabling NUMA does help:
>
> NUMA + mke2fs on LVM: OOPS (mailed earlier)
> no NUMA + mke2fs on LVM: okay
> NUMA + mke2fs on sdb1: OOPS (below)
> no NUMA + mke2fs on sdb1: okay
>
> no NUMA + mount e2fs on LVM: okay
> no NUMA + mount e2fs on sb1: okay
> NUMA + mount e2fs on LVM: okay
> NUMA + mount e2fs on sb1: untested, assume okay
Honestly it smells like a devmapper bug, with the pertinent test case
being some sort of behavior not normally generated by filesystem use
(the most common use case for block devices).
But I could be wrong...
Jeff
On Wed, 13 Oct 2004, Jeff Garzik wrote:
> Brad Fitzpatrick wrote:
> > I was just about to mail and report that disabling NUMA does help:
> >
> > NUMA + mke2fs on LVM: OOPS (mailed earlier)
> > no NUMA + mke2fs on LVM: okay
> > NUMA + mke2fs on sdb1: OOPS (below)
> > no NUMA + mke2fs on sdb1: okay
> >
> > no NUMA + mount e2fs on LVM: okay
> > no NUMA + mount e2fs on sb1: okay
> > NUMA + mount e2fs on LVM: okay
> > NUMA + mount e2fs on sb1: untested, assume okay
>
>
> Honestly it smells like a devmapper bug, with the pertinent test case
> being some sort of behavior not normally generated by filesystem use
> (the most common use case for block devices).
Except for the data point above:
NUMA + mke2fs on sdb1: OOPS (below)
That's with devmapper totally out of the picture.
- Brad
On Wed, Oct 13, 2004 at 01:49:08PM -0700, Brad Fitzpatrick wrote:
> On Wed, 13 Oct 2004, Jeff Garzik wrote:
>
> > Brad Fitzpatrick wrote:
> > > I was just about to mail and report that disabling NUMA does help:
> > >
> > > NUMA + mke2fs on LVM: OOPS (mailed earlier)
> > > no NUMA + mke2fs on LVM: okay
> > > NUMA + mke2fs on sdb1: OOPS (below)
> > > no NUMA + mke2fs on sdb1: okay
> > >
> > > no NUMA + mount e2fs on LVM: okay
> > > no NUMA + mount e2fs on sb1: okay
> > > NUMA + mount e2fs on LVM: okay
> > > NUMA + mount e2fs on sb1: untested, assume okay
> >
> >
> > Honestly it smells like a devmapper bug, with the pertinent test case
> > being some sort of behavior not normally generated by filesystem use
> > (the most common use case for block devices).
>
> Except for the data point above:
>
> NUMA + mke2fs on sdb1: OOPS (below)
>
> That's with devmapper totally out of the picture.
Ahh, indeed.
I think Andi did NUMA on x86-64?
Jeff
On Wed, 2004-10-13 at 16:38, Stef van der Made wrote:
> I'm trying to get kernel-2.6.9-rc4-mm1 to work with gnome-2.8. While
> 2.6.9-rc4 works fine with gnome-2.8 the mm1 version has an issue. Any
> process that I'm trying to start that uses gnome libraries crashes
> immediatly after startup. Mozilla, nautilus and gnome terminal to name a
> few. The reason for using mm1 is that I'm using reiser4 for one of my
> partitions.
>
Hi
I had the same issue,
here's a fix
cd /usr/src/linux-2.6.9-rc4-mm1
wget
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/broken-out/optimize-profile-path-slightly.patch
patch -R -p1 < optimize-profile-path-slightly.patch
Jesse
--
Jesse Stockall <[email protected]>
Unpatching optimize-profile-path-slightly.patchsolved the issue. Thanks
for the tip. You also gave the tip that I should compile without
-fomit-frame-pointer. I've done this for years without any problem. What
problems could this cause ?
Thanks again for your help,
Stef
Radoslaw Szkodzinski wrote:
>On Wed, 13 Oct 2004 22:38:19 +0200, Stef van der Made <[email protected]> wrote:
>
>
>>I'm trying to get kernel-2.6.9-rc4-mm1 to work with gnome-2.8. While
>>2.6.9-rc4 works fine with gnome-2.8 the mm1 version has an issue. Any
>>process that I'm trying to start that uses gnome libraries crashes
>>immediatly after startup. Mozilla, nautilus and gnome terminal to name a
>>few. The reason for using mm1 is that I'm using reiser4 for one of my
>>partitions.
>>
>>The output that I get in bugbuddy is as following:
>>
>>Backtrace was generated from '/usr/test/garnome2/lib/nautilus'
>>
>><snip>
>>
>>
>This was useless, w/o any information.
>Next time please compile with debugging (-g) and without
>-fomit-frame-pointer. (in case of gcc <3.5)
>
>
>
>>And on the terminal that I started X windows:
>>
>>"/usr/test/garnome2/lib/nautilus": not in executable format: Is a directory
>>/home/stef/405: No such file or directory.
>>/usr/local/mozilla/run-mozilla.sh: line 423: 460 Segmentation
>>fault "$prog" ${1+"$@"}
>><snip>
>>/usr/local/mozilla/run-mozilla.sh: line 423: 483 Segmentation
>>fault "$prog" ${1+"$@"}
>>
>>
>>
>
>Looks like you need to back out this patch, the root of all evil:
>http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/broken-out/optimize-profile-path-slightly.patch
>like this:
>cd linux-2.6.9-rc4-mm1
>patch -p1 -R < /download-dir/optimize-profile-path-slightly.patch
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to [email protected]
>More majordomo info at http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at http://www.tux.org/lkml/
>
>
>
On Wed, 13 Oct 2004 13:34:01 -0700
"Randy.Dunlap" <[email protected]> wrote:
>
> > Who's responsible for the K8_NUMA stuff? I'd love to work with them to
> > narrow this down.
>
> Andi Kleen (SUSE). Copied.
It looks like memory corruption somewhere. I suspect not directly related to NUMA,
but the different memory layout with NUMA may trigger it.
I would enable CONFIG_DEBUG_SLAB and CONFIG_DEBUG_PAGEALLOC and see if that
triggers it elsewhere.
First suspection would be the device driver. Perhaps you can test it with
a different block device?
-Andi
On Wed, 13 Oct 2004 23:16:34 +0200, Stef van der Made <[email protected]> wrote:
>
> Unpatching optimize-profile-path-slightly.patchsolved the issue. Thanks
> for the tip. You also gave the tip that I should compile without
> -fomit-frame-pointer. I've done this for years without any problem. What
> problems could this cause ?
Compiling with -fomit-frame-pointer removes information from the
binary that could be used for debugging. So the bugbuddy information
you provided was not optimal, because it lacked the crucial stuff. ;-)
The flag does not cause any problems, it is routinely used for
compiling stuff that doesn't need debugging.
Groeten,
Buddy
> Thanks again for your help,
>
> Stef
>
>
>
>
> Radoslaw Szkodzinski wrote:
>
> >On Wed, 13 Oct 2004 22:38:19 +0200, Stef van der Made <[email protected]> wrote:
> >
> >
> >>I'm trying to get kernel-2.6.9-rc4-mm1 to work with gnome-2.8. While
> >>2.6.9-rc4 works fine with gnome-2.8 the mm1 version has an issue. Any
> >>process that I'm trying to start that uses gnome libraries crashes
> >>immediatly after startup. Mozilla, nautilus and gnome terminal to name a
> >>few. The reason for using mm1 is that I'm using reiser4 for one of my
> >>partitions.
> >>
> >>The output that I get in bugbuddy is as following:
> >>
> >>Backtrace was generated from '/usr/test/garnome2/lib/nautilus'
> >>
> >><snip>
> >>
> >>
> >This was useless, w/o any information.
> >Next time please compile with debugging (-g) and without
> >-fomit-frame-pointer. (in case of gcc <3.5)
> >
> >
> >
> >>And on the terminal that I started X windows:
> >>
> >>"/usr/test/garnome2/lib/nautilus": not in executable format: Is a directory
> >>/home/stef/405: No such file or directory.
> >>/usr/local/mozilla/run-mozilla.sh: line 423: 460 Segmentation
> >>fault "$prog" ${1+"$@"}
> >><snip>
> >>/usr/local/mozilla/run-mozilla.sh: line 423: 483 Segmentation
> >>fault "$prog" ${1+"$@"}
> >>
> >>
> >>
> >
> >Looks like you need to back out this patch, the root of all evil:
> >http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/broken-out/optimize-profile-path-slightly.patch
> >like this:
> >cd linux-2.6.9-rc4-mm1
> >patch -p1 -R < /download-dir/optimize-profile-path-slightly.patch
> >-
> >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> >the body of a message to [email protected]
> >More majordomo info at http://vger.kernel.org/majordomo-info.html
> >Please read the FAQ at http://www.tux.org/lkml/
> >
> >
> >
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
On Thu, 14 Oct 2004 00:08:24 +0200, Buddy Lucas <[email protected]> wrote:
> Compiling with -fomit-frame-pointer removes information from the
> binary that could be used for debugging. So the bugbuddy information
> you provided was not optimal, because it lacked the crucial stuff. ;-)
> The flag does not cause any problems, it is routinely used for
> compiling stuff that doesn't need debugging.
It's worth adding that the reason why programs are often built with
-fomit-frame-pointer is for overhead reduction on stack frames, to
make the binary slightly smaller and run maybe a bit faster so it's
become quite popular. This discussion is beyond the scope of the
original mail, but worthy of noting - and it ends up going down the
path of differences between architectures, so let's just leave it at
this clarification.
Jon.
On Wed, 13 Oct 2004, Andi Kleen wrote:
> On Wed, 13 Oct 2004 13:34:01 -0700 "Randy.Dunlap" <[email protected]> wrote:
> >
> > > Who's responsible for the K8_NUMA stuff? I'd love to work with them to
> > > narrow this down.
> >
> > Andi Kleen (SUSE). Copied.
>
> It looks like memory corruption somewhere. I suspect not directly related to NUMA,
> but the different memory layout with NUMA may trigger it.
>
> I would enable CONFIG_DEBUG_SLAB and CONFIG_DEBUG_PAGEALLOC and see if that
> triggers it elsewhere.
>
> First suspection would be the device driver. Perhaps you can test it with
> a different block device?
>
> -Andi
Andi,
CONFIG_DEBUG_SLAB and CONFIG_DEBUG_PAGEALLOC show no corruption.
It turns out that NUMA is the culprit and LVM has no effect on any
configuration. The machine has 6 memory slots. If I have 4 sticks of
2GB in the machine, it makes zone 0 w/ 8GB and zone 1 disabled. If I
add two more 2GB sticks (total 12GB, filling all possible 6 slots),
then I have 8GB in zone 0 and 4GB in zone 1, and then a mke2fs on a
280GB /dev/sdb1 works fine.
In conclusion:
NUMA + 12 GB -> works
NUMA + 8 GB -> OOPS
no numa + 8 GB -> works
And I suspect that without CONFIG_SMP, 8GB will also work, by virtue
of there only being one NUMA zone. I haven't tested that yet, but
Randy referred me to these posts, where somebody with the same problem
confirms that disabling NUMA and SMP fixes his problem:
http://marc.theaimsgroup.com/?l=linux-kernel&m=109328505204081&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=109330259511819&w=2
When I boot NUMA w/ 12GB, I get:
Scanning NUMA topology in Northbridge 24
Number of nodes 2 (10010)
Node 0 MemBase 0000000000000000 Limit 00000001ffffffff
Node 1 MemBase 0000000200000000 Limit 00000002ffffffff
node 1 shift 24 addr 200000000 conflict 0
node 1 shift 25 addr 200000000 conflict 0
Using node hash shift of 26
Bootmem setup node 0 0000000000000000-00000001ffffffff
Bootmem setup node 1 0000000200000000-00000002ffffffff
No mptable found.
On node 0 totalpages: 2097151
DMA zone: 4096 pages, LIFO batch:1
Normal zone: 2093055 pages, LIFO batch:16
HighMem zone: 0 pages, LIFO batch:1
On node 1 totalpages: 1048575
DMA zone: 0 pages, LIFO batch:1
Normal zone: 1048575 pages, LIFO batch:16
HighMem zone: 0 pages, LIFO batch:1
When I boot NUMA w/ 8GB, k8topology.c prints out:
...
Node 0 MemBase 0000000000000000 Limit 00000001ffffffff
Skipping disabled node 1
...
I can get you the full dmesg of both configs tomorrow.
Anything else you'd like to see?
- Brad
On Wed, 13 Oct 2004, Brad Fitzpatrick wrote:
...
>
> It turns out that NUMA is the culprit and LVM has no effect on any
> configuration. The machine has 6 memory slots. If I have 4 sticks of
> 2GB in the machine, it makes zone 0 w/ 8GB and zone 1 disabled. If I
> add two more 2GB sticks (total 12GB, filling all possible 6 slots),
> then I have 8GB in zone 0 and 4GB in zone 1, and then a mke2fs on a
> 280GB /dev/sdb1 works fine.
>
> In conclusion:
>
> NUMA + 12 GB -> works
> NUMA + 8 GB -> OOPS
> no numa + 8 GB -> works
>
> And I suspect that without CONFIG_SMP, 8GB will also work, by virtue
> of there only being one NUMA zone. I haven't tested that yet, but
> Randy referred me to these posts, where somebody with the same problem
> confirms that disabling NUMA and SMP fixes his problem:
>
> http://marc.theaimsgroup.com/?l=linux-kernel&m=109328505204081&w=2
> http://marc.theaimsgroup.com/?l=linux-kernel&m=109330259511819&w=2
>
Replying to myself, in hopes it'll benefit others.
It also works with NUMA and 8GB if I balance the memory between the two
processors, 2 sticks of 2GB on each.
Then the zones look like:
Scanning NUMA topology in Northbridge 24
Number of nodes 2 (10010)
Node 0 MemBase 0000000000000000 Limit 00000000ffffffff
Node 1 MemBase 0000000100000000 Limit 00000001ffffffff
node 1 shift 24 addr 100000000 conflict 0
node 1 shift 25 addr 1fe000000 conflict 0
Using node hash shift of 26
Bootmem setup node 0 0000000000000000-00000000ffffffff
Bootmem setup node 1 0000000100000000-00000001ffffffff
No mptable found.
On node 0 totalpages: 1048575
DMA zone: 4096 pages, LIFO batch:1
Normal zone: 1044479 pages, LIFO batch:16
HighMem zone: 0 pages, LIFO batch:1
On node 1 totalpages: 1048575
DMA zone: 0 pages, LIFO batch:1
Normal zone: 1048575 pages, LIFO batch:16
HighMem zone: 0 pages, LIFO batch:1
The eServer 325 manual in one place says you must add memory in
pairs, and in other places seems to suggest (but not say it's required?)
that you add memory in 1 & 2, then 5 & 6 (with SMP), and then 3 & 4.
It would've been nice if Linux either flat-out halted on boot saying,
"This is stupid, move your memory around" or ran in a degraded memory
configuration reliably without crashing. The scary part is that the BIOS
was cool with the configuration and Linux was /mostly/ cool with it, most
of the time, until certain usage patterns (like mke2fs) made it crash.
Andi, I can still get you information, but for now I'll mark this one down
as "user error". Perhals you'll have an idea on how to make Linux more
robust if another user winds up with this same issue, as Christopher
Swingley and I did.
- Brad
On Thu, Oct 14, 2004 at 11:29:52AM -0700, Brad Fitzpatrick wrote:
> On Wed, 13 Oct 2004, Brad Fitzpatrick wrote:
>
> ...
> >
> > It turns out that NUMA is the culprit and LVM has no effect on any
> > configuration. The machine has 6 memory slots. If I have 4 sticks of
> > 2GB in the machine, it makes zone 0 w/ 8GB and zone 1 disabled. If I
> > add two more 2GB sticks (total 12GB, filling all possible 6 slots),
> > then I have 8GB in zone 0 and 4GB in zone 1, and then a mke2fs on a
> > 280GB /dev/sdb1 works fine.
> >
> > In conclusion:
> >
> > NUMA + 12 GB -> works
> > NUMA + 8 GB -> OOPS
> > no numa + 8 GB -> works
> >
> > And I suspect that without CONFIG_SMP, 8GB will also work, by virtue
> > of there only being one NUMA zone. I haven't tested that yet, but
> > Randy referred me to these posts, where somebody with the same problem
> > confirms that disabling NUMA and SMP fixes his problem:
> >
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=109328505204081&w=2
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=109330259511819&w=2
> >
>
> Replying to myself, in hopes it'll benefit others.
>
> It also works with NUMA and 8GB if I balance the memory between the two
> processors, 2 sticks of 2GB on each.
>
> Then the zones look like:
>
> Scanning NUMA topology in Northbridge 24
> Number of nodes 2 (10010)
> Node 0 MemBase 0000000000000000 Limit 00000000ffffffff
> Node 1 MemBase 0000000100000000 Limit 00000001ffffffff
> node 1 shift 24 addr 100000000 conflict 0
> node 1 shift 25 addr 1fe000000 conflict 0
> Using node hash shift of 26
> Bootmem setup node 0 0000000000000000-00000000ffffffff
> Bootmem setup node 1 0000000100000000-00000001ffffffff
> No mptable found.
> On node 0 totalpages: 1048575
> DMA zone: 4096 pages, LIFO batch:1
> Normal zone: 1044479 pages, LIFO batch:16
> HighMem zone: 0 pages, LIFO batch:1
> On node 1 totalpages: 1048575
> DMA zone: 0 pages, LIFO batch:1
> Normal zone: 1048575 pages, LIFO batch:16
> HighMem zone: 0 pages, LIFO batch:1
>
>
> The eServer 325 manual in one place says you must add memory in
> pairs, and in other places seems to suggest (but not say it's required?)
> that you add memory in 1 & 2, then 5 & 6 (with SMP), and then 3 & 4.
>
> It would've been nice if Linux either flat-out halted on boot saying,
> "This is stupid, move your memory around" or ran in a degraded memory
> configuration reliably without crashing. The scary part is that the BIOS
> was cool with the configuration and Linux was /mostly/ cool with it, most
> of the time, until certain usage patterns (like mke2fs) made it crash.
Linux has no such limitations in the NUMA code, so it wouldn't
make sense to test for it. Most likely it was just an hardware problem -
specific memory configurations don't work correctly in your motherboard
and in specific memory access patterns triggered by mkfs, possible including
bus master IO from the RAID controller, it corrupted data.
It's not possible in Linux to test for all the limitations all the systems
it could run on may have in their memory configuration. Linux just assumes that it can use the memory it gets passed. I would talk to your BIOS vendor about
better sanity checking or perhaps your motherboard vendor about not
adding such restrictions
> Andi, I can still get you information, but for now I'll mark this one down
> as "user error". Perhals you'll have an idea on how to make Linux more
> robust if another user winds up with this same issue, as Christopher
> Swingley and I did.
There is no way to make Linux robust with unreliable memory subsystems,
sorry. It would be like trying to make a human more robust
with an unreliable O2 supply. Memory just has to work.
There are tools like memtest86 that can be used to diagnose such
things in a controlled environment. But in your case it probably
involved PCI traffic, and that is hard to simulate with standard tools.
-Andi