I hit the below oops while playing with "wakeup" tracer. Just enabled
tracing and ran "while true; do cat trace; done" on a 8-way systems with
-tip at commit 9e25bacc221318a29d37425e054c01047edad7f5.
Attached the .config.
Thanks
Maneesh
BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
PGD 219147067 PUD 1e449e067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:0c/0000:0c:00.0/irq
CPU 5
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
Pid: 5937, comm: cat Not tainted 2.6.29-tip-test #1 eserver xSeries 366-[88632RA]-
RIP: 0010:[<ffffffff802d2258>] [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
RSP: 0000:ffff8801cf4cdea0 EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff88022f232820 RCX: 0000000000000046
RDX: ffff8800281989a0 RSI: ffffffff8020d56d RDI: ffffffff811de480
RBP: ffff8801cf4cded0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000046
R13: 0000000000000000 R14: ffff8801e4471820 R15: 0000000000000005
FS: 00007f607411c6e0(0000) GS:ffff88002818b000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000001da19a000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process cat (pid: 5937, threadinfo ffff8801cf4cc000, task ffff8801e4471820)
Stack:
ffff8800281989a0 ffff8802191e14b0 ffff88002819dd40 00007fff7c130640
0000000000000000 ffff88002819dd40 ffff8801cf4cdf70 ffffffff806f8264
0000000000000002 ffff8801cf4cdf48 ffff8801e4471820 ffff88022f232820
Call Trace:
[<ffffffff806f8264>] schedule+0xd99/0x10f9
[<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
[<ffffffff8020d56d>] retint_careful+0x12/0x2e
Code: f2 83 3d 1b b0 9f 00 00 74 77 48 3b 1d a2 de f2 00 75 6e 48 8b 4d 00 48 8b 3d 8d de f2 00 45 89 e8 48 8b 01 48 8b 71 08 4c 89 e1 <48> 8b 50 08 e8 f7 9f ff ff 48 8b 3d 70 de f2 00 45 89 e8 48 89
RIP [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
RSP <ffff8801cf4cdea0>
CR2: 0000000000000008
---[ end trace ab11daa90bb147ba ]---
On Thu, Mar 26, 2009 at 05:58:07PM +0530, Maneesh Soni wrote:
>
> I hit the below oops while playing with "wakeup" tracer. Just enabled
> tracing and ran "while true; do cat trace; done" on a 8-way systems with
> -tip at commit 9e25bacc221318a29d37425e054c01047edad7f5.
>
>
> Attached the .config.
>
> Thanks
> Maneesh
>
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> IP: [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
> PGD 219147067 PUD 1e449e067 PMD 0
> Oops: 0000 [#1] SMP
> last sysfs file: /sys/devices/pci0000:0c/0000:0c:00.0/irq
> CPU 5
> Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
> Pid: 5937, comm: cat Not tainted 2.6.29-tip-test #1 eserver xSeries 366-[88632RA]-
> RIP: 0010:[<ffffffff802d2258>] [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
> RSP: 0000:ffff8801cf4cdea0 EFLAGS: 00010046
> RAX: 0000000000000000 RBX: ffff88022f232820 RCX: 0000000000000046
> RDX: ffff8800281989a0 RSI: ffffffff8020d56d RDI: ffffffff811de480
> RBP: ffff8801cf4cded0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000046
> R13: 0000000000000000 R14: ffff8801e4471820 R15: 0000000000000005
> FS: 00007f607411c6e0(0000) GS:ffff88002818b000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000008 CR3: 00000001da19a000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process cat (pid: 5937, threadinfo ffff8801cf4cc000, task ffff8801e4471820)
> Stack:
> ffff8800281989a0 ffff8802191e14b0 ffff88002819dd40 00007fff7c130640
> 0000000000000000 ffff88002819dd40 ffff8801cf4cdf70 ffffffff806f8264
> 0000000000000002 ffff8801cf4cdf48 ffff8801e4471820 ffff88022f232820
> Call Trace:
> [<ffffffff806f8264>] schedule+0xd99/0x10f9
> [<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> [<ffffffff8020d56d>] retint_careful+0x12/0x2e
> Code: f2 83 3d 1b b0 9f 00 00 74 77 48 3b 1d a2 de f2 00 75 6e 48 8b 4d 00 48 8b 3d 8d de f2 00 45 89 e8 48 8b 01 48 8b 71 08 4c 89 e1 <48> 8b 50 08 e8 f7 9f ff ff 48 8b 3d 70 de f2 00 45 89 e8 48 89
> RIP [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
> RSP <ffff8801cf4cdea0>
> CR2: 0000000000000008
> ---[ end trace ab11daa90bb147ba ]---
>
I've tested the same config and ran your script with wakeup tracer but nothing
weird happened.
Though I've tested on UP so may be I should try on SMP.
Thanks,
Frederic.
> #
> # Automatically generated make config: don't edit
> # Linux kernel version: 2.6.29
> # Thu Mar 26 17:19:45 2009
> #
> CONFIG_64BIT=y
> # CONFIG_X86_32 is not set
> CONFIG_X86_64=y
> CONFIG_X86=y
> CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
> CONFIG_GENERIC_TIME=y
> CONFIG_GENERIC_CMOS_UPDATE=y
> CONFIG_CLOCKSOURCE_WATCHDOG=y
> CONFIG_GENERIC_CLOCKEVENTS=y
> CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
> CONFIG_LOCKDEP_SUPPORT=y
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_LATENCYTOP_SUPPORT=y
> CONFIG_FAST_CMPXCHG_LOCAL=y
> CONFIG_MMU=y
> CONFIG_ZONE_DMA=y
> CONFIG_GENERIC_ISA_DMA=y
> CONFIG_GENERIC_IOMAP=y
> CONFIG_GENERIC_BUG=y
> CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
> CONFIG_GENERIC_HWEIGHT=y
> CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> CONFIG_RWSEM_GENERIC_SPINLOCK=y
> # CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
> CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
> CONFIG_GENERIC_CALIBRATE_DELAY=y
> CONFIG_GENERIC_TIME_VSYSCALL=y
> CONFIG_ARCH_HAS_CPU_RELAX=y
> CONFIG_ARCH_HAS_DEFAULT_IDLE=y
> CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
> CONFIG_HAVE_SETUP_PER_CPU_AREA=y
> CONFIG_HAVE_DYNAMIC_PER_CPU_AREA=y
> CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
> CONFIG_ARCH_HIBERNATION_POSSIBLE=y
> CONFIG_ARCH_SUSPEND_POSSIBLE=y
> CONFIG_ZONE_DMA32=y
> CONFIG_ARCH_POPULATES_NODE_MAP=y
> CONFIG_AUDIT_ARCH=y
> CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
> CONFIG_GENERIC_HARDIRQS=y
> CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y
> CONFIG_GENERIC_IRQ_PROBE=y
> CONFIG_GENERIC_PENDING_IRQ=y
> CONFIG_USE_GENERIC_SMP_HELPERS=y
> CONFIG_X86_64_SMP=y
> CONFIG_X86_HT=y
> CONFIG_X86_TRAMPOLINE=y
> # CONFIG_KTIME_SCALAR is not set
> CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
>
> #
> # General setup
> #
> CONFIG_EXPERIMENTAL=y
> CONFIG_LOCK_KERNEL=y
> CONFIG_INIT_ENV_ARG_LIMIT=32
> CONFIG_LOCALVERSION="-test"
> # CONFIG_LOCALVERSION_AUTO is not set
> CONFIG_HAVE_KERNEL_GZIP=y
> CONFIG_HAVE_KERNEL_BZIP2=y
> CONFIG_HAVE_KERNEL_LZMA=y
> CONFIG_KERNEL_GZIP=y
> # CONFIG_KERNEL_BZIP2 is not set
> # CONFIG_KERNEL_LZMA is not set
> CONFIG_SWAP=y
> CONFIG_SYSVIPC=y
> CONFIG_SYSVIPC_SYSCTL=y
> CONFIG_POSIX_MQUEUE=y
> CONFIG_BSD_PROCESS_ACCT=y
> # CONFIG_BSD_PROCESS_ACCT_V3 is not set
> CONFIG_TASKSTATS=y
> CONFIG_TASK_DELAY_ACCT=y
> # CONFIG_TASK_XACCT is not set
> CONFIG_AUDIT=y
> CONFIG_AUDITSYSCALL=y
> CONFIG_AUDIT_TREE=y
>
> #
> # RCU Subsystem
> #
> CONFIG_CLASSIC_RCU=y
> # CONFIG_TREE_RCU is not set
> # CONFIG_PREEMPT_RCU is not set
> # CONFIG_TREE_RCU_TRACE is not set
> # CONFIG_PREEMPT_RCU_TRACE is not set
> # CONFIG_IKCONFIG is not set
> CONFIG_LOG_BUF_SHIFT=17
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> # CONFIG_GROUP_SCHED is not set
> # CONFIG_CGROUPS is not set
> CONFIG_SYSFS_DEPRECATED=y
> CONFIG_SYSFS_DEPRECATED_V2=y
> CONFIG_RELAY=y
> CONFIG_NAMESPACES=y
> # CONFIG_UTS_NS is not set
> # CONFIG_IPC_NS is not set
> # CONFIG_USER_NS is not set
> # CONFIG_PID_NS is not set
> # CONFIG_NET_NS is not set
> CONFIG_BLK_DEV_INITRD=y
> CONFIG_INITRAMFS_SOURCE=""
> CONFIG_RD_GZIP=y
> # CONFIG_RD_BZIP2 is not set
> # CONFIG_RD_LZMA is not set
> CONFIG_INITRAMFS_COMPRESSION_NONE=y
> # CONFIG_INITRAMFS_COMPRESSION_GZIP is not set
> # CONFIG_INITRAMFS_COMPRESSION_BZIP2 is not set
> # CONFIG_INITRAMFS_COMPRESSION_LZMA is not set
> CONFIG_CC_OPTIMIZE_FOR_SIZE=y
> CONFIG_SYSCTL=y
> CONFIG_ANON_INODES=y
> # CONFIG_EMBEDDED is not set
> CONFIG_UID16=y
> CONFIG_SYSCTL_SYSCALL=y
> CONFIG_KALLSYMS=y
> # CONFIG_KALLSYMS_ALL is not set
> CONFIG_KALLSYMS_EXTRA_PASS=y
> CONFIG_HOTPLUG=y
> CONFIG_PRINTK=y
> CONFIG_BUG=y
> CONFIG_ELF_CORE=y
> CONFIG_PCSPKR_PLATFORM=y
> CONFIG_BASE_FULL=y
> CONFIG_FUTEX=y
> CONFIG_EPOLL=y
> CONFIG_SIGNALFD=y
> CONFIG_TIMERFD=y
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_AIO=y
> CONFIG_HAVE_PERF_COUNTERS=y
>
> #
> # Performance Counters
> #
> CONFIG_PERF_COUNTERS=y
> CONFIG_EVENT_PROFILE=y
> CONFIG_VM_EVENT_COUNTERS=y
> CONFIG_PCI_QUIRKS=y
> CONFIG_COMPAT_BRK=y
> CONFIG_SLAB=y
> # CONFIG_SLUB is not set
> # CONFIG_SLOB is not set
> CONFIG_PROFILING=y
> CONFIG_TRACEPOINTS=y
> CONFIG_MARKERS=y
> # CONFIG_OPROFILE is not set
> CONFIG_HAVE_OPROFILE=y
> CONFIG_KPROBES=y
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
> CONFIG_KRETPROBES=y
> CONFIG_HAVE_IOREMAP_PROT=y
> CONFIG_HAVE_KPROBES=y
> CONFIG_HAVE_KRETPROBES=y
> CONFIG_HAVE_ARCH_TRACEHOOK=y
> CONFIG_HAVE_DMA_API_DEBUG=y
> # CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
> CONFIG_SLABINFO=y
> CONFIG_RT_MUTEXES=y
> CONFIG_BASE_SMALL=0
> CONFIG_MODULES=y
> # CONFIG_MODULE_FORCE_LOAD is not set
> CONFIG_MODULE_UNLOAD=y
> # CONFIG_MODULE_FORCE_UNLOAD is not set
> CONFIG_MODVERSIONS=y
> CONFIG_MODULE_SRCVERSION_ALL=y
> CONFIG_STOP_MACHINE=y
> CONFIG_BLOCK=y
> # CONFIG_BLK_DEV_BSG is not set
> # CONFIG_BLK_DEV_INTEGRITY is not set
> CONFIG_BLOCK_COMPAT=y
>
> #
> # IO Schedulers
> #
> CONFIG_IOSCHED_NOOP=y
> CONFIG_IOSCHED_AS=y
> CONFIG_IOSCHED_DEADLINE=y
> CONFIG_IOSCHED_CFQ=y
> # CONFIG_DEFAULT_AS is not set
> # CONFIG_DEFAULT_DEADLINE is not set
> CONFIG_DEFAULT_CFQ=y
> # CONFIG_DEFAULT_NOOP is not set
> CONFIG_DEFAULT_IOSCHED="cfq"
> CONFIG_FREEZER=y
>
> #
> # Processor type and features
> #
> # CONFIG_NO_HZ is not set
> # CONFIG_HIGH_RES_TIMERS is not set
> CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
> CONFIG_SMP=y
> # CONFIG_X86_X2APIC is not set
> # CONFIG_SPARSE_IRQ is not set
> CONFIG_X86_MPPARSE=y
> CONFIG_X86_EXTENDED_PLATFORM=y
> # CONFIG_X86_VSMP is not set
> # CONFIG_X86_UV is not set
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> # CONFIG_PARAVIRT_GUEST is not set
> # CONFIG_MEMTEST is not set
> # CONFIG_M386 is not set
> # CONFIG_M486 is not set
> # CONFIG_M586 is not set
> # CONFIG_M586TSC is not set
> # CONFIG_M586MMX is not set
> # CONFIG_M686 is not set
> # CONFIG_MPENTIUMII is not set
> # CONFIG_MPENTIUMIII is not set
> # CONFIG_MPENTIUMM is not set
> # CONFIG_MPENTIUM4 is not set
> # CONFIG_MK6 is not set
> # CONFIG_MK7 is not set
> # CONFIG_MK8 is not set
> # CONFIG_MCRUSOE is not set
> # CONFIG_MEFFICEON is not set
> # CONFIG_MWINCHIPC6 is not set
> # CONFIG_MWINCHIP3D is not set
> # CONFIG_MGEODEGX1 is not set
> # CONFIG_MGEODE_LX is not set
> # CONFIG_MCYRIXIII is not set
> # CONFIG_MVIAC3_2 is not set
> # CONFIG_MVIAC7 is not set
> # CONFIG_MPSC is not set
> # CONFIG_MCORE2 is not set
> CONFIG_GENERIC_CPU=y
> CONFIG_X86_CPU=y
> CONFIG_X86_L1_CACHE_BYTES=64
> CONFIG_X86_INTERNODE_CACHE_BYTES=64
> CONFIG_X86_CMPXCHG=y
> CONFIG_X86_L1_CACHE_SHIFT=6
> CONFIG_X86_WP_WORKS_OK=y
> CONFIG_X86_TSC=y
> CONFIG_X86_CMPXCHG64=y
> CONFIG_X86_CMOV=y
> CONFIG_X86_MINIMUM_CPU_FAMILY=64
> CONFIG_X86_DEBUGCTLMSR=y
> CONFIG_CPU_SUP_INTEL=y
> CONFIG_CPU_SUP_AMD=y
> CONFIG_CPU_SUP_CENTAUR=y
> CONFIG_X86_DS=y
> CONFIG_X86_PTRACE_BTS=y
> CONFIG_HPET_TIMER=y
> CONFIG_HPET_EMULATE_RTC=y
> CONFIG_DMI=y
> CONFIG_GART_IOMMU=y
> CONFIG_CALGARY_IOMMU=y
> # CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT is not set
> CONFIG_AMD_IOMMU=y
> # CONFIG_AMD_IOMMU_STATS is not set
> CONFIG_SWIOTLB=y
> CONFIG_IOMMU_HELPER=y
> CONFIG_IOMMU_API=y
> # CONFIG_MAXSMP is not set
> CONFIG_NR_CPUS=255
> CONFIG_SCHED_SMT=y
> CONFIG_SCHED_MC=y
> # CONFIG_PREEMPT_NONE is not set
> CONFIG_PREEMPT_VOLUNTARY=y
> # CONFIG_PREEMPT is not set
> CONFIG_X86_LOCAL_APIC=y
> CONFIG_X86_IO_APIC=y
> # CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
> CONFIG_X86_MCE=y
> CONFIG_X86_MCE_INTEL=y
> CONFIG_X86_MCE_AMD=y
> CONFIG_X86_MCE_THRESHOLD=y
> # CONFIG_I8K is not set
> # CONFIG_MICROCODE is not set
> CONFIG_X86_MSR=y
> CONFIG_X86_CPUID=y
> # CONFIG_X86_CPU_DEBUG is not set
> CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
> CONFIG_DIRECT_GBPAGES=y
> CONFIG_NUMA=y
> CONFIG_K8_NUMA=y
> CONFIG_X86_64_ACPI_NUMA=y
> CONFIG_NODES_SPAN_OTHER_NODES=y
> # CONFIG_NUMA_EMU is not set
> CONFIG_NODES_SHIFT=6
> CONFIG_ARCH_SPARSEMEM_DEFAULT=y
> CONFIG_ARCH_SPARSEMEM_ENABLE=y
> CONFIG_ARCH_SELECT_MEMORY_MODEL=y
> CONFIG_ARCH_MEMORY_PROBE=y
> CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
> CONFIG_SELECT_MEMORY_MODEL=y
> # CONFIG_FLATMEM_MANUAL is not set
> # CONFIG_DISCONTIGMEM_MANUAL is not set
> CONFIG_SPARSEMEM_MANUAL=y
> CONFIG_SPARSEMEM=y
> CONFIG_NEED_MULTIPLE_NODES=y
> CONFIG_HAVE_MEMORY_PRESENT=y
> CONFIG_SPARSEMEM_EXTREME=y
> CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
> CONFIG_SPARSEMEM_VMEMMAP=y
> CONFIG_MEMORY_HOTPLUG=y
> CONFIG_MEMORY_HOTPLUG_SPARSE=y
> # CONFIG_MEMORY_HOTREMOVE is not set
> CONFIG_PAGEFLAGS_EXTENDED=y
> CONFIG_SPLIT_PTLOCK_CPUS=4
> CONFIG_MIGRATION=y
> CONFIG_PHYS_ADDR_T_64BIT=y
> CONFIG_ZONE_DMA_FLAG=1
> CONFIG_BOUNCE=y
> CONFIG_VIRT_TO_BUS=y
> CONFIG_UNEVICTABLE_LRU=y
> # CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
> CONFIG_X86_RESERVE_LOW_64K=y
> CONFIG_MTRR=y
> CONFIG_MTRR_SANITIZER=y
> CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
> CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
> # CONFIG_X86_PAT is not set
> # CONFIG_EFI is not set
> # CONFIG_SECCOMP is not set
> # CONFIG_CC_STACKPROTECTOR is not set
> # CONFIG_HZ_100 is not set
> # CONFIG_HZ_250 is not set
> # CONFIG_HZ_300 is not set
> CONFIG_HZ_1000=y
> CONFIG_HZ=1000
> # CONFIG_SCHED_HRTICK is not set
> CONFIG_KEXEC=y
> CONFIG_CRASH_DUMP=y
> CONFIG_PHYSICAL_START=0x200000
> CONFIG_RELOCATABLE=y
> CONFIG_PHYSICAL_ALIGN=0x200000
> CONFIG_HOTPLUG_CPU=y
> CONFIG_COMPAT_VDSO=y
> # CONFIG_CMDLINE_BOOL is not set
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
> CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID=y
>
> #
> # Power management and ACPI options
> #
> CONFIG_PM=y
> # CONFIG_PM_DEBUG is not set
> CONFIG_PM_SLEEP_SMP=y
> CONFIG_PM_SLEEP=y
> CONFIG_SUSPEND=y
> CONFIG_SUSPEND_FREEZER=y
> # CONFIG_HIBERNATION is not set
> CONFIG_ACPI=y
> CONFIG_ACPI_SLEEP=y
> # CONFIG_ACPI_PROCFS is not set
> CONFIG_ACPI_PROCFS_POWER=y
> CONFIG_ACPI_SYSFS_POWER=y
> CONFIG_ACPI_PROC_EVENT=y
> CONFIG_ACPI_AC=m
> CONFIG_ACPI_BATTERY=m
> CONFIG_ACPI_BUTTON=m
> CONFIG_ACPI_FAN=y
> CONFIG_ACPI_DOCK=y
> CONFIG_ACPI_PROCESSOR=y
> CONFIG_ACPI_HOTPLUG_CPU=y
> CONFIG_ACPI_THERMAL=y
> CONFIG_ACPI_NUMA=y
> # CONFIG_ACPI_CUSTOM_DSDT is not set
> CONFIG_ACPI_BLACKLIST_YEAR=0
> # CONFIG_ACPI_DEBUG is not set
> # CONFIG_ACPI_PCI_SLOT is not set
> CONFIG_X86_PM_TIMER=y
> CONFIG_ACPI_CONTAINER=y
> CONFIG_ACPI_HOTPLUG_MEMORY=m
> CONFIG_ACPI_SBS=m
>
> #
> # CPU Frequency scaling
> #
> CONFIG_CPU_FREQ=y
> CONFIG_CPU_FREQ_DEBUG=y
> # CONFIG_CPU_FREQ_STAT is not set
> # CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
> # CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
> CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
> # CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
> # CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
> CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> # CONFIG_CPU_FREQ_GOV_POWERSAVE is not set
> CONFIG_CPU_FREQ_GOV_USERSPACE=y
> # CONFIG_CPU_FREQ_GOV_ONDEMAND is not set
> # CONFIG_CPU_FREQ_GOV_CONSERVATIVE is not set
>
> #
> # CPUFreq processor drivers
> #
> # CONFIG_X86_ACPI_CPUFREQ is not set
> # CONFIG_X86_POWERNOW_K8 is not set
> # CONFIG_X86_SPEEDSTEP_CENTRINO is not set
> # CONFIG_X86_P4_CLOCKMOD is not set
>
> #
> # shared options
> #
> # CONFIG_X86_SPEEDSTEP_LIB is not set
> CONFIG_CPU_IDLE=y
> CONFIG_CPU_IDLE_GOV_LADDER=y
>
> #
> # Memory power savings
> #
> # CONFIG_I7300_IDLE is not set
>
> #
> # Bus options (PCI etc.)
> #
> CONFIG_PCI=y
> CONFIG_PCI_DIRECT=y
> CONFIG_PCI_MMCONFIG=y
> CONFIG_PCI_DOMAINS=y
> # CONFIG_DMAR is not set
> # CONFIG_INTR_REMAP is not set
> CONFIG_PCIEPORTBUS=y
> # CONFIG_HOTPLUG_PCI_PCIE is not set
> CONFIG_PCIEAER=y
> # CONFIG_PCIEASPM is not set
> CONFIG_ARCH_SUPPORTS_MSI=y
> CONFIG_PCI_MSI=y
> CONFIG_PCI_LEGACY=y
> # CONFIG_PCI_DEBUG is not set
> # CONFIG_PCI_STUB is not set
> CONFIG_HT_IRQ=y
> CONFIG_ISA_DMA_API=y
> CONFIG_K8_NB=y
> CONFIG_PCCARD=y
> # CONFIG_PCMCIA_DEBUG is not set
> CONFIG_PCMCIA=y
> CONFIG_PCMCIA_LOAD_CIS=y
> CONFIG_PCMCIA_IOCTL=y
> CONFIG_CARDBUS=y
>
> #
> # PC-card bridges
> #
> CONFIG_YENTA=y
> CONFIG_YENTA_O2=y
> CONFIG_YENTA_RICOH=y
> CONFIG_YENTA_TI=y
> CONFIG_YENTA_ENE_TUNE=y
> CONFIG_YENTA_TOSHIBA=y
> # CONFIG_PD6729 is not set
> # CONFIG_I82092 is not set
> CONFIG_PCCARD_NONSTATIC=y
> CONFIG_HOTPLUG_PCI=y
> # CONFIG_HOTPLUG_PCI_FAKE is not set
> # CONFIG_HOTPLUG_PCI_ACPI is not set
> # CONFIG_HOTPLUG_PCI_CPCI is not set
> # CONFIG_HOTPLUG_PCI_SHPC is not set
>
> #
> # Executable file formats / Emulations
> #
> CONFIG_BINFMT_ELF=y
> CONFIG_COMPAT_BINFMT_ELF=y
> # CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
> # CONFIG_HAVE_AOUT is not set
> CONFIG_BINFMT_MISC=y
> CONFIG_IA32_EMULATION=y
> # CONFIG_IA32_AOUT is not set
> CONFIG_COMPAT=y
> CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
> CONFIG_SYSVIPC_COMPAT=y
> CONFIG_NET=y
>
> #
> # Networking options
> #
> CONFIG_COMPAT_NET_DEV_OPS=y
> CONFIG_PACKET=y
> CONFIG_PACKET_MMAP=y
> CONFIG_UNIX=y
> CONFIG_XFRM=y
> CONFIG_XFRM_USER=y
> # CONFIG_XFRM_SUB_POLICY is not set
> # CONFIG_XFRM_MIGRATE is not set
> # CONFIG_XFRM_STATISTICS is not set
> # CONFIG_NET_KEY is not set
> CONFIG_INET=y
> CONFIG_IP_MULTICAST=y
> CONFIG_IP_ADVANCED_ROUTER=y
> CONFIG_ASK_IP_FIB_HASH=y
> # CONFIG_IP_FIB_TRIE is not set
> CONFIG_IP_FIB_HASH=y
> CONFIG_IP_MULTIPLE_TABLES=y
> CONFIG_IP_ROUTE_MULTIPATH=y
> CONFIG_IP_ROUTE_VERBOSE=y
> # CONFIG_IP_PNP is not set
> # CONFIG_NET_IPIP is not set
> # CONFIG_NET_IPGRE is not set
> CONFIG_IP_MROUTE=y
> CONFIG_IP_PIMSM_V1=y
> CONFIG_IP_PIMSM_V2=y
> # CONFIG_ARPD is not set
> CONFIG_SYN_COOKIES=y
> # CONFIG_INET_AH is not set
> # CONFIG_INET_ESP is not set
> # CONFIG_INET_IPCOMP is not set
> # CONFIG_INET_XFRM_TUNNEL is not set
> # CONFIG_INET_TUNNEL is not set
> # CONFIG_INET_XFRM_MODE_TRANSPORT is not set
> # CONFIG_INET_XFRM_MODE_TUNNEL is not set
> CONFIG_INET_XFRM_MODE_BEET=y
> CONFIG_INET_LRO=y
> # CONFIG_INET_DIAG is not set
> CONFIG_TCP_CONG_ADVANCED=y
> CONFIG_TCP_CONG_BIC=y
> # CONFIG_TCP_CONG_CUBIC is not set
> # CONFIG_TCP_CONG_WESTWOOD is not set
> # CONFIG_TCP_CONG_HTCP is not set
> # CONFIG_TCP_CONG_HSTCP is not set
> # CONFIG_TCP_CONG_HYBLA is not set
> # CONFIG_TCP_CONG_VEGAS is not set
> # CONFIG_TCP_CONG_SCALABLE is not set
> # CONFIG_TCP_CONG_LP is not set
> # CONFIG_TCP_CONG_VENO is not set
> # CONFIG_TCP_CONG_YEAH is not set
> # CONFIG_TCP_CONG_ILLINOIS is not set
> CONFIG_DEFAULT_BIC=y
> # CONFIG_DEFAULT_CUBIC is not set
> # CONFIG_DEFAULT_HTCP is not set
> # CONFIG_DEFAULT_VEGAS is not set
> # CONFIG_DEFAULT_WESTWOOD is not set
> # CONFIG_DEFAULT_RENO is not set
> CONFIG_DEFAULT_TCP_CONG="bic"
> # CONFIG_TCP_MD5SIG is not set
> CONFIG_IPV6=m
> CONFIG_IPV6_PRIVACY=y
> CONFIG_IPV6_ROUTER_PREF=y
> CONFIG_IPV6_ROUTE_INFO=y
> CONFIG_IPV6_OPTIMISTIC_DAD=y
> # CONFIG_INET6_AH is not set
> # CONFIG_INET6_ESP is not set
> # CONFIG_INET6_IPCOMP is not set
> # CONFIG_IPV6_MIP6 is not set
> # CONFIG_INET6_XFRM_TUNNEL is not set
> # CONFIG_INET6_TUNNEL is not set
> # CONFIG_INET6_XFRM_MODE_TRANSPORT is not set
> # CONFIG_INET6_XFRM_MODE_TUNNEL is not set
> # CONFIG_INET6_XFRM_MODE_BEET is not set
> # CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set
> # CONFIG_IPV6_SIT is not set
> # CONFIG_IPV6_TUNNEL is not set
> CONFIG_IPV6_MULTIPLE_TABLES=y
> # CONFIG_IPV6_SUBTREES is not set
> # CONFIG_IPV6_MROUTE is not set
> CONFIG_NETLABEL=y
> CONFIG_NETWORK_SECMARK=y
> CONFIG_NETFILTER=y
> # CONFIG_NETFILTER_DEBUG is not set
> CONFIG_NETFILTER_ADVANCED=y
>
> #
> # Core Netfilter Configuration
> #
> # CONFIG_NETFILTER_NETLINK_QUEUE is not set
> # CONFIG_NETFILTER_NETLINK_LOG is not set
> # CONFIG_NF_CONNTRACK is not set
> CONFIG_NETFILTER_XTABLES=m
> # CONFIG_NETFILTER_XT_TARGET_CLASSIFY is not set
> # CONFIG_NETFILTER_XT_TARGET_MARK is not set
> # CONFIG_NETFILTER_XT_TARGET_NFLOG is not set
> # CONFIG_NETFILTER_XT_TARGET_NFQUEUE is not set
> # CONFIG_NETFILTER_XT_TARGET_RATEEST is not set
> # CONFIG_NETFILTER_XT_TARGET_SECMARK is not set
> # CONFIG_NETFILTER_XT_TARGET_TCPMSS is not set
> # CONFIG_NETFILTER_XT_MATCH_COMMENT is not set
> # CONFIG_NETFILTER_XT_MATCH_DCCP is not set
> # CONFIG_NETFILTER_XT_MATCH_DSCP is not set
> # CONFIG_NETFILTER_XT_MATCH_ESP is not set
> # CONFIG_NETFILTER_XT_MATCH_HASHLIMIT is not set
> # CONFIG_NETFILTER_XT_MATCH_IPRANGE is not set
> # CONFIG_NETFILTER_XT_MATCH_LENGTH is not set
> # CONFIG_NETFILTER_XT_MATCH_LIMIT is not set
> # CONFIG_NETFILTER_XT_MATCH_MAC is not set
> # CONFIG_NETFILTER_XT_MATCH_MARK is not set
> # CONFIG_NETFILTER_XT_MATCH_MULTIPORT is not set
> # CONFIG_NETFILTER_XT_MATCH_OWNER is not set
> # CONFIG_NETFILTER_XT_MATCH_POLICY is not set
> # CONFIG_NETFILTER_XT_MATCH_PKTTYPE is not set
> # CONFIG_NETFILTER_XT_MATCH_QUOTA is not set
> # CONFIG_NETFILTER_XT_MATCH_RATEEST is not set
> # CONFIG_NETFILTER_XT_MATCH_REALM is not set
> # CONFIG_NETFILTER_XT_MATCH_RECENT is not set
> # CONFIG_NETFILTER_XT_MATCH_SCTP is not set
> # CONFIG_NETFILTER_XT_MATCH_STATISTIC is not set
> # CONFIG_NETFILTER_XT_MATCH_STRING is not set
> # CONFIG_NETFILTER_XT_MATCH_TCPMSS is not set
> # CONFIG_NETFILTER_XT_MATCH_TIME is not set
> # CONFIG_NETFILTER_XT_MATCH_U32 is not set
> # CONFIG_IP_VS is not set
>
> #
> # IP: Netfilter Configuration
> #
> # CONFIG_NF_DEFRAG_IPV4 is not set
> # CONFIG_IP_NF_QUEUE is not set
> CONFIG_IP_NF_IPTABLES=m
> # CONFIG_IP_NF_MATCH_ADDRTYPE is not set
> # CONFIG_IP_NF_MATCH_AH is not set
> # CONFIG_IP_NF_MATCH_ECN is not set
> # CONFIG_IP_NF_MATCH_TTL is not set
> CONFIG_IP_NF_FILTER=m
> # CONFIG_IP_NF_TARGET_REJECT is not set
> # CONFIG_IP_NF_TARGET_LOG is not set
> # CONFIG_IP_NF_TARGET_ULOG is not set
> # CONFIG_IP_NF_MANGLE is not set
> # CONFIG_IP_NF_RAW is not set
> # CONFIG_IP_NF_SECURITY is not set
> # CONFIG_IP_NF_ARPTABLES is not set
>
> #
> # IPv6: Netfilter Configuration
> #
> # CONFIG_IP6_NF_QUEUE is not set
> CONFIG_IP6_NF_IPTABLES=m
> # CONFIG_IP6_NF_MATCH_AH is not set
> # CONFIG_IP6_NF_MATCH_EUI64 is not set
> # CONFIG_IP6_NF_MATCH_FRAG is not set
> # CONFIG_IP6_NF_MATCH_OPTS is not set
> # CONFIG_IP6_NF_MATCH_HL is not set
> # CONFIG_IP6_NF_MATCH_IPV6HEADER is not set
> # CONFIG_IP6_NF_MATCH_MH is not set
> # CONFIG_IP6_NF_MATCH_RT is not set
> # CONFIG_IP6_NF_TARGET_LOG is not set
> CONFIG_IP6_NF_FILTER=m
> CONFIG_IP6_NF_TARGET_REJECT=m
> # CONFIG_IP6_NF_MANGLE is not set
> # CONFIG_IP6_NF_RAW is not set
> # CONFIG_IP6_NF_SECURITY is not set
> # CONFIG_IP_DCCP is not set
> # CONFIG_IP_SCTP is not set
> # CONFIG_TIPC is not set
> # CONFIG_ATM is not set
> # CONFIG_BRIDGE is not set
> # CONFIG_NET_DSA is not set
> # CONFIG_VLAN_8021Q is not set
> # CONFIG_DECNET is not set
> CONFIG_LLC=y
> # CONFIG_LLC2 is not set
> # CONFIG_IPX is not set
> # CONFIG_ATALK is not set
> # CONFIG_X25 is not set
> # CONFIG_LAPB is not set
> # CONFIG_ECONET is not set
> # CONFIG_WAN_ROUTER is not set
> CONFIG_NET_SCHED=y
>
> #
> # Queueing/Scheduling
> #
> # CONFIG_NET_SCH_CBQ is not set
> # CONFIG_NET_SCH_HTB is not set
> # CONFIG_NET_SCH_HFSC is not set
> # CONFIG_NET_SCH_PRIO is not set
> # CONFIG_NET_SCH_MULTIQ is not set
> # CONFIG_NET_SCH_RED is not set
> # CONFIG_NET_SCH_SFQ is not set
> # CONFIG_NET_SCH_TEQL is not set
> # CONFIG_NET_SCH_TBF is not set
> # CONFIG_NET_SCH_GRED is not set
> # CONFIG_NET_SCH_DSMARK is not set
> # CONFIG_NET_SCH_NETEM is not set
> # CONFIG_NET_SCH_DRR is not set
> # CONFIG_NET_SCH_INGRESS is not set
>
> #
> # Classification
> #
> CONFIG_NET_CLS=y
> # CONFIG_NET_CLS_BASIC is not set
> # CONFIG_NET_CLS_TCINDEX is not set
> # CONFIG_NET_CLS_ROUTE4 is not set
> # CONFIG_NET_CLS_FW is not set
> # CONFIG_NET_CLS_U32 is not set
> # CONFIG_NET_CLS_RSVP is not set
> # CONFIG_NET_CLS_RSVP6 is not set
> # CONFIG_NET_CLS_FLOW is not set
> CONFIG_NET_EMATCH=y
> CONFIG_NET_EMATCH_STACK=32
> # CONFIG_NET_EMATCH_CMP is not set
> # CONFIG_NET_EMATCH_NBYTE is not set
> # CONFIG_NET_EMATCH_U32 is not set
> # CONFIG_NET_EMATCH_META is not set
> # CONFIG_NET_EMATCH_TEXT is not set
> CONFIG_NET_CLS_ACT=y
> # CONFIG_NET_ACT_POLICE is not set
> # CONFIG_NET_ACT_GACT is not set
> # CONFIG_NET_ACT_MIRRED is not set
> # CONFIG_NET_ACT_IPT is not set
> # CONFIG_NET_ACT_NAT is not set
> # CONFIG_NET_ACT_PEDIT is not set
> # CONFIG_NET_ACT_SIMP is not set
> # CONFIG_NET_ACT_SKBEDIT is not set
> CONFIG_NET_SCH_FIFO=y
> # CONFIG_DCB is not set
>
> #
> # Network testing
> #
> # CONFIG_NET_PKTGEN is not set
> # CONFIG_NET_TCPPROBE is not set
> # CONFIG_HAMRADIO is not set
> # CONFIG_CAN is not set
> # CONFIG_IRDA is not set
> CONFIG_BT=m
> CONFIG_BT_L2CAP=m
> # CONFIG_BT_SCO is not set
> CONFIG_BT_RFCOMM=m
> CONFIG_BT_RFCOMM_TTY=y
> # CONFIG_BT_BNEP is not set
> CONFIG_BT_HIDP=m
>
> #
> # Bluetooth device drivers
> #
> # CONFIG_BT_HCIBTUSB is not set
> # CONFIG_BT_HCIUART is not set
> # CONFIG_BT_HCIBCM203X is not set
> # CONFIG_BT_HCIBPA10X is not set
> # CONFIG_BT_HCIBFUSB is not set
> # CONFIG_BT_HCIDTL1 is not set
> # CONFIG_BT_HCIBT3C is not set
> # CONFIG_BT_HCIBLUECARD is not set
> # CONFIG_BT_HCIBTUART is not set
> # CONFIG_BT_HCIVHCI is not set
> # CONFIG_AF_RXRPC is not set
> # CONFIG_PHONET is not set
> CONFIG_FIB_RULES=y
> CONFIG_WIRELESS=y
> # CONFIG_CFG80211 is not set
> CONFIG_WIRELESS_OLD_REGULATORY=y
> CONFIG_WIRELESS_EXT=y
> CONFIG_WIRELESS_EXT_SYSFS=y
> # CONFIG_LIB80211 is not set
> # CONFIG_MAC80211 is not set
> # CONFIG_WIMAX is not set
> # CONFIG_RFKILL is not set
>
> #
> # Device Drivers
> #
>
> #
> # Generic Driver Options
> #
> CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
> CONFIG_STANDALONE=y
> CONFIG_PREVENT_FIRMWARE_BUILD=y
> CONFIG_FW_LOADER=y
> CONFIG_FIRMWARE_IN_KERNEL=y
> CONFIG_EXTRA_FIRMWARE=""
> # CONFIG_DEBUG_DRIVER is not set
> # CONFIG_DEBUG_DEVRES is not set
> # CONFIG_SYS_HYPERVISOR is not set
> CONFIG_CONNECTOR=y
> CONFIG_PROC_EVENTS=y
> # CONFIG_MTD is not set
> CONFIG_PARPORT=m
> CONFIG_PARPORT_PC=m
> # CONFIG_PARPORT_SERIAL is not set
> # CONFIG_PARPORT_PC_FIFO is not set
> # CONFIG_PARPORT_PC_SUPERIO is not set
> # CONFIG_PARPORT_PC_PCMCIA is not set
> # CONFIG_PARPORT_GSC is not set
> # CONFIG_PARPORT_AX88796 is not set
> CONFIG_PARPORT_1284=y
> CONFIG_PNP=y
> CONFIG_PNP_DEBUG_MESSAGES=y
>
> #
> # Protocols
> #
> CONFIG_PNPACPI=y
> CONFIG_BLK_DEV=y
> # CONFIG_BLK_DEV_FD is not set
> # CONFIG_PARIDE is not set
> # CONFIG_BLK_CPQ_DA is not set
> # CONFIG_BLK_CPQ_CISS_DA is not set
> # CONFIG_BLK_DEV_DAC960 is not set
> # CONFIG_BLK_DEV_UMEM is not set
> # CONFIG_BLK_DEV_COW_COMMON is not set
> # CONFIG_BLK_DEV_LOOP is not set
> # CONFIG_BLK_DEV_NBD is not set
> # CONFIG_BLK_DEV_SX8 is not set
> # CONFIG_BLK_DEV_UB is not set
> CONFIG_BLK_DEV_RAM=y
> CONFIG_BLK_DEV_RAM_COUNT=16
> CONFIG_BLK_DEV_RAM_SIZE=16384
> # CONFIG_BLK_DEV_XIP is not set
> # CONFIG_CDROM_PKTCDVD is not set
> # CONFIG_ATA_OVER_ETH is not set
> # CONFIG_BLK_DEV_HD is not set
> CONFIG_MISC_DEVICES=y
> # CONFIG_IBM_ASM is not set
> # CONFIG_PHANTOM is not set
> # CONFIG_SGI_IOC4 is not set
> # CONFIG_TIFM_CORE is not set
> # CONFIG_ICS932S401 is not set
> # CONFIG_ENCLOSURE_SERVICES is not set
> # CONFIG_HP_ILO is not set
> # CONFIG_C2PORT is not set
>
> #
> # EEPROM support
> #
> # CONFIG_EEPROM_AT24 is not set
> # CONFIG_EEPROM_LEGACY is not set
> # CONFIG_EEPROM_93CX6 is not set
> CONFIG_HAVE_IDE=y
> CONFIG_IDE=y
>
> #
> # Please see Documentation/ide/ide.txt for help/info on IDE drives
> #
> CONFIG_IDE_TIMINGS=y
> CONFIG_IDE_ATAPI=y
> # CONFIG_BLK_DEV_IDE_SATA is not set
> CONFIG_IDE_GD=y
> CONFIG_IDE_GD_ATA=y
> # CONFIG_IDE_GD_ATAPI is not set
> # CONFIG_BLK_DEV_IDECS is not set
> # CONFIG_BLK_DEV_DELKIN is not set
> CONFIG_BLK_DEV_IDECD=m
> CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
> # CONFIG_BLK_DEV_IDETAPE is not set
> # CONFIG_BLK_DEV_IDEACPI is not set
> CONFIG_IDE_TASK_IOCTL=y
> CONFIG_IDE_PROC_FS=y
>
> #
> # IDE chipset support/bugfixes
> #
> CONFIG_IDE_GENERIC=y
> # CONFIG_BLK_DEV_PLATFORM is not set
> # CONFIG_BLK_DEV_CMD640 is not set
> CONFIG_BLK_DEV_IDEPNP=y
> CONFIG_BLK_DEV_IDEDMA_SFF=y
>
> #
> # PCI IDE chipsets support
> #
> CONFIG_BLK_DEV_IDEPCI=y
> CONFIG_IDEPCI_PCIBUS_ORDER=y
> # CONFIG_BLK_DEV_OFFBOARD is not set
> CONFIG_BLK_DEV_GENERIC=y
> # CONFIG_BLK_DEV_OPTI621 is not set
> # CONFIG_BLK_DEV_RZ1000 is not set
> CONFIG_BLK_DEV_IDEDMA_PCI=y
> CONFIG_BLK_DEV_AEC62XX=y
> CONFIG_BLK_DEV_ALI15X3=y
> CONFIG_BLK_DEV_AMD74XX=y
> CONFIG_BLK_DEV_ATIIXP=y
> CONFIG_BLK_DEV_CMD64X=y
> # CONFIG_BLK_DEV_TRIFLEX is not set
> # CONFIG_BLK_DEV_CS5520 is not set
> # CONFIG_BLK_DEV_CS5530 is not set
> CONFIG_BLK_DEV_HPT366=y
> # CONFIG_BLK_DEV_JMICRON is not set
> # CONFIG_BLK_DEV_SC1200 is not set
> CONFIG_BLK_DEV_PIIX=y
> # CONFIG_BLK_DEV_IT8172 is not set
> # CONFIG_BLK_DEV_IT8213 is not set
> CONFIG_BLK_DEV_IT821X=y
> # CONFIG_BLK_DEV_NS87415 is not set
> CONFIG_BLK_DEV_PDC202XX_OLD=y
> CONFIG_BLK_DEV_PDC202XX_NEW=y
> CONFIG_BLK_DEV_SVWKS=y
> CONFIG_BLK_DEV_SIIMAGE=y
> CONFIG_BLK_DEV_SIS5513=y
> # CONFIG_BLK_DEV_SLC90E66 is not set
> # CONFIG_BLK_DEV_TRM290 is not set
> CONFIG_BLK_DEV_VIA82CXXX=y
> # CONFIG_BLK_DEV_TC86C001 is not set
> CONFIG_BLK_DEV_IDEDMA=y
>
> #
> # SCSI device support
> #
> # CONFIG_RAID_ATTRS is not set
> CONFIG_SCSI=m
> CONFIG_SCSI_DMA=y
> # CONFIG_SCSI_TGT is not set
> # CONFIG_SCSI_NETLINK is not set
> CONFIG_SCSI_PROC_FS=y
>
> #
> # SCSI support type (disk, tape, CD-ROM)
> #
> CONFIG_BLK_DEV_SD=m
> # CONFIG_CHR_DEV_ST is not set
> # CONFIG_CHR_DEV_OSST is not set
> CONFIG_BLK_DEV_SR=m
> CONFIG_BLK_DEV_SR_VENDOR=y
> CONFIG_CHR_DEV_SG=m
> # CONFIG_CHR_DEV_SCH is not set
>
> #
> # Some SCSI devices (e.g. CD jukebox) support multiple LUNs
> #
> CONFIG_SCSI_MULTI_LUN=y
> CONFIG_SCSI_CONSTANTS=y
> CONFIG_SCSI_LOGGING=y
> # CONFIG_SCSI_SCAN_ASYNC is not set
> CONFIG_SCSI_WAIT_SCAN=m
>
> #
> # SCSI Transports
> #
> # CONFIG_SCSI_SPI_ATTRS is not set
> # CONFIG_SCSI_FC_ATTRS is not set
> # CONFIG_SCSI_ISCSI_ATTRS is not set
> # CONFIG_SCSI_SAS_LIBSAS is not set
> # CONFIG_SCSI_SRP_ATTRS is not set
> CONFIG_SCSI_LOWLEVEL=y
> # CONFIG_ISCSI_TCP is not set
> # CONFIG_SCSI_CXGB3_ISCSI is not set
> # CONFIG_BLK_DEV_3W_XXXX_RAID is not set
> # CONFIG_SCSI_3W_9XXX is not set
> # CONFIG_SCSI_ACARD is not set
> CONFIG_SCSI_AACRAID=m
> # CONFIG_SCSI_AIC7XXX is not set
> # CONFIG_SCSI_AIC7XXX_OLD is not set
> # CONFIG_SCSI_AIC79XX is not set
> # CONFIG_SCSI_AIC94XX is not set
> # CONFIG_SCSI_DPT_I2O is not set
> # CONFIG_SCSI_ADVANSYS is not set
> # CONFIG_SCSI_ARCMSR is not set
> CONFIG_MEGARAID_NEWGEN=y
> # CONFIG_MEGARAID_MM is not set
> # CONFIG_MEGARAID_LEGACY is not set
> # CONFIG_MEGARAID_SAS is not set
> # CONFIG_SCSI_HPTIOP is not set
> # CONFIG_SCSI_BUSLOGIC is not set
> # CONFIG_LIBFC is not set
> # CONFIG_FCOE is not set
> # CONFIG_SCSI_DMX3191D is not set
> # CONFIG_SCSI_EATA is not set
> # CONFIG_SCSI_FUTURE_DOMAIN is not set
> # CONFIG_SCSI_GDTH is not set
> # CONFIG_SCSI_IPS is not set
> # CONFIG_SCSI_INITIO is not set
> # CONFIG_SCSI_INIA100 is not set
> # CONFIG_SCSI_PPA is not set
> # CONFIG_SCSI_IMM is not set
> # CONFIG_SCSI_MVSAS is not set
> # CONFIG_SCSI_STEX is not set
> # CONFIG_SCSI_SYM53C8XX_2 is not set
> # CONFIG_SCSI_QLOGIC_1280 is not set
> # CONFIG_SCSI_QLA_FC is not set
> # CONFIG_SCSI_QLA_ISCSI is not set
> # CONFIG_SCSI_LPFC is not set
> # CONFIG_SCSI_DC395x is not set
> # CONFIG_SCSI_DC390T is not set
> # CONFIG_SCSI_DEBUG is not set
> # CONFIG_SCSI_SRP is not set
> # CONFIG_SCSI_LOWLEVEL_PCMCIA is not set
> CONFIG_SCSI_DH=m
> # CONFIG_SCSI_DH_RDAC is not set
> # CONFIG_SCSI_DH_HP_SW is not set
> # CONFIG_SCSI_DH_EMC is not set
> # CONFIG_SCSI_DH_ALUA is not set
> # CONFIG_ATA is not set
> CONFIG_MD=y
> CONFIG_BLK_DEV_MD=y
> CONFIG_MD_AUTODETECT=y
> # CONFIG_MD_LINEAR is not set
> # CONFIG_MD_RAID0 is not set
> # CONFIG_MD_RAID1 is not set
> # CONFIG_MD_RAID10 is not set
> # CONFIG_MD_RAID456 is not set
> # CONFIG_MD_MULTIPATH is not set
> # CONFIG_MD_FAULTY is not set
> CONFIG_BLK_DEV_DM=m
> # CONFIG_DM_DEBUG is not set
> # CONFIG_DM_CRYPT is not set
> # CONFIG_DM_SNAPSHOT is not set
> CONFIG_DM_MIRROR=m
> # CONFIG_DM_ZERO is not set
> CONFIG_DM_MULTIPATH=m
> # CONFIG_DM_DELAY is not set
> CONFIG_DM_UEVENT=y
> CONFIG_FUSION=y
> # CONFIG_FUSION_SPI is not set
> # CONFIG_FUSION_FC is not set
> # CONFIG_FUSION_SAS is not set
> CONFIG_FUSION_MAX_SGE=128
> CONFIG_FUSION_LOGGING=y
>
> #
> # IEEE 1394 (FireWire) support
> #
>
> #
> # Enable only one of the two stacks, unless you know what you are doing
> #
> # CONFIG_FIREWIRE is not set
> # CONFIG_IEEE1394 is not set
> # CONFIG_I2O is not set
> # CONFIG_MACINTOSH_DRIVERS is not set
> CONFIG_NETDEVICES=y
> # CONFIG_IFB is not set
> # CONFIG_DUMMY is not set
> # CONFIG_BONDING is not set
> # CONFIG_MACVLAN is not set
> # CONFIG_EQUALIZER is not set
> # CONFIG_TUN is not set
> # CONFIG_VETH is not set
> # CONFIG_NET_SB1000 is not set
> # CONFIG_ARCNET is not set
> CONFIG_PHYLIB=m
>
> #
> # MII PHY device drivers
> #
> # CONFIG_MARVELL_PHY is not set
> # CONFIG_DAVICOM_PHY is not set
> # CONFIG_QSEMI_PHY is not set
> # CONFIG_LXT_PHY is not set
> # CONFIG_CICADA_PHY is not set
> # CONFIG_VITESSE_PHY is not set
> # CONFIG_SMSC_PHY is not set
> # CONFIG_BROADCOM_PHY is not set
> # CONFIG_ICPLUS_PHY is not set
> # CONFIG_REALTEK_PHY is not set
> # CONFIG_NATIONAL_PHY is not set
> # CONFIG_STE10XP is not set
> # CONFIG_LSI_ET1011C_PHY is not set
> # CONFIG_MDIO_BITBANG is not set
> CONFIG_NET_ETHERNET=y
> # CONFIG_MII is not set
> # CONFIG_HAPPYMEAL is not set
> # CONFIG_SUNGEM is not set
> # CONFIG_CASSINI is not set
> CONFIG_NET_VENDOR_3COM=y
> # CONFIG_VORTEX is not set
> # CONFIG_TYPHOON is not set
> # CONFIG_DNET is not set
> CONFIG_NET_TULIP=y
> # CONFIG_DE2104X is not set
> # CONFIG_TULIP is not set
> # CONFIG_DE4X5 is not set
> # CONFIG_WINBOND_840 is not set
> # CONFIG_DM9102 is not set
> # CONFIG_ULI526X is not set
> # CONFIG_PCMCIA_XIRCOM is not set
> # CONFIG_HP100 is not set
> # CONFIG_IBM_NEW_EMAC_ZMII is not set
> # CONFIG_IBM_NEW_EMAC_RGMII is not set
> # CONFIG_IBM_NEW_EMAC_TAH is not set
> # CONFIG_IBM_NEW_EMAC_EMAC4 is not set
> # CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
> # CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
> # CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
> CONFIG_NET_PCI=y
> # CONFIG_PCNET32 is not set
> # CONFIG_AMD8111_ETH is not set
> # CONFIG_ADAPTEC_STARFIRE is not set
> # CONFIG_B44 is not set
> # CONFIG_FORCEDETH is not set
> # CONFIG_E100 is not set
> # CONFIG_FEALNX is not set
> # CONFIG_NATSEMI is not set
> # CONFIG_NE2K_PCI is not set
> # CONFIG_8139CP is not set
> # CONFIG_8139TOO is not set
> # CONFIG_R6040 is not set
> # CONFIG_SIS900 is not set
> # CONFIG_EPIC100 is not set
> # CONFIG_SMSC9420 is not set
> # CONFIG_SUNDANCE is not set
> # CONFIG_TLAN is not set
> # CONFIG_VIA_RHINE is not set
> # CONFIG_SC92031 is not set
> CONFIG_NET_POCKET=y
> # CONFIG_ATP is not set
> # CONFIG_DE600 is not set
> # CONFIG_DE620 is not set
> # CONFIG_ATL2 is not set
> CONFIG_NETDEV_1000=y
> # CONFIG_ACENIC is not set
> # CONFIG_DL2K is not set
> # CONFIG_E1000 is not set
> # CONFIG_E1000E is not set
> # CONFIG_IP1000 is not set
> # CONFIG_IGB is not set
> # CONFIG_NS83820 is not set
> # CONFIG_HAMACHI is not set
> # CONFIG_YELLOWFIN is not set
> # CONFIG_R8169 is not set
> # CONFIG_SIS190 is not set
> # CONFIG_SKGE is not set
> # CONFIG_SKY2 is not set
> # CONFIG_VIA_VELOCITY is not set
> CONFIG_TIGON3=m
> # CONFIG_BNX2 is not set
> # CONFIG_QLA3XXX is not set
> # CONFIG_ATL1 is not set
> # CONFIG_ATL1E is not set
> # CONFIG_ATL1C is not set
> # CONFIG_JME is not set
> CONFIG_NETDEV_10000=y
> # CONFIG_CHELSIO_T1 is not set
> CONFIG_CHELSIO_T3_DEPENDS=y
> # CONFIG_CHELSIO_T3 is not set
> # CONFIG_ENIC is not set
> # CONFIG_IXGBE is not set
> # CONFIG_IXGB is not set
> # CONFIG_S2IO is not set
> # CONFIG_MYRI10GE is not set
> # CONFIG_NIU is not set
> # CONFIG_MLX4_EN is not set
> # CONFIG_MLX4_CORE is not set
> # CONFIG_TEHUTI is not set
> # CONFIG_BNX2X is not set
> # CONFIG_QLGE is not set
> # CONFIG_SFC is not set
> # CONFIG_BE2NET is not set
> CONFIG_TR=y
> # CONFIG_IBMOL is not set
> # CONFIG_3C359 is not set
> # CONFIG_TMS380TR is not set
>
> #
> # Wireless LAN
> #
> # CONFIG_WLAN_PRE80211 is not set
> # CONFIG_WLAN_80211 is not set
> # CONFIG_IWLWIFI_LEDS is not set
>
> #
> # Enable WiMAX (Networking options) to see the WiMAX drivers
> #
>
> #
> # USB Network Adapters
> #
> # CONFIG_USB_CATC is not set
> # CONFIG_USB_KAWETH is not set
> # CONFIG_USB_PEGASUS is not set
> # CONFIG_USB_RTL8150 is not set
> # CONFIG_USB_USBNET is not set
> CONFIG_NET_PCMCIA=y
> # CONFIG_PCMCIA_3C589 is not set
> # CONFIG_PCMCIA_3C574 is not set
> # CONFIG_PCMCIA_FMVJ18X is not set
> # CONFIG_PCMCIA_PCNET is not set
> # CONFIG_PCMCIA_NMCLAN is not set
> # CONFIG_PCMCIA_SMC91C92 is not set
> # CONFIG_PCMCIA_XIRC2PS is not set
> # CONFIG_PCMCIA_AXNET is not set
> # CONFIG_PCMCIA_IBMTR is not set
> # CONFIG_WAN is not set
> CONFIG_FDDI=y
> # CONFIG_DEFXX is not set
> # CONFIG_SKFP is not set
> # CONFIG_HIPPI is not set
> # CONFIG_PLIP is not set
> # CONFIG_PPP is not set
> # CONFIG_SLIP is not set
> CONFIG_NET_FC=y
> # CONFIG_NETCONSOLE is not set
> # CONFIG_NETPOLL is not set
> # CONFIG_NET_POLL_CONTROLLER is not set
> # CONFIG_ISDN is not set
> # CONFIG_PHONE is not set
>
> #
> # Input device support
> #
> CONFIG_INPUT=y
> CONFIG_INPUT_FF_MEMLESS=y
> # CONFIG_INPUT_POLLDEV is not set
>
> #
> # Userland interfaces
> #
> CONFIG_INPUT_MOUSEDEV=y
> # CONFIG_INPUT_MOUSEDEV_PSAUX is not set
> CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
> CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
> # CONFIG_INPUT_JOYDEV is not set
> CONFIG_INPUT_EVDEV=y
> # CONFIG_INPUT_EVBUG is not set
>
> #
> # Input Device Drivers
> #
> CONFIG_INPUT_KEYBOARD=y
> CONFIG_KEYBOARD_ATKBD=y
> # CONFIG_KEYBOARD_SUNKBD is not set
> # CONFIG_KEYBOARD_LKKBD is not set
> # CONFIG_KEYBOARD_XTKBD is not set
> # CONFIG_KEYBOARD_NEWTON is not set
> # CONFIG_KEYBOARD_STOWAWAY is not set
> CONFIG_INPUT_MOUSE=y
> CONFIG_MOUSE_PS2=y
> CONFIG_MOUSE_PS2_ALPS=y
> CONFIG_MOUSE_PS2_LOGIPS2PP=y
> CONFIG_MOUSE_PS2_SYNAPTICS=y
> CONFIG_MOUSE_PS2_LIFEBOOK=y
> CONFIG_MOUSE_PS2_TRACKPOINT=y
> # CONFIG_MOUSE_PS2_ELANTECH is not set
> # CONFIG_MOUSE_PS2_TOUCHKIT is not set
> # CONFIG_MOUSE_SERIAL is not set
> # CONFIG_MOUSE_APPLETOUCH is not set
> # CONFIG_MOUSE_BCM5974 is not set
> # CONFIG_MOUSE_VSXXXAA is not set
> CONFIG_INPUT_JOYSTICK=y
> # CONFIG_JOYSTICK_ANALOG is not set
> # CONFIG_JOYSTICK_A3D is not set
> # CONFIG_JOYSTICK_ADI is not set
> # CONFIG_JOYSTICK_COBRA is not set
> # CONFIG_JOYSTICK_GF2K is not set
> # CONFIG_JOYSTICK_GRIP is not set
> # CONFIG_JOYSTICK_GRIP_MP is not set
> # CONFIG_JOYSTICK_GUILLEMOT is not set
> # CONFIG_JOYSTICK_INTERACT is not set
> # CONFIG_JOYSTICK_SIDEWINDER is not set
> # CONFIG_JOYSTICK_TMDC is not set
> # CONFIG_JOYSTICK_IFORCE is not set
> # CONFIG_JOYSTICK_WARRIOR is not set
> # CONFIG_JOYSTICK_MAGELLAN is not set
> # CONFIG_JOYSTICK_SPACEORB is not set
> # CONFIG_JOYSTICK_SPACEBALL is not set
> # CONFIG_JOYSTICK_STINGER is not set
> # CONFIG_JOYSTICK_TWIDJOY is not set
> # CONFIG_JOYSTICK_ZHENHUA is not set
> # CONFIG_JOYSTICK_DB9 is not set
> # CONFIG_JOYSTICK_GAMECON is not set
> # CONFIG_JOYSTICK_TURBOGRAFX is not set
> # CONFIG_JOYSTICK_JOYDUMP is not set
> # CONFIG_JOYSTICK_XPAD is not set
> # CONFIG_INPUT_TABLET is not set
> CONFIG_INPUT_TOUCHSCREEN=y
> # CONFIG_TOUCHSCREEN_FUJITSU is not set
> # CONFIG_TOUCHSCREEN_GUNZE is not set
> # CONFIG_TOUCHSCREEN_ELO is not set
> # CONFIG_TOUCHSCREEN_WACOM_W8001 is not set
> # CONFIG_TOUCHSCREEN_MTOUCH is not set
> # CONFIG_TOUCHSCREEN_INEXIO is not set
> # CONFIG_TOUCHSCREEN_MK712 is not set
> # CONFIG_TOUCHSCREEN_PENMOUNT is not set
> # CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set
> # CONFIG_TOUCHSCREEN_TOUCHWIN is not set
> # CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set
> # CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
> # CONFIG_TOUCHSCREEN_TSC2007 is not set
> CONFIG_INPUT_MISC=y
> CONFIG_INPUT_PCSPKR=m
> # CONFIG_INPUT_APANEL is not set
> # CONFIG_INPUT_ATLAS_BTNS is not set
> # CONFIG_INPUT_ATI_REMOTE is not set
> # CONFIG_INPUT_ATI_REMOTE2 is not set
> # CONFIG_INPUT_KEYSPAN_REMOTE is not set
> # CONFIG_INPUT_POWERMATE is not set
> # CONFIG_INPUT_YEALINK is not set
> # CONFIG_INPUT_CM109 is not set
> # CONFIG_INPUT_UINPUT is not set
>
> #
> # Hardware I/O ports
> #
> CONFIG_SERIO=y
> CONFIG_SERIO_I8042=y
> CONFIG_SERIO_SERPORT=y
> # CONFIG_SERIO_CT82C710 is not set
> # CONFIG_SERIO_PARKBD is not set
> # CONFIG_SERIO_PCIPS2 is not set
> CONFIG_SERIO_LIBPS2=y
> CONFIG_SERIO_RAW=m
> # CONFIG_GAMEPORT is not set
>
> #
> # Character devices
> #
> CONFIG_VT=y
> CONFIG_CONSOLE_TRANSLATIONS=y
> CONFIG_VT_CONSOLE=y
> CONFIG_HW_CONSOLE=y
> CONFIG_VT_HW_CONSOLE_BINDING=y
> CONFIG_DEVKMEM=y
> CONFIG_SERIAL_NONSTANDARD=y
> # CONFIG_COMPUTONE is not set
> # CONFIG_ROCKETPORT is not set
> # CONFIG_CYCLADES is not set
> # CONFIG_DIGIEPCA is not set
> # CONFIG_MOXA_INTELLIO is not set
> # CONFIG_MOXA_SMARTIO is not set
> # CONFIG_ISI is not set
> # CONFIG_SYNCLINK is not set
> # CONFIG_SYNCLINKMP is not set
> # CONFIG_SYNCLINK_GT is not set
> # CONFIG_N_HDLC is not set
> # CONFIG_RISCOM8 is not set
> # CONFIG_SPECIALIX is not set
> # CONFIG_SX is not set
> # CONFIG_RIO is not set
> # CONFIG_STALDRV is not set
> # CONFIG_NOZOMI is not set
>
> #
> # Serial drivers
> #
> CONFIG_SERIAL_8250=y
> CONFIG_SERIAL_8250_CONSOLE=y
> CONFIG_FIX_EARLYCON_MEM=y
> CONFIG_SERIAL_8250_PCI=y
> CONFIG_SERIAL_8250_PNP=y
> # CONFIG_SERIAL_8250_CS is not set
> CONFIG_SERIAL_8250_NR_UARTS=32
> CONFIG_SERIAL_8250_RUNTIME_UARTS=4
> CONFIG_SERIAL_8250_EXTENDED=y
> CONFIG_SERIAL_8250_MANY_PORTS=y
> CONFIG_SERIAL_8250_SHARE_IRQ=y
> CONFIG_SERIAL_8250_DETECT_IRQ=y
> CONFIG_SERIAL_8250_RSA=y
>
> #
> # Non-8250 serial port support
> #
> CONFIG_SERIAL_CORE=y
> CONFIG_SERIAL_CORE_CONSOLE=y
> # CONFIG_SERIAL_JSM is not set
> CONFIG_UNIX98_PTYS=y
> # CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
> # CONFIG_LEGACY_PTYS is not set
> CONFIG_PRINTER=m
> CONFIG_LP_CONSOLE=y
> # CONFIG_PPDEV is not set
> # CONFIG_IPMI_HANDLER is not set
> CONFIG_HW_RANDOM=y
> # CONFIG_HW_RANDOM_INTEL is not set
> # CONFIG_HW_RANDOM_AMD is not set
> CONFIG_NVRAM=y
> CONFIG_RTC=y
> # CONFIG_R3964 is not set
> # CONFIG_APPLICOM is not set
>
> #
> # PCMCIA character devices
> #
> # CONFIG_SYNCLINK_CS is not set
> # CONFIG_CARDMAN_4000 is not set
> # CONFIG_CARDMAN_4040 is not set
> # CONFIG_IPWIRELESS is not set
> # CONFIG_MWAVE is not set
> # CONFIG_PC8736x_GPIO is not set
> CONFIG_RAW_DRIVER=y
> CONFIG_MAX_RAW_DEVS=8192
> CONFIG_HPET=y
> # CONFIG_HPET_MMAP is not set
> # CONFIG_HANGCHECK_TIMER is not set
> # CONFIG_TCG_TPM is not set
> # CONFIG_TELCLOCK is not set
> CONFIG_DEVPORT=y
> CONFIG_I2C=m
> CONFIG_I2C_BOARDINFO=y
> # CONFIG_I2C_CHARDEV is not set
> CONFIG_I2C_HELPER_AUTO=y
>
> #
> # I2C Hardware Bus support
> #
>
> #
> # PC SMBus host controller drivers
> #
> # CONFIG_I2C_ALI1535 is not set
> # CONFIG_I2C_ALI1563 is not set
> # CONFIG_I2C_ALI15X3 is not set
> # CONFIG_I2C_AMD756 is not set
> # CONFIG_I2C_AMD8111 is not set
> # CONFIG_I2C_I801 is not set
> # CONFIG_I2C_ISCH is not set
> CONFIG_I2C_PIIX4=m
> # CONFIG_I2C_NFORCE2 is not set
> # CONFIG_I2C_SIS5595 is not set
> # CONFIG_I2C_SIS630 is not set
> # CONFIG_I2C_SIS96X is not set
> # CONFIG_I2C_VIA is not set
> # CONFIG_I2C_VIAPRO is not set
>
> #
> # I2C system bus drivers (mostly embedded / system-on-chip)
> #
> # CONFIG_I2C_OCORES is not set
> # CONFIG_I2C_SIMTEC is not set
>
> #
> # External I2C/SMBus adapter drivers
> #
> # CONFIG_I2C_PARPORT is not set
> # CONFIG_I2C_PARPORT_LIGHT is not set
> # CONFIG_I2C_TAOS_EVM is not set
> # CONFIG_I2C_TINY_USB is not set
>
> #
> # Graphics adapter I2C/DDC channel drivers
> #
> # CONFIG_I2C_VOODOO3 is not set
>
> #
> # Other I2C/SMBus bus drivers
> #
> # CONFIG_I2C_PCA_PLATFORM is not set
> # CONFIG_I2C_STUB is not set
>
> #
> # Miscellaneous I2C Chip support
> #
> # CONFIG_DS1682 is not set
> # CONFIG_SENSORS_PCF8574 is not set
> # CONFIG_PCF8575 is not set
> # CONFIG_SENSORS_PCA9539 is not set
> # CONFIG_SENSORS_PCF8591 is not set
> # CONFIG_SENSORS_MAX6875 is not set
> # CONFIG_SENSORS_TSL2550 is not set
> # CONFIG_I2C_DEBUG_CORE is not set
> # CONFIG_I2C_DEBUG_ALGO is not set
> # CONFIG_I2C_DEBUG_BUS is not set
> # CONFIG_I2C_DEBUG_CHIP is not set
> # CONFIG_SPI is not set
> CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
> # CONFIG_GPIOLIB is not set
> # CONFIG_W1 is not set
> CONFIG_POWER_SUPPLY=y
> # CONFIG_POWER_SUPPLY_DEBUG is not set
> # CONFIG_PDA_POWER is not set
> # CONFIG_BATTERY_DS2760 is not set
> # CONFIG_BATTERY_BQ27x00 is not set
> # CONFIG_HWMON is not set
> CONFIG_THERMAL=y
> CONFIG_WATCHDOG=y
> # CONFIG_WATCHDOG_NOWAYOUT is not set
>
> #
> # Watchdog Device Drivers
> #
> # CONFIG_SOFT_WATCHDOG is not set
> # CONFIG_ACQUIRE_WDT is not set
> # CONFIG_ADVANTECH_WDT is not set
> # CONFIG_ALIM1535_WDT is not set
> # CONFIG_ALIM7101_WDT is not set
> # CONFIG_SC520_WDT is not set
> # CONFIG_EUROTECH_WDT is not set
> # CONFIG_IB700_WDT is not set
> # CONFIG_IBMASR is not set
> # CONFIG_WAFER_WDT is not set
> # CONFIG_I6300ESB_WDT is not set
> # CONFIG_ITCO_WDT is not set
> # CONFIG_IT8712F_WDT is not set
> # CONFIG_IT87_WDT is not set
> # CONFIG_HP_WATCHDOG is not set
> # CONFIG_SC1200_WDT is not set
> # CONFIG_PC87413_WDT is not set
> # CONFIG_60XX_WDT is not set
> # CONFIG_SBC8360_WDT is not set
> # CONFIG_CPU5_WDT is not set
> # CONFIG_SMSC_SCH311X_WDT is not set
> # CONFIG_SMSC37B787_WDT is not set
> # CONFIG_W83627HF_WDT is not set
> # CONFIG_W83697HF_WDT is not set
> # CONFIG_W83697UG_WDT is not set
> # CONFIG_W83877F_WDT is not set
> # CONFIG_W83977F_WDT is not set
> # CONFIG_MACHZ_WDT is not set
> # CONFIG_SBC_EPX_C3_WATCHDOG is not set
>
> #
> # PCI-based Watchdog Cards
> #
> # CONFIG_PCIPCWATCHDOG is not set
> # CONFIG_WDTPCI is not set
>
> #
> # USB-based Watchdog Cards
> #
> # CONFIG_USBPCWATCHDOG is not set
> CONFIG_SSB_POSSIBLE=y
>
> #
> # Sonics Silicon Backplane
> #
> # CONFIG_SSB is not set
>
> #
> # Multifunction device drivers
> #
> # CONFIG_MFD_CORE is not set
> # CONFIG_MFD_SM501 is not set
> # CONFIG_HTC_PASIC3 is not set
> # CONFIG_MFD_TMIO is not set
> # CONFIG_MFD_WM8400 is not set
> # CONFIG_MFD_PCF50633 is not set
> # CONFIG_REGULATOR is not set
>
> #
> # Multimedia devices
> #
>
> #
> # Multimedia core support
> #
> # CONFIG_VIDEO_DEV is not set
> # CONFIG_DVB_CORE is not set
> # CONFIG_VIDEO_MEDIA is not set
>
> #
> # Multimedia drivers
> #
> # CONFIG_DAB is not set
>
> #
> # Graphics support
> #
> CONFIG_AGP=y
> CONFIG_AGP_AMD64=y
> CONFIG_AGP_INTEL=y
> CONFIG_AGP_SIS=y
> CONFIG_AGP_VIA=y
> # CONFIG_DRM is not set
> # CONFIG_VGASTATE is not set
> # CONFIG_VIDEO_OUTPUT_CONTROL is not set
> CONFIG_FB=y
> # CONFIG_FIRMWARE_EDID is not set
> # CONFIG_FB_DDC is not set
> CONFIG_FB_BOOT_VESA_SUPPORT=y
> CONFIG_FB_CFB_FILLRECT=y
> CONFIG_FB_CFB_COPYAREA=y
> CONFIG_FB_CFB_IMAGEBLIT=y
> # CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
> # CONFIG_FB_SYS_FILLRECT is not set
> # CONFIG_FB_SYS_COPYAREA is not set
> # CONFIG_FB_SYS_IMAGEBLIT is not set
> # CONFIG_FB_FOREIGN_ENDIAN is not set
> # CONFIG_FB_SYS_FOPS is not set
> # CONFIG_FB_SVGALIB is not set
> # CONFIG_FB_MACMODES is not set
> # CONFIG_FB_BACKLIGHT is not set
> CONFIG_FB_MODE_HELPERS=y
> CONFIG_FB_TILEBLITTING=y
>
> #
> # Frame buffer hardware drivers
> #
> # CONFIG_FB_CIRRUS is not set
> # CONFIG_FB_PM2 is not set
> # CONFIG_FB_CYBER2000 is not set
> # CONFIG_FB_ARC is not set
> # CONFIG_FB_ASILIANT is not set
> # CONFIG_FB_IMSTT is not set
> # CONFIG_FB_VGA16 is not set
> # CONFIG_FB_UVESA is not set
> CONFIG_FB_VESA=y
> # CONFIG_FB_N411 is not set
> # CONFIG_FB_HGA is not set
> # CONFIG_FB_S1D13XXX is not set
> # CONFIG_FB_NVIDIA is not set
> # CONFIG_FB_RIVA is not set
> # CONFIG_FB_LE80578 is not set
> # CONFIG_FB_INTEL is not set
> # CONFIG_FB_MATROX is not set
> # CONFIG_FB_RADEON is not set
> # CONFIG_FB_ATY128 is not set
> # CONFIG_FB_ATY is not set
> # CONFIG_FB_S3 is not set
> # CONFIG_FB_SAVAGE is not set
> # CONFIG_FB_SIS is not set
> # CONFIG_FB_VIA is not set
> # CONFIG_FB_NEOMAGIC is not set
> # CONFIG_FB_KYRO is not set
> # CONFIG_FB_3DFX is not set
> # CONFIG_FB_VOODOO1 is not set
> # CONFIG_FB_VT8623 is not set
> # CONFIG_FB_TRIDENT is not set
> # CONFIG_FB_ARK is not set
> # CONFIG_FB_PM3 is not set
> # CONFIG_FB_CARMINE is not set
> # CONFIG_FB_GEODE is not set
> # CONFIG_FB_VIRTUAL is not set
> # CONFIG_FB_METRONOME is not set
> # CONFIG_FB_MB862XX is not set
> CONFIG_BACKLIGHT_LCD_SUPPORT=y
> # CONFIG_LCD_CLASS_DEVICE is not set
> # CONFIG_BACKLIGHT_CLASS_DEVICE is not set
>
> #
> # Display device support
> #
> # CONFIG_DISPLAY_SUPPORT is not set
>
> #
> # Console display driver support
> #
> CONFIG_VGA_CONSOLE=y
> CONFIG_VGACON_SOFT_SCROLLBACK=y
> CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
> CONFIG_DUMMY_CONSOLE=y
> CONFIG_FRAMEBUFFER_CONSOLE=y
> # CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
> CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
> # CONFIG_FONTS is not set
> CONFIG_FONT_8x8=y
> CONFIG_FONT_8x16=y
> CONFIG_LOGO=y
> # CONFIG_LOGO_LINUX_MONO is not set
> # CONFIG_LOGO_LINUX_VGA16 is not set
> CONFIG_LOGO_LINUX_CLUT224=y
> # CONFIG_SOUND is not set
> CONFIG_HID_SUPPORT=y
> CONFIG_HID=y
> CONFIG_HID_DEBUG=y
> # CONFIG_HIDRAW is not set
>
> #
> # USB Input Devices
> #
> CONFIG_USB_HID=y
> CONFIG_HID_PID=y
> CONFIG_USB_HIDDEV=y
>
> #
> # Special HID drivers
> #
> CONFIG_HID_COMPAT=y
> CONFIG_HID_A4TECH=y
> CONFIG_HID_APPLE=y
> CONFIG_HID_BELKIN=y
> CONFIG_HID_CHERRY=y
> CONFIG_HID_CHICONY=y
> CONFIG_HID_CYPRESS=y
> CONFIG_HID_EZKEY=y
> CONFIG_HID_GYRATION=y
> CONFIG_HID_LOGITECH=y
> CONFIG_LOGITECH_FF=y
> # CONFIG_LOGIRUMBLEPAD2_FF is not set
> CONFIG_HID_MICROSOFT=y
> CONFIG_HID_MONTEREY=y
> CONFIG_HID_NTRIG=y
> CONFIG_HID_PANTHERLORD=y
> # CONFIG_PANTHERLORD_FF is not set
> CONFIG_HID_PETALYNX=y
> CONFIG_HID_SAMSUNG=y
> CONFIG_HID_SONY=y
> CONFIG_HID_SUNPLUS=y
> # CONFIG_GREENASIA_FF is not set
> CONFIG_HID_TOPSEED=y
> CONFIG_THRUSTMASTER_FF=y
> # CONFIG_ZEROPLUS_FF is not set
> CONFIG_USB_SUPPORT=y
> CONFIG_USB_ARCH_HAS_HCD=y
> CONFIG_USB_ARCH_HAS_OHCI=y
> CONFIG_USB_ARCH_HAS_EHCI=y
> CONFIG_USB=y
> # CONFIG_USB_DEBUG is not set
> # CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set
>
> #
> # Miscellaneous USB options
> #
> CONFIG_USB_DEVICEFS=y
> CONFIG_USB_DEVICE_CLASS=y
> # CONFIG_USB_DYNAMIC_MINORS is not set
> # CONFIG_USB_SUSPEND is not set
> # CONFIG_USB_OTG is not set
> CONFIG_USB_MON=y
> # CONFIG_USB_WUSB is not set
> # CONFIG_USB_WUSB_CBAF is not set
>
> #
> # USB Host Controller Drivers
> #
> # CONFIG_USB_C67X00_HCD is not set
> CONFIG_USB_EHCI_HCD=m
> CONFIG_USB_EHCI_ROOT_HUB_TT=y
> CONFIG_USB_EHCI_TT_NEWSCHED=y
> # CONFIG_USB_OXU210HP_HCD is not set
> # CONFIG_USB_ISP116X_HCD is not set
> # CONFIG_USB_ISP1760_HCD is not set
> CONFIG_USB_OHCI_HCD=m
> # CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
> # CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
> CONFIG_USB_OHCI_LITTLE_ENDIAN=y
> CONFIG_USB_UHCI_HCD=m
> # CONFIG_USB_SL811_HCD is not set
> # CONFIG_USB_R8A66597_HCD is not set
> # CONFIG_USB_HWA_HCD is not set
>
> #
> # USB Device Class drivers
> #
> # CONFIG_USB_ACM is not set
> # CONFIG_USB_PRINTER is not set
> # CONFIG_USB_WDM is not set
> # CONFIG_USB_TMC is not set
>
> #
> # NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may also be needed;
> #
>
> #
> # see USB_STORAGE Help for more information
> #
> CONFIG_USB_STORAGE=m
> # CONFIG_USB_STORAGE_DEBUG is not set
> CONFIG_USB_STORAGE_DATAFAB=y
> CONFIG_USB_STORAGE_FREECOM=y
> CONFIG_USB_STORAGE_ISD200=y
> CONFIG_USB_STORAGE_USBAT=y
> CONFIG_USB_STORAGE_SDDR09=y
> CONFIG_USB_STORAGE_SDDR55=y
> CONFIG_USB_STORAGE_JUMPSHOT=y
> CONFIG_USB_STORAGE_ALAUDA=y
> # CONFIG_USB_STORAGE_ONETOUCH is not set
> # CONFIG_USB_STORAGE_KARMA is not set
> # CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
> # CONFIG_USB_LIBUSUAL is not set
>
> #
> # USB Imaging devices
> #
> # CONFIG_USB_MDC800 is not set
> # CONFIG_USB_MICROTEK is not set
>
> #
> # USB port drivers
> #
> # CONFIG_USB_USS720 is not set
> # CONFIG_USB_SERIAL is not set
>
> #
> # USB Miscellaneous drivers
> #
> # CONFIG_USB_EMI62 is not set
> # CONFIG_USB_EMI26 is not set
> # CONFIG_USB_ADUTUX is not set
> # CONFIG_USB_SEVSEG is not set
> # CONFIG_USB_RIO500 is not set
> # CONFIG_USB_LEGOTOWER is not set
> # CONFIG_USB_LCD is not set
> # CONFIG_USB_BERRY_CHARGE is not set
> # CONFIG_USB_LED is not set
> # CONFIG_USB_CYPRESS_CY7C63 is not set
> # CONFIG_USB_CYTHERM is not set
> # CONFIG_USB_PHIDGET is not set
> # CONFIG_USB_IDMOUSE is not set
> # CONFIG_USB_FTDI_ELAN is not set
> # CONFIG_USB_APPLEDISPLAY is not set
> # CONFIG_USB_SISUSBVGA is not set
> # CONFIG_USB_LD is not set
> # CONFIG_USB_TRANCEVIBRATOR is not set
> # CONFIG_USB_IOWARRIOR is not set
> # CONFIG_USB_TEST is not set
> # CONFIG_USB_ISIGHTFW is not set
> # CONFIG_USB_VST is not set
>
> #
> # OTG and related infrastructure
> #
> # CONFIG_UWB is not set
> # CONFIG_MMC is not set
> # CONFIG_MEMSTICK is not set
> CONFIG_NEW_LEDS=y
> CONFIG_LEDS_CLASS=y
>
> #
> # LED drivers
> #
> # CONFIG_LEDS_ALIX2 is not set
> # CONFIG_LEDS_PCA9532 is not set
> # CONFIG_LEDS_CLEVO_MAIL is not set
> # CONFIG_LEDS_PCA955X is not set
>
> #
> # LED Triggers
> #
> CONFIG_LEDS_TRIGGERS=y
> # CONFIG_LEDS_TRIGGER_TIMER is not set
> CONFIG_LEDS_TRIGGER_IDE_DISK=y
> # CONFIG_LEDS_TRIGGER_HEARTBEAT is not set
> # CONFIG_LEDS_TRIGGER_BACKLIGHT is not set
> # CONFIG_LEDS_TRIGGER_DEFAULT_ON is not set
> # CONFIG_ACCESSIBILITY is not set
> CONFIG_EDAC=y
>
> #
> # Reporting subsystems
> #
> # CONFIG_EDAC_DEBUG is not set
> # CONFIG_EDAC_MM_EDAC is not set
> # CONFIG_RTC_CLASS is not set
> # CONFIG_DMADEVICES is not set
> # CONFIG_AUXDISPLAY is not set
> # CONFIG_UIO is not set
> # CONFIG_STAGING is not set
> CONFIG_X86_PLATFORM_DEVICES=y
> # CONFIG_ASUS_LAPTOP is not set
> # CONFIG_SONY_LAPTOP is not set
> # CONFIG_THINKPAD_ACPI is not set
> # CONFIG_INTEL_MENLOW is not set
> # CONFIG_EEEPC_LAPTOP is not set
> # CONFIG_ACPI_WMI is not set
> # CONFIG_ACPI_ASUS is not set
> # CONFIG_ACPI_TOSHIBA is not set
>
> #
> # Firmware Drivers
> #
> # CONFIG_EDD is not set
> CONFIG_FIRMWARE_MEMMAP=y
> # CONFIG_DELL_RBU is not set
> # CONFIG_DCDBAS is not set
> CONFIG_DMIID=y
> CONFIG_ISCSI_IBFT_FIND=y
> # CONFIG_ISCSI_IBFT is not set
>
> #
> # File systems
> #
> CONFIG_EXT2_FS=y
> CONFIG_EXT2_FS_XATTR=y
> CONFIG_EXT2_FS_POSIX_ACL=y
> CONFIG_EXT2_FS_SECURITY=y
> CONFIG_EXT2_FS_XIP=y
> CONFIG_EXT3_FS=m
> CONFIG_EXT3_FS_XATTR=y
> CONFIG_EXT3_FS_POSIX_ACL=y
> CONFIG_EXT3_FS_SECURITY=y
> # CONFIG_EXT4_FS is not set
> CONFIG_FS_XIP=y
> CONFIG_JBD=m
> # CONFIG_JBD_DEBUG is not set
> CONFIG_FS_MBCACHE=y
> # CONFIG_REISERFS_FS is not set
> # CONFIG_JFS_FS is not set
> CONFIG_FS_POSIX_ACL=y
> CONFIG_FILE_LOCKING=y
> # CONFIG_XFS_FS is not set
> # CONFIG_GFS2_FS is not set
> # CONFIG_OCFS2_FS is not set
> # CONFIG_BTRFS_FS is not set
> CONFIG_DNOTIFY=y
> CONFIG_INOTIFY=y
> CONFIG_INOTIFY_USER=y
> CONFIG_QUOTA=y
> # CONFIG_QUOTA_NETLINK_INTERFACE is not set
> CONFIG_PRINT_QUOTA_WARNING=y
> CONFIG_QUOTA_TREE=y
> # CONFIG_QFMT_V1 is not set
> CONFIG_QFMT_V2=y
> CONFIG_QUOTACTL=y
> # CONFIG_AUTOFS_FS is not set
> CONFIG_AUTOFS4_FS=m
> # CONFIG_FUSE_FS is not set
>
> #
> # CD-ROM/DVD Filesystems
> #
> CONFIG_ISO9660_FS=y
> CONFIG_JOLIET=y
> CONFIG_ZISOFS=y
> # CONFIG_UDF_FS is not set
>
> #
> # DOS/FAT/NT Filesystems
> #
> # CONFIG_MSDOS_FS is not set
> # CONFIG_VFAT_FS is not set
> # CONFIG_NTFS_FS is not set
>
> #
> # Pseudo filesystems
> #
> CONFIG_PROC_FS=y
> CONFIG_PROC_KCORE=y
> CONFIG_PROC_VMCORE=y
> CONFIG_PROC_SYSCTL=y
> CONFIG_PROC_PAGE_MONITOR=y
> CONFIG_SYSFS=y
> CONFIG_TMPFS=y
> # CONFIG_TMPFS_POSIX_ACL is not set
> CONFIG_HUGETLBFS=y
> CONFIG_HUGETLB_PAGE=y
> # CONFIG_CONFIGFS_FS is not set
> CONFIG_MISC_FILESYSTEMS=y
> # CONFIG_ADFS_FS is not set
> # CONFIG_AFFS_FS is not set
> # CONFIG_ECRYPT_FS is not set
> # CONFIG_HFS_FS is not set
> # CONFIG_HFSPLUS_FS is not set
> # CONFIG_BEFS_FS is not set
> # CONFIG_BFS_FS is not set
> # CONFIG_EFS_FS is not set
> # CONFIG_CRAMFS is not set
> # CONFIG_SQUASHFS is not set
> # CONFIG_VXFS_FS is not set
> # CONFIG_MINIX_FS is not set
> # CONFIG_OMFS_FS is not set
> # CONFIG_HPFS_FS is not set
> # CONFIG_QNX4FS_FS is not set
> # CONFIG_ROMFS_FS is not set
> # CONFIG_SYSV_FS is not set
> # CONFIG_UFS_FS is not set
> CONFIG_NETWORK_FILESYSTEMS=y
> # CONFIG_NFS_FS is not set
> # CONFIG_NFSD is not set
> # CONFIG_SMB_FS is not set
> # CONFIG_CIFS is not set
> # CONFIG_NCP_FS is not set
> # CONFIG_CODA_FS is not set
> # CONFIG_AFS_FS is not set
>
> #
> # Partition Types
> #
> CONFIG_PARTITION_ADVANCED=y
> # CONFIG_ACORN_PARTITION is not set
> CONFIG_OSF_PARTITION=y
> CONFIG_AMIGA_PARTITION=y
> # CONFIG_ATARI_PARTITION is not set
> CONFIG_MAC_PARTITION=y
> CONFIG_MSDOS_PARTITION=y
> CONFIG_BSD_DISKLABEL=y
> CONFIG_MINIX_SUBPARTITION=y
> CONFIG_SOLARIS_X86_PARTITION=y
> CONFIG_UNIXWARE_DISKLABEL=y
> # CONFIG_LDM_PARTITION is not set
> CONFIG_SGI_PARTITION=y
> # CONFIG_ULTRIX_PARTITION is not set
> CONFIG_SUN_PARTITION=y
> CONFIG_KARMA_PARTITION=y
> CONFIG_EFI_PARTITION=y
> # CONFIG_SYSV68_PARTITION is not set
> CONFIG_NLS=y
> CONFIG_NLS_DEFAULT="utf8"
> CONFIG_NLS_CODEPAGE_437=y
> # CONFIG_NLS_CODEPAGE_737 is not set
> # CONFIG_NLS_CODEPAGE_775 is not set
> # CONFIG_NLS_CODEPAGE_850 is not set
> # CONFIG_NLS_CODEPAGE_852 is not set
> # CONFIG_NLS_CODEPAGE_855 is not set
> # CONFIG_NLS_CODEPAGE_857 is not set
> # CONFIG_NLS_CODEPAGE_860 is not set
> # CONFIG_NLS_CODEPAGE_861 is not set
> # CONFIG_NLS_CODEPAGE_862 is not set
> # CONFIG_NLS_CODEPAGE_863 is not set
> # CONFIG_NLS_CODEPAGE_864 is not set
> # CONFIG_NLS_CODEPAGE_865 is not set
> # CONFIG_NLS_CODEPAGE_866 is not set
> # CONFIG_NLS_CODEPAGE_869 is not set
> # CONFIG_NLS_CODEPAGE_936 is not set
> # CONFIG_NLS_CODEPAGE_950 is not set
> # CONFIG_NLS_CODEPAGE_932 is not set
> # CONFIG_NLS_CODEPAGE_949 is not set
> # CONFIG_NLS_CODEPAGE_874 is not set
> # CONFIG_NLS_ISO8859_8 is not set
> # CONFIG_NLS_CODEPAGE_1250 is not set
> # CONFIG_NLS_CODEPAGE_1251 is not set
> CONFIG_NLS_ASCII=y
> # CONFIG_NLS_ISO8859_1 is not set
> # CONFIG_NLS_ISO8859_2 is not set
> # CONFIG_NLS_ISO8859_3 is not set
> # CONFIG_NLS_ISO8859_4 is not set
> # CONFIG_NLS_ISO8859_5 is not set
> # CONFIG_NLS_ISO8859_6 is not set
> # CONFIG_NLS_ISO8859_7 is not set
> # CONFIG_NLS_ISO8859_9 is not set
> # CONFIG_NLS_ISO8859_13 is not set
> # CONFIG_NLS_ISO8859_14 is not set
> # CONFIG_NLS_ISO8859_15 is not set
> # CONFIG_NLS_KOI8_R is not set
> # CONFIG_NLS_KOI8_U is not set
> # CONFIG_NLS_UTF8 is not set
> # CONFIG_DLM is not set
>
> #
> # Kernel hacking
> #
> CONFIG_TRACE_IRQFLAGS_SUPPORT=y
> # CONFIG_PRINTK_TIME is not set
> CONFIG_ALLOW_WARNINGS=y
> CONFIG_ENABLE_WARN_DEPRECATED=y
> CONFIG_ENABLE_MUST_CHECK=y
> CONFIG_FRAME_WARN=2048
> CONFIG_MAGIC_SYSRQ=y
> # CONFIG_UNUSED_SYMBOLS is not set
> CONFIG_DEBUG_FS=y
> # CONFIG_HEADERS_CHECK is not set
> # CONFIG_DEBUG_SECTION_MISMATCH is not set
> CONFIG_DEBUG_KERNEL=y
> # CONFIG_DEBUG_SHIRQ is not set
> CONFIG_DETECT_SOFTLOCKUP=y
> # CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
> CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
> CONFIG_DETECT_HUNG_TASK=y
> # CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
> CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
> CONFIG_SCHED_DEBUG=y
> CONFIG_SCHEDSTATS=y
> # CONFIG_TIMER_STATS is not set
> # CONFIG_DEBUG_OBJECTS is not set
> # CONFIG_DEBUG_SLAB is not set
> # CONFIG_DEBUG_RT_MUTEXES is not set
> # CONFIG_RT_MUTEX_TESTER is not set
> # CONFIG_DEBUG_SPINLOCK is not set
> # CONFIG_DEBUG_MUTEXES is not set
> # CONFIG_DEBUG_LOCK_ALLOC is not set
> # CONFIG_PROVE_LOCKING is not set
> # CONFIG_LOCK_STAT is not set
> CONFIG_TRACE_IRQFLAGS=y
> # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
> # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
> CONFIG_STACKTRACE=y
> # CONFIG_DEBUG_KOBJECT is not set
> CONFIG_DEBUG_BUGVERBOSE=y
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_VM is not set
> # CONFIG_DEBUG_VIRTUAL is not set
> # CONFIG_DEBUG_WRITECOUNT is not set
> CONFIG_DEBUG_MEMORY_INIT=y
> CONFIG_DEBUG_LIST=y
> # CONFIG_DEBUG_SG is not set
> # CONFIG_DEBUG_NOTIFIERS is not set
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_FRAME_POINTER=y
> # CONFIG_BOOT_PRINTK_DELAY is not set
> # CONFIG_RCU_TORTURE_TEST is not set
> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> # CONFIG_KPROBES_SANITY_TEST is not set
> # CONFIG_BACKTRACE_SELF_TEST is not set
> # CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
> # CONFIG_LKDTM is not set
> # CONFIG_FAULT_INJECTION is not set
> # CONFIG_LATENCYTOP is not set
> # CONFIG_SYSCTL_SYSCALL_CHECK is not set
> CONFIG_USER_STACKTRACE_SUPPORT=y
> CONFIG_NOP_TRACER=y
> CONFIG_HAVE_FTRACE_NMI_ENTER=y
> CONFIG_HAVE_FUNCTION_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
> CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
> CONFIG_HAVE_DYNAMIC_FTRACE=y
> CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
> CONFIG_HAVE_HW_BRANCH_TRACER=y
> CONFIG_HAVE_FTRACE_SYSCALLS=y
> CONFIG_TRACER_MAX_TRACE=y
> CONFIG_RING_BUFFER=y
> CONFIG_FTRACE_NMI_ENTER=y
> CONFIG_TRACING=y
> CONFIG_TRACING_SUPPORT=y
>
> #
> # Tracers
> #
> CONFIG_FUNCTION_TRACER=y
> CONFIG_FUNCTION_GRAPH_TRACER=y
> CONFIG_IRQSOFF_TRACER=y
> CONFIG_SYSPROF_TRACER=y
> CONFIG_SCHED_TRACER=y
> CONFIG_CONTEXT_SWITCH_TRACER=y
> CONFIG_EVENT_TRACER=y
> CONFIG_FTRACE_SYSCALLS=y
> CONFIG_BOOT_TRACER=y
> CONFIG_TRACE_BRANCH_PROFILING=y
> CONFIG_PROFILE_ALL_BRANCHES=y
> CONFIG_TRACING_BRANCHES=y
> CONFIG_BRANCH_TRACER=y
> CONFIG_POWER_TRACER=y
> CONFIG_STACK_TRACER=y
> CONFIG_HW_BRANCH_TRACER=y
> CONFIG_KMEMTRACE=y
> CONFIG_WORKQUEUE_TRACER=y
> CONFIG_BLK_DEV_IO_TRACE=y
> CONFIG_DYNAMIC_FTRACE=y
> CONFIG_FUNCTION_PROFILER=y
> CONFIG_FTRACE_MCOUNT_RECORD=y
> CONFIG_FTRACE_SELFTEST=y
> CONFIG_FTRACE_STARTUP_TEST=y
> CONFIG_MMIOTRACE=y
> # CONFIG_MMIOTRACE_TEST is not set
> # CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
> # CONFIG_DYNAMIC_PRINTK_DEBUG is not set
> # CONFIG_DMA_API_DEBUG is not set
> CONFIG_SAMPLES=y
> # CONFIG_SAMPLE_MARKERS is not set
> # CONFIG_SAMPLE_TRACEPOINTS is not set
> # CONFIG_SAMPLE_KOBJECT is not set
> # CONFIG_SAMPLE_KPROBES is not set
> CONFIG_HAVE_ARCH_KGDB=y
> # CONFIG_KGDB is not set
> CONFIG_HAVE_ARCH_KMEMCHECK=y
> # CONFIG_STRICT_DEVMEM is not set
> CONFIG_X86_VERBOSE_BOOTUP=y
> CONFIG_EARLY_PRINTK=y
> # CONFIG_EARLY_PRINTK_DBGP is not set
> CONFIG_DEBUG_STACKOVERFLOW=y
> # CONFIG_DEBUG_STACK_USAGE is not set
> # CONFIG_DEBUG_PAGEALLOC is not set
> # CONFIG_DEBUG_PER_CPU_MAPS is not set
> # CONFIG_X86_PTDUMP is not set
> CONFIG_DEBUG_RODATA=y
> # CONFIG_DEBUG_RODATA_TEST is not set
> # CONFIG_DEBUG_NX_TEST is not set
> # CONFIG_IOMMU_DEBUG is not set
> CONFIG_HAVE_MMIOTRACE_SUPPORT=y
> CONFIG_IO_DELAY_TYPE_0X80=0
> CONFIG_IO_DELAY_TYPE_0XED=1
> CONFIG_IO_DELAY_TYPE_UDELAY=2
> CONFIG_IO_DELAY_TYPE_NONE=3
> CONFIG_IO_DELAY_0X80=y
> # CONFIG_IO_DELAY_0XED is not set
> # CONFIG_IO_DELAY_UDELAY is not set
> # CONFIG_IO_DELAY_NONE is not set
> CONFIG_DEFAULT_IO_DELAY_TYPE=0
> # CONFIG_DEBUG_BOOT_PARAMS is not set
> # CONFIG_CPA_DEBUG is not set
> # CONFIG_OPTIMIZE_INLINING is not set
>
> #
> # Security options
> #
> CONFIG_KEYS=y
> CONFIG_KEYS_DEBUG_PROC_KEYS=y
> CONFIG_SECURITY=y
> # CONFIG_SECURITYFS is not set
> CONFIG_SECURITY_NETWORK=y
> CONFIG_SECURITY_NETWORK_XFRM=y
> # CONFIG_SECURITY_PATH is not set
> # CONFIG_SECURITY_FILE_CAPABILITIES is not set
> # CONFIG_SECURITY_ROOTPLUG is not set
> CONFIG_SECURITY_DEFAULT_MMAP_MIN_ADDR=0
> CONFIG_SECURITY_SELINUX=y
> CONFIG_SECURITY_SELINUX_BOOTPARAM=y
> CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
> CONFIG_SECURITY_SELINUX_DISABLE=y
> CONFIG_SECURITY_SELINUX_DEVELOP=y
> CONFIG_SECURITY_SELINUX_AVC_STATS=y
> CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
> # CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
> # CONFIG_SECURITY_SMACK is not set
> CONFIG_CRYPTO=y
>
> #
> # Crypto core or helper
> #
> CONFIG_CRYPTO_FIPS=y
> CONFIG_CRYPTO_ALGAPI=y
> CONFIG_CRYPTO_ALGAPI2=y
> CONFIG_CRYPTO_AEAD2=y
> CONFIG_CRYPTO_BLKCIPHER2=y
> CONFIG_CRYPTO_HASH=y
> CONFIG_CRYPTO_HASH2=y
> CONFIG_CRYPTO_RNG2=y
> CONFIG_CRYPTO_MANAGER=y
> CONFIG_CRYPTO_MANAGER2=y
> # CONFIG_CRYPTO_GF128MUL is not set
> # CONFIG_CRYPTO_NULL is not set
> # CONFIG_CRYPTO_CRYPTD is not set
> # CONFIG_CRYPTO_AUTHENC is not set
> # CONFIG_CRYPTO_TEST is not set
>
> #
> # Authenticated Encryption with Associated Data
> #
> # CONFIG_CRYPTO_CCM is not set
> # CONFIG_CRYPTO_GCM is not set
> # CONFIG_CRYPTO_SEQIV is not set
>
> #
> # Block modes
> #
> # CONFIG_CRYPTO_CBC is not set
> # CONFIG_CRYPTO_CTR is not set
> # CONFIG_CRYPTO_CTS is not set
> # CONFIG_CRYPTO_ECB is not set
> # CONFIG_CRYPTO_LRW is not set
> # CONFIG_CRYPTO_PCBC is not set
> # CONFIG_CRYPTO_XTS is not set
>
> #
> # Hash modes
> #
> CONFIG_CRYPTO_HMAC=y
> # CONFIG_CRYPTO_XCBC is not set
>
> #
> # Digest
> #
> CONFIG_CRYPTO_CRC32C=y
> # CONFIG_CRYPTO_CRC32C_INTEL is not set
> # CONFIG_CRYPTO_MD4 is not set
> # CONFIG_CRYPTO_MD5 is not set
> # CONFIG_CRYPTO_MICHAEL_MIC is not set
> # CONFIG_CRYPTO_RMD128 is not set
> # CONFIG_CRYPTO_RMD160 is not set
> # CONFIG_CRYPTO_RMD256 is not set
> # CONFIG_CRYPTO_RMD320 is not set
> CONFIG_CRYPTO_SHA1=y
> # CONFIG_CRYPTO_SHA256 is not set
> # CONFIG_CRYPTO_SHA512 is not set
> # CONFIG_CRYPTO_TGR192 is not set
> # CONFIG_CRYPTO_WP512 is not set
>
> #
> # Ciphers
> #
> # CONFIG_CRYPTO_AES is not set
> # CONFIG_CRYPTO_AES_X86_64 is not set
> # CONFIG_CRYPTO_ANUBIS is not set
> # CONFIG_CRYPTO_ARC4 is not set
> # CONFIG_CRYPTO_BLOWFISH is not set
> # CONFIG_CRYPTO_CAMELLIA is not set
> # CONFIG_CRYPTO_CAST5 is not set
> # CONFIG_CRYPTO_CAST6 is not set
> # CONFIG_CRYPTO_DES is not set
> # CONFIG_CRYPTO_FCRYPT is not set
> # CONFIG_CRYPTO_KHAZAD is not set
> # CONFIG_CRYPTO_SALSA20 is not set
> # CONFIG_CRYPTO_SALSA20_X86_64 is not set
> # CONFIG_CRYPTO_SEED is not set
> # CONFIG_CRYPTO_SERPENT is not set
> # CONFIG_CRYPTO_TEA is not set
> # CONFIG_CRYPTO_TWOFISH is not set
> # CONFIG_CRYPTO_TWOFISH_X86_64 is not set
>
> #
> # Compression
> #
> # CONFIG_CRYPTO_DEFLATE is not set
> # CONFIG_CRYPTO_LZO is not set
>
> #
> # Random Number Generation
> #
> # CONFIG_CRYPTO_ANSI_CPRNG is not set
> CONFIG_CRYPTO_HW=y
> # CONFIG_CRYPTO_DEV_HIFN_795X is not set
> CONFIG_HAVE_KVM=y
> CONFIG_VIRTUALIZATION=y
> # CONFIG_KVM is not set
> # CONFIG_VIRTIO_PCI is not set
> # CONFIG_VIRTIO_BALLOON is not set
> CONFIG_BINARY_PRINTF=y
>
> #
> # Library routines
> #
> CONFIG_BITREVERSE=y
> CONFIG_GENERIC_FIND_FIRST_BIT=y
> CONFIG_GENERIC_FIND_NEXT_BIT=y
> CONFIG_GENERIC_FIND_LAST_BIT=y
> # CONFIG_CRC_CCITT is not set
> # CONFIG_CRC16 is not set
> # CONFIG_CRC_T10DIF is not set
> # CONFIG_CRC_ITU_T is not set
> CONFIG_CRC32=y
> # CONFIG_CRC7 is not set
> CONFIG_LIBCRC32C=y
> CONFIG_ZLIB_INFLATE=y
> CONFIG_DECOMPRESS_GZIP=y
> CONFIG_HAS_IOMEM=y
> CONFIG_HAS_IOPORT=y
> CONFIG_HAS_DMA=y
On Thu, 26 Mar 2009, Frederic Weisbecker wrote:
> On Thu, Mar 26, 2009 at 05:58:07PM +0530, Maneesh Soni wrote:
> >
> > I hit the below oops while playing with "wakeup" tracer. Just enabled
> > tracing and ran "while true; do cat trace; done" on a 8-way systems with
> > -tip at commit 9e25bacc221318a29d37425e054c01047edad7f5.
> >
> >
> > Attached the .config.
> >
> > Thanks
> > Maneesh
> >
> > BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> > IP: [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
> > PGD 219147067 PUD 1e449e067 PMD 0
> > Oops: 0000 [#1] SMP
> > last sysfs file: /sys/devices/pci0000:0c/0000:0c:00.0/irq
> > CPU 5
> > Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
> > Pid: 5937, comm: cat Not tainted 2.6.29-tip-test #1 eserver xSeries 366-[88632RA]-
> > RIP: 0010:[<ffffffff802d2258>] [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
> > RSP: 0000:ffff8801cf4cdea0 EFLAGS: 00010046
> > RAX: 0000000000000000 RBX: ffff88022f232820 RCX: 0000000000000046
> > RDX: ffff8800281989a0 RSI: ffffffff8020d56d RDI: ffffffff811de480
> > RBP: ffff8801cf4cded0 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000046
> > R13: 0000000000000000 R14: ffff8801e4471820 R15: 0000000000000005
> > FS: 00007f607411c6e0(0000) GS:ffff88002818b000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 0000000000000008 CR3: 00000001da19a000 CR4: 00000000000006e0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process cat (pid: 5937, threadinfo ffff8801cf4cc000, task ffff8801e4471820)
> > Stack:
> > ffff8800281989a0 ffff8802191e14b0 ffff88002819dd40 00007fff7c130640
> > 0000000000000000 ffff88002819dd40 ffff8801cf4cdf70 ffffffff806f8264
> > 0000000000000002 ffff8801cf4cdf48 ffff8801e4471820 ffff88022f232820
> > Call Trace:
> > [<ffffffff806f8264>] schedule+0xd99/0x10f9
> > [<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> > [<ffffffff8020d56d>] retint_careful+0x12/0x2e
> > Code: f2 83 3d 1b b0 9f 00 00 74 77 48 3b 1d a2 de f2 00 75 6e 48 8b 4d 00 48 8b 3d 8d de f2 00 45 89 e8 48 8b 01 48 8b 71 08 4c 89 e1 <48> 8b 50 08 e8 f7 9f ff ff 48 8b 3d 70 de f2 00 45 89 e8 48 89
> > RIP [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
> > RSP <ffff8801cf4cdea0>
> > CR2: 0000000000000008
> > ---[ end trace ab11daa90bb147ba ]---
> >
>
>
>
> I've tested the same config and ran your script with wakeup tracer but nothing
> weird happened.
> Though I've tested on UP so may be I should try on SMP.
Yeah, that may make a difference ;-)
-- Steve
On Thu, 26 Mar 2009, Steven Rostedt wrote:
>
> On Thu, 26 Mar 2009, Frederic Weisbecker wrote:
>
> > On Thu, Mar 26, 2009 at 05:58:07PM +0530, Maneesh Soni wrote:
> > >
> > > I hit the below oops while playing with "wakeup" tracer. Just enabled
> > > tracing and ran "while true; do cat trace; done" on a 8-way systems with
> > > -tip at commit 9e25bacc221318a29d37425e054c01047edad7f5.
> > >
> > >
> > > Attached the .config.
> > >
> > > Thanks
> > > Maneesh
> > >
> > > BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> > > IP: [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
> > > PGD 219147067 PUD 1e449e067 PMD 0
> > > Oops: 0000 [#1] SMP
> > > last sysfs file: /sys/devices/pci0000:0c/0000:0c:00.0/irq
> > > CPU 5
> > > Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
> > > Pid: 5937, comm: cat Not tainted 2.6.29-tip-test #1 eserver xSeries 366-[88632RA]-
> > > RIP: 0010:[<ffffffff802d2258>] [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
> > > RSP: 0000:ffff8801cf4cdea0 EFLAGS: 00010046
> > > RAX: 0000000000000000 RBX: ffff88022f232820 RCX: 0000000000000046
> > > RDX: ffff8800281989a0 RSI: ffffffff8020d56d RDI: ffffffff811de480
> > > RBP: ffff8801cf4cded0 R08: 0000000000000000 R09: 0000000000000000
> > > R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000046
> > > R13: 0000000000000000 R14: ffff8801e4471820 R15: 0000000000000005
> > > FS: 00007f607411c6e0(0000) GS:ffff88002818b000(0000) knlGS:0000000000000000
> > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > CR2: 0000000000000008 CR3: 00000001da19a000 CR4: 00000000000006e0
> > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > Process cat (pid: 5937, threadinfo ffff8801cf4cc000, task ffff8801e4471820)
> > > Stack:
> > > ffff8800281989a0 ffff8802191e14b0 ffff88002819dd40 00007fff7c130640
> > > 0000000000000000 ffff88002819dd40 ffff8801cf4cdf70 ffffffff806f8264
> > > 0000000000000002 ffff8801cf4cdf48 ffff8801e4471820 ffff88022f232820
> > > Call Trace:
> > > [<ffffffff806f8264>] schedule+0xd99/0x10f9
> > > [<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> > > [<ffffffff8020d56d>] retint_careful+0x12/0x2e
> > > Code: f2 83 3d 1b b0 9f 00 00 74 77 48 3b 1d a2 de f2 00 75 6e 48 8b 4d 00 48 8b 3d 8d de f2 00 45 89 e8 48 8b 01 48 8b 71 08 4c 89 e1 <48> 8b 50 08 e8 f7 9f ff ff 48 8b 3d 70 de f2 00 45 89 e8 48 89
> > > RIP [<ffffffff802d2258>] probe_wakeup_sched_switch+0xd1/0x1ae
> > > RSP <ffff8801cf4cdea0>
> > > CR2: 0000000000000008
> > > ---[ end trace ab11daa90bb147ba ]---
> > >
> >
> >
> >
> > I've tested the same config and ran your script with wakeup tracer but nothing
> > weird happened.
> > Though I've tested on UP so may be I should try on SMP.
>
> Yeah, that may make a difference ;-)
Looking at the code, I would say it is definitely SMP related. The issue
is that he's reading the trace file over and over again. Reading the trace
tries to stop the current tracer while it reads. When it stops reading, it
restarts the tracer. This does a reset of the buffers. There is the wakeup
lock that is suppose to protect it.
But I've changed this code slightly to allow the stopping to be done in
atomic operations. Thus I no longer remove the trace points on a stop.
This now changes the way things work since I first wrote the locking code.
So I need to go back and review it to see if I've just opened up a race
window.
Thanks for testing and reporting this!
-- Steve
Ingo,
I believe this is the fix for the oops that Maneesh saw. What he explains
he did sounds like it would trigger the race. Reading the trace output
resets wakeup_task to NULL and wakeup_cpu to -1. And he hit the bug by
just reading the trace in a while loop.
Please pull the latest tip/tracing/ftrace-1 tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/ftrace-1
Steven Rostedt (1):
tracing/wakeup: move access to wakeup_cpu into spinlock
----
kernel/trace/trace_sched_wakeup.c | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
---------------------------
commit 8ac070ca952d14c57e3d772e203fd117161cc596
Author: Steven Rostedt <[email protected]>
Date: Thu Mar 26 10:25:24 2009 -0400
tracing/wakeup: move access to wakeup_cpu into spinlock
Impact: fix for race condition
The code had the following outside the lock:
if (next != wakeup_task)
return;
pc = preempt_count();
/* The task we are waiting for is waking up */
data = wakeup_trace->data[wakeup_cpu];
On initialization, wakeup_task is NULL and wakeup_cpu -1. This code
is not under a lock. If wakeup_task is set on another CPU as that
task is waking up, we can see the wakeup_task before wakeup_cpu is
set. If we read wakeup_cpu while it is still -1 then we will have
a bad data pointer.
This patch moves the reading of wakeup_cpu within the protection of
the spinlock used to protect the writing of wakeup_cpu and wakeup_task.
Reported-by: Maneesh Soni <[email protected]>
Signed-off-by: Steven Rostedt <[email protected]>
diff --git a/kernel/trace/trace_sched_wakeup.c b/kernel/trace/trace_sched_wakeup.c
index 3c5ad6b..9e4ce4c 100644
--- a/kernel/trace/trace_sched_wakeup.c
+++ b/kernel/trace/trace_sched_wakeup.c
@@ -138,9 +138,6 @@ probe_wakeup_sched_switch(struct rq *rq, struct task_struct *prev,
pc = preempt_count();
- /* The task we are waiting for is waking up */
- data = wakeup_trace->data[wakeup_cpu];
-
/* disable local data, not wakeup_cpu data */
cpu = raw_smp_processor_id();
disabled = atomic_inc_return(&wakeup_trace->data[cpu]->disabled);
@@ -154,6 +151,9 @@ probe_wakeup_sched_switch(struct rq *rq, struct task_struct *prev,
if (unlikely(!tracer_enabled || next != wakeup_task))
goto out_unlock;
+ /* The task we are waiting for is waking up */
+ data = wakeup_trace->data[wakeup_cpu];
+
trace_function(wakeup_trace, CALLER_ADDR1, CALLER_ADDR2, flags, pc);
tracing_sched_switch_trace(wakeup_trace, prev, next, flags, pc);
On Thu, Mar 26, 2009 at 10:40:47AM -0400, Steven Rostedt wrote:
>
> Ingo,
>
> I believe this is the fix for the oops that Maneesh saw. What he explains
> he did sounds like it would trigger the race. Reading the trace output
> resets wakeup_task to NULL and wakeup_cpu to -1. And he hit the bug by
> just reading the trace in a while loop.
>
> Please pull the latest tip/tracing/ftrace-1 tree, which can be found at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
> tip/tracing/ftrace-1
>
>
> Steven Rostedt (1):
> tracing/wakeup: move access to wakeup_cpu into spinlock
>
> ----
> kernel/trace/trace_sched_wakeup.c | 6 +++---
> 1 files changed, 3 insertions(+), 3 deletions(-)
> ---------------------------
> commit 8ac070ca952d14c57e3d772e203fd117161cc596
> Author: Steven Rostedt <[email protected]>
> Date: Thu Mar 26 10:25:24 2009 -0400
>
> tracing/wakeup: move access to wakeup_cpu into spinlock
>
> Impact: fix for race condition
>
> The code had the following outside the lock:
>
> if (next != wakeup_task)
> return;
>
> pc = preempt_count();
>
> /* The task we are waiting for is waking up */
> data = wakeup_trace->data[wakeup_cpu];
>
> On initialization, wakeup_task is NULL and wakeup_cpu -1. This code
> is not under a lock. If wakeup_task is set on another CPU as that
> task is waking up, we can see the wakeup_task before wakeup_cpu is
> set. If we read wakeup_cpu while it is still -1 then we will have
> a bad data pointer.
>
> This patch moves the reading of wakeup_cpu within the protection of
> the spinlock used to protect the writing of wakeup_cpu and wakeup_task.
>
> Reported-by: Maneesh Soni <[email protected]>
> Signed-off-by: Steven Rostedt <[email protected]>
>
> diff --git a/kernel/trace/trace_sched_wakeup.c b/kernel/trace/trace_sched_wakeup.c
> index 3c5ad6b..9e4ce4c 100644
> --- a/kernel/trace/trace_sched_wakeup.c
> +++ b/kernel/trace/trace_sched_wakeup.c
> @@ -138,9 +138,6 @@ probe_wakeup_sched_switch(struct rq *rq, struct task_struct *prev,
>
> pc = preempt_count();
>
> - /* The task we are waiting for is waking up */
> - data = wakeup_trace->data[wakeup_cpu];
> -
> /* disable local data, not wakeup_cpu data */
> cpu = raw_smp_processor_id();
> disabled = atomic_inc_return(&wakeup_trace->data[cpu]->disabled);
> @@ -154,6 +151,9 @@ probe_wakeup_sched_switch(struct rq *rq, struct task_struct *prev,
> if (unlikely(!tracer_enabled || next != wakeup_task))
> goto out_unlock;
>
> + /* The task we are waiting for is waking up */
> + data = wakeup_trace->data[wakeup_cpu];
> +
> trace_function(wakeup_trace, CALLER_ADDR1, CALLER_ADDR2, flags, pc);
> tracing_sched_switch_trace(wakeup_trace, prev, next, flags, pc);
>
>
Hi Steven,
I still the oops with the patch,
BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
PGD 1d68e9067 PUD 1e6cdf067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:0c/0000:0c:00.0/irq
CPU 5
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
Pid: 6095, comm: cat Not tainted 2.6.29-tip-test #2 eserver xSeries 366-[88632RA]-
RIP: 0010:[<ffffffff802d2259>] [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
RSP: 0000:ffff8801e6d03ea0 EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff8800281989a0 RCX: 0000000000000046
RDX: ffff8800281989a0 RSI: ffffffff8020d56d RDI: ffffffff811de480
RBP: ffff8801e6d03ed0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000046
R13: ffff88022f232820 R14: 0000000000000000 R15: 0000000000000005
FS: 00007f98b94cb6e0(0000) GS:ffff88002818b000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000224595000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process cat (pid: 6095, threadinfo ffff8801e6d02000, task ffff8801d6935040)
Stack:
ffff8801d6935040 ffff880224981230 ffff88002819dd40 0000003b4ae00aa0
00007fffc14df830 ffff88002819dd40 ffff8801e6d03f70 ffffffff806f8264
0000000000000002 ffff8801e6d03f48 ffff8801d6935040 ffff88022f232820
Call Trace:
[<ffffffff806f8264>] schedule+0xd99/0x10f9
[<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
[<ffffffff8027d2cb>] ? delayed_work_timer_fn+0x0/0x38
[<ffffffff8020d56d>] retint_careful+0x12/0x2e
Code: 2d ad de f2 00 75 77 48 63 05 ac de f2 00 48 8b 4d 00 45 89 f0 48 8b 3d 8e de f2 00 48 8b 71 08 48 8b 5c c7 28 48 8b 01 4c 89 e1 <48> 8b 50 08 e8 f6 9f ff ff 48 8b 3d 6f de f2 00 48 8b 75 d0 45
RIP [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
RSP <ffff8801e6d03ea0>
CR2: 0000000000000008
thanks
Maneesh
--
Maneesh Soni
Linux Technology Center
IBM India Systems and Technology Lab,
Bangalore, India.
On Fri, Mar 27, 2009 at 02:11:44PM +0530, Maneesh Soni wrote:
> On Thu, Mar 26, 2009 at 10:40:47AM -0400, Steven Rostedt wrote:
> >
> > Ingo,
> >
> > I believe this is the fix for the oops that Maneesh saw. What he explains
> > he did sounds like it would trigger the race. Reading the trace output
> > resets wakeup_task to NULL and wakeup_cpu to -1. And he hit the bug by
> > just reading the trace in a while loop.
> >
> > Please pull the latest tip/tracing/ftrace-1 tree, which can be found at:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
> > tip/tracing/ftrace-1
> >
> >
> > Steven Rostedt (1):
> > tracing/wakeup: move access to wakeup_cpu into spinlock
> >
> > ----
> > kernel/trace/trace_sched_wakeup.c | 6 +++---
> > 1 files changed, 3 insertions(+), 3 deletions(-)
> > ---------------------------
> > commit 8ac070ca952d14c57e3d772e203fd117161cc596
> > Author: Steven Rostedt <[email protected]>
> > Date: Thu Mar 26 10:25:24 2009 -0400
> >
> > tracing/wakeup: move access to wakeup_cpu into spinlock
> >
> > Impact: fix for race condition
> >
> > The code had the following outside the lock:
> >
> > if (next != wakeup_task)
> > return;
> >
> > pc = preempt_count();
> >
> > /* The task we are waiting for is waking up */
> > data = wakeup_trace->data[wakeup_cpu];
> >
> > On initialization, wakeup_task is NULL and wakeup_cpu -1. This code
> > is not under a lock. If wakeup_task is set on another CPU as that
> > task is waking up, we can see the wakeup_task before wakeup_cpu is
> > set. If we read wakeup_cpu while it is still -1 then we will have
> > a bad data pointer.
> >
> > This patch moves the reading of wakeup_cpu within the protection of
> > the spinlock used to protect the writing of wakeup_cpu and wakeup_task.
> >
> > Reported-by: Maneesh Soni <[email protected]>
> > Signed-off-by: Steven Rostedt <[email protected]>
> >
> > diff --git a/kernel/trace/trace_sched_wakeup.c b/kernel/trace/trace_sched_wakeup.c
> > index 3c5ad6b..9e4ce4c 100644
> > --- a/kernel/trace/trace_sched_wakeup.c
> > +++ b/kernel/trace/trace_sched_wakeup.c
> > @@ -138,9 +138,6 @@ probe_wakeup_sched_switch(struct rq *rq, struct task_struct *prev,
> >
> > pc = preempt_count();
> >
> > - /* The task we are waiting for is waking up */
> > - data = wakeup_trace->data[wakeup_cpu];
> > -
> > /* disable local data, not wakeup_cpu data */
> > cpu = raw_smp_processor_id();
> > disabled = atomic_inc_return(&wakeup_trace->data[cpu]->disabled);
> > @@ -154,6 +151,9 @@ probe_wakeup_sched_switch(struct rq *rq, struct task_struct *prev,
> > if (unlikely(!tracer_enabled || next != wakeup_task))
> > goto out_unlock;
> >
> > + /* The task we are waiting for is waking up */
> > + data = wakeup_trace->data[wakeup_cpu];
> > +
> > trace_function(wakeup_trace, CALLER_ADDR1, CALLER_ADDR2, flags, pc);
> > tracing_sched_switch_trace(wakeup_trace, prev, next, flags, pc);
> >
> >
>
> Hi Steven,
>
> I still the oops with the patch,
>
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> IP: [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> PGD 1d68e9067 PUD 1e6cdf067 PMD 0
> Oops: 0000 [#1] SMP
> last sysfs file: /sys/devices/pci0000:0c/0000:0c:00.0/irq
> CPU 5
> Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
> Pid: 6095, comm: cat Not tainted 2.6.29-tip-test #2 eserver xSeries 366-[88632RA]-
> RIP: 0010:[<ffffffff802d2259>] [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> RSP: 0000:ffff8801e6d03ea0 EFLAGS: 00010046
> RAX: 0000000000000000 RBX: ffff8800281989a0 RCX: 0000000000000046
> RDX: ffff8800281989a0 RSI: ffffffff8020d56d RDI: ffffffff811de480
> RBP: ffff8801e6d03ed0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000046
> R13: ffff88022f232820 R14: 0000000000000000 R15: 0000000000000005
> FS: 00007f98b94cb6e0(0000) GS:ffff88002818b000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000008 CR3: 0000000224595000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process cat (pid: 6095, threadinfo ffff8801e6d02000, task ffff8801d6935040)
> Stack:
> ffff8801d6935040 ffff880224981230 ffff88002819dd40 0000003b4ae00aa0
> 00007fffc14df830 ffff88002819dd40 ffff8801e6d03f70 ffffffff806f8264
> 0000000000000002 ffff8801e6d03f48 ffff8801d6935040 ffff88022f232820
> Call Trace:
> [<ffffffff806f8264>] schedule+0xd99/0x10f9
> [<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> [<ffffffff8027d2cb>] ? delayed_work_timer_fn+0x0/0x38
> [<ffffffff8020d56d>] retint_careful+0x12/0x2e
> Code: 2d ad de f2 00 75 77 48 63 05 ac de f2 00 48 8b 4d 00 45 89 f0 48 8b 3d 8e de f2 00 48 8b 71 08 48 8b 5c c7 28 48 8b 01 4c 89 e1 <48> 8b 50 08 e8 f6 9f ff ff 48 8b 3d 6f de f2 00 48 8b 75 d0 45
> RIP [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> RSP <ffff8801e6d03ea0>
> CR2: 0000000000000008
>
I could get a kdump in my next attempt. Let me know if you want me to upload the dump
somewhere. (700 MB compressed)
Some info from the dump using gdb & crash-utility.
(gdb) bt full
#0 0xffffffff802d2259 in probe_wakeup_sched_switch (rq=<value optimized out>,
prev=Cannot access memory at address 0xffff8801d05c7ea0
) at kernel/trace/trace_sched_wakeup.c:157
data = (struct trace_array_cpu *) 0xffff8800281629a0
T0 = <value optimized out>
T1 = <value optimized out>
delta = <value optimized out>
flags = 70
pc = 0
Cannot access memory at address 0xffff8801d05c7ed8
(gdb) print wakeup_trace
$7 = (struct trace_array *) 0xffffffff811de480
(gdb) print *(struct trace_array *) 0xffffffff811de480
$8 = {buffer = 0xffff88022f090440, entries = 1441792, cpu = 0,
time_start = 437360893965, waiter = 0x0, data = {0xffff8800281119a0,
0xffff88002812c9a0, 0xffff8800281479a0, 0xffff8800281629a0,
0xffff88002817d9a0, 0xffff8800281989a0, 0xffff8800281b39a0,
0xffff8800281ce9a0, 0x0 <repeats 247 times>}}
(gdb) print tracer_enabled
$9 = 1
(gdb) print wakeup_cpu
$10 = 3
(gdb) info threads
8 process 0 0xffffffff80217024 in mwait_idle ()
at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
7 process 0 0xffffffff80217024 in mwait_idle ()
at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
6 process 0 0xffffffff80217024 in mwait_idle ()
at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
5 process 0 0xffffffff80217024 in mwait_idle ()
at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
* 4 process 5773 0xffffffff802d2259 in probe_wakeup_sched_switch (
rq=<value optimized out>, prev=Cannot access memory at address 0xffff8801d05c7ea0
) at kernel/trace/trace_sched_wakeup.c:157
3 process 0 0xffffffff80217024 in mwait_idle ()
at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
2 process 0 0xffffffff80217024 in mwait_idle ()
at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
1 process 0 0xffffffff80217024 in mwait_idle ()
at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
(gdb) info registers
rax 0x0 0
rbx 0xffff8800281629a0 -131940722792032
rcx 0x46 70
rdx 0xffff8800281629a0 -131940722792032
rsi 0xffffffff8020d56d -2145331859
rdi 0xffffffff811de480 -2128747392
rbp 0xffff8801d05c7ed0 0xffff8801d05c7ed0
rsp 0xffff8801d05c7ea0 0xffff8801d05c7ea0
r8 0x0 0
r9 0x0 0
r10 0xffff8801d05c7eb0 -131933604643152
r11 0x1 1
r12 0x46 70
r13 0xffff88022f21d820 -131932014651360
r14 0x0 0
r15 0x3 3
rip 0xffffffff802d2259 0xffffffff802d2259 <probe_wakeup_sched_switch+210>
eflags 0x10046 [ PF ZF RF ]
cs 0x10 16
ss 0x0 0
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
fctrl 0x0 0
fstat 0x0 0
ftag 0x0 0
fiseg 0x0 0
fioff 0x0 0
foseg 0x0 0
fooff 0x0 0
fop 0x0 0
mxcsr 0x0 [ ]
(gdb)
...
...
0xffffffff802d2208 <probe_wakeup_sched_switch+129>: lock xadd %ax,0xf2debf(%rip) # 0xffffffff812000d0 <wakeup_lock>
0xffffffff802d2211 <probe_wakeup_sched_switch+138>: cmp %ah,%al
0xffffffff802d2213 <probe_wakeup_sched_switch+140>: je 0xffffffff802d221f <probe_wakeup_sched_switch+152>
0xffffffff802d2215 <probe_wakeup_sched_switch+142>: pause
0xffffffff802d2217 <probe_wakeup_sched_switch+144>: mov 0xf2deb3(%rip),%al # 0xffffffff812000d0 <wakeup_lock>
0xffffffff802d221d <probe_wakeup_sched_switch+150>: jmp 0xffffffff802d2211 <probe_wakeup_sched_switch+138>
0xffffffff802d221f <probe_wakeup_sched_switch+152>: cmpl $0x0,0x9fb02a(%rip) # 0xffffffff80ccd250 <tracer_enabled>
0xffffffff802d2226 <probe_wakeup_sched_switch+159>: je 0xffffffff802d22ac <probe_wakeup_sched_switch+293>
0xffffffff802d222c <probe_wakeup_sched_switch+165>: cmp 0xf2dead(%rip),%r13 # 0xffffffff812000e0 <wakeup_task>
0xffffffff802d2233 <probe_wakeup_sched_switch+172>: jne 0xffffffff802d22ac <probe_wakeup_sched_switch+293>
0xffffffff802d2235 <probe_wakeup_sched_switch+174>: movslq 0xf2deac(%rip),%rax # 0xffffffff812000e8 <wakeup_cpu>
0xffffffff802d223c <probe_wakeup_sched_switch+181>: mov 0x0(%rbp),%rcx
0xffffffff802d2240 <probe_wakeup_sched_switch+185>: mov %r14d,%r8d
0xffffffff802d2243 <probe_wakeup_sched_switch+188>: mov 0xf2de8e(%rip),%rdi # 0xffffffff812000d8 <wakeup_trace>
0xffffffff802d224a <probe_wakeup_sched_switch+195>: mov 0x8(%rcx),%rsi
0xffffffff802d224e <probe_wakeup_sched_switch+199>: mov 0x28(%rdi,%rax,8),%rbx
0xffffffff802d2253 <probe_wakeup_sched_switch+204>: mov (%rcx),%rax
0xffffffff802d2256 <probe_wakeup_sched_switch+207>: mov %r12,%rcx
0xffffffff802d2259 <probe_wakeup_sched_switch+210>: mov 0x8(%rax),%rdx <<<<<<<<<===================== Here.
0xffffffff802d225d <probe_wakeup_sched_switch+214>: callq 0xffffffff802cc258 <trace_function>
0xffffffff802d2262 <probe_wakeup_sched_switch+219>: mov 0xf2de6f(%rip),%rdi # 0xffffffff812000d8 <wakeup_trace>
0xffffffff802d2269 <probe_wakeup_sched_switch+226>: mov -0x30(%rbp),%rsi
0xffffffff802d226d <probe_wakeup_sched_switch+230>: mov %r14d,%r8d
0xffffffff802d2270 <probe_wakeup_sched_switch+233>: mov %r13,%rdx
0xffffffff802d2273 <probe_wakeup_sched_switch+236>: mov %r12,%rcx
0xffffffff802d2276 <probe_wakeup_sched_switch+239>: callq 0xffffffff802cd5b---Type <return> to continue, or q <return> to quit---
2 <tracing_sched_switch_trace>
Some stack from rbp
crash> rd 0xffff8801d05c7ed0 50
ffff8801d05c7ed0: ffff8801d05c7f70 ffffffff806f8264 p.\.....d.o.....
ffff8801d05c7ee0: 0000000000000002 ffff8801d05c7f48 ........H.\.....
ffff8801d05c7ef0: ffff8801d043d040 ffff88022f21d820 @.C..... .!/....
ffff8801d05c7f00: ffff8801d043d3e8 0000000300000000 ..C.............
ffff8801d05c7f10: 00007fff656c34d0 0000000000000000 .4le............
ffff8801d05c7f20: 0000000000000000 0000000000000000 ................
ffff8801d05c7f30: ffffffff806fb61b 00000000000cf7bd ..o.............
ffff8801d05c7f40: 0000000000000857 00000000008c3fc0 W........?......
ffff8801d05c7f50: 0000000000000000 00007fff656c34d0 .........4le....
ffff8801d05c7f60: 0000000000000000 0000000000000000 ................
ffff8801d05c7f70: 0000000000000000 ffffffff8020d56d ........m. .....
ffff8801d05c7f80: 000000000000fe2e 0000000000000202 ................
ffff8801d05c7f90: 0000003b4b151a30 0000000000000003 0..K;...........
ffff8801d05c7fa0: 0000000000000004 00000000008c3fc0 .........?......
ffff8801d05c7fb0: 00000000008c30b0 0000000000000000 .0..............
ffff8801d05c7fc0: 00007f8e5a102de8 0000000000000000 .-.Z............
ffff8801d05c7fd0: ffffffffffffffff 00000000004533f6 .........3E.....
ffff8801d05c7fe0: 0000000000000033 0000000000010206 3...............
ffff8801d05c7ff0: 00007fff656c3270 000000000000002b p2le....+.......
ffff8801d05c8000: 0000000000000000 0000000000000000 ................
and some relvant text addresses
crash> sym ffffffff806f8264
ffffffff806f8264 (T) schedule+3481 include/trace/sched_event_types.h: 149
crash> sym ffffffff8020d56d
ffffffff8020d56d (t) retint_careful+18 include/linux/kdev_t.h: 52
crash> sym ffffffff806fb61b
ffffffff806fb61b (T) trace_hardirqs_on_thunk+58 include/linux/hrtimer.h: 199
crash> sym ffff880028167d40
It seems CALLER_ADDR is not correct?
Thanks
Maneesh
--
Maneesh Soni
Linux Technology Center
IBM India Systems and Technology Lab,
Bangalore, India.
On Fri, 27 Mar 2009, Maneesh Soni wrote:
> >
> > Hi Steven,
> >
> > I still the oops with the patch,
> >
> > BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> > IP: [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> > PGD 1d68e9067 PUD 1e6cdf067 PMD 0
> > Oops: 0000 [#1] SMP
> > last sysfs file: /sys/devices/pci0000:0c/0000:0c:00.0/irq
> > CPU 5
> > Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
> > Pid: 6095, comm: cat Not tainted 2.6.29-tip-test #2 eserver xSeries 366-[88632RA]-
> > RIP: 0010:[<ffffffff802d2259>] [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> > RSP: 0000:ffff8801e6d03ea0 EFLAGS: 00010046
> > RAX: 0000000000000000 RBX: ffff8800281989a0 RCX: 0000000000000046
> > RDX: ffff8800281989a0 RSI: ffffffff8020d56d RDI: ffffffff811de480
> > RBP: ffff8801e6d03ed0 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000046
> > R13: ffff88022f232820 R14: 0000000000000000 R15: 0000000000000005
> > FS: 00007f98b94cb6e0(0000) GS:ffff88002818b000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 0000000000000008 CR3: 0000000224595000 CR4: 00000000000006e0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process cat (pid: 6095, threadinfo ffff8801e6d02000, task ffff8801d6935040)
> > Stack:
> > ffff8801d6935040 ffff880224981230 ffff88002819dd40 0000003b4ae00aa0
> > 00007fffc14df830 ffff88002819dd40 ffff8801e6d03f70 ffffffff806f8264
> > 0000000000000002 ffff8801e6d03f48 ffff8801d6935040 ffff88022f232820
> > Call Trace:
> > [<ffffffff806f8264>] schedule+0xd99/0x10f9
> > [<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> > [<ffffffff8027d2cb>] ? delayed_work_timer_fn+0x0/0x38
> > [<ffffffff8020d56d>] retint_careful+0x12/0x2e
> > Code: 2d ad de f2 00 75 77 48 63 05 ac de f2 00 48 8b 4d 00 45 89 f0 48 8b 3d 8e de f2 00 48 8b 71 08 48 8b 5c c7 28 48 8b 01 4c 89 e1 <48> 8b 50 08 e8 f6 9f ff ff 48 8b 3d 6f de f2 00 48 8b 75 d0 45
> > RIP [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> > RSP <ffff8801e6d03ea0>
> > CR2: 0000000000000008
> >
>
> I could get a kdump in my next attempt. Let me know if you want me to upload the dump
> somewhere. (700 MB compressed)
>
> Some info from the dump using gdb & crash-utility.
Thanks, this is indeed helpful.
>
> (gdb) bt full
> #0 0xffffffff802d2259 in probe_wakeup_sched_switch (rq=<value optimized out>,
> prev=Cannot access memory at address 0xffff8801d05c7ea0
> ) at kernel/trace/trace_sched_wakeup.c:157
> data = (struct trace_array_cpu *) 0xffff8800281629a0
> T0 = <value optimized out>
> T1 = <value optimized out>
> delta = <value optimized out>
> flags = 70
> pc = 0
> Cannot access memory at address 0xffff8801d05c7ed8
> (gdb) print wakeup_trace
> $7 = (struct trace_array *) 0xffffffff811de480
> (gdb) print *(struct trace_array *) 0xffffffff811de480
> $8 = {buffer = 0xffff88022f090440, entries = 1441792, cpu = 0,
> time_start = 437360893965, waiter = 0x0, data = {0xffff8800281119a0,
> 0xffff88002812c9a0, 0xffff8800281479a0, 0xffff8800281629a0,
> 0xffff88002817d9a0, 0xffff8800281989a0, 0xffff8800281b39a0,
> 0xffff8800281ce9a0, 0x0 <repeats 247 times>}}
> (gdb) print tracer_enabled
> $9 = 1
> (gdb) print wakeup_cpu
> $10 = 3
> (gdb) info threads
> 8 process 0 0xffffffff80217024 in mwait_idle ()
> at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> 7 process 0 0xffffffff80217024 in mwait_idle ()
> at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> 6 process 0 0xffffffff80217024 in mwait_idle ()
> at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> 5 process 0 0xffffffff80217024 in mwait_idle ()
> at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> * 4 process 5773 0xffffffff802d2259 in probe_wakeup_sched_switch (
> rq=<value optimized out>, prev=Cannot access memory at address 0xffff8801d05c7ea0
> ) at kernel/trace/trace_sched_wakeup.c:157
> 3 process 0 0xffffffff80217024 in mwait_idle ()
> at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> 2 process 0 0xffffffff80217024 in mwait_idle ()
> at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> 1 process 0 0xffffffff80217024 in mwait_idle ()
> at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> (gdb) info registers
> rax 0x0 0
> rbx 0xffff8800281629a0 -131940722792032
> rcx 0x46 70
> rdx 0xffff8800281629a0 -131940722792032
> rsi 0xffffffff8020d56d -2145331859
> rdi 0xffffffff811de480 -2128747392
> rbp 0xffff8801d05c7ed0 0xffff8801d05c7ed0
> rsp 0xffff8801d05c7ea0 0xffff8801d05c7ea0
> r8 0x0 0
> r9 0x0 0
> r10 0xffff8801d05c7eb0 -131933604643152
> r11 0x1 1
> r12 0x46 70
> r13 0xffff88022f21d820 -131932014651360
> r14 0x0 0
> r15 0x3 3
> rip 0xffffffff802d2259 0xffffffff802d2259 <probe_wakeup_sched_switch+210>
> eflags 0x10046 [ PF ZF RF ]
> cs 0x10 16
> ss 0x0 0
> ds 0x0 0
> es 0x0 0
> fs 0x0 0
> gs 0x0 0
> fctrl 0x0 0
> fstat 0x0 0
> ftag 0x0 0
> fiseg 0x0 0
> fioff 0x0 0
> foseg 0x0 0
> fooff 0x0 0
> fop 0x0 0
> mxcsr 0x0 [ ]
> (gdb)
> ...
> ...
> 0xffffffff802d2208 <probe_wakeup_sched_switch+129>: lock xadd %ax,0xf2debf(%rip) # 0xffffffff812000d0 <wakeup_lock>
> 0xffffffff802d2211 <probe_wakeup_sched_switch+138>: cmp %ah,%al
> 0xffffffff802d2213 <probe_wakeup_sched_switch+140>: je 0xffffffff802d221f <probe_wakeup_sched_switch+152>
> 0xffffffff802d2215 <probe_wakeup_sched_switch+142>: pause
> 0xffffffff802d2217 <probe_wakeup_sched_switch+144>: mov 0xf2deb3(%rip),%al # 0xffffffff812000d0 <wakeup_lock>
> 0xffffffff802d221d <probe_wakeup_sched_switch+150>: jmp 0xffffffff802d2211 <probe_wakeup_sched_switch+138>
> 0xffffffff802d221f <probe_wakeup_sched_switch+152>: cmpl $0x0,0x9fb02a(%rip) # 0xffffffff80ccd250 <tracer_enabled>
> 0xffffffff802d2226 <probe_wakeup_sched_switch+159>: je 0xffffffff802d22ac <probe_wakeup_sched_switch+293>
> 0xffffffff802d222c <probe_wakeup_sched_switch+165>: cmp 0xf2dead(%rip),%r13 # 0xffffffff812000e0 <wakeup_task>
> 0xffffffff802d2233 <probe_wakeup_sched_switch+172>: jne 0xffffffff802d22ac <probe_wakeup_sched_switch+293>
> 0xffffffff802d2235 <probe_wakeup_sched_switch+174>: movslq 0xf2deac(%rip),%rax # 0xffffffff812000e8 <wakeup_cpu>
> 0xffffffff802d223c <probe_wakeup_sched_switch+181>: mov 0x0(%rbp),%rcx
Here we load the frame pointer into %rcx.
> 0xffffffff802d2240 <probe_wakeup_sched_switch+185>: mov %r14d,%r8d
> 0xffffffff802d2243 <probe_wakeup_sched_switch+188>: mov 0xf2de8e(%rip),%rdi # 0xffffffff812000d8 <wakeup_trace>
> 0xffffffff802d224a <probe_wakeup_sched_switch+195>: mov 0x8(%rcx),%rsi
CALLER_ADDR1 gets loaded into %rsi
> 0xffffffff802d224e <probe_wakeup_sched_switch+199>: mov 0x28(%rdi,%rax,8),%rbx
> 0xffffffff802d2253 <probe_wakeup_sched_switch+204>: mov (%rcx),%rax
We move what should be the caller of the next frame into %rax but this
ends up being NULL?
> 0xffffffff802d2256 <probe_wakeup_sched_switch+207>: mov %r12,%rcx
> 0xffffffff802d2259 <probe_wakeup_sched_switch+210>: mov 0x8(%rax),%rdx <<<<<<<<<===================== Here.
Hence, we crash.
This is really strange. CALLER_ADDR* are internal gcc
__builtin_return_address(*) functions.
Do you have CONFIG_FRAME_POINTER set?
Is this running in a virtual box or bare metal? (probably should not matter)
What version of gcc do you have?
Hmm, looking back at the call stack:
> > [<ffffffff806f8264>] schedule+0xd99/0x10f9
> > [<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> > [<ffffffff8027d2cb>] ? delayed_work_timer_fn+0x0/0x38
> > [<ffffffff8020d56d>] retint_careful+0x12/0x2e
I wonder if we do not set up a stack frame properly. retint_careful is
called by assembly. I'll have a look at that.
Thanks,
-- Steve
> 0xffffffff802d225d <probe_wakeup_sched_switch+214>: callq 0xffffffff802cc258 <trace_function>
> 0xffffffff802d2262 <probe_wakeup_sched_switch+219>: mov 0xf2de6f(%rip),%rdi # 0xffffffff812000d8 <wakeup_trace>
> 0xffffffff802d2269 <probe_wakeup_sched_switch+226>: mov -0x30(%rbp),%rsi
> 0xffffffff802d226d <probe_wakeup_sched_switch+230>: mov %r14d,%r8d
> 0xffffffff802d2270 <probe_wakeup_sched_switch+233>: mov %r13,%rdx
> 0xffffffff802d2273 <probe_wakeup_sched_switch+236>: mov %r12,%rcx
> 0xffffffff802d2276 <probe_wakeup_sched_switch+239>: callq 0xffffffff802cd5b---Type <return> to continue, or q <return> to quit---
> 2 <tracing_sched_switch_trace>
>
>
> Some stack from rbp
>
> crash> rd 0xffff8801d05c7ed0 50
> ffff8801d05c7ed0: ffff8801d05c7f70 ffffffff806f8264 p.\.....d.o.....
> ffff8801d05c7ee0: 0000000000000002 ffff8801d05c7f48 ........H.\.....
> ffff8801d05c7ef0: ffff8801d043d040 ffff88022f21d820 @.C..... .!/....
> ffff8801d05c7f00: ffff8801d043d3e8 0000000300000000 ..C.............
> ffff8801d05c7f10: 00007fff656c34d0 0000000000000000 .4le............
> ffff8801d05c7f20: 0000000000000000 0000000000000000 ................
> ffff8801d05c7f30: ffffffff806fb61b 00000000000cf7bd ..o.............
> ffff8801d05c7f40: 0000000000000857 00000000008c3fc0 W........?......
> ffff8801d05c7f50: 0000000000000000 00007fff656c34d0 .........4le....
> ffff8801d05c7f60: 0000000000000000 0000000000000000 ................
> ffff8801d05c7f70: 0000000000000000 ffffffff8020d56d ........m. .....
> ffff8801d05c7f80: 000000000000fe2e 0000000000000202 ................
> ffff8801d05c7f90: 0000003b4b151a30 0000000000000003 0..K;...........
> ffff8801d05c7fa0: 0000000000000004 00000000008c3fc0 .........?......
> ffff8801d05c7fb0: 00000000008c30b0 0000000000000000 .0..............
> ffff8801d05c7fc0: 00007f8e5a102de8 0000000000000000 .-.Z............
> ffff8801d05c7fd0: ffffffffffffffff 00000000004533f6 .........3E.....
> ffff8801d05c7fe0: 0000000000000033 0000000000010206 3...............
> ffff8801d05c7ff0: 00007fff656c3270 000000000000002b p2le....+.......
> ffff8801d05c8000: 0000000000000000 0000000000000000 ................
>
> and some relvant text addresses
> crash> sym ffffffff806f8264
> ffffffff806f8264 (T) schedule+3481 include/trace/sched_event_types.h: 149
> crash> sym ffffffff8020d56d
> ffffffff8020d56d (t) retint_careful+18 include/linux/kdev_t.h: 52
> crash> sym ffffffff806fb61b
> ffffffff806fb61b (T) trace_hardirqs_on_thunk+58 include/linux/hrtimer.h: 199
> crash> sym ffff880028167d40
>
> It seems CALLER_ADDR is not correct?
>
> Thanks
> Maneesh
>
>
> --
> Maneesh Soni
> Linux Technology Center
> IBM India Systems and Technology Lab,
> Bangalore, India.
>
On Fri, Mar 27, 2009 at 09:15:29AM -0400, Steven Rostedt wrote:
>
> On Fri, 27 Mar 2009, Maneesh Soni wrote:
> > >
> > > Hi Steven,
> > >
> > > I still the oops with the patch,
> > >
> > > BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> > > IP: [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> > > PGD 1d68e9067 PUD 1e6cdf067 PMD 0
> > > Oops: 0000 [#1] SMP
> > > last sysfs file: /sys/devices/pci0000:0c/0000:0c:00.0/irq
> > > CPU 5
> > > Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
> > > Pid: 6095, comm: cat Not tainted 2.6.29-tip-test #2 eserver xSeries 366-[88632RA]-
> > > RIP: 0010:[<ffffffff802d2259>] [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> > > RSP: 0000:ffff8801e6d03ea0 EFLAGS: 00010046
> > > RAX: 0000000000000000 RBX: ffff8800281989a0 RCX: 0000000000000046
> > > RDX: ffff8800281989a0 RSI: ffffffff8020d56d RDI: ffffffff811de480
> > > RBP: ffff8801e6d03ed0 R08: 0000000000000000 R09: 0000000000000000
RBP is very close to RSP, so it's very likely that FRAME_POINTER is enabled.
> > > R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000046
> > > R13: ffff88022f232820 R14: 0000000000000000 R15: 0000000000000005
> > > FS: 00007f98b94cb6e0(0000) GS:ffff88002818b000(0000) knlGS:0000000000000000
> > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > CR2: 0000000000000008 CR3: 0000000224595000 CR4: 00000000000006e0
> > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > Process cat (pid: 6095, threadinfo ffff8801e6d02000, task ffff8801d6935040)
> > > Stack:
> > > ffff8801d6935040 ffff880224981230 ffff88002819dd40 0000003b4ae00aa0
> > > 00007fffc14df830 ffff88002819dd40 ffff8801e6d03f70 ffffffff806f8264
> > > 0000000000000002 ffff8801e6d03f48 ffff8801d6935040 ffff88022f232820
> > > Call Trace:
> > > [<ffffffff806f8264>] schedule+0xd99/0x10f9
> > > [<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> > > [<ffffffff8027d2cb>] ? delayed_work_timer_fn+0x0/0x38
> > > [<ffffffff8020d56d>] retint_careful+0x12/0x2e
> > > Code: 2d ad de f2 00 75 77 48 63 05 ac de f2 00 48 8b 4d 00 45 89 f0 48 8b 3d 8e de f2 00 48 8b 71 08 48 8b 5c c7 28 48 8b 01 4c 89 e1 <48> 8b 50 08 e8 f6 9f ff ff 48 8b 3d 6f de f2 00 48 8b 75 d0 45
> > > RIP [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> > > RSP <ffff8801e6d03ea0>
> > > CR2: 0000000000000008
> > >
> >
> > I could get a kdump in my next attempt. Let me know if you want me to upload the dump
> > somewhere. (700 MB compressed)
> >
> > Some info from the dump using gdb & crash-utility.
>
> Thanks, this is indeed helpful.
>
> >
> > (gdb) bt full
> > #0 0xffffffff802d2259 in probe_wakeup_sched_switch (rq=<value optimized out>,
> > prev=Cannot access memory at address 0xffff8801d05c7ea0
> > ) at kernel/trace/trace_sched_wakeup.c:157
> > data = (struct trace_array_cpu *) 0xffff8800281629a0
> > T0 = <value optimized out>
> > T1 = <value optimized out>
> > delta = <value optimized out>
> > flags = 70
> > pc = 0
> > Cannot access memory at address 0xffff8801d05c7ed8
> > (gdb) print wakeup_trace
> > $7 = (struct trace_array *) 0xffffffff811de480
> > (gdb) print *(struct trace_array *) 0xffffffff811de480
> > $8 = {buffer = 0xffff88022f090440, entries = 1441792, cpu = 0,
> > time_start = 437360893965, waiter = 0x0, data = {0xffff8800281119a0,
> > 0xffff88002812c9a0, 0xffff8800281479a0, 0xffff8800281629a0,
> > 0xffff88002817d9a0, 0xffff8800281989a0, 0xffff8800281b39a0,
> > 0xffff8800281ce9a0, 0x0 <repeats 247 times>}}
> > (gdb) print tracer_enabled
> > $9 = 1
> > (gdb) print wakeup_cpu
> > $10 = 3
> > (gdb) info threads
> > 8 process 0 0xffffffff80217024 in mwait_idle ()
> > at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> > 7 process 0 0xffffffff80217024 in mwait_idle ()
> > at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> > 6 process 0 0xffffffff80217024 in mwait_idle ()
> > at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> > 5 process 0 0xffffffff80217024 in mwait_idle ()
> > at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> > * 4 process 5773 0xffffffff802d2259 in probe_wakeup_sched_switch (
> > rq=<value optimized out>, prev=Cannot access memory at address 0xffff8801d05c7ea0
> > ) at kernel/trace/trace_sched_wakeup.c:157
> > 3 process 0 0xffffffff80217024 in mwait_idle ()
> > at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> > 2 process 0 0xffffffff80217024 in mwait_idle ()
> > at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> > 1 process 0 0xffffffff80217024 in mwait_idle ()
> > at /home/maneesh/linux-2.6-tip/arch/x86/include/asm/processor.h:732
> > (gdb) info registers
> > rax 0x0 0
> > rbx 0xffff8800281629a0 -131940722792032
> > rcx 0x46 70
> > rdx 0xffff8800281629a0 -131940722792032
> > rsi 0xffffffff8020d56d -2145331859
> > rdi 0xffffffff811de480 -2128747392
> > rbp 0xffff8801d05c7ed0 0xffff8801d05c7ed0
> > rsp 0xffff8801d05c7ea0 0xffff8801d05c7ea0
> > r8 0x0 0
> > r9 0x0 0
> > r10 0xffff8801d05c7eb0 -131933604643152
> > r11 0x1 1
> > r12 0x46 70
> > r13 0xffff88022f21d820 -131932014651360
> > r14 0x0 0
> > r15 0x3 3
> > rip 0xffffffff802d2259 0xffffffff802d2259 <probe_wakeup_sched_switch+210>
> > eflags 0x10046 [ PF ZF RF ]
> > cs 0x10 16
> > ss 0x0 0
> > ds 0x0 0
> > es 0x0 0
> > fs 0x0 0
> > gs 0x0 0
> > fctrl 0x0 0
> > fstat 0x0 0
> > ftag 0x0 0
> > fiseg 0x0 0
> > fioff 0x0 0
> > foseg 0x0 0
> > fooff 0x0 0
> > fop 0x0 0
> > mxcsr 0x0 [ ]
> > (gdb)
> > ...
> > ...
> > 0xffffffff802d2208 <probe_wakeup_sched_switch+129>: lock xadd %ax,0xf2debf(%rip) # 0xffffffff812000d0 <wakeup_lock>
> > 0xffffffff802d2211 <probe_wakeup_sched_switch+138>: cmp %ah,%al
> > 0xffffffff802d2213 <probe_wakeup_sched_switch+140>: je 0xffffffff802d221f <probe_wakeup_sched_switch+152>
> > 0xffffffff802d2215 <probe_wakeup_sched_switch+142>: pause
> > 0xffffffff802d2217 <probe_wakeup_sched_switch+144>: mov 0xf2deb3(%rip),%al # 0xffffffff812000d0 <wakeup_lock>
> > 0xffffffff802d221d <probe_wakeup_sched_switch+150>: jmp 0xffffffff802d2211 <probe_wakeup_sched_switch+138>
> > 0xffffffff802d221f <probe_wakeup_sched_switch+152>: cmpl $0x0,0x9fb02a(%rip) # 0xffffffff80ccd250 <tracer_enabled>
> > 0xffffffff802d2226 <probe_wakeup_sched_switch+159>: je 0xffffffff802d22ac <probe_wakeup_sched_switch+293>
> > 0xffffffff802d222c <probe_wakeup_sched_switch+165>: cmp 0xf2dead(%rip),%r13 # 0xffffffff812000e0 <wakeup_task>
> > 0xffffffff802d2233 <probe_wakeup_sched_switch+172>: jne 0xffffffff802d22ac <probe_wakeup_sched_switch+293>
> > 0xffffffff802d2235 <probe_wakeup_sched_switch+174>: movslq 0xf2deac(%rip),%rax # 0xffffffff812000e8 <wakeup_cpu>
>
> > 0xffffffff802d223c <probe_wakeup_sched_switch+181>: mov 0x0(%rbp),%rcx
>
> Here we load the frame pointer into %rcx.
>
> > 0xffffffff802d2240 <probe_wakeup_sched_switch+185>: mov %r14d,%r8d
> > 0xffffffff802d2243 <probe_wakeup_sched_switch+188>: mov 0xf2de8e(%rip),%rdi # 0xffffffff812000d8 <wakeup_trace>
>
> > 0xffffffff802d224a <probe_wakeup_sched_switch+195>: mov 0x8(%rcx),%rsi
>
> CALLER_ADDR1 gets loaded into %rsi
>
> > 0xffffffff802d224e <probe_wakeup_sched_switch+199>: mov 0x28(%rdi,%rax,8),%rbx
>
>
> > 0xffffffff802d2253 <probe_wakeup_sched_switch+204>: mov (%rcx),%rax
>
> We move what should be the caller of the next frame into %rax but this
> ends up being NULL?
>
> > 0xffffffff802d2256 <probe_wakeup_sched_switch+207>: mov %r12,%rcx
> > 0xffffffff802d2259 <probe_wakeup_sched_switch+210>: mov 0x8(%rax),%rdx <<<<<<<<<===================== Here.
>
> Hence, we crash.
>
> This is really strange. CALLER_ADDR* are internal gcc
> __builtin_return_address(*) functions.
>
> Do you have CONFIG_FRAME_POINTER set?
>
> Is this running in a virtual box or bare metal? (probably should not matter)
>
> What version of gcc do you have?
>
> Hmm, looking back at the call stack:
>
> > > [<ffffffff806f8264>] schedule+0xd99/0x10f9
> > > [<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> > > [<ffffffff8027d2cb>] ? delayed_work_timer_fn+0x0/0x38
> > > [<ffffffff8020d56d>] retint_careful+0x12/0x2e
>
> I wonder if we do not set up a stack frame properly. retint_careful is
> called by assembly. I'll have a look at that
Aah, indeed.
May be should probe_kernel_address before saving
the addresses.
Frederic.
.
>
> Thanks,
>
> -- Steve
>
>
>
>
> > 0xffffffff802d225d <probe_wakeup_sched_switch+214>: callq 0xffffffff802cc258 <trace_function>
> > 0xffffffff802d2262 <probe_wakeup_sched_switch+219>: mov 0xf2de6f(%rip),%rdi # 0xffffffff812000d8 <wakeup_trace>
> > 0xffffffff802d2269 <probe_wakeup_sched_switch+226>: mov -0x30(%rbp),%rsi
> > 0xffffffff802d226d <probe_wakeup_sched_switch+230>: mov %r14d,%r8d
> > 0xffffffff802d2270 <probe_wakeup_sched_switch+233>: mov %r13,%rdx
> > 0xffffffff802d2273 <probe_wakeup_sched_switch+236>: mov %r12,%rcx
> > 0xffffffff802d2276 <probe_wakeup_sched_switch+239>: callq 0xffffffff802cd5b---Type <return> to continue, or q <return> to quit---
> > 2 <tracing_sched_switch_trace>
> >
> >
> > Some stack from rbp
> >
> > crash> rd 0xffff8801d05c7ed0 50
> > ffff8801d05c7ed0: ffff8801d05c7f70 ffffffff806f8264 p.\.....d.o.....
> > ffff8801d05c7ee0: 0000000000000002 ffff8801d05c7f48 ........H.\.....
> > ffff8801d05c7ef0: ffff8801d043d040 ffff88022f21d820 @.C..... .!/....
> > ffff8801d05c7f00: ffff8801d043d3e8 0000000300000000 ..C.............
> > ffff8801d05c7f10: 00007fff656c34d0 0000000000000000 .4le............
> > ffff8801d05c7f20: 0000000000000000 0000000000000000 ................
> > ffff8801d05c7f30: ffffffff806fb61b 00000000000cf7bd ..o.............
> > ffff8801d05c7f40: 0000000000000857 00000000008c3fc0 W........?......
> > ffff8801d05c7f50: 0000000000000000 00007fff656c34d0 .........4le....
> > ffff8801d05c7f60: 0000000000000000 0000000000000000 ................
> > ffff8801d05c7f70: 0000000000000000 ffffffff8020d56d ........m. .....
> > ffff8801d05c7f80: 000000000000fe2e 0000000000000202 ................
> > ffff8801d05c7f90: 0000003b4b151a30 0000000000000003 0..K;...........
> > ffff8801d05c7fa0: 0000000000000004 00000000008c3fc0 .........?......
> > ffff8801d05c7fb0: 00000000008c30b0 0000000000000000 .0..............
> > ffff8801d05c7fc0: 00007f8e5a102de8 0000000000000000 .-.Z............
> > ffff8801d05c7fd0: ffffffffffffffff 00000000004533f6 .........3E.....
> > ffff8801d05c7fe0: 0000000000000033 0000000000010206 3...............
> > ffff8801d05c7ff0: 00007fff656c3270 000000000000002b p2le....+.......
> > ffff8801d05c8000: 0000000000000000 0000000000000000 ................
> >
> > and some relvant text addresses
> > crash> sym ffffffff806f8264
> > ffffffff806f8264 (T) schedule+3481 include/trace/sched_event_types.h: 149
> > crash> sym ffffffff8020d56d
> > ffffffff8020d56d (t) retint_careful+18 include/linux/kdev_t.h: 52
> > crash> sym ffffffff806fb61b
> > ffffffff806fb61b (T) trace_hardirqs_on_thunk+58 include/linux/hrtimer.h: 199
> > crash> sym ffff880028167d40
> >
> > It seems CALLER_ADDR is not correct?
> >
> > Thanks
> > Maneesh
> >
> >
> > --
> > Maneesh Soni
> > Linux Technology Center
> > IBM India Systems and Technology Lab,
> > Bangalore, India.
> >
On Fri, 27 Mar 2009, Frederic Weisbecker wrote:
> >
> > Hmm, looking back at the call stack:
> >
> > > > [<ffffffff806f8264>] schedule+0xd99/0x10f9
> > > > [<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> > > > [<ffffffff8027d2cb>] ? delayed_work_timer_fn+0x0/0x38
> > > > [<ffffffff8020d56d>] retint_careful+0x12/0x2e
> >
> > I wonder if we do not set up a stack frame properly. retint_careful is
> > called by assembly. I'll have a look at that
>
>
> Aah, indeed.
> May be should probe_kernel_address before saving
> the addresses.
>
Looking at the retint_careful, I'm 90% sure that is the bug.
Unfortunately, the CALLER_ADDR2 is a built in gcc function.
Looks like only the wakeup latency tracer calls it, perhaps we should
simply pass 0 instead?
-- Steve
On Fri, Mar 27, 2009 at 09:15:29AM -0400, Steven Rostedt wrote:
...
> Do you have CONFIG_FRAME_POINTER set?
>
Yes, CONFIG_FRAME_POINTER=y
> Is this running in a virtual box or bare metal? (probably should not matter)
>
Bare metal.
> What version of gcc do you have?
>
gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-44)
Thanks
Maneesh
--
Maneesh Soni
Linux Technology Center
IBM India Systems and Technology Lab,
Bangalore, India.
On Fri, 27 Mar 2009, Maneesh Soni wrote:
> > Hi Steven,
> >
> > I still the oops with the patch,
> >
> > BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> > IP: [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> > PGD 1d68e9067 PUD 1e6cdf067 PMD 0
> > Oops: 0000 [#1] SMP
> > last sysfs file: /sys/devices/pci0000:0c/0000:0c:00.0/irq
> > CPU 5
> > Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
> > Pid: 6095, comm: cat Not tainted 2.6.29-tip-test #2 eserver xSeries 366-[88632RA]-
> > RIP: 0010:[<ffffffff802d2259>] [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> > RSP: 0000:ffff8801e6d03ea0 EFLAGS: 00010046
> > RAX: 0000000000000000 RBX: ffff8800281989a0 RCX: 0000000000000046
> > RDX: ffff8800281989a0 RSI: ffffffff8020d56d RDI: ffffffff811de480
> > RBP: ffff8801e6d03ed0 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000046
> > R13: ffff88022f232820 R14: 0000000000000000 R15: 0000000000000005
> > FS: 00007f98b94cb6e0(0000) GS:ffff88002818b000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 0000000000000008 CR3: 0000000224595000 CR4: 00000000000006e0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process cat (pid: 6095, threadinfo ffff8801e6d02000, task ffff8801d6935040)
> > Stack:
> > ffff8801d6935040 ffff880224981230 ffff88002819dd40 0000003b4ae00aa0
> > 00007fffc14df830 ffff88002819dd40 ffff8801e6d03f70 ffffffff806f8264
> > 0000000000000002 ffff8801e6d03f48 ffff8801d6935040 ffff88022f232820
> > Call Trace:
> > [<ffffffff806f8264>] schedule+0xd99/0x10f9
> > [<ffffffff806fb61b>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> > [<ffffffff8027d2cb>] ? delayed_work_timer_fn+0x0/0x38
> > [<ffffffff8020d56d>] retint_careful+0x12/0x2e
> > Code: 2d ad de f2 00 75 77 48 63 05 ac de f2 00 48 8b 4d 00 45 89 f0 48 8b 3d 8e de f2 00 48 8b 71 08 48 8b 5c c7 28 48 8b 01 4c 89 e1 <48> 8b 50 08 e8 f6 9f ff ff 48 8b 3d 6f de f2 00 48 8b 75 d0 45
> > RIP [<ffffffff802d2259>] probe_wakeup_sched_switch+0xd2/0x1ac
> > RSP <ffff8801e6d03ea0>
> > CR2: 0000000000000008
> >
> 0xffffffff802d2208 <probe_wakeup_sched_switch+129>: lock xadd %ax,0xf2debf(%rip) # 0xffffffff812000d0 <wakeup_lock>
> 0xffffffff802d2211 <probe_wakeup_sched_switch+138>: cmp %ah,%al
> 0xffffffff802d2213 <probe_wakeup_sched_switch+140>: je 0xffffffff802d221f <probe_wakeup_sched_switch+152>
> 0xffffffff802d2215 <probe_wakeup_sched_switch+142>: pause
> 0xffffffff802d2217 <probe_wakeup_sched_switch+144>: mov 0xf2deb3(%rip),%al # 0xffffffff812000d0 <wakeup_lock>
> 0xffffffff802d221d <probe_wakeup_sched_switch+150>: jmp 0xffffffff802d2211 <probe_wakeup_sched_switch+138>
> 0xffffffff802d221f <probe_wakeup_sched_switch+152>: cmpl $0x0,0x9fb02a(%rip) # 0xffffffff80ccd250 <tracer_enabled>
> 0xffffffff802d2226 <probe_wakeup_sched_switch+159>: je 0xffffffff802d22ac <probe_wakeup_sched_switch+293>
> 0xffffffff802d222c <probe_wakeup_sched_switch+165>: cmp 0xf2dead(%rip),%r13 # 0xffffffff812000e0 <wakeup_task>
> 0xffffffff802d2233 <probe_wakeup_sched_switch+172>: jne 0xffffffff802d22ac <probe_wakeup_sched_switch+293>
> 0xffffffff802d2235 <probe_wakeup_sched_switch+174>: movslq 0xf2deac(%rip),%rax # 0xffffffff812000e8 <wakeup_cpu>
> 0xffffffff802d223c <probe_wakeup_sched_switch+181>: mov 0x0(%rbp),%rcx
> 0xffffffff802d2240 <probe_wakeup_sched_switch+185>: mov %r14d,%r8d
> 0xffffffff802d2243 <probe_wakeup_sched_switch+188>: mov 0xf2de8e(%rip),%rdi # 0xffffffff812000d8 <wakeup_trace>
> 0xffffffff802d224a <probe_wakeup_sched_switch+195>: mov 0x8(%rcx),%rsi
> 0xffffffff802d224e <probe_wakeup_sched_switch+199>: mov 0x28(%rdi,%rax,8),%rbx
> 0xffffffff802d2253 <probe_wakeup_sched_switch+204>: mov (%rcx),%rax
> 0xffffffff802d2256 <probe_wakeup_sched_switch+207>: mov %r12,%rcx
> 0xffffffff802d2259 <probe_wakeup_sched_switch+210>: mov 0x8(%rax),%rdx <<<<<<<<<===================== Here.
> 0xffffffff802d225d <probe_wakeup_sched_switch+214>: callq 0xffffffff802cc258 <trace_function>
> 0xffffffff802d2262 <probe_wakeup_sched_switch+219>: mov 0xf2de6f(%rip),%rdi # 0xffffffff812000d8 <wakeup_trace>
> 0xffffffff802d2269 <probe_wakeup_sched_switch+226>: mov -0x30(%rbp),%rsi
> 0xffffffff802d226d <probe_wakeup_sched_switch+230>: mov %r14d,%r8d
> 0xffffffff802d2270 <probe_wakeup_sched_switch+233>: mov %r13,%rdx
> 0xffffffff802d2273 <probe_wakeup_sched_switch+236>: mov %r12,%rcx
> 0xffffffff802d2276 <probe_wakeup_sched_switch+239>: callq 0xffffffff802cd5b---Type <return> to continue, or q <return> to quit---
> 2 <tracing_sched_switch_trace>
> It seems CALLER_ADDR is not correct?
Hi Maneesh,
Could you try this patch and see if it keeps your system from crashing?
Thanks,
-- Steve
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index a331ec3..dbf3f8f 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -917,10 +917,15 @@ retint_careful:
TRACE_IRQS_ON
ENABLE_INTERRUPTS(CLBR_NONE)
pushq %rdi
- CFI_ADJUST_CFA_OFFSET 8
+ pushq %rbp
+ call 1f
+1: mov %rsp, %rbp
+ CFI_ADJUST_CFA_OFFSET 24
call schedule
+ addq $8, %rsp /* skip call */
+ popq %rbp
popq %rdi
- CFI_ADJUST_CFA_OFFSET -8
+ CFI_ADJUST_CFA_OFFSET -24
GET_THREAD_INFO(%rcx)
DISABLE_INTERRUPTS(CLBR_NONE)
TRACE_IRQS_OFF
On Wed, Apr 01, 2009 at 07:42:58PM -0400, Steven Rostedt wrote:
....
>
>
>
> Hi Maneesh,
>
> Could you try this patch and see if it keeps your system from crashing?
>
> Thanks,
>
> -- Steve
>
> diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
> index a331ec3..dbf3f8f 100644
> --- a/arch/x86/kernel/entry_64.S
> +++ b/arch/x86/kernel/entry_64.S
> @@ -917,10 +917,15 @@ retint_careful:
> TRACE_IRQS_ON
> ENABLE_INTERRUPTS(CLBR_NONE)
> pushq %rdi
> - CFI_ADJUST_CFA_OFFSET 8
> + pushq %rbp
> + call 1f
> +1: mov %rsp, %rbp
> + CFI_ADJUST_CFA_OFFSET 24
> call schedule
> + addq $8, %rsp /* skip call */
> + popq %rbp
> popq %rdi
> - CFI_ADJUST_CFA_OFFSET -8
> + CFI_ADJUST_CFA_OFFSET -24
> GET_THREAD_INFO(%rcx)
> DISABLE_INTERRUPTS(CLBR_NONE)
> TRACE_IRQS_OFF
Hi Steve
I tried the above patch but similar oops again
BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff80292349>] probe_wakeup_sched_switch+0x11f/0x1e8
PGD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:01/0000:01:01.1/irq
CPU 3
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
Pid: 16589, comm: sshd Not tainted 2.6.29-tip-test #3 eserver xSeries 366-[88632RA]-
RIP: 0010:[<ffffffff80292349>] [<ffffffff80292349>] probe_wakeup_sched_switch+0x11f/0x1e8
RSP: 0018:ffff8801da1b5e90 EFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000046
RDX: 0000000000000000 RSI: ffffffff8020bf85 RDI: ffffffff80d6f460
RBP: ffff8801da1b5ed0 R08: 0000000000000000 R09: 0000000100000003
R10: ffff8801da1b5ed0 R11: ffff88022d152078 R12: 0000000000000046
R13: ffff88022f352040 R14: 0000000000000000 R15: 0000000000000003
FS: 00007f748364d710(0000) GS:ffff880028155000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000001cfd8e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: ffffffff80d91980 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Process sshd (pid: 16589, threadinfo ffff8801da1b4000, task ffff88022d152040)
Stack:
ffff88022d152040 ffff88022d152040 ffff880028162960 ffff880224d79810
ffff880028167d00 00007fff8b6c7190 0000000000000005 00007fff8b6c7190
ffff8801da1b5f70 ffffffff805210b7 ffff8802295b8558 0000000000000001
Call Trace:
[<ffffffff805210b7>] schedule+0x82f/0xb39
[<ffffffff802d95a4>] ? sys_write+0x72/0x8d
[<ffffffff8020bf85>] sysret_careful+0xd/0x10
Code: d2 48 c7 c7 38 ef 73 80 89 f3 e8 83 20 00 00 85 db 75 6f 48 8b 4d 00 48 8b 3d 54 ed af 00 45 89 f0 48 8b 01 48 8b 71 08 4c 89 e1 <48> 8b 50 08 e8 5f a1 ff ff 48 8b 3d 37 ed af 00 48 8b 75 c8 45
RIP [<ffffffff80292349>] probe_wakeup_sched_switch+0x11f/0x1e8
RSP <ffff8801da1b5e90>
CR2: 0000000000000008
<probe_wakeup_sched_switch>:
:
:
ffffffff8029195f: 75 6f jne ffffffff802919d0 <probe_wakeup_sched_switch+0x176>
ffffffff80291961: 48 8b 4d 00 mov 0x0(%rbp),%rcx
ffffffff80291965: 48 8b 3d 24 f7 af 00 mov 11532068(%rip),%rdi # ffffffff80d91090 <wakeup_trace>
ffffffff8029196c: 45 89 f0 mov %r14d,%r8d
ffffffff8029196f: 48 8b 01 mov (%rcx),%rax
ffffffff80291972: 48 8b 71 08 mov 0x8(%rcx),%rsi
ffffffff80291976: 4c 89 e1 mov %r12,%rcx
ffffffff80291979: 48 8b 50 08 mov 0x8(%rax),%rdx
ffffffff8029197d: e8 af a7 ff ff callq ffffffff8028c131 <trace_function>
ffffffff80291982: 48 8b 3d 07 f7 af 00 mov 11532039(%rip),%rdi # ffffffff80d91090 <wakeup_trace>
ffffffff80291989: 48 8b 75 c8 mov 0xffffffffffffffc8(%rbp),%rsi
ffffffff8029198d: 45 89 f0 mov %r14d,%r8d
:
:
ffffffff8020c93c <retint_careful>:
ffffffff8020c93c: 0f ba e2 03 bt $0x3,%edx
ffffffff8020c940: 73 29 jae ffffffff8020c96b <retint_signal>
ffffffff8020c942: fb sti
ffffffff8020c943: 57 push %rdi
ffffffff8020c944: 55 push %rbp
ffffffff8020c945: e8 00 00 00 00 callq ffffffff8020c94a <retint_careful+0xe>
ffffffff8020c94a: 48 89 e5 mov %rsp,%rbp
ffffffff8020c94d: e8 d6 34 31 00 callq ffffffff8051fe28 <schedule>
ffffffff8020c952: 48 83 c4 08 add $0x8,%rsp
ffffffff8020c956: 5d pop %rbp
ffffffff8020c957: 5f pop %rdi
ffffffff8020c958: 65 48 8b 0c 25 e8 b4 mov %gs:0xb4e8,%rcx
ffffffff8020c95f: 00 00
ffffffff8020c961: 48 81 e9 d8 1f 00 00 sub $0x1fd8,%rcx
ffffffff8020c968: fa cli
ffffffff8020c969: eb 91 jmp ffffffff8020c8fc <retint_check>
--
Maneesh Soni
Linux Technology Center
IBM India Systems and Technology Lab,
Bangalore, India.
On Thu, 2 Apr 2009, Maneesh Soni wrote:
> On Wed, Apr 01, 2009 at 07:42:58PM -0400, Steven Rostedt wrote:
>
> ....
>
> >
> >
> >
> > Hi Maneesh,
> >
> > Could you try this patch and see if it keeps your system from crashing?
> >
> > Thanks,
> >
> > -- Steve
> >
> > diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
> > index a331ec3..dbf3f8f 100644
> > --- a/arch/x86/kernel/entry_64.S
> > +++ b/arch/x86/kernel/entry_64.S
> > @@ -917,10 +917,15 @@ retint_careful:
> > TRACE_IRQS_ON
> > ENABLE_INTERRUPTS(CLBR_NONE)
> > pushq %rdi
> > - CFI_ADJUST_CFA_OFFSET 8
> > + pushq %rbp
> > + call 1f
> > +1: mov %rsp, %rbp
> > + CFI_ADJUST_CFA_OFFSET 24
> > call schedule
> > + addq $8, %rsp /* skip call */
> > + popq %rbp
> > popq %rdi
> > - CFI_ADJUST_CFA_OFFSET -8
> > + CFI_ADJUST_CFA_OFFSET -24
> > GET_THREAD_INFO(%rcx)
> > DISABLE_INTERRUPTS(CLBR_NONE)
> > TRACE_IRQS_OFF
>
>
>
> Hi Steve
>
> I tried the above patch but similar oops again
Thanks, that helps a lot.
>
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> IP: [<ffffffff80292349>] probe_wakeup_sched_switch+0x11f/0x1e8
> PGD 0
> Oops: 0000 [#1] SMP
> last sysfs file: /sys/devices/pci0000:01/0000:01:01.1/irq
> CPU 3
> Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
> Pid: 16589, comm: sshd Not tainted 2.6.29-tip-test #3 eserver xSeries 366-[88632RA]-
> RIP: 0010:[<ffffffff80292349>] [<ffffffff80292349>] probe_wakeup_sched_switch+0x11f/0x1e8
> RSP: 0018:ffff8801da1b5e90 EFLAGS: 00010046
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000046
> RDX: 0000000000000000 RSI: ffffffff8020bf85 RDI: ffffffff80d6f460
> RBP: ffff8801da1b5ed0 R08: 0000000000000000 R09: 0000000100000003
> R10: ffff8801da1b5ed0 R11: ffff88022d152078 R12: 0000000000000046
> R13: ffff88022f352040 R14: 0000000000000000 R15: 0000000000000003
> FS: 00007f748364d710(0000) GS:ffff880028155000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000008 CR3: 00000001cfd8e000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: ffffffff80d91980 DR6: 00000000ffff0ff0 DR7: 0000000000000600
> Process sshd (pid: 16589, threadinfo ffff8801da1b4000, task ffff88022d152040)
> Stack:
> ffff88022d152040 ffff88022d152040 ffff880028162960 ffff880224d79810
> ffff880028167d00 00007fff8b6c7190 0000000000000005 00007fff8b6c7190
> ffff8801da1b5f70 ffffffff805210b7 ffff8802295b8558 0000000000000001
> Call Trace:
> [<ffffffff805210b7>] schedule+0x82f/0xb39
> [<ffffffff802d95a4>] ? sys_write+0x72/0x8d
> [<ffffffff8020bf85>] sysret_careful+0xd/0x10
This is what I was afraid of. Your other crashes were intret_careful,
now we are hitting sysret_careful. I'm going to pull out all references to
CALLER_ADDR2. The above patch was simply me manually putting in a call
frame in intret_careful. But this is unreliable, any caller from an
interrupt (or syscall) to schedule will cause an error. I'm not sure we
need the CALLER_ADDR2 anyway.
Thanks!
-- Steve
On Thu, Apr 02, 2009 at 09:18:35AM -0400, Steven Rostedt wrote:
[..]
>
> >
> > BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> > IP: [<ffffffff80292349>] probe_wakeup_sched_switch+0x11f/0x1e8
> > PGD 0
> > Oops: 0000 [#1] SMP
> > last sysfs file: /sys/devices/pci0000:01/0000:01:01.1/irq
> > CPU 3
> > Modules linked in: autofs4 hidp rfcomm l2cap bluetooth iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw acpi_memhotplug button tg3 libphy i2c_piix4 i2c_core pcspkr usb_storage uhci_hcd ohci_hcd ehci_hcd aacraid sd_mod scsi_mod ext3 jbd
> > Pid: 16589, comm: sshd Not tainted 2.6.29-tip-test #3 eserver xSeries 366-[88632RA]-
> > RIP: 0010:[<ffffffff80292349>] [<ffffffff80292349>] probe_wakeup_sched_switch+0x11f/0x1e8
> > RSP: 0018:ffff8801da1b5e90 EFLAGS: 00010046
> > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000046
> > RDX: 0000000000000000 RSI: ffffffff8020bf85 RDI: ffffffff80d6f460
> > RBP: ffff8801da1b5ed0 R08: 0000000000000000 R09: 0000000100000003
> > R10: ffff8801da1b5ed0 R11: ffff88022d152078 R12: 0000000000000046
> > R13: ffff88022f352040 R14: 0000000000000000 R15: 0000000000000003
> > FS: 00007f748364d710(0000) GS:ffff880028155000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 0000000000000008 CR3: 00000001cfd8e000 CR4: 00000000000006e0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: ffffffff80d91980 DR6: 00000000ffff0ff0 DR7: 0000000000000600
> > Process sshd (pid: 16589, threadinfo ffff8801da1b4000, task ffff88022d152040)
> > Stack:
> > ffff88022d152040 ffff88022d152040 ffff880028162960 ffff880224d79810
> > ffff880028167d00 00007fff8b6c7190 0000000000000005 00007fff8b6c7190
> > ffff8801da1b5f70 ffffffff805210b7 ffff8802295b8558 0000000000000001
> > Call Trace:
> > [<ffffffff805210b7>] schedule+0x82f/0xb39
> > [<ffffffff802d95a4>] ? sys_write+0x72/0x8d
> > [<ffffffff8020bf85>] sysret_careful+0xd/0x10
>
> This is what I was afraid of. Your other crashes were intret_careful,
> now we are hitting sysret_careful. I'm going to pull out all references to
> CALLER_ADDR2. The above patch was simply me manually putting in a call
> frame in intret_careful. But this is unreliable, any caller from an
> interrupt (or syscall) to schedule will cause an error. I'm not sure we
> need the CALLER_ADDR2 anyway.
>
I had also tested with passing 0 instead of CALLER_ADDR2 and didn't see any crash.
Thanks
Maneesh
--
Maneesh Soni
Linux Technology Center
IBM India Systems and Technology Lab,
Bangalore, India.