LinuxLists.cc - Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Quoting Dhaval Giani ([email protected]):
> Hi,
>
> On mounting cpusets using containers, I have been hitting the following
> bug.
>
>
> -----------[ cut here ]------------
> kernel BUG at kernel/cpuset.c:331!
> invalid opcode: 0000 [#1] SMP
> Modules linked in:
>
> Pid: 4293, comm: mkdir Not tainted (2.6.23-rc2-mm2 #3)
> EIP: 0060:[<c0147daa>] EFLAGS: 00010246 CPU: 7
> EIP is at guarantee_online_mems+0x37/0x3e
> EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: ca64bf48
> ESI: c05374a0 EDI: ca940000 EBP: ca54becc ESP: ca54bec8
> DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
> Process mkdir (pid: 4293, ti=ca54a000 task=ca64b510 task.ti=ca54a000)
> Stack: ca64b510 ca54bedc c0147e0d ca96f740 c05374a0 ca54beec c01488dc c05378e0
> cb0a1680 ca54bf10 c01470a6 cb495800 00000000 000041ed caf73734 c98fe488
> c01471b5 000001ed ca54bf18 c01471c6 ca54bf30 c0179f56 caf73734 caf73734
> Call Trace:
> [<c0105b08>] show_trace_log_lvl+0x12/0x22
> [<c0105bae>] show_stack_log_lvl+0x87/0x96
> [<c0105c97>] show_registers+0x9a/0x15f
> [<c0105e8d>] die+0xec/0x1c3
> [<c03eec14>] do_trap+0x8e/0xa8
> [<c01060d4>] do_invalid_op+0x6c/0x73
> [<c03ee9e2>] error_code+0x72/0x78
> [<c0147e0d>] cpuset_update_task_memory_state+0x5c/0xb0
> [<c01488dc>] cpuset_create+0x49/0xa0
> [<c01470a6>] container_create+0x9e/0x1ad
> [<c01471c6>] container_mkdir+0x11/0x13
> [<c0179f56>] vfs_mkdir+0x5b/0xbd
> [<c017a03e>] sys_mkdirat+0x86/0xba
> [<c017a082>] sys_mkdir+0x10/0x12
> [<c0104cfa>] sysenter_past_esp+0x5f/0x99
> =======================
> INFO: lockdep is turned off.
> Code: eb 03 8b 49 18 85 c9 74 09 8b 41 14 21 d8 a8 01 74 f0 85 c9 a1 80 7d 59 c0 74 03 23 41 14 89 02 a1 80 7d 59 c0 23 02 a8 01 75 04 <0f> 0b eb fe 5b 5d c3 55 89 e5 56 53 64 8b 1d 00 40 5d c0 8b 93
> EIP: [<c0147daa>] guarantee_online_mems+0x37/0x3e SS:ESP 0068:ca54bec8
> BUG: soft lockup - CPU#7 stuck for 11s! [mkdir:4293]
>
> Pid: 4293, comm: mkdir Tainted: G D (2.6.23-rc2-mm2 #3)
> EIP: 0060:[<c023916d>] EFLAGS: 00000246 CPU: 7
> EIP is at delay_tsc+0x8/0x17
> EAX: dc5d84ee EBX: 00000001 ECX: dc5d83a2 EDX: 000005e6
> ESI: 00000000 EDI: ca64b9e0 EBP: ca54bd34 ESP: ca54bd30
> DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> CR0: 8005003b CR2: 4a419520 CR3: 0b9e5000 CR4: 000006f0
> DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> DR6: ffff0ff0 DR7: 00000400
> [<c0105b08>] show_trace_log_lvl+0x12/0x22
> [<c0105b25>] show_trace+0xd/0xf
> [<c010342e>] show_regs+0x1c/0x1f
> [<c0150fe6>] softlockup_tick+0xf2/0x10b
> [<c01297ee>] run_local_timers+0x12/0x14
> [<c012964e>] update_process_times+0x3e/0x63
> [<c0138d41>] tick_periodic+0x66/0x72
> [<c0138d64>] tick_handle_periodic+0x17/0x65
> [<c0114408>] local_apic_timer_interrupt+0x47/0x4c
> [<c0114437>] smp_apic_timer_interrupt+0x2a/0x39
> [<c010584f>] apic_timer_interrupt+0x33/0x38
> [<c02391b0>] __delay+0x9/0xb
> [<c023b257>] __spin_lock_debug+0x46/0x9b
> [<c023b302>] _raw_spin_lock+0x56/0x6b
> [<c03ee5c5>] _spin_lock+0x20/0x25
> [<c0123988>] exit_mm+0x90/0xd4
> [<c01240b9>] do_exit+0x1ed/0x39e
> [<c0105f5c>] die+0x1bb/0x1c3
> [<c03eec14>] do_trap+0x8e/0xa8
> [<c01060d4>] do_invalid_op+0x6c/0x73
> [<c03ee9e2>] error_code+0x72/0x78
> [<c0147e0d>] cpuset_update_task_memory_state+0x5c/0xb0
> [<c01488dc>] cpuset_create+0x49/0xa0
> [<c01470a6>] container_create+0x9e/0x1ad
> [<c01471c6>] container_mkdir+0x11/0x13
> [<c0179f56>] vfs_mkdir+0x5b/0xbd
> [<c017a03e>] sys_mkdirat+0x86/0xba
> [<c017a082>] sys_mkdir+0x10/0x12
> [<c0104cfa>] sysenter_past_esp+0x5f/0x99
> =======================
> BUG: soft lockup - CPU#7 stuck for 11s! [mkdir:4293]
>
> Pid: 4293, comm: mkdir Tainted: G D (2.6.23-rc2-mm2 #3)
> EIP: 0060:[<c023b249>] EFLAGS: 00000246 CPU: 7
> EIP is at __spin_lock_debug+0x38/0x9b
> EAX: 00000000 EBX: 09f4b5fd ECX: cf8afa92 EDX: 000005ed
> ESI: 00000000 EDI: ca64b9e0 EBP: ca54bd5c ESP: ca54bd44
> DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> CR0: 8005003b CR2: 4a419520 CR3: 0b9e5000 CR4: 000006f0
> DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> DR6: ffff0ff0 DR7: 00000400
> [<c0105b08>] show_trace_log_lvl+0x12/0x22
> [<c0105b25>] show_trace+0xd/0xf
> [<c010342e>] show_regs+0x1c/0x1f
> [<c0150fe6>] softlockup_tick+0xf2/0x10b
> [<c01297ee>] run_local_timers+0x12/0x14
> [<c012964e>] update_process_times+0x3e/0x63
> [<c0138d41>] tick_periodic+0x66/0x72
> [<c0138d64>] tick_handle_periodic+0x17/0x65
> [<c0114408>] local_apic_timer_interrupt+0x47/0x4c
> [<c0114437>] smp_apic_timer_interrupt+0x2a/0x39
> [<c010584f>] apic_timer_interrupt+0x33/0x38
> [<c023b302>] _raw_spin_lock+0x56/0x6b
> [<c03ee5c5>] _spin_lock+0x20/0x25
> [<c0123988>] exit_mm+0x90/0xd4
> [<c01240b9>] do_exit+0x1ed/0x39e
> [<c0105f5c>] die+0x1bb/0x1c3
> [<c03eec14>] do_trap+0x8e/0xa8
> [<c01060d4>] do_invalid_op+0x6c/0x73
> [<c03ee9e2>] error_code+0x72/0x78
> [<c0147e0d>] cpuset_update_task_memory_state+0x5c/0xb0
> [<c01488dc>] cpuset_create+0x49/0xa0
> [<c01470a6>] container_create+0x9e/0x1ad
> [<c01471c6>] container_mkdir+0x11/0x13
> [<c0179f56>] vfs_mkdir+0x5b/0xbd
> [<c017a03e>] sys_mkdirat+0x86/0xba
> [<c017a082>] sys_mkdir+0x10/0x12
> [<c0104cfa>] sysenter_past_esp+0x5f/0x99
> =======================
> BUG: soft lockup - CPU#7 stuck for 11s! [mkdir:4293]
>
> Pid: 4293, comm: mkdir Tainted: G D (2.6.23-rc2-mm2 #3)
> EIP: 0060:[<c023b249>] EFLAGS: 00000246 CPU: 7
> EIP is at __spin_lock_debug+0x38/0x9b
> EAX: 00000000 EBX: 0f1ef797 ECX: c48010c2 EDX: 000005f4
> ESI: 00000000 EDI: ca64b9e0 EBP: ca54bd5c ESP: ca54bd44
> DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> CR0: 8005003b CR2: 4a419520 CR3: 0b9e5000 CR4: 000006f0
> DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> DR6: ffff0ff0 DR7: 00000400
> [<c0105b08>] show_trace_log_lvl+0x12/0x22
> [<c0105b25>] show_trace+0xd/0xf
> [<c010342e>] show_regs+0x1c/0x1f
> [<c0150fe6>] softlockup_tick+0xf2/0x10b
> [<c01297ee>] run_local_timers+0x12/0x14
> [<c012964e>] update_process_times+0x3e/0x63
> [<c0138d41>] tick_periodic+0x66/0x72
> [<c0138d64>] tick_handle_periodic+0x17/0x65
> [<c0114408>] local_apic_timer_interrupt+0x47/0x4c
> [<c0114437>] smp_apic_timer_interrupt+0x2a/0x39
> [<c010584f>] apic_timer_interrupt+0x33/0x38
> [<c023b302>] _raw_spin_lock+0x56/0x6b
> [<c03ee5c5>] _spin_lock+0x20/0x25
> [<c0123988>] exit_mm+0x90/0xd4
> [<c01240b9>] do_exit+0x1ed/0x39e
> [<c0105f5c>] die+0x1bb/0x1c3
> [<c03eec14>] do_trap+0x8e/0xa8
> [<c01060d4>] do_invalid_op+0x6c/0x73
> [<c03ee9e2>] error_code+0x72/0x78
> [<c0147e0d>] cpuset_update_task_memory_state+0x5c/0xb0
> [<c01488dc>] cpuset_create+0x49/0xa0
> [<c01470a6>] container_create+0x9e/0x1ad
> [<c01471c6>] container_mkdir+0x11/0x13
> [<c0179f56>] vfs_mkdir+0x5b/0xbd
> [<c017a03e>] sys_mkdirat+0x86/0xba
> [<c017a082>] sys_mkdir+0x10/0x12
> [<c0104cfa>] sysenter_past_esp+0x5f/0x99
> =======================
> BUG: soft lockup - CPU#7 stuck for 11s! [mkdir:4293]
>
> Pid: 4293, comm: mkdir Tainted: G D (2.6.23-rc2-mm2 #3)
> EIP: 0060:[<c023916d>] EFLAGS: 00000246 CPU: 7
> EIP is at delay_tsc+0x8/0x17
> EAX: b975273a EBX: 00000001 ECX: b97525ee EDX: 000005fb
> ESI: 00000000 EDI: ca64b9e0 EBP: ca54bd34 ESP: ca54bd30
> DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> CR0: 8005003b CR2: 4a419520 CR3: 0b9e5000 CR4: 000006f0
> DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> DR6: ffff0ff0 DR7: 00000400
> [<c0105b08>] show_trace_log_lvl+0x12/0x22
> [<c0105b25>] show_trace+0xd/0xf
> [<c010342e>] show_regs+0x1c/0x1f
> [<c0150fe6>] softlockup_tick+0xf2/0x10b
> [<c01297ee>] run_local_timers+0x12/0x14
> [<c012964e>] update_process_times+0x3e/0x63
> [<c0138d41>] tick_periodic+0x66/0x72
> [<c0138d64>] tick_handle_periodic+0x17/0x65
> [<c0114408>] local_apic_timer_interrupt+0x47/0x4c
> [<c0114437>] smp_apic_timer_interrupt+0x2a/0x39
> [<c010584f>] apic_timer_interrupt+0x33/0x38
> [<c02391b0>] __delay+0x9/0xb
> [<c023b257>] __spin_lock_debug+0x46/0x9b
> [<c023b302>] _raw_spin_lock+0x56/0x6b
> [<c03ee5c5>] _spin_lock+0x20/0x25
> [<c0123988>] exit_mm+0x90/0xd4
> [<c01240b9>] do_exit+0x1ed/0x39e
> [<c0105f5c>] die+0x1bb/0x1c3
> [<c03eec14>] do_trap+0x8e/0xa8
> [<c01060d4>] do_invalid_op+0x6c/0x73
> [<c03ee9e2>] error_code+0x72/0x78
> [<c0147e0d>] cpuset_update_task_memory_state+0x5c/0xb0
> [<c01488dc>] cpuset_create+0x49/0xa0
> [<c01470a6>] container_create+0x9e/0x1ad
> [<c01471c6>] container_mkdir+0x11/0x13
> [<c0179f56>] vfs_mkdir+0x5b/0xbd
> [<c017a03e>] sys_mkdirat+0x86/0xba
> [<c017a082>] sys_mkdir+0x10/0x12
> [<c0104cfa>] sysenter_past_esp+0x5f/0x99
> =======================
>
> On reverting memoryless-nodes-use-n_high_memory-for-cpusets.patch, I no
> longer hit this bug.
>
> .config follows
>
> #
> # Automatically generated make config: don't edit
> # Linux kernel version: 2.6.23-rc2-mm2
> # Sat Aug 11 18:32:23 2007
> #
> CONFIG_X86_32=y
> CONFIG_GENERIC_TIME=y
> CONFIG_GENERIC_CMOS_UPDATE=y
> CONFIG_CLOCKSOURCE_WATCHDOG=y
> CONFIG_GENERIC_CLOCKEVENTS=y
> CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
> CONFIG_NONIRQ_WAKEUP=y
> CONFIG_LOCKDEP_SUPPORT=y
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_SEMAPHORE_SLEEPERS=y
> CONFIG_X86=y
> CONFIG_MMU=y
> CONFIG_ZONE_DMA=y
> CONFIG_QUICKLIST=y
> CONFIG_GENERIC_ISA_DMA=y
> CONFIG_GENERIC_IOMAP=y
> CONFIG_GENERIC_BUG=y
> CONFIG_GENERIC_HWEIGHT=y
> CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> CONFIG_DMI=y
> CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
>
> #
> # General setup
> #
> CONFIG_EXPERIMENTAL=y
> CONFIG_LOCK_KERNEL=y
> CONFIG_INIT_ENV_ARG_LIMIT=32
> CONFIG_LOCALVERSION=""
> CONFIG_LOCALVERSION_AUTO=y
> CONFIG_SWAP=y
> CONFIG_SWAP_PREFETCH=y
> CONFIG_SYSVIPC=y
> CONFIG_SYSVIPC_SYSCTL=y
> CONFIG_POSIX_MQUEUE=y
> # CONFIG_BSD_PROCESS_ACCT is not set
> # CONFIG_TASKSTATS is not set
> # CONFIG_USER_NS is not set
> CONFIG_AUDIT=y
> CONFIG_AUDITSYSCALL=y
> CONFIG_IKCONFIG=y
> CONFIG_IKCONFIG_PROC=y
> CONFIG_LOG_BUF_SHIFT=15
> CONFIG_CONTAINERS=y
> # CONFIG_CONTAINER_DEBUG is not set
> CONFIG_CONTAINER_NS=y
> # CONFIG_CONTAINER_CPUACCT is not set
> CONFIG_CPUSETS=y
> CONFIG_SYSFS_DEPRECATED=y
> CONFIG_PROC_PID_CPUSET=y
> # CONFIG_RELAY is not set
> CONFIG_BLK_DEV_INITRD=y
> CONFIG_INITRAMFS_SOURCE=""
> CONFIG_CC_OPTIMIZE_FOR_SIZE=y
> CONFIG_SYSCTL=y
> # CONFIG_EMBEDDED is not set
> CONFIG_UID16=y
> CONFIG_SYSCTL_SYSCALL=y
> CONFIG_KALLSYMS=y
> CONFIG_KALLSYMS_ALL=y
> # CONFIG_KALLSYMS_EXTRA_PASS is not set
> CONFIG_HOTPLUG=y
> CONFIG_PRINTK=y
> CONFIG_BUG=y
> CONFIG_ELF_CORE=y
> CONFIG_BASE_FULL=y
> CONFIG_FUTEX=y
> CONFIG_ANON_INODES=y
> CONFIG_EPOLL=y
> CONFIG_SIGNALFD=y
> CONFIG_TIMERFD=y
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_VM_EVENT_COUNTERS=y
> CONFIG_SLAB=y
> # CONFIG_SLUB is not set
> # CONFIG_SLOB is not set
> CONFIG_PROC_PAGE_MONITOR=y
> CONFIG_PROC_KPAGEMAP=y
> CONFIG_RT_MUTEXES=y
> # CONFIG_TINY_SHMEM is not set
> CONFIG_BASE_SMALL=0
> CONFIG_MODULES=y
> CONFIG_MODULE_UNLOAD=y
> CONFIG_MODULE_FORCE_UNLOAD=y
> # CONFIG_MODVERSIONS is not set
> # CONFIG_MODULE_SRCVERSION_ALL is not set
> # CONFIG_KMOD is not set
> CONFIG_STOP_MACHINE=y
> CONFIG_BLOCK=y
> CONFIG_LBD=y
> # CONFIG_BLK_DEV_IO_TRACE is not set
> # CONFIG_LSF is not set
> # CONFIG_BLK_DEV_BSG is not set
>
> #
> # IO Schedulers
> #
> CONFIG_IOSCHED_NOOP=y
> CONFIG_IOSCHED_AS=y
> CONFIG_IOSCHED_DEADLINE=y
> CONFIG_IOSCHED_CFQ=y
> CONFIG_DEFAULT_AS=y
> # CONFIG_DEFAULT_DEADLINE is not set
> # CONFIG_DEFAULT_CFQ is not set
> # CONFIG_DEFAULT_NOOP is not set
> CONFIG_DEFAULT_IOSCHED="anticipatory"
>
> #
> # Processor type and features
> #
> # CONFIG_TICK_ONESHOT is not set
> # CONFIG_NO_HZ is not set
> # CONFIG_HIGH_RES_TIMERS is not set
> CONFIG_SMP=y
> # CONFIG_X86_PC is not set
> # CONFIG_X86_ELAN is not set
> # CONFIG_X86_VOYAGER is not set
> # CONFIG_X86_NUMAQ is not set
> # CONFIG_X86_SUMMIT is not set
> # CONFIG_X86_BIGSMP is not set
> # CONFIG_X86_VISWS is not set
> CONFIG_X86_GENERICARCH=y
> # CONFIG_X86_ES7000 is not set
> # CONFIG_PARAVIRT is not set
> CONFIG_X86_CYCLONE_TIMER=y
> # CONFIG_M386 is not set
> # CONFIG_M486 is not set
> # CONFIG_M586 is not set
> # CONFIG_M586TSC is not set
> # CONFIG_M586MMX is not set
> # CONFIG_M686 is not set
> # CONFIG_MPENTIUMII is not set
> # CONFIG_MPENTIUMIII is not set
> # CONFIG_MPENTIUMM is not set
> CONFIG_MPENTIUM4=y
> # CONFIG_MCORE2 is not set
> # CONFIG_MK6 is not set
> # CONFIG_MK7 is not set
> # CONFIG_MK8 is not set
> # CONFIG_MCRUSOE is not set
> # CONFIG_MEFFICEON is not set
> # CONFIG_MWINCHIPC6 is not set
> # CONFIG_MWINCHIP2 is not set
> # CONFIG_MWINCHIP3D is not set
> # CONFIG_MGEODEGX1 is not set
> # CONFIG_MGEODE_LX is not set
> # CONFIG_MCYRIXIII is not set
> # CONFIG_MVIAC3_2 is not set
> # CONFIG_MVIAC7 is not set
> CONFIG_X86_GENERIC=y
> CONFIG_X86_CMPXCHG=y
> CONFIG_X86_L1_CACHE_SHIFT=7
> CONFIG_X86_XADD=y
> CONFIG_RWSEM_XCHGADD_ALGORITHM=y
> # CONFIG_ARCH_HAS_ILOG2_U32 is not set
> # CONFIG_ARCH_HAS_ILOG2_U64 is not set
> CONFIG_GENERIC_CALIBRATE_DELAY=y
> CONFIG_X86_WP_WORKS_OK=y
> CONFIG_X86_INVLPG=y
> CONFIG_X86_BSWAP=y
> CONFIG_X86_POPAD_OK=y
> CONFIG_X86_GOOD_APIC=y
> CONFIG_X86_INTEL_USERCOPY=y
> CONFIG_X86_USE_PPRO_CHECKSUM=y
> CONFIG_X86_TSC=y
> CONFIG_X86_CMOV=y
> CONFIG_X86_MINIMUM_CPU_FAMILY=4
> CONFIG_HPET_TIMER=y
> CONFIG_HPET_EMULATE_RTC=y
> CONFIG_NR_CPUS=32
> CONFIG_SCHED_SMT=y
> CONFIG_SCHED_MC=y
> # CONFIG_PREEMPT_NONE is not set
> CONFIG_PREEMPT_VOLUNTARY=y
> # CONFIG_PREEMPT is not set
> CONFIG_PREEMPT_BKL=y
> CONFIG_X86_LOCAL_APIC=y
> CONFIG_X86_IO_APIC=y
> CONFIG_X86_MCE=y
> CONFIG_X86_MCE_NONFATAL=y
> CONFIG_X86_MCE_P4THERMAL=y
> CONFIG_VM86=y
> # CONFIG_TOSHIBA is not set
> # CONFIG_I8K is not set
> # CONFIG_X86_REBOOTFIXUPS is not set
> CONFIG_MICROCODE=y
> CONFIG_MICROCODE_OLD_INTERFACE=y
> CONFIG_X86_MSR=y
> CONFIG_X86_CPUID=y
>
> #
> # Firmware Drivers
> #
> # CONFIG_EDD is not set
> # CONFIG_DELL_RBU is not set
> # CONFIG_DCDBAS is not set
> CONFIG_DMIID=y
> # CONFIG_NOHIGHMEM is not set
> # CONFIG_HIGHMEM4G is not set
> CONFIG_HIGHMEM64G=y
> CONFIG_PAGE_OFFSET=0xC0000000
> CONFIG_HIGHMEM=y
> CONFIG_X86_PAE=y
> # CONFIG_NUMA is not set
> CONFIG_ARCH_POPULATES_NODE_MAP=y
> CONFIG_SELECT_MEMORY_MODEL=y
> CONFIG_FLATMEM_MANUAL=y
> # CONFIG_DISCONTIGMEM_MANUAL is not set
> # CONFIG_SPARSEMEM_MANUAL is not set
> CONFIG_FLATMEM=y
> CONFIG_FLAT_NODE_MEM_MAP=y
> # CONFIG_SPARSEMEM_STATIC is not set
> CONFIG_SPLIT_PTLOCK_CPUS=4
> CONFIG_RESOURCES_64BIT=y
> CONFIG_ZONE_DMA_FLAG=1
> CONFIG_BOUNCE=y
> CONFIG_NR_QUICK=1
> CONFIG_VIRT_TO_BUS=y
> # CONFIG_HIGHPTE is not set
> # CONFIG_MATH_EMULATION is not set
> CONFIG_MTRR=y
> # CONFIG_EFI is not set
> # CONFIG_IRQBALANCE is not set
> CONFIG_SECCOMP=y
> # CONFIG_HZ_100 is not set
> CONFIG_HZ_250=y
> # CONFIG_HZ_300 is not set
> # CONFIG_HZ_1000 is not set
> CONFIG_HZ=250
> CONFIG_KEXEC=y
> # CONFIG_CRASH_DUMP is not set
> CONFIG_PHYSICAL_START=0x100000
> # CONFIG_RELOCATABLE is not set
> CONFIG_PHYSICAL_ALIGN=0x100000
> CONFIG_HOTPLUG_CPU=y
> CONFIG_COMPAT_VDSO=y
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
>
> #
> # Power management options (ACPI, APM)
> #
> CONFIG_PM=y
> CONFIG_PM_LEGACY=y
> CONFIG_PM_DEBUG=y
> # CONFIG_PM_VERBOSE is not set
> CONFIG_SUSPEND_SMP_POSSIBLE=y
> # CONFIG_SUSPEND is not set
> CONFIG_HIBERNATION_SMP_POSSIBLE=y
> # CONFIG_HIBERNATION is not set
> CONFIG_ACPI=y
> # CONFIG_ACPI_PROCFS is not set
> CONFIG_ACPI_AC=y
> CONFIG_ACPI_BATTERY=y
> CONFIG_ACPI_BUTTON=y
> CONFIG_ACPI_FAN=y
> # CONFIG_ACPI_DOCK is not set
> CONFIG_ACPI_PROCESSOR=y
> CONFIG_ACPI_HOTPLUG_CPU=y
> CONFIG_ACPI_THERMAL=y
> # CONFIG_ACPI_ASUS is not set
> # CONFIG_ACPI_TOSHIBA is not set
> CONFIG_ACPI_BLACKLIST_YEAR=2001
> CONFIG_ACPI_DEBUG=y
> # CONFIG_ACPI_DEBUG_FUNC_TRACE is not set
> CONFIG_ACPI_EC=y
> CONFIG_ACPI_POWER=y
> CONFIG_ACPI_SYSTEM=y
> CONFIG_X86_PM_TIMER=y
> CONFIG_ACPI_CONTAINER=y
> # CONFIG_ACPI_SBS is not set
>
> #
> # CPU Frequency scaling
> #
> CONFIG_CPU_FREQ=y
> CONFIG_CPU_FREQ_TABLE=y
> CONFIG_CPU_FREQ_DEBUG=y
> CONFIG_CPU_FREQ_STAT=y
> CONFIG_CPU_FREQ_STAT_DETAILS=y
> CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
> # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
> # CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
> # CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
> CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> CONFIG_CPU_FREQ_GOV_POWERSAVE=y
> CONFIG_CPU_FREQ_GOV_USERSPACE=y
> CONFIG_CPU_FREQ_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
>
> #
> # CPUFreq processor drivers
> #
> # CONFIG_X86_ACPI_CPUFREQ is not set
> # CONFIG_X86_POWERNOW_K6 is not set
> # CONFIG_X86_POWERNOW_K7 is not set
> # CONFIG_X86_POWERNOW_K8 is not set
> # CONFIG_X86_GX_SUSPMOD is not set
> # CONFIG_X86_SPEEDSTEP_CENTRINO is not set
> # CONFIG_X86_SPEEDSTEP_ICH is not set
> # CONFIG_X86_SPEEDSTEP_SMI is not set
> CONFIG_X86_P4_CLOCKMOD=y
> # CONFIG_X86_CPUFREQ_NFORCE2 is not set
> # CONFIG_X86_LONGRUN is not set
> # CONFIG_X86_LONGHAUL is not set
> # CONFIG_X86_E_POWERSAVER is not set
>
> #
> # shared options
> #
> CONFIG_X86_SPEEDSTEP_LIB=y
>
> #
> # CPU idle PM support
> #
> # CONFIG_CPU_IDLE is not set
>
> #
> # Bus options (PCI, PCMCIA, EISA, MCA, ISA)
> #
> CONFIG_PCI=y
> # CONFIG_PCI_GOBIOS is not set
> # CONFIG_PCI_GOMMCONFIG is not set
> # CONFIG_PCI_GODIRECT is not set
> CONFIG_PCI_GOANY=y
> CONFIG_PCI_BIOS=y
> CONFIG_PCI_DIRECT=y
> CONFIG_PCI_MMCONFIG=y
> # CONFIG_PCIEPORTBUS is not set
> CONFIG_ARCH_SUPPORTS_MSI=y
> CONFIG_PCI_MSI=y
> # CONFIG_PCI_DEBUG is not set
> CONFIG_HT_IRQ=y
> CONFIG_ISA_DMA_API=y
> # CONFIG_ISA is not set
> # CONFIG_MCA is not set
> # CONFIG_SCx200 is not set
> CONFIG_K8_NB=y
> # CONFIG_PCCARD is not set
> # CONFIG_HOTPLUG_PCI is not set
>
> #
> # Executable file formats
> #
> CONFIG_BINFMT_ELF=y
> # CONFIG_BINFMT_AOUT is not set
> # CONFIG_BINFMT_MISC is not set
>
> #
> # Networking
> #
> CONFIG_NET=y
>
> #
> # Networking options
> #
> CONFIG_PACKET=y
> # CONFIG_PACKET_MMAP is not set
> CONFIG_UNIX=y
> CONFIG_XFRM=y
> # CONFIG_XFRM_USER is not set
> # CONFIG_XFRM_SUB_POLICY is not set
> # CONFIG_XFRM_MIGRATE is not set
> # CONFIG_NET_KEY is not set
> CONFIG_INET=y
> CONFIG_IP_MULTICAST=y
> # CONFIG_IP_ADVANCED_ROUTER is not set
> CONFIG_IP_FIB_HASH=y
> CONFIG_IP_PNP=y
> CONFIG_IP_PNP_DHCP=y
> # CONFIG_IP_PNP_BOOTP is not set
> # CONFIG_IP_PNP_RARP is not set
> # CONFIG_NET_IPIP is not set
> # CONFIG_NET_IPGRE is not set
> # CONFIG_IP_MROUTE is not set
> # CONFIG_ARPD is not set
> # CONFIG_SYN_COOKIES is not set
> # CONFIG_INET_AH is not set
> # CONFIG_INET_ESP is not set
> # CONFIG_INET_IPCOMP is not set
> # CONFIG_INET_XFRM_TUNNEL is not set
> # CONFIG_INET_TUNNEL is not set
> CONFIG_INET_XFRM_MODE_TRANSPORT=y
> CONFIG_INET_XFRM_MODE_TUNNEL=y
> CONFIG_INET_XFRM_MODE_BEET=y
> CONFIG_INET_DIAG=y
> CONFIG_INET_TCP_DIAG=y
> # CONFIG_TCP_CONG_ADVANCED is not set
> CONFIG_TCP_CONG_CUBIC=y
> CONFIG_DEFAULT_TCP_CONG="cubic"
> # CONFIG_TCP_MD5SIG is not set
> CONFIG_IPV6=m
> # CONFIG_IPV6_PRIVACY is not set
> # CONFIG_IPV6_ROUTER_PREF is not set
> # CONFIG_IPV6_OPTIMISTIC_DAD is not set
> # CONFIG_INET6_AH is not set
> # CONFIG_INET6_ESP is not set
> # CONFIG_INET6_IPCOMP is not set
> # CONFIG_IPV6_MIP6 is not set
> # CONFIG_INET6_XFRM_TUNNEL is not set
> # CONFIG_INET6_TUNNEL is not set
> # CONFIG_INET6_XFRM_MODE_TRANSPORT is not set
> # CONFIG_INET6_XFRM_MODE_TUNNEL is not set
> # CONFIG_INET6_XFRM_MODE_BEET is not set
> # CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set
> # CONFIG_IPV6_SIT is not set
> # CONFIG_IPV6_TUNNEL is not set
> # CONFIG_IPV6_MULTIPLE_TABLES is not set
> # CONFIG_NETWORK_SECMARK is not set
> # CONFIG_NETFILTER is not set
> # CONFIG_IP_DCCP is not set
> # CONFIG_IP_SCTP is not set
> # CONFIG_TIPC is not set
> # CONFIG_ATM is not set
> # CONFIG_BRIDGE is not set
> # CONFIG_VLAN_8021Q is not set
> # CONFIG_DECNET is not set
> # CONFIG_LLC2 is not set
> # CONFIG_IPX is not set
> # CONFIG_ATALK is not set
> # CONFIG_X25 is not set
> # CONFIG_LAPB is not set
> # CONFIG_ECONET is not set
> # CONFIG_WAN_ROUTER is not set
>
> #
> # QoS and/or fair queueing
> #
> # CONFIG_NET_SCHED is not set
>
> #
> # Network testing
> #
> # CONFIG_NET_PKTGEN is not set
> # CONFIG_NET_TCPPROBE is not set
> # CONFIG_HAMRADIO is not set
> # CONFIG_IRDA is not set
> # CONFIG_BT is not set
> # CONFIG_AF_RXRPC is not set
>
> #
> # Wireless
> #
> # CONFIG_CFG80211 is not set
> # CONFIG_WIRELESS_EXT is not set
> # CONFIG_MAC80211 is not set
> # CONFIG_IEEE80211 is not set
> # CONFIG_RFKILL is not set
> # CONFIG_NET_9P is not set
>
> #
> # Device Drivers
> #
>
> #
> # Generic Driver Options
> #
> CONFIG_STANDALONE=y
> CONFIG_PREVENT_FIRMWARE_BUILD=y
> CONFIG_FW_LOADER=y
> # CONFIG_DEBUG_DRIVER is not set
> # CONFIG_DEBUG_DEVRES is not set
> # CONFIG_SYS_HYPERVISOR is not set
> CONFIG_CONNECTOR=y
> CONFIG_PROC_EVENTS=y
> # CONFIG_MTD is not set
> # CONFIG_PARPORT is not set
> CONFIG_PNP=y
> # CONFIG_PNP_DEBUG is not set
>
> #
> # Protocols
> #
> CONFIG_PNPACPI=y
> CONFIG_BLK_DEV=y
> CONFIG_BLK_DEV_FD=y
> # CONFIG_BLK_CPQ_DA is not set
> # CONFIG_BLK_CPQ_CISS_DA is not set
> # CONFIG_BLK_DEV_DAC960 is not set
> # CONFIG_BLK_DEV_UMEM is not set
> # CONFIG_BLK_DEV_COW_COMMON is not set
> CONFIG_BLK_DEV_LOOP=y
> # CONFIG_BLK_DEV_CRYPTOLOOP is not set
> # CONFIG_BLK_DEV_NBD is not set
> # CONFIG_BLK_DEV_SX8 is not set
> CONFIG_BLK_DEV_RAM=y
> CONFIG_BLK_DEV_RAM_COUNT=16
> CONFIG_BLK_DEV_RAM_SIZE=4096
> CONFIG_BLK_DEV_RAM_BLOCKSIZE=1024
> # CONFIG_CDROM_PKTCDVD is not set
> # CONFIG_ATA_OVER_ETH is not set
> CONFIG_MISC_DEVICES=y
> # CONFIG_IBM_ASM is not set
> # CONFIG_PHANTOM is not set
> # CONFIG_EEPROM_93CX6 is not set
> # CONFIG_SGI_IOC4 is not set
> # CONFIG_TIFM_CORE is not set
> # CONFIG_SONY_LAPTOP is not set
> # CONFIG_THINKPAD_ACPI is not set
> CONFIG_IDE=y
> CONFIG_BLK_DEV_IDE=y
>
> #
> # Please see Documentation/ide.txt for help/info on IDE drives
> #
> # CONFIG_BLK_DEV_IDE_SATA is not set
> # CONFIG_BLK_DEV_HD_IDE is not set
> CONFIG_BLK_DEV_IDEDISK=y
> CONFIG_IDEDISK_MULTI_MODE=y
> CONFIG_BLK_DEV_IDECD=y
> # CONFIG_BLK_DEV_IDETAPE is not set
> # CONFIG_BLK_DEV_IDEFLOPPY is not set
> # CONFIG_BLK_DEV_IDESCSI is not set
> # CONFIG_BLK_DEV_IDEACPI is not set
> # CONFIG_IDE_TASK_IOCTL is not set
> # CONFIG_IDE_PROC_FS is not set
>
> #
> # IDE chipset support/bugfixes
> #
> CONFIG_IDE_GENERIC=y
> # CONFIG_BLK_DEV_PLATFORM is not set
> # CONFIG_BLK_DEV_CMD640 is not set
> # CONFIG_BLK_DEV_IDEPNP is not set
>
> #
> # PCI IDE chipsets support
> #
> CONFIG_BLK_DEV_IDEPCI=y
> # CONFIG_IDEPCI_SHARE_IRQ is not set
> CONFIG_IDEPCI_PCIBUS_ORDER=y
> # CONFIG_BLK_DEV_OFFBOARD is not set
> # CONFIG_BLK_DEV_GENERIC is not set
> # CONFIG_BLK_DEV_OPTI621 is not set
> # CONFIG_BLK_DEV_RZ1000 is not set
> CONFIG_BLK_DEV_IDEDMA_PCI=y
> # CONFIG_BLK_DEV_IDEDMA_FORCED is not set
> # CONFIG_IDEDMA_ONLYDISK is not set
> # CONFIG_BLK_DEV_AEC62XX is not set
> # CONFIG_BLK_DEV_ALI15X3 is not set
> CONFIG_BLK_DEV_AMD74XX=y
> # CONFIG_BLK_DEV_ATIIXP is not set
> # CONFIG_BLK_DEV_CMD64X is not set
> # CONFIG_BLK_DEV_TRIFLEX is not set
> # CONFIG_BLK_DEV_CY82C693 is not set
> # CONFIG_BLK_DEV_CS5520 is not set
> # CONFIG_BLK_DEV_CS5530 is not set
> # CONFIG_BLK_DEV_CS5535 is not set
> # CONFIG_BLK_DEV_HPT34X is not set
> # CONFIG_BLK_DEV_HPT366 is not set
> # CONFIG_BLK_DEV_JMICRON is not set
> # CONFIG_BLK_DEV_SC1200 is not set
> CONFIG_BLK_DEV_PIIX=y
> # CONFIG_BLK_DEV_IT8213 is not set
> # CONFIG_BLK_DEV_IT821X is not set
> # CONFIG_BLK_DEV_NS87415 is not set
> # CONFIG_BLK_DEV_PDC202XX_OLD is not set
> # CONFIG_BLK_DEV_PDC202XX_NEW is not set
> # CONFIG_BLK_DEV_SVWKS is not set
> # CONFIG_BLK_DEV_SIIMAGE is not set
> # CONFIG_BLK_DEV_SIS5513 is not set
> # CONFIG_BLK_DEV_SLC90E66 is not set
> # CONFIG_BLK_DEV_TRM290 is not set
> # CONFIG_BLK_DEV_VIA82CXXX is not set
> # CONFIG_BLK_DEV_TC86C001 is not set
> # CONFIG_IDE_ARM is not set
> CONFIG_BLK_DEV_IDEDMA=y
> # CONFIG_BLK_DEV_HD is not set
>
> #
> # SCSI device support
> #
> # CONFIG_RAID_ATTRS is not set
> CONFIG_SCSI=y
> CONFIG_SCSI_DMA=y
> # CONFIG_SCSI_TGT is not set
> CONFIG_SCSI_NETLINK=y
> # CONFIG_SCSI_PROC_FS is not set
>
> #
> # SCSI support type (disk, tape, CD-ROM)
> #
> CONFIG_BLK_DEV_SD=y
> CONFIG_CHR_DEV_ST=y
> CONFIG_CHR_DEV_OSST=y
> CONFIG_BLK_DEV_SR=y
> # CONFIG_BLK_DEV_SR_VENDOR is not set
> CONFIG_CHR_DEV_SG=y
> # CONFIG_CHR_DEV_SCH is not set
>
> #
> # Some SCSI devices (e.g. CD jukebox) support multiple LUNs
> #
> # CONFIG_SCSI_MULTI_LUN is not set
> # CONFIG_SCSI_CONSTANTS is not set
> # CONFIG_SCSI_LOGGING is not set
> # CONFIG_SCSI_SCAN_ASYNC is not set
> CONFIG_SCSI_WAIT_SCAN=m
>
> #
> # SCSI Transports
> #
> CONFIG_SCSI_SPI_ATTRS=y
> CONFIG_SCSI_FC_ATTRS=y
> # CONFIG_SCSI_ISCSI_ATTRS is not set
> CONFIG_SCSI_SAS_ATTRS=y
> # CONFIG_SCSI_SAS_LIBSAS is not set
> # CONFIG_SCSI_SRP_ATTRS is not set
> CONFIG_SCSI_LOWLEVEL=y
> # CONFIG_ISCSI_TCP is not set
> # CONFIG_BLK_DEV_3W_XXXX_RAID is not set
> # CONFIG_SCSI_3W_9XXX is not set
> # CONFIG_SCSI_ACARD is not set
> # CONFIG_SCSI_AACRAID is not set
> CONFIG_SCSI_AIC7XXX=y
> CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
> CONFIG_AIC7XXX_RESET_DELAY_MS=5000
> CONFIG_AIC7XXX_DEBUG_ENABLE=y
> CONFIG_AIC7XXX_DEBUG_MASK=0
> CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
> CONFIG_SCSI_AIC7XXX_OLD=y
> # CONFIG_SCSI_AIC79XX is not set
> # CONFIG_SCSI_AIC94XX is not set
> # CONFIG_SCSI_DPT_I2O is not set
> # CONFIG_SCSI_ADVANSYS is not set
> # CONFIG_SCSI_ARCMSR is not set
> CONFIG_MEGARAID_NEWGEN=y
> CONFIG_MEGARAID_MM=y
> CONFIG_MEGARAID_MAILBOX=y
> CONFIG_MEGARAID_LEGACY=y
> CONFIG_MEGARAID_SAS=y
> # CONFIG_SCSI_HPTIOP is not set
> # CONFIG_SCSI_BUSLOGIC is not set
> # CONFIG_SCSI_DMX3191D is not set
> # CONFIG_SCSI_EATA is not set
> # CONFIG_SCSI_FUTURE_DOMAIN is not set
> # CONFIG_SCSI_GDTH is not set
> # CONFIG_SCSI_IPS is not set
> # CONFIG_SCSI_INITIO is not set
> # CONFIG_SCSI_INIA100 is not set
> # CONFIG_SCSI_STEX is not set
> # CONFIG_SCSI_SYM53C8XX_2 is not set
> # CONFIG_SCSI_QLOGIC_1280 is not set
> # CONFIG_SCSI_QLA_FC is not set
> # CONFIG_SCSI_QLA_ISCSI is not set
> # CONFIG_SCSI_LPFC is not set
> # CONFIG_SCSI_DC395x is not set
> # CONFIG_SCSI_DC390T is not set
> # CONFIG_SCSI_NSP32 is not set
> # CONFIG_SCSI_DEBUG is not set
> # CONFIG_SCSI_SRP is not set
> # CONFIG_ATA is not set
> CONFIG_MD=y
> # CONFIG_BLK_DEV_MD is not set
> CONFIG_BLK_DEV_DM=y
> # CONFIG_DM_DEBUG is not set
> # CONFIG_DM_CRYPT is not set
> # CONFIG_DM_SNAPSHOT is not set
> # CONFIG_DM_MIRROR is not set
> # CONFIG_DM_ZERO is not set
> # CONFIG_DM_MULTIPATH is not set
> # CONFIG_DM_DELAY is not set
> CONFIG_FUSION=y
> CONFIG_FUSION_SPI=y
> CONFIG_FUSION_FC=y
> CONFIG_FUSION_SAS=y
> CONFIG_FUSION_MAX_SGE=128
> CONFIG_FUSION_CTL=y
> # CONFIG_FUSION_LOGGING is not set
>
> #
> # IEEE 1394 (FireWire) support
> #
> # CONFIG_FIREWIRE is not set
> CONFIG_IEEE1394=y
>
> #
> # Subsystem Options
> #
> # CONFIG_IEEE1394_VERBOSEDEBUG is not set
>
> #
> # Controllers
> #
>
> #
> # Texas Instruments PCILynx requires I2C
> #
> CONFIG_IEEE1394_OHCI1394=y
>
> #
> # Protocols
> #
> # CONFIG_IEEE1394_VIDEO1394 is not set
> # CONFIG_IEEE1394_SBP2 is not set
> # CONFIG_IEEE1394_ETH1394_ROM_ENTRY is not set
> # CONFIG_IEEE1394_ETH1394 is not set
> # CONFIG_IEEE1394_DV1394 is not set
> CONFIG_IEEE1394_RAWIO=y
> # CONFIG_I2O is not set
> CONFIG_MACINTOSH_DRIVERS=y
> # CONFIG_MAC_EMUMOUSEBTN is not set
> CONFIG_NETDEVICES=y
> # CONFIG_NETDEVICES_MULTIQUEUE is not set
> # CONFIG_DUMMY is not set
> CONFIG_BONDING=y
> # CONFIG_MACVLAN is not set
> # CONFIG_EQUALIZER is not set
> # CONFIG_TUN is not set
> # CONFIG_NET_SB1000 is not set
> # CONFIG_ARCNET is not set
> # CONFIG_PHYLIB is not set
> CONFIG_NET_ETHERNET=y
> CONFIG_MII=y
> # CONFIG_HAPPYMEAL is not set
> # CONFIG_SUNGEM is not set
> # CONFIG_CASSINI is not set
> # CONFIG_NET_VENDOR_3COM is not set
> CONFIG_NET_TULIP=y
> # CONFIG_DE2104X is not set
> CONFIG_TULIP=y
> # CONFIG_TULIP_MWI is not set
> # CONFIG_TULIP_MMIO is not set
> # CONFIG_TULIP_NAPI is not set
> # CONFIG_DE4X5 is not set
> # CONFIG_WINBOND_840 is not set
> # CONFIG_DM9102 is not set
> # CONFIG_ULI526X is not set
> # CONFIG_HP100 is not set
> CONFIG_NET_PCI=y
> # CONFIG_PCNET32 is not set
> # CONFIG_AMD8111_ETH is not set
> # CONFIG_ADAPTEC_STARFIRE is not set
> CONFIG_FORCEDETH=y
> # CONFIG_FORCEDETH_NAPI is not set
> # CONFIG_DGRS is not set
> # CONFIG_EEPRO100 is not set
> # CONFIG_E100 is not set
> # CONFIG_FEALNX is not set
> # CONFIG_NATSEMI is not set
> # CONFIG_NE2K_PCI is not set
> # CONFIG_8139CP is not set
> # CONFIG_8139TOO is not set
> # CONFIG_SIS900 is not set
> # CONFIG_EPIC100 is not set
> # CONFIG_SUNDANCE is not set
> # CONFIG_TLAN is not set
> # CONFIG_VIA_RHINE is not set
> # CONFIG_SC92031 is not set
> CONFIG_NETDEV_1000=y
> # CONFIG_ACENIC is not set
> # CONFIG_DL2K is not set
> # CONFIG_E1000 is not set
> # CONFIG_E1000E is not set
> # CONFIG_NS83820 is not set
> # CONFIG_HAMACHI is not set
> # CONFIG_YELLOWFIN is not set
> # CONFIG_R8169 is not set
> # CONFIG_SIS190 is not set
> # CONFIG_SKGE is not set
> # CONFIG_SKY2 is not set
> # CONFIG_VIA_VELOCITY is not set
> CONFIG_TIGON3=y
> CONFIG_BNX2=y
> # CONFIG_QLA3XXX is not set
> # CONFIG_ATL1 is not set
> CONFIG_NETDEV_10000=y
> # CONFIG_CHELSIO_T1 is not set
> # CONFIG_CHELSIO_T3 is not set
> # CONFIG_IXGB is not set
> # CONFIG_IXGBE is not set
> # CONFIG_S2IO is not set
> # CONFIG_MYRI10GE is not set
> # CONFIG_NETXEN_NIC is not set
> # CONFIG_MLX4_CORE is not set
> # CONFIG_TR is not set
>
> #
> # Wireless LAN
> #
> # CONFIG_WLAN_PRE80211 is not set
> # CONFIG_WLAN_80211 is not set
> # CONFIG_WAN is not set
> # CONFIG_FDDI is not set
> # CONFIG_HIPPI is not set
> CONFIG_PPP=y
> CONFIG_PPP_MULTILINK=y
> CONFIG_PPP_FILTER=y
> # CONFIG_PPP_ASYNC is not set
> # CONFIG_PPP_SYNC_TTY is not set
> # CONFIG_PPP_DEFLATE is not set
> # CONFIG_PPP_BSDCOMP is not set
> # CONFIG_PPP_MPPE is not set
> # CONFIG_PPPOE is not set
> # CONFIG_PPPOL2TP is not set
> # CONFIG_SLIP is not set
> CONFIG_SLHC=y
> # CONFIG_NET_FC is not set
> # CONFIG_SHAPER is not set
> CONFIG_NETCONSOLE=y
> # CONFIG_NETCONSOLE_DYNAMIC is not set
> CONFIG_NETPOLL=y
> # CONFIG_NETPOLL_TRAP is not set
> CONFIG_NET_POLL_CONTROLLER=y
> # CONFIG_ISDN is not set
> # CONFIG_PHONE is not set
>
> #
> # Input device support
> #
> CONFIG_INPUT=y
> # CONFIG_INPUT_FF_MEMLESS is not set
> # CONFIG_INPUT_POLLDEV is not set
>
> #
> # Userland interfaces
> #
> CONFIG_INPUT_MOUSEDEV=y
> CONFIG_INPUT_MOUSEDEV_PSAUX=y
> CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
> CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
> # CONFIG_INPUT_JOYDEV is not set
> # CONFIG_INPUT_TSDEV is not set
> CONFIG_INPUT_EVDEV=y
> # CONFIG_INPUT_EVBUG is not set
>
> #
> # Input Device Drivers
> #
> CONFIG_INPUT_KEYBOARD=y
> CONFIG_KEYBOARD_ATKBD=y
> # CONFIG_KEYBOARD_SUNKBD is not set
> # CONFIG_KEYBOARD_LKKBD is not set
> # CONFIG_KEYBOARD_XTKBD is not set
> # CONFIG_KEYBOARD_NEWTON is not set
> # CONFIG_KEYBOARD_STOWAWAY is not set
> CONFIG_INPUT_MOUSE=y
> CONFIG_MOUSE_PS2=y
> CONFIG_MOUSE_PS2_ALPS=y
> CONFIG_MOUSE_PS2_LOGIPS2PP=y
> CONFIG_MOUSE_PS2_SYNAPTICS=y
> CONFIG_MOUSE_PS2_LIFEBOOK=y
> CONFIG_MOUSE_PS2_TRACKPOINT=y
> # CONFIG_MOUSE_PS2_TOUCHKIT is not set
> # CONFIG_MOUSE_SERIAL is not set
> # CONFIG_MOUSE_APPLETOUCH is not set
> # CONFIG_MOUSE_VSXXXAA is not set
> # CONFIG_INPUT_JOYSTICK is not set
> # CONFIG_INPUT_TABLET is not set
> # CONFIG_INPUT_TOUCHSCREEN is not set
> # CONFIG_INPUT_MISC is not set
>
> #
> # Hardware I/O ports
> #
> CONFIG_SERIO=y
> CONFIG_SERIO_I8042=y
> # CONFIG_SERIO_SERPORT is not set
> # CONFIG_SERIO_CT82C710 is not set
> # CONFIG_SERIO_PCIPS2 is not set
> CONFIG_SERIO_LIBPS2=y
> # CONFIG_SERIO_RAW is not set
> # CONFIG_GAMEPORT is not set
>
> #
> # Character devices
> #
> CONFIG_VT=y
> CONFIG_VT_CONSOLE=y
> CONFIG_HW_CONSOLE=y
> # CONFIG_VT_HW_CONSOLE_BINDING is not set
> CONFIG_SERIAL_NONSTANDARD=y
> # CONFIG_COMPUTONE is not set
> # CONFIG_ROCKETPORT is not set
> # CONFIG_CYCLADES is not set
> # CONFIG_DIGIEPCA is not set
> # CONFIG_MOXA_INTELLIO is not set
> # CONFIG_MOXA_SMARTIO is not set
> # CONFIG_MOXA_SMARTIO_NEW is not set
> # CONFIG_ISI is not set
> # CONFIG_SYNCLINK is not set
> # CONFIG_SYNCLINKMP is not set
> # CONFIG_SYNCLINK_GT is not set
> # CONFIG_N_HDLC is not set
> # CONFIG_SPECIALIX is not set
> # CONFIG_SX is not set
> # CONFIG_RIO is not set
> CONFIG_STALDRV=y
> # CONFIG_NOZOMI is not set
>
> #
> # Serial drivers
> #
> CONFIG_SERIAL_8250=y
> CONFIG_SERIAL_8250_CONSOLE=y
> CONFIG_FIX_EARLYCON_MEM=y
> CONFIG_SERIAL_8250_PCI=y
> CONFIG_SERIAL_8250_PNP=y
> CONFIG_SERIAL_8250_NR_UARTS=4
> CONFIG_SERIAL_8250_RUNTIME_UARTS=4
> CONFIG_SERIAL_8250_EXTENDED=y
> # CONFIG_SERIAL_8250_MANY_PORTS is not set
> CONFIG_SERIAL_8250_SHARE_IRQ=y
> CONFIG_SERIAL_8250_DETECT_IRQ=y
> CONFIG_SERIAL_8250_RSA=y
>
> #
> # Non-8250 serial port support
> #
> CONFIG_SERIAL_CORE=y
> CONFIG_SERIAL_CORE_CONSOLE=y
> # CONFIG_SERIAL_JSM is not set
> CONFIG_UNIX98_PTYS=y
> CONFIG_LEGACY_PTYS=y
> CONFIG_LEGACY_PTY_COUNT=256
> # CONFIG_IPMI_HANDLER is not set
> # CONFIG_WATCHDOG is not set
> # CONFIG_HW_RANDOM is not set
> # CONFIG_NVRAM is not set
> CONFIG_RTC=y
> # CONFIG_R3964 is not set
> # CONFIG_APPLICOM is not set
> # CONFIG_SONYPI is not set
> CONFIG_AGP=y
> CONFIG_AGP_ALI=y
> CONFIG_AGP_ATI=y
> # CONFIG_AGP_AMD is not set
> CONFIG_AGP_AMD64=y
> CONFIG_AGP_INTEL=y
> # CONFIG_AGP_NVIDIA is not set
> # CONFIG_AGP_SIS is not set
> # CONFIG_AGP_SWORKS is not set
> # CONFIG_AGP_VIA is not set
> # CONFIG_AGP_EFFICEON is not set
> CONFIG_DRM=y
> # CONFIG_DRM_TDFX is not set
> # CONFIG_DRM_R128 is not set
> CONFIG_DRM_RADEON=y
> # CONFIG_DRM_I810 is not set
> # CONFIG_DRM_I830 is not set
> # CONFIG_DRM_I915 is not set
> # CONFIG_DRM_MGA is not set
> # CONFIG_DRM_SIS is not set
> # CONFIG_DRM_VIA is not set
> # CONFIG_DRM_SAVAGE is not set
> # CONFIG_MWAVE is not set
> # CONFIG_PC8736x_GPIO is not set
> # CONFIG_NSC_GPIO is not set
> # CONFIG_CS5535_GPIO is not set
> CONFIG_RAW_DRIVER=y
> CONFIG_MAX_RAW_DEVS=256
> CONFIG_HPET=y
> # CONFIG_HPET_RTC_IRQ is not set
> CONFIG_HPET_MMAP=y
> CONFIG_HANGCHECK_TIMER=y
> # CONFIG_TCG_TPM is not set
> # CONFIG_TELCLOCK is not set
> CONFIG_DEVPORT=y
> # CONFIG_I2C is not set
>
> #
> # SPI support
> #
> # CONFIG_SPI is not set
> # CONFIG_SPI_MASTER is not set
> # CONFIG_W1 is not set
> # CONFIG_POWER_SUPPLY is not set
> # CONFIG_HWMON is not set
>
> #
> # Sonics Silicon Backplane
> #
> # CONFIG_SSB is not set
>
> #
> # Multifunction device drivers
> #
> # CONFIG_MFD_SM501 is not set
>
> #
> # Multimedia devices
> #
> # CONFIG_VIDEO_DEV is not set
> # CONFIG_DVB_CORE is not set
> CONFIG_DAB=y
>
> #
> # Graphics support
> #
> # CONFIG_BACKLIGHT_LCD_SUPPORT is not set
>
> #
> # Display device support
> #
> # CONFIG_DISPLAY_SUPPORT is not set
> # CONFIG_VGASTATE is not set
> CONFIG_VIDEO_OUTPUT_CONTROL=m
> # CONFIG_FB is not set
>
> #
> # Console display driver support
> #
> CONFIG_VGA_CONSOLE=y
> CONFIG_VGACON_SOFT_SCROLLBACK=y
> CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=128
> CONFIG_VIDEO_SELECT=y
> CONFIG_DUMMY_CONSOLE=y
>
> #
> # Sound
> #
> CONFIG_SOUND=y
>
> #
> # Advanced Linux Sound Architecture
> #
> # CONFIG_SND is not set
>
> #
> # Open Sound System
> #
> CONFIG_SOUND_PRIME=y
> # CONFIG_SOUND_TRIDENT is not set
> # CONFIG_SOUND_MSNDCLAS is not set
> # CONFIG_SOUND_MSNDPIN is not set
> # CONFIG_SOUND_OSS is not set
> CONFIG_HID_SUPPORT=y
> CONFIG_HID=y
> # CONFIG_HID_DEBUG is not set
> # CONFIG_HIDRAW is not set
> CONFIG_USB_SUPPORT=y
> CONFIG_USB_ARCH_HAS_HCD=y
> CONFIG_USB_ARCH_HAS_OHCI=y
> CONFIG_USB_ARCH_HAS_EHCI=y
> # CONFIG_USB is not set
>
> #
> # NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
> #
>
> #
> # USB Gadget Support
> #
> # CONFIG_USB_GADGET is not set
> # CONFIG_MMC is not set
> # CONFIG_NEW_LEDS is not set
> # CONFIG_INFINIBAND is not set
> # CONFIG_EDAC is not set
> # CONFIG_RTC_CLASS is not set
>
> #
> # DMA Engine support
> #
> # CONFIG_DMA_ENGINE is not set
>
> #
> # DMA Clients
> #
>
> #
> # DMA Devices
> #
> # CONFIG_DCA is not set
> CONFIG_VIRTUALIZATION=y
> # CONFIG_KVM is not set
>
> #
> # Userspace I/O
> #
> # CONFIG_UIO is not set
>
> #
> # File systems
> #
> CONFIG_EXT2_FS=y
> CONFIG_EXT2_FS_XATTR=y
> CONFIG_EXT2_FS_POSIX_ACL=y
> CONFIG_EXT2_FS_SECURITY=y
> CONFIG_EXT2_FS_XIP=y
> CONFIG_FS_XIP=y
> CONFIG_EXT3_FS=y
> CONFIG_EXT3_FS_XATTR=y
> CONFIG_EXT3_FS_POSIX_ACL=y
> CONFIG_EXT3_FS_SECURITY=y
> # CONFIG_EXT4DEV_FS is not set
> CONFIG_JBD=y
> # CONFIG_JBD_DEBUG is not set
> CONFIG_FS_MBCACHE=y
> # CONFIG_REISER4_FS is not set
> CONFIG_REISERFS_FS=y
> CONFIG_REISERFS_CHECK=y
> CONFIG_REISERFS_PROC_INFO=y
> CONFIG_REISERFS_FS_XATTR=y
> CONFIG_REISERFS_FS_POSIX_ACL=y
> CONFIG_REISERFS_FS_SECURITY=y
> # CONFIG_JFS_FS is not set
> CONFIG_FS_POSIX_ACL=y
> # CONFIG_XFS_FS is not set
> # CONFIG_GFS2_FS is not set
> # CONFIG_OCFS2_FS is not set
> # CONFIG_MINIX_FS is not set
> # CONFIG_ROMFS_FS is not set
> CONFIG_INOTIFY=y
> CONFIG_INOTIFY_USER=y
> # CONFIG_QUOTA is not set
> CONFIG_DNOTIFY=y
> # CONFIG_AUTOFS_FS is not set
> CONFIG_AUTOFS4_FS=y
> # CONFIG_FUSE_FS is not set
>
> #
> # CD-ROM/DVD Filesystems
> #
> CONFIG_ISO9660_FS=y
> # CONFIG_JOLIET is not set
> # CONFIG_ZISOFS is not set
> # CONFIG_UDF_FS is not set
>
> #
> # DOS/FAT/NT Filesystems
> #
> CONFIG_FAT_FS=y
> CONFIG_MSDOS_FS=y
> CONFIG_VFAT_FS=y
> CONFIG_FAT_DEFAULT_CODEPAGE=437
> CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
> # CONFIG_NTFS_FS is not set
>
> #
> # Pseudo filesystems
> #
> CONFIG_PROC_FS=y
> # CONFIG_PROC_KCORE is not set
> CONFIG_PROC_SYSCTL=y
> CONFIG_SYSFS=y
> CONFIG_TMPFS=y
> # CONFIG_TMPFS_POSIX_ACL is not set
> CONFIG_HUGETLBFS=y
> CONFIG_HUGETLB_PAGE=y
> CONFIG_RAMFS=y
> CONFIG_CONFIGFS_FS=y
>
> #
> # Layered filesystems
> #
> # CONFIG_UNION_FS is not set
>
> #
> # Miscellaneous filesystems
> #
> # CONFIG_ADFS_FS is not set
> # CONFIG_AFFS_FS is not set
> # CONFIG_HFS_FS is not set
> # CONFIG_HFSPLUS_FS is not set
> # CONFIG_BEFS_FS is not set
> # CONFIG_BFS_FS is not set
> # CONFIG_EFS_FS is not set
> # CONFIG_CRAMFS is not set
> # CONFIG_VXFS_FS is not set
> # CONFIG_HPFS_FS is not set
> # CONFIG_QNX4FS_FS is not set
> # CONFIG_SYSV_FS is not set
> # CONFIG_UFS_FS is not set
>
> #
> # Network File Systems
> #
> # CONFIG_NFS_FS is not set
> # CONFIG_NFSD is not set
> CONFIG_SMB_FS=y
> # CONFIG_SMB_NLS_DEFAULT is not set
> CONFIG_CIFS=y
> # CONFIG_CIFS_STATS is not set
> # CONFIG_CIFS_WEAK_PW_HASH is not set
> # CONFIG_CIFS_XATTR is not set
> # CONFIG_CIFS_DEBUG2 is not set
> # CONFIG_CIFS_EXPERIMENTAL is not set
> # CONFIG_NCP_FS is not set
> # CONFIG_CODA_FS is not set
> # CONFIG_AFS_FS is not set
>
> #
> # Partition Types
> #
> # CONFIG_PARTITION_ADVANCED is not set
> CONFIG_MSDOS_PARTITION=y
>
> #
> # Native Language Support
> #
> CONFIG_NLS=y
> CONFIG_NLS_DEFAULT="iso8859-1"
> CONFIG_NLS_CODEPAGE_437=y
> # CONFIG_NLS_CODEPAGE_737 is not set
> # CONFIG_NLS_CODEPAGE_775 is not set
> # CONFIG_NLS_CODEPAGE_850 is not set
> # CONFIG_NLS_CODEPAGE_852 is not set
> # CONFIG_NLS_CODEPAGE_855 is not set
> # CONFIG_NLS_CODEPAGE_857 is not set
> # CONFIG_NLS_CODEPAGE_860 is not set
> # CONFIG_NLS_CODEPAGE_861 is not set
> # CONFIG_NLS_CODEPAGE_862 is not set
> # CONFIG_NLS_CODEPAGE_863 is not set
> # CONFIG_NLS_CODEPAGE_864 is not set
> # CONFIG_NLS_CODEPAGE_865 is not set
> # CONFIG_NLS_CODEPAGE_866 is not set
> # CONFIG_NLS_CODEPAGE_869 is not set
> # CONFIG_NLS_CODEPAGE_936 is not set
> # CONFIG_NLS_CODEPAGE_950 is not set
> # CONFIG_NLS_CODEPAGE_932 is not set
> # CONFIG_NLS_CODEPAGE_949 is not set
> # CONFIG_NLS_CODEPAGE_874 is not set
> # CONFIG_NLS_ISO8859_8 is not set
> # CONFIG_NLS_CODEPAGE_1250 is not set
> # CONFIG_NLS_CODEPAGE_1251 is not set
> CONFIG_NLS_ASCII=y
> CONFIG_NLS_ISO8859_1=y
> # CONFIG_NLS_ISO8859_2 is not set
> # CONFIG_NLS_ISO8859_3 is not set
> # CONFIG_NLS_ISO8859_4 is not set
> # CONFIG_NLS_ISO8859_5 is not set
> # CONFIG_NLS_ISO8859_6 is not set
> # CONFIG_NLS_ISO8859_7 is not set
> # CONFIG_NLS_ISO8859_9 is not set
> # CONFIG_NLS_ISO8859_13 is not set
> # CONFIG_NLS_ISO8859_14 is not set
> CONFIG_NLS_ISO8859_15=y
> # CONFIG_NLS_KOI8_R is not set
> # CONFIG_NLS_KOI8_U is not set
> CONFIG_NLS_UTF8=y
>
> #
> # Distributed Lock Manager
> #
> # CONFIG_DLM is not set
> CONFIG_INSTRUMENTATION=y
> CONFIG_PROFILING=y
> CONFIG_OPROFILE=y
> CONFIG_KPROBES=y
>
> #
> # Kernel hacking
> #
> CONFIG_TRACE_IRQFLAGS_SUPPORT=y
> # CONFIG_PRINTK_TIME is not set
> CONFIG_ENABLE_MUST_CHECK=y
> CONFIG_MAGIC_SYSRQ=y
> CONFIG_UNUSED_SYMBOLS=y
> # CONFIG_PAGE_OWNER is not set
> # CONFIG_DEBUG_FS is not set
> # CONFIG_HEADERS_CHECK is not set
> CONFIG_DEBUG_KERNEL=y
> # CONFIG_DEBUG_SHIRQ is not set
> CONFIG_DETECT_SOFTLOCKUP=y
> # CONFIG_SCHED_DEBUG is not set
> # CONFIG_SCHEDSTATS is not set
> # CONFIG_TIMER_STATS is not set
> # CONFIG_DEBUG_SLAB is not set
> # CONFIG_DEBUG_RT_MUTEXES is not set
> # CONFIG_RT_MUTEX_TESTER is not set
> CONFIG_DEBUG_SPINLOCK=y
> CONFIG_DEBUG_MUTEXES=y
> CONFIG_DEBUG_LOCK_ALLOC=y
> CONFIG_PROVE_LOCKING=y
> CONFIG_LOCKDEP=y
> # CONFIG_LOCK_STAT is not set
> CONFIG_DEBUG_LOCKDEP=y
> CONFIG_TRACE_IRQFLAGS=y
> CONFIG_DEBUG_SPINLOCK_SLEEP=y
> # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
> CONFIG_STACKTRACE=y
> # CONFIG_DEBUG_KOBJECT is not set
> # CONFIG_DEBUG_HIGHMEM is not set
> CONFIG_DEBUG_BUGVERBOSE=y
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_VM is not set
> # CONFIG_DEBUG_LIST is not set
> CONFIG_FRAME_POINTER=y
> # CONFIG_PROFILE_LIKELY is not set
> CONFIG_FORCED_INLINING=y
> # CONFIG_BOOT_PRINTK_DELAY is not set
> # CONFIG_DEBUG_SYNCHRO_TEST is not set
> # CONFIG_RCU_TORTURE_TEST is not set
> # CONFIG_LKDTM is not set
> # CONFIG_FAULT_INJECTION is not set
> # CONFIG_WANT_EXTRA_DEBUG_INFORMATION is not set
> # CONFIG_UNWIND_INFO is not set
> # CONFIG_KGDB is not set
> CONFIG_EARLY_PRINTK=y
> # CONFIG_DEBUG_STACKOVERFLOW is not set
> # CONFIG_DEBUG_STACK_USAGE is not set
> # CONFIG_DEBUG_RODATA is not set
> # CONFIG_4KSTACKS is not set
> CONFIG_X86_FIND_SMP_CONFIG=y
> CONFIG_X86_MPPARSE=y
> CONFIG_DOUBLEFAULT=y
>
> #
> # Security options
> #
> # CONFIG_KEYS is not set
> # CONFIG_SECURITY is not set
> # CONFIG_SECURITY_FILE_CAPABILITIES is not set
> # CONFIG_CRYPTO is not set
>
> #
> # Library routines
> #
> CONFIG_BITREVERSE=y
> # CONFIG_CRC_CCITT is not set
> # CONFIG_CRC16 is not set
> # CONFIG_CRC_ITU_T is not set
> CONFIG_CRC32=y
> # CONFIG_CRC7 is not set
> # CONFIG_LIBCRC32C is not set
> CONFIG_AUDIT_GENERIC=y
> CONFIG_ZLIB_INFLATE=y
> CONFIG_PLIST=y
> CONFIG_HAS_IOMEM=y
> CONFIG_HAS_IOPORT=y
> CONFIG_HAS_DMA=y
> CONFIG_GENERIC_HARDIRQS=y
> CONFIG_GENERIC_IRQ_PROBE=y
> CONFIG_GENERIC_PENDING_IRQ=y
> CONFIG_X86_SMP=y
> CONFIG_X86_HT=y
> CONFIG_X86_BIOS_REBOOT=y
> CONFIG_X86_TRAMPOLINE=y
> CONFIG_KTIME_SCALAR=y

Yeah, I'm seeing the same thing. Oddly, my node_states[N_NORMAL_MEMORY]
and node_states[N_HIGH_MEMORY] are empty, while node_states[N_ONLINE]
contains my single cpu (on i386 kvm image).

-serge

2007-08-14 16:31:41

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Mon, 2007-08-13 at 15:12 -0500, Serge E. Hallyn wrote:
> Quoting Dhaval Giani ([email protected]):
> > Hi,
> >
> > On mounting cpusets using containers, I have been hitting the following
> > bug.
> >
> >
> > -----------[ cut here ]------------
> > kernel BUG at kernel/cpuset.c:331!
<snip>
> > CONFIG_HIGHMEM=y
> > CONFIG_X86_PAE=y
> > # CONFIG_NUMA is not set
> > CONFIG_ARCH_POPULATES_NODE_MAP=y
> > CONFIG_SELECT_MEMORY_MODEL=y
> > CONFIG_FLATMEM_MANUAL=y
> > # CONFIG_DISCONTIGMEM_MANUAL is not set
> > # CONFIG_SPARSEMEM_MANUAL is not set
> > CONFIG_FLATMEM=y
> > CONFIG_FLAT_NODE_MEM_MAP=y
> > # CONFIG_SPARSEMEM_STATIC is not set
> > CONFIG_SPLIT_PTLOCK_CPUS=4
> > CONFIG_RESOURCES_64BIT=y
> > CONFIG_ZONE_DMA_FLAG=1
<snip>
>
> Yeah, I'm seeing the same thing. Oddly, my node_states[N_NORMAL_MEMORY]
> and node_states[N_HIGH_MEMORY] are empty, while node_states[N_ONLINE]
> contains my single cpu (on i386 kvm image).
>
> -serge

Yes, you'll definitely hit that BUG if the N_HIGH_MEMORY mask is empty.
So far, I can't see how this could be, tho'. __build_all_zonelists()
should be called for non-NUMA as well as NUMA. It iterates over "all
on-line nodes"--the N_ONLINE state, which is initialized to have node 0
on-line. That should work for a non-NUMA i386, right?

In the loop over all on-line nodes, __build_all_zonelists() sets
N_HIGH_MEMORY for each [on-line] node with ANY present pages--i.e.,
non-zero pgdat->node_present_pages. Then, check_for_regular_memory()
will set N_NORMAL_MEMORY for any nodes with non-zero zone->present_pages
for any zone <= ZONE_NORMAL.

I don't have an i386 test system to verify this, but from cscope, it
looks like the pgdat, including node_present_pages should be setup in
the following call path:

start_kernel()->setup_arch()->zone_sizes_init()->
free_area_init_zones()->free_area_init_zone->
calculate_node_total_pages()

This is called before build_all_zonelists()--also from start_kernel(),
so node_present_pages should be non-zero when we populate the N_*_MEMORY
states...

Any ideas, anyone?

Lee

2007-08-14 18:04:19

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Quoting Lee Schermerhorn ([email protected]):
> On Mon, 2007-08-13 at 15:12 -0500, Serge E. Hallyn wrote:
> > Quoting Dhaval Giani ([email protected]):
> > > Hi,
> > >
> > > On mounting cpusets using containers, I have been hitting the following
> > > bug.
> > >
> > >
> > > -----------[ cut here ]------------
> > > kernel BUG at kernel/cpuset.c:331!
> <snip>
> > > CONFIG_HIGHMEM=y
> > > CONFIG_X86_PAE=y
> > > # CONFIG_NUMA is not set
> > > CONFIG_ARCH_POPULATES_NODE_MAP=y
> > > CONFIG_SELECT_MEMORY_MODEL=y
> > > CONFIG_FLATMEM_MANUAL=y
> > > # CONFIG_DISCONTIGMEM_MANUAL is not set
> > > # CONFIG_SPARSEMEM_MANUAL is not set
> > > CONFIG_FLATMEM=y
> > > CONFIG_FLAT_NODE_MEM_MAP=y
> > > # CONFIG_SPARSEMEM_STATIC is not set
> > > CONFIG_SPLIT_PTLOCK_CPUS=4
> > > CONFIG_RESOURCES_64BIT=y
> > > CONFIG_ZONE_DMA_FLAG=1
> <snip>
> >
> > Yeah, I'm seeing the same thing. Oddly, my node_states[N_NORMAL_MEMORY]
> > and node_states[N_HIGH_MEMORY] are empty, while node_states[N_ONLINE]
> > contains my single cpu (on i386 kvm image).
> >
> > -serge
>
> Yes, you'll definitely hit that BUG if the N_HIGH_MEMORY mask is empty.
> So far, I can't see how this could be, tho'. __build_all_zonelists()
> should be called for non-NUMA as well as NUMA. It iterates over "all

Yup, and it is, and some debug statements insist that
node_set_state(nid, N_HIGH_MEMORY) is being called for cpu 0,
and the state is in fact being correctly set.

So it must get cleared later...

-serge

2007-08-14 18:14:48

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Tue, 2007-08-14 at 13:03 -0500, Serge E. Hallyn wrote:
> Quoting Lee Schermerhorn ([email protected]):
> > On Mon, 2007-08-13 at 15:12 -0500, Serge E. Hallyn wrote:
> > > Quoting Dhaval Giani ([email protected]):
> > > > Hi,
> > > >
> > > > On mounting cpusets using containers, I have been hitting the following
> > > > bug.
> > > >
> > > >
> > > > -----------[ cut here ]------------
> > > > kernel BUG at kernel/cpuset.c:331!
> > <snip>
> > > > CONFIG_HIGHMEM=y
> > > > CONFIG_X86_PAE=y
> > > > # CONFIG_NUMA is not set
> > > > CONFIG_ARCH_POPULATES_NODE_MAP=y
> > > > CONFIG_SELECT_MEMORY_MODEL=y
> > > > CONFIG_FLATMEM_MANUAL=y
> > > > # CONFIG_DISCONTIGMEM_MANUAL is not set
> > > > # CONFIG_SPARSEMEM_MANUAL is not set
> > > > CONFIG_FLATMEM=y
> > > > CONFIG_FLAT_NODE_MEM_MAP=y
> > > > # CONFIG_SPARSEMEM_STATIC is not set
> > > > CONFIG_SPLIT_PTLOCK_CPUS=4
> > > > CONFIG_RESOURCES_64BIT=y
> > > > CONFIG_ZONE_DMA_FLAG=1
> > <snip>
> > >
> > > Yeah, I'm seeing the same thing. Oddly, my node_states[N_NORMAL_MEMORY]
> > > and node_states[N_HIGH_MEMORY] are empty, while node_states[N_ONLINE]
> > > contains my single cpu (on i386 kvm image).
> > >
> > > -serge
> >
> > Yes, you'll definitely hit that BUG if the N_HIGH_MEMORY mask is empty.
> > So far, I can't see how this could be, tho'. __build_all_zonelists()
> > should be called for non-NUMA as well as NUMA. It iterates over "all
>
> Yup, and it is, and some debug statements insist that
> node_set_state(nid, N_HIGH_MEMORY) is being called for cpu 0,
> and the state is in fact being correctly set.
>
> So it must get cleared later...

OK. That helps. I'll see what I can find...

Thanks,
Lee

2007-08-14 19:24:10

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Quoting Lee Schermerhorn ([email protected]):
> On Tue, 2007-08-14 at 13:03 -0500, Serge E. Hallyn wrote:
> > Quoting Lee Schermerhorn ([email protected]):
> > > On Mon, 2007-08-13 at 15:12 -0500, Serge E. Hallyn wrote:
> > > > Quoting Dhaval Giani ([email protected]):
> > > > > Hi,
> > > > >
> > > > > On mounting cpusets using containers, I have been hitting the following
> > > > > bug.
> > > > >
> > > > >
> > > > > -----------[ cut here ]------------
> > > > > kernel BUG at kernel/cpuset.c:331!
> > > <snip>
> > > > > CONFIG_HIGHMEM=y
> > > > > CONFIG_X86_PAE=y
> > > > > # CONFIG_NUMA is not set
> > > > > CONFIG_ARCH_POPULATES_NODE_MAP=y
> > > > > CONFIG_SELECT_MEMORY_MODEL=y
> > > > > CONFIG_FLATMEM_MANUAL=y
> > > > > # CONFIG_DISCONTIGMEM_MANUAL is not set
> > > > > # CONFIG_SPARSEMEM_MANUAL is not set
> > > > > CONFIG_FLATMEM=y
> > > > > CONFIG_FLAT_NODE_MEM_MAP=y
> > > > > # CONFIG_SPARSEMEM_STATIC is not set
> > > > > CONFIG_SPLIT_PTLOCK_CPUS=4
> > > > > CONFIG_RESOURCES_64BIT=y
> > > > > CONFIG_ZONE_DMA_FLAG=1
> > > <snip>
> > > >
> > > > Yeah, I'm seeing the same thing. Oddly, my node_states[N_NORMAL_MEMORY]
> > > > and node_states[N_HIGH_MEMORY] are empty, while node_states[N_ONLINE]
> > > > contains my single cpu (on i386 kvm image).
> > > >
> > > > -serge
> > >
> > > Yes, you'll definitely hit that BUG if the N_HIGH_MEMORY mask is empty.
> > > So far, I can't see how this could be, tho'. __build_all_zonelists()
> > > should be called for non-NUMA as well as NUMA. It iterates over "all
> >
> > Yup, and it is, and some debug statements insist that
> > node_set_state(nid, N_HIGH_MEMORY) is being called for cpu 0,
> > and the state is in fact being correctly set.
> >
> > So it must get cleared later...
>
> OK. That helps. I'll see what I can find...

Well apparently I lied? I don't think it's getting cleared in node_states.
I'm doing:

static void guarantee_online_mems(const struct cpuset *cs, nodemask_t *pmask)
{
int nid=0;
printk(KERN_NOTICE "%s: at top, node %d is %d\n",
__FUNCTION__, nid, node_state(nid, N_HIGH_MEMORY));
while (cs && !nodes_intersects(cs->mems_allowed,
node_states[N_HIGH_MEMORY])) {
printk(KERN_NOTICE "%s: in loop\n", __FUNCTION__);
cs = cs->parent;
}
if (cs) {
printk(KERN_NOTICE "%s: cs was true\n", __FUNCTION__);
nodes_and(*pmask, cs->mems_allowed,
node_states[N_HIGH_MEMORY]);
}
else
{
printk(KERN_NOTICE "%s: cs was false\n", __FUNCTION__);
*pmask = node_states[N_HIGH_MEMORY];
}
printk(KERN_NOTICE "%s: at bottom, node %d is %d\n",
__FUNCTION__, nid, node_state(nid, N_HIGH_MEMORY));
printk(KERN_NOTICE "%s: at bottom, in pmask node %d is %d\n",
__FUNCTION__, nid, node_isset(nid, *pmask));
BUG_ON(!nodes_intersects(*pmask, node_states[N_HIGH_MEMORY]));
}

and getting:

guarantee_online_mems: in loop
guarantee_online_mems: cs was false
guarantee_online_mems: at bottom, node 0 is 1
guarantee_online_mems: at bottom, in pmask node 0 is 0

So I'm guess that
*pmask = node_states[N_HIGH_MEMORY];
isn't doing what it was intended to do? That or I've got node_state()
wrong...

-serge

2007-08-14 20:50:21

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Quoting Serge E. Hallyn ([email protected]):
> Quoting Lee Schermerhorn ([email protected]):
> > On Tue, 2007-08-14 at 13:03 -0500, Serge E. Hallyn wrote:
> > > Quoting Lee Schermerhorn ([email protected]):
> > > > On Mon, 2007-08-13 at 15:12 -0500, Serge E. Hallyn wrote:
> > > > > Quoting Dhaval Giani ([email protected]):
> > > > > > Hi,
> > > > > >
> > > > > > On mounting cpusets using containers, I have been hitting the following
> > > > > > bug.
> > > > > >
> > > > > >
> > > > > > -----------[ cut here ]------------
> > > > > > kernel BUG at kernel/cpuset.c:331!
> > > > <snip>
> > > > > > CONFIG_HIGHMEM=y
> > > > > > CONFIG_X86_PAE=y
> > > > > > # CONFIG_NUMA is not set
> > > > > > CONFIG_ARCH_POPULATES_NODE_MAP=y
> > > > > > CONFIG_SELECT_MEMORY_MODEL=y
> > > > > > CONFIG_FLATMEM_MANUAL=y
> > > > > > # CONFIG_DISCONTIGMEM_MANUAL is not set
> > > > > > # CONFIG_SPARSEMEM_MANUAL is not set
> > > > > > CONFIG_FLATMEM=y
> > > > > > CONFIG_FLAT_NODE_MEM_MAP=y
> > > > > > # CONFIG_SPARSEMEM_STATIC is not set
> > > > > > CONFIG_SPLIT_PTLOCK_CPUS=4
> > > > > > CONFIG_RESOURCES_64BIT=y
> > > > > > CONFIG_ZONE_DMA_FLAG=1
> > > > <snip>
> > > > >
> > > > > Yeah, I'm seeing the same thing. Oddly, my node_states[N_NORMAL_MEMORY]
> > > > > and node_states[N_HIGH_MEMORY] are empty, while node_states[N_ONLINE]
> > > > > contains my single cpu (on i386 kvm image).
> > > > >
> > > > > -serge
> > > >
> > > > Yes, you'll definitely hit that BUG if the N_HIGH_MEMORY mask is empty.
> > > > So far, I can't see how this could be, tho'. __build_all_zonelists()
> > > > should be called for non-NUMA as well as NUMA. It iterates over "all
> > >
> > > Yup, and it is, and some debug statements insist that
> > > node_set_state(nid, N_HIGH_MEMORY) is being called for cpu 0,
> > > and the state is in fact being correctly set.
> > >
> > > So it must get cleared later...
> >
> > OK. That helps. I'll see what I can find...
>
> Well apparently I lied? I don't think it's getting cleared in node_states.
> I'm doing:
>
> static void guarantee_online_mems(const struct cpuset *cs, nodemask_t *pmask)
> {
> int nid=0;
> printk(KERN_NOTICE "%s: at top, node %d is %d\n",
> __FUNCTION__, nid, node_state(nid, N_HIGH_MEMORY));
> while (cs && !nodes_intersects(cs->mems_allowed,
> node_states[N_HIGH_MEMORY])) {
> printk(KERN_NOTICE "%s: in loop\n", __FUNCTION__);
> cs = cs->parent;
> }
> if (cs) {
> printk(KERN_NOTICE "%s: cs was true\n", __FUNCTION__);
> nodes_and(*pmask, cs->mems_allowed,
> node_states[N_HIGH_MEMORY]);
> }
> else
> {
> printk(KERN_NOTICE "%s: cs was false\n", __FUNCTION__);
> *pmask = node_states[N_HIGH_MEMORY];
> }
> printk(KERN_NOTICE "%s: at bottom, node %d is %d\n",
> __FUNCTION__, nid, node_state(nid, N_HIGH_MEMORY));
> printk(KERN_NOTICE "%s: at bottom, in pmask node %d is %d\n",
> __FUNCTION__, nid, node_isset(nid, *pmask));
> BUG_ON(!nodes_intersects(*pmask, node_states[N_HIGH_MEMORY]));
> }
>
> and getting:
>
> guarantee_online_mems: in loop
> guarantee_online_mems: cs was false
> guarantee_online_mems: at bottom, node 0 is 1
> guarantee_online_mems: at bottom, in pmask node 0 is 0
>
>
> So I'm guess that
> *pmask = node_states[N_HIGH_MEMORY];
> isn't doing what it was intended to do? That or I've got node_state()
> wrong...

Argh.

CONFIG_NODES_SHIFT was unset, so MAX_NUMNODES=1, so
node_state() and node_set_state() are dummies.

So __build_all_zonelists is using the dummy node_state() and
not actually setting node_states[N_HIGH_MEMORY], and my debug
statements were using the dumy node_state() which returns 1
if n==0 :)

Here is a patch which fixes the bug on my system. As noted in
the patch description, there are other calls to node_set_state() in
mm/page_alloc.c which might also need to be switched to node_set(),
if this is deemed the right solution.

thanks,
-serge

>From 8cffdf9bc60af3894858bcc9412aa80938064435 Mon Sep 17 00:00:00 2001
From: [email protected] <hallyn@kernel.(none)>
Date: Tue, 14 Aug 2007 13:47:51 -0700
Subject: [PATCH 1/1] memoryless: actually set node_states[N_HIGH_MEMORY]

If CONFIG_NODES_SHIFT is unset, then MAX_NUMNODES=1, and
node_set_state() does nothing. This means that __build_all_zonelists()
does not actually set node_states[N_HIGH_MEMORY]. cpusets needs
node_states[N_HIGH_MEMORY] to be correctly set, or it will raise a
bug when a new cpuset is created.

Switch to using node_set() in __build_all_zonelists() to
make sure it is actually set.

Note there are other calls to node_set_state() in mm/page_alloc.c,
which perhaps should also be modified. Since this area is completely
foreign to me I thought I'd first make sure this is the right approach.

Signed-off-by: Serge E. Hallyn <[email protected]>
Signed-off-by: Dhaval Giani <[email protected]>
---
mm/page_alloc.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 1a40099..111e311 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2443,7 +2443,7 @@ static int __build_all_zonelists(void *dummy)

/* Any memory on that node */
if (pgdat->node_present_pages)
- node_set_state(nid, N_HIGH_MEMORY);
+ node_set(nid, node_states[N_HIGH_MEMORY]);
check_for_regular_memory(pgdat);
}
return 0;
--
1.5.1

2007-08-14 21:02:25

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Tue, 2007-08-14 at 14:23 -0500, Serge E. Hallyn wrote:
> Quoting Lee Schermerhorn ([email protected]):
> > On Tue, 2007-08-14 at 13:03 -0500, Serge E. Hallyn wrote:
> > > Quoting Lee Schermerhorn ([email protected]):
> > > > On Mon, 2007-08-13 at 15:12 -0500, Serge E. Hallyn wrote:
> > > > > Quoting Dhaval Giani ([email protected]):
> > > > > > Hi,
> > > > > >
> > > > > > On mounting cpusets using containers, I have been hitting the following
> > > > > > bug.
> > > > > >
> > > > > >
> > > > > > -----------[ cut here ]------------
> > > > > > kernel BUG at kernel/cpuset.c:331!
> > > > <snip>
> > > > > > CONFIG_HIGHMEM=y
> > > > > > CONFIG_X86_PAE=y
> > > > > > # CONFIG_NUMA is not set
> > > > > > CONFIG_ARCH_POPULATES_NODE_MAP=y
> > > > > > CONFIG_SELECT_MEMORY_MODEL=y
> > > > > > CONFIG_FLATMEM_MANUAL=y
> > > > > > # CONFIG_DISCONTIGMEM_MANUAL is not set
> > > > > > # CONFIG_SPARSEMEM_MANUAL is not set
> > > > > > CONFIG_FLATMEM=y
> > > > > > CONFIG_FLAT_NODE_MEM_MAP=y
> > > > > > # CONFIG_SPARSEMEM_STATIC is not set
> > > > > > CONFIG_SPLIT_PTLOCK_CPUS=4
> > > > > > CONFIG_RESOURCES_64BIT=y
> > > > > > CONFIG_ZONE_DMA_FLAG=1
> > > > <snip>
> > > > >
> > > > > Yeah, I'm seeing the same thing. Oddly, my node_states[N_NORMAL_MEMORY]
> > > > > and node_states[N_HIGH_MEMORY] are empty, while node_states[N_ONLINE]
> > > > > contains my single cpu (on i386 kvm image).
> > > > >
> > > > > -serge
> > > >
> > > > Yes, you'll definitely hit that BUG if the N_HIGH_MEMORY mask is empty.
> > > > So far, I can't see how this could be, tho'. __build_all_zonelists()
> > > > should be called for non-NUMA as well as NUMA. It iterates over "all
> > >
> > > Yup, and it is, and some debug statements insist that
> > > node_set_state(nid, N_HIGH_MEMORY) is being called for cpu 0,
> > > and the state is in fact being correctly set.
> > >
> > > So it must get cleared later...
> >
> > OK. That helps. I'll see what I can find...
>
> Well apparently I lied? I don't think it's getting cleared in node_states.
> I'm doing:
>
> static void guarantee_online_mems(const struct cpuset *cs, nodemask_t *pmask)
> {
> int nid=0;
> printk(KERN_NOTICE "%s: at top, node %d is %d\n",
> __FUNCTION__, nid, node_state(nid, N_HIGH_MEMORY));
> while (cs && !nodes_intersects(cs->mems_allowed,
> node_states[N_HIGH_MEMORY])) {
> printk(KERN_NOTICE "%s: in loop\n", __FUNCTION__);
> cs = cs->parent;
> }
> if (cs) {
> printk(KERN_NOTICE "%s: cs was true\n", __FUNCTION__);
> nodes_and(*pmask, cs->mems_allowed,
> node_states[N_HIGH_MEMORY]);
> }
> else
> {
> printk(KERN_NOTICE "%s: cs was false\n", __FUNCTION__);
> *pmask = node_states[N_HIGH_MEMORY];
> }
> printk(KERN_NOTICE "%s: at bottom, node %d is %d\n",
> __FUNCTION__, nid, node_state(nid, N_HIGH_MEMORY));
> printk(KERN_NOTICE "%s: at bottom, in pmask node %d is %d\n",
> __FUNCTION__, nid, node_isset(nid, *pmask));
> BUG_ON(!nodes_intersects(*pmask, node_states[N_HIGH_MEMORY]));
> }
>
> and getting:
>
> guarantee_online_mems: in loop
> guarantee_online_mems: cs was false

The above message indicates that you went off the top of the cpuset
hierarchy w/o finding a non-empty intersection with
node_states[N_HIGH_MEMORY]. The top cpuset should include all nodes
with memory--and there has to be at least one, right?. See
cpuset_init_smp(). That would indicate that node_states[N_HIGH_MEMORY]
was empty when cpuset_init_smp() was called or when you entered
guarantee_online_mems(). One can't tell because you didn't include the
"at top" message [are you seeing it?] and node_state() always returns
true for node zero on non-NUMA platforms [!(MAX_NUMANODES > 1)].

If you're still investigating, I'd suggest a printk in
cpuset_init_smp(). And use node_isset() directly, instead of
node_state. Or you can really look beneath the covers of nodemask_t and
print 'node_states[N_HIGH_MEMORY].bits[0]' using a hex format.

I should note that this works on my ia64 NUMA platform and other folks'
x86_64 platforms. So, it's looking like an i386 specific issue.

> guarantee_online_mems: at bottom, node 0 is 1
> guarantee_online_mems: at bottom, in pmask node 0 is 0
>
>
> So I'm guess that
> *pmask = node_states[N_HIGH_MEMORY];
> isn't doing what it was intended to do?

Should just do the structure assignment, right? That's used pretty
heavily throughout the mem policy and cpuset sources.

Hmmm, including to initialize the top cpuset's mems_allowed. If that
assignment failed, as well, it might indicate that the compiler is
having difficulty handling the indexed array of structs?

Try the patch below to see if the changes the behavior. I didn't change
all of the struct assignments. Just the ones involving the indexed
node_state array element.

> That or I've got node_state()
> wrong...

Uh, no. you're using it correctly. But as I mentioned above, for
MAX_NUMANODES <= 1, node_state() just does "return node == 0;" Better
to use node_isset() directly for this case.

Lee

===========================================================
PATCH -- use memcpy() for node_state[] assignment

to see if this fixes the i386 cpusets empty mems_allowed
problem.

N.B., compile-tested only on ia64.

Signed-off-by: Lee Schermerhorn <[email protected]>

kernel/cpuset.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)

Index: Linux/kernel/cpuset.c
===================================================================
--- Linux.orig/kernel/cpuset.c 2007-08-14 16:28:27.000000000 -0400
+++ Linux/kernel/cpuset.c 2007-08-14 16:37:27.000000000 -0400
@@ -327,7 +327,7 @@ static void guarantee_online_mems(const
nodes_and(*pmask, cs->mems_allowed,
node_states[N_HIGH_MEMORY]);
else
- *pmask = node_states[N_HIGH_MEMORY];
+ memcpy(pmask, &node_states[N_HIGH_MEMORY], sizeof(*pmask));
BUG_ON(!nodes_intersects(*pmask, node_states[N_HIGH_MEMORY]));
}

@@ -1396,7 +1396,8 @@ static void common_cpu_mem_hotplug_unplu

guarantee_online_cpus_mems_in_subtree(&top_cpuset);
top_cpuset.cpus_allowed = cpu_online_map;
- top_cpuset.mems_allowed = node_states[N_HIGH_MEMORY];
+ memcpy(&top_cpuset.mems_allowed, &node_states[N_HIGH_MEMORY],
+ sizeof(top_cpuset.mems_allowed));

mutex_unlock(&callback_mutex);
container_unlock();
@@ -1445,7 +1446,8 @@ void cpuset_track_online_nodes(void)
void __init cpuset_init_smp(void)
{
top_cpuset.cpus_allowed = cpu_online_map;
- top_cpuset.mems_allowed = node_states[N_HIGH_MEMORY];
+ memcpy(&top_cpuset.mems_allowed, &node_states[N_HIGH_MEMORY],
+ sizeof(top_cpuset.mems_allowed));

hotcpu_notifier(cpuset_handle_cpuhp, 0);
}

2007-08-14 21:08:11

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Tue, 2007-08-14 at 15:49 -0500, Serge E. Hallyn wrote:
> Quoting Serge E. Hallyn ([email protected]):
> > Quoting Lee Schermerhorn ([email protected]):
> > > On Tue, 2007-08-14 at 13:03 -0500, Serge E. Hallyn wrote:
> > > > Quoting Lee Schermerhorn ([email protected]):
> > > > > On Mon, 2007-08-13 at 15:12 -0500, Serge E. Hallyn wrote:
> > > > > > Quoting Dhaval Giani ([email protected]):
> > > > > > > Hi,
> > > > > > >
> > > > > > > On mounting cpusets using containers, I have been hitting the following
> > > > > > > bug.
> > > > > > >
> > > > > > >
> > > > > > > -----------[ cut here ]------------
> > > > > > > kernel BUG at kernel/cpuset.c:331!
> > > > > <snip>
> > > > > > > CONFIG_HIGHMEM=y
> > > > > > > CONFIG_X86_PAE=y
> > > > > > > # CONFIG_NUMA is not set
> > > > > > > CONFIG_ARCH_POPULATES_NODE_MAP=y
> > > > > > > CONFIG_SELECT_MEMORY_MODEL=y
> > > > > > > CONFIG_FLATMEM_MANUAL=y
> > > > > > > # CONFIG_DISCONTIGMEM_MANUAL is not set
> > > > > > > # CONFIG_SPARSEMEM_MANUAL is not set
> > > > > > > CONFIG_FLATMEM=y
> > > > > > > CONFIG_FLAT_NODE_MEM_MAP=y
> > > > > > > # CONFIG_SPARSEMEM_STATIC is not set
> > > > > > > CONFIG_SPLIT_PTLOCK_CPUS=4
> > > > > > > CONFIG_RESOURCES_64BIT=y
> > > > > > > CONFIG_ZONE_DMA_FLAG=1
> > > > > <snip>
> > > > > >
> > > > > > Yeah, I'm seeing the same thing. Oddly, my node_states[N_NORMAL_MEMORY]
> > > > > > and node_states[N_HIGH_MEMORY] are empty, while node_states[N_ONLINE]
> > > > > > contains my single cpu (on i386 kvm image).
> > > > > >
> > > > > > -serge
> > > > >
> > > > > Yes, you'll definitely hit that BUG if the N_HIGH_MEMORY mask is empty.
> > > > > So far, I can't see how this could be, tho'. __build_all_zonelists()
> > > > > should be called for non-NUMA as well as NUMA. It iterates over "all
> > > >
> > > > Yup, and it is, and some debug statements insist that
> > > > node_set_state(nid, N_HIGH_MEMORY) is being called for cpu 0,
> > > > and the state is in fact being correctly set.
> > > >
> > > > So it must get cleared later...
> > >
> > > OK. That helps. I'll see what I can find...
> >
> > Well apparently I lied? I don't think it's getting cleared in node_states.
> > I'm doing:
> >
> > static void guarantee_online_mems(const struct cpuset *cs, nodemask_t *pmask)
> > {
> > int nid=0;
> > printk(KERN_NOTICE "%s: at top, node %d is %d\n",
> > __FUNCTION__, nid, node_state(nid, N_HIGH_MEMORY));
> > while (cs && !nodes_intersects(cs->mems_allowed,
> > node_states[N_HIGH_MEMORY])) {
> > printk(KERN_NOTICE "%s: in loop\n", __FUNCTION__);
> > cs = cs->parent;
> > }
> > if (cs) {
> > printk(KERN_NOTICE "%s: cs was true\n", __FUNCTION__);
> > nodes_and(*pmask, cs->mems_allowed,
> > node_states[N_HIGH_MEMORY]);
> > }
> > else
> > {
> > printk(KERN_NOTICE "%s: cs was false\n", __FUNCTION__);
> > *pmask = node_states[N_HIGH_MEMORY];
> > }
> > printk(KERN_NOTICE "%s: at bottom, node %d is %d\n",
> > __FUNCTION__, nid, node_state(nid, N_HIGH_MEMORY));
> > printk(KERN_NOTICE "%s: at bottom, in pmask node %d is %d\n",
> > __FUNCTION__, nid, node_isset(nid, *pmask));
> > BUG_ON(!nodes_intersects(*pmask, node_states[N_HIGH_MEMORY]));
> > }
> >
> > and getting:
> >
> > guarantee_online_mems: in loop
> > guarantee_online_mems: cs was false
> > guarantee_online_mems: at bottom, node 0 is 1
> > guarantee_online_mems: at bottom, in pmask node 0 is 0
> >
> >
> > So I'm guess that
> > *pmask = node_states[N_HIGH_MEMORY];
> > isn't doing what it was intended to do? That or I've got node_state()
> > wrong...
>
> Argh.
>
> CONFIG_NODES_SHIFT was unset, so MAX_NUMNODES=1, so
> node_state() and node_set_state() are dummies.
>
> So __build_all_zonelists is using the dummy node_state() and
> not actually setting node_states[N_HIGH_MEMORY], and my debug
> statements were using the dumy node_state() which returns 1
> if n==0 :)

Yeah, I saw that, but I was barking up the wrong tree vis a vis the
structure assignment vs memcpy() :-( Please disregard previous mail.

>
> Here is a patch which fixes the bug on my system. As noted in
> the patch description, there are other calls to node_set_state() in
> mm/page_alloc.c which might also need to be switched to node_set(),
> if this is deemed the right solution.

Yeah. I think that in the node_state initialization functions, we want
to avoid the node_state() wrappers. They do the right thing for users
of the interfaces, but not for setting up the state masks.

>
> thanks,
> -serge

Thank you for tracking this down.

Lee

2007-08-14 21:52:25

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Tue, 14 Aug 2007, Serge E. Hallyn wrote:

> CONFIG_NODES_SHIFT was unset, so MAX_NUMNODES=1, so
> node_state() and node_set_state() are dummies.

Ok then you did not have a NUMA system configured. So its okay for the
dummies to ignore the stuff. CONFIG_NODES_SHIFT is a constant and does not
change. The first bit is always set.

2007-08-14 21:56:57

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Tue, 2007-08-14 at 15:49 -0500, Serge E. Hallyn wrote:
<snip>
>
> Here is a patch which fixes the bug on my system. As noted in
> the patch description, there are other calls to node_set_state() in
> mm/page_alloc.c which might also need to be switched to node_set(),
> if this is deemed the right solution.

Here's an update to that patch to handle the other 2 places I found
where node_set_state() would be a no-op when MAX_NUMNODES == 1.

I'm continuing to look at uses of node_online_map throughout the kernel.
I'll look at current usage of the node state wrappers as well. However,
you probably want to fix these other 2 places as well, or you might run
into other issues with empty 'NORMAL_MEMORY and 'CPU node state masks...

Lee

[PATCH 1/1] memoryless: actually set node_states[N_HIGH_MEMORY] - v2

v1 -> v2"
+ switch to node_set() to initialize N_NORMAL_MEMORY and N_CPU
node_states[] as well.

If CONFIG_NODES_SHIFT is unset, then MAX_NUMNODES=1, and
node_set_state() does nothing. This means that __build_all_zonelists()
does not actually set node_states[N_HIGH_MEMORY]. cpusets needs
node_states[N_HIGH_MEMORY] to be correctly set, or it will raise a
bug when a new cpuset is created.

Switch to using node_set() in __build_all_zonelists() to
make sure it is actually set.

Note there are other calls to node_set_state() in mm/page_alloc.c,
which perhaps should also be modified. Since this area is completely
foreign to me I thought I'd first make sure this is the right approach.

addressed in v2 --lts

Signed-off-by: Serge E. Hallyn <[email protected]>
Signed-off-by: Dhaval Giani <[email protected]>
Signed-off-by: Lee Schermerhorn <[email protected]>

mm/page_alloc.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)

Index: Linux/mm/page_alloc.c
===================================================================
--- Linux.orig/mm/page_alloc.c 2007-08-14 16:48:28.000000000 -0400
+++ Linux/mm/page_alloc.c 2007-08-14 16:57:24.000000000 -0400
@@ -2425,7 +2425,8 @@ static void check_for_regular_memory(pg_
for (zone_type = 0; zone_type <= ZONE_NORMAL; zone_type++) {
struct zone *zone = &pgdat->node_zones[zone_type];
if (zone->present_pages)
- node_set_state(zone_to_nid(zone), N_NORMAL_MEMORY);
+ node_set(zone_to_nid(zone),
+ node_states[N_NORMAL_MEMORY]);
}
#endif
}
@@ -2443,7 +2444,7 @@ static int __build_all_zonelists(void *d

/* Any memory on that node */
if (pgdat->node_present_pages)
- node_set_state(nid, N_HIGH_MEMORY);
+ node_set(nid, node_states[N_HIGH_MEMORY]);
check_for_regular_memory(pgdat);
}
return 0;
@@ -2793,7 +2794,7 @@ static int __cpuinit process_zones(int c
(zone->present_pages / percpu_pagelist_fraction));
}

- node_set_state(node, N_CPU);
+ node_set(node, node_states[N_CPU]);
return 0;
bad:
for_each_zone(dzone) {

2007-08-14 21:58:20

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Tue, 14 Aug 2007, Lee Schermerhorn wrote:

> Here's an update to that patch to handle the other 2 places I found
> where node_set_state() would be a no-op when MAX_NUMNODES == 1.

What is the problem? node_set_state is not needed when !NUMA. All flags
are set for the single node?

2007-08-14 22:13:05

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Tue, 2007-08-14 at 14:28 -0700, Christoph Lameter wrote:
> On Tue, 14 Aug 2007, Serge E. Hallyn wrote:
>
> > CONFIG_NODES_SHIFT was unset, so MAX_NUMNODES=1, so
> > node_state() and node_set_state() are dummies.
>
> Ok then you did not have a NUMA system configured. So its okay for the
> dummies to ignore the stuff. CONFIG_NODES_SHIFT is a constant and does not
> change. The first bit is always set.

The first bit [node 0] is only set for the N_ONLINE [and N_POSSIBLE]
mask. We could add the static init for the other masks, but since
non-numa platforms are going through the __build_all_zonelists, they
might as well set the MEMORY bits explicitly. Or, maybe you'll
disagree ;-).

Lee

2007-08-14 22:14:25

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Tue, 14 Aug 2007, Lee Schermerhorn wrote:

> > Ok then you did not have a NUMA system configured. So its okay for the
> > dummies to ignore the stuff. CONFIG_NODES_SHIFT is a constant and does not
> > change. The first bit is always set.
>
> The first bit [node 0] is only set for the N_ONLINE [and N_POSSIBLE]
> mask. We could add the static init for the other masks, but since
> non-numa platforms are going through the __build_all_zonelists, they
> might as well set the MEMORY bits explicitly. Or, maybe you'll
> disagree ;-).

The bitmaps can be completely ignored if !NUMA.

In the non NUMA case we define

static inline int node_state(int node, enum node_states state)
{
return node == 0;
}

So its always true for node 0. The "bit" is set.

We are trying to get cpusets to work with !NUMA?

2007-08-15 13:43:57

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Tue, 2007-08-14 at 14:56 -0700, Christoph Lameter wrote:
> On Tue, 14 Aug 2007, Lee Schermerhorn wrote:
>
> > > Ok then you did not have a NUMA system configured. So its okay for the
> > > dummies to ignore the stuff. CONFIG_NODES_SHIFT is a constant and does not
> > > change. The first bit is always set.
> >
> > The first bit [node 0] is only set for the N_ONLINE [and N_POSSIBLE]
> > mask. We could add the static init for the other masks, but since
> > non-numa platforms are going through the __build_all_zonelists, they
> > might as well set the MEMORY bits explicitly. Or, maybe you'll
> > disagree ;-).
>
> The bitmaps can be completely ignored if !NUMA.
>
> In the non NUMA case we define
>
> static inline int node_state(int node, enum node_states state)
> {
> return node == 0;
> }
>
> So its always true for node 0. The "bit" is set.

The issue is with the N_*_MEMORY masks. They don't get initialized
properly because node_set_state() is a no-op if !NUMA. So, where we
look for intersections with or where we AND with the N_*_MEMORY masks we
get the empty set.

>
> We are trying to get cpusets to work with !NUMA?

Well, yes. In Serge's case, he's trying to use cpusets with !NUMA.
He'll have to comment on the reasons for that. Looking at all of the
#ifdefs and init/Kconfig, CPUSET does not depend on NUMA--only SMP and
CONTAINERS [altho' methinks CPUSET should select CONTAINERS rather than
depend on it...]. So, you can use cpusets to partition of cpus in
non-NUMA configs.

In the more general case, tho', I'm looking at all uses of the
node_online_map and for_each_online_node, for instances where they
should be replaced with one of the *_MEMORY masks. IMO, generic code
that is compiled independent of any CONFIG option, like NUMA, should
just work, independent of the config. Currently, as Serge has shown,
this is not the case. So, I think we should fix the *_MEMORY maps to be
correctly populated in both the NUMA and !NUMA cases. A couple of
options:

1) just use node_set() when populating the masks,

2) initialize all masks to include at least cpu/node 0 in the !NUMA
case.

Serge chose #1 to fix his problem. I followed his lead to fix the other
2 places where node_set_state() was being used to initialize the NORMAL
memory node mask and the CPU node mask. This will add a few unnecessary
instructions to !NUMA configs, so we could change to #2.

Thoughts?

Lee

2007-08-15 14:31:37

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Quoting Lee Schermerhorn ([email protected]):
> On Tue, 2007-08-14 at 14:56 -0700, Christoph Lameter wrote:
> > On Tue, 14 Aug 2007, Lee Schermerhorn wrote:
> >
> > > > Ok then you did not have a NUMA system configured. So its okay for the
> > > > dummies to ignore the stuff. CONFIG_NODES_SHIFT is a constant and does not
> > > > change. The first bit is always set.
> > >
> > > The first bit [node 0] is only set for the N_ONLINE [and N_POSSIBLE]
> > > mask. We could add the static init for the other masks, but since
> > > non-numa platforms are going through the __build_all_zonelists, they
> > > might as well set the MEMORY bits explicitly. Or, maybe you'll
> > > disagree ;-).
> >
> > The bitmaps can be completely ignored if !NUMA.
> >
> > In the non NUMA case we define
> >
> > static inline int node_state(int node, enum node_states state)
> > {
> > return node == 0;
> > }
> >
> > So its always true for node 0. The "bit" is set.
>
> The issue is with the N_*_MEMORY masks. They don't get initialized
> properly because node_set_state() is a no-op if !NUMA. So, where we
> look for intersections with or where we AND with the N_*_MEMORY masks we
> get the empty set.
>
> >
> > We are trying to get cpusets to work with !NUMA?
>
>
> Well, yes. In Serge's case, he's trying to use cpusets with !NUMA.
> He'll have to comment on the reasons for that. Looking at all of the

So I can lock a container to a cpu on a non-numa machine.

> #ifdefs and init/Kconfig, CPUSET does not depend on NUMA--only SMP and
> CONTAINERS [altho' methinks CPUSET should select CONTAINERS rather than
> depend on it...]. So, you can use cpusets to partition of cpus in
> non-NUMA configs.
>
> In the more general case, tho', I'm looking at all uses of the
> node_online_map and for_each_online_node, for instances where they
> should be replaced with one of the *_MEMORY masks. IMO, generic code
> that is compiled independent of any CONFIG option, like NUMA, should
> just work, independent of the config. Currently, as Serge has shown,
> this is not the case. So, I think we should fix the *_MEMORY maps to be
> correctly populated in both the NUMA and !NUMA cases. A couple of
> options:
>
> 1) just use node_set() when populating the masks,
>
> 2) initialize all masks to include at least cpu/node 0 in the !NUMA
> case.
>
> Serge chose #1 to fix his problem. I followed his lead to fix the other
> 2 places where node_set_state() was being used to initialize the NORMAL
> memory node mask and the CPU node mask. This will add a few unnecessary
> instructions to !NUMA configs, so we could change to #2.
>
> Thoughts?

Paul, is the mems stuff in cpusets only really useful for NUMA cases?
(I think it is... but am not sure) If so I suppose one alternative
could be to just disable that when !NUMA. But disabling cpusets when
!NUMA is completely wrong.

I personally would think that 1) is still the best option. Otherwise
the action

echo $SOME_CPU > /cpusets/set1/cpu
echo $SOME_CPU > /cpusets/set1/mems

works on a numa machine, and is wrong on a non-numa machine. With
option 1, the second part doesn't actually restrict the memory, but
at least /cpusets/set1/mems exists and $SOME_CPU doesn't have to be 0 to
be valid.

-serge

2007-08-15 16:25:28

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Wed, 2007-08-15 at 09:31 -0500, Serge E. Hallyn wrote:
> Quoting Lee Schermerhorn ([email protected]):
> > On Tue, 2007-08-14 at 14:56 -0700, Christoph Lameter wrote:
> > > On Tue, 14 Aug 2007, Lee Schermerhorn wrote:
> > >
> > > > > Ok then you did not have a NUMA system configured. So its okay for the
> > > > > dummies to ignore the stuff. CONFIG_NODES_SHIFT is a constant and does not
> > > > > change. The first bit is always set.
> > > >
> > > > The first bit [node 0] is only set for the N_ONLINE [and N_POSSIBLE]
> > > > mask. We could add the static init for the other masks, but since
> > > > non-numa platforms are going through the __build_all_zonelists, they
> > > > might as well set the MEMORY bits explicitly. Or, maybe you'll
> > > > disagree ;-).
> > >
> > > The bitmaps can be completely ignored if !NUMA.
> > >
> > > In the non NUMA case we define
> > >
> > > static inline int node_state(int node, enum node_states state)
> > > {
> > > return node == 0;
> > > }
> > >
> > > So its always true for node 0. The "bit" is set.
> >
> > The issue is with the N_*_MEMORY masks. They don't get initialized
> > properly because node_set_state() is a no-op if !NUMA. So, where we
> > look for intersections with or where we AND with the N_*_MEMORY masks we
> > get the empty set.
> >
> > >
> > > We are trying to get cpusets to work with !NUMA?

Sounds reasonable.

> >
> >
> > Well, yes. In Serge's case, he's trying to use cpusets with !NUMA.
> > He'll have to comment on the reasons for that. Looking at all of the
>
> So I can lock a container to a cpu on a non-numa machine.
>
> > #ifdefs and init/Kconfig, CPUSET does not depend on NUMA--only SMP and
> > CONTAINERS [altho' methinks CPUSET should select CONTAINERS rather than
> > depend on it...]. So, you can use cpusets to partition of cpus in
> > non-NUMA configs.
> >
> > In the more general case, tho', I'm looking at all uses of the
> > node_online_map and for_each_online_node, for instances where they
> > should be replaced with one of the *_MEMORY masks. IMO, generic code
> > that is compiled independent of any CONFIG option, like NUMA, should
> > just work, independent of the config. Currently, as Serge has shown,
> > this is not the case. So, I think we should fix the *_MEMORY maps to be
> > correctly populated in both the NUMA and !NUMA cases. A couple of
> > options:
> >
> > 1) just use node_set() when populating the masks,
> >
> > 2) initialize all masks to include at least cpu/node 0 in the !NUMA
> > case.
> >
> > Serge chose #1 to fix his problem. I followed his lead to fix the other
> > 2 places where node_set_state() was being used to initialize the NORMAL
> > memory node mask and the CPU node mask. This will add a few unnecessary
> > instructions to !NUMA configs, so we could change to #2.
> >
> > Thoughts?
>
> Paul, is the mems stuff in cpusets only really useful for NUMA cases?
> (I think it is... but am not sure) If so I suppose one alternative
> could be to just disable that when !NUMA. But disabling cpusets when
> !NUMA is completely wrong.
>
> I personally would think that 1) is still the best option. Otherwise
> the action
>
> echo $SOME_CPU > /cpusets/set1/cpu
> echo $SOME_CPU > /cpusets/set1/mems
>
> works on a numa machine, and is wrong on a non-numa machine. With
> option 1, the second part doesn't actually restrict the memory, but
> at least /cpusets/set1/mems exists and $SOME_CPU doesn't have to be 0 to
> be valid.

Well, you really shouldn't be writing cpu ids to the cpuset mems file.
Rather, it takes node ids. And on !NUMA configs, only node 0 exists.

Can you actually write a !0 cpuid to the mems file with the current
option #1 patch [that uses node_set() to populate the node_states[]]?
It should allow something like:

echo 0,1<and maybe others> >/cpusets/set1/mems

As long as one of the specified node ids has memory, it will silently
ignore any that don't.

If you're up for it, you could try the following patch to statically
initialize the node state masks, in place of the "option 1" patch. Be
aware, tho', that I've only tested on my ia64 NUMA platform. I did
compile it [page_alloc.c] without error under allnoconfig.

Lee

-----------------
PATCH Initialize N_*_MEMORY and N_CPU masks for non-NUMA config

Against: 2.6.23-rc2-mm2

Statically initialize the N_*_MEMORY and N_CPU node state masks
for !NUMA configurations. This static initialization is required
because the node_set_state() function becomes a no-op for !NUMA.
Other generic code assumes that these masks are set correctly.

Note that in NUMA configurations, these masks will be populated
correctly, so don't bother with static initialization. No sense
in making assumptions that could be broken down the road, resulting
in extra work for someone to debug. Unlikely, perhaps, but who
needs the aggravation...

Signed-off-by: Lee Schermerhorn <[email protected]>

mm/page_alloc.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)

Index: Linux/mm/page_alloc.c
===================================================================
--- Linux.orig/mm/page_alloc.c 2007-08-15 10:01:23.000000000 -0400
+++ Linux/mm/page_alloc.c 2007-08-15 10:05:41.000000000 -0400
@@ -52,7 +52,14 @@
*/
nodemask_t node_states[NR_NODE_STATES] __read_mostly = {
[N_POSSIBLE] = NODE_MASK_ALL,
- [N_ONLINE] = { { [0] = 1UL } }
+ [N_ONLINE] = { { [0] = 1UL } },
+#ifndef CONFIG_NUMA
+ [N_NORMAL_MEMORY] = { { [0] = 1UL } },
+#ifdef CONFIG_HIGHMEM
+ [N_HIGH_MEMORY] = { { [0] = 1UL } },
+#endif
+ [N_CPU] = { { [0] = 1UL } },
+#endif /* NUMA */
};
EXPORT_SYMBOL(node_states);

2007-08-15 16:30:48

by Paul Jackson

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Lee wrote:
> [altho' methinks CPUSET should select CONTAINERS rather than
> depend on it...]

Good point -- what do you think, Paul Menage?

--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401

2007-08-15 16:32:39

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Quoting Lee Schermerhorn ([email protected]):
> On Wed, 2007-08-15 at 09:31 -0500, Serge E. Hallyn wrote:
> > Quoting Lee Schermerhorn ([email protected]):
> > > On Tue, 2007-08-14 at 14:56 -0700, Christoph Lameter wrote:
> > > > On Tue, 14 Aug 2007, Lee Schermerhorn wrote:
> > > >
> > > > > > Ok then you did not have a NUMA system configured. So its okay for the
> > > > > > dummies to ignore the stuff. CONFIG_NODES_SHIFT is a constant and does not
> > > > > > change. The first bit is always set.
> > > > >
> > > > > The first bit [node 0] is only set for the N_ONLINE [and N_POSSIBLE]
> > > > > mask. We could add the static init for the other masks, but since
> > > > > non-numa platforms are going through the __build_all_zonelists, they
> > > > > might as well set the MEMORY bits explicitly. Or, maybe you'll
> > > > > disagree ;-).
> > > >
> > > > The bitmaps can be completely ignored if !NUMA.
> > > >
> > > > In the non NUMA case we define
> > > >
> > > > static inline int node_state(int node, enum node_states state)
> > > > {
> > > > return node == 0;
> > > > }
> > > >
> > > > So its always true for node 0. The "bit" is set.
> > >
> > > The issue is with the N_*_MEMORY masks. They don't get initialized
> > > properly because node_set_state() is a no-op if !NUMA. So, where we
> > > look for intersections with or where we AND with the N_*_MEMORY masks we
> > > get the empty set.
> > >
> > > >
> > > > We are trying to get cpusets to work with !NUMA?
>
> Sounds reasonable.
>
> > >
> > >
> > > Well, yes. In Serge's case, he's trying to use cpusets with !NUMA.
> > > He'll have to comment on the reasons for that. Looking at all of the
> >
> > So I can lock a container to a cpu on a non-numa machine.
> >
> > > #ifdefs and init/Kconfig, CPUSET does not depend on NUMA--only SMP and
> > > CONTAINERS [altho' methinks CPUSET should select CONTAINERS rather than
> > > depend on it...]. So, you can use cpusets to partition of cpus in
> > > non-NUMA configs.
> > >
> > > In the more general case, tho', I'm looking at all uses of the
> > > node_online_map and for_each_online_node, for instances where they
> > > should be replaced with one of the *_MEMORY masks. IMO, generic code
> > > that is compiled independent of any CONFIG option, like NUMA, should
> > > just work, independent of the config. Currently, as Serge has shown,
> > > this is not the case. So, I think we should fix the *_MEMORY maps to be
> > > correctly populated in both the NUMA and !NUMA cases. A couple of
> > > options:
> > >
> > > 1) just use node_set() when populating the masks,
> > >
> > > 2) initialize all masks to include at least cpu/node 0 in the !NUMA
> > > case.
> > >
> > > Serge chose #1 to fix his problem. I followed his lead to fix the other
> > > 2 places where node_set_state() was being used to initialize the NORMAL
> > > memory node mask and the CPU node mask. This will add a few unnecessary
> > > instructions to !NUMA configs, so we could change to #2.
> > >
> > > Thoughts?
> >
> > Paul, is the mems stuff in cpusets only really useful for NUMA cases?
> > (I think it is... but am not sure) If so I suppose one alternative
> > could be to just disable that when !NUMA. But disabling cpusets when
> > !NUMA is completely wrong.
> >
> > I personally would think that 1) is still the best option. Otherwise
> > the action
> >
> > echo $SOME_CPU > /cpusets/set1/cpu
> > echo $SOME_CPU > /cpusets/set1/mems
> >
> > works on a numa machine, and is wrong on a non-numa machine. With
> > option 1, the second part doesn't actually restrict the memory, but
> > at least /cpusets/set1/mems exists and $SOME_CPU doesn't have to be 0 to
> > be valid.
>
> Well, you really shouldn't be writing cpu ids to the cpuset mems file.
> Rather, it takes node ids. And on !NUMA configs, only node 0 exists.
>
> Can you actually write a !0 cpuid to the mems file with the current
> option #1 patch [that uses node_set() to populate the node_states[]]?
> It should allow something like:
>
> echo 0,1<and maybe others> >/cpusets/set1/mems
>
> As long as one of the specified node ids has memory, it will silently
> ignore any that don't.
>
> If you're up for it, you could try the following patch to statically
> initialize the node state masks, in place of the "option 1" patch. Be
> aware, tho', that I've only tested on my ia64 NUMA platform. I did
> compile it [page_alloc.c] without error under allnoconfig.
>
> Lee
>
> -----------------
> PATCH Initialize N_*_MEMORY and N_CPU masks for non-NUMA config
>
> Against: 2.6.23-rc2-mm2
>
> Statically initialize the N_*_MEMORY and N_CPU node state masks
> for !NUMA configurations. This static initialization is required
> because the node_set_state() function becomes a no-op for !NUMA.
> Other generic code assumes that these masks are set correctly.
>
> Note that in NUMA configurations, these masks will be populated
> correctly, so don't bother with static initialization. No sense
> in making assumptions that could be broken down the road, resulting
> in extra work for someone to debug. Unlikely, perhaps, but who
> needs the aggravation...
>
> Signed-off-by: Lee Schermerhorn <[email protected]>
>
> mm/page_alloc.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> Index: Linux/mm/page_alloc.c
> ===================================================================
> --- Linux.orig/mm/page_alloc.c 2007-08-15 10:01:23.000000000 -0400
> +++ Linux/mm/page_alloc.c 2007-08-15 10:05:41.000000000 -0400
> @@ -52,7 +52,14 @@
> */
> nodemask_t node_states[NR_NODE_STATES] __read_mostly = {
> [N_POSSIBLE] = NODE_MASK_ALL,
> - [N_ONLINE] = { { [0] = 1UL } }
> + [N_ONLINE] = { { [0] = 1UL } },
> +#ifndef CONFIG_NUMA
> + [N_NORMAL_MEMORY] = { { [0] = 1UL } },
> +#ifdef CONFIG_HIGHMEM
> + [N_HIGH_MEMORY] = { { [0] = 1UL } },
> +#endif
> + [N_CPU] = { { [0] = 1UL } },
> +#endif /* NUMA */
> };
> EXPORT_SYMBOL(node_states);

Ok, I'll try this soon as I have a chance. To be honest when I first
looked at this and saw that node_states[] was only being initialized
for the first two entries this was my first instinct, except I had
no idea what the right way to initilaize it was :) Cause I am
completely numa-challenged.

Will let you know how it works.

thanks,
-serge

2007-08-15 16:33:19

by Paul Jackson

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Serge wrote:
> Paul, is the mems stuff in cpusets only really useful for NUMA cases?

I haven't been following this closely, but offhand, my take is that:
1) CONFIG_CPUSETS is primarily useful on SMP or NUMA systems.
2) It can be configured on non-NUMA systems (it still seems to
need CONFIG_SMP ... not sure why.)
3) Kernels configured for CONFIG_CPUSETS can be (and often are)
run on systems with just one CPU, or just one memory node.
4) To the user, running a CONFIG_CPUSETS enabled kernel w/o
CONFIG_NUMA, or on a system with just one memory node
should allow all the usual CPUSET operations, as if
one just had one memory node, number '0'.

If all systems in our universe only had one memory node, then
cpusets wouldn't do memory -- it would not be useful.

But for consistency of interface between various configured systems,
it is useful to have cpusets working the same across both NUMA and
non-NUMA systems. Some systems just happen to have only one memory
node.

--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401

2007-08-15 17:00:24

by Dhaval Giani

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Hi,

On Wed, Aug 15, 2007 at 11:31:42AM -0500, Serge E. Hallyn wrote:
> Quoting Lee Schermerhorn ([email protected]):
> > On Wed, 2007-08-15 at 09:31 -0500, Serge E. Hallyn wrote:
> > > Quoting Lee Schermerhorn ([email protected]):
> > > > On Tue, 2007-08-14 at 14:56 -0700, Christoph Lameter wrote:
> > > > > On Tue, 14 Aug 2007, Lee Schermerhorn wrote:
> > > > >
> Ok, I'll try this soon as I have a chance. To be honest when I first
> looked at this and saw that node_states[] was only being initialized
> for the first two entries this was my first instinct, except I had
> no idea what the right way to initilaize it was :) Cause I am
> completely numa-challenged.
>

I just tried it out and it is not triggering the bug. So i guess it
works :).

--
regards,
Dhaval

I would like to change the world but they don't give me the source code!

2007-08-15 17:08:40

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Quoting Dhaval Giani ([email protected]):
> Hi,
>
> On Wed, Aug 15, 2007 at 11:31:42AM -0500, Serge E. Hallyn wrote:
> > Quoting Lee Schermerhorn ([email protected]):
> > > On Wed, 2007-08-15 at 09:31 -0500, Serge E. Hallyn wrote:
> > > > Quoting Lee Schermerhorn ([email protected]):
> > > > > On Tue, 2007-08-14 at 14:56 -0700, Christoph Lameter wrote:
> > > > > > On Tue, 14 Aug 2007, Lee Schermerhorn wrote:
> > > > > >
> > Ok, I'll try this soon as I have a chance. To be honest when I first
> > looked at this and saw that node_states[] was only being initialized
> > for the first two entries this was my first instinct, except I had
> > no idea what the right way to initilaize it was :) Cause I am
> > completely numa-challenged.
> >
>
> I just tried it out and it is not triggering the bug. So i guess it
> works :).

Great, thanks for checking.

I vote let's go with Lee's patch, then.

-serge

2007-08-15 17:12:23

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Quoting Paul Jackson ([email protected]):
> Lee wrote:
> > [altho' methinks CPUSET should select CONTAINERS rather than
> > depend on it...]
>
> Good point -- what do you think, Paul Menage?

Paul mentioned (http://www.spinics.net/lists/linux-containers/msg03775.html)
that he was asked not to use select in Kconfigs any more.

-serge

2007-08-15 18:07:58

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Wed, 2007-08-15 at 12:12 -0500, Serge E. Hallyn wrote:
> Quoting Paul Jackson ([email protected]):
> > Lee wrote:
> > > [altho' methinks CPUSET should select CONTAINERS rather than
> > > depend on it...]
> >
> > Good point -- what do you think, Paul Menage?
>
> Paul mentioned (http://www.spinics.net/lists/linux-containers/msg03775.html)
> that he was asked not to use select in Kconfigs any more.

That is unfortunate. I view CPUSET is a "first class" option that
happens to be built on CONTAINERS. IMO, I shouldn't have to know this
implementation detail in order to select CPUSET--indeed, even to be
offered the optio--in 'make *config'. However, it will be hidden if I
haven't already selected containers. Or do I misunderstand the
implications of "depends on"?

Andrew: is the problem with select something fatal or just too
difficult to fix?

Lee

2007-08-15 18:08:41

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Wed, 2007-08-15 at 22:22 +0530, Dhaval Giani wrote:
> Hi,
>
> On Wed, Aug 15, 2007 at 11:31:42AM -0500, Serge E. Hallyn wrote:
> > Quoting Lee Schermerhorn ([email protected]):
> > > On Wed, 2007-08-15 at 09:31 -0500, Serge E. Hallyn wrote:
> > > > Quoting Lee Schermerhorn ([email protected]):
> > > > > On Tue, 2007-08-14 at 14:56 -0700, Christoph Lameter wrote:
> > > > > > On Tue, 14 Aug 2007, Lee Schermerhorn wrote:
> > > > > >
> > Ok, I'll try this soon as I have a chance. To be honest when I first
> > looked at this and saw that node_states[] was only being initialized
> > for the first two entries this was my first instinct, except I had
> > no idea what the right way to initilaize it was :) Cause I am
> > completely numa-challenged.
> >
>
> I just tried it out and it is not triggering the bug. So i guess it
> works :).
>

Thanks for testing. If Christoph agrees, I'll repost to -mm to request
merging.

Lee

2007-08-15 18:34:29

by Andrew Morton

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Wed, 15 Aug 2007 14:00:29 -0400
Lee Schermerhorn <[email protected]> wrote:

> On Wed, 2007-08-15 at 12:12 -0500, Serge E. Hallyn wrote:
> > Quoting Paul Jackson ([email protected]):
> > > Lee wrote:
> > > > [altho' methinks CPUSET should select CONTAINERS rather than
> > > > depend on it...]
> > >
> > > Good point -- what do you think, Paul Menage?
> >
> > Paul mentioned (http://www.spinics.net/lists/linux-containers/msg03775.html)
> > that he was asked not to use select in Kconfigs any more.
>
> That is unfortunate. I view CPUSET is a "first class" option that
> happens to be built on CONTAINERS. IMO, I shouldn't have to know this
> implementation detail in order to select CPUSET--indeed, even to be
> offered the optio--in 'make *config'. However, it will be hidden if I
> haven't already selected containers. Or do I misunderstand the
> implications of "depends on"?
>
> Andrew: is the problem with select something fatal or just too
> difficult to fix?
>

I don't recall what specifically went wrong when we were using select
in there, but it was pretty ugly.

We're hitting more and more problems with this "select is
broken"-versus-"depends-on user inferface sucks" problem. I do think that
a suitable fix could be found at the user interface level: show the
"cpusets" option to the user, but don't let them select it. Instead, tell
the user what needs to be enabled so they can access this option.

But I haven't seen the slightest sign of anyone thinking about doing
anything along those lines.

2007-08-15 20:37:29

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Wed, 15 Aug 2007, Lee Schermerhorn wrote:

> > So its always true for node 0. The "bit" is set.
>
> The issue is with the N_*_MEMORY masks. They don't get initialized
> properly because node_set_state() is a no-op if !NUMA. So, where we
> look for intersections with or where we AND with the N_*_MEMORY masks we
> get the empty set.

That is intentional. Memory is always present if you are on !NUMA. You can
simply use a default nodemask where only node 0 is set. That is what the
fallback provides. Maybe it does not provide the right thing for cpusets?

> > We are trying to get cpusets to work with !NUMA?
>
> Well, yes. In Serge's case, he's trying to use cpusets with !NUMA.
> He'll have to comment on the reasons for that. Looking at all of the
> #ifdefs and init/Kconfig, CPUSET does not depend on NUMA--only SMP and
> CONTAINERS [altho' methinks CPUSET should select CONTAINERS rather than
> depend on it...]. So, you can use cpusets to partition of cpus in
> non-NUMA configs.

Looks like we need to fix cpuset nodemasks for the !NUMA case then?
It cannot expect to find valid nodemasks if !NUMA.

> In the more general case, tho', I'm looking at all uses of the
> node_online_map and for_each_online_node, for instances where they
> should be replaced with one of the *_MEMORY masks. IMO, generic code
> that is compiled independent of any CONFIG option, like NUMA, should
> just work, independent of the config. Currently, as Serge has shown,

AFAIK this works except for cpusets.

> this is not the case. So, I think we should fix the *_MEMORY maps to be
> correctly populated in both the NUMA and !NUMA cases. A couple of
> options:

There is no point in having a variable if you know the results because of
!NUMA. That is the way nodemask.h has always operated.

> Thoughts?

Lets get either rid of the definitions for the nodemasks in the !NUMA
case or fix their contents to have the right constant value expected in
cpusets.

2007-08-15 20:39:47

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Wed, 15 Aug 2007, Serge E. Hallyn wrote:

> > Index: Linux/mm/page_alloc.c
> > ===================================================================
> > --- Linux.orig/mm/page_alloc.c 2007-08-15 10:01:23.000000000 -0400
> > +++ Linux/mm/page_alloc.c 2007-08-15 10:05:41.000000000 -0400
> > @@ -52,7 +52,14 @@
> > */
> > nodemask_t node_states[NR_NODE_STATES] __read_mostly = {
> > [N_POSSIBLE] = NODE_MASK_ALL,
> > - [N_ONLINE] = { { [0] = 1UL } }
> > + [N_ONLINE] = { { [0] = 1UL } },
> > +#ifndef CONFIG_NUMA
> > + [N_NORMAL_MEMORY] = { { [0] = 1UL } },
> > +#ifdef CONFIG_HIGHMEM
> > + [N_HIGH_MEMORY] = { { [0] = 1UL } },
> > +#endif
> > + [N_CPU] = { { [0] = 1UL } },
> > +#endif /* NUMA */
> > };
> > EXPORT_SYMBOL(node_states);

> Will let you know how it works.

The patch above looks like we are heading in the right direction.

2007-08-15 20:40:40

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Wed, 15 Aug 2007, Lee Schermerhorn wrote:

> Thanks for testing. If Christoph agrees, I'll repost to -mm to request
> merging.

Please do. Looks right from what I can tell so far.

2007-08-15 20:49:49

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On Wed, 2007-08-15 at 13:36 -0700, Christoph Lameter wrote:
> On Wed, 15 Aug 2007, Lee Schermerhorn wrote:
>
> > > So its always true for node 0. The "bit" is set.
> >
> > The issue is with the N_*_MEMORY masks. They don't get initialized
> > properly because node_set_state() is a no-op if !NUMA. So, where we
> > look for intersections with or where we AND with the N_*_MEMORY masks we
> > get the empty set.
>
> That is intentional. Memory is always present if you are on !NUMA. You can
> simply use a default nodemask where only node 0 is set. That is what the
> fallback provides. Maybe it does not provide the right thing for cpusets?
>
> > > We are trying to get cpusets to work with !NUMA?
> >
> > Well, yes. In Serge's case, he's trying to use cpusets with !NUMA.
> > He'll have to comment on the reasons for that. Looking at all of the
> > #ifdefs and init/Kconfig, CPUSET does not depend on NUMA--only SMP and
> > CONTAINERS [altho' methinks CPUSET should select CONTAINERS rather than
> > depend on it...]. So, you can use cpusets to partition of cpus in
> > non-NUMA configs.
>
> Looks like we need to fix cpuset nodemasks for the !NUMA case then?
> It cannot expect to find valid nodemasks if !NUMA.

Well, OK. But Paul really hates #ifdefs in kernel/cpusets.c. He's
asked me to remove them before, so I avoided them here. Cpusets really
should use only nodes with memory--i.e., the N_HIGH_MEMORY state.

>
> > In the more general case, tho', I'm looking at all uses of the
> > node_online_map and for_each_online_node, for instances where they
> > should be replaced with one of the *_MEMORY masks. IMO, generic code
> > that is compiled independent of any CONFIG option, like NUMA, should
> > just work, independent of the config. Currently, as Serge has shown,
>
> AFAIK this works except for cpusets.

So far. I'm replacing other usage of node_online_map with the
N_HIGH_MEMORY mask, as we discussed. I should have that patch ready to
post tomorrow.

>
> > this is not the case. So, I think we should fix the *_MEMORY maps to be
> > correctly populated in both the NUMA and !NUMA cases. A couple of
> > options:
>
> There is no point in having a variable if you know the results because of
> !NUMA. That is the way nodemask.h has always operated.

But, the mask--the N_HIGH_MEMORY array element, that is--is there for
both NUMA and !NUMA [== N_NORMAL_MEMORY for !CONFIG_HIGHMEM]. We just
don't initialize it for the !NUMA case, currently.

>
> > Thoughts?
>
> Lets get either rid of the definitions for the nodemasks in the !NUMA
> case or fix their contents to have the right constant value expected in
> cpusets.

That's what the patch I posted today [option 2] does--statically
initializes the N_*_MEMORY and N_CPU masks to indicate that node 0
exists. Serge and Dhaval have tested it on their platform and it solves
the cpuset mount problem.

Lee

2007-08-16 02:37:00

by Paul Jackson

[permalink] [raw]

Subject: Re: [ckrm-tech] Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Christoph wrote (and Lee quoted):
> Looks like we need to fix cpuset nodemasks for the !NUMA case then?
> It cannot expect to find valid nodemasks if !NUMA.

The code in kernel/cpuset.c should be written as if there are always
valid nodemasks. In the case of !NUMA, it will happen that there is
only one node, but kernel/cpuset.c should not manifest that detail.

--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401

2007-08-16 04:47:17

by Paul Menage

[permalink] [raw]

Subject: Re: [ckrm-tech] Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

On 8/15/07, Paul Jackson <[email protected]> wrote:
> Lee wrote:
> > [altho' methinks CPUSET should select CONTAINERS rather than
> > depend on it...]
>
> Good point -- what do you think, Paul Menage?

That's how I made the configs originally; akpm asked me to invert the
dependencies to use "depends" rather than "select" due to some general
problems with using "select". I agree that "select" seems more
natural.

Paul

2007-08-16 13:28:14

[permalink] [raw]

Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang

Andrew: this one should go with the other memoryless node patches.

Lee

---------------------------------------

PATCH Initialize N_*_MEMORY and N_CPU masks for non-NUMA config

Against: 2.6.23-rc2-mm2

The N_*_MEMORY and N_CPU node state masks are not being initialized
for !NUMA because node_set_state() becomes a no-op for that config.
This causes one to hit a BUG_ON() attempting to mount cpusets to partition
cpus. Discovered and diagnosed by Serge Hallyn and Dhaval Giani.

Statically initialize the N_*_MEMORY and N_CPU node state masks
for !NUMA configurations. This static initialization is required
because the node_set_state() function becomes a no-op for !NUMA.
Other generic code assumes that these masks are set correctly.

Note that in NUMA configurations, these masks will be populated
correctly, so don't bother with static initialization. No sense
in making assumptions that could be broken down the road, resulting
in extra work for someone to debug. Unlikely, perhaps, but who
needs the aggravation...

Tested on i386 !NUMA by Dhaval Giani. Fixes cpuset BUG.

Signed-off-by: Lee Schermerhorn <[email protected]>

mm/page_alloc.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)

Index: Linux/mm/page_alloc.c
===================================================================
--- Linux.orig/mm/page_alloc.c 2007-08-15 10:01:23.000000000 -0400
+++ Linux/mm/page_alloc.c 2007-08-15 10:05:41.000000000 -0400
@@ -52,7 +52,14 @@
*/
nodemask_t node_states[NR_NODE_STATES] __read_mostly = {
[N_POSSIBLE] = NODE_MASK_ALL,
- [N_ONLINE] = { { [0] = 1UL } }
+ [N_ONLINE] = { { [0] = 1UL } },
+#ifndef CONFIG_NUMA
+ [N_NORMAL_MEMORY] = { { [0] = 1UL } },
+#ifdef CONFIG_HIGHMEM
+ [N_HIGH_MEMORY] = { { [0] = 1UL } },
+#endif
+ [N_CPU] = { { [0] = 1UL } },
+#endif /* NUMA */
};
EXPORT_SYMBOL(node_states);

2007-08-16 19:14:52