Dinakar Guniguntala's "dynamic sched domains" functionality was been merged
into 2.6.13-rcN, although it was disabled at the last minute in the final
2.6.13 because it triggers a fatal bug for NUMA systems with more than one
CPU per node.
Conceptually, when a user/sysadmin declares a cpu-exclusive (a.k.a.
"isolated") cpuset, the cpuset code calls build_sched_domains() to isolate
the cpu-exclusive CPU(s) such that these CPU(s) only load-balance among
themselves and the remaining CPU(s) only load-balance among themselves.
Thus, the non-isolated CPU(s) spend no load-balancing effort trying to
offload tasks cannot be migrated away from their isolated CPU(s).
Otherwise, for example, if an isolated CPU were to be the systemwide
most-heavily-loaded CPU, then this would effectively disable all dynamic
load-balancing in the Scheduler because the non-isolated CPU(s) would
keep making futile efforts to offload isolated, non-migratable tasks.
Unfortunately, the 2.6.13 bug is that build_sched_domains() expects that
a sched domain will include all the CPUs of each node in the domain; more
accurately, that no node will belong in both an isolated cpuset and a
non-isolated cpuset. Declaring a cpuset that violates this presumption
will produce flawed data structures and will oops the kernel. Hence,
for 2.6.13, the cpuset code that would otherwise call build_sched_domains()
is #ifdef'ed disabled.
To trigger the problem (on a NUMA system with >1 CPUs per node, if the
kernel/cpuset.c disabling #ifdef is removed):
cd /dev/cpuset
mkdir newcpuset
cd newcpuset
echo 0 >cpus
echo 0 >mems
echo 1 >cpu_exclusive
The fix is in three parts:
1) A contribution from Ingo Molnar to pull the arch-specific ia64
build_sched_domains() (et al) routines into kernel/sched.c to form
a unified set of build and destroy routines.
2) My fix to the 2.6.13 problem: dynamically allocate sched_group_nodes[]
and sched_group_allnodes[] for each invocation of build_sched_domains(),
rather than use global arrays for these structures, taking care to
remember kmalloc() addresses so that arch_destroy_sched_domains() can
properly kfree() them.
3) Undo the #ifdef disabling hack that was put into 2.6.13 to disable
dynamic sched domains.
This is a patch against 2.6.13. I have posted a previous similar patch
against 2.6.13-mm1.
John Hawkes