Linus,
Erich Focht has written scheduler extensions in support of
NUMA systems. These extensions are being used at customer
sites. I have branched off and done some similar NUMA scheduler
extensions, though on a much smaller scale. We have combined
efforts and produced two patches which provide minimal NUMA
scheduler capabilities.
The first patch provides node awareness to the load balancer.
This is derived directly from Erich's Node aware NUMA scheduler.
The second patch adds load balancing at exec. This is derived
from work that I have done. Together, these patches provide
performance gains for kernel compilation of between 5 and 10%.
On memory bandwidth extensive microbenchmarks we have seen gains
of 50-100%.
Please consider for inclusion in 2.5.
--
Michael Hohnbaum 503-578-5486
[email protected] T/L 775-5486
On Thu, 2002-10-31 at 15:31, Michael Hohnbaum wrote:
> Linus,
>
> Erich Focht has written scheduler extensions in support of
> NUMA systems. These extensions are being used at customer
> sites. I have branched off and done some similar NUMA scheduler
> extensions, though on a much smaller scale. We have combined
> efforts and produced two patches which provide minimal NUMA
> scheduler capabilities.
>
> The first patch provides node awareness to the load balancer.
> This is derived directly from Erich's Node aware NUMA scheduler.
> The second patch adds load balancing at exec. This is derived
> from work that I have done. Together, these patches provide
> performance gains for kernel compilation of between 5 and 10%.
> On memory bandwidth extensive microbenchmarks we have seen gains
> of 50-100%.
>
> Please consider for inclusion in 2.5.
>
Here is the second patch.
--
Michael Hohnbaum 503-578-5486
[email protected] T/L 775-5486
> Erich Focht has written scheduler extensions in support of
> NUMA systems. These extensions are being used at customer
> sites. I have branched off and done some similar NUMA scheduler
> extensions, though on a much smaller scale. We have combined
> efforts and produced two patches which provide minimal NUMA
> scheduler capabilities.
Just wanted to add that everyone that's been involved in this is
now in harmonious agreement about this combined solution. If you're
curious as to where the benefits come from, the differences in
kernel profiles are included below from a 16-way NUMA-Q doing a
kernel compile.
Positive numbers got worse with the patch, negative got better:
(differences below 50 ticks cut off to increase signal:noise ratio)
132 d_lookup
80 strnlen_user
72 atomic_dec_and_lock
...
-50 file_move
-58 pte_alloc_one
-83 __set_page_dirty_buffers
-84 do_wp_page
-109 free_hot_cold_page
-119 clear_page_tables
-128 __copy_to_user
-175 zap_pte_range
-194 buffered_rmqueue
-237 page_remove_rmap
-897 __copy_from_user
-907 do_anonymous_page
As would be expected most of the gain is in the memory management
functions (do_anonymous page is doing pre-zeroing of pages, and
is always the biggest item on these profiles).
M.
On Thu, 2002-10-31 at 18:52, Martin J. Bligh wrote:
> Just wanted to add that everyone that's been involved in this is
> now in harmonious agreement about this combined solution. If you're
> curious as to where the benefits come from, the differences in
> kernel profiles are included below from a 16-way NUMA-Q doing a
> kernel compile.
Linus, although these patches are fairly straightforward and
non-impacting in the !CONFIG_NUMA case, would you prefer it if a
non-NUMA person who knew the scheduler (say, me) went over these patches
and merged them with you?
Ingo, do you have an opinion either way? I think basic NUMA support,
especially in the load balancer, should make it in before 2.6.
Robert Love
Hum. A last minute change broke UP compilation.
Attatched ... should come out as text/plain so you can read
it, but if it all goes wrong, it just removes:
if (cache_decay_ticks)
cache_decay_ticks=1;
from sched_init.
M.
On Friday 01 November 2002 00:52, Martin J. Bligh wrote:
> > Erich Focht has written scheduler extensions in support of
> > NUMA systems. These extensions are being used at customer
> > sites. I have branched off and done some similar NUMA scheduler
> > extensions, though on a much smaller scale. We have combined
> > efforts and produced two patches which provide minimal NUMA
> > scheduler capabilities.
>
> Just wanted to add that everyone that's been involved in this is
> now in harmonious agreement about this combined solution.
Yes, I'd like to confirm this, too. Just couldn't be online last night
(european time) so I'm glad that Michael sent it all out. Happily
we have a holiday over here today, after Helloween ;-)
Regards,
Erich