2008-10-02 23:57:41

by Chris Friesen

[permalink] [raw]
Subject: [bug report] sched: stop_machine() usage causes load balancer to misbehave

I mentioned before that ftrace (specifically the ftraced daemon) seems
to be interfering with the load balancer. After some experimenting, it
appears that any regular calls to stop_machine() will end up confusing
the load balancer.

As an experiment, I disabled ftraced (which would normally result in
correct load balancing) but added a single kernel thread which simply
runs the following loop, where "chrisd2" is a dummy function.

while(1) {
set_current_state(TASK_INTERRUPTIBLE);
schedule_timeout(HZ);
stop_machine(chrisd2, NULL, NULL);
}

With the modified kernel, my testcase shows that the load balancer
doesn't balance--all tasks remain on one cpu while the other one stays idle.

Most of the users of stop_machine() (kprobes on s390, cpu hotplug,
module load/unload, numa_zonelist_order, etc.) don't seem to be called
on a regular basis. Only ftrace behaves this way, which is why it
appeared to be the source of the problem.

I haven't tracked down the specific reasons for the misbehaviour, but it
seems undesirable.

Anyone have any ideas what might be causing this? Is it a problem with
the load balancer, or an unavoidable consequence of what stop_machine()
is doing?

Thanks,

Chris


2008-10-03 00:03:14

by Steven Rostedt

[permalink] [raw]
Subject: Re: [bug report] sched: stop_machine() usage causes load balancer to misbehave


On Thu, 2 Oct 2008, Chris Friesen wrote:

> I mentioned before that ftrace (specifically the ftraced daemon) seems to be
> interfering with the load balancer. After some experimenting, it appears that
> any regular calls to stop_machine() will end up confusing the load balancer.
>
> As an experiment, I disabled ftraced (which would normally result in correct
> load balancing) but added a single kernel thread which simply runs the
> following loop, where "chrisd2" is a dummy function.


Chris,

Thanks for looking into this. I don't have an answer for your question,
but I'll work harder to get the MCOUNT_REC working in PPC.

Does this happen with x86 too?

-- Steve

2008-10-04 05:27:39

by Chris Friesen

[permalink] [raw]
Subject: Re: [bug report] sched: stop_machine() usage causes load balancer to misbehave

Steven Rostedt wrote:
> On Thu, 2 Oct 2008, Chris Friesen wrote:
>
>
>>I mentioned before that ftrace (specifically the ftraced daemon) seems to be
>>interfering with the load balancer. After some experimenting, it appears that
>>any regular calls to stop_machine() will end up confusing the load balancer.
>>
>>As an experiment, I disabled ftraced (which would normally result in correct
>>load balancing) but added a single kernel thread which simply runs the
>>following loop, where "chrisd2" is a dummy function.
>
>
>
> Chris,
>
> Thanks for looking into this. I don't have an answer for your question,
> but I'll work harder to get the MCOUNT_REC working in PPC.
>
> Does this happen with x86 too?

Unfortunately I don't have an x86 box handy for testing...I could
probably get some time on a lab machine next week.

Chris