2013-04-15 13:57:04

by Mats Liljegren

[permalink] [raw]
Subject: Trouble with full nohz and lttng

When I use lttng, I keep getting ticks. Without lttng, no ticks. It's
the same application I run, the same way. The only difference is
whether lttng logging has been started or not.

The trace says "sched" as reason for not disabled ticks. Using "top" I
can't see any extra process/thread being created while my application
runs, my application gets 99-101% of the CPU. There are a couple of
kworker threads (3), a migration thread and a ksoftirq thread on the
same CPU as my application, though "top" gives no indication that they
do any execution.

Is this a known limitation of lttng, or is there something else going on here?

/Mats


2013-04-15 14:10:24

by Frederic Weisbecker

[permalink] [raw]
Subject: Re: Trouble with full nohz and lttng

2013/4/15 Mats Liljegren <[email protected]>:
> When I use lttng, I keep getting ticks. Without lttng, no ticks. It's the
> same application I run, the same way. The only difference is whether lttng
> logging has been started or not.
>
> The trace says "sched" as reason for not disabled ticks. Using "top" I can't
> see any extra process/thread being created while my application runs, my
> application gets 99-101% of the CPU. There are a couple of kworker threads
> (3), a migration thread and a ksoftirq thread on the same CPU as my
> application, though "top" gives no indication that they do any execution.
>
> Is this a known limitation of lttng, or is there something else going on
> here?

I have no idea. But the "sched" reason means there is more than one
task in the runqueue at some point. It can be a kthread indeed. Or a
user task that gets migrated / woken up in the CPU.

You'll need to check the sched:sched_switch tracepoint.

2013-04-16 08:46:36

by Mats Liljegren

[permalink] [raw]
Subject: Re: Trouble with full nohz and lttng

On Mon, Apr 15, 2013 at 4:10 PM, Frederic Weisbecker <[email protected]> wrote:
> 2013/4/15 Mats Liljegren <[email protected]>:
>> When I use lttng, I keep getting ticks. Without lttng, no ticks. It's the
>> same application I run, the same way. The only difference is whether lttng
>> logging has been started or not.
>>
>> The trace says "sched" as reason for not disabled ticks. Using "top" I can't
>> see any extra process/thread being created while my application runs, my
>> application gets 99-101% of the CPU. There are a couple of kworker threads
>> (3), a migration thread and a ksoftirq thread on the same CPU as my
>> application, though "top" gives no indication that they do any execution.
>>
>> Is this a known limitation of lttng, or is there something else going on
>> here?
>
> I have no idea. But the "sched" reason means there is more than one
> task in the runqueue at some point. It can be a kthread indeed. Or a
> user task that gets migrated / woken up in the CPU.
>
> You'll need to check the sched:sched_switch tracepoint.

It is ksoftirqd/1 and kworker/1:1 that are run every now and then, and
I'm unable to move them out of the CPU in question. So apparently
lttng causes more work in either these threads which then prohibits
full nohz mode for me.

Not sure what to do about this though...

/Mats