DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 92F6A4DD4B
Message-ID: <1490818125.28917.11.camel@redhat.com>
Subject: Re: [BUG nohz]: wrong user and system time accounting
From: Rik van Riel <riel@redhat.com>
To: Luiz Capitulino <lcapitulino@redhat.com>
Cc: Wanpeng Li <kernellwp@gmail.com>,
        Frederic Weisbecker <fweisbec@gmail.com>, linux-kernel@vger.kernel.org
Date: Wed, 29 Mar 2017 16:08:45 -0400
In-Reply-To: <20170329131656.1d6cb743@redhat.com>
References: <20170323165512.60945ac6@redhat.com>
        <CANRm+CxcgSP2-x+A822DmHLvFLzFmTptS6oYwYtwVdErTpiB=Q@mail.gmail.com>
        <1490636129.8850.76.camel@redhat.com>
        <20170328132406.7d23579c@redhat.com> <20170329131656.1d6cb743@redhat.com>
Organization: Red Hat, Inc.
Content-Type: text/plain; charset="UTF-8"
Mime-Version: 1.0
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1532
Lines: 42

On Wed, 2017-03-29 at 13:16 -0400, Luiz Capitulino wrote:
> On Tue, 28 Mar 2017 13:24:06 -0400
> Luiz Capitulino <lcapitulino@redhat.com> wrote:
> 
> >  1. In my tracing I'm seeing that sometimes (always?) the
> >     time interval between two timer interrupts is less than 1ms
> 
> I think that's the root cause.
> 
> In this trace, we see the following:
> 
>  1. On CPU15, we transition from user-space to kernel-space because
>     of a timer interrupt (it's the tick)
> 
>  2. vtimer_delta() returns 0, because jiffies didn't change since the
>     last accounting
> 
>  3. While CPU15 is executing in kernel-space, jiffies is updated
>     by CPU0
> 
>  4. When going back to user-space, vtime_delta() returns non-zero
>     and the whole time is accounted for system time (observe how
>     the cputime parameter in account_system_time() is less than 1ms)

In other words, the tick on cpu0 is aligned
with the tick on the nohz_full cpus, and
jiffies is advanced while the nohz_full cpus
with an active tick happen to be in kernel
mode?

Frederic, can you think of any reason why
the tick on nohz_full CPUs would end up aligned
with the tick on cpu0, instead of running at some
random offset?

A random offset, or better yet a somewhat randomized
tick length to make sure that simultaneous ticks are
fairly rare and the vtime sampling does not end up
"in phase" with the jiffies incrementing, could make
the accounting work right again.

Of course, that assumes the above hypothesis is correct :)