Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932896AbdC2UJf (ORCPT ); Wed, 29 Mar 2017 16:09:35 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46342 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932818AbdC2UIt (ORCPT ); Wed, 29 Mar 2017 16:08:49 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 92F6A4DD4B Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=riel@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 92F6A4DD4B Message-ID: <1490818125.28917.11.camel@redhat.com> Subject: Re: [BUG nohz]: wrong user and system time accounting From: Rik van Riel To: Luiz Capitulino Cc: Wanpeng Li , Frederic Weisbecker , linux-kernel@vger.kernel.org Date: Wed, 29 Mar 2017 16:08:45 -0400 In-Reply-To: <20170329131656.1d6cb743@redhat.com> References: <20170323165512.60945ac6@redhat.com> <1490636129.8850.76.camel@redhat.com> <20170328132406.7d23579c@redhat.com> <20170329131656.1d6cb743@redhat.com> Organization: Red Hat, Inc. Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Wed, 29 Mar 2017 20:08:48 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1532 Lines: 42 On Wed, 2017-03-29 at 13:16 -0400, Luiz Capitulino wrote: > On Tue, 28 Mar 2017 13:24:06 -0400 > Luiz Capitulino wrote: > > >  1. In my tracing I'm seeing that sometimes (always?) the > >     time interval between two timer interrupts is less than 1ms > > I think that's the root cause. >  > In this trace, we see the following: > >  1. On CPU15, we transition from user-space to kernel-space because >     of a timer interrupt (it's the tick) > >  2. vtimer_delta() returns 0, because jiffies didn't change since the >     last accounting > >  3. While CPU15 is executing in kernel-space, jiffies is updated >     by CPU0 > >  4. When going back to user-space, vtime_delta() returns non-zero >     and the whole time is accounted for system time (observe how >     the cputime parameter in account_system_time() is less than 1ms) In other words, the tick on cpu0 is aligned with the tick on the nohz_full cpus, and jiffies is advanced while the nohz_full cpus with an active tick happen to be in kernel mode? Frederic, can you think of any reason why the tick on nohz_full CPUs would end up aligned with the tick on cpu0, instead of running at some random offset? A random offset, or better yet a somewhat randomized tick length to make sure that simultaneous ticks are fairly rare and the vtime sampling does not end up "in phase" with the jiffies incrementing, could make the accounting work right again. Of course, that assumes the above hypothesis is correct :)