Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751187AbbD2Upm (ORCPT ); Wed, 29 Apr 2015 16:45:42 -0400 Received: from g4t3427.houston.hp.com ([15.201.208.55]:36560 "EHLO g4t3427.houston.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750930AbbD2Upk (ORCPT ); Wed, 29 Apr 2015 16:45:40 -0400 Message-ID: <1430340334.2475.11.camel@j-VirtualBox> Subject: Re: [PATCH v2 3/5] sched, timer: Use atomics in thread_group_cputimer to improve scalability From: Jason Low To: Rik van Riel Cc: Peter Zijlstra , Ingo Molnar , Thomas Gleixner , linux-kernel@vger.kernel.org, "Paul E. McKenney" , Andrew Morton , Oleg Nesterov , Frederic Weisbecker , Mel Gorman , Steven Rostedt , Preeti U Murthy , Mike Galbraith , Davidlohr Bueso , Waiman Long , Aswin Chandramouleeswaran , Scott J Norton , jason.low2@hp.com Date: Wed, 29 Apr 2015 13:45:34 -0700 In-Reply-To: <5540ECDA.6060308@redhat.com> References: <1430251224-5764-1-git-send-email-jason.low2@hp.com> <1430251224-5764-4-git-send-email-jason.low2@hp.com> <5540ECDA.6060308@redhat.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.2.3-0ubuntu6 Content-Transfer-Encoding: 7bit Mime-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1873 Lines: 40 On Wed, 2015-04-29 at 10:38 -0400, Rik van Riel wrote: > On 04/28/2015 04:00 PM, Jason Low wrote: > > While running a database workload, we found a scalability issue with itimers. > > > > Much of the problem was caused by the thread_group_cputimer spinlock. > > Each time we account for group system/user time, we need to obtain a > > thread_group_cputimer's spinlock to update the timers. On larger systems > > (such as a 16 socket machine), this caused more than 30% of total time > > spent trying to obtain this kernel lock to update these group timer stats. > > > > This patch converts the timers to 64 bit atomic variables and use > > atomic add to update them without a lock. With this patch, the percent > > of total time spent updating thread group cputimer timers was reduced > > from 30% down to less than 1%. > > > > Note: On 32 bit systems using the generic 64 bit atomics, this causes > > sample_group_cputimer() to take locks 3 times instead of just 1 time. > > However, we tested this patch on a 32 bit system ARM system using the > > generic atomics and did not find the overhead to be much of an issue. > > An explanation for why this isn't an issue is that 32 bit systems usually > > have small numbers of CPUs, and cacheline contention from extra spinlocks > > called periodically is not really apparent on smaller systems. > > I don't see 32 bit systems ever getting so many CPUs > that this becomes an issue :) Yeah, the generic 64 bit atomics are meant to be used on systems with (<=4 or so) CPUs. > > Signed-off-by: Jason Low > > Acked-by: Rik van Riel Thanks! -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/