Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753337AbdGNDrh (ORCPT ); Thu, 13 Jul 2017 23:47:37 -0400 Received: from mga06.intel.com ([134.134.136.31]:50614 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753182AbdGNDrg (ORCPT ); Thu, 13 Jul 2017 23:47:36 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.40,357,1496127600"; d="scan'208";a="127168037" Subject: Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods To: paulmck@linux.vnet.ibm.com, Peter Zijlstra Cc: Andi Kleen , Frederic Weisbecker , Christoph Lameter , Aubrey Li , tglx@linutronix.de, len.brown@intel.com, rjw@rjwysocki.net, tim.c.chen@linux.intel.com, arjan@linux.intel.com, yang.zhang.wz@gmail.com, x86@kernel.org, linux-kernel@vger.kernel.org References: <20170711094157.5xcwkloxnjehieqv@hirez.programming.kicks-ass.net> <20170711160926.GA18805@lerouge> <20170711163422.etydkhhtgfthpfi5@hirez.programming.kicks-ass.net> <496d4921-5768-cd1e-654b-38630b7d2e13@linux.intel.com> <20170712083410.ualmvnvzoohyami5@hirez.programming.kicks-ass.net> <20170712213240.GE3441@tassilo.jf.intel.com> <20170713083649.febfflfl5hafkko5@hirez.programming.kicks-ass.net> <16e12e23-6b28-f174-7c4b-4d719225cd3b@linux.intel.com> <20170713145311.z4zxlyd2dospeoqg@hirez.programming.kicks-ass.net> <20170713152032.GM2393@linux.vnet.ibm.com> From: "Li, Aubrey" Message-ID: <58a312d3-bd3d-9427-109b-5d81a5fd7c76@linux.intel.com> Date: Fri, 14 Jul 2017 11:47:32 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <20170713152032.GM2393@linux.vnet.ibm.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1748 Lines: 41 On 2017/7/13 23:20, Paul E. McKenney wrote: > On Thu, Jul 13, 2017 at 04:53:11PM +0200, Peter Zijlstra wrote: >> On Thu, Jul 13, 2017 at 10:48:55PM +0800, Li, Aubrey wrote: >> >>> - totally from arch_cpu_idle_enter entry to arch_cpu_idle_exit return costs >>> 9122ns - 15318ns. >>> ---- In this period(arch idle), rcu_idle_enter costs 1985ns - 2262ns, rcu_idle_exit >>> costs 1813ns - 3507ns >>> >>> Besides RCU, >> >> So Paul wants more details on where RCU hurts so we can try to fix. > > More specifically: rcu_needs_cpu(), rcu_prepare_for_idle(), > rcu_cleanup_after_idle(), rcu_eqs_enter(), rcu_eqs_enter_common(), > rcu_dynticks_eqs_enter(), do_nocb_deferred_wakeup(), > rcu_dynticks_task_enter(), rcu_eqs_exit(), rcu_eqs_exit_common(), > rcu_dynticks_task_exit(), rcu_dynticks_eqs_exit(). > > The first three (rcu_needs_cpu(), rcu_prepare_for_idle(), and > rcu_cleanup_after_idle()) should not be significant unless you have > CONFIG_RCU_FAST_NO_HZ=y. If you do, it would be interesting to learn > how often invoke_rcu_core() is invoked from rcu_prepare_for_idle() > and rcu_cleanup_after_idle(), as this can raise softirq. Also > rcu_accelerate_cbs() and rcu_try_advance_all_cbs(). > > Knowing which of these is causing the most trouble might help me > reduce the overhead in the current idle path. > I don't have details of these functions, I can measure if you want. Do you have preferred workload for the measurement? > Also, how big is this system? If you can say, about what is the cost > of a cache miss to some other CPU's cache? > The system has two NUMA nodes. nproc returns 104. local memory access is ~100 ns and remote memory access is ~200ns, reported by mgen. Does this address your question? Thanks, -Aubrey