Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752497AbdGMPUl (ORCPT ); Thu, 13 Jul 2017 11:20:41 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:38899 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751280AbdGMPUk (ORCPT ); Thu, 13 Jul 2017 11:20:40 -0400 Date: Thu, 13 Jul 2017 08:20:32 -0700 From: "Paul E. McKenney" To: Peter Zijlstra Cc: "Li, Aubrey" , Andi Kleen , Frederic Weisbecker , Christoph Lameter , Aubrey Li , tglx@linutronix.de, len.brown@intel.com, rjw@rjwysocki.net, tim.c.chen@linux.intel.com, arjan@linux.intel.com, yang.zhang.wz@gmail.com, x86@kernel.org, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH v1 00/11] Create fast idle path for short idle periods Reply-To: paulmck@linux.vnet.ibm.com References: <20170711094157.5xcwkloxnjehieqv@hirez.programming.kicks-ass.net> <20170711160926.GA18805@lerouge> <20170711163422.etydkhhtgfthpfi5@hirez.programming.kicks-ass.net> <496d4921-5768-cd1e-654b-38630b7d2e13@linux.intel.com> <20170712083410.ualmvnvzoohyami5@hirez.programming.kicks-ass.net> <20170712213240.GE3441@tassilo.jf.intel.com> <20170713083649.febfflfl5hafkko5@hirez.programming.kicks-ass.net> <16e12e23-6b28-f174-7c4b-4d719225cd3b@linux.intel.com> <20170713145311.z4zxlyd2dospeoqg@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170713145311.z4zxlyd2dospeoqg@hirez.programming.kicks-ass.net> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 17071315-0024-0000-0000-000002AF7E30 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00007361; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000214; SDB=6.00887015; UDB=6.00442828; IPR=6.00667155; BA=6.00005470; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00016213; XFM=3.00000015; UTC=2017-07-13 15:20:37 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17071315-0025-0000-0000-000044BE183A Message-Id: <20170713152032.GM2393@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-07-13_08:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1706020000 definitions=main-1707130240 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1853 Lines: 43 On Thu, Jul 13, 2017 at 04:53:11PM +0200, Peter Zijlstra wrote: > On Thu, Jul 13, 2017 at 10:48:55PM +0800, Li, Aubrey wrote: > > > - totally from arch_cpu_idle_enter entry to arch_cpu_idle_exit return costs > > 9122ns - 15318ns. > > ---- In this period(arch idle), rcu_idle_enter costs 1985ns - 2262ns, rcu_idle_exit > > costs 1813ns - 3507ns > > > > Besides RCU, > > So Paul wants more details on where RCU hurts so we can try to fix. More specifically: rcu_needs_cpu(), rcu_prepare_for_idle(), rcu_cleanup_after_idle(), rcu_eqs_enter(), rcu_eqs_enter_common(), rcu_dynticks_eqs_enter(), do_nocb_deferred_wakeup(), rcu_dynticks_task_enter(), rcu_eqs_exit(), rcu_eqs_exit_common(), rcu_dynticks_task_exit(), rcu_dynticks_eqs_exit(). The first three (rcu_needs_cpu(), rcu_prepare_for_idle(), and rcu_cleanup_after_idle()) should not be significant unless you have CONFIG_RCU_FAST_NO_HZ=y. If you do, it would be interesting to learn how often invoke_rcu_core() is invoked from rcu_prepare_for_idle() and rcu_cleanup_after_idle(), as this can raise softirq. Also rcu_accelerate_cbs() and rcu_try_advance_all_cbs(). Knowing which of these is causing the most trouble might help me reduce the overhead in the current idle path. Also, how big is this system? If you can say, about what is the cost of a cache miss to some other CPU's cache? Thanx, Paul > > the period includes c-state selection on X86, a few timestamp updates > > and a few computations in menu governor. Also, deep HW-cstate latency can be up > > to 100+ microseconds, even if the system is very busy, CPU still has chance to enter > > deep cstate, which I guess some outburst workloads are not happy with it. > > > > That's my major concern without a fast idle path. > > Fixing C-state selection by creating an alternative idle path sounds so > very wrong. >