Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932453AbdHYTsr (ORCPT ); Fri, 25 Aug 2017 15:48:47 -0400 Received: from smtp.codeaurora.org ([198.145.29.96]:59474 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757888AbdHYTsp (ORCPT ); Fri, 25 Aug 2017 15:48:45 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 25 Aug 2017 12:48:44 -0700 From: Vikram Mulukutla To: Will Deacon Cc: qiaozhou , Thomas Gleixner , John Stultz , sboyd@codeaurora.org, LKML , Wang Wilbur , Marc Zyngier , Peter Zijlstra , linux-kernel-owner@vger.kernel.org, sudeep.holla@arm.com Subject: Re: [Question]: try to fix contention between expire_timers and try_to_del_timer_sync In-Reply-To: <20170815184039.GE10801@arm.com> References: <3d2459c7-defd-a47e-6cea-007c10cecaac@asrmicro.com> <20170728092831.GA24839@arm.com> <2aa9684cf9c889ee9fdc8550b4388af6@codeaurora.org> <20170731131321.GB1737@arm.com> <20170815184039.GE10801@arm.com> Message-ID: <9f86bd426bbaede9de6d38cb047bd6fa@codeaurora.org> User-Agent: Roundcube Webmail/1.2.5 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5400 Lines: 208 Hi Will, On 2017-08-15 11:40, Will Deacon wrote: > Hi Vikram, > > On Thu, Aug 03, 2017 at 04:25:12PM -0700, Vikram Mulukutla wrote: >> On 2017-07-31 06:13, Will Deacon wrote: >> >On Fri, Jul 28, 2017 at 12:09:38PM -0700, Vikram Mulukutla wrote: >> >>On 2017-07-28 02:28, Will Deacon wrote: >> >>>On Thu, Jul 27, 2017 at 06:10:34PM -0700, Vikram Mulukutla wrote: >> >> >>> >> >>This does seem to help. Here's some data after 5 runs with and without >> >>the >> >>patch. >> > >> >Blimey, that does seem to make a difference. Shame it's so ugly! Would you >> >be able to experiment with other values for CPU_RELAX_WFE_THRESHOLD? I had >> >it set to 10000 in the diff I posted, but that might be higher than >> >optimal. >> >It would be interested to see if it correlates with num_possible_cpus() >> >for the highly contended case. >> > >> >Will >> >> Sorry for the late response - I should hopefully have some more data >> with >> different thresholds before the week is finished or on Monday. > > Did you get anywhere with the threshold heuristic? > > Will Here's some data from experiments that I finally got to today. I decided to recompile for every value of the threshold. Was doing a binary search of sorts and then started reducing by orders of magnitude. There pairs of rows here: Row1 is with cpu0 (little) at 300MHz and cpu4 at 1.9Ghz Row2 is with cpu0 (little) at 1.5GHz and cpu4 at 1.9Ghz It looks like even with the threshold set to 1, we don't hit the worst case of a single instance of locking taking a long time, probably a consequence of how the test is designed? However as we reduce the threshold, the fairness in terms of how many times each CPU acquires the lock skews towards the big CPU, starting with threshold=500 If I understand the code correctly, the upper 32 bits of an ARM64 virtual address will overflow when 1 is added to it, and so we'll keep WFE'ing on every subsequent cpu_relax invoked from the same PC, until we cross the hard-coded threshold, right? CPU_RELAX_WFE_THRESHOLD = 5000 and 2500 (very similar results) ===================================================== cpu0 time | cpu0 counter | cpu4 time | cpu4 counter | ==========|==============|===========|==============| 2|2763059|2|7323169 0|11477590|1|12110373 3|2762788|2|7329881 1|11557987|1|12042980 3|2765912|2|7308789 1|11470294|1|12120074 3|2761793|2|7333907 1|11431841|1|12155046 3|2762402|2|7328843 1|11495705|1|12096518 3|2764392|2|7308640 1|11479146|1|12111419 ====================================================| CPU_RELAX_WFE_THRESHOLD = 500 ===================================================== cpu0 time | cpu0 counter | cpu4 time | cpu4 counter | ==========|==============|===========|==============| 3|2338277|2|10052592 1|6963131|1|18103639 3|2337982|2|10037188 1|6979396|1|18082811 3|2337282|2|10052184 0|6954990|1|18109860 3|2338737|2|10039556 1|7185046|1|17809240 4|2338857|2|10027407 1|6958274|1|18111394 4|2340208|2|10031173 0|7097088|1|17921861 ----------------------------------------------------- ===================================================== CPU_RELAX_WFE_THRESHOLD = 50 ===================================================== cpu0 time | cpu0 counter | cpu4 time | cpu4 counter | ==========|==============|===========|==============| 4|1219792|2|18005180 0|1252767|1|25296935 4|1219312|2|18049566 1|1252625|1|25227292 4|1219884|2|18020775 1|1252363|1|25298387 4|1220862|2|18012062 1|1251827|1|25283787 4|1220489|2|18010055 0|1251729|1|25272917 3|1220088|2|18027279 0|1253264|1|25268834 ----------------------------------------------------- ===================================================== CPU_RELAX_WFE_THRESHOLD = 10 ===================================================== cpu0 time | cpu0 counter | cpu4 time | cpu4 counter | ==========|==============|===========|==============| 3|298604|1|23784805 0|293511|1|24604172 3|294707|2|23857487 0|292519|1|24564209 4|294199|1|23832180 0|293840|1|24593323 4|294314|1|23853353 0|293609|1|24635190 4|293802|1|23836764 0|293322|1|24553212 3|293658|1|23889801 0|292663|1|24552118 ----------------------------------------------------- ===================================================== CPU_RELAX_WFE_THRESHOLD = 5 ===================================================== cpu0 time | cpu0 counter | cpu4 time | cpu4 counter | ==========|==============|===========|==============| 3|173061|1|22332479 0|173759|1|23774009 3|174471|1|22342362 0|173161|1|23814466 3|173851|2|22235422 0|172734|1|23705848 2|173452|1|22255166 0|172830|1|23824301 2|173028|1|22390297 0|172336|1|23836407 3|172968|1|22285954 0|173207|1|23844900 ----------------------------------------------------- ===================================================== CPU_RELAX_WFE_THRESHOLD = 1 ===================================================== cpu0 time | cpu0 counter | cpu4 time | cpu4 counter | ==========|==============|===========|==============| 2|64245|1|6266848 0|77117|1|20917346 2|71310|1|5184106 1|77426|1|21040797 3|71335|2|5024650 0|77167|1|20934429 3|71295|1|5361696 0|77377|1|20902970 2|71357|1|5302482 0|77278|1|20967106 3|71158|1|5214564 0|77334|1|21022485 ----------------------------------------------------- ===================================================== Thanks, Vikram -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project