Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933730AbbLPM4T (ORCPT ); Wed, 16 Dec 2015 07:56:19 -0500 Received: from mail-qk0-f180.google.com ([209.85.220.180]:33569 "EHLO mail-qk0-f180.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753219AbbLPM4S (ORCPT ); Wed, 16 Dec 2015 07:56:18 -0500 MIME-Version: 1.0 In-Reply-To: References: <1449903863.9638.8.camel@gmail.com> <1449931045.7411.8.camel@gmail.com> <566F7715.3020905@redhat.com> Date: Wed, 16 Dec 2015 13:56:17 +0100 Message-ID: Subject: Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel From: Jirka Hladky To: Rik van Riel Cc: Mike Galbraith , linux-kernel@vger.kernel.org, Peter Zijlstra , Ingo Molnar , "kkolakow@redhat.com" Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3605 Lines: 86 Hi Rik, I have redone the bisecting and have new results: # first bad commit: [2a595721a1fa6b684c1c818f379bef834ac3d65e] sched/numa: Convert sched_numa_balancing to a static_branch Could you please have a look what went wrong? Thanks a lot! Jirka git bisect start '--' 'kernel/sched' # good: [6a13feb9c82803e2b815eca72fa7a9f5561d7861] Linux 4.3 git bisect good 6a13feb9c82803e2b815eca72fa7a9f5561d7861 # bad: [527e9316f8ec44bd53d90fb9f611fa7ffff52bb9] Linux 4.4-rc4 git bisect bad 527e9316f8ec44bd53d90fb9f611fa7ffff52bb9 # bad: [b99def8b961448f5b9a550dddeeb718e3975e7a6] sched/core: Rework TASK_DEAD preemption exception git bisect bad b99def8b961448f5b9a550dddeeb718e3975e7a6 # skip: [8cd5601c50603caa195ce86cc465cb04079ed488] sched/fair: Convert arch_scale_cpu_capacity() from weak function to #define git bisect skip 8cd5601c50603caa195ce86cc465cb04079ed488 # bad: [fe19159225d8516f3f57a5fe8f735c01684f0ddd] Merge branch 'sched/urgent' into sched/core, to pick up fixes before applying new changes git bisect bad fe19159225d8516f3f57a5fe8f735c01684f0ddd # good: [78a9c54649ea220065aad9902460a1d137c7eafd] sched/numa: Rename numabalancing_enabled to sched_numa_balancing git bisect good 78a9c54649ea220065aad9902460a1d137c7eafd # bad: [54a21385facbdcd89a78e8c3e5025f04c5f2b59c] sched/fair: Rename scale() to cap_scale() git bisect bad 54a21385facbdcd89a78e8c3e5025f04c5f2b59c # bad: [9e91d61d9b0ca8d865dbd59af8d0d5c5b68003e9] sched/fair: Name utilization related data and functions consistently git bisect bad 9e91d61d9b0ca8d865dbd59af8d0d5c5b68003e9 # bad: [2a595721a1fa6b684c1c818f379bef834ac3d65e] sched/numa: Convert sched_numa_balancing to a static_branch git bisect bad 2a595721a1fa6b684c1c818f379bef834ac3d65e # good: [2b49d84b259fc18e131026e5d38e7855352f71b9] sched/numa: Remove the NUMA sched_feature git bisect good 2b49d84b259fc18e131026e5d38e7855352f71b9 # first bad commit: [2a595721a1fa6b684c1c818f379bef834ac3d65e] sched/numa: Convert sched_numa_balancing to a static_branch On Tue, Dec 15, 2015 at 9:49 AM, Jirka Hladky wrote: > Hi Rik, > > I have reviewed the data and you are right. The trouble is that even > with 4.3 kernel there is 20% change that results will be bad. I have > repeated tests 100 times on 4.3 kernel over the night. In 20 cases I > see that runtime went up from 12 seconds to 28 seconds due to the > wrong NUMA placement. I will try to replay the bisect once again. > > Jirka > > On Tue, Dec 15, 2015 at 3:12 AM, Rik van Riel wrote: >> On 12/14/2015 06:52 PM, Jirka Hladky wrote: >>> Hi all, >>> >>> I have the results of bisecting: >>> >>> first bad commit: [973759c80db96ed4b4c5cb85ac7d48107f801371] Merge tag >>> 'v4.3-rc1' into sched/core, to refresh the branch >>> >>> Could you please have a look at this commit why it has caused the >>> performance regression when running 4 stream benchmarks in parallel on 4 >>> NUMA node server? >> >> That is a merge commit. It contains no actual code changes. >> >>> Please let me know if you need additional data. git bisect log is bellow. >> >> It looks like "git bisect" may have led you astray. >> >> I am not sure what debugging tool to use to figure out which >> of the patches from some merged-in branch caused the issue, >> but hopefully one of the people reading this email know a trick. >> >> -- >> All rights reversed -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/