Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: 0qTLG80XTVR5pZtTdv2U6FJnVlJN2vRGgD6VUYoQTIuzG6BuXGP4WU78xzL5SQaJ5ZpWGnOydH
 /PULSr5une2w==
IronPort-SDR: paL3/EwV1myg1aP9Iv1Py3BJX2j/Qc1tLpcLyw0bBYKrWCIeYTGoLp50KXWosEPR+9z2tHYrcK
 35fELMgquAMw==
Date:   Thu, 15 Apr 2021 21:17:19 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     Borislav Petkov <bp@suse.de>,
        Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
Cc:     kernel test robot <oliver.sang@intel.com>,
        Peter Zijlstra <peterz@infradead.org>,
        LKML <linux-kernel@vger.kernel.org>, lkp@lists.01.org,
        lkp@intel.com, ying.huang@intel.com, zhengjun.xing@intel.com
Subject: Re: [thermal]  9223d0dccb:  stress-ng.msg.ops_per_sec -27.4%
 regression
Message-ID: <20210415131719.GA77225@shbuild999.sh.intel.com>
References: <20210413135800.GA10266@xsang-OptiPlex-9020>
 <20210413172827.GC17097@zn.tnic>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20210413172827.GC17097@zn.tnic>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk

Hi Boris, Srinivas,

On Tue, Apr 13, 2021 at 07:28:27PM +0200, Borislav Petkov wrote:
> On Tue, Apr 13, 2021 at 09:58:01PM +0800, kernel test robot wrote:
> > Greeting,
> > 
> > FYI, we noticed a -27.4% regression of stress-ng.msg.ops_per_sec due to commit:
> > 
> > 
> > commit: 9223d0dccb8f8523754122f68316dd1a4f39f7f8 ("thermal: Move therm_throt there from x86/mce")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

This seems to be another case that performance jump is caused by 
kernel's data alignment change triggered by an irrelevant patch.  

With a debug patch to force aligned all data sections of .o files,
the performance diff is reduced from -27.4 to -2.8%.

And from perf profile and c2c data, we did see differenc about spinlock
around calling do_msgrcv/do_msgsnd with the 2 commits

> Hmm, so I went and ran your reproducer, but simplified (see end of
> mail), on a KBL box here. The kernel is tip:x86/urgent from last week:
> 
> 5.12.0-rc6+
> -----------
> stress-ng: info:  [1430] dispatching hogs: 9 msg
> stress-ng: info:  [1430] successful run completed in 60.01s (1 min, 0.01 secs)
> stress-ng: info:  [1430] stressor       bogo ops real time  usr time  sys time   bogo ops/s   bogo ops/s
> stress-ng: info:  [1430]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
> stress-ng: info:  [1430] msg           237390147     60.01    104.03    255.85   3955872.56    659636.95
> stress-ng: info:  [1430] for a 60.01s run time:
> stress-ng: info:  [1430]     360.08s available CPU time
> stress-ng: info:  [1430]     104.11s user time   ( 28.91%)
> stress-ng: info:  [1430]     255.93s system time ( 71.08%)
> stress-ng: info:  [1430]     360.04s total time  ( 99.99%)
> stress-ng: info:  [1430] load average: 8.47 3.71 1.48
> 
> Now the same kernel with
> 
> >   4f432e8bb1 ("x86/mce: Get rid of mcheck_intel_therm_init()")
> >   9223d0dccb ("thermal: Move therm_throt there from x86/mce")
> 
> reverted.
> 
> 5.12.0-rc6-rev+
> ---------------
> stress-ng: info:  [1246] dispatching hogs: 9 msg
> stress-ng: info:  [1246] successful run completed in 60.02s (1 min, 0.02 secs)
> stress-ng: info:  [1246] stressor       bogo ops real time  usr time  sys time   bogo ops/s   bogo ops/s
> stress-ng: info:  [1246]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
> stress-ng: info:  [1246] msg           215174467     60.01     99.64    260.24   3585438.79    597906.15
> stress-ng: info:  [1246] for a 60.02s run time:
> stress-ng: info:  [1246]     360.10s available CPU time
> stress-ng: info:  [1246]      99.72s user time   ( 27.69%)
> stress-ng: info:  [1246]     260.32s system time ( 72.29%)
> stress-ng: info:  [1246]     360.04s total time  ( 99.98%)
> stress-ng: info:  [1246] load average: 7.98 2.33 0.80
> 
> so if I'm reading this correctly, reverting the patches here brings the
> *slow-down*.
> 
> What's up?
> 
> reproducer:
> ----------
> 
> #!/usr/bin/bash
> 
> for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
> do
>         online_file="$cpu_dir"/online
>         [ -f "$online_file" ] && [ "$(cat "$online_file")" -eq 0 ] && continue
> 
>         file="$cpu_dir"/cpufreq/scaling_governor
>         [ -f "$file" ] && echo "performance" > "$file"
> done
> 
> stress-ng --timeout 60 --times --verify --metrics-brief --msg 9

The original test case is for 'nr_threads=10%' which turns to '9' for the
96 CPU 2-sockets  Cascade Lake platform. So I guess it may not be reproduced
on 1 socket platform, and sometimes kernel config also matters for
micro-benchmark like 'stress-ng' 

Thanks,
Feng

> -- 
> Regards/Gruss,
>     Boris.
> 
> SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer, HRB 36809, AG Nürnberg