Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2;
Date:   Wed, 18 Oct 2023 15:07:20 +0800
From:   Oliver Sang <oliver.sang@intel.com>
To:     Greg Kroah-Hartman <gregkh@linuxfoundation.org>
CC:     Max Kellermann <max.kellermann@ionos.com>,
        <oe-lkp@lists.linux.dev>, <lkp@intel.com>,
        Linux Memory Management List <linux-mm@kvack.org>,
        <linux-kernel@vger.kernel.org>, <ying.huang@intel.com>,
        <feng.tang@intel.com>, <fengwei.yin@intel.com>,
        <oliver.sang@intel.com>
Subject: Re: [linux-next:master] [drivers/char/mem]  1b057bd800:
 stress-ng.splice.ops_per_sec -99.8% regression
Message-ID: <ZS+EKPJA+BMhM0yO@xsang-OptiPlex-9020>
References: <202310172247.b9959bd4-oliver.sang@intel.com>
 <2023101759-front-ember-6354@gregkh>
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <2023101759-front-ember-6354@gregkh>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?c76C7dDrxLoH0VfjJKozhv12275kyI7lRjZLdnhNdUChAUEimZuXhz0oAB?=
 =?iso-8859-1?Q?Eiz2jf3IEclfxK5St/YYt5GPdwrBPytRuI9VbvXXxFBBoFjNyuhbMMCgYe?=
 =?iso-8859-1?Q?smSumxz5aUISZIvopX02Ek6m4u5hLUJHXIQck6KutdoVoM64Q2PDPaWCrT?=
 =?iso-8859-1?Q?HmvMxE4zAUP7ALhBJQi6x0sZNMLf12FD8ENs8JFjtVGpKsi/uMn5XaA/EW?=
 =?iso-8859-1?Q?oE03Qd7OU4mg0q8t/3EJHarVkfkV0UsyDrft8xfBwz00o25KfGwDhi1dkg?=
 =?iso-8859-1?Q?wqqjl0+iwb6Q+UfLYfMPbwfIkNel/keoOsHcf3LcVfiufxCCtAwuBaE8TI?=
 =?iso-8859-1?Q?PMsHuj3iym7GWWd9paMz1qPLYVhbw4vpoCMp7Z8abg0bY3Zu1CFupkZCco?=
 =?iso-8859-1?Q?jNiQ/L/6oBgoghL44kDEvCnKi3C5Nv/DKm3W6oFKVFFvM+OOChjzeDT8U/?=
 =?iso-8859-1?Q?gujABli9i8KOXGpSxMo2ICldMnxF7jXGd6HlM/9XchvZx2lcK0N7lqiB26?=
 =?iso-8859-1?Q?YbhnKd0XCwwAbP2/sktcYz1nTADgpUUGxQakV0HCcLI6S4rTCBQrtWe0Bm?=
 =?iso-8859-1?Q?XGVgiQWjvSZNZfBbie3FD3a3k4T87wi0YkeJ+BJJlYxBlo9P3HrXPE1i1b?=
 =?iso-8859-1?Q?yfpCsMPRDkyjBqYVJUOm+O2+0x0iUbN5PSfkNLBQQ2v/SEIIrryQ1uen0i?=
 =?iso-8859-1?Q?Qvb/RyJlEagbym1GLArxYAXJvQeqaEu3qNN4iB0WW/DyFXHV3AVEgjUCnS?=
 =?iso-8859-1?Q?5O5nVeN2QnGC5YZsRYJZDf79UsZVJlFam+wpT7N6gFAYt1d0/IgWNkaoDh?=
 =?iso-8859-1?Q?CxXJS5LgWz6E8T/gdvGq+tjYLUfw/zAo6xCjvpZ7TEqVcgLO0VOcds7L/y?=
 =?iso-8859-1?Q?aByGWtqHy4cvMa1hyCeW3DPuwB956RcKlCr/6aiCXehY1tQoEkVY+lH3mn?=
 =?iso-8859-1?Q?HzNS/KFoJf9pha4pGvq38ci8ldbK1bZT8ij01PPh7LgoyiAkXoJbKHIbNp?=
 =?iso-8859-1?Q?ehPWgN5oS7NPfnp9qDKDh2u54qkvwKYtj433PDdNq8j22Jt3irwhLgbeZS?=
 =?iso-8859-1?Q?gzS9EliEVtjNVdzNHzlZOSU3WhW7NiUaQbXlb/OcA+6FsRg3X0EoeYHkaF?=
 =?iso-8859-1?Q?z3x0FUg00fiLr/HBKYpjkKpKwKrO4jUtmJUYgfOyu69bHIzZ4hc93VwDb/?=
 =?iso-8859-1?Q?TB4fB4Jof/2nyRJvQ8Gu8Bb6kZhC7VdHQiD1i9E8y3gnnWTOWvPajodH/h?=
 =?iso-8859-1?Q?LgOdofXp8opBIx5Nq9xO9zSV9xny8uO5AwBzc3liGfz23g85w6aYR5zngz?=
 =?iso-8859-1?Q?pJE9GMGL4AHZvcDJaDCqb57sU2RHKkRDZ1cXe+idF66t0t9h/9X3WCN2Ik?=
 =?iso-8859-1?Q?aXdWlCSAotLXhjUk51Tjlh5P/MVMgtNyH3mrHwbdOJchfH184rIJ+rIFhO?=
 =?iso-8859-1?Q?TFT59+EXKFWHhPYViLH3o0wYQdMi5PjmgxdLgOnCXhHvfPShqsbtvJDVBS?=
 =?iso-8859-1?Q?44YMb97cXFfK8bBhlWQeoUDbCGMyHw9IkwnrVCaROmqpf7SdHG07bt4oA5?=
 =?iso-8859-1?Q?E9dyuF/GuRR8ZXnDzDbZhn+PVjjdTTyvLLIJzWGpmpyXtRZpz/bALVzKV9?=
 =?iso-8859-1?Q?FVPBGxhqLLJ+E7ZgfnN9YQrfltjVOJ75B3g4r0DHKdi1BmHOyh3lZEqw?=
 =?iso-8859-1?Q?=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8ed42c18-d9e4-4dbf-5889-08dbcfa8e3e8
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB6779.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 07:07:29.1236
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zLKVV2wMntHSF/mHTZzMg2wQn+kOUC5eoI7dDVgvG0/+N6yfuDz7xw7XMvTOODHxz2gaNbsTorRVKgugh2Exaw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB6567
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 18 Oct 2023 00:07:49 -0700 (PDT)

hi, Greg Kroah-Hartman,

On Tue, Oct 17, 2023 at 06:56:56PM +0200, Greg Kroah-Hartman wrote:
> On Tue, Oct 17, 2023 at 11:06:42PM +0800, kernel test robot wrote:
> > 
> > 
> > Hello,
> > 
> > kernel test robot noticed a -99.8% regression of stress-ng.splice.ops_per_sec on:
> > 
> > 
> > commit: 1b057bd800c3ea0c926191d7950cd2365eddc9bb ("drivers/char/mem: implement splice() for /dev/zero, /dev/full")
> > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master
> > 
> > testcase: stress-ng
> > test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
> > parameters:
> > 
> > 	nr_threads: 100%
> > 	testtime: 60s
> > 	class: pipe
> > 	test: splice
> > 	cpufreq_governor: performance
> > 
> > 
> > In addition to that, the commit also has significant impact on the following tests:
> > 
> > +------------------+-------------------------------------------------------------------------------------------------+
> > | testcase: change | stress-ng: stress-ng.splice.ops_per_sec 38.9% improvement                                       |
> 
> So everything now goes faster, right?  -99.8% regression means 99.8%
> faster?

let me clarify.

our auto bisect captured this commit as 'first bad commit' in two tests.

Test 1:

it found a (very big) regression comparing to parent commit.

19e3e6cdfdc73400 1b057bd800c3ea0c926191d7950 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
  12433266           -99.8%      22893 ?  3%  stress-ng.splice.ops_per_sec

the detail data for parent in multi-runs:
  "stress-ng.splice.ops_per_sec": [
    12444442.19,
    12599010.87,
    12416009.38,
    12494132.89,
    12286766.76,
    12359235.82
  ],

for 1b057bd800:
  "stress-ng.splice.ops_per_sec": [
    24055.57,
    23235.46,
    22142.13,
    23782.13,
    21732.13,
    22415.46
  ],

so this is much slower.

the config for this Test 1 is:
testcase: stress-ng
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
parameters:
	nr_threads: 100%
	testtime: 60s
	class: pipe
	test: splice
	cpufreq_governor: performance


Test 2:

this is still a stress-ng test, but the config is different with Test 1
(the bare metal machine config, and stress-ng parameters):

testcase: stress-ng
test machine: 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz (Cascade Lake) with 128G memory
parameters:
	nr_threads=1
	testtime=60s
	class=os
	test=splice
	disk=1HDD
	fs=ext4
	cpufreq_governor=performance

Test 2 shows a big improvement:

19e3e6cdfdc73400 1b057bd800c3ea0c926191d7950 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    171798           +38.9%     238710 ?  4%  stress-ng.splice.ops_per_sec

the detail data:
for parent:
  "stress-ng.splice.ops_per_sec": [
    173056.44,
    172030.08,
    171401.68,
    171694.23,
    171001.19,
    171606.93
  ],

for 1b057bd800:
  "stress-ng.splice.ops_per_sec": [
    244347.89,
    259085.63,
    231423.88,
    232897.93,
    226714.77,
    237792.34
  ],


there are monitoring data such like perf data in original report. FYI

> 
> thanks,
> 
> greg k-h