Received-SPF: pass (google.com: domain of linux-ext4+bounces-648-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Wed, 3 Jan 2024 13:53:32 +0800
From: Oliver Sang <oliver.sang@intel.com>
To: Zhang Yi <yi.zhang@huawei.com>
CC: <oe-lkp@lists.linux.dev>, <lkp@intel.com>, <linux-kernel@vger.kernel.org>,
	Theodore Ts'o <tytso@mit.edu>, Jan Kara <jack@suse.cz>,
	<linux-ext4@vger.kernel.org>, <ying.huang@intel.com>, <feng.tang@intel.com>,
	<fengwei.yin@intel.com>, <yukuai3@huawei.com>, <oliver.sang@intel.com>
Subject: Re: [linus:master] [jbd2] 6a3afb6ac6: fileio.latency_95th_ms 92.5%
 regression
Message-ID: <ZZT2XInayfY0eRu+@xsang-OptiPlex-9020>
References: <202401021525.a27b9444-oliver.sang@intel.com>
 <dcc72d34-89e1-6181-3556-a1a981256cc6@huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <dcc72d34-89e1-6181-3556-a1a981256cc6@huawei.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?mKtOx/dFc9kR0pDGy8cYpgUUdQ+uY1WDFt94DJ/UyHV10NutVfdq9vVEzJl9?=
 =?us-ascii?Q?f9T84pfsP/S7x5Sr4ZmqOioejqfKSSKa14rTfa75xpXMr1We8g2fqqutot0L?=
 =?us-ascii?Q?JcAlmfcHy3qluyiERjWRcs69w/Jcr+sy6FJdBc5CvuHdALOIyPb3unbOEAUW?=
 =?us-ascii?Q?KE+YkyiF8wfv3oUdszRm6DGEXWe4X8sWoT/5tFzD1oTD9Afej6unL4qXyviF?=
 =?us-ascii?Q?eNQxbT11ZxQOV25z9fXA64/BjAXt0Ker5APt/fQqmRbcSv9FGP6hvNaGmXgp?=
 =?us-ascii?Q?i/jX2bYuHe63p5plIVA/o1tvCSBh/dg4kHceoLe1clzApP/UHgfmGBIlv5HJ?=
 =?us-ascii?Q?f9v8dlweVO6s5k5aPLxDhEvm6xaNRNM2/gtbB1DSVQirokIFc2Ix4bW7OyTt?=
 =?us-ascii?Q?VXSP798UX+tdnYSI5epFYZ9jqv4G5qQascDKoc+s81KR/vpUWAbhrsblNc57?=
 =?us-ascii?Q?yDySnZs5Tb5s17mygVIqOiKeit9UVvT2GrTp6xlLsveKwKtuKr9vIbRtOWb5?=
 =?us-ascii?Q?b/R3vUbeE8zmPm3viJl0sDbTY/VKWS5j5O8S/yLzx2izGiZbJXE3olImTh9Y?=
 =?us-ascii?Q?T75G+s9k+/UgiQEDi6ziCmfeAPSJa6TLITtbOz3FAwiYrPwNugEa0C00VphO?=
 =?us-ascii?Q?ROOeBX/gNZFgAbuvzGgD47LFkvlBcI6pGG8/EiQfNyGUpFVvq8nzTOK9Q3x4?=
 =?us-ascii?Q?pFNGBg671w+4gpOoo0HwaCjc/9RQ1wZVmmL87Y4nipNmo79QqSMDz4vZQ7ol?=
 =?us-ascii?Q?lREY3R24ijp58rFiA70LWves791l25MkyJsd+DaJc06dei6JkGd47WxpHuTx?=
 =?us-ascii?Q?n2w/M3byvns3QZCnqUuxLZtEiw0xGc55teRpkDjhPGN3Y3t5061EGtsGhcxn?=
 =?us-ascii?Q?gRan4o4vXHCQuxtmPp03R7nXVixEdooTFS5U9TI0BCkT6IumBOPZT3vOYH9y?=
 =?us-ascii?Q?QjDqvX4ZKFDmTHGE/aNSrQkGHkPgUVR4fMSXIsqs8oOnRB1UghRHRRXvzq6E?=
 =?us-ascii?Q?trYnnnG3Rf5qx5jTdazsFQkrb4yPt7MJExBtK9iYqRchw7oQFtcu6yebwX4J?=
 =?us-ascii?Q?dNV/e7q78K1obTCGrSukYKjJ+D59eoU0rlFg/8vJCb9geXUnFE5OnJkwwRi1?=
 =?us-ascii?Q?3MaHehCfv11h3ZyC6K94r3pfRFIy+3kHUgX1u80Iq8Uk41DlQy3aFW8BQr8F?=
 =?us-ascii?Q?hPLHvQ2Mo5yuCkNWc2GahzRO3uR4pFe+yU+L/j2a2GrNRjOzXGZ5d13VoNLr?=
 =?us-ascii?Q?LqgCzE1E15vOHTDbORldgT8WXAr5UzIipdyjzRImeBPOZboR/XyoiE4hieVj?=
 =?us-ascii?Q?sSoIknx0GHZErBoR1DJYcxr5vWo1k35vSEgC3aQOV9+kjg78qLny4xS+SNAs?=
 =?us-ascii?Q?MOGXNocM1AbqpLIejDa2XRPROTYs86ZcdgSZH4k8kefJ5OYGGny6aKwKBa0C?=
 =?us-ascii?Q?weZ5isDsdebiyOuJP+3CyxnQ9Y14kfIyz50pB0EYRO4J9gwuHOa90cbj6iNE?=
 =?us-ascii?Q?kiFsHmITltW2Z0aHoGODaaoPjdpl8H2QbJp1lAgb3RkA49LGgvBLLcYhcGM4?=
 =?us-ascii?Q?XqXx1NW0dgIyG4VJXZBL19+f+x/09J88mTjR6H1Hk78MD0caDBxlTatONjY1?=
 =?us-ascii?Q?hQ=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e24f8149-ff25-4849-347f-08dc0c2058c4
X-MS-Exchange-CrossTenant-AuthSource: LV3PR11MB8603.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2024 05:53:45.0719
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: lAVqqRPBiSkD3XDvp0UBHUdDg0B703PRA+/lb1DVg2sDlpn6ZAQj26tZbEINRyhfWDBGvDY1fGu/Fr6sOXmr7w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB7566
X-OriginatorOrg: intel.com

hi, Yi,

On Wed, Jan 03, 2024 at 11:31:39AM +0800, Zhang Yi wrote:
> On 2024/1/2 15:31, kernel test robot wrote:
> > 
> > 
> > Hello,
> > 
> > kernel test robot noticed a 92.5% regression of fileio.latency_95th_ms on:
> 
> Hello,
> 
> This seems a little weird, the tests doesn't use blk-cgroup, and the patch
> increase IO priority in WBT, so there shouldn't be any negative influence in
> theory. I've tested sysbench on my machine with Intel Xeon Gold 6240 CPU,
> 400GB memory with HDD disk, and couldn't reproduce this regression.

we double confirmed the latency_95th_ms data is stable in multi runs on our
test platform (Ice Lake).
however, we cannot gurantee the regression could be reproduced on another test
machine.

our team's goal is to report issues (both performance and func) we observed in
our tests to community to help author/community to review then improve code
quality if possible. so just FYI. it's up to author/community to judge the next
step.

regarding this report, if you want to dig further, we will be very happen to
do further tests if you give us some debug patches. Thanks a lot!


> 
> ==
> Without 6a3afb6ac6 ("jbd2: increase the journal IO's priority")
> ==
> 
>  $ sysbench fileio --events=0 --threads=128 --time=600 --file-test-mode=seqwr --file-total-size=68719476736 --file-io-mode=sync --file-num=1024 run
> 
>   sysbench 1.1.0-df89d34 (using bundled LuaJIT 2.1.0-beta3)
> 
>   Running the test with following options:
>   Number of threads: 128
>   Initializing random number generator from current time
> 
> 
>   Extra file open flags: (none)
>   1024 files, 64MiB each
>   64GiB total file size
>   Block size 16KiB
>   Periodic FSYNC enabled, calling fsync() each 100 requests.
>   Calling fsync() at the end of test, Enabled.
>   Using synchronous I/O mode
>   Doing sequential write (creation) test
>   Initializing worker threads...
> 
>   Threads started!
> 
> 
>   Throughput:
>            read:  IOPS=0.00 0.00 MiB/s (0.00 MB/s)
>            write: IOPS=31961.19 499.39 MiB/s (523.65 MB/s)
>            fsync: IOPS=327500.24
> 
>   Latency (ms):
>            min:                                  0.00
>            avg:                                  0.33
>            max:                                920.89
>            95th percentile:                      0.33
>            sum:                            71212319.19
> 
> ==
> With 6a3afb6ac6 ("jbd2: increase the journal IO's priority")
> ==
> 
>  $ sysbench fileio --events=0 --threads=128 --time=600 --file-test-mode=seqwr --file-total-size=68719476736 --file-io-mode=sync --file-num=1024 run
> 
>   sysbench 1.1.0-df89d34 (using bundled LuaJIT 2.1.0-beta3)
> 
>   Running the test with following options:
>   Number of threads: 128
>   Initializing random number generator from current time
> 
> 
>   Extra file open flags: (none)
>   1024 files, 64MiB each
>   64GiB total file size
>   Block size 16KiB
>   Periodic FSYNC enabled, calling fsync() each 100 requests.
>   Calling fsync() at the end of test, Enabled.
>   Using synchronous I/O mode
>   Doing sequential write (creation) test
>   Initializing worker threads...
> 
>   Threads started!
> 
> 
>   Throughput:
>            read:  IOPS=0.00 0.00 MiB/s (0.00 MB/s)
>            write: IOPS=31710.38 495.47 MiB/s (519.54 MB/s)
>            fsync: IOPS=324931.88
> 
>   Latency (ms):
>            min:                                  0.00
>            avg:                                  0.33
>            max:                               1051.69
>            95th percentile:                      0.32
>            sum:                            71309894.62
> 
> Thanks,
> Yi.
> 
> > 
> > 
> > commit: 6a3afb6ac6dfab158ebdd4b87941178f58c8939f ("jbd2: increase the journal IO's priority")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > 
> > testcase: fileio
> > test machine: 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (Ice Lake) with 128G memory
> > parameters:
> > 
> > 	period: 600s
> > 	nr_threads: 100%
> > 	disk: 1HDD
> > 	fs: ext4
> > 	size: 64G
> > 	filenum: 1024f
> > 	rwmode: seqwr
> > 	iomode: sync
> > 	cpufreq_governor: performance
> >