Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 16 May 2023 10:46:17 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     Dave Chinner <david@fromorbit.com>,
        "Darrick J. Wong" <djwong@kernel.org>
CC:     Oliver Sang <oliver.sang@intel.com>,
        Dave Chinner <dchinner@redhat.com>, <oe-lkp@lists.linux.dev>,
        <lkp@intel.com>, <linux-kernel@vger.kernel.org>,
        "Darrick J. Wong" <djwong@kernel.org>, <linux-xfs@vger.kernel.org>,
        <ying.huang@intel.com>, <fengwei.yin@intel.com>,
        <rui.zhang@intel.com>
Subject: Re: [linus:master] [xfs]  2edf06a50f:  fsmark.files_per_sec -5.7%
 regression
Message-ID: <ZGLuea93wBdzJipX@feng-clx>
References: <202305090905.aff4e0e6-oliver.sang@intel.com>
 <20230509065433.GT3223426@dread.disaster.area>
 <20230509071053.GE2651828@dread.disaster.area>
 <ZF3uXe+cjAsfCLic@xsang-OptiPlex-9020>
 <20230512230504.GF3223426@dread.disaster.area>
 <ZGDyAOewWqjY5xvJ@feng-clx>
 <20230515222034.GG3223426@dread.disaster.area>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230515222034.GG3223426@dread.disaster.area>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Vd/UzFtnx1y9egayuj6B9N5ZDMkXaEQVJm1w+mCyOXGEdIiYUiQJI+EGWkwp?=
 =?us-ascii?Q?VoNNz300VsmWmNRO92fQgiZaLqahgOTRFM9/RTJkf+PrUtlz3u/6phRvbkMm?=
 =?us-ascii?Q?723f9iYUqEkBT52BP6OikmzDOjbShwBQU/LcKi+B2RotQy4B4rXywD+1hVBT?=
 =?us-ascii?Q?ucezLrpedteqAzMpXIfgeOV69QD0b+/PCdy4ieTTcMcio5fjrVVXbhjR8TSy?=
 =?us-ascii?Q?gcLMDS0odhX7G7KZIBWayCk7wDTP93koN15k7ui26g3kglzP5F3KwXY6+Q3L?=
 =?us-ascii?Q?AYhN4OJvclA9DuSpn94S4+mzR/xawMDe2Y3mAXZzHkowZjHK5JSftA2d+bvG?=
 =?us-ascii?Q?UujLWe+rTaGBsMjOrRcYfUzScDYF1OFOSXO89+hj8rhgrPFWUNfq2gYBI3VG?=
 =?us-ascii?Q?HWTTYJfIjAeh5vg6G+UoBMwNNHXNj4jzFr2rxR8OOU8UGsY1zwRmRNu8Mn5B?=
 =?us-ascii?Q?/NeL0rjc4/aq1ATe8Tl/kfhfilOfIquMu26oSLPW9CAwbDTYnIhkFoh5AS9+?=
 =?us-ascii?Q?PEVzlR+H1UZsWNuCvv0cMO0EAfnI2waGCq0TxiYLW3VsLfMZwQtPBaKvFIg6?=
 =?us-ascii?Q?0j6QlMoDPKqXoE6VF6enlwbU0cWUMPgVc/LTipTjP922vMCmlYzfY3j+4W17?=
 =?us-ascii?Q?uFsHcZfwPy5xNHWRjT+pj9jnDIHYxMLyezh0g8c8Qf/A8w4TiFUfaqT/splB?=
 =?us-ascii?Q?oe9wVrBCyhfr0TMFfO0s91VT/7m44v+qqfY4PXysCY0YHDcm2BWejzF1ghWk?=
 =?us-ascii?Q?sjpsAD9Cyzj2hyGSreHaNKGAxuZsLP0XwbHm+sEv84yyfkSY6o0fxm//RFLA?=
 =?us-ascii?Q?KuTcsQmCZI2KlCAuAjezM7xktQbh5drnWy4as92c3xCmdESQa2R60rw2rfdq?=
 =?us-ascii?Q?TGJI7nV1w5mQSVczxsKPc6Pnb5qKvHO3jtDBAUE/rLkEDRCj046cZLHk1uSE?=
 =?us-ascii?Q?ooPSHhIUGaxiWOwf8KOBQJYUNGptJCRK7u9lAbHVTwAcXkFZIy/dhBox4cWS?=
 =?us-ascii?Q?NEyhhfs054fGi0Y6mhWlTWeEo9kAwHq++3RIzD0mbxsxndrd1J+o4PYRY7zd?=
 =?us-ascii?Q?s+PMuatt+VGkAS81pWT50JQQUGbr3mbSxkaxvmQaoBSpM7Kul8lxiFcEkNlN?=
 =?us-ascii?Q?6eq3m4fPROzYx20BMW5qlq+OC9bGgW9pT/jB9xtBDr4Q6q2aN0CD9K54HvE2?=
 =?us-ascii?Q?cWtM0b9Bcmdx+GKwwF4JFCQSA2+3hYbLAyrkDZTvU5M0+ggtVsjCAqleekYz?=
 =?us-ascii?Q?Ep52cYnGlg/BfOVLU23C3712g+bQMb7M0Cke+g9v0DM8JDTACYZSqys0FLV7?=
 =?us-ascii?Q?tVZE52STuYGg/b4MlFfrfoNCjubFlwrawn9Pu2/j29xo0br5oYHpOD/PANlR?=
 =?us-ascii?Q?HyGM1vskl+qZCU/QvMioyeX8CxesKVH0sLFEHYswQS4ueUnp818ZyYKu7bP9?=
 =?us-ascii?Q?xoAOrW7RUUPcTY6/boqZJLrDjcns2Lu1WlCVSqf0MD24aRtfTNT16P2uXi5K?=
 =?us-ascii?Q?yOqGtGC0iJiWiRbP0ek9SNIiM2ILJGDNLFZFZr4gGU+3cMcEMUW8LEROfAL1?=
 =?us-ascii?Q?JTvV8Yh7ORR3i+NwltfetqceJgZYnch8D8lr2xF1?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d166a7e4-c482-4ef0-eeb5-08db55b8877f
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2023 02:52:04.0924
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 875otovou7wyHvnyJ+ZC66roxyIrUjZX6NARTOZ/elAXskGZztK81GEVdCm1KxNMUzL+vzgf/82G2NHtVnN57w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR11MB5991
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, May 16, 2023 at 08:20:34AM +1000, Dave Chinner wrote:
[...]
> > 
> > For commit 2edf06a50f5, it seems to change the semantics a little
> > about handling of 'flags' for xfs_alloc_fix_freelist(). With the debug
> > below, the performance is restored.
> > 
> > 
> > ecd788a92460eef4 2edf06a50f5bbe664283f3c55c4 68721405630744da1c07c9c1c3c 
> > ---------------- --------------------------- --------------------------- 
> > 
> >      14349            -5.7%      13527            +0.6%      14437        fsmark.files_per_sec
> >     486.29            +5.8%     514.28            -0.5%     483.70        fsmark.time.elapsed_time
> > 
> > Please help to review if the debug patch miss anything as I don't
> > know the internals of xfs, thanks.
> 
> Well spotted. :)
> 
> The relevant commit dropped the trylock flag, so the perf regression
> and change of allocator behaviour is due to it blocking on a busy AG
> instead of skipping to the next uncontended on and so all
> allocations came from extents in the last block of the free space
> btree in that AG.

Thanks for the confirming and analysis!

Late yesterday, I added trace printk in xfs_alloc_vextent_iterate_ags() 
and it did show the flag has XFS_ALLOC_FLAG_TRYLOCK bit set.

	fs_mark-28005   [016] ..... 14993.945487: xfs_alloc_vextent_iterate_ags: flags = 0x1
	fs_mark-28004   [002] ..... 14993.945487: xfs_alloc_vextent_iterate_ags: flags = 0x1
	fs_mark-27986   [006] ..... 14993.945497: xfs_alloc_vextent_iterate_ags: flags = 0x1

> > 
> > ---
> > diff --git a/fs/xfs/libxfs/xfs_alloc.c b/fs/xfs/libxfs/xfs_alloc.c
> > index 98defd19e09e..8c85cc68c5f4 100644
> > --- a/fs/xfs/libxfs/xfs_alloc.c
> > +++ b/fs/xfs/libxfs/xfs_alloc.c
> > @@ -3246,12 +3246,12 @@ xfs_alloc_vextent_set_fsbno(
> >   */
> >  static int
> >  __xfs_alloc_vextent_this_ag(
> > -	struct xfs_alloc_arg	*args)
> > +	struct xfs_alloc_arg	*args, int flag)
> >  {
> >  	struct xfs_mount	*mp = args->mp;
> >  	int			error;
> >  
> > -	error = xfs_alloc_fix_freelist(args, 0);
> > +	error = xfs_alloc_fix_freelist(args, flag);
> >  	if (error) {
> >  		trace_xfs_alloc_vextent_nofix(args);
> >  		return error;
> > @@ -3289,7 +3289,7 @@ xfs_alloc_vextent_this_ag(
> >  	}
> >  
> >  	args->pag = xfs_perag_get(mp, args->agno);
> > -	error = __xfs_alloc_vextent_this_ag(args);
> > +	error = __xfs_alloc_vextent_this_ag(args, 0);
> >  
> >  	xfs_alloc_vextent_set_fsbno(args, minimum_agno);
> >  	xfs_perag_put(args->pag);
> > @@ -3329,7 +3329,7 @@ xfs_alloc_vextent_iterate_ags(
> >  	args->agno = start_agno;
> >  	for (;;) {
> >  		args->pag = xfs_perag_get(mp, args->agno);
> > -		error = __xfs_alloc_vextent_this_ag(args);
> > +		error = __xfs_alloc_vextent_this_ag(args, flags);
> >  		if (error) {
> >  			args->agbno = NULLAGBLOCK;
> >  			break;
> 
> I don't think this is the right way to fix this.

I see. I understand this commit is in the middle of a series.

I should have stated clearly the debug hack patch was tried
mainly for debugging the regression.

> The code is -very-
> different at the end of the series that this is in the middle of,
> and I need to check what callers now use the trylock behaviour to
> determine how the trylock flag shold be passed around...

Sure. When the patch is fine, we can test it.


> > Also for the turbostat.Bzy_MHz diff, IIUC, 0Day always uses
> > 'performance' cpufreq governor. And as the test case is running
> > 32 thread in a platform with 96 CPUs, there are many CPUs in idle
> > state in average, and I suspect the Bzy_MHz may be calculated 
> > considering those cpufreq and cpuidle factors.
> 
> If "busy MHz" includes the speed of idle CPUs, then it's not really
> a measure of the speed of "busy" CPUs. If what you say is true, then
> it is, at best, badly names - it would just be the "average Mhz",
> right?


I found the turbostat.c in kernel tree tools/power/x86/turbostat/

if (DO_BIC(BIC_Bzy_MHz)) {
	if (has_base_hz)
		outp +=
		    sprintf(outp, "%s%.0f", (printed++ ? delim : ""), base_hz / units * t->aperf / t->mperf);
	else
		outp += sprintf(outp, "%s%.0f", (printed++ ? delim : ""),
				tsc / units * t->aperf / t->mperf / interval_float);
}

Rui Zhang told me the 'aperf' is the actual cpu cycles of a CPU in a
period of time, and it only count when CPU is in C0 state, and will
stop counting when cpu is in idle power state. Like in one second
interval, if the CPU spends 500 ms running at 1000 MHz, and the other
500 ms in idle, then the Bzy_MHz will be shown 500 MHz.

Thanks,
Feng

> -Dave.
> 
> -- 
> Dave Chinner
> david@fromorbit.com