Received-SPF: pass (google.com: domain of linux-kernel+bounces-18062-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Fri, 5 Jan 2024 11:27:04 -0500
From: "Liam R. Howlett" <Liam.Howlett@Oracle.com>
To: Chuck Lever III <chuck.lever@oracle.com>
Cc: Matthew Wilcox <willy@infradead.org>, Feng Tang <feng.tang@intel.com>,
        "Sang, Oliver" <oliver.sang@intel.com>,
        "oe-lkp@lists.linux.dev" <oe-lkp@lists.linux.dev>, lkp <lkp@intel.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Christian Brauner <brauner@kernel.org>, linux-mm <linux-mm@kvack.org>,
        "Huang, Ying" <ying.huang@intel.com>,
        "Yin, Fengwei" <fengwei.yin@intel.com>
Subject: Re: [linus:master] [shmem]  a2e459555c:  aim9.disk_src.ops_per_sec
 -19.0% regression
Message-ID: <20240105162704.7gulfnerq5tvmjbg@revolver>
Mail-Followup-To: "Liam R. Howlett" <Liam.Howlett@Oracle.com>,
	Chuck Lever III <chuck.lever@oracle.com>,
	Matthew Wilcox <willy@infradead.org>,
	Feng Tang <feng.tang@intel.com>,
	"Sang, Oliver" <oliver.sang@intel.com>,
	"oe-lkp@lists.linux.dev" <oe-lkp@lists.linux.dev>,
	lkp <lkp@intel.com>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Christian Brauner <brauner@kernel.org>,
	linux-mm <linux-mm@kvack.org>, "Huang, Ying" <ying.huang@intel.com>,
	"Yin, Fengwei" <fengwei.yin@intel.com>
References: <202309081306.3ecb3734-oliver.sang@intel.com>
 <C85F44FD-BC7D-421B-9585-C5FDFAEA9400@oracle.com>
 <ZP++GV9WURg1GhoY@xsang-OptiPlex-9020>
 <84984801-F885-4739-B4B3-DE8DE4ABE378@oracle.com>
 <ZQCAYpqu+5iD0rhh@feng-clx>
 <ZQCLdzmtVcjxZWXt@casper.infradead.org>
 <D00399D9-D629-4CE3-AC32-636FD6F06C24@oracle.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <D00399D9-D629-4CE3-AC32-636FD6F06C24@oracle.com>
User-Agent: NeoMutt/20220429
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?bjdvU09NNTF4akptZHFocEVaa1RlK3BETTNiOXgvUzg1NUFWYzZqcmUvai9V?=
 =?utf-8?B?aktlSlpxUU9EODNvSWhuN1ByN0tqUjB1ZG9XWXRQMWQ5cWhKdnJ4b2luNGpU?=
 =?utf-8?B?dTVOQjRrQUJtK3QvUlp1dm1scThWa0VVNTV3TnoyMzBwempCT3pwdGszeFhC?=
 =?utf-8?B?a2xUVmxyUHdRamhUQnU4c2krTkJVK2RReXU1Z2c4SnZlYjNvcDNqSjJ3ZkUw?=
 =?utf-8?B?ZWNkbWNBNmxPUXNLTlYyb3BRSTVmbGZONUg3TythL0UzRVA0YWxnaENRVVRH?=
 =?utf-8?B?QjVWRkphZnp6TVUrdjBxV0RSV0U1NkZUNFltdWJQSmN6OFdQTU4wbmNWcVhh?=
 =?utf-8?B?UFdSN2czVTIrR1pnUG4xUmJTdHg2K1lCcTNRVjUwVWNaVm9VbTFaZWxKb0kr?=
 =?utf-8?B?Z011dUhoZEZZMlozUU9VblhmekdGTUdkZ2NScm51SHJOaDZRQlU4NmoveGNh?=
 =?utf-8?B?dEkxOU5HQWN0MGM3WDN2eW9TQ1NpRVAyU0kwNnFWb091SlVKZ2lQcHdrai9J?=
 =?utf-8?B?TWF0MEN4NTQ2TVRURHQwaCtrVmxzL3ZCT2llMENtSHVrMlduR3hPdk1XVW5r?=
 =?utf-8?B?ZHd1dVhZRnN1NUF3SVZ1Y2tFUHY0dkkyeWN1NjlhQ2IrV09tdHM2U2NodDd1?=
 =?utf-8?B?WkRqNy9NZzQwSTlnZVY5TTdKOXpXY0dHR2tHT0FTQmEwamtBdXlrYzBqT21C?=
 =?utf-8?B?VlV1NWZ1eDNNM1d6NE0yS0doM0N0SmlqWG1RQnFVVFd1UmhBRVdZTHBiRE1O?=
 =?utf-8?B?UnVmYUJRNUM1M0ZMUVhyZDRtYWw3V0lvL1FWalJyM29XaXNXQXh4NkV2cEVW?=
 =?utf-8?B?MlFUdUNtSlJjOHFhT2FJQUVXbkZCNTNqd0VhWFdML09CVzlFdjMyK1lXeU1n?=
 =?utf-8?B?VEhvSjlxMzA5ZEFGekZQM1JSZlZ2QjE1R2VNcHhrYU53T1RvbUxhOU9FUDJF?=
 =?utf-8?B?RzROWVUzeUxKM2xGU3p3M1NIR2t1Q0tvNGxrdHcxS0s2Wllva3VrTkNZS0pK?=
 =?utf-8?B?T3pzeWdzbndWTDNodE9tejB6bXlDOTJVUExmdmU1VERQa2U4RDJ0VGU1ODh1?=
 =?utf-8?B?TTFBTlBXSE9WSnh3MU1aWFdZanFLY1IxZGlBY3Y1ZjhERVljVjhrcnlTLzdD?=
 =?utf-8?B?bjNDVEY3WDFsV3BLMDZtelZJdWxreENoKzBYZ3E0TFVjY1JTeFNNRExrcFQ3?=
 =?utf-8?B?T0o1Mi85UTFSOVZyb0pnSEphMG5aVmJyeUpMUGtYZmxWV1Q3Vjc3VWRoeTZR?=
 =?utf-8?B?dTFHMEVPK1BkNyt6YXBKWVZKQnZvcVczSFdJMVBrdnFNSmJVSzBZdXFoTlBy?=
 =?utf-8?B?OSttam5BYzdiSFRHZjRRVGxyVUhhdld1R2ZONWtUUGh4UVBVWGZ0cmdteTNy?=
 =?utf-8?B?Z1R3NzJQRTFyTzdnL0JuR1h0bGNtSnZCelFCMjc3MEtucUt2VEdCMUlnckM5?=
 =?utf-8?B?Q2dZZ0t2M0xqUG5SVG95WnN6aU1ldnZNTUhYZ2xwMXI1K3NxRGFJUFZUb2Vk?=
 =?utf-8?B?ZWpBV0Y3bW5HK0tVRGIvb09SRDBuc2pzZ1VqeXZtSmtSNHVuVUV6djNEZUtK?=
 =?utf-8?B?TFY3WjlWWXBwQlNYN05CMHpLYXZiMWJ2SXhMQnFNQmdyNkRMc3B0dGV5ZldV?=
 =?utf-8?B?WnZPQUMvd29KbmtyVitQeWt3SFdMSDEvYTJyT3Y2QUxnSzF1T0liSW9GK082?=
 =?utf-8?B?QXBCTm8rSitJK0o5emI0ZVdlNlJJMUhnWC9SQmRHWmhFYm03NkN5ZG1VM0FS?=
 =?utf-8?B?R04xUkRRQ3k3WWxqaXlSQXZ5eCtTa21GZnF3SGtraWFEVGRra1R1YjlYSWhx?=
 =?utf-8?B?ZWZ6cmlRdTFRSlc2Z2RUemltUUVWWXNCK0VRQ1BMbzVab1cwOUs3VkFLQmxG?=
 =?utf-8?B?Nkc4QUVzVjhBSzNnN3JiY0dGQnQ4TzByRVo1QjFJNVhmMTBEaGtZcHQrcEx6?=
 =?utf-8?B?UHk3NStMdjA5RFlENC93YXJkTzdVSWx1MDRJNTFzaFhFZTVSWjN2RHUxcTN3?=
 =?utf-8?B?d3FVZDBSV1B5QVZxbUtJR3AyWWxVNTFHU3dmNkhTY1RwZmpFNW83b0RNWUhj?=
 =?utf-8?B?RG9PYkM0ZVR2UlNURVdYZ1FMYWNxQTdoMUZjenBZUUphcHVXMG5ZRWlhY2s0?=
 =?utf-8?Q?cfvxF8ph9ayDdg6Odfr/HHZDa?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	TyJOFIg3WqVShtm8T49wroEFH8TRAL4O2U8JerVEYFwNb3rQYRxTMcF0lW1vJC6MLqQDYZerOdoEZzwr1rDaGf8IwOkt1WpE9Malbgq5PZ1y4P+2mc9HvJeWP5Zn6cv0GoXjGZJJTFn1sOrh/MIdYkrTBz2vAO1hQVVDmFXlNVEAIFM2yxSrchwTe6CaMbKNdcz1GLK1I2pWGScF+TUVLd/fPeL7CBdrdKGImO9WWMlX4DCDpVYKIRm0fmwEjseWvdm6DjubOlVaLm9+9atHd0JoW2OePSzLA/Q1yf+upFnvg3ynFW9ODm2FykLpG//FBUPRQZVjEUxHjnoOo723dJhYloeVhki6edeLjvvlOu9Dsq+jE41jhOLxTJeCDK7A7AvglDkf1TiWfQXNdcEONddtprqTErdFQdxCfKXnG2GFD/rzwMFl8r+2gA+bbssGzvtfaTkojN9mTFEfSn9SYb08IYTppAyPdLE985b78lS5ElCDVHwCrtTyZQEFr4hBSMX3Fdf78reQOyVYEjEJD3wz/fon+nvj5SyG+1LoFGc7kLx5Qm4RTmGLdITQ15vyOn8kao0eZsou3q7zmYItiQ7/SO7/dpSDOm1lBvi7CFI=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cda2d99c-b28b-4011-1ce8-08dc0e0b2870
X-MS-Exchange-CrossTenant-AuthSource: DS0PR10MB7933.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jan 2024 16:27:06.7582
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GLO3IAcyJ0+iUDbhacK8yWgx1mtHIlqINwQyStHFSs3HT+J3GrtwxHaLnGT72ayuoimD+5vwOzfAej1Hx9Nt7g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR10MB7820
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-01-05_08,2024-01-05_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 phishscore=0 spamscore=0
 mlxlogscore=999 malwarescore=0 suspectscore=0 adultscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2401050135
X-Proofpoint-ORIG-GUID: Np3YLvrAFKCVfEHCsDRTM7VYw79NhoTD
X-Proofpoint-GUID: Np3YLvrAFKCVfEHCsDRTM7VYw79NhoTD

* Chuck Lever III <chuck.lever@oracle.com> [240104 14:33]:
>=20
>=20
> > On Sep 12, 2023, at 12:01=E2=80=AFPM, Matthew Wilcox <willy@infradead.o=
rg> wrote:
> >=20
> > On Tue, Sep 12, 2023 at 11:14:42PM +0800, Feng Tang wrote:
> >>> Well that's the problem. Since I can't run the reproducer, there's
> >>> nothing I can do to troubleshoot the problem myself.
> >>=20
> >> We dug more into the perf and other profiling data from 0Day server
> >> running this case, and it seems that the new simple_offset_add()
> >> called by shmem_mknod() brings extra cost related with slab,
> >> specifically the 'radix_tree_node', which cause the regression.
> >>=20
> >> Here is some slabinfo diff for commit a2e459555c5f and its parent:
> >>=20
> >> 23a31d87645c6527 a2e459555c5f9da3e619b7e47a6=20
> >> ---------------- ---------------------------=20
> >>=20
> >>     26363           +40.2%      36956        slabinfo.radix_tree_node.=
active_objs
> >>    941.00           +40.4%       1321        slabinfo.radix_tree_node.=
active_slabs
> >>     26363           +40.3%      37001        slabinfo.radix_tree_node.=
num_objs
> >>    941.00           +40.4%       1321        slabinfo.radix_tree_node.=
num_slabs
> >=20
> > I can't find the benchmark source, but my suspicion is that this
> > creates and deletes a lot of files in a directory.  The 'stable
> > directory offsets' series uses xa_alloc_cyclic(), so we'll end up
> > with a very sparse radix tree.  ie it'll look something like this:
> >=20
> > 0 - "."
> > 1 - ".."
> > 6 - "d"
> > 27 - "y"
> > 4000 - "fzz"
> > 65537 - "czzz"
> > 643289767 - "bzzzzzz"
> >=20
> > (i didn't work out the names precisely here, but this is approximately
> > what you'd get if you create files a-z, aa-zz, aaa-zzz, etc and delete
> > almost all of them)
> >=20
> > The radix tree does not handle this well.  It'll allocate one node for:
> >=20
> > entries 0-63 (covers the first 4 entries)
> > entries 0-4095
> > entries 3968-4031 (the first 5)
> > entries 0-262143
> > entries 65536-69631
> > entries 65536-65599 (the first 6)
> > entries 0-16777215
> > entries 0-1073741823
> > entries 637534208-654311423
> > entries 643039232-643301375
> > entries 643289088-643293183
> > entries 643289728-643289791 (all 7)
> >=20
> > That ends up being 12 nodes (you get 7 nodes per page) to store 7
> > pointers.  Admittedly to get here, you have to do 643289765 creations
> > and nearly as many deletions, so are we going to see it in a
> > non-benchmark situation?
> >=20
> > The maple tree is more resilient against this kind of shenanigan, but
> > we're not there in terms of supporting the kind of allocation you
> > want.  For this kind of allocation pattern, you'd get all 7 pointers
> > in a single 256-byte node.
>=20
> Hello Matthew, it's been a couple of kernel releases, so
> following up.
>=20
> Is Maple tree ready for libfs to use it for managing directory
> offsets?

The feature you are looking for is dense nodes.  It will allow for
a compact tree when you have a number of single indexes mapping to
entries (ideal for many ranges of 1).

I'm actively working on dense nodes, which will yield 31 entries per
node when they are single index mappings.  I'm hoping to have it
completed in the next few weeks and start beating it up with tests
before pushing it out.

>=20
> Should we just go for broke and convert libfs from xarray to
> Maple tree now?

We are trying to keep the API close for both the xarray and the maple
tree, so if you do the conversion to one then switching shouldn't be
much work.  I'd try the maple tree to see if the performance is
acceptable today (I may be biased), but I don't know how big of an
effort this conversion would entail.

The maple tree will compress the NULL indexes to a single entry of NULL.
My main concern is the density of information and the number of
allocations the tree will do to keep up with the workload - both will
improve with the dense nodes feature.

If you convert to maple tree, you will get the update for free later as
the node type the tree chooses will be transparent to users.

If you need tagging then you should use the xarray as I haven't started
that feature yet - but I don't think you need that?

I also noticed that Matthew mentioned xa_alloc_cyclic() as the potential
call into the xarray.  The maple tree counterpart isn't used much today
and may need to be optimised.  If you can verify what this test does, we
could produce a test case for the maple tree test suite and optimise if
necessary.

Let us know if you have any other questions or need some pointers on how
to get started with a conversion.

Thanks,
Liam