Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
Date:   Thu, 23 Nov 2023 09:34:52 -0500
From:   "Liam R. Howlett" <Liam.Howlett@Oracle.com>
To:     Bagas Sanjaya <bagasdotme@gmail.com>
Cc:     Chun Ng <chunn@nvidia.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Linux Regressions <regressions@lists.linux.dev>,
        Andrew Morton <akpm@linux-foundation.org>,
        Linux Memory Management List <linux-mm@kvack.org>,
        Ankita Garg <ankitag@nvidia.com>,
        Suren Baghdasaryan <surenb@google.com>,
        "Matthew Wilcox (Oracle)" <willy@infradead.org>
Subject: Re: [REGRESSION]: mmap performance regression starting with k-6.1
Message-ID: <20231123143452.erzar3sqhg37hjxz@revolver>
Mail-Followup-To: "Liam R. Howlett" <Liam.Howlett@Oracle.com>,
        Bagas Sanjaya <bagasdotme@gmail.com>, Chun Ng <chunn@nvidia.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Linux Regressions <regressions@lists.linux.dev>,
        Andrew Morton <akpm@linux-foundation.org>,
        Linux Memory Management List <linux-mm@kvack.org>,
        Ankita Garg <ankitag@nvidia.com>,
        Suren Baghdasaryan <surenb@google.com>,
        "Matthew Wilcox (Oracle)" <willy@infradead.org>
References: <PH7PR12MB7937B0DF19E7E8539703D0E3D6BAA@PH7PR12MB7937.namprd12.prod.outlook.com>
 <ZV7eHE2Fxb75oRpG@archie.me>
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <ZV7eHE2Fxb75oRpG@archie.me>
User-Agent: NeoMutt/20220429
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?xgXYNqjeL/aanth1shx1xMrH5LY/8dk4YbkuFuoNlkmvDTaxrT9im2eI85?=
 =?iso-8859-1?Q?/knd3Ry5Wy82sM9NCFSnjpEnawuvh5FNr6pXdaT3vKd25p/8cuEfTIlV08?=
 =?iso-8859-1?Q?rQD9ufC1v6EqMQv9JsFXzhCD3OLlsKwCuG7zwQnYiY3r0SKlSAFW8UKF/Z?=
 =?iso-8859-1?Q?UZLOk3sQ+kK9SM0Kk3SKIVrHFOibzCg0B2BCityoWnanzpvtxuapmiHd2e?=
 =?iso-8859-1?Q?7O69cn6OYr3f0ktUCYXau4HG2GxD4ItPw4Ih87RsWjW6XiNoQXh2Rg/xhm?=
 =?iso-8859-1?Q?S27rT+7Ywj0zdDXIvV2C3me/mBzwJYsNljQGO5g5sQT9lbEWutAe0c3MSd?=
 =?iso-8859-1?Q?hZVC3fyjGUcm9KE7oCJ1rABy/EXDD9Oe05BXR/QLz9PP8ItHNNjPBMEkYg?=
 =?iso-8859-1?Q?ni3UvUESDXBbfM5Bmkl24WW8bAs0ArOAqebqSnvxK5Z7jnPCPAdLmVwrwm?=
 =?iso-8859-1?Q?wEeUw3Ll2uPSsQd5U5MYgDflNDL8MRsAF0JOen3fAqNAZSOKaZ6QTeR9Z2?=
 =?iso-8859-1?Q?Drm85OwV+7sCZeOw86XnWcC8V0k/mHLuqLKz6gUfSPBGLOsQ04MDdf+rD6?=
 =?iso-8859-1?Q?xRClt2J4nuWX2QK3kfOJUJ/X7E372vydXSM9zvtERHi4FwU4hqnKAjZcCM?=
 =?iso-8859-1?Q?ht5QhBgQA1/sA3lMgHmXCRl2zKueylWd1gvnqeGrXPbhycpzpmbbTkjA1o?=
 =?iso-8859-1?Q?hpfJXl91rGAg5x79HlsRW+B2jIBWWciSfdr++FqowdKpYHLcoBwXQY7fVx?=
 =?iso-8859-1?Q?+06YuDzINTjKTiYUbTVp47fQl5RscGVT3z/Apdkwzx31C/D7UFuvBbedyN?=
 =?iso-8859-1?Q?NQVFc7cHxp/8BRPjvu1KHVr+KcmPnkQX1cUhBQhMSbMckMuRAgy2tp4BK6?=
 =?iso-8859-1?Q?XCLWGgq6Q3blx6Gieg3LAncZM7ebjLK6UC/0UEaHyBn9VnCWOWrhUkMYhY?=
 =?iso-8859-1?Q?8rIeLCKztJK9yJRY/XWnAMw6ghVo0NZ5vLxZy76G/qDOFC2phJRQpOXNV1?=
 =?iso-8859-1?Q?PDQuSzLco90+tUkdxq6phyr+ceWQqO0dyt8a9FXLuw/ue9GolmSj1LeUKo?=
 =?iso-8859-1?Q?rSYp/fjTV86e6msSqD9vQN34+PQMxatpPyc1eFJ2qbWNFfI4HY9b8Px56J?=
 =?iso-8859-1?Q?Bc7oVBx/OBa2Ah6KRu4woG3OXIGFH79gvQNSRZgSX4p6t6pyrD3m5fr1xL?=
 =?iso-8859-1?Q?DxDpwk7sfnyUukhSVDfr1A37rSxcd1SL3yIg7AR/vU3zUdyGVR/rn5EdiS?=
 =?iso-8859-1?Q?mVu5R1tlQhKp0rQZOB7dkYA6vWgn0cfsZPzk5IBXbNx5ArCzCP9L6/hwn5?=
 =?iso-8859-1?Q?w3t4rXM+w1UDrtZpFLn9HAnMApU/5zEuvjdpdNFT0oa9RK0Iac0g9bBVx3?=
 =?iso-8859-1?Q?464meca6NDBID/W8wveMTYM4s9YctTDfhR3DjXZgIb2Xtqg8M9sy4xowPU?=
 =?iso-8859-1?Q?XfoU2JNVlAMRU5nUx0nKhhsA7whD5laSPr4n0t2IIdxggHV0oJHDpHdfHs?=
 =?iso-8859-1?Q?sr0cEwL0+pf7t+gujPeu1e8Cyrq46/a3s43pWWtmWPjsxTOmA8Qk3DbHEk?=
 =?iso-8859-1?Q?phSzWcDZnXtt1TBtpXyM1l7k0jzsEB1JZNvyAnRUkQWc9sV1pyMTGZv3+l?=
 =?iso-8859-1?Q?I3NGpNNLdbliXDAYb+1E/Rsg9ttdBf7SgE?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?iso-8859-1?Q?NB5xLdAi5EHxkHc6+M/ZgzLaXyZ6xSRv2pttGBwHy3qJYJdFIcX5duE6VF?=
 =?iso-8859-1?Q?YWNIMHCRz/34hI6XWz25DiR69CR+zbp/hxuXks22qr41rlk0emIV0HfQof?=
 =?iso-8859-1?Q?h+I37FYh7PxDAvn1EaxiD8l+AJuQSSlDJOqxfSlmdwG3gEQsSebPiR/f1n?=
 =?iso-8859-1?Q?o3Hu5buRKUDS0hutmG5/TyffQPdK9R+4FU+D8SK6cUauIh7vWjLR/uQ5jR?=
 =?iso-8859-1?Q?33jE4/q2C235NQKSZCPYyksqMDuiNMZKiUttOBwWHUX30Ja867rV/mGZvH?=
 =?iso-8859-1?Q?EKT2bvXQB2ruF4mFBI0wOgTExBzW3uy4gNUbhraUe1wjZDuiwX4pHrXoJU?=
 =?iso-8859-1?Q?7Ag4VZK8ZMt8fmUCH9/Ve3QPPyU+3dTXXZ8hSVkhKcLL5CKj8T/o0NzVxQ?=
 =?iso-8859-1?Q?p9EtcIDhiaWeVggv2BMbkhJZpnHTMtzyRUlxq4LpegEkvitrzJY1PvLIjf?=
 =?iso-8859-1?Q?HViBPbKe+11/R42Mw9x3GmcNqgoWp2JoBMOLoh1M0ps/x75StfoORKHn7L?=
 =?iso-8859-1?Q?eqgeAM+cqVuQaLaGWkQlEBivyTjl9UBW+loip6tVOXBfPcvBM3FrUGDFPU?=
 =?iso-8859-1?Q?q2IGSsfRQZzf17Ua1hvsHkRdwz5qZZ9KJ7KWkpuU+lt2ixQ+rQCF/7+oPj?=
 =?iso-8859-1?Q?qkH9ZYNVaAef3jiTBUOOLolh8G1DDjuPV8u9cQ9kySZPo3FtTr1TF5+qUp?=
 =?iso-8859-1?Q?seDInrqvw7PXGa4tRYp+mswiNW9UyzJjUHOUgYrlZj+mYkNbOjzL+Y8nmU?=
 =?iso-8859-1?Q?iEgFYbNbjqia/UndvFe4N9zw821MKyQB1FlHSYocHGkNFxNMMoQ/rFnvxD?=
 =?iso-8859-1?Q?N/9uVKE2DQtZZWzp/gVJHXNS7GMiz6tTMqxDiueuZTAoDnRcIljfH+7kNT?=
 =?iso-8859-1?Q?9TGmsh8f97rAbQFhvj3RcWkFukZSXnyPrvFvlgjZsgbjgN7uKJRZJPK5HZ?=
 =?iso-8859-1?Q?Qy69rTTSUZH/GS069aVMTotqYVopMNAq6JJLkqf0csjAFV3eAabXUpWdLi?=
 =?iso-8859-1?Q?ZkelXfA/ojc0sm2Uj+GjvtUZcn474hp6LmsTxO?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b6b914e7-6e40-46a8-1088-08dbec315c9c
X-MS-Exchange-CrossTenant-AuthSource: SN6PR10MB3022.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Nov 2023 14:34:55.7400
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2lUtdVVS9lNc4uJ4v13J6WrSTda9u6DDnZKmwEEZoy4ameuw6q4awrTZb+L6engA46T2ELvyr7VM55uV8m5YoA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB7324
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.987,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-11-23_12,2023-11-22_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 bulkscore=0
 malwarescore=0 mlxlogscore=513 adultscore=0 phishscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311060000
 definitions=main-2311230105
X-Proofpoint-GUID: 0CbcD6ZLXzdD70dVNFhM3Focs99bEnWk
X-Proofpoint-ORIG-GUID: 0CbcD6ZLXzdD70dVNFhM3Focs99bEnWk
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Thu, 23 Nov 2023 06:35:50 -0800 (PST)

* Bagas Sanjaya <bagasdotme@gmail.com> [231123 00:07]:
> On Wed, Nov 22, 2023 at 08:03:19PM +0000, Chun Ng wrote:
> > Hi,
> >=20
> > Recently I observed there is performance regression on system call mmap=
(..). I tried both vanilla kernels and Raspberry Pi kernels on a Raspberry =
Pi 4 box and the results are pretty consistent among them.
> >=20
> > Bisection showed that the regression starts from k-6.1, and the latest =
vanilla k-6.7 is still showing the same regression.

This is almost certainly the maple tree.  The tree is slower on writes
than the rbtree and so if the benchmark mmaps/munmaps in a tight loop
you will see this slow down.  What you are doing is measuring the speed
of inserting and removing a VMA with this benchmark, so it's not really
something that happens - we usually use the mapping between adding and
removing it.

What this gains us is the ability to remove contention on the mmap lock
during page faults.  If you were to test contention around that lock,
you will see a slowdown until you reach v6.4, where per-vma locking
started to show up.  More benchmarking will show different types of
fault handling outside of the mmap lock until (I believe) 6.6, where
most (or all?) types are supported.

Although this is expected, I am still looking to reduce any real
workloads that may suffer.  I've been reducing the allocations, for
example.

> >=20
> > The test program calls mmap/munmap for a 4K page with MAP_ANON and MAP_=
PRIVATE flags, and ftrace is used to measure the time spent on the do_mmap(=
..) call.=A0 Measured time of a sample run with different vanilla kernel ve=
rsions are:
> > k-5.10 and k-6.0: ~157us
> > k-6.1: ~194us
> > k-6.7: ~214us

I would have expected v6.7 to remain closer to v6.1, but that may depend
on the minor versions you have been testing and what fixes have landed
there.


> > Results are pretty consistent across multiple runs with a small percent=
age variance.=A0 Ftrace shows that latency of mmap_region(...) has increase=
d since k-6.1.=A0=A0An application that makes frequent mmap(..) calls the a=
ccumulated extra latency is very noticeable.=20
> >=20
> > Please find the ftrace results and kernel config files in this folder:
> > https://drive.google.com/drive/folders/1qy8YTBqxu8Gdbs7IigYbSd4FXldId5s=
d?usp=3Ddrive_link
> >=20
> > The test program can be found in here:
> > https://drive.google.com/file/d/1tG6_BbQMCHwfKebvAIAg_xqbM_lpPcuM/view?=
usp=3Dsharing
> >=20
> > Info on the testing environment:
> > cpufreq_governor: performance
> > Test machine: Raspberry Pi 4, 8GB DDR
> > SCHED_FIFO with priority 99 for running the test program
> >=20
> > Vanilla kernels are not tainted. However on k-6.0 and k-6.7, I have to =
patch the drivers/clk/bcm/clk-raspberrypi.c file with the version in Raspbe=
rry Pi kernel tree for the CPU frequency governor to work.
> >=20
>=20
> The next step is to find the commit that introduces your regression with
> `git bisect`. If you haven't done so, see
> Documentation/admin-guide/bug-bisect.rst for instructions.
>=20
> Anyway, I'm adding this regression to regzbot:
>=20
> #regzbot ^introduced: v6.0..v6.1
>=20
> Thanks.
>=20
> --=20
> An old man doll... just what I always wanted! - Clara