Received-SPF: pass (google.com: domain of linux-kernel+bounces-18407-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
From: Zi Yan <ziy@nvidia.com>
To: Ryan Roberts <ryan.roberts@arm.com>
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
 "\"Huang, Ying\"" <ying.huang@intel.com>,
 Andrew Morton <akpm@linux-foundation.org>,
 "\"Matthew Wilcox (Oracle)\"" <willy@infradead.org>,
 David Hildenbrand <david@redhat.com>,
 "\"Yin, Fengwei\"" <fengwei.yin@intel.com>, Yu Zhao <yuzhao@google.com>,
 Vlastimil Babka <vbabka@suse.cz>,
 "\"Kirill A . Shutemov\"" <kirill.shutemov@linux.intel.com>,
 Johannes Weiner <hannes@cmpxchg.org>,
 Baolin Wang <baolin.wang@linux.alibaba.com>,
 Kemeng Shi <shikemeng@huaweicloud.com>,
 Mel Gorman <mgorman@techsingularity.net>,
 Rohan Puri <rohan.puri15@gmail.com>, Mcgrof Chamberlain <mcgrof@kernel.org>,
 Adam Manzanares <a.manzanares@samsung.com>,
 "\"Vishal Moola (Oracle)\"" <vishal.moola@gmail.com>
Subject: Re: [PATCH v1 0/4] Enable >0 order folio memory compaction
Date: Fri, 05 Jan 2024 17:56:08 -0500
Message-ID: <954192F7-8174-4E4F-82C6-36BF808CDE35@nvidia.com>
In-Reply-To: <DFFD13A9-2578-4A57-AA46-068EEA77FBC4@nvidia.com>
References: <20231113170157.280181-1-zi.yan@sent.com>
 <1e7cf758-ac0e-4eda-a8e2-90db5eed7f7a@arm.com>
 <7ED1378A-AC39-48A2-8A2A-E06C7858DCE1@nvidia.com>
 <df51bbb7-f791-4747-9077-e9f7f37e9518@arm.com>
 <E22D7AC8-0891-4B4D-BB90-129B1CB77CB2@nvidia.com>
 <7acefbe0-9413-48fd-ad98-56fecc6f29da@arm.com>
 <DFFD13A9-2578-4A57-AA46-068EEA77FBC4@nvidia.com>
Content-Type: multipart/signed;
 boundary="=_MailMate_766B8054-C4BE-4A77-A7D0-9AE1EF0771F6_=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?aDVxVGZSQlVVT20zdndnWnZYQXVRSk02Ti9mSWhBMEpheUovcEJPMWx3THpt?=
 =?utf-8?B?YW1kTUhBaDNlUE14QlNpakpjNUxNK3RLTmQyTitkblNtcVJXUnZjYitPRmRL?=
 =?utf-8?B?UjBPRGR5RVFickpyaGtrdkhkTjBlcEJtNjBaOU80azNtdFBVcFozYmYyQllW?=
 =?utf-8?B?NWhYMFFaVHBmZ0dDaER2QUZseThISUFEUkgvT1lVbXNZaXFMaDJSNWI4RDFX?=
 =?utf-8?B?cmVrTmhpUU1EdWhtVS9zdGhvM1hWZVA3VmhVMWd5NWk0S1ZPOWkwRTNPdmF5?=
 =?utf-8?B?ZFBOamZCc0hyU3B3NXVvbXBXaGtEaStja09SZ2RIUUg5RWtjVnZqbWlUYit6?=
 =?utf-8?B?bmoraUhJcVBvWXNZNXYxbjk4aXZHOG54azQ5ei9Yb2RGODRISkpJSUw1TElz?=
 =?utf-8?B?ZVN0UzNEWExtemhiQUZqOUorWnE4NVIyYzZrRFJSNmVIM2JqNFRqQWhWbTlz?=
 =?utf-8?B?bHZ5aURhOHFFNVoxa2xWcEtWWlNDOHI0cGtBd3pyTWJhenNzOTloU04wK3Ri?=
 =?utf-8?B?ak9WZGFzaXVqcnBuUmdobi84YzNmVG1ZbkxHNU9DbHdPNjBVZUZCRzAyWWJ6?=
 =?utf-8?B?KzV0czZvYmdNNnJXTDd6VXF0bVRFRUJNN1RoYm5FTGJXSTM2OG9reGVkZmJL?=
 =?utf-8?B?RmRZNWgxYzYrVTZ1dWZScHMveVpZamVnT2tvWjlQWS9Ja1RZSTcrK1JRa1Y4?=
 =?utf-8?B?aWJjdjMyREFqS3kvM2tjbDR4YWRtVXFBbzRQRjkzcTVhVGtqc3hsWEg1ZFRn?=
 =?utf-8?B?UzdEKzREZFlFOGR6TTI4TE1zOTNab3g4Mi9OQnF6Ny9WSG5NYjc3Rnc0aG1y?=
 =?utf-8?B?Y1l1TVMwaFN3QTFxNVZsa3IrOEdtTzh2M1djU0pPd2FwdjlrRU82L3hNQ2Za?=
 =?utf-8?B?a3hZOVBkb0dtcGw0eHZ0V1VTbW4xNGx3clRQL25IZlN2ZmlxQXAza25CK3RT?=
 =?utf-8?B?R2dnaEU5eThCcW8vRWhEUk9tMFRuektTa1JCTlRKRWZCNjhLYmlwa3VDZHBE?=
 =?utf-8?B?VDBkT2F0cFo3WnRtR1BlRDU2dGhjTjRhemJBNUdRU0lyNmc4TG5KcGZXcVVS?=
 =?utf-8?B?d0xla29PY0krOGhkV3JUcFlPN2M0MTI2SWM3a2JxUENaNEt2TnBzbnFkNG9D?=
 =?utf-8?B?Ti9Ram95SlI3Q0pIa0IxZlEzcCtBSnpWYnkvNVF0UVBFRWFzSGNJem40OU91?=
 =?utf-8?B?RmlUdTR3dnVwU3NxT1JyMU1xZFlpVzUyamxBTTFYeGFWSUJPUHlIY05JODR3?=
 =?utf-8?B?L0JobG4vdnpway9PeHIzcytGWEJNVEdML3Z6ZEQvdWtTS0pySXNxRVRrSVhQ?=
 =?utf-8?B?Q1cvTWdEVkovcGhML1Nsc3pzY3dUY1JYY3p6dktCc25NL3JjbU8wVnNqMkZV?=
 =?utf-8?B?aUxwMlFwd2JDVlBxdkswbzZYb0ZQSmJSRFVxaGtHMmZjSEF4aEcrT1ZwTllF?=
 =?utf-8?B?bVRiSVhYVlh0c3JWR3VHU0lnSjBhMXFFbmdjdk5ObHlPSkdQb0VkdGdQMGlU?=
 =?utf-8?B?SFFHMTNDVTF3Sk4ybG5UYno1aCtHRjBsQW5xNTROTmlkMGVkQm51OEVnTUh5?=
 =?utf-8?B?L0Q3eHZseFgrMC82Znh6cmk2NVFEM2Q3bUxuTlBYWk00Z3c1cWptbW5BNzcv?=
 =?utf-8?B?RE5hcnd6cktBTFhUUUFRR3U1dzI4ckhBT1BYOUM4UWMvaG1IRnFQYVJZWmUz?=
 =?utf-8?B?U0xueXZQVXk1by9yS0ZDUS9yZjFBTGNJdmUrN212VU9iaFArSmUySXg1TDhO?=
 =?utf-8?B?WlRMa2h5YmFZQXlNQUVmZEMyWWZzR2tsdDNRV1ZIRkdFL2ZoRGZGOGVpWlY0?=
 =?utf-8?B?VXB5cHRzSGVtUGFhTHBSYjJycDMvTXd4RGhXS2QyYllGQlNlOHNFYXN6clI3?=
 =?utf-8?B?bHJWajN4QlJEc0R2bFBEd3FFWWNlZlhaOEREU2JYU1hsNE02TlB0cVhJS3dN?=
 =?utf-8?B?LzE1YUVFV1BWTTc3Q01lOUVrZ2M1cDVYRWxCeExLTWEvdTNldk5QSGJrV3dH?=
 =?utf-8?B?K3VKbk4rYXgvakNBSElyTmR2VVZibmpUbjN4Q0E5S2k4WTZUZDcxTWh6T253?=
 =?utf-8?B?QUNMbzdyU3N4THhLdG5pZCtXOWtLcTlPTVN4bDFlWTFaNzFMc3orTDZjM0VG?=
 =?utf-8?Q?6vfg95fw2npeMSpkXbps3BOr8?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b77cdd25-8730-4870-0226-08dc0e41830d
X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB5744.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jan 2024 22:56:11.5655
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: iMsNQJgnVEFp+6sInFrhcnYssqUL1n4V+QJeo6texstC4DCxnJgSczY9rPd8bZhP
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8286

--=_MailMate_766B8054-C4BE-4A77-A7D0-9AE1EF0771F6_=
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 3 Jan 2024, at 10:51, Zi Yan wrote:

> On 3 Jan 2024, at 4:12, Ryan Roberts wrote:
>
>> On 02/01/2024 20:50, Zi Yan wrote:
>>> On 21 Nov 2023, at 12:11, Ryan Roberts wrote:
>>>
>>>> On 21/11/2023 16:45, Zi Yan wrote:
>>>>> On 21 Nov 2023, at 10:46, Ryan Roberts wrote:
>>>>>
>>>>>>>
>>>>>>> vm-scalability results
>>>>>>> =3D=3D=3D
>>>>>>>
>>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>>>>>>>   gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scala=
bility
>>>>>>>
>>>>>>> commit:
>>>>>>>   6.6.0-rc4-mm-everything-2023-10-21-02-40+
>>>>>>>   6.6.0-rc4-split-folio-in-compaction+
>>>>>>>   6.6.0-rc4-folio-migration-in-compaction+
>>>>>>>   6.6.0-rc4-folio-migration-free-page-split+
>>>>>>>   6.6.0-rc4-folio-migration-free-page-split-sort-src+
>>>>>>>
>>>>>>> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migr=
ation-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
>>>>>>> ---------------- --------------------------- --------------------=
------- --------------------------- ---------------------------
>>>>>>>          %stddev     %change         %stddev     %change         =
%stddev     %change         %stddev     %change         %stddev
>>>>>>>              \          |                \          |            =
    \          |                \          |                \
>>>>>>>   12896955            +2.7%   13249322            -4.0%   1238517=
5 =C2=B1  5%      +1.1%   13033951            -0.4%   12845698        vm-=
scalability.throughput
>>>>>>
>>>>>> Hi Zi,
>>>>>>
>>>>>> Are you able to add any commentary to these results as I'm struggl=
ing to
>>>>>> interpret them; Is a positive or negative change better (are they =
times or
>>>>>> rates?). What are the stddev values? The title suggests percent bu=
t the values
>>>>>> are huge - I'm trying to understand what the error bars look like =
- are the
>>>>>> swings real or noise?
>>>>>
>>>>> The metric is vm-scalability.throughput, so the larger the better. =
Some %stddev
>>>>> are not present since they are too small. For 6.6.0-rc4-folio-migra=
tion-in-compaction+,
>>>>> %stddev is greater than %change, so the change might be noise.
>>>>
>>>> Ahh got it - thanks!
>>>>
>>>>>
>>>>> Also, I talked to DavidH in last THP Cabal meeting about this. He s=
uggested that
>>>>> there are a lot of noise in vm-scalability like what I have here an=
d I should
>>>>> run more iterations and on bare metal. I am currently rerun them on=
 a baremetal
>>>>> and more iterations on the existing VM and report the results later=
=2E Please
>>>>> note that the runs really take some time.
>>>>
>>>> Ahh ok, I'll wait for the bare metal numbers and will disregard thes=
e for now.
>>>> Thanks!
>>>
>>> It seems that the unexpected big mmap-pread-seq-mt perf drop came fro=
m the mistake I
>>> made in patch 1. After fixing that, mmap-pread-seq-mt perf only drops=
 0.5%. The new
>>> results on top of 6.7.0-rc1-mm-everything-2023-11-15-00-17 are at the=
 end of the email.
>>
>> Good news! I don't see the results for mmap-pread-seq-mt below - perha=
ps you
>> forgot to include it?
>
> The stats below only shows significant changes and mmap-pread-seq-mt de=
lta is less
> than 5%, thus it is not shown.
>
>>
>>>
>>> I am preparing v2 and will send it out soon.
>>>
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>>>   gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalabili=
ty
>>>
>>> commit:
>>>   6.7.0-rc1-mm-everything-2023-11-15-00-17+
>>>   6.7.0-rc1-split-folio-in-compaction+
>>>   6.7.0-rc1-folio-migration-in-compaction+
>>>   6.7.0-rc1-folio-migration-free-page-split+
>>>   6.7.0-rc1-folio-migration-free-page-split-sort-src+
>>>
>>> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migratio=
n-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
>>> ---------------- --------------------------- ------------------------=
--- --------------------------- ---------------------------
>>>          %stddev     %change         %stddev     %change         %std=
dev     %change         %stddev     %change         %stddev
>>>              \          |                \          |                =
\          |                \          |                \
>>>   13041962           +16.1%   15142976            +5.0%   13690666 =C2=
=B1  6%      +6.7%   13920441            +5.5%   13762582        vm-scala=
bility.throughput
>>
>> I'm still not sure I'm interpretting this correctly; is %change always=
 relative
>> to 6.7.0-rc1-mm-everything-2023-11-15-00-17 or is it relative to the p=
revious
>> commit?
>
> The former, always relative to 6.7.0-rc1-mm-everything-2023-11-15-00-17=
=2E
>
>>
>> If the former, then it looks like splitting the folios is actually fas=
ter than
>> migrating them whole?
>
> Yes, I will look into it when I am preparing the next version.
>

The reason seems to be that compaction ends early when migrating folios a=
s a whole.
It happens when a order-0 folio is migrated and there is no order-0 free =
page,
then migrate_pages() returns -ENOMEM making compact_zone() stop compactio=
n (for
higher order folios, they would be split). This should be fixed by enabli=
ng
free page split optimization, but the perf number does not say so. Let me=
 dig more.


--
Best Regards,
Yan, Zi

--=_MailMate_766B8054-C4BE-4A77-A7D0-9AE1EF0771F6_=
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename=signature.asc
Content-Type: application/pgp-signature; name=signature.asc

-----BEGIN PGP SIGNATURE-----

iQJDBAEBCgAtFiEE6rR4j8RuQ2XmaZol4n+egRQHKFQFAmWYiQgPHHppeUBudmlk
aWEuY29tAAoJEOJ/noEUByhUJ4AP/jA7Zggdqmy5eEmBbcncN8crMnivfT+A4LO7
3NfpIpZ7U9w+90LPGPD9BjPPHjna/5KrH7IrUC51Wb/s5FGP3+UhC3YZFyh8RL51
1doKJ/IKJ3At/vSagPbrGJKz5zIy+Fb/a3FRxUOGOOzyPstQ9W1qabg4ggcRsLer
B1RjiuaxXa/8PRH03ZbQ+vNtt8Es642kXexuAh1YjTAaJnaJh6rkOmNPg89lUU09
cZsEuu/bkvlukDRKrTXYWRzncEl2f9k62vnn6Gs8TSjLUKOih6M/42M3qYAlVYva
K3pZbXY4hWM19ki+oEVtxromBTxZaqFiO7MEyrcO8of/P8bwTojdFjwLObGs4s1K
GT/oohXVG7FaMkJCmoI5PEb4Y3TW7eKj5h6aXv9ectK4+zh3FIUi5xpfUP5S7uZ4
4sA7lxQMQvD4NFNGF66zjZMKxiuDfXjzFOIPKtLj/D7duUiPoAKl4PFNOCmhiXej
0OoafCvyEQXnQG7U2xrSKo703j6lWVx/iD4LCHgiWTimRjZb+G/Q06nsavlfiSN8
pkasjQnef0oA2n10dSEiQbPbPtCEcztbSrexlL2RsfWPGFwC8AbKf2cIHtlcJFP/
q58+akyG+i9+ex+/rK0b+i/pR0o9/7NIayXPfm1jEiCoymcdy+5j5RPpB8TDlKuD
i9rOeGyn
=Z9u6
-----END PGP SIGNATURE-----

--=_MailMate_766B8054-C4BE-4A77-A7D0-9AE1EF0771F6_=--