Received-SPF: pass (google.com: domain of linux-kernel+bounces-107432-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: "Prasad, Aravinda" <aravinda.prasad@intel.com>
To: SeongJae Park <sj@kernel.org>
CC: "damon@lists.linux.dev" <damon@lists.linux.dev>, "linux-mm@kvack.org"
	<linux-mm@kvack.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "s2322819@ed.ac.uk" <s2322819@ed.ac.uk>,
	"Kumar, Sandeep4" <sandeep4.kumar@intel.com>, "Huang, Ying"
	<ying.huang@intel.com>, "Hansen, Dave" <dave.hansen@intel.com>, "Williams,
 Dan J" <dan.j.williams@intel.com>, "Subramoney, Sreenivas"
	<sreenivas.subramoney@intel.com>, "Kervinen, Antti"
	<antti.kervinen@intel.com>, "Kanevskiy, Alexander"
	<alexander.kanevskiy@intel.com>
Subject: RE: [PATCH v2 0/3] mm/damon: Profiling enhancements for DAMON
Thread-Topic: [PATCH v2 0/3] mm/damon: Profiling enhancements for DAMON
Thread-Index: AQHaeTeYeVMaFIFnV0yMrKTBAlKavbE+h/4AgABRr6A=
Date: Tue, 19 Mar 2024 10:56:42 +0000
Message-ID: <MW5PR11MB5907F44D802C298E8182A1B6F22C2@MW5PR11MB5907.namprd11.prod.outlook.com>
References: <20240318132848.82686-1-aravinda.prasad@intel.com>
 <20240319052054.100167-1-sj@kernel.org>
In-Reply-To: <20240319052054.100167-1-sj@kernel.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?WG2FOibTLTNKbBbf7+nkJVE3jncr+RDbGRB6ipG3RT+nuO+1MCNwHInCYo3O?=
 =?us-ascii?Q?kd4Abwp1/WKXmclWtaKVMNIp1B2nizUgTBxtHeOW3mIRdRJzSVumibopmNr5?=
 =?us-ascii?Q?uFqpGFA9W6TSyjGKpY8/fzY42rVqTVgUEGiniAGqiyr/X/JjSPYnGF0LMadk?=
 =?us-ascii?Q?Y4TjY3zTZeP2+A7DatymQ0RPgdPsOsFF3FQeJJX7HKOExk9vD5tHUdUQ/e0n?=
 =?us-ascii?Q?JDwnchU+9h+SeHxaSeGzCYUkaOu9uB19ZqDEKeFzpWG6kEyFWcNB3H2uzYsl?=
 =?us-ascii?Q?hsTHoK0ncgZ/TCNNuW8wFwFf/Ur7QugXTlc1tctnYQmcsPM1UjysUfycaxyX?=
 =?us-ascii?Q?AzOsTXmvwvKCNnLu2sn9m8ayhVVEalHbd0or8V0AOAC5xNlqYmuVkt3jkr5F?=
 =?us-ascii?Q?ta9+ZsUhYjpUTG0sbHE+89SgG2CUUCqYlfbh0FT4smhn3+024sWeNYnXJq30?=
 =?us-ascii?Q?4z/3mA4vVbOhz1GhmIVre3zCCNZV7rFLH5hsD4BGmhyNovbOxT0Qn/l2FyAv?=
 =?us-ascii?Q?+EoLuSB/F63sxD8Sim3S6jBu7H9dpFL4bQ5CmzVLfXvCmasMZ69WYQAvamLS?=
 =?us-ascii?Q?JPPWTgHQ3hjlu73gHisxRhw5MJKv92Xkjf/mkAog7doo1N7UY/9ikOpNGHFd?=
 =?us-ascii?Q?dAsAFt8bZhitDJR/Z9744kwISjas79Obx5ox0R/MG5mx7O1t5u8zl6N3PIJV?=
 =?us-ascii?Q?+Z/nvgYOfUvLairpmtOJtlX4ufH40NmTWtpGG2r+WVXkTsEN9o/xeDw9yaQH?=
 =?us-ascii?Q?FUiPhZc21Le6WME0Vul8sPH6aOjr/BIgk7ACVdCb56f/wc6o7GRYR56ZIgEH?=
 =?us-ascii?Q?y5HfPNImFsQWDYdMEJncCPsta6HUkAmuTjXLmQGW50jzVS1A2S+JnBwWVETL?=
 =?us-ascii?Q?N3+dcq6mE0YcqjgNrAB3HIbwTdRQ1wwllXYVhXX9svRFCca04on7qIXdL9p8?=
 =?us-ascii?Q?4CuAbRtbqacV7DgpRoRvGVbELcFV1jirDDTQZw2HDvMx8JL71yAhrhaFo4QK?=
 =?us-ascii?Q?pjgCBSLTzcjr6HgOuZF5UdRzTCTpeNxABY5t5BYZebiO9ZwyEWhrjJtLAxo8?=
 =?us-ascii?Q?GSCFRbhwzLub1gdZxXJdFgcXvgVnXn5DR7SO0ovMNGxemwqI34Bu9g0c4Piy?=
 =?us-ascii?Q?ZDQJuGCIJUZVPAfF+P3YLH1Q1eTuMsOqkgCEKX2dPbJn3fc6Sy8NJUQiSU6o?=
 =?us-ascii?Q?6RwfomSapBhbz/zqlZslnHfb9Enf9ehOOaF03+2DD7IbhtOZGOUR/HUlGTDk?=
 =?us-ascii?Q?Bs2XjOXK16+4a2w2XbqPcKfTHfWQuvfSGxVh4u86I6E6jGXucv+Jx+q6bvGF?=
 =?us-ascii?Q?hUOJK1mAHFeKG16fZ4yLWPR0Tt3QBAtSPQxRc/pTQvSf5oLaZ/4Z7XZZ6O9J?=
 =?us-ascii?Q?l9SGYoPnZV7wyeIV+AgwpO7+O5ZXGMz99r5BTQ3d30iOF/LDg3Ioc39sImMs?=
 =?us-ascii?Q?KfULCRZ0vQbVAfaNLgbN4brnoNcfKvc+LQ0jpz8j8EpXNJA7tqdo4aDICk94?=
 =?us-ascii?Q?srTezN8xQMi5PoFVYuxcX2R9ziq9gXCZZvAyhP4DP3Wi6a8xGBdviloph3+t?=
 =?us-ascii?Q?VGADSdJrWSrFoaeXhtN2na1qnOipOZZNK1JyJyU+?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MW5PR11MB5907.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 199eda8a-c28b-4960-dbbe-08dc480342cb
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Mar 2024 10:56:42.1971
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Z0I89CnRtne+0duoTlng1YRWvWU+bHjpCTDZ1pDOF682t9ib5njurUtTFLVzwWFYT2lDIGN3C+kQRKYeXhlJy03Sk8h7eCy9fRTsFblPYaE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR11MB4948
X-OriginatorOrg: intel.com


> -----Original Message-----
> From: SeongJae Park <sj@kernel.org>
> Sent: Tuesday, March 19, 2024 10:51 AM
> To: Prasad, Aravinda <aravinda.prasad@intel.com>
> Cc: damon@lists.linux.dev; linux-mm@kvack.org; sj@kernel.org; linux-
> kernel@vger.kernel.org; s2322819@ed.ac.uk; Kumar, Sandeep4
> <sandeep4.kumar@intel.com>; Huang, Ying <ying.huang@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Williams, Dan J
> <dan.j.williams@intel.com>; Subramoney, Sreenivas
> <sreenivas.subramoney@intel.com>; Kervinen, Antti
> <antti.kervinen@intel.com>; Kanevskiy, Alexander
> <alexander.kanevskiy@intel.com>
> Subject: Re: [PATCH v2 0/3] mm/damon: Profiling enhancements for DAMON
>=20
> Hi Aravinda,
>=20
>=20
> Thank you for posting this new revision!
>=20
> I remember I told you that I don't see a high level significant problems =
on on
> the reply to the previous revision of this patch[1], but I show a concern=
 now.
> Sorry for not raising this earlier, but let me explain my humble concerns=
 before
> being even more late.

Sure, no problem. We can discuss. I will get back to you with a detailed no=
te.

Regards,
Aravinda

>=20
> On Mon, 18 Mar 2024 18:58:45 +0530 Aravinda Prasad
> <aravinda.prasad@intel.com> wrote:
>=20
> > DAMON randomly samples one or more pages in every region and tracks
> > accesses to them using the ACCESSED bit in PTE (or PMD for 2MB pages).
> > When the region size is large (e.g., several GBs), which is common for
> > large footprint applications, detecting whether the region is accessed
> > or not completely depends on whether the pages that are actively
> > accessed in the region are picked during random sampling.
> > If such pages are not picked for sampling, DAMON fails to identify the
> > region as accessed. However, increasing the sampling rate or
> > increasing the number of regions increases CPU overheads of kdamond.
>=20
> DAMON uses sampling because it considers a region as accessed if a portio=
n of
> the region that big enough to be detected via sampling is all accessed.  =
If a
> region is having some pages that really accessed but the proportion is to=
o
> small to be found via sampling, I think DAMON could say the overall acces=
s to
> the region is only modest and could even be ignored.  In my humble opinio=
n,
> this fits with the definition of DAMON region: A memory address range tha=
t
> constructed with pages having similar access frequency.


>=20
> >
> > This patch proposes profiling different levels of the
> > application\u2019s page table tree to detect whether a region is
> > accessed or not. This patch set is based on the observation that, when
> > the accessed bit for a page is set, the accessed bits at the higher
> > levels of the page table tree (PMD/PUD/PGD) corresponding to the path
> > of the page table walk are also set. Hence, it is efficient to check
> > the accessed bits at the higher levels of the page table tree to
> > detect whether a region is accessed or not. For example, if the access
> > bit for a PUD entry is set, then one or more pages in the 1GB PUD
> > subtree is accessed as each PUD entry covers 1GB mapping. Hence,
> > instead of sampling thousands of 4K/2M pages to detect accesses in a
> > large region, sampling at the higher level of page table tree is faster=
 and
> efficient.
>=20
> Due to the above reason, I concern this could result in making DAMON
> monitoring results be inaccurately biased to report more than real access=
es.
>=20
> >
> > This patch set is based on 6.8-rc5 kernel (commit: f48159f8,
> > mm-unstable
> > tree)
> >
> > Changes since v1 [1]
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >
> >  - Added support for 5-level page table tree
> >  - Split the patch to mm infrastructure changes and DAMON enhancements
> >  - Code changes as per comments on v1
> >  - Added kerneldoc comments
> >
> > [1] https://lkml.org/lkml/2023/12/15/272
> >
> > Evaluation:
> >
> > - MASIM benchmark with 1GB, 10GB, 100GB footprint with 10% hot data
> >   and 5TB with 10GB hot data.
> > - DAMON: 5ms sampling, 200ms aggregation interval. Rest all
> >   parameters set to default value.
> > - DAMON+PTP: Page table profiling applied to DAMON with the above
> >   parameters.
> >
> > Profiling efficiency in detecting hot data:
> >
> > Footprint	1GB	10GB	100GB	5TB
> > ---------------------------------------------
> > DAMON		>90%	<50%	 ~0%	  0%
> > DAMON+PTP	>90%	>90%	>90%	>90%
>=20
> Sampling interval is the time interval that assumed to be large enough fo=
r the
> workload to make meaningful amount of accesses within the interval.  Henc=
e,
> meaningful amount of sampling interval depends on the workload's
> characteristic and system's memory bandwidth.
>=20
> Here, the size of the hot memory region is about 100MB, 1GB, 10GB, and
> 10GB for the four cases, respectively.  And you set the sampling interval=
 as
> 5ms.  Let's assume the system can access, say, 50 GB per second, and henc=
e it
> could be able to access only up to 250 MB per 5ms.  So, in case of 1GB an=
d
> footprint, all hot memory region would be accessed while DAMON is waiting
> for next sampling interval.  Hence, DAMON would be able to see most
> accesses via sampling.  But for 100GB footprint case, only 250MB / 10GB =
=3D
> about 2.5% of the hot memory region would be accessed between the
> sampling interval.  DAMON cannot see whole accesses, and hence the
> precision could be low.
>=20
> I don't know exact memory bandwith of the system, but to detect the 10 GB
> hot region with 5ms sampling interval, the system should be able to acces=
s
> 2GB memory per millisecond, or about 2TB memory per second.  I think
> systems of such memory bandwidth is not that common.
>=20
> I show you also explored a configuration setting the aggregation interval
> higher.  But because each sampling checks only access between the samplin=
g
> interval, that might not help in this setup.  I'm wondering if you also e=
xplored
> increasing sampling interval.
>=20
> Sorry again for finding this concern not early enough.  But I think we ma=
y need
> to discuss about this first.
>=20
> [1] https://lkml.kernel.org/r/20231215201159.73845-1-sj@kernel.org
>=20
>=20
> Thanks,
> SJ
>=20
>=20
> >
> > CPU overheads (in billion cycles) for kdamond:
> >
> > Footprint	1GB	10GB	100GB	5TB
> > ---------------------------------------------
> > DAMON		1.15	19.53	3.52	9.55
> > DAMON+PTP	0.83	 3.20	1.27	2.55
> >
> > A detailed explanation and evaluation can be found in the arXiv paper:
> > https://arxiv.org/pdf/2311.10275.pdf
> >
> >
> > Aravinda Prasad (3):
> >   mm/damon: mm infrastructure support
> >   mm/damon: profiling enhancement
> >   mm/damon: documentation updates
> >
> >  Documentation/mm/damon/design.rst |  42 ++++++
> >  arch/x86/include/asm/pgtable.h    |  20 +++
> >  arch/x86/mm/pgtable.c             |  28 +++-
> >  include/linux/mmu_notifier.h      |  36 +++++
> >  include/linux/pgtable.h           |  79 ++++++++++
> >  mm/damon/vaddr.c                  | 233 ++++++++++++++++++++++++++++--
> >  6 files changed, 424 insertions(+), 14 deletions(-)
> >
> > --
> > 2.21.3