Received-SPF: pass (google.com: domain of linux-kernel+bounces-173535-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: Zi Yan <ziy@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Lance Yang <ioworker0@gmail.com>, Alistair Popple <apopple@nvidia.com>,
 akpm@linux-foundation.org, willy@infradead.org, sj@kernel.org,
 maskray@google.com, ryan.roberts@arm.com, david@redhat.com,
 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com,
 zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com,
 libang.li@antgroup.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com,
 peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org,
 linux-kernel@vger.kernel.org, Baolin Wang <baolin.wang@linux.alibaba.com>
Subject: Re: [PATCH v4 2/3] mm/rmap: integrate PMD-mapped folio splitting into
 pagewalk loop
Date: Wed, 08 May 2024 12:22:08 -0400
Message-ID: <30469615-2DDC-467E-A810-5EE8E1CFCB43@nvidia.com>
In-Reply-To: <20240508155253.GK4650@nvidia.com>
References: <20240501042700.83974-1-ioworker0@gmail.com>
 <20240501042700.83974-3-ioworker0@gmail.com>
 <0077A412-0AF1-4022-8F49-EE77AE601ECB@nvidia.com>
 <CAK1f24mRVam3w9CZkq4O0smGUTVUQfdvA48mqsU+NKxMUxdz1A@mail.gmail.com>
 <B1DA2DA4-AFEA-4541-8DFC-0D815249A064@nvidia.com>
 <CAK1f24mpt+ZGow4LLdXKZcwopmkZDrM_eh15SnkEgpiBXTmOwA@mail.gmail.com>
 <10BA9EED-A2BB-44C2-B80A-26527CDFFA50@nvidia.com>
 <1B2017A4-A252-4C1F-9608-D43ECEAD53B1@nvidia.com>
 <20240508155253.GK4650@nvidia.com>
Content-Type: multipart/signed;
 boundary="=_MailMate_4B9AA9E3-1544-402F-88E0-F46055751397_=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?oquja3nTjDYeP1NRMZNAWHp3VkyIOimTr8zgmeovniFxsXdVc6lfrDtmgoeM?=
 =?us-ascii?Q?2oPWYnTppkEYF8k6GEE+sJkxuhgd4QtymwU4+CbgeMUygzDywXsJxe2kGBeG?=
 =?us-ascii?Q?GKraXsrhxcL+uEDscZuEzP+qKgKC0oDYsHzMtHF1lKLQyEc8s6imqNltRZ+p?=
 =?us-ascii?Q?R/S7zU5VSsG+B19lEjlbGiB/IBhgxgimLiX8Zc5JxtIIUh3gcE5vcitgjmlD?=
 =?us-ascii?Q?Ai+NhKkmzCtDSNajxa3z9pjM6u8/g6cGpQisrZ3dX7Fo7TdigXA14DYENf0X?=
 =?us-ascii?Q?Xoo40LqchSm8sfdSH/qvwYL6mWY2cGuTRAiO1qguu6nGDIxGVzgp3EyW7YQD?=
 =?us-ascii?Q?9KkZUebk/O+7lp+3aN/ErzvDabmkDaDwrAEO4y13fOe25J/82EQqG8vz6QzY?=
 =?us-ascii?Q?Fc7npPt5lkkAUR8DUfrklFnDcqKBHD58FbHjWsq/IBG3zuOCXL7iw6vQlVAv?=
 =?us-ascii?Q?rx+XoRhigEHs4uU5a8pcE5zfXEpT3Fg8au6Jo5BENjCkZhEjh2ZiOdNgjoWN?=
 =?us-ascii?Q?4TlCBX6x9zLGnKbQDDsUGy9dtN4hhvnb6psZCZcYSuyJzXfTMV1gfGo2wzkt?=
 =?us-ascii?Q?E+ReFlHRMEQwzLveKmdq4r1crRKE6BPvCbOmlqS0XNm7NVidjPfG4H9XlkfA?=
 =?us-ascii?Q?8kleHqie4lXIkHmFp+EDtcj5slU6/x9P/iFkhPDuoXdZ+87fXq1nwdQ8XxII?=
 =?us-ascii?Q?e5Yz05L9RnOs7BODgWSx53FHxXuKikVee8lCagt4P/hrO9S73BJqR654OOiz?=
 =?us-ascii?Q?GNkVH343IwbMOJrZDZVF1UHPi2yeQjRz+d73EulZcR31c7LESOMMQMOHEvtb?=
 =?us-ascii?Q?U/n/iEIWQk3QBQvh2uIgDNGZQHPv05Cf9Uq+CM7lZL6p+T0khENZCmJhdISg?=
 =?us-ascii?Q?pUT9SP9iRf4jAsaugoRYgthvCrj4xRZWQXgAnLYuw8eFTrWdLyACs/sm2YVf?=
 =?us-ascii?Q?HY9HdrO7gx90QRRzP2CY6hZtXHY5ffv/de08aNC1qQrAHKNIMPgf0riTEUZB?=
 =?us-ascii?Q?rBdrfk1pBACCA76RSKn5AXy5FzI6prt/XlexeWngwdnios7SnNDYTPCzIcb6?=
 =?us-ascii?Q?UdpI/xXi2DO4yif55o/vgDbMO3NVwbdvNUDD7Wqvxd4v14sdRGejwTI97zm1?=
 =?us-ascii?Q?KPb2bcm5mFayY5Cz7nMV3wM0b3rUdbZxQEi1KqF6hPNIt3NrYe4KScBqrQUb?=
 =?us-ascii?Q?Cxm6J+w+vWvy75+7rCXHud+mYRDzsaM0X8rzQ3eVSX9OjhKCY06XpbGA16Tv?=
 =?us-ascii?Q?G6uFtrFPjdyBA3LughV+3JKMvUBcYf+mU1UE55YBRKWQIDUVBMiHg1rDts5U?=
 =?us-ascii?Q?vP0QDJ4yONXyw9sUEyYj+NvmZXbjIy5kMSogCl5o6nrOB66M2R5XGSJOHLR+?=
 =?us-ascii?Q?2gekjOJBUkJjKdtYe+hPoegsJxsysUXTdLHGgI8vz6Z5ooBsEkfsr4KltuZw?=
 =?us-ascii?Q?hiGSrNCCWMSRQ1/pP+Jl6fwvcgIvIWRSO1v+SlNVNQOWidJ/ZX8chJGzEUza?=
 =?us-ascii?Q?ZUWBBS8fDdpktfB0eZOpU4wUrqYXhZPawOGpdPJ2c1Yx0FWbZVJrQGAzkUcD?=
 =?us-ascii?Q?myzs9qTVYw6TMryjcQq+4xIRbGp4X8/ggEvEvsal?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1fd65404-8b2d-44ce-ee69-08dc6f7b03e2
X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB5744.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2024 16:22:11.8195
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: NJpsVsqm3qVAuwPpwN1PXsFuZfIRNxrL5dPsBJgj2ZTfsUBu//Q1+Y2ElBLbC6yu
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR12MB9385

--=_MailMate_4B9AA9E3-1544-402F-88E0-F46055751397_=
Content-Type: text/plain

On 8 May 2024, at 11:52, Jason Gunthorpe wrote:

> On Wed, May 08, 2024 at 10:56:34AM -0400, Zi Yan wrote:
>
>> Lance is improving try_to_unmap_one() to support unmapping PMD THP as a whole,
>> so he moves split_huge_pmd_address() inside while (page_vma_mapped_walk(&pvmw))
>> and after mmu_notifier_invalidate_range_start() as split_huge_pmd_locked()
>> and does not include the mmu notifier ops inside split_huge_pmd_address().
>> I wonder if that could cause issues, since the mmu_notifier_invalidate_range_start()
>> before the while loop only has range of the original address and
>> split huge pmd can affect the entire PMD address range and these two ranges
>> might not be the same.
>
> That does not sound entirely good..
>
> I suppose it depends on what split does, if the MM page table has the
> same translation before and after split then perhaps no invalidation
> is even necessary.

Before split, it is a PMD mapping to a PMD THP (order-9). After split,
they are 512 PTEs mapping to the same THP. Unless the secondary TLB
does not support PMD mapping and use 512 PTEs instead, it seems to
be an issue from my understanding.

In terms of two mmu_notifier ranges, first is in the split_huge_pmd_address()[1]
and second is in try_to_unmap_one()[2]. When try_to_unmap_one() is unmapping
a subpage in the middle of a PMD THP, the former notifies about the PMD range
change due to one PMD split into 512 PTEs and the latter only needs to notify
about the invalidation of the unmapped PTE. I do not think the latter can
replace the former, although a potential optimization can be that the latter
can be removed as it is included in the range of the former.

Regarding Lance's current code change, is it OK to change mmu_notifier range
after mmu_notifier_invalidate_range_start()? Since in Lance's code, first
mmu_notifier is gone and second, whose range is only about the single PTE,
starts mmu_notifier_invalidate_range_start(), then the code finds that
a PMD needs to be split into 512 PTEs. Would changing the range from PTE
to PMD suffice? Or the code should call mmu_notifier_invalidate_range_start()
with the new PMD range? I am not even sure two start with one end is legit.

[1] https://elixir.bootlin.com/linux/v6.9-rc7/source/mm/huge_memory.c#L2658
[2] https://elixir.bootlin.com/linux/v6.9-rc7/source/mm/rmap.c#L1650

--
Best Regards,
Yan, Zi

--=_MailMate_4B9AA9E3-1544-402F-88E0-F46055751397_=
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename=signature.asc
Content-Type: application/pgp-signature; name=signature.asc

-----BEGIN PGP SIGNATURE-----

iQJDBAEBCgAtFiEE6rR4j8RuQ2XmaZol4n+egRQHKFQFAmY7prAPHHppeUBudmlk
aWEuY29tAAoJEOJ/noEUByhUwJEP/juV2ZebEx7Bo8cgIA4UiWMq7pG/iMExCP2U
8loEtknGwzcx1Qit4aYVLlXpan16egJnOGJFmy8G1702cqlFb3LpoUNNb1xSeGd4
x5zNH4hrKN3xpP00in0cpWl3iYmuW8JTgdkO3BjGoESwH7fRAROy4JiR037epoWP
fumQnKc7GlbZRap1KqbtABM0pCb1+Ajg5B/+feojUJXLSAcGjXs2X2wmjqApvkYX
19E2brS+NjOwxamnQMPWUSOFydIhvnOrc6t5bpajJRU0qWd85PFjP/Vcp7Gr0O1Z
wJt/bZvlYeaoOd3Rm7WUDdy6JGwMmXdpz3fN9FfFld7/FZZCT8uygBHKWaeE1jHs
tRvN/OVEU7hZU8ghQyxsQtUjwaSArGX6llCs3+piLGsjVZQ9eTxot19Gx/UF98ho
WATzR6GqAIL+1yXC3RDegLAVRv949Ucxx3sh6aJOvd47JmZ7eWaRfkVEgk2YDSY/
nBTizzkciX++1ehyygQ+CQQP7bz7vdNXmO4g72kUw1KPdm+YgZ5JmwOyHBp1izcN
Y0jBlMdRTv8tHuf4i7cYNMymBILqwNv+lHnKJ2goX/BVMIosMcoAnk+WW7GpNSVM
jtCTE9Zco+m4jWJZ+y2L82r2WuUAjaB1lRpegraaw1oQrTDr0YTtqWyC7IOzRkIl
3CZmqGRN
=w7YB
-----END PGP SIGNATURE-----

--=_MailMate_4B9AA9E3-1544-402F-88E0-F46055751397_=--