Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Wed, 10 Nov 2021 08:56:01 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Qi Zheng <zhengqi.arch@bytedance.com>
Cc:     akpm@linux-foundation.org, tglx@linutronix.de,
        kirill.shutemov@linux.intel.com, mika.penttila@nextfour.com,
        david@redhat.com, linux-doc@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        songmuchun@bytedance.com, zhouchengming@bytedance.com
Subject: Re: [PATCH v3 00/15] Free user PTE page table pages
Message-ID: <20211110125601.GQ1740502@nvidia.com>
References: <20211110105428.32458-1-zhengqi.arch@bytedance.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20211110105428.32458-1-zhengqi.arch@bytedance.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?wzil/l9ET96mgnLcJBj3f7AXJcoBPgrgADsAsMbtSogKNPzGZDCuz6CRev1b?=
 =?us-ascii?Q?xhYPM6Jd5P0b6d3g2FaW7IG+jsXj0+OZQJPUj8LXZUn31YqvUiuy8DafJieT?=
 =?us-ascii?Q?yJAXqcq+QbT4W4hmG6w0cufE/lsJIU2sdPBPrXIWAjdUmgEN+wPmndaV47bL?=
 =?us-ascii?Q?hGEZGhoPZy6IiQpj/3k9wdQK9TyGJ2fNVVu5bof6I1c27N0zBOrhEH9cz6yb?=
 =?us-ascii?Q?BZH8fPd3Sua9ozhkkeLn9Y2PPX3V26nI56CzMGfXqndG7vUSB0kz//ksiYUw?=
 =?us-ascii?Q?IxPvgVGv/QMPMlHsEs5glYvxRD2UpPJ+NP0c5pZlCa6Zzj0bEmXz+5nUrtcw?=
 =?us-ascii?Q?JbOuU4M7uXeuOwmgkjISkGxcGJk6Hm6+iMw9/eyhExdiQHujJ5cPkH8qAc9H?=
 =?us-ascii?Q?67symdBA+1QC+gyJHKC8Nx2h9iWJ3hli+Rw5qZVuEJD3BBAmfgL1zktG5Yzx?=
 =?us-ascii?Q?Z10upu+FfJRmSwkGrtLE0mB4NI1ANC1AiBEn/53boDQeAdW3pe2bkp+ORAiJ?=
 =?us-ascii?Q?KJp9Qodf08l+S7f2ruyfBGoRgKAAh5Iu1Nbt8uflWGsuxu957SRGfF63XEra?=
 =?us-ascii?Q?nDglTyeU6CoQR7dFOkA+vgVq3cb3lu3bJ5zo5NIZSne7lLnsGeK9i6Gbd19j?=
 =?us-ascii?Q?83evTb/9jfTh6cvFWK025epdpvcKUHTC2A0xTT0QuPwC1DDL4r44oYQKV9Cj?=
 =?us-ascii?Q?VyA6+A/4b/CGOqfmheE2aq6hWjbtvL8EW01gO9oaN4Zy5iGJTzuHAzLReFRv?=
 =?us-ascii?Q?CPHCB1freT/KFbqfN38FG6/p8CqD/gQD4Rf5QTZZvhgoIab0j49ltfWmmh6f?=
 =?us-ascii?Q?LoKYZo36o4ETBXTWyxPJZJuj60xNNv7GUMxagunKdVm2YdlTBwK/a08A1UuC?=
 =?us-ascii?Q?JYReZhamwIorBHtSm/PDssUQaTkchyRTBy1copSAPMiERCN6Gc8nRgbvnjJh?=
 =?us-ascii?Q?j1PFrkwC5xkW8wAcSgwMsCNfDkO+lXZhXmC40gPnx2cadsGH0z+ZXUc/bbYX?=
 =?us-ascii?Q?kDhG7EaqmbAXA6FVlTnQoSffaQhjelMN6lw6meXCjvNfKdeNAi8KcO02PXlZ?=
 =?us-ascii?Q?Yp3vZ01P5hvt4Ii/6qQyLxDqX1oCFt5z0lwwdo33UuPj0upULdsYH1EUWw0D?=
 =?us-ascii?Q?Q8SXWnOpJLuzw23m1imRVjm8uO+ngN+uPKph2XoISCrz4/VDKRYZqTD0PLFm?=
 =?us-ascii?Q?dX7526RquuLw5JcY4OwBTaqFVXWim4N1ivw5Av1yof+S2puWRaE9NLn9jHLD?=
 =?us-ascii?Q?jfthX2n3atg0OF8uTQl/aFDVXpzZrP6mXQXFZt/5s1Qs7+T8JdVNib8Lfauq?=
 =?us-ascii?Q?mx1TPZXB4CYQK6LzaIufaEkL5K7GIEnAby8PEMIydXY0OzUV+kslS3wK7K2C?=
 =?us-ascii?Q?GN6OUhAgRPeBaLFpSGJBTtKBRbcrf213n5Hgx7KJ0tipMqL3IMa3c21inhR9?=
 =?us-ascii?Q?v0cAmTV1jcJ8pfLJwQyWPx870XqoIme9YvvjoKzNoQXzMx9nEpg6nwJpzCVq?=
 =?us-ascii?Q?EVH7+SHthuq8L/cAZr5Ny1W1eW8PFnKLiCEtcExe9jcfQi5kUMcVo2Y45B9a?=
 =?us-ascii?Q?37jLXT3DMURM7wS1JHM=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ba9f13bd-2c85-4be2-eca9-08d9a449739c
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Nov 2021 12:56:03.0052
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3NCjXxYPVuIi5lhdpHXNcTuk+o25lpqtL/tDzf/yQKOtoKl6DOe4cWZI7tcAM1f5
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB5505
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Nov 10, 2021 at 06:54:13PM +0800, Qi Zheng wrote:

> In this patch series, we add a pte_refcount field to the struct page of page
> table to track how many users of PTE page table. Similar to the mechanism of
> page refcount, the user of PTE page table should hold a refcount to it before
> accessing. The PTE page table page will be freed when the last refcount is
> dropped.

So, this approach basically adds two atomics on every PTE map

If I have it right the reason that zap cannot clean the PTEs today is
because zap cannot obtain the mmap lock due to a lock ordering issue
with the inode lock vs mmap lock.

If it could obtain the mmap lock then it could do the zap using the
write side as unmapping a vma does.

Rather than adding a new "lock" to ever PTE I wonder if it would be
more efficient to break up the mmap lock and introduce a specific
rwsem for the page table itself, in addition to the PTL. Currently the
mmap lock is protecting both the vma list and the page table.

I think that would allow the lock ordering issue to be resolved and
zap could obtain a page table rwsem.

Compared to two atomics per PTE this would just be two atomic per
page table walk operation, it is conceptually a lot simpler, and would
allow freeing all the page table levels, not just PTEs.

?

Jason