Received-SPF: pass (google.com: domain of linux-kernel+bounces-212912-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <02f8cbd0-8b2b-4c2d-ad96-f854d25bf3c2@redhat.com>
Date: Thu, 13 Jun 2024 11:04:57 +0200
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH 0/3] asynchronously scan and free empty user PTE pages
To: Qi Zheng <zhengqi.arch@bytedance.com>, hughd@google.com,
 willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev,
 akpm@linux-foundation.org
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org
References: <cover.1718267194.git.zhengqi.arch@bytedance.com>
Content-Language: en-US
From: David Hildenbrand <david@redhat.com>
Autocrypt: addr=david@redhat.com; keydata=
 xsFNBFXLn5EBEAC+zYvAFJxCBY9Tr1xZgcESmxVNI/0ffzE/ZQOiHJl6mGkmA1R7/uUpiCjJ
 dBrn+lhhOYjjNefFQou6478faXE6o2AhmebqT4KiQoUQFV4R7y1KMEKoSyy8hQaK1umALTdL
 QZLQMzNE74ap+GDK0wnacPQFpcG1AE9RMq3aeErY5tujekBS32jfC/7AnH7I0v1v1TbbK3Gp
 XNeiN4QroO+5qaSr0ID2sz5jtBLRb15RMre27E1ImpaIv2Jw8NJgW0k/D1RyKCwaTsgRdwuK
 Kx/Y91XuSBdz0uOyU/S8kM1+ag0wvsGlpBVxRR/xw/E8M7TEwuCZQArqqTCmkG6HGcXFT0V9
 PXFNNgV5jXMQRwU0O/ztJIQqsE5LsUomE//bLwzj9IVsaQpKDqW6TAPjcdBDPLHvriq7kGjt
 WhVhdl0qEYB8lkBEU7V2Yb+SYhmhpDrti9Fq1EsmhiHSkxJcGREoMK/63r9WLZYI3+4W2rAc
 UucZa4OT27U5ZISjNg3Ev0rxU5UH2/pT4wJCfxwocmqaRr6UYmrtZmND89X0KigoFD/XSeVv
 jwBRNjPAubK9/k5NoRrYqztM9W6sJqrH8+UWZ1Idd/DdmogJh0gNC0+N42Za9yBRURfIdKSb
 B3JfpUqcWwE7vUaYrHG1nw54pLUoPG6sAA7Mehl3nd4pZUALHwARAQABzSREYXZpZCBIaWxk
 ZW5icmFuZCA8ZGF2aWRAcmVkaGF0LmNvbT7CwZgEEwEIAEICGwMGCwkIBwMCBhUIAgkKCwQW
 AgMBAh4BAheAAhkBFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAl8Ox4kFCRKpKXgACgkQTd4Q
 9wD/g1oHcA//a6Tj7SBNjFNM1iNhWUo1lxAja0lpSodSnB2g4FCZ4R61SBR4l/psBL73xktp
 rDHrx4aSpwkRP6Epu6mLvhlfjmkRG4OynJ5HG1gfv7RJJfnUdUM1z5kdS8JBrOhMJS2c/gPf
 wv1TGRq2XdMPnfY2o0CxRqpcLkx4vBODvJGl2mQyJF/gPepdDfcT8/PY9BJ7FL6Hrq1gnAo4
 3Iv9qV0JiT2wmZciNyYQhmA1V6dyTRiQ4YAc31zOo2IM+xisPzeSHgw3ONY/XhYvfZ9r7W1l
 pNQdc2G+o4Di9NPFHQQhDw3YTRR1opJaTlRDzxYxzU6ZnUUBghxt9cwUWTpfCktkMZiPSDGd
 KgQBjnweV2jw9UOTxjb4LXqDjmSNkjDdQUOU69jGMUXgihvo4zhYcMX8F5gWdRtMR7DzW/YE
 BgVcyxNkMIXoY1aYj6npHYiNQesQlqjU6azjbH70/SXKM5tNRplgW8TNprMDuntdvV9wNkFs
 9TyM02V5aWxFfI42+aivc4KEw69SE9KXwC7FSf5wXzuTot97N9Phj/Z3+jx443jo2NR34XgF
 89cct7wJMjOF7bBefo0fPPZQuIma0Zym71cP61OP/i11ahNye6HGKfxGCOcs5wW9kRQEk8P9
 M/k2wt3mt/fCQnuP/mWutNPt95w9wSsUyATLmtNrwccz63XOwU0EVcufkQEQAOfX3n0g0fZz
 Bgm/S2zF/kxQKCEKP8ID+Vz8sy2GpDvveBq4H2Y34XWsT1zLJdvqPI4af4ZSMxuerWjXbVWb
 T6d4odQIG0fKx4F8NccDqbgHeZRNajXeeJ3R7gAzvWvQNLz4piHrO/B4tf8svmRBL0ZB5P5A
 2uhdwLU3NZuK22zpNn4is87BPWF8HhY0L5fafgDMOqnf4guJVJPYNPhUFzXUbPqOKOkL8ojk
 CXxkOFHAbjstSK5Ca3fKquY3rdX3DNo+EL7FvAiw1mUtS+5GeYE+RMnDCsVFm/C7kY8c2d0G
 NWkB9pJM5+mnIoFNxy7YBcldYATVeOHoY4LyaUWNnAvFYWp08dHWfZo9WCiJMuTfgtH9tc75
 7QanMVdPt6fDK8UUXIBLQ2TWr/sQKE9xtFuEmoQGlE1l6bGaDnnMLcYu+Asp3kDT0w4zYGsx
 5r6XQVRH4+5N6eHZiaeYtFOujp5n+pjBaQK7wUUjDilPQ5QMzIuCL4YjVoylWiBNknvQWBXS
 lQCWmavOT9sttGQXdPCC5ynI+1ymZC1ORZKANLnRAb0NH/UCzcsstw2TAkFnMEbo9Zu9w7Kv
 AxBQXWeXhJI9XQssfrf4Gusdqx8nPEpfOqCtbbwJMATbHyqLt7/oz/5deGuwxgb65pWIzufa
 N7eop7uh+6bezi+rugUI+w6DABEBAAHCwXwEGAEIACYCGwwWIQQb2cqtc1xMOkYN/MpN3hD3
 AP+DWgUCXw7HsgUJEqkpoQAKCRBN3hD3AP+DWrrpD/4qS3dyVRxDcDHIlmguXjC1Q5tZTwNB
 boaBTPHSy/Nksu0eY7x6HfQJ3xajVH32Ms6t1trDQmPx2iP5+7iDsb7OKAb5eOS8h+BEBDeq
 3ecsQDv0fFJOA9ag5O3LLNk+3x3q7e0uo06XMaY7UHS341ozXUUI7wC7iKfoUTv03iO9El5f
 XpNMx/YrIMduZ2+nd9Di7o5+KIwlb2mAB9sTNHdMrXesX8eBL6T9b+MZJk+mZuPxKNVfEQMQ
 a5SxUEADIPQTPNvBewdeI80yeOCrN+Zzwy/Mrx9EPeu59Y5vSJOx/z6OUImD/GhX7Xvkt3kq
 Er5KTrJz3++B6SH9pum9PuoE/k+nntJkNMmQpR4MCBaV/J9gIOPGodDKnjdng+mXliF3Ptu6
 3oxc2RCyGzTlxyMwuc2U5Q7KtUNTdDe8T0uE+9b8BLMVQDDfJjqY0VVqSUwImzTDLX9S4g/8
 kC4HRcclk8hpyhY2jKGluZO0awwTIMgVEzmTyBphDg/Gx7dZU1Xf8HFuE+UZ5UDHDTnwgv7E
 th6RC9+WrhDNspZ9fJjKWRbveQgUFCpe1sa77LAw+XFrKmBHXp9ZVIe90RMe2tRL06BGiRZr
 jPrnvUsUUsjRoRNJjKKA/REq+sAnhkNPPZ/NNMjaZ5b8Tovi8C0tmxiCHaQYqj7G2rgnT0kt
 WNyWQQ==
Organization: Red Hat
In-Reply-To: <cover.1718267194.git.zhengqi.arch@bytedance.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 13.06.24 10:38, Qi Zheng wrote:
> Hi all,
> 
> This series aims to asynchronously scan and free empty user PTE pages.
> 
> 1. Background
> =============
> 
> We often find huge user PTE memory usage on our servers, such as the following:
> 
>          VIRT:  55t
>          RES:   590g
>          VmPTE: 110g
>          
> The root cause is that these processes use some high-performance mmeory
> allocators (such as jemalloc, tcmalloc, etc). These memory allocators use
> madvise(MADV_DONTNEED or MADV_FREE) to release physical memory, but neither
> MADV_DONTNEED nor MADV_FREE will release page table memory, which may cause
> huge page table memory usage.
> 
> This issue has been discussed on LSFMM 2022 (led by David Hildenbrand):
> 
>          topic link: https://lore.kernel.org/linux-mm/7b908208-02f8-6fde-4dfc-13d5e00310a6@redhat.com/
>          youtube link: https://www.youtube.com/watch?v=naO_BRhcU68
>          
> In the past, I have tried to introduce refcount for PTE pages to solve this
> problem, but these methods [1][2][3] introduced too much complexity.
> 
> [1]. https://lore.kernel.org/lkml/20211110105428.32458-1-zhengqi.arch@bytedance.com/
> [2]. https://lore.kernel.org/lkml/20220429133552.33768-1-zhengqi.arch@bytedance.com/
> [3]. https://lore.kernel.org/lkml/20220825101037.96517-1-zhengqi.arch@bytedance.com/
> 
> 2. Infrastructure
> =================
> 
> Later, in order to freeing retracted page table, Hugh Dickins added a lot of
> PTE-related infrastructure[4][5][6]:
> 
>      - allow pte_offset_map_lock() etc to fail
>      - make PTE pages can be removed without mmap or rmap locks
>        (see collapse_pte_mapped_thp() and retract_page_tables())
>      - make PTE pages can be freed by RCU (via pte_free_defer())
>      - etc
>      
> These are all beneficial to freeing empty PTE pages.
> 
> [4]. https://lore.kernel.org/all/a4963be9-7aa6-350-66d0-2ba843e1af44@google.com/
> [5]. https://lore.kernel.org/all/c1c9a74a-bc5b-15ea-e5d2-8ec34bc921d@google.com/
> [6]. https://lore.kernel.org/all/7cd843a9-aa80-14f-5eb2-33427363c20@google.com/
> 

I'm a big fan for virtio-mem.


> 3. Implementation
> =================
> 
> For empty user PTE pages, we don't actually need to free it immediately, nor do
> we need to free all of it.
> 
> Therefore, in this patchset, we register a task_work for the user tasks to
> asyncronously scan and free empty PTE pages when they return to user space.
> (The scanning time interval and address space size can be adjusted.)

The question is, if we really have to scan asynchronously, or if would 
be reasonable for most use cases to trigger a madvise(MADV_PT_RECLAIM) 
every now and then. For virtio-mem, and likely most memory allocators, 
that might be feasible, and valuable independent of system-wide 
automatic scanning.

> 
> When scanning, we can filter out some unsuitable vmas:
> 
>      - VM_HUGETLB vma
>      - VM_UFFD_WP vma

Why is UFFD_WP unsuitable? It should be suitable as long as you make 
sure to really only remove page tables that are all pte_none().

>      - etc
>      
> And for some PTE pages that spans multiple vmas, we can also skip.
> 
> For locking:
> 
>      - use the mmap read lock to traverse the vma tree and pgtable
>      - use pmd lock for clearing pmd entry
>      - use pte lock for checking empty PTE page, and release it after clearing
>        pmd entry, then we can capture the changed pmd in pte_offset_map_lock()
>        etc after holding this pte lock. Thanks to this, we don't need to hold the
>        rmap-related locks.
>      - users of pte_offset_map_lock() etc all expect the PTE page to be stable by
>        using rcu lock, so use pte_free_defer() to free PTE pages.
>     

I once had a protoype that would scan similar to GUP-fast, using the 
mmap lock in read mode and disabling local IRQs and then walking the 
page table locklessly (no PTLs). Only when identifying an empty page and 
ripping out the page table, it would have to do more heavy locking (back 
when we required the mmap lock in write mode and other things).

I can try digging up that patch if you're interested.

We'll have to double check whether all anon memory cases can *properly* 
handle pte_offset_map_lock() failing (not just handling it, but doing 
the right thing; most of that anon-only code didn't ever run into that 
issue so far, so these code paths were likely never triggered).


> For the path that will also free PTE pages in THP, we need to recheck whether the
> content of pmd entry is valid after holding pmd lock or pte lock.
> 
> 4. TODO
> =======
> 
> Some applications may be concerned about the overhead of scanning and rebuilding
> page tables, so the following features are considered for implementation in the
> future:
> 
>      - add per-process switch (via prctl)
>      - add a madvise option (like THP)
>      - add MM_PGTABLE_SCAN_DELAY/MM_PGTABLE_SCAN_SIZE control (via procfs file)
>      
> Perhaps we can add the refcount to PTE pages in the future as well, which would
> help improve the scanning speed.

I didn't like the added complexity last time, and the problem of 
handling situations where we squeeze multiple page tables into a single 
"struct page".

-- 
Cheers,

David / dhildenb