*Changes in v20*
- Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO
*Changes in v19*
- Minor changes and interface updates
*Changes in v18*
- Rebase on top of next-20230613
- Minor updates
*Changes in v17*
- Rebase on top of next-20230606
- Minor improvements in PAGEMAP_SCAN IOCTL patch
*Changes in v16*
- Fix a corner case
- Add exclusive PM_SCAN_OP_WP back
*Changes in v15*
- Build fix (Add missed build fix in RESEND)
*Changes in v14*
- Fix build error caused by #ifdef added at last minute in some configs
*Changes in v13*
- Rebase on top of next-20230414
- Give-up on using uffd_wp_range() and write new helpers, flush tlb only
once
*Changes in v12*
- Update and other memory types to UFFD_FEATURE_WP_ASYNC
- Rebaase on top of next-20230406
- Review updates
*Changes in v11*
- Rebase on top of next-20230307
- Base patches on UFFD_FEATURE_WP_UNPOPULATED
- Do a lot of cosmetic changes and review updates
- Remove ENGAGE_WP + !GET operation as it can be performed with
UFFDIO_WRITEPROTECT
*Changes in v10*
- Add specific condition to return error if hugetlb is used with wp
async
- Move changes in tools/include/uapi/linux/fs.h to separate patch
- Add documentation
*Changes in v9:*
- Correct fault resolution for userfaultfd wp async
- Fix build warnings and errors which were happening on some configs
- Simplify pagemap ioctl's code
*Changes in v8:*
- Update uffd async wp implementation
- Improve PAGEMAP_IOCTL implementation
*Changes in v7:*
- Add uffd wp async
- Update the IOCTL to use uffd under the hood instead of soft-dirty
flags
*Motivation*
The real motivation for adding PAGEMAP_SCAN IOCTL is to emulate Windows
GetWriteWatch() syscall [1]. The GetWriteWatch{} retrieves the addresses of
the pages that are written to in a region of virtual memory.
This syscall is used in Windows applications and games etc. This syscall is
being emulated in pretty slow manner in userspace. Our purpose is to
enhance the kernel such that we translate it efficiently in a better way.
Currently some out of tree hack patches are being used to efficiently
emulate it in some kernels. We intend to replace those with these patches.
So the whole gaming on Linux can effectively get benefit from this. It
means there would be tons of users of this code.
CRIU use case [2] was mentioned by Andrei and Danylo:
> Use cases for migrating sparse VMAs are binaries sanitized with ASAN,
> MSAN or TSAN [3]. All of these sanitizers produce sparse mappings of
> shadow memory [4]. Being able to migrate such binaries allows to highly
> reduce the amount of work needed to identify and fix post-migration
> crashes, which happen constantly.
Andrei's defines the following uses of this code:
* it is more granular and allows us to track changed pages more
effectively. The current interface can clear dirty bits for the entire
process only. In addition, reading info about pages is a separate
operation. It means we must freeze the process to read information
about all its pages, reset dirty bits, only then we can start dumping
pages. The information about pages becomes more and more outdated,
while we are processing pages. The new interface solves both these
downsides. First, it allows us to read pte bits and clear the
soft-dirty bit atomically. It means that CRIU will not need to freeze
processes to pre-dump their memory. Second, it clears soft-dirty bits
for a specified region of memory. It means CRIU will have actual info
about pages to the moment of dumping them.
* The new interface has to be much faster because basic page filtering
is happening in the kernel. With the old interface, we have to read
pagemap for each page.
*Implementation Evolution (Short Summary)*
From the definition of GetWriteWatch(), we feel like kernel's soft-dirty
feature can be used under the hood with some additions like:
* reset soft-dirty flag for only a specific region of memory instead of
clearing the flag for the entire process
* get and clear soft-dirty flag for a specific region atomically
So we decided to use ioctl on pagemap file to read or/and reset soft-dirty
flag. But using soft-dirty flag, sometimes we get extra pages which weren't
even written. They had become soft-dirty because of VMA merging and
VM_SOFTDIRTY flag. This breaks the definition of GetWriteWatch(). We were
able to by-pass this short coming by ignoring VM_SOFTDIRTY until David
reported that mprotect etc messes up the soft-dirty flag while ignoring
VM_SOFTDIRTY [5]. This wasn't happening until [6] got introduced. We
discussed if we can revert these patches. But we could not reach to any
conclusion. So at this point, I made couple of tries to solve this whole
VM_SOFTDIRTY issue by correcting the soft-dirty implementation:
* [7] Correct the bug fixed wrongly back in 2014. It had potential to cause
regression. We left it behind.
* [8] Keep a list of soft-dirty part of a VMA across splits and merges. I
got the reply don't increase the size of the VMA by 8 bytes.
At this point, we left soft-dirty considering it is too much delicate and
userfaultfd [9] seemed like the only way forward. From there onward, we
have been basing soft-dirty emulation on userfaultfd wp feature where
kernel resolves the faults itself when WP_ASYNC feature is used. It was
straight forward to add WP_ASYNC feature in userfautlfd. Now we get only
those pages dirty or written-to which are really written in reality. (PS
There is another WP_UNPOPULATED userfautfd feature is required which is
needed to avoid pre-faulting memory before write-protecting [9].)
All the different masks were added on the request of CRIU devs to create
interface more generic and better.
[1] https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getwritewatch
[2] https://lore.kernel.org/all/[email protected]
[3] https://github.com/google/sanitizers
[4] https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit
[5] https://lore.kernel.org/all/[email protected]
[6] https://lore.kernel.org/all/[email protected]/
[7] https://lore.kernel.org/all/[email protected]
[8] https://lore.kernel.org/all/[email protected]
[9] https://lore.kernel.org/all/[email protected]
[10] https://lore.kernel.org/all/[email protected]
* Original Cover letter from v8*
Hello,
Note:
Soft-dirty pages and pages which have been written-to are synonyms. As
kernel already has soft-dirty feature inside which we have given up to
use, we are using written-to terminology while using UFFD async WP under
the hood.
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are
supported in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
(PAGE_IS_SWAPPED).
- Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which
pages have been written-to.
- Find pages which have been written-to and write protect the pages
(atomic PAGE_IS_WRITTEN + PAGEMAP_WP_ENGAGE)
It is possible to find and clear soft-dirty pages entirely in userspace.
But it isn't efficient:
- The mprotect and SIGSEGV handler for bookkeeping
- The userfaultfd wp (synchronous) with the handler for bookkeeping
Some benchmarks can be seen here[1]. This series adds features that weren't
present earlier:
- There is no atomic get soft-dirty/Written-to status and clear present in
the kernel.
- The pages which have been written-to can not be found in accurate way.
(Kernel's soft-dirty PTE bit + sof_dirty VMA bit shows more soft-dirty
pages than there actually are.)
Historically, soft-dirty PTE bit tracking has been used in the CRIU
project. The procfs interface is enough for finding the soft-dirty bit
status and clearing the soft-dirty bit of all the pages of a process.
We have the use case where we need to track the soft-dirty PTE bit for
only specific pages on-demand. We need this tracking and clear mechanism
of a region of memory while the process is running to emulate the
getWriteWatch() syscall of Windows.
*(Moved to using UFFD instead of soft-dirtyi feature to find pages which
have been written-to from v7 patch series)*:
Stop using the soft-dirty flags for finding which pages have been
written to. It is too delicate and wrong as it shows more soft-dirty
pages than the actual soft-dirty pages. There is no interest in
correcting it [2][3] as this is how the feature was written years ago.
It shouldn't be updated to changed behaviour. Peter Xu has suggested
using the async version of the UFFD WP [4] as it is based inherently
on the PTEs.
So in this patch series, I've added a new mode to the UFFD which is
asynchronous version of the write protect. When this variant of the
UFFD WP is used, the page faults are resolved automatically by the
kernel. The pages which have been written-to can be found by reading
pagemap file (!PM_UFFD_WP). This feature can be used successfully to
find which pages have been written to from the time the pages were
write protected. This works just like the soft-dirty flag without
showing any extra pages which aren't soft-dirty in reality.
The information related to pages if the page is file mapped, present and
swapped is required for the CRIU project [5][6]. The addition of the
required mask, any mask, excluded mask and return masks are also required
for the CRIU project [5].
The IOCTL returns the addresses of the pages which match the specific
masks. The page addresses are returned in struct page_region in a compact
form. The max_pages is needed to support a use case where user only wants
to get a specific number of pages. So there is no need to find all the
pages of interest in the range when max_pages is specified. The IOCTL
returns when the maximum number of the pages are found. The max_pages is
optional. If max_pages is specified, it must be equal or greater than the
vec_size. This restriction is needed to handle worse case when one
page_region only contains info of one page and it cannot be compacted.
This is needed to emulate the Windows getWriteWatch() syscall.
The patch series include the detailed selftest which can be used as an
example for the uffd async wp test and PAGEMAP_IOCTL. It shows the
interface usages as well.
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/all/[email protected]
[3] https://lore.kernel.org/all/[email protected]
[4] https://lore.kernel.org/all/Y6Hc2d+7eTKs7AiH@x1n
[5] https://lore.kernel.org/all/[email protected]/
[6] https://lore.kernel.org/all/[email protected]/
Regards,
Muhammad Usama Anjum
Muhammad Usama Anjum (4):
fs/proc/task_mmu: Implement IOCTL to get and optionally clear info
about PTEs
tools headers UAPI: Update linux/fs.h with the kernel sources
mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL
selftests: mm: add pagemap ioctl tests
Peter Xu (1):
userfaultfd: UFFD_FEATURE_WP_ASYNC
Documentation/admin-guide/mm/pagemap.rst | 58 +
Documentation/admin-guide/mm/userfaultfd.rst | 35 +
fs/proc/task_mmu.c | 560 +++++++
fs/userfaultfd.c | 26 +-
include/linux/hugetlb.h | 1 +
include/linux/userfaultfd_k.h | 21 +-
include/uapi/linux/fs.h | 54 +
include/uapi/linux/userfaultfd.h | 9 +-
mm/hugetlb.c | 34 +-
mm/memory.c | 27 +-
tools/include/uapi/linux/fs.h | 54 +
tools/testing/selftests/mm/.gitignore | 2 +
tools/testing/selftests/mm/Makefile | 3 +-
tools/testing/selftests/mm/config | 1 +
tools/testing/selftests/mm/pagemap_ioctl.c | 1464 ++++++++++++++++++
tools/testing/selftests/mm/run_vmtests.sh | 4 +
16 files changed, 2329 insertions(+), 24 deletions(-)
create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c
mode change 100644 => 100755 tools/testing/selftests/mm/run_vmtests.sh
--
2.39.2
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are supported
in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT), swapped
(PAGE_IS_SWAPPED) or page has pfn zero (PAGE_IS_PFNZERO).
- Find pages which have been written-to and/or write protect the pages
(atomic PM_SCAN_OP_GET + PM_SCAN_OP_WP)
This IOCTL can be extended to get information about more PTE bits.
Signed-off-by: Muhammad Usama Anjum <[email protected]>
---
Changes in v20:
- Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO
Changes in v19:
- Interface changes such as renaming, return mask and WP can be used
with any flags specified in masks
- Internal code changes
Changes in v18:
- Rebased on top of next-20230613
- ptep_get() updates
- remove pmd_trans_unstable() and add ACTION_AGAIN
- Review updates (Micheal)
Changes in v17:
- Rebased on next-20230606
- Made make_uffd_wp_*_pte() better and minor changes
Changes in v16:
- Fixed a corner case where kernel writes beyond user buffer by one
element
- Bring back exclusive PM_SCAN_OP_WP
- Cosmetic changes
Changes in v15:
- Build fix:
- Use generic tlb flush function in pagemap_scan_pmd_entry() instead of
using x86 specific flush function in do_pagemap_scan()
- Remove #ifdef from pagemap_scan_hugetlb_entry()
- Use mm instead of undefined vma->vm_mm
Changes in v14:
- Fix build error caused by #ifdef added at last minute in some configs
Changes in v13:
- Review updates
- mmap_read_lock_killable() instead of mmap_read_lock()
- Replace uffd_wp_range() with helpers which increases performance
drastically for OP_WP operations by reducing the number of tlb
flushing etc
- Add MMU_NOTIFY_PROTECTION_VMA notification for the memory range
Changes in v12:
- Add hugetlb support to cover all memory types
- Merge "userfaultfd: Define dummy uffd_wp_range()" with this patch
- Review updates to the code
Changes in v11:
- Find written pages in a better way
- Fix a corner case (thanks Paul)
- Improve the code/comments
- remove ENGAGE_WP + ! GET operation
- shorten the commit message in favour of moving documentation to
pagemap.rst
Changes in v10:
- move changes in tools/include/uapi/linux/fs.h to separate patch
- update commit message
Change in v8:
- Correct is_pte_uffd_wp()
- Improve readability and error checks
- Remove some un-needed code
Changes in v7:
- Rebase on top of latest next
- Fix some corner cases
- Base soft-dirty on the uffd wp async
- Update the terminologies
- Optimize the memory usage inside the ioctl
---
fs/proc/task_mmu.c | 560 ++++++++++++++++++++++++++++++++++++++++
include/linux/hugetlb.h | 1 +
include/uapi/linux/fs.h | 54 ++++
mm/hugetlb.c | 2 +-
4 files changed, 616 insertions(+), 1 deletion(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 507cd4e59d07..c9d096db6974 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -19,6 +19,7 @@
#include <linux/shmem_fs.h>
#include <linux/uaccess.h>
#include <linux/pkeys.h>
+#include <linux/minmax.h>
#include <asm/elf.h>
#include <asm/tlb.h>
@@ -1765,11 +1766,570 @@ static int pagemap_release(struct inode *inode, struct file *file)
return 0;
}
+#define PM_SCAN_REQUIRE_UFFD (1ULL << 63)
+
+#define PM_SCAN_FOUND_MAX_PAGES (1)
+#define PM_SCAN_BUFFER_FULL (-256)
+
+#define PM_SCAN_BITS_ALL (PAGE_IS_WRITTEN | PAGE_IS_FILE | \
+ PAGE_IS_PRESENT | PAGE_IS_SWAPPED | \
+ PAGE_IS_PFNZERO)
+#define PM_SCAN_OPS (PM_SCAN_OP_GET | PM_SCAN_OP_WP)
+#define IS_PM_SCAN_GET(flags) (flags & PM_SCAN_OP_GET)
+#define IS_PM_SCAN_WP(flags) (flags & PM_SCAN_OP_WP)
+
+#define PM_SCAN_FLAGS(wt, file, present, swap, pfnzero) \
+ ((wt) | ((file) << 1) | ((present) << 2) | \
+ ((swap) << 3) | ((pfnzero) << 4))
+
+struct pagemap_scan_private {
+ struct page_region *vec_buf, cur_buf;
+ unsigned long long vec_buf_len, vec_buf_index, max_pages, found_pages, flags;
+ unsigned long long required_mask, anyof_mask, excluded_mask, return_mask;
+};
+
+static inline bool is_pte_uffd_wp(pte_t pte)
+{
+ return (pte_present(pte) && pte_uffd_wp(pte)) ||
+ pte_swp_uffd_wp_any(pte);
+}
+
+static inline void make_uffd_wp_pte(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *pte)
+{
+ pte_t ptent = ptep_get(pte);
+
+ if (pte_present(ptent)) {
+ pte_t old_pte;
+
+ old_pte = ptep_modify_prot_start(vma, addr, pte);
+ ptent = pte_mkuffd_wp(ptent);
+ ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
+ } else if (is_swap_pte(ptent)) {
+ ptent = pte_swp_mkuffd_wp(ptent);
+ set_pte_at(vma->vm_mm, addr, pte, ptent);
+ } else {
+ set_pte_at(vma->vm_mm, addr, pte,
+ make_pte_marker(PTE_MARKER_UFFD_WP));
+ }
+}
+
+static inline bool pagemap_scan_is_file(struct vm_area_struct *vma, pte_t ptent,
+ unsigned long addr)
+{
+ struct page *page = NULL;
+ swp_entry_t entry;
+
+ if (pte_present(ptent)) {
+ page = vm_normal_page(vma, addr, ptent);
+ } else {
+ entry = pte_to_swp_entry(ptent);
+ if (is_pfn_swap_entry(entry))
+ page = pfn_swap_entry_to_page(entry);
+ }
+
+ if (page && !PageAnon(page))
+ return true;
+
+ return false;
+}
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static inline bool is_pmd_uffd_wp(pmd_t pmd)
+{
+ return (pmd_present(pmd) && pmd_uffd_wp(pmd)) ||
+ (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd));
+}
+
+static inline void make_uffd_wp_pmd(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmdp)
+{
+ pmd_t old, pmd = *pmdp;
+
+ if (pmd_present(pmd)) {
+ old = pmdp_invalidate_ad(vma, addr, pmdp);
+ pmd = pmd_mkuffd_wp(old);
+ set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
+ } else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
+ pmd = pmd_swp_mkuffd_wp(pmd);
+ set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
+ }
+}
+#endif
+
+#ifdef CONFIG_HUGETLB_PAGE
+static inline bool is_huge_pte_uffd_wp(pte_t pte)
+{
+ return (pte_present(pte) && huge_pte_uffd_wp(pte)) ||
+ pte_swp_uffd_wp_any(pte);
+}
+
+static inline void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ pte_t ptent)
+{
+ if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent))
+ return;
+
+ if (is_hugetlb_entry_migration(ptent))
+ set_huge_pte_at(vma->vm_mm, addr, ptep,
+ pte_swp_mkuffd_wp(ptent));
+ else if (!huge_pte_none(ptent))
+ huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
+ huge_pte_mkuffd_wp(ptent));
+ else
+ set_huge_pte_at(vma->vm_mm, addr, ptep,
+ make_pte_marker(PTE_MARKER_UFFD_WP));
+}
+
+static inline bool pagemap_scan_is_huge_file(pte_t pte)
+{
+ if (pte_present(pte) && (!PageAnon(pte_page(pte))))
+ return true;
+
+ return false;
+}
+#endif
+
+static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
+ struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+
+ if ((p->flags & PM_SCAN_REQUIRE_UFFD) && (!userfaultfd_wp_async(vma) ||
+ !userfaultfd_wp_use_markers(vma)))
+ return -EPERM;
+
+ if (vma->vm_flags & VM_PFNMAP)
+ return 1;
+
+ return 0;
+}
+
+static int pagemap_scan_output(unsigned long bitmap,
+ struct pagemap_scan_private *p,
+ unsigned long addr, unsigned int n_pages)
+{
+ struct page_region *cur_buf = &p->cur_buf;
+
+ if (!n_pages)
+ return -EINVAL;
+
+ bitmap &= p->return_mask;
+
+ if (cur_buf->flags == bitmap &&
+ cur_buf->start + cur_buf->len * PAGE_SIZE == addr) {
+ cur_buf->len += n_pages;
+ p->found_pages += n_pages;
+ } else {
+ if (cur_buf->len) {
+ if (p->vec_buf_index >= p->vec_buf_len)
+ return PM_SCAN_BUFFER_FULL;
+
+ memcpy(&p->vec_buf[p->vec_buf_index], cur_buf,
+ sizeof(*p->vec_buf));
+ p->vec_buf_index++;
+ }
+
+ cur_buf->start = addr;
+ cur_buf->len = n_pages;
+ cur_buf->flags = bitmap;
+ p->found_pages += n_pages;
+ }
+
+ if (p->found_pages == p->max_pages)
+ return PM_SCAN_FOUND_MAX_PAGES;
+
+ return 0;
+}
+
+static bool pagemap_scan_is_interesting_page(unsigned long bitmap,
+ struct pagemap_scan_private *p)
+{
+ if ((p->required_mask & bitmap) != p->required_mask)
+ return false;
+ if (p->anyof_mask && !(p->anyof_mask & bitmap))
+ return false;
+ if (p->excluded_mask & bitmap)
+ return false;
+
+ return true;
+}
+
+static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
+ unsigned long end, struct mm_walk *walk)
+{
+ bool is_written, flush = false, is_interesting = true;
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ unsigned long bitmap, addr = end;
+ pte_t *pte, *orig_pte, ptent;
+ spinlock_t *ptl;
+ int ret = 0;
+
+ arch_enter_lazy_mmu_mode();
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ ptl = pmd_trans_huge_lock(pmd, vma);
+ if (ptl) {
+ unsigned long n_pages = (end - start)/PAGE_SIZE;
+
+ if (p->max_pages && n_pages > p->max_pages - p->found_pages)
+ n_pages = p->max_pages - p->found_pages;
+
+ is_written = !is_pmd_uffd_wp(*pmd);
+
+ /*
+ * Break huge page into small pages if the WP operation need to
+ * be performed is on a portion of the huge page.
+ */
+ if (is_written && IS_PM_SCAN_WP(p->flags) &&
+ n_pages < HPAGE_SIZE/PAGE_SIZE) {
+ spin_unlock(ptl);
+
+ split_huge_pmd(vma, pmd, start);
+ goto process_smaller_pages;
+ }
+
+ bitmap = PM_SCAN_FLAGS(is_written, false,
+ pmd_present(*pmd), is_swap_pmd(*pmd),
+ pmd_present(*pmd) && is_zero_pfn(pmd_pfn(*pmd)));
+
+ if (IS_PM_SCAN_GET(p->flags)) {
+ is_interesting = pagemap_scan_is_interesting_page(bitmap, p);
+ if (is_interesting)
+ ret = pagemap_scan_output(bitmap, p, start, n_pages);
+ }
+
+ if (IS_PM_SCAN_WP(p->flags) && is_written && is_interesting &&
+ ret >= 0) {
+ make_uffd_wp_pmd(vma, start, pmd);
+ flush_tlb_range(vma, start, end);
+ }
+
+ spin_unlock(ptl);
+
+ arch_leave_lazy_mmu_mode();
+ return ret;
+ }
+
+process_smaller_pages:
+#endif
+
+ orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
+ if (!pte) {
+ walk->action = ACTION_AGAIN;
+ return 0;
+ }
+
+ for (addr = start; addr < end && !ret; pte++, addr += PAGE_SIZE) {
+ ptent = ptep_get(pte);
+ is_written = !is_pte_uffd_wp(ptent);
+
+ bitmap = PM_SCAN_FLAGS(is_written, pagemap_scan_is_file(vma, ptent, addr),
+ pte_present(ptent), is_swap_pte(ptent),
+ pte_present(ptent) && is_zero_pfn(pte_pfn(ptent)));
+
+ if (IS_PM_SCAN_GET(p->flags)) {
+ is_interesting = pagemap_scan_is_interesting_page(bitmap, p);
+ if (is_interesting)
+ ret = pagemap_scan_output(bitmap, p, addr, 1);
+ }
+
+ if (IS_PM_SCAN_WP(p->flags) && is_written && is_interesting &&
+ ret >= 0) {
+ make_uffd_wp_pte(vma, addr, pte);
+ flush = true;
+ }
+ }
+
+ if (flush)
+ flush_tlb_range(vma, start, addr);
+
+ pte_unmap_unlock(orig_pte, ptl);
+ arch_leave_lazy_mmu_mode();
+
+ cond_resched();
+ return ret;
+}
+
+#ifdef CONFIG_HUGETLB_PAGE
+static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
+ unsigned long start, unsigned long end,
+ struct mm_walk *walk)
+{
+ unsigned long n_pages = (end - start)/PAGE_SIZE;
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ bool is_written, is_interesting = true;
+ struct hstate *h = hstate_vma(vma);
+ unsigned long bitmap;
+ spinlock_t *ptl;
+ int ret = 0;
+ pte_t ptent;
+
+ if (IS_PM_SCAN_WP(p->flags) && n_pages < HPAGE_SIZE/PAGE_SIZE)
+ return -EINVAL;
+
+ if (p->max_pages && n_pages > p->max_pages - p->found_pages)
+ n_pages = p->max_pages - p->found_pages;
+
+ if (IS_PM_SCAN_WP(p->flags)) {
+ i_mmap_lock_write(vma->vm_file->f_mapping);
+ ptl = huge_pte_lock(h, vma->vm_mm, ptep);
+ }
+
+ ptent = huge_ptep_get(ptep);
+ is_written = !is_huge_pte_uffd_wp(ptent);
+
+ /*
+ * Partial hugetlb page clear isn't supported
+ */
+ if (is_written && IS_PM_SCAN_WP(p->flags) &&
+ n_pages < HPAGE_SIZE/PAGE_SIZE) {
+ ret = -ENOSPC;
+ goto unlock_and_return;
+ }
+
+ bitmap = PM_SCAN_FLAGS(is_written, pagemap_scan_is_huge_file(ptent),
+ pte_present(ptent), is_swap_pte(ptent),
+ pte_present(ptent) && is_zero_pfn(pte_pfn(ptent)));
+
+ if (IS_PM_SCAN_GET(p->flags)) {
+ is_interesting = pagemap_scan_is_interesting_page(bitmap, p);
+ if (is_interesting)
+ ret = pagemap_scan_output(bitmap, p, start, n_pages);
+ }
+
+ if (IS_PM_SCAN_WP(p->flags) && is_written && is_interesting &&
+ ret >= 0) {
+ make_uffd_wp_huge_pte(vma, start, ptep, ptent);
+ flush_hugetlb_tlb_range(vma, start, end);
+ }
+
+unlock_and_return:
+ if (IS_PM_SCAN_WP(p->flags)) {
+ spin_unlock(ptl);
+ i_mmap_unlock_write(vma->vm_file->f_mapping);
+ }
+
+ return ret;
+}
+#else
+#define pagemap_scan_hugetlb_entry NULL
+#endif
+
+static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
+ int depth, struct mm_walk *walk)
+{
+ unsigned long n_pages = (end - addr)/PAGE_SIZE;
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ int ret = 0;
+
+ if (!vma || !IS_PM_SCAN_GET(p->flags))
+ return 0;
+
+ if (p->max_pages && n_pages > p->max_pages - p->found_pages)
+ n_pages = p->max_pages - p->found_pages;
+
+ ret = pagemap_scan_output(PM_SCAN_FLAGS(false, false, false, false, false),
+ p, addr, n_pages);
+
+ return ret;
+}
+
+static const struct mm_walk_ops pagemap_scan_ops = {
+ .test_walk = pagemap_scan_test_walk,
+ .pmd_entry = pagemap_scan_pmd_entry,
+ .pte_hole = pagemap_scan_pte_hole,
+ .hugetlb_entry = pagemap_scan_hugetlb_entry,
+};
+
+static int pagemap_scan_args_valid(struct pm_scan_arg *arg, unsigned long start,
+ struct page_region __user *vec)
+{
+ /* Detect illegal size, flags, len and masks */
+ if (arg->size != sizeof(struct pm_scan_arg))
+ return -EINVAL;
+ if (!arg->flags)
+ return -EINVAL;
+ if (arg->flags & ~PM_SCAN_OPS)
+ return -EINVAL;
+ if (!arg->len)
+ return -EINVAL;
+ if ((arg->required_mask | arg->anyof_mask | arg->excluded_mask |
+ arg->return_mask) & ~PM_SCAN_BITS_ALL)
+ return -EINVAL;
+ if (!arg->required_mask && !arg->anyof_mask &&
+ !arg->excluded_mask)
+ return -EINVAL;
+ if (!arg->return_mask)
+ return -EINVAL;
+
+ /* Validate memory range */
+ if (!IS_ALIGNED(start, PAGE_SIZE))
+ return -EINVAL;
+ if (!access_ok((void __user *)start, arg->len))
+ return -EFAULT;
+
+ if (IS_PM_SCAN_GET(arg->flags)) {
+ if (arg->vec_len == 0)
+ return -EINVAL;
+ if (!vec)
+ return -EINVAL;
+ if (!access_ok((void __user *)vec,
+ arg->vec_len * sizeof(struct page_region)))
+ return -EFAULT;
+ }
+
+ if (IS_PM_SCAN_WP(arg->flags) && !IS_PM_SCAN_GET(arg->flags) &&
+ arg->max_pages)
+ return -EINVAL;
+
+ return 0;
+}
+
+static long do_pagemap_scan(struct mm_struct *mm, unsigned long __arg)
+{
+ struct pm_scan_arg __user *uarg = (struct pm_scan_arg __user *)__arg;
+ unsigned long long start, end, walk_start, walk_end;
+ unsigned long long empty_slots, vec_index = 0;
+ struct mmu_notifier_range range;
+ struct page_region __user *vec;
+ struct pagemap_scan_private p;
+ struct pm_scan_arg arg;
+ int ret = 0;
+
+ if (copy_from_user(&arg, uarg, sizeof(arg)))
+ return -EFAULT;
+
+ start = untagged_addr((unsigned long)arg.start);
+ vec = (struct page_region *)untagged_addr((unsigned long)arg.vec);
+
+ ret = pagemap_scan_args_valid(&arg, start, vec);
+ if (ret)
+ return ret;
+
+ end = start + arg.len;
+ p.max_pages = arg.max_pages;
+ p.found_pages = 0;
+ p.required_mask = arg.required_mask;
+ p.anyof_mask = arg.anyof_mask;
+ p.excluded_mask = arg.excluded_mask;
+ p.return_mask = arg.return_mask;
+ p.flags = arg.flags;
+ p.flags |= ((p.required_mask | p.anyof_mask | p.excluded_mask) &
+ PAGE_IS_WRITTEN) ? PM_SCAN_REQUIRE_UFFD : 0;
+ p.cur_buf.start = p.cur_buf.len = p.cur_buf.flags = 0;
+ p.vec_buf = NULL;
+ p.vec_buf_len = PAGEMAP_WALK_SIZE >> PAGE_SHIFT;
+
+ /*
+ * Allocate smaller buffer to get output from inside the page walk
+ * functions and walk page range in PAGEMAP_WALK_SIZE size chunks. As
+ * we want to return output to user in compact form where no two
+ * consecutive regions should be continuous and have the same flags.
+ * So store the latest element in p.cur_buf between different walks and
+ * store the p.cur_buf at the end of the walk to the user buffer.
+ */
+ if (IS_PM_SCAN_GET(p.flags)) {
+ p.vec_buf = kmalloc_array(p.vec_buf_len, sizeof(*p.vec_buf),
+ GFP_KERNEL);
+ if (!p.vec_buf)
+ return -ENOMEM;
+ }
+
+ if (IS_PM_SCAN_WP(p.flags)) {
+ mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
+ mm, start, end);
+ mmu_notifier_invalidate_range_start(&range);
+ }
+
+ walk_start = walk_end = start;
+ while (walk_end < end && !ret) {
+ if (IS_PM_SCAN_GET(p.flags)) {
+ p.vec_buf_index = 0;
+
+ /*
+ * All data is copied to cur_buf first. When more data
+ * is found, we push cur_buf to vec_buf and copy new
+ * data to cur_buf. Subtract 1 from length as the
+ * index of cur_buf isn't counted in length.
+ */
+ empty_slots = arg.vec_len - vec_index;
+ p.vec_buf_len = min(p.vec_buf_len, empty_slots - 1);
+ }
+
+ walk_end = (walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK;
+ if (walk_end > end)
+ walk_end = end;
+
+ ret = mmap_read_lock_killable(mm);
+ if (ret)
+ goto free_data;
+ ret = walk_page_range(mm, walk_start, walk_end,
+ &pagemap_scan_ops, &p);
+ mmap_read_unlock(mm);
+
+ if (ret && ret != PM_SCAN_BUFFER_FULL &&
+ ret != PM_SCAN_FOUND_MAX_PAGES)
+ goto free_data;
+
+ walk_start = walk_end;
+ if (IS_PM_SCAN_GET(p.flags) && p.vec_buf_index) {
+ if (copy_to_user(&vec[vec_index], p.vec_buf,
+ p.vec_buf_index * sizeof(*p.vec_buf))) {
+ /*
+ * Return error even though the OP succeeded
+ */
+ ret = -EFAULT;
+ goto free_data;
+ }
+ vec_index += p.vec_buf_index;
+ }
+ }
+
+ if (p.cur_buf.len) {
+ if (copy_to_user(&vec[vec_index], &p.cur_buf, sizeof(p.cur_buf))) {
+ ret = -EFAULT;
+ goto free_data;
+ }
+ vec_index++;
+ }
+
+ ret = vec_index;
+
+free_data:
+ if (IS_PM_SCAN_WP(p.flags))
+ mmu_notifier_invalidate_range_end(&range);
+
+ kfree(p.vec_buf);
+ return ret;
+}
+
+static long do_pagemap_cmd(struct file *file, unsigned int cmd,
+ unsigned long arg)
+{
+ struct mm_struct *mm = file->private_data;
+
+ switch (cmd) {
+ case PAGEMAP_SCAN:
+ return do_pagemap_scan(mm, arg);
+
+ default:
+ return -EINVAL;
+ }
+}
+
const struct file_operations proc_pagemap_operations = {
.llseek = mem_lseek, /* borrow this */
.read = pagemap_read,
.open = pagemap_open,
.release = pagemap_release,
+ .unlocked_ioctl = do_pagemap_cmd,
+ .compat_ioctl = do_pagemap_cmd,
};
#endif /* CONFIG_PROC_PAGE_MONITOR */
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index beb7c63d2871..a6e773c3e2b4 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -261,6 +261,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
unsigned long cp_flags);
bool is_hugetlb_entry_migration(pte_t pte);
+bool is_hugetlb_entry_hwpoisoned(pte_t pte);
void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
#else /* !CONFIG_HUGETLB_PAGE */
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index b7b56871029c..00dba03d4b8f 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -305,4 +305,58 @@ typedef int __bitwise __kernel_rwf_t;
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND)
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg)
+
+/* Bits are set in flags of the page_region and masks in pm_scan_args */
+#define PAGE_IS_WRITTEN (1 << 0)
+#define PAGE_IS_FILE (1 << 1)
+#define PAGE_IS_PRESENT (1 << 2)
+#define PAGE_IS_SWAPPED (1 << 3)
+#define PAGE_IS_PFNZERO (1 << 4)
+
+/*
+ * struct page_region - Page region with flags
+ * @start: Start of the region
+ * @len: Length of the region in pages
+ * @bitmap: Bits sets for the region
+ */
+struct page_region {
+ __u64 start;
+ __u64 len;
+ __u64 flags;
+};
+
+/*
+ * struct pm_scan_arg - Pagemap ioctl argument
+ * @size: Size of the structure
+ * @flags: Flags for the IOCTL
+ * @start: Starting address of the region
+ * @len: Length of the region (All the pages in this length are included)
+ * @vec: Address of page_region struct array for output
+ * @vec_len: Length of the page_region struct array
+ * @max_pages: Optional max return pages
+ * @required_mask: Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask: Any mask - Any of these bits are set in the PTE
+ * @excluded_mask: Exclude mask - None of these bits are set in the PTE
+ * @return_mask: Bits that are to be reported in page_region
+ */
+struct pm_scan_arg {
+ __u64 size;
+ __u64 flags;
+ __u64 start;
+ __u64 len;
+ __u64 vec;
+ __u64 vec_len;
+ __u64 max_pages;
+ __u64 required_mask;
+ __u64 anyof_mask;
+ __u64 excluded_mask;
+ __u64 return_mask;
+};
+
+/* Supported flags */
+#define PM_SCAN_OP_GET (1 << 0)
+#define PM_SCAN_OP_WP (1 << 1)
+
#endif /* _UAPI_LINUX_FS_H */
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 0db13167b1ee..7e60f0f3fd03 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4980,7 +4980,7 @@ bool is_hugetlb_entry_migration(pte_t pte)
return false;
}
-static bool is_hugetlb_entry_hwpoisoned(pte_t pte)
+bool is_hugetlb_entry_hwpoisoned(pte_t pte)
{
swp_entry_t swp;
--
2.39.2
New IOCTL and macros has been added in the kernel sources. Update the
tools header file as well.
Signed-off-by: Muhammad Usama Anjum <[email protected]>
---
Changes in v20:
- Update tools/include/uapi/linux/fs.h
Changes in v19:
- Update fs.h accourding to precious patch
---
tools/include/uapi/linux/fs.h | 54 +++++++++++++++++++++++++++++++++++
1 file changed, 54 insertions(+)
diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h
index b7b56871029c..00dba03d4b8f 100644
--- a/tools/include/uapi/linux/fs.h
+++ b/tools/include/uapi/linux/fs.h
@@ -305,4 +305,58 @@ typedef int __bitwise __kernel_rwf_t;
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND)
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg)
+
+/* Bits are set in flags of the page_region and masks in pm_scan_args */
+#define PAGE_IS_WRITTEN (1 << 0)
+#define PAGE_IS_FILE (1 << 1)
+#define PAGE_IS_PRESENT (1 << 2)
+#define PAGE_IS_SWAPPED (1 << 3)
+#define PAGE_IS_PFNZERO (1 << 4)
+
+/*
+ * struct page_region - Page region with flags
+ * @start: Start of the region
+ * @len: Length of the region in pages
+ * @bitmap: Bits sets for the region
+ */
+struct page_region {
+ __u64 start;
+ __u64 len;
+ __u64 flags;
+};
+
+/*
+ * struct pm_scan_arg - Pagemap ioctl argument
+ * @size: Size of the structure
+ * @flags: Flags for the IOCTL
+ * @start: Starting address of the region
+ * @len: Length of the region (All the pages in this length are included)
+ * @vec: Address of page_region struct array for output
+ * @vec_len: Length of the page_region struct array
+ * @max_pages: Optional max return pages
+ * @required_mask: Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask: Any mask - Any of these bits are set in the PTE
+ * @excluded_mask: Exclude mask - None of these bits are set in the PTE
+ * @return_mask: Bits that are to be reported in page_region
+ */
+struct pm_scan_arg {
+ __u64 size;
+ __u64 flags;
+ __u64 start;
+ __u64 len;
+ __u64 vec;
+ __u64 vec_len;
+ __u64 max_pages;
+ __u64 required_mask;
+ __u64 anyof_mask;
+ __u64 excluded_mask;
+ __u64 return_mask;
+};
+
+/* Supported flags */
+#define PM_SCAN_OP_GET (1 << 0)
+#define PM_SCAN_OP_WP (1 << 1)
+
#endif /* _UAPI_LINUX_FS_H */
--
2.39.2
Add some explanation and method to use write-protection and written-to
on memory range.
Signed-off-by: Muhammad Usama Anjum <[email protected]>
---
Changes in v20:
- Add new flag in documentation
Changes in v19:
- Improved documentatiom with punctuation marks
Changes in v16:
- Update the documentation
Changes in v11:
- Add more documentation
---
Documentation/admin-guide/mm/pagemap.rst | 58 ++++++++++++++++++++++++
1 file changed, 58 insertions(+)
diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst
index c8f380271cad..84c14ce86ad4 100644
--- a/Documentation/admin-guide/mm/pagemap.rst
+++ b/Documentation/admin-guide/mm/pagemap.rst
@@ -227,3 +227,61 @@ Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is
always 12 at most architectures). Since Linux 3.11 their meaning changes
after first clear of soft-dirty bits. Since Linux 4.2 they are used for
flags unconditionally.
+
+Pagemap Scan IOCTL
+==================
+
+The ``PAGEMAP_SCAN`` IOCTL on the pagemap file can be used to get or optionally
+clear the info about page table entries. The following operations are supported
+in this IOCTL:
+- Get the information if the pages have been written to (``PAGE_IS_WRITTEN``),
+ file mapped (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``), swapped
+ (``PAGE_IS_SWAPPED``) or page has pfn zero (``PAGE_IS_PFNZERO``).
+- Find pages which have been written to and/or write protect the pages atomically
+ (atomic ``PM_SCAN_OP_GET + PM_SCAN_OP_WP``)
+
+The ``struct pm_scan_arg`` is used as the argument of the IOCTL.
+ 1. The size of the ``struct pm_scan_arg`` must be specified in the ``size``
+ field. This field will be helpful in recognizing the structure if extensions
+ are done later.
+ 2. The flags can be specified in the ``flags`` field. The ``PM_SCAN_OP_GET``
+ and ``PM_SCAN_OP_WP`` are the only added flags at this time.
+ 3. The range is specified through ``start`` and ``len``.
+ 4. The output buffer of ``struct page_region`` array and size is specified in
+ ``vec`` and ``vec_len``.
+ 5. The optional maximum requested pages are specified in the ``max_pages``.
+ 6. The masks are specified in ``required_mask``, ``anyof_mask``,
+ ``excluded_ mask`` and ``return_mask``.
+ 1. To find if ``PAGE_IS_WRITTEN`` flag is set for pages which have
+ ``PAGE_IS_FILE`` set and ``PAGE_IS_SWAPPED`` unset, ``required_mask``
+ is set to ``PAGE_IS_FILE``, ``exclude_mask`` is set to
+ ``PAGE_IS_SWAPPED`` and ``return_mask`` is set to ``PAGE_IS_WRITTEN``.
+ The output buffer in ``vec`` and length must be specified in ``vec_len``.
+ 2. To find pages which have either ``PAGE_IS_FILE`` or ``PAGE_IS_SWAPPED``
+ set, ``anyof_masks`` is set to ``PAGE_IS_FILE | PAGE_IS_SWAPPED``.
+ 3. To find written pages and engage write protect, ``PAGE_IS_WRITTEN`` is
+ specified in ``required_mask`` and ``return_mask``. In addition to
+ specifying the output buffer in ``vec`` and length in ``vec_len``, the
+ ``PM_SCAN_OP_WP`` is specified in ``flags`` to perform write protect
+ on the range as well.
+
+The ``PAGE_IS_WRITTEN`` flag can be considered as the better and correct
+alternative of soft-dirty flag. It doesn't get affected by housekeeping chores
+(VMA merging) of the kernel and hence the user can find the true soft-dirty pages
+only. This IOCTL adds the atomic way to find which pages have been written and
+write protect those pages again. This kind of operation is needed to efficiently
+find out which pages have changed in the memory.
+
+To get information about which pages have been written to or optionally write
+protect the pages, following must be performed first in order:
+ 1. The userfaultfd file descriptor is created with ``userfaultfd`` syscall.
+ 2. The ``UFFD_FEATURE_WP_UNPOPULATED`` and ``UFFD_FEATURE_WP_ASYNC`` features
+ are set by ``UFFDIO_API`` IOCTL.
+ 3. The memory range is registered with ``UFFDIO_REGISTER_MODE_WP`` mode
+ through ``UFFDIO_REGISTER`` IOCTL.
+ 4. Then any part of the registered memory or the whole memory region must
+ be write protected using ``PAGEMAP_SCAN`` IOCTL with flag ``PM_SCAN_OP_WP``
+ or the ``UFFDIO_WRITEPROTECT`` IOCTL can be used. Both of these perform the
+ same operation. The former is better in terms of performance.
+ 5. Now the ``PAGEMAP_SCAN`` IOCTL can be used to either just find pages which
+ have been written to and/or optionally write protect the pages as well.
--
2.39.2
On Wed, 21 Jun 2023 at 09:24, Muhammad Usama Anjum
<[email protected]> wrote:
[...]
> +static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
> + struct mm_walk *walk)
> +{
> + struct pagemap_scan_private *p = walk->private;
> + struct vm_area_struct *vma = walk->vma;
> +
> + if ((p->flags & PM_SCAN_REQUIRE_UFFD) && (!userfaultfd_wp_async(vma) ||
> + !userfaultfd_wp_use_markers(vma)))
> + return -EPERM;
> +
> + if (vma->vm_flags & VM_PFNMAP)
> + return 1;
> +
> + return 0;
> +}
This could actually short-circuit all vma flags (e.g. IS_FILE): if
(required_mask & IS_FILE && vma is not file-backed) return 0;
Best Regards
Michał Mirosław
On 6/21/23 6:42 PM, Michał Mirosław wrote:
> On Wed, 21 Jun 2023 at 09:24, Muhammad Usama Anjum
> <[email protected]> wrote:
> [...]
>> +static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
>> + struct mm_walk *walk)
>> +{
>> + struct pagemap_scan_private *p = walk->private;
>> + struct vm_area_struct *vma = walk->vma;
>> +
>> + if ((p->flags & PM_SCAN_REQUIRE_UFFD) && (!userfaultfd_wp_async(vma) ||
>> + !userfaultfd_wp_use_markers(vma)))
>> + return -EPERM;
>> +
>> + if (vma->vm_flags & VM_PFNMAP)
>> + return 1;
>> +
>> + return 0;
>> +}
>
> This could actually short-circuit all vma flags (e.g. IS_FILE): if
> (required_mask & IS_FILE && vma is not file-backed) return 0;
Sorry, unable to understand you. Should we do something here?
>
> Best Regards
> Michał Mirosław
--
BR,
Muhammad Usama Anjum
On Wed, 21 Jun 2023 at 16:16, Muhammad Usama Anjum
<[email protected]> wrote:
> On 6/21/23 6:42 PM, Michał Mirosław wrote:
> > On Wed, 21 Jun 2023 at 09:24, Muhammad Usama Anjum
> > <[email protected]> wrote:
> > [...]
> >> +static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
> >> + struct mm_walk *walk)
> >> +{
> >> + struct pagemap_scan_private *p = walk->private;
> >> + struct vm_area_struct *vma = walk->vma;
> >> +
> >> + if ((p->flags & PM_SCAN_REQUIRE_UFFD) && (!userfaultfd_wp_async(vma) ||
> >> + !userfaultfd_wp_use_markers(vma)))
> >> + return -EPERM;
> >> +
> >> + if (vma->vm_flags & VM_PFNMAP)
> >> + return 1;
> >> +
> >> + return 0;
> >> +}
> >
> > This could actually short-circuit all vma flags (e.g. IS_FILE): if
> > (required_mask & IS_FILE && vma is not file-backed) return 0;
> Sorry, unable to understand you. Should we do something here?
It seems I had an earlier version of the patch in my mind, where
PAGE_IS_FILE was based on vma type. Sorry for the noise.
Best Regards
Michał Mirosław
>
> >
> > Best Regards
> > Michał Mirosław
>
> --
> BR,
> Muhammad Usama Anjum
Hi Muhammad,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on next-20230623]
[cannot apply to linus/master v6.4-rc7]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Muhammad-Usama-Anjum/userfaultfd-UFFD_FEATURE_WP_ASYNC/20230621-171253
base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link: https://lore.kernel.org/r/20230621072404.2918101-3-usama.anjum%40collabora.com
patch subject: [PATCH v20 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs
config: s390-randconfig-s043-20230622 (https://download.01.org/0day-ci/archive/20230624/[email protected]/config)
compiler: s390-linux-gcc (GCC) 12.3.0
reproduce: (https://download.01.org/0day-ci/archive/20230624/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
>> fs/proc/task_mmu.c:2209:13: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct page_region [noderef] __user *vec @@ got struct page_region * @@
fs/proc/task_mmu.c:2209:13: sparse: expected struct page_region [noderef] __user *vec
fs/proc/task_mmu.c:2209:13: sparse: got struct page_region *
vim +2209 fs/proc/task_mmu.c
2193
2194 static long do_pagemap_scan(struct mm_struct *mm, unsigned long __arg)
2195 {
2196 struct pm_scan_arg __user *uarg = (struct pm_scan_arg __user *)__arg;
2197 unsigned long long start, end, walk_start, walk_end;
2198 unsigned long long empty_slots, vec_index = 0;
2199 struct mmu_notifier_range range;
2200 struct page_region __user *vec;
2201 struct pagemap_scan_private p;
2202 struct pm_scan_arg arg;
2203 int ret = 0;
2204
2205 if (copy_from_user(&arg, uarg, sizeof(arg)))
2206 return -EFAULT;
2207
2208 start = untagged_addr((unsigned long)arg.start);
> 2209 vec = (struct page_region *)untagged_addr((unsigned long)arg.vec);
2210
2211 ret = pagemap_scan_args_valid(&arg, start, vec);
2212 if (ret)
2213 return ret;
2214
2215 end = start + arg.len;
2216 p.max_pages = arg.max_pages;
2217 p.found_pages = 0;
2218 p.required_mask = arg.required_mask;
2219 p.anyof_mask = arg.anyof_mask;
2220 p.excluded_mask = arg.excluded_mask;
2221 p.return_mask = arg.return_mask;
2222 p.flags = arg.flags;
2223 p.flags |= ((p.required_mask | p.anyof_mask | p.excluded_mask) &
2224 PAGE_IS_WRITTEN) ? PM_SCAN_REQUIRE_UFFD : 0;
2225 p.cur_buf.start = p.cur_buf.len = p.cur_buf.flags = 0;
2226 p.vec_buf = NULL;
2227 p.vec_buf_len = PAGEMAP_WALK_SIZE >> PAGE_SHIFT;
2228
2229 /*
2230 * Allocate smaller buffer to get output from inside the page walk
2231 * functions and walk page range in PAGEMAP_WALK_SIZE size chunks. As
2232 * we want to return output to user in compact form where no two
2233 * consecutive regions should be continuous and have the same flags.
2234 * So store the latest element in p.cur_buf between different walks and
2235 * store the p.cur_buf at the end of the walk to the user buffer.
2236 */
2237 if (IS_PM_SCAN_GET(p.flags)) {
2238 p.vec_buf = kmalloc_array(p.vec_buf_len, sizeof(*p.vec_buf),
2239 GFP_KERNEL);
2240 if (!p.vec_buf)
2241 return -ENOMEM;
2242 }
2243
2244 if (IS_PM_SCAN_WP(p.flags)) {
2245 mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
2246 mm, start, end);
2247 mmu_notifier_invalidate_range_start(&range);
2248 }
2249
2250 walk_start = walk_end = start;
2251 while (walk_end < end && !ret) {
2252 if (IS_PM_SCAN_GET(p.flags)) {
2253 p.vec_buf_index = 0;
2254
2255 /*
2256 * All data is copied to cur_buf first. When more data
2257 * is found, we push cur_buf to vec_buf and copy new
2258 * data to cur_buf. Subtract 1 from length as the
2259 * index of cur_buf isn't counted in length.
2260 */
2261 empty_slots = arg.vec_len - vec_index;
2262 p.vec_buf_len = min(p.vec_buf_len, empty_slots - 1);
2263 }
2264
2265 walk_end = (walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK;
2266 if (walk_end > end)
2267 walk_end = end;
2268
2269 ret = mmap_read_lock_killable(mm);
2270 if (ret)
2271 goto free_data;
2272 ret = walk_page_range(mm, walk_start, walk_end,
2273 &pagemap_scan_ops, &p);
2274 mmap_read_unlock(mm);
2275
2276 if (ret && ret != PM_SCAN_BUFFER_FULL &&
2277 ret != PM_SCAN_FOUND_MAX_PAGES)
2278 goto free_data;
2279
2280 walk_start = walk_end;
2281 if (IS_PM_SCAN_GET(p.flags) && p.vec_buf_index) {
2282 if (copy_to_user(&vec[vec_index], p.vec_buf,
2283 p.vec_buf_index * sizeof(*p.vec_buf))) {
2284 /*
2285 * Return error even though the OP succeeded
2286 */
2287 ret = -EFAULT;
2288 goto free_data;
2289 }
2290 vec_index += p.vec_buf_index;
2291 }
2292 }
2293
2294 if (p.cur_buf.len) {
2295 if (copy_to_user(&vec[vec_index], &p.cur_buf, sizeof(p.cur_buf))) {
2296 ret = -EFAULT;
2297 goto free_data;
2298 }
2299 vec_index++;
2300 }
2301
2302 ret = vec_index;
2303
2304 free_data:
2305 if (IS_PM_SCAN_WP(p.flags))
2306 mmu_notifier_invalidate_range_end(&range);
2307
2308 kfree(p.vec_buf);
2309 return ret;
2310 }
2311
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki