On Thu, Feb 4, 2021 at 10:34 AM Axel Rasmussen <[email protected]> wrote:
>
> Changelog
> =========
>
> v3->v4:
> - Reordered if() branches in hugetlb_mcopy_atomic_pte, so the conditions are
> simpler and easier to read.
> - Reverted most of the mfill_atomic_pte change (the anon / shmem path). Just
> return -EINVAL for CONTINUE, and set zeropage = (mode ==
> MCOPY_ATOMIC_ZEROPAGE), so we can keep the delta small.
> - Split out adding #ifdef CONFIG_USERFAULTFD to a separate patch (instead of
> lumping it together with adding UFFDIO_CONTINUE). Also, extended it to make
> the same change for shmem as well as suggested by Hugh Dickins.
> - Fixed signature of hugetlb_mcopy_atomic_pte for !CONFIG_HUGETLB_PAGE
> (signature must be the same in either case).
> - Rebased onto a newer version of Peter's patches to disable huge PMD sharing.
Apologies for the email noise, of course immediately after doing git
send-email I realized I forgot to document one item here:
- Relaxed restriction for minor registration to allow any hugetlb
VMAs, not just those with VM_SHARED. Fixed setting VM_WRITE flag in a
CONTINUE ioctl for non-VM_SHARED VMAs.
>
> v2->v3:
> - Added #ifdef CONFIG_USERFAULTFD around hugetlb helper functions, to fix build
> errors when building without CONFIG_USERFAULTFD set.
>
> v1->v2:
> - Fixed a bug in the hugetlb_mcopy_atomic_pte retry case. We now plumb in the
> enum mcopy_atomic_mode, so we can differentiate between the three cases this
> function needs to handle:
> 1) We're doing a COPY op, and need to allocate a page, add to cache, etc.
> 2) We're doing a COPY op, but allocation in this function failed previously;
> we're in the retry path. The page was allocated, but not e.g. added to page
> cache, so that still needs to be done.
> 3) We're doing a CONTINUE op, we need to look up an existing page instead of
> allocating a new one.
> - Rebased onto a newer version of Peter's patches to disable huge PMD sharing,
> which fixes syzbot complaints on some non-x86 architectures.
> - Moved __VM_UFFD_FLAGS into userfaultfd_k.h, so inline helpers can use it.
> - Renamed UFFD_FEATURE_MINOR_FAULT_HUGETLBFS to UFFD_FEATURE_MINOR_HUGETLBFS,
> for consistency with other existing feature flags.
> - Moved the userfaultfd_minor hook in hugetlb.c into the else block, so we don't
> have to explicitly check for !new_page.
>
> RFC->v1:
> - Rebased onto Peter Xu's patches for disabling huge PMD sharing for certain
> userfaultfd-registered areas.
> - Added commits which update documentation, and add a self test which exercises
> the new feature.
> - Fixed reporting CONTINUE as a supported ioctl even for non-MINOR ranges.
>
> Overview
> ========
>
> This series adds a new userfaultfd registration mode,
> UFFDIO_REGISTER_MODE_MINOR. This allows userspace to intercept "minor" faults.
> By "minor" fault, I mean the following situation:
>
> Let there exist two mappings (i.e., VMAs) to the same page(s) (shared memory).
> One of the mappings is registered with userfaultfd (in minor mode), and the
> other is not. Via the non-UFFD mapping, the underlying pages have already been
> allocated & filled with some contents. The UFFD mapping has not yet been
> faulted in; when it is touched for the first time, this results in what I'm
> calling a "minor" fault. As a concrete example, when working with hugetlbfs, we
> have huge_pte_none(), but find_lock_page() finds an existing page.
>
> We also add a new ioctl to resolve such faults: UFFDIO_CONTINUE. The idea is,
> userspace resolves the fault by either a) doing nothing if the contents are
> already correct, or b) updating the underlying contents using the second,
> non-UFFD mapping (via memcpy/memset or similar, or something fancier like RDMA,
> or etc...). In either case, userspace issues UFFDIO_CONTINUE to tell the kernel
> "I have ensured the page contents are correct, carry on setting up the mapping".
>
> Use Case
> ========
>
> Consider the use case of VM live migration (e.g. under QEMU/KVM):
>
> 1. While a VM is still running, we copy the contents of its memory to a
> target machine. The pages are populated on the target by writing to the
> non-UFFD mapping, using the setup described above. The VM is still running
> (and therefore its memory is likely changing), so this may be repeated
> several times, until we decide the target is "up to date enough".
>
> 2. We pause the VM on the source, and start executing on the target machine.
> During this gap, the VM's user(s) will *see* a pause, so it is desirable to
> minimize this window.
>
> 3. Between the last time any page was copied from the source to the target, and
> when the VM was paused, the contents of that page may have changed - and
> therefore the copy we have on the target machine is out of date. Although we
> can keep track of which pages are out of date, for VMs with large amounts of
> memory, it is "slow" to transfer this information to the target machine. We
> want to resume execution before such a transfer would complete.
>
> 4. So, the guest begins executing on the target machine. The first time it
> touches its memory (via the UFFD-registered mapping), userspace wants to
> intercept this fault. Userspace checks whether or not the page is up to date,
> and if not, copies the updated page from the source machine, via the non-UFFD
> mapping. Finally, whether a copy was performed or not, userspace issues a
> UFFDIO_CONTINUE ioctl to tell the kernel "I have ensured the page contents
> are correct, carry on setting up the mapping".
>
> We don't have to do all of the final updates on-demand. The userfaultfd manager
> can, in the background, also copy over updated pages once it receives the map of
> which pages are up-to-date or not.
>
> Interaction with Existing APIs
> ==============================
>
> Because it's possible to combine registration modes (e.g. a single VMA can be
> userfaultfd-registered MINOR | MISSING), and because it's up to userspace how to
> resolve faults once they are received, I spent some time thinking through how
> the existing API interacts with the new feature.
>
> UFFDIO_CONTINUE cannot be used to resolve non-minor faults, as it does not
> allocate a new page. If UFFDIO_CONTINUE is used on a non-minor fault:
>
> - For non-shared memory or shmem, -EINVAL is returned.
> - For hugetlb, -EFAULT is returned.
>
> UFFDIO_COPY and UFFDIO_ZEROPAGE cannot be used to resolve minor faults. Without
> modifications, the existing codepath assumes a new page needs to be allocated.
> This is okay, since userspace must have a second non-UFFD-registered mapping
> anyway, thus there isn't much reason to want to use these in any case (just
> memcpy or memset or similar).
>
> - If UFFDIO_COPY is used on a minor fault, -EEXIST is returned.
> - If UFFDIO_ZEROPAGE is used on a minor fault, -EEXIST is returned (or -EINVAL
> in the case of hugetlb, as UFFDIO_ZEROPAGE is unsupported in any case).
> - UFFDIO_WRITEPROTECT simply doesn't work with shared memory, and returns
> -ENOENT in that case (regardless of the kind of fault).
>
> Dependencies
> ============
>
> I've included 4 commits from Peter Xu's larger series
> (https://lore.kernel.org/patchwork/cover/1366017/) in this series. My changes
> depend on his work, to disable huge PMD sharing for MINOR registered userfaultfd
> areas. I included the 4 commits directly because a) it lets this series just be
> applied and work as-is, and b) they are fairly standalone, and could potentially
> be merged even without the rest of the larger series Peter submitted. Thanks
> Peter!
>
> Also, although it doesn't affect minor fault handling, I did notice that the
> userfaultfd self test sometimes experienced memory corruption
> (https://lore.kernel.org/patchwork/cover/1356755/). For anyone testing this
> series, it may be useful to apply that series first to fix the selftest
> flakiness. That series doesn't have to be merged into mainline / maintaner
> branches before mine, though.
>
> Future Work
> ===========
>
> Currently the patchset only supports hugetlbfs. There is no reason it can't work
> with shmem, but I expect hugetlbfs to be much more commonly used since we're
> talking about backing guest memory for VMs. I plan to implement shmem support in
> a follow-up patch series.
>
> Axel Rasmussen (6):
> userfaultfd: add minor fault registration mode
> userfaultfd: disable huge PMD sharing for MINOR registered VMAs
> userfaultfd: hugetlbfs: only compile UFFD helpers if config enabled
> userfaultfd: add UFFDIO_CONTINUE ioctl
> userfaultfd: update documentation to describe minor fault handling
> userfaultfd/selftests: add test exercising minor fault handling
>
> Peter Xu (4):
> hugetlb: Pass vma into huge_pte_alloc() and huge_pmd_share()
> hugetlb/userfaultfd: Forbid huge pmd sharing when uffd enabled
> mm/hugetlb: Move flush_hugetlb_tlb_range() into hugetlb.h
> hugetlb/userfaultfd: Unshare all pmds for hugetlbfs when register wp
>
> Documentation/admin-guide/mm/userfaultfd.rst | 107 ++++++----
> arch/arm64/mm/hugetlbpage.c | 7 +-
> arch/ia64/mm/hugetlbpage.c | 3 +-
> arch/mips/mm/hugetlbpage.c | 4 +-
> arch/parisc/mm/hugetlbpage.c | 2 +-
> arch/powerpc/mm/hugetlbpage.c | 3 +-
> arch/s390/mm/hugetlbpage.c | 2 +-
> arch/sh/mm/hugetlbpage.c | 2 +-
> arch/sparc/mm/hugetlbpage.c | 6 +-
> fs/proc/task_mmu.c | 1 +
> fs/userfaultfd.c | 196 ++++++++++++++++---
> include/linux/hugetlb.h | 22 ++-
> include/linux/mm.h | 1 +
> include/linux/mmu_notifier.h | 1 +
> include/linux/userfaultfd_k.h | 49 ++++-
> include/trace/events/mmflags.h | 1 +
> include/uapi/linux/userfaultfd.h | 36 +++-
> mm/hugetlb.c | 113 +++++++----
> mm/userfaultfd.c | 51 +++--
> tools/testing/selftests/vm/userfaultfd.c | 147 +++++++++++++-
> 20 files changed, 601 insertions(+), 153 deletions(-)
>
> --
> 2.30.0.365.g02bc693789-goog
>