Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 8 Dec 2020 10:34:34 +0200
From:   Mike Rapoport <rppt@linux.ibm.com>
To:     Nadav Amit <nadav.amit@gmail.com>
Cc:     Mike Rapoport <rppt@linux.vnet.ibm.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        linux-mm <linux-mm@kvack.org>,
        lkml <linux-kernel@vger.kernel.org>,
        Andrea Arcangeli <aarcange@redhat.com>,
        Mike Kravetz <mike.kravetz@oracle.com>,
        Pavel Emelyanov <xemul@virtuozzo.com>,
        Andrei Vagin <avagin@virtuozzo.com>
Subject: Re: [PATCH] userfaultfd: prevent non-cooperative events vs
 mcopy_atomic races
Message-ID: <20201208083434.GA1164013@linux.ibm.com>
References: <1527061324-19949-1-git-send-email-rppt@linux.vnet.ibm.com>
 <31DA12CC-E9CC-497D-A2EE-B83549D95CE8@gmail.com>
 <20201206093703.GY123287@linux.ibm.com>
 <5921BA80-F263-4F8D-B7E6-316CEB602B51@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <5921BA80-F263-4F8D-B7E6-316CEB602B51@gmail.com>
Precedence: bulk

On Sun, Dec 06, 2020 at 08:31:39PM -0800, Nadav Amit wrote:
> 
> Whenever I run into a non-standard and non-trivial synchronization algorithm
> in the kernel (and elsewhere), I become very confused and concerned. I
> raised my question since I wanted to modify the code and could not figure
> out how to properly do so. Based on your input that the monitor is expected
> to know the child mappings according to userfaultfd events, I now think that
> the kernel does not provide this ability and the locking scheme is broken.
> 
> Here are some scenarios that I think are broken - please correct me if I am
> wrong:
> 
> * Scenario 1: MADV_DONTNEED racing with userfaultfd page-faults
> 
> userfaultfd_remove() only holds the mmap_lock for read, so these events
> cannot be ordered with userfaultfd page-faults.
> 
> * Scenario 2: MADV_DONTNEED racing with fork()
> 
> As userfaultfd_remove() releases mmap_lock after the user notification and
> before the actual unmapping, concurrent fork() might happen before or after
> the actual unmapping in MADV_DONTNEED and the user therefore has no way of
> knowing whether the actual unmapping took place before or after the fork().
> 
> * Scenario 3: Concurrent MADV_DONTNEED can cause userfaultfd_remove() to
> clear mmap_changing cleared before all the notifications are completed.
> 
> As mmap_lock is only taken for read, the first thread the completed
> userfaultfd_remove() would clear the indication that was set by the other
> one.
> 
> * Scenario 4: Fork starts and ends between copying of two pages.
> 
> As mmap_lock might be released during ioctl_copy() (inside
> __mcopy_atomic()), some pages might be mapped in the child and others not:
> 
> 
> CPU0				CPU1
> ----				----
> ioctl_copy():
>  __mcopy_atomic()
>   mmap_read_lock()
>   !mmap_changing [ok]
>   mfill_atomic_pte() == 0 [page0 copied]
>   mfill_atomic_pte() == -ENOENT [page1 will be retried]
>   mmap_read_unlock()
>   goto retry
> 
> 				fork():
> 				 dup_userfaultfd()
> 				 -> mmap_changing=true
> 				 userfaultfd_event_wait_completion()
> 				 -> mmap_changing=false
> 
>   mmap_read_lock()
>   !mmap_changing [ok]
>   mfill_atomic_pte() == 0 [page1 copied]
>   mmap_read_unlock()
>  
>  return: 2 pages were mapped, while the first is present in the child and
>  the second one is non-present.
> 
> Bottom-line: it seems to me that mmap_changing should be a counter (not
> boolean) that is protected by mmap_lock. This counter should be kept
> elevated throughout the entire operation (in regard to MADV_DONTNEED).
> Perhaps mmap_lock does not have to be taken to decrease the counter, but
> then an smp_wmb() would be needed before the counter is decreased.
> 
> Let me know whether I am completely off or missing something.

I tried to remember what's going on there and wrap my head around your
examples. I'm not sure if userspace cannot workaround some of those, but
I can't say I can propose it right now.

There is for sure userspace is helpless in Scenario 4, but I think it is
very unlikely that fork() will be fast enough to grab and release
mmap_lock while uffd_copy() waits for CPU to retry.

I agree that a making mmap_changing a counter would be more robust
anyway.

> Thanks,
> Nadav
> 

-- 
Sincerely yours,
Mike.