Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CADyq12wPW69ovpW4akDY5PGBbrvnwsLO86=sSKTU4CB3dNwG3Q@mail.gmail.com>
 <20200414214516.GA182757@xz-x1> <20200415031602.22348-1-hdanton@sina.com> <20200415142546.GO5100@ziepe.ca>
In-Reply-To: <20200415142546.GO5100@ziepe.ca>
From:   Brian Geffon <bgeffon@google.com>
Date:   Wed, 15 Apr 2020 08:16:59 -0700
Message-ID: <CADyq12w2ZG+VehAuS763Zb1xJB8SfEbOHkopMPhC2hVqhM5yzw@mail.gmail.com>
Subject: Re: Userfaultfd doesn't seem to break out of poll on fd close
To:     Jason Gunthorpe <jgg@ziepe.ca>
Cc:     Hillf Danton <hdanton@sina.com>, Peter Xu <peterx@redhat.com>,
        Andrea Arcangeli <aarcange@redhat.com>,
        linux-mm <linux-mm@kvack.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Sonny Rao <sonnyrao@google.com>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Thanks everyone for the insights!


On Wed, Apr 15, 2020 at 7:25 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Wed, Apr 15, 2020 at 11:16:02AM +0800, Hillf Danton wrote:
> >
> > On Tue, 14 Apr 2020 19:34:10 -0300 Jason Gunthorpe wrote:
> > >
> > > On Tue, Apr 14, 2020 at 05:45:16PM -0400, Peter Xu wrote:
> > > > On Sun, Apr 12, 2020 at 01:10:40PM -0700, Brian Geffon wrote:
> > > > > Hi,
> > > > > It seems that userfaultfd isn't woken from a poll when the file
> > > > > descriptor is closed. It seems that it should be from the code in
> > > > > userfault_ctx_release, but it appears that's not actually called
> > > > > immediately. I have a simple standalone example that shows this
> > > > > behavior. It's straight forward: one thread creates a userfaultfd and
> > > > > then closes it after a second thread has entered a poll syscall, some
> > > > > abbreviated strace output is below showing this and the code can be
> > > > > seen here: https://gist.github.com/bgaff/9a8fbbe8af79c0e18502430d416df77e
> > > > >
> > > > > Given that it's probably very common to have a dedicated thread remain
> > > > > blocked indefinitely in a poll(2) waiting for faults there must be a
> > > > > way to break it out early when it's closed. Am I missing something?
> > > >
> > > > Hi, Brian,
> > > >
> > > > I might be wrong below, just to share my understanding...
> > > >
> > > > IMHO a well-behaved userspace should not close() on a file descriptor
> > > > if it's still in use within another thread.  In this case, the poll()
> > > > thread is still using the userfaultfd handle
> > >
> > > I also don't think concurrant close() on a file descriptor that is
> > > under poll() is well defined, or should be relied upon.
> > >
> > > > IIUC userfaultfd_release() is only called when the file descriptor
> > > > destructs itself.  But shouldn't the poll() take a refcount of that
> > > > file descriptor too before waiting?  Not sure userfaultfd_release() is
> > > > the place to kick then, because if so, close() will only decrease the
> > > > fd refcount from 2->1, and I'm not sure userfaultfd_release() will be
> > > > triggered.
> > >
> > > This is most probably true.
> > >
> > > eventfd, epoll and pthread_join is the robust answer to these
> > > problems.
> > >
> >
> > See the difference EPOLLHUP makes.
>
> The whole idea is completely racey:
>
>           CPU1                            CPU2                  CPU3
>  fds[i]->fd = userfaultfd;
>  while()
>                                        close(userfaultfd)
>                                        pthread_join()
>                                                             someother_fd = open()
>                                                             userfaultfd == someother_fd
>      poll(fds)   // <- Still sleeps
>
> The kernel should not be trying to wake poll from fd release, and
> userspace should not close a FD that is currently under poll.
>
> Besides, it really does look like poll holds the fget while doing its
> work (see poll_freewait), so fops release() won't be called anyhow..
>
> Jason