Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CAOp6jLZ8GR8DKCZLQJKPS7oeP1CX4L3ijBFDBePGX4q6z-DQhg@mail.gmail.com>
In-Reply-To: <CAOp6jLZ8GR8DKCZLQJKPS7oeP1CX4L3ijBFDBePGX4q6z-DQhg@mail.gmail.com>
From:   Axel Rasmussen <axelrasmussen@google.com>
Date:   Tue, 11 May 2021 11:11:58 -0700
Message-ID: <CAJHvVci_q5rxuo-N+EH_CNxX_M3oxWcLyitvbdySa_PMH3e_Dg@mail.gmail.com>
Subject: Re: Userspace notifications for observing userfaultfd faults
To:     "Robert O'Callahan" <roc@pernos.co>
Cc:     Andrea Arcangeli <aarcange@redhat.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Hugh Dickins <hughd@google.com>, Peter Xu <peterx@redhat.com>,
        Ingo Molnar <mingo@redhat.com>, Linux-MM <linux-mm@kvack.org>,
        LKML <linux-kernel@vger.kernel.org>, Kyle Huey <khuey@pernos.co>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Mon, May 10, 2021 at 5:38 PM Robert O'Callahan <roc@pernos.co> wrote:
>
> For rr (https://rr-project.org) to support recording and replaying
> applications that use userfaultfd, we need to observe that a task we
> are controlling has blocked on a userfault. Currently this is very
> difficult to do, especially if a task blocks on a userfault on a page
> where some other task has already triggered a userfault, so no new
> userfaultfd event is generated. We also need to observe which page has
> been faulted on so we can determine when the fault has been serviced
> and the task is ready to run again.
>
> I've tried to find workarounds with existing APIs and it doesn't seem
> tractable. See https://github.com/rr-debugger/rr/issues/2852#issuecomment-837514946
> for some thoughts about that.
>
> It seems to me that a sufficient API for us would be a new software
> perf event, e.g. PERF_COUNT_SW_USERFAULTS, with an associated
> PERF_SAMPLE_ADDR that would give us the address of the page. Does that
> sounds like a reasonable thing to add?

Is some combination of bpf and kprobes a possible solution? There are
some seemingly relevant examples here:
https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md

I haven't tried it, but it seems like attaching to handle_userfault()
would give similar information to perf_count_sw_page_faults, but for
userfaults.

>
> Robert O'Callahan