Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <20181018005420.82993-1-namit@vmware.com> <20181018005420.82993-2-namit@vmware.com>
 <D247AB7C-2890-49CE-B102-163E5B368FC2@amacapital.net> <07255D2B-0243-4254-B62A-37050C44207E@vmware.com>
 <CALCETrXEG8OrMpB3VBS9RF41O_MwtPmXBazkdSGE0ohZmXDcBw@mail.gmail.com>
 <925F22EA-F8CB-4194-B96B-378409ED7918@vmware.com> <2626124E-7344-42F3-AD07-0BB34D62A9EE@amacapital.net>
 <2054C1A9-37C1-4A5A-A716-EDAC90564D2A@vmware.com>
In-Reply-To: <2054C1A9-37C1-4A5A-A716-EDAC90564D2A@vmware.com>
From:   Andy Lutomirski <luto@amacapital.net>
Date:   Thu, 18 Oct 2018 10:29:13 -0700
Message-ID: <CALCETrUQwHysVOJ2G8AQF25TRVaqUi7k93K8nks4Yp+fFehVBA@mail.gmail.com>
Subject: Re: [RFC PATCH 1/5] x86: introduce preemption disable prefix
To:     Nadav Amit <namit@vmware.com>
Cc:     Ingo Molnar <mingo@redhat.com>,
        Andrew Lutomirski <luto@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        "H. Peter Anvin" <hpa@zytor.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        LKML <linux-kernel@vger.kernel.org>, X86 ML <x86@kernel.org>,
        Borislav Petkov <bp@alien8.de>,
        "Woodhouse, David" <dwmw@amazon.co.uk>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, Oct 18, 2018 at 10:25 AM Nadav Amit <namit@vmware.com> wrote:
>
> at 10:00 AM, Andy Lutomirski <luto@amacapital.net> wrote:
>
> >
> >
> >> On Oct 18, 2018, at 9:47 AM, Nadav Amit <namit@vmware.com> wrote:
> >>
> >> at 8:51 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> >>
> >>>> On Wed, Oct 17, 2018 at 8:12 PM Nadav Amit <namit@vmware.com> wrote:
> >>>> at 6:22 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> >>>>
> >>>>>> On Oct 17, 2018, at 5:54 PM, Nadav Amit <namit@vmware.com> wrote:
> >>>>>>
> >>>>>> It is sometimes beneficial to prevent preemption for very few
> >>>>>> instructions, or prevent preemption for some instructions that pre=
cede
> >>>>>> a branch (this latter case will be introduced in the next patches)=
.
> >>>>>>
> >>>>>> To provide such functionality on x86-64, we use an empty REX-prefi=
x
> >>>>>> (opcode 0x40) as an indication that preemption is disabled for the
> >>>>>> following instruction.
> >>>>>
> >>>>> Nifty!
> >>>>>
> >>>>> That being said, I think you have a few bugs. First, you can=E2=80=
=99t just ignore
> >>>>> a rescheduling interrupt, as you introduce unbounded latency when t=
his
> >>>>> happens =E2=80=94 you=E2=80=99re effectively emulating preempt_enab=
le_no_resched(), which
> >>>>> is not a drop-in replacement for preempt_enable(). To fix this, you=
 may
> >>>>> need to jump to a slow-path trampoline that calls schedule() at the=
 end or
> >>>>> consider rewinding one instruction instead. Or use TF, which is onl=
y a
> >>>>> little bit terrifying=E2=80=A6
> >>>>
> >>>> Yes, I didn=E2=80=99t pay enough attention here. For my use-case, I =
think that the
> >>>> easiest solution would be to make synchronize_sched() ignore preempt=
ions
> >>>> that happen while the prefix is detected. It would slightly change t=
he
> >>>> meaning of the prefix.
> >>
> >> So thinking about it further, rewinding the instruction seems the easi=
est
> >> and most robust solution. I=E2=80=99ll do it.
> >>
> >>>>> You also aren=E2=80=99t accounting for the case where you get an ex=
ception that
> >>>>> is, in turn, preempted.
> >>>>
> >>>> Hmm.. Can you give me an example for such an exception in my use-cas=
e? I
> >>>> cannot think of an exception that might be preempted (assuming #BP, =
#MC
> >>>> cannot be preempted).
> >>>
> >>> Look for cond_local_irq_enable().
> >>
> >> I looked at it. Yet, I still don=E2=80=99t see how exceptions might ha=
ppen in my
> >> use-case, but having said that - this can be fixed too.
> >
> > I=E2=80=99m not totally certain there=E2=80=99s a case that matters.  B=
ut it=E2=80=99s worth checking
> >
> >> To be frank, I paid relatively little attention to this subject. Any
> >> feedback about the other parts and especially on the high-level approa=
ch? Is
> >> modifying the retpolines in the proposed manner (assembly macros)
> >> acceptable?
> >
> > It=E2=80=99s certainly a neat idea, and it could be a real speedup.
>
> Great. So I=E2=80=99ll try to shape things up, and I still wait for other=
 comments
> (from others).
>
> I=E2=80=99ll just mention two more patches I need to cleanup (I know I st=
ill owe you some
> work, so obviously it will be done later):
>
> 1. Seccomp trampolines. On my Ubuntu, when I run Redis, systemd installs =
17
> BPF filters on the Redis server process that are invoked on each
> system-call. Invoking each one requires an indirect branch. The patch kee=
ps
> a per-process kernel code-page that holds trampolines for these functions=
.

I wonder how many levels of branches are needed before the branches
involved exceed the retpoline cost.

>
> 2. Binary-search for system-calls. Use the per-process kernel code-page a=
lso
> to hold multiple trampolines for the 16 common system calls of a certain
> process. The patch uses an indirection table and a binary-search to find =
the
> proper trampoline.

Same comment applies here.

>
> Thanks again,
> Nadav


--=20
Andy Lutomirski
AMA Capital Management, LLC