Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (1.0)
Subject: Re: [PATCH 03/10] smp,cpumask: introduce on_each_cpu_cond_mask
From:   Andy Lutomirski <luto@amacapital.net>
In-Reply-To: <1532886673.28585.24.camel@surriel.com>
Date:   Sun, 29 Jul 2018 11:55:30 -0700
Cc:     Paolo Bonzini <pbonzini@redhat.com>,
        Andy Lutomirski <luto@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        kernel-team <kernel-team@fb.com>,
        Peter Zijlstra <peterz@infradead.org>, X86 ML <x86@kernel.org>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Ingo Molnar <mingo@kernel.org>, Mike Galbraith <efault@gmx.de>,
        Dave Hansen <dave.hansen@intel.com>, will.daecon@arm.com,
        Catalin Marinas <catalin.marinas@arm.com>,
        Benjamin Herrenschmidt <benh@kernel.crashing.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <8A3221EE-0D94-487E-B53D-885A555634BD@amacapital.net>
References: <20180728215357.3249-1-riel@surriel.com> <20180728215357.3249-4-riel@surriel.com> <CALCETrUipqat-78++E8fL7d2=77Cbj72jCTZMO=yNF8nX9HhCA@mail.gmail.com> <1532865634.28585.2.camel@surriel.com> <E710FBA5-CC5E-4941-ACBF-4AB3424F1F68@amacapital.net> <1532886673.28585.24.camel@surriel.com>
To:     Rik van Riel <riel@surriel.com>, torvalds@linux-foundation.org
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On Jul 29, 2018, at 10:51 AM, Rik van Riel <riel@surriel.com> wrote:
>=20
>> On Sun, 2018-07-29 at 08:36 -0700, Andy Lutomirski wrote:
>>> On Jul 29, 2018, at 5:00 AM, Rik van Riel <riel@surriel.com> wrote:
>>>=20
>>>> On Sat, 2018-07-28 at 19:57 -0700, Andy Lutomirski wrote:
>>>> On Sat, Jul 28, 2018 at 2:53 PM, Rik van Riel <riel@surriel.com>
>>>> wrote:
>>>>> Introduce a variant of on_each_cpu_cond that iterates only over
>>>>> the
>>>>> CPUs in a cpumask, in order to avoid making callbacks for every
>>>>> single
>>>>> CPU in the system when we only need to test a subset.
>>>> Nice.
>>>> Although, if you want to be really fancy, you could optimize this
>>>> (or
>>>> add a variant) that does the callback on the local CPU in
>>>> parallel
>>>> with the remote ones.  That would give a small boost to TLB
>>>> flushes.
>>>=20
>>> The test_func callbacks are not run remotely, but on
>>> the local CPU, before deciding who to send callbacks
>>> to.
>>>=20
>>> The actual IPIs are sent in parallel, if the cpumask
>>> allocation succeeds (it always should in many kernel
>>> configurations, and almost always in the rest).
>>>=20
>>=20
>> What I meant is that on_each_cpu_mask does:
>>=20
>> smp_call_function_many(mask, func, info, wait);
>> if (cpumask_test_cpu(cpu, mask)) {
>>   unsigned long flags;
>>   local_irq_save(flags); func(info);
>>   local_irq_restore(flags);
>> }
>>=20
>> So it IPIs all the remote CPUs in parallel, then waits, then does the
>> local work.  In principle, the local flush could be done after
>> triggering the IPIs but before they all finish.
>=20
> Grepping around the code, I found a few examples where the
> calling code appears to expect that smp_call_function_many
> also calls "func" on the local CPU.
>=20
> For example, kvm_emulate_wbinvd_noskip has this:
>=20
>        if (kvm_x86_ops->has_wbinvd_exit()) {
>                int cpu =3D get_cpu();
>=20
>                cpumask_set_cpu(cpu, vcpu->arch.wbinvd_dirty_mask);
>                smp_call_function_many(vcpu->arch.wbinvd_dirty_mask,
>                                wbinvd_ipi, NULL, 1);
>                put_cpu();
>                cpumask_clear(vcpu->arch.wbinvd_dirty_mask);
>        } else
>                wbinvd();
>=20
> This seems to result in systems with ->has_wbinvd_exit
> only calling wbinvd_ipi on OTHER CPUs, and not on the
> CPU where the guest exited with wbinvd?
>=20
> This seems unintended.
>=20
> I guess looking into on_each_cpu_mask might be a little
> higher priority than waiting until the next Outreachy
> season :)
>=20

The right approach might be a tree wise rename from smp_call_... to on_other=
_cpus_mask() it similar. The current naming and semantics are extremely conf=
using.

Linus, this is the kind of thing you seem to like taking outside the merge w=
indow. What do you think about a straight-up search_and_replace to make rena=
me the smp_call_... functions to exactly match the corresponding on_each_cpu=
 functions except with =E2=80=9Ceach=E2=80=9D replaced with =E2=80=9Cother=E2=
=80=9D?=