Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   KY Srinivasan <kys@microsoft.com>
To:     Vitaly Kuznetsov <vkuznets@redhat.com>,
        "Michael Kelley (EOSG)" <Michael.H.Kelley@microsoft.com>
CC:     "x86@kernel.org" <x86@kernel.org>,
        "devel@linuxdriverproject.org" <devel@linuxdriverproject.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Haiyang Zhang <haiyangz@microsoft.com>,
        Stephen Hemminger <sthemmin@microsoft.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>,
        "H. Peter Anvin" <hpa@zytor.com>,
        Tianyu Lan <Tianyu.Lan@microsoft.com>
Subject: RE: [PATCH] x86/hyper-v: use cheaper
 HVCALL_FLUSH_VIRTUAL_ADDRESS_{LIST,SPACE} hypercalls when possible
Thread-Topic: [PATCH] x86/hyper-v: use cheaper
 HVCALL_FLUSH_VIRTUAL_ADDRESS_{LIST,SPACE} hypercalls when possible
Thread-Index: AQHUBMYlSJQ3WMa140uivtTO6PwhI6Rn5ECAgADygfiAAJ8AEA==
Date:   Wed, 20 Jun 2018 17:55:40 +0000
Message-ID: <MW2PR2101MB111318C0735C2C7D6CD93ADBA0770@MW2PR2101MB1113.namprd21.prod.outlook.com>
References: <20180615163010.20381-1-vkuznets@redhat.com>
        <SN6PR2101MB11209236A3C18C7CA01F4061DC700@SN6PR2101MB1120.namprd21.prod.outlook.com>
 <87tvpxkgr5.fsf@vitty.brq.redhat.com>
In-Reply-To: <87tvpxkgr5.fsf@vitty.brq.redhat.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: microsoft.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: b695d383-bcb0-42b8-da77-08d5d6d7099c
X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jun 2018 17:55:40.9469
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR2101MB0939
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> -----Original Message-----
> From: Vitaly Kuznetsov <vkuznets@redhat.com>
> Sent: Wednesday, June 20, 2018 1:24 AM
> To: Michael Kelley (EOSG) <Michael.H.Kelley@microsoft.com>
> Cc: x86@kernel.org; devel@linuxdriverproject.org; linux-
> kernel@vger.kernel.org; KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> <haiyangz@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>; Thomas Gleixner <tglx@linutronix.de>; Ingo
> Molnar <mingo@redhat.com>; H. Peter Anvin <hpa@zytor.com>; Tianyu Lan
> <Tianyu.Lan@microsoft.com>
> Subject: Re: [PATCH] x86/hyper-v: use cheaper
> HVCALL_FLUSH_VIRTUAL_ADDRESS_{LIST,SPACE} hypercalls when possible
>=20
> "Michael Kelley (EOSG)" <Michael.H.Kelley@microsoft.com> writes:
>=20
> >> -----Original Message-----
> >> From: linux-kernel-owner@vger.kernel.org <linux-kernel-
> owner@vger.kernel.org> On Behalf
> >> Of Vitaly Kuznetsov
> >> Sent: Friday, June 15, 2018 9:30 AM
> >> To: x86@kernel.org
> >> Cc: devel@linuxdriverproject.org; linux-kernel@vger.kernel.org; KY
> Srinivasan
> >> <kys@microsoft.com>; Haiyang Zhang <haiyangz@microsoft.com>;
> Stephen Hemminger
> >> <sthemmin@microsoft.com>; Thomas Gleixner <tglx@linutronix.de>;
> Ingo Molnar
> >> <mingo@redhat.com>; H. Peter Anvin <hpa@zytor.com>; Tianyu Lan
> >> <Tianyu.Lan@microsoft.com>
> >> Subject: [PATCH] x86/hyper-v: use cheaper
> HVCALL_FLUSH_VIRTUAL_ADDRESS_{LIST,SPACE}
> >> hypercalls when possible
> >>
> >> While working on Hyper-V style PV TLB flush support in KVM I noticed t=
hat
> >> real Windows guests use TLB flush hypercall in a somewhat smarter way:
> when
> >> the flush needs to be performed on a subset of first 64 vCPUs or on al=
l
> >> present vCPUs Windows avoids more expensive hypercalls which support
> >> sparse CPU sets and uses their 'cheap' counterparts. This means that
> >> HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED name is actually a
> misnomer: EX
> >> hypercalls (which support sparse CPU sets) are "available", not
> >> "recommended". This makes sense as they are actually harder to parse.
> >>
> >> Nothing stops us from being equally 'smart' in Linux too. Switch to
> >> doing cheaper hypercalls whenever possible.
> >>
> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> >> ---
> >
> > This is a good idea.  We should probably do the same with the hypercall=
s for
> sending
> > IPIs -- try the simpler version first and move to the more complex _EX
> version only
> > if necessary.
> >
> > A complication:  We've recently found a problem with the code for doing
> IPI
> > hypercalls, and the bug affects the TLB flush code as well.  As seconda=
ry
> CPUs
> > are started, there's a window of time where the hv_vp_index entry for a
> > secondary CPU is uninitialized.  We are seeing IPIs happening in that
> window, and
> > the IPI hypercall code uses the uninitialized hv_vp_index entry.   Same
> thing could
> > happen with the TLB flush hypercall code.  I didn't actually see any
> occurrences of
> > the TLB case in my tracing, but we should fix it anyway in case a TLB f=
lush
> gets
> > added at some point in the future.
> >
> > KY has a patch coming.  In the patch, hv_cpu_number_to_vp_number()
> > and cpumask_to_vpset() can both return U32_MAX if they encounter an
> > uninitialized hv_vp_index entry, and the code needs to be able to bail =
out
> to
> > the native functions for that particular IPI or TLB flush operation.  O=
nce the
> > initialization of secondary CPUs is complete, the uninitialized situati=
on
> won't
> > happen again, and the hypercall path will always be used.
>=20
> Sure,

I am surprised that we have not seen this issue in tlb flush enlightenments=
.

K. Y
>=20
> with TLB flush we can always fall back to doing it natively (by sending
> IPIs).
>=20
> >
> > We'll need to coordinate on these patches.  Be aware that the IPI flavo=
r of
> the
> > bug is currently causing random failures when booting 4.18 RC1 on Hyper=
-V
> VMs
> > with large vCPU counts.
>=20
> Thanks for the heads up! This particular patch is just an optimization
> so there's no rush, IPI fix is definitely more important.
>=20
> >
> > Reviewed-by:  Michael Kelley <mikelley@microsoft.com>
>=20
> Thanks!
>=20
> --
>   Vitaly