Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Sunil Muthuswamy <sunilmut@microsoft.com>
To:     Praveen Kumar <kumarpraveen@linux.microsoft.com>,
        "linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
CC:     KY Srinivasan <kys@microsoft.com>,
        Haiyang Zhang <haiyangz@microsoft.com>,
        Stephen Hemminger <sthemmin@microsoft.com>,
        "wei.liu@kernel.org" <wei.liu@kernel.org>,
        Dexuan Cui <decui@microsoft.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "bp@alien8.de" <bp@alien8.de>, "x86@kernel.org" <x86@kernel.org>,
        "hpa@zytor.com" <hpa@zytor.com>,
        "viremana@linux.microsoft.com" <viremana@linux.microsoft.com>,
        "nunodasneves@linux.microsoft.com" <nunodasneves@linux.microsoft.com>
Subject: RE: [PATCH v5] hyperv: root partition faults writing to VP ASSIST MSR
 PAGE
Thread-Topic: [PATCH v5] hyperv: root partition faults writing to VP ASSIST
 MSR PAGE
Thread-Index: AQHXhgRi9VkW0Z/FOUWxjTerh/knP6tdiy6Q
Date:   Sat, 31 Jul 2021 20:36:39 +0000
Message-ID: <MW4PR21MB20020E5A5C33831B4FB20AC9C0ED9@MW4PR21MB2002.namprd21.prod.outlook.com>
References: <20210731120519.17154-1-kumarpraveen@linux.microsoft.com>
In-Reply-To: <20210731120519.17154-1-kumarpraveen@linux.microsoft.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?ShJ4S0xVESVjtZC/y/vyRDq/yKgLjaXZSLiZgGeF6JaIKbHjcmekni++kIE3?=
 =?us-ascii?Q?b3VVJwLYL9Y9dIVybJuIQCmRr96RijB79vazKMNiQArT48LFTRy8By634y8a?=
 =?us-ascii?Q?bMKThw/0ku4Cg/4zViaFIwEnDuCqClUsoF1aMLWYYjZhZANYN1lCGbl2PqVs?=
 =?us-ascii?Q?LjL0l4UN3NLJkrtMiE3stafK0e3DE7zQSdWDzLJY/GMro5Ab+c2MoqhPruQe?=
 =?us-ascii?Q?LcglBn+p1YG1wn4W3nLS50hTloOSNCDDX4HV4OmaDI9PcGGVX1s/ZFv5dI/9?=
 =?us-ascii?Q?xyHg/lui1ezAqV50H6xbbI6dot4+vVvNfxC65LUC/J092i8pZUGqfND8C9xx?=
 =?us-ascii?Q?5mJDNR+IFfrUAxydfBBUxa2N6dX/Dw4409iUvul0LBGnQLK2bBfoomjOTLwf?=
 =?us-ascii?Q?NmTgNjVSxAJ9AeCcR132t1Njn1J+F2QrbXgT07bRHuFzzxqQT5R+IHfP1HcC?=
 =?us-ascii?Q?dXO4f9lOslc2jobAoVlj8w01O/CrMjIB6nOu263MY8ZxODIYoarSmrS6yoz9?=
 =?us-ascii?Q?3gyMqsYGizUsbmrtEYbE5cUfTZDqWbzX60SELjrAG5ZfwY3pTbBXMxh2O6yK?=
 =?us-ascii?Q?LnX/WljHo6SRUvguSfcxQtU+f75T6A2ppKyJx+HOjkK17v9kiTYsqux/4xQO?=
 =?us-ascii?Q?SmDaSgTETCFNcn42TEfwUErCDTt5U6s4gSDNjP8Ws7USK5eT+P8r86jCtI/Y?=
 =?us-ascii?Q?b3S9tVPvvrhwdb0ywvjUgv1uiIFiFT+KUnab82sEJJYnxhoX8vFIOT8qNApk?=
 =?us-ascii?Q?m7sjuPB81SAQMSH1U/sur5ZU4XKWUMKmAtkzsKGknus7hYQiCQaXHWn0aWvZ?=
 =?us-ascii?Q?VZDp00eYaY2AMWlleAWX+SDC1ymwIN2T+m2JZTC2Z+dj92TydsxLcuw2oj6x?=
 =?us-ascii?Q?p3CRFqn/9kmbECwhw3F7YJvWu16AcYbhrlZ9i785rrDba4Fq1XoHerA47aWY?=
 =?us-ascii?Q?uPZqmPaT1Ek2DmH5mJt+J+zZdh7RH7wIk5c5AWh6UBwLDJlFcP7bhUulGCDP?=
 =?us-ascii?Q?qWvibZLYKq5ZBZq46/E1xx1PbVxR15EnmELC1pzjxDYXkw/rlcfVVlxZ/rEA?=
 =?us-ascii?Q?guQbOSPt3518UImV61oOrAVR7B/PawF44cbS5vYOKO7UWf+blLP16gmwT872?=
 =?us-ascii?Q?bbUdZMHDq+aytmfL5UspYP0TVQySGJ+2W5B+vR8dq/o0cYR7GjB43IIbMnPE?=
 =?us-ascii?Q?xZDnF9VRLZSBkTN8XCoQ6YSH6GB8uXEdrgS2GvsRUKz4khaprUj0iMtPTm+r?=
 =?us-ascii?Q?WQhPo9UBMdGQlz+yDU14sfMhTzawSDInWO6NOibv9C0olBNwBojt8nYZysO4?=
 =?us-ascii?Q?q+YVao1FRsJdXTT8ez9De1z5faw4NvrGSytj0BDiTFI22c0llIyXgcRfpOP/?=
 =?us-ascii?Q?io4eFXr95w9MQwD1QddO0ZsuMcrH?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MW4PR21MB2002.namprd21.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b5d13c90-72d8-46bf-62f5-08d95462e618
X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Jul 2021 20:36:39.1998
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: RjAEmsouwZe4KaiOYsoa+kanQ5VkSz4JoLp5aSCjJZ8thSO0drwIcS94ULGcLba8BYDf1tMgacGW5FPNVI3GoR1wd9aH8+T285pxkvSycAU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR21MB0829
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> For Root partition the VP assist pages are pre-determined by the
> hypervisor. The Root kernel is not allowed to change them to
> different locations. And thus, we are getting below stack as in
> current implementation Root is trying to perform write to specific
> MSR.
>=20
> [ 2.778197] unchecked MSR access error: WRMSR to 0x40000073 (tried to
> write 0x0000000145ac5001) at rIP: 0xffffffff810c1084
> (native_write_msr+0x4/0x30)
> [ 2.784867] Call Trace:
> [ 2.791507] hv_cpu_init+0xf1/0x1c0
> [ 2.798144] ? hyperv_report_panic+0xd0/0xd0
> [ 2.804806] cpuhp_invoke_callback+0x11a/0x440
> [ 2.811465] ? hv_resume+0x90/0x90
> [ 2.818137] cpuhp_issue_call+0x126/0x130
> [ 2.824782] __cpuhp_setup_state_cpuslocked+0x102/0x2b0
> [ 2.831427] ? hyperv_report_panic+0xd0/0xd0
> [ 2.838075] ? hyperv_report_panic+0xd0/0xd0
> [ 2.844723] ? hv_resume+0x90/0x90
> [ 2.851375] __cpuhp_setup_state+0x3d/0x90
> [ 2.858030] hyperv_init+0x14e/0x410
> [ 2.864689] ? enable_IR_x2apic+0x190/0x1a0
> [ 2.871349] apic_intr_mode_init+0x8b/0x100
> [ 2.878017] x86_late_time_init+0x20/0x30
> [ 2.884675] start_kernel+0x459/0x4fb
> [ 2.891329] secondary_startup_64_no_verify+0xb0/0xbb
>=20
> Since, the hypervisor already provides the VP assist page for root
> partition, we need to memremap the memory from hypervisor for root
> kernel to use. The mapping is done in hv_cpu_init during bringup and
> is unmaped in hv_cpu_die during teardown.
>=20
> Signed-off-by: Praveen Kumar <kumarpraveen@linux.microsoft.com>
> ---
>  arch/x86/hyperv/hv_init.c          | 64 ++++++++++++++++++++----------
>  arch/x86/include/asm/hyperv-tlfs.h |  9 +++++
>  2 files changed, 53 insertions(+), 20 deletions(-)
>=20
> changelog:
> v1: initial patch
> v2: commit message changes, removal of HV_MSR_APIC_ACCESS_AVAILABLE
>     check and addition of null check before reading the VP assist MSR
>     for root partition
> v3: added new data structure to handle VP ASSIST MSR page and done
>     handling in hv_cpu_init and hv_cpu_die
> v4: better code alignment, VP ASSIST handling correction for root
>     partition in hv_cpu_die and renaming of hv_vp_assist_msr_contents
>     attribute
> v5: disable VP ASSIST page for root partition during hv_cpu_die
> ---
> diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
> index 6f247e7e07eb..a46bd92c532a 100644
> --- a/arch/x86/hyperv/hv_init.c
> +++ b/arch/x86/hyperv/hv_init.c
> @@ -44,6 +44,7 @@ EXPORT_SYMBOL_GPL(hv_vp_assist_page);
>=20
>  static int hv_cpu_init(unsigned int cpu)
>  {
> +	union hv_vp_assist_msr_contents msr =3D {0};
>  	struct hv_vp_assist_page **hvp =3D &hv_vp_assist_page[smp_processor_id(=
)];
>  	int ret;
>=20
> @@ -54,25 +55,34 @@ static int hv_cpu_init(unsigned int cpu)
>  	if (!hv_vp_assist_page)
>  		return 0;
>=20
> -	/*
> -	 * The VP ASSIST PAGE is an "overlay" page (see Hyper-V TLFS's Section
> -	 * 5.2.1 "GPA Overlay Pages"). Here it must be zeroed out to make sure
> -	 * we always write the EOI MSR in hv_apic_eoi_write() *after* the
> -	 * EOI optimization is disabled in hv_cpu_die(), otherwise a CPU may
> -	 * not be stopped in the case of CPU offlining and the VM will hang.
> -	 */
>  	if (!*hvp) {
> -		*hvp =3D __vmalloc(PAGE_SIZE, GFP_KERNEL | __GFP_ZERO);
> -	}
> -
> -	if (*hvp) {
> -		u64 val;
> -
> -		val =3D vmalloc_to_pfn(*hvp);
> -		val =3D (val << HV_X64_MSR_VP_ASSIST_PAGE_ADDRESS_SHIFT) |
> -			HV_X64_MSR_VP_ASSIST_PAGE_ENABLE;
> -
> -		wrmsrl(HV_X64_MSR_VP_ASSIST_PAGE, val);
> +		if (hv_root_partition) {
> +			/*
> +			 * For Root partition we get the hypervisor provided VP ASSIST
> +			 * PAGE, instead of allocating a new page.
> +			 */
> +			rdmsrl(HV_X64_MSR_VP_ASSIST_PAGE, msr.as_uint64);
> +			*hvp =3D memremap(msr.pfn <<
> +					HV_X64_MSR_VP_ASSIST_PAGE_ADDRESS_SHIFT,
> +					PAGE_SIZE, MEMREMAP_WB);
> +		} else {
> +			/*
> +			 * The VP ASSIST PAGE is an "overlay" page (see Hyper-V TLFS's
> +			 * Section 5.2.1 "GPA Overlay Pages"). Here it must be zeroed
> +			 * out to make sure we always write the EOI MSR in
> +			 * hv_apic_eoi_write() *after* theEOI optimization is disabled
> +			 * in hv_cpu_die(), otherwise a CPU may not be stopped in the
> +			 * case of CPU offlining and the VM will hang.
> +			 */
> +			*hvp =3D __vmalloc(PAGE_SIZE, GFP_KERNEL | __GFP_ZERO);
> +			if (*hvp)
> +				msr.pfn =3D vmalloc_to_pfn(*hvp);
> +		}
> +		WARN_ON(!(*hvp));
> +		if (*hvp) {
> +			msr.enable =3D 1;
> +			wrmsrl(HV_X64_MSR_VP_ASSIST_PAGE, msr.as_uint64);
> +		}
>  	}
>=20
>  	return 0;
> @@ -170,8 +180,22 @@ static int hv_cpu_die(unsigned int cpu)
>=20
>  	hv_common_cpu_die(cpu);
>=20
> -	if (hv_vp_assist_page && hv_vp_assist_page[cpu])
> -		wrmsrl(HV_X64_MSR_VP_ASSIST_PAGE, 0);
> +	if (hv_vp_assist_page && hv_vp_assist_page[cpu]) {
> +		union hv_vp_assist_msr_contents msr =3D {0};
> +		if (hv_root_partition) {
> +			/*
> +			 * For Root partition the VP ASSIST page is mapped to
> +			 * hypervisor provided page, and thus, we unmap the
> +			 * page here and nullify it, so that in future we have
> +			 * correct page address mapped in hv_cpu_init.
> +			 */
> +			memunmap(hv_vp_assist_page[cpu]);
> +			hv_vp_assist_page[cpu] =3D NULL;
> +			rdmsrl(HV_X64_MSR_VP_ASSIST_PAGE, msr.as_uint64);
> +			msr.enable =3D 0;
> +		}
> +		wrmsrl(HV_X64_MSR_VP_ASSIST_PAGE, msr.as_uint64);
> +	}
>=20
>  	if (hv_reenlightenment_cb =3D=3D NULL)
>  		return 0;
> diff --git a/arch/x86/include/asm/hyperv-tlfs.h b/arch/x86/include/asm/hy=
perv-tlfs.h
> index f1366ce609e3..2322d6bd5883 100644
> --- a/arch/x86/include/asm/hyperv-tlfs.h
> +++ b/arch/x86/include/asm/hyperv-tlfs.h
> @@ -288,6 +288,15 @@ union hv_x64_msr_hypercall_contents {
>  	} __packed;
>  };
>=20
> +union hv_vp_assist_msr_contents {
> +	u64 as_uint64;
> +	struct {
> +		u64 enable:1;
> +		u64 reserved:11;
> +		u64 pfn:52;
> +	} __packed;
> +};
> +
>  struct hv_reenlightenment_control {
>  	__u64 vector:8;
>  	__u64 reserved1:8;
> --
> 2.25.1

Reviewed-by: Sunil Muthuswamy <sunilmut@microsoft.com>