Received: by 2002:a05:6a10:c7c6:0:0:0:0 with SMTP id h6csp1954075pxy; Mon, 2 Aug 2021 14:59:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzmGf76s7FzOvvzD+ZGtDa/wIiXcjYdl9OYQBQG1CTbtsUTgaMSKAoePYQs6HbOeQxCFvQd X-Received: by 2002:a17:906:3019:: with SMTP id 25mr17240700ejz.91.1627941576544; Mon, 02 Aug 2021 14:59:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1627941576; cv=none; d=google.com; s=arc-20160816; b=A7Wha7ccbPybdvYrcvdSHOi7ss/S+RSs1GlpUCuOtN7T8+5iUSzHhoznHBZwsJXbAl X81O6D05NDTR63pz17JFpWIviuXI8VDeZye6BPMEz9f3A0Y90MRQEatwzBiTO9PID0Dj gCHbUvnW7yA5UWmMtS04c5JmMZktDfBPhurBv5GV10Tu7C5VM+GWXf1060yXkX76cUUr 3rsfHYG789fWakfYzJOgMiApnrqRh/irUyhqtHoRuIaP+l+ty47O5LHF5krg1mnFX/mX ZKFC+0Yuqj7K1YZPAiaXA50F4DOjUSD7CfVq0MHA+q8gVwhUY44ZALsqiX+Io/6fMt75 dM/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=d788AXnqtR7xmaTEEUS9xe8Hn4Sx6OLWF7vdJSotwpM=; b=lRTXF0WrEesGr3QEOrlJz812X4PvvoeI7jgO0iYRqaO3YiU29k7XsPLbUPasFrqsTv b04elOMGNVgGGEfeHNG0C27F9EA/6WhrhneiyEk5jrceVwng5XvKfKjdZfgovrac1jIL TFieMkEYnJ+J5/7sA5Jd1l/dLB73UVl44zDazLEdStexpSVtsIK1/4mnAd3/TYlX2TJf eyiJvLPUlNtDxI8l8LdTGUrGxrJpoP2z5OI9a98W6AWre0OKoQDFEWmzKXqq20WM1npG 7lq6zjkVsqldnUvgXgjNxm3Lb/w4oIyfFt2mrGpJhI4PiUhoZ2XTSyw8del/XOkRRRU9 Tjnw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@soleen.com header.s=google header.b=HNBk5sK5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id o1si637073eji.88.2021.08.02.14.59.13; Mon, 02 Aug 2021 14:59:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@soleen.com header.s=google header.b=HNBk5sK5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231983AbhHBVzE (ORCPT + 99 others); Mon, 2 Aug 2021 17:55:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233092AbhHBVyi (ORCPT ); Mon, 2 Aug 2021 17:54:38 -0400 Received: from mail-qv1-xf2c.google.com (mail-qv1-xf2c.google.com [IPv6:2607:f8b0:4864:20::f2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 16AC7C0613D5 for ; Mon, 2 Aug 2021 14:54:27 -0700 (PDT) Received: by mail-qv1-xf2c.google.com with SMTP id em4so4786461qvb.0 for ; Mon, 02 Aug 2021 14:54:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=d788AXnqtR7xmaTEEUS9xe8Hn4Sx6OLWF7vdJSotwpM=; b=HNBk5sK5nIpwScV6ruErOFK1CnoHkcV1P3nVWVA+EofJS6HR91CV8HytnoU/ushldA MRjHNHRXIM4W8s+NiJIQDZnDMGJIdVUqv2UB0tAAKFikB0jmrT/clURsgEJWX4geXOdz 2WqPpFsHfJQvDcc4rf7QESPXd3q4gpGbOgVh4NP9veiztWV31qIpsMdqEZSVGsOuzjb0 MtaidV+U/5sKOJec1RaWIhLB2+aAFaf8nKGoldWdrjWtcBz0akMXkEM5ixzEh28shuon dcSJdfb/4ZGjju1DaTmmrePZ0fX08LAvpowX/PF1wjrYL/mhzepoVLWuX0kmU/3dp87J gYtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=d788AXnqtR7xmaTEEUS9xe8Hn4Sx6OLWF7vdJSotwpM=; b=Lepvcg0EcN3mISMzCQgQWcfot7CI+pDRU45vyHyNDOgdCJlGj1GG6KcTmUOjkgNDaW gPUORZLssN29ZAuj156vUIHgngYR5bLyXZQ5OT/lS/eoV5fuDfgbfWCwC1JunFDDR/fl TgmzPbEeKD4F4sQEEPzoNZCbMO7EAHOUAs0RdYVKimgtk42sDV7Bxcnbx6YHnil+S/gT tIqruuTlXdOM1mJHJvi3kvZyokAyCwq9FHp3tujA18JqW6WpClg5Zu9MRpuoG8b6Q/YS 3N2ARY46bYe4ucj93/v75/IbnGr1DEs23I3EulugKB8YVFCfiTK0RkMOnq5dr3IBCUd5 2OeA== X-Gm-Message-State: AOAM530B2yXPkdVJNhfFWH+JPbg89/QRVZAG+J8/xFkCX0Xq17Sjq/6Y ypkbajNL7fuHn4nLSk6bp9ZEN6pRILVxkeup X-Received: by 2002:a0c:c612:: with SMTP id v18mr18349323qvi.20.1627941266265; Mon, 02 Aug 2021 14:54:26 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id v11sm5479216qtc.0.2021.08.02.14.54.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Aug 2021 14:54:25 -0700 (PDT) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com, kernelfans@gmail.com, akpm@linux-foundation.org, madvenka@linux.microsoft.com Subject: [PATCH v16 11/15] arm64: kexec: install a copy of the linear-map Date: Mon, 2 Aug 2021 17:54:04 -0400 Message-Id: <20210802215408.804942-12-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210802215408.804942-1-pasha.tatashin@soleen.com> References: <20210802215408.804942-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org To perform the kexec relocation with the MMU enabled, we need a copy of the linear map. Create one, and install it from the relocation code. This has to be done from the assembly code as it will be idmapped with TTBR0. The kernel runs in TTRB1, so can't use the break-before-make sequence on the mapping it is executing from. The makes no difference yet as the relocation code runs with the MMU disabled. Suggested-by: James Morse Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/assembler.h | 19 +++++++++++++++++++ arch/arm64/include/asm/kexec.h | 2 ++ arch/arm64/kernel/asm-offsets.c | 2 ++ arch/arm64/kernel/hibernate-asm.S | 20 -------------------- arch/arm64/kernel/machine_kexec.c | 16 ++++++++++++++-- arch/arm64/kernel/relocate_kernel.S | 3 +++ 6 files changed, 40 insertions(+), 22 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 71999a325055..4289c4e1c2a3 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -483,6 +483,25 @@ alternative_endif _cond_extable .Licache_op\@, \fixup .endm +/* + * To prevent the possibility of old and new partial table walks being visible + * in the tlb, switch the ttbr to a zero page when we invalidate the old + * records. D4.7.1 'General TLB maintenance requirements' in ARM DDI 0487A.i + * Even switching to our copied tables will cause a changed output address at + * each stage of the walk. + */ + .macro break_before_make_ttbr_switch zero_page, page_table, tmp, tmp2 + phys_to_ttbr \tmp, \zero_page + msr ttbr1_el1, \tmp + isb + tlbi vmalle1 + dsb nsh + phys_to_ttbr \tmp, \page_table + offset_ttbr1 \tmp, \tmp2 + msr ttbr1_el1, \tmp + isb + .endm + /* * reset_pmuserenr_el0 - reset PMUSERENR_EL0 if PMUv3 present */ diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 753a1c398898..d678f0ceb7ee 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -97,6 +97,8 @@ struct kimage_arch { phys_addr_t dtb_mem; phys_addr_t kern_reloc; phys_addr_t el2_vectors; + phys_addr_t ttbr1; + phys_addr_t zero_page; }; #ifdef CONFIG_KEXEC_FILE diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 47ccb57b787e..a98384d13ee7 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -175,6 +175,8 @@ int main(void) #ifdef CONFIG_KEXEC_CORE DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); DEFINE(KIMAGE_ARCH_EL2_VECTORS, offsetof(struct kimage, arch.el2_vectors)); + DEFINE(KIMAGE_ARCH_ZERO_PAGE, offsetof(struct kimage, arch.zero_page)); + DEFINE(KIMAGE_ARCH_TTBR1, offsetof(struct kimage, arch.ttbr1)); DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); DEFINE(KIMAGE_START, offsetof(struct kimage, start)); BLANK(); diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S index a30a2c3f905e..0e1d9c3c6a93 100644 --- a/arch/arm64/kernel/hibernate-asm.S +++ b/arch/arm64/kernel/hibernate-asm.S @@ -15,26 +15,6 @@ #include #include -/* - * To prevent the possibility of old and new partial table walks being visible - * in the tlb, switch the ttbr to a zero page when we invalidate the old - * records. D4.7.1 'General TLB maintenance requirements' in ARM DDI 0487A.i - * Even switching to our copied tables will cause a changed output address at - * each stage of the walk. - */ -.macro break_before_make_ttbr_switch zero_page, page_table, tmp, tmp2 - phys_to_ttbr \tmp, \zero_page - msr ttbr1_el1, \tmp - isb - tlbi vmalle1 - dsb nsh - phys_to_ttbr \tmp, \page_table - offset_ttbr1 \tmp, \tmp2 - msr ttbr1_el1, \tmp - isb -.endm - - /* * Resume from hibernate * diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 83da6045cd45..50bc0a265c86 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -159,6 +159,8 @@ static void *kexec_page_alloc(void *arg) int machine_kexec_post_load(struct kimage *kimage) { + int rc; + pgd_t *trans_pgd; void *reloc_code = page_to_virt(kimage->control_code_page); long reloc_size; struct trans_pgd_info info = { @@ -175,12 +177,22 @@ int machine_kexec_post_load(struct kimage *kimage) kimage->arch.el2_vectors = 0; if (is_hyp_nvhe()) { - int rc = trans_pgd_copy_el2_vectors(&info, - &kimage->arch.el2_vectors); + rc = trans_pgd_copy_el2_vectors(&info, + &kimage->arch.el2_vectors); if (rc) return rc; } + /* Create a copy of the linear map */ + trans_pgd = kexec_page_alloc(kimage); + if (!trans_pgd) + return -ENOMEM; + rc = trans_pgd_create_copy(&info, &trans_pgd, PAGE_OFFSET, PAGE_END); + if (rc) + return rc; + kimage->arch.ttbr1 = __pa(trans_pgd); + kimage->arch.zero_page = __pa(empty_zero_page); + reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code); diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 9d2400855ee4..a07b737533c3 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -29,10 +29,13 @@ */ SYM_CODE_START(arm64_relocate_new_kernel) /* Setup the list loop variables. */ + ldr x18, [x0, #KIMAGE_ARCH_ZERO_PAGE] /* x18 = zero page for BBM */ + ldr x17, [x0, #KIMAGE_ARCH_TTBR1] /* x17 = linear map copy */ ldr x16, [x0, #KIMAGE_HEAD] /* x16 = kimage_head */ mov x14, xzr /* x14 = entry ptr */ mov x13, xzr /* x13 = copy dest */ raw_dcache_line_size x15, x1 /* x15 = dcache line size */ + break_before_make_ttbr_switch x18, x17, x1, x2 /* set linear map */ .Lloop: and x12, x16, PAGE_MASK /* x12 = addr */ -- 2.25.1