Received: by 2002:a05:6a10:16a7:0:0:0:0 with SMTP id gp39csp1111171pxb; Fri, 6 Nov 2020 00:56:11 -0800 (PST) X-Google-Smtp-Source: ABdhPJx+fDMRZn1NUukeZxoE8KLqB5ascV7q98RXW6sf3O0CZqlmQBIYmRCse4XIayP04RFe/wCa X-Received: by 2002:a17:906:2444:: with SMTP id a4mr1055859ejb.415.1604652970856; Fri, 06 Nov 2020 00:56:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1604652970; cv=none; d=google.com; s=arc-20160816; b=c/vJQjZGrdb7XEbaYSPwbhgNyDSoFW3QbkPBuiLjgW+0LwozzhTo6fXi1mFJzqQJ8p 5u3a8wvTBtbjQ4L/hfnmHWQne7zwBx7MbsSmQhd2idVDx+aMF6tfvwMMJr1lzADdmHiu t6+9h4/6rqr3VTJ2PMWTv4KgMPN5ezPYRkbAjY+b66r+C8kd62Yh3l6t+c1i1jdhORKa vWSx8zEhbVjWRuVweUpoznIfPbNOBSFQPHkOVvfofets893d3E6CyrBvxbBBOOAIe2J6 ss9DBCKtljhlmjEXDRJNDtxyz5uNtKUyjBL/UW2Iv5FJt08OSX03rmNnoquVWuQA+DnH bB6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:in-reply-to :references:mime-version:dkim-signature; bh=1SlIddTQa6YnvwCKFNoc0Zc6N2GDs/YOmt73lg/h6iM=; b=NSSrwGAuAB0kTh9mu/MLRRd9ATrWv7BX5oTRaRTV7X1N0s7gVzVysXUSkIS4qhA9Gv VYjEoAN1nu1qxf/rg3iW9Wi/NvT5nKpv6G9c9qoFNMSfHmiDJYFeII+8iCgv2LMSwThg FSeUrOEi7KWSfnoWzF6RUMHqJw9I7ma+Mhg1A+zRdALbpz+ZDqyIaj4+qGFGJFwbloyN tEx9BLerdIpvoqU3wCDl0zlZdhvTK2RmCJrEE6QEC6ceEeFnIekIzBOLfPd6+P1MyS+L ebvfGJ/zQ5egBhpMPo7QWW9ETSvv9nFPhnq/q17hOAJB/4qazBKHmqCUNQp1cbUnFm7y Ny7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@brainfault-org.20150623.gappssmtp.com header.s=20150623 header.b=CMIX+ykz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id d2si462101ejm.262.2020.11.06.00.55.47; Fri, 06 Nov 2020 00:56:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@brainfault-org.20150623.gappssmtp.com header.s=20150623 header.b=CMIX+ykz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726250AbgKFIwy (ORCPT + 99 others); Fri, 6 Nov 2020 03:52:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725835AbgKFIwy (ORCPT ); Fri, 6 Nov 2020 03:52:54 -0500 Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF70AC0613CF for ; Fri, 6 Nov 2020 00:52:53 -0800 (PST) Received: by mail-wr1-x441.google.com with SMTP id c17so439716wrc.11 for ; Fri, 06 Nov 2020 00:52:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=brainfault-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=1SlIddTQa6YnvwCKFNoc0Zc6N2GDs/YOmt73lg/h6iM=; b=CMIX+ykzx4ZW4zP5Hx5PbPAkifwInBnOURHV+C70jv97Jl8yMGbM/8E5kIi+AG8/hZ QHPHPFmSM2stfBD58jaxUtDqBwoLloMXo8bXDz3YLFJNP7d6CEX6mRRmCb//e4MkQiNh sv4PKoknzxggLZ/6vKGJXczN65VuIgI3kDjHMzz2b5NQj4DkjDRBGpVg+4o3lzlaeMoV pjK3ZPBNUqk5TXqXaXsf+d1F4dgB3nYGdR4c5sm2qgRpiUDjEJfIgrHgHv0NFp9mZt4Q P8L8+P1JkF2HXC2yclg8kntyuKKsP0hH9xJ7V6CWj3IQ0ltdLrSluOdR6SM27gMkVKWy xLtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=1SlIddTQa6YnvwCKFNoc0Zc6N2GDs/YOmt73lg/h6iM=; b=SoUpAxvskjoyaX7eNBmw3dEdfocHbxnCM5pILpvm2wqvM7tDppmvyjc/EURkM1Ih/1 okBbBCrVCZnnIKEix/1HR64wnXm1byEGijGsFr0hmcDiXF9F6QQ/cecQneD6zE3nxE2u 9BgYT/uIURxXRgEJMETejuKzjku6XGJkR9GF4eMs5oeyREXtMUdPhZr4u5PkP7Jd4z7k 6lgf+X+6zmwc7O9kUI7iCAkOZCpvnmNRiNc+s6c4jONyuq8Idlh2OE7432A3Uap1FQNo KJqPTqNN1xzZI1mljmOlsm6nqG+Mx1t1e692ssWtfYirLmsPThDaNhSuUDrC4bTx3brO Lg8g== X-Gm-Message-State: AOAM530tW8s20zuEC8CIK46QFJzF2dCp8ZZgG1FlD3F7ggt/bAKyLNov lr0wdVpDWgWR87M16zv7YlEA38DTyS+ilsqtf/ApeQ== X-Received: by 2002:adf:eec2:: with SMTP id a2mr1532933wrp.128.1604652772606; Fri, 06 Nov 2020 00:52:52 -0800 (PST) MIME-Version: 1.0 References: <20201104063713.3039035-1-anup.patel@wdc.com> In-Reply-To: From: Anup Patel Date: Fri, 6 Nov 2020 14:22:40 +0530 Message-ID: Subject: Re: [PATCH] RISC-V: Use non-PGD mappings for early DTB access To: Palmer Dabbelt Cc: Anup Patel , Paul Walmsley , Albert Ou , Atish Patra , Alistair Francis , linux-riscv , "linux-kernel@vger.kernel.org List" , Bin Meng , Bin Meng Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Nov 6, 2020 at 1:30 PM Palmer Dabbelt wrote: > > On Tue, 03 Nov 2020 22:37:13 PST (-0800), Anup Patel wrote: > > Currently, we use PGD mappings for early DTB mapping in early_pgd > > but this breaks Linux kernel on SiFive Unleashed because on SiFive > > Unleashed PMP checks don't work correctly for PGD mappings. > > > > To fix early DTB mappings on SiFive Unleashed, we use non-PGD > > mappings (i.e. PMD) for early DTB access. > > > > Fixes: 8f3a2b4a96dc ("RISC-V: Move DT mapping outof fixmap") > > Signed-off-by: Anup Patel > > --- > > arch/riscv/mm/init.c | 14 ++++++++++++++ > > 1 file changed, 14 insertions(+) > > > > diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c > > index ea933b789a88..0d13d0c36a7d 100644 > > --- a/arch/riscv/mm/init.c > > +++ b/arch/riscv/mm/init.c > > @@ -297,6 +297,7 @@ pmd_t fixmap_pmd[PTRS_PER_PMD] __page_aligned_bss; > > #define NUM_EARLY_PMDS (1UL + MAX_EARLY_MAPPING_SIZE / PGDIR_SIZE) > > #endif > > pmd_t early_pmd[PTRS_PER_PMD * NUM_EARLY_PMDS] __initdata __aligned(PAGE_SIZE); > > +pmd_t early_dtb_pmd[PTRS_PER_PMD] __initdata __aligned(PAGE_SIZE); > > > > static pmd_t *__init get_pmd_virt_early(phys_addr_t pa) > > { > > @@ -494,6 +495,18 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) > > load_pa + (va - PAGE_OFFSET), > > map_size, PAGE_KERNEL_EXEC); > > > > +#ifndef __PAGETABLE_PMD_FOLDED > > + /* Setup early PMD for DTB */ > > + create_pgd_mapping(early_pg_dir, DTB_EARLY_BASE_VA, > > + (uintptr_t)early_dtb_pmd, PGDIR_SIZE, PAGE_TABLE); > > + /* Create two consecutive PMD mappings for FDT early scan */ > > + pa = dtb_pa & ~(PMD_SIZE - 1); > > + create_pmd_mapping(early_dtb_pmd, DTB_EARLY_BASE_VA, > > + pa, PMD_SIZE, PAGE_KERNEL); > > + create_pmd_mapping(early_dtb_pmd, DTB_EARLY_BASE_VA + PMD_SIZE, > > + pa + PMD_SIZE, PMD_SIZE, PAGE_KERNEL); > > + dtb_early_va = (void *)DTB_EARLY_BASE_VA + (dtb_pa & (PMD_SIZE - 1)); > > +#else > > /* Create two consecutive PGD mappings for FDT early scan */ > > pa = dtb_pa & ~(PGDIR_SIZE - 1); > > create_pgd_mapping(early_pg_dir, DTB_EARLY_BASE_VA, > > @@ -501,6 +514,7 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) > > create_pgd_mapping(early_pg_dir, DTB_EARLY_BASE_VA + PGDIR_SIZE, > > pa + PGDIR_SIZE, PGDIR_SIZE, PAGE_KERNEL); > > dtb_early_va = (void *)DTB_EARLY_BASE_VA + (dtb_pa & (PGDIR_SIZE - 1)); > > +#endif > > dtb_early_pa = dtb_pa; > > > > /* > > We're starting to build up a handful of workarounds for these sorts of things. > The PMP trap vs WARL one was the last I could remember, but that's a bit > different as both of those behaviors were allowed by specifications at some > point. IIRC there were also some TLB shootdown issues floating > > The best I can come up with is to add both some sort of "minimum support > specification version" Kconfig entry and an "quirks" set of Kconfig entries. > That would allow us to gradually jettison old ISAs as well as more cleanly add > support for broken hardware like this. > > Do you have a pointer to some datasheet type document that describes the issue? Unfortunately, we still don't know where all SiFive erratums are documented. Maybe you know the right person who can publish this ?? The PMP checks not working correctly for PGD-mappings was discovered independently by Bin Meng on some other OS (almost 1.5 years back). Few discussions about this erratum have happened on OpenSBI mailing list as well although in a very different context. In fact, Andrew had confirmed about this erratum in sw-dev google groups. (Refer, https://groups.google.com/a/groups.riscv.org/g/sw-dev/c/CfyT5Los5uA/m/yY0ES0dZBAAJ?pli=1) (Refer, https://github.com/riscv/opensbi/issues/103) Till now we have encountered two erratums for SiFive U54 cores: 1. SFENCE.VMA not working with "virtual address" parameter 2. PMP checks not working for PGD mappings The Microchip ICICLE kit has U54 cores but the PMP checks issue is only seen on SiFive Unleashed so I guess this issue is fixed in-between but we can't be sure without detailed documentation. > That'd probably be the line I'd like to draw for adding workarounds like this, > as otherwise we can't really be sure something is a hardware issue. > > That said, it's better to have the fix so this is on fixes. I'll try to > remember it for the list of workarounds. I agree. At the moment, 5.10-rc2 does not boot on SiFive Unleashed but it boots on Microchip icicle kit and everywhere else because of the PMP checks issue. Regards, Anup