Received: by 2002:a05:6358:d09b:b0:dc:cd0c:909e with SMTP id jc27csp1810600rwb; Fri, 11 Nov 2022 00:11:46 -0800 (PST) X-Google-Smtp-Source: AA0mqf6bGG/GwyhBo37aaBStlVm4AbMUTXmlA5ym3q1nGBmx0n+w2NomxTgCczIQGTCs+I4Rvyo0 X-Received: by 2002:a17:902:e84a:b0:186:7b95:f773 with SMTP id t10-20020a170902e84a00b001867b95f773mr1437369plg.152.1668154306558; Fri, 11 Nov 2022 00:11:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668154306; cv=none; d=google.com; s=arc-20160816; b=YAE17KeSPlF46rJLmoyyg8YemdiWBtMmWp5+Nv1tK936WVoPfC80Ceg7B65LRU350f 6QXUdqsbZPqOnQCwtGbLwHv7CTHV13o2EgOF4KuUwNSd/6gVsadHOagkrNuI3v5+MxtZ obQThiJ6x7hHa0/DffA1zhFZ+uAdMJTzEk7ShXbSbxUv2mc57cheu3NViqeHl2YdRGqC v+R44bCRxPJ1EtUkjRkee+J9a9ygifu2dX9j06SfWCtbjYkBkdw17wg7NdxzDE9Xt3Ya yUAJcW2+U82vwRHD8/q04k9mV1ncWtzpT/bxHp4tuY9UTT828dLV59pjW6e4/gbf8Bxq ojpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=k/4nQL0TR/hVIZLQspnDL/8PNOIAe2T5HzPC+CIyjTc=; b=fsCHWLy9Bvps/n0AiqMsESAnvW2ejUHwCO/THEcF64F/szctEOyOrvFc2jLMqGLYHD Q1FdrrFY8mHd8E83QfUOcPo9jXj9v7Oo6b7uqGFuXkGL9vYEPpOmH+Mc1g4utnYEJELn C4z9MkW2hP+1+oi/yvDYamneE2dka7w4p6qRo93JQpCw1M7f0ESLNTozjcSD2GWtbJhB q3lGs0qnBU7lNpGk1vs4IbMaYEwRR/ITnhryhR27CMZWgnZS/t8xB8BCHltBpw+jzux5 vplXyCHhG44pQU7Dd8AllidSMbqDi7RJd3hzhKi1DBkHTxQ83dOzCgqzg4EDDDlwl3C1 V8HA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=bjKMK8gR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l1-20020a170902ec0100b00186f1a1fb83si1676469pld.352.2022.11.11.00.11.33; Fri, 11 Nov 2022 00:11:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=bjKMK8gR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232855AbiKKH70 (ORCPT + 93 others); Fri, 11 Nov 2022 02:59:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232574AbiKKH7Y (ORCPT ); Fri, 11 Nov 2022 02:59:24 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A558C45A3B for ; Thu, 10 Nov 2022 23:59:23 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id F158FCE257B for ; Fri, 11 Nov 2022 07:59:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 62CDAC433D6; Fri, 11 Nov 2022 07:59:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668153560; bh=7YM4Ka0ebCuIEuz50vxHPcJkntgQCHzI1KnrlWMijZQ=; h=From:To:Cc:Subject:Date:From; b=bjKMK8gRmG1/YFfOrdCMYD4UqRsJ0z9YnF9AnNT90pFjQrCMqnd1FgjRxql1JeKX1 f1r1SlN0S0fPWiJnDgrXXPboYMK2OLqIaAO0+d9cl9Wk05qISLZjLf4IlHFP/fRNsU 4NvqEDnRiV8tTlAjqGJ/XfhXPKAH5U2Rdw6wDUmRbajXIX2DqIpE7k48zKLZrzWxYa ZNgU1Qw5/Sug37W0rZbq2wOTMLvOkbY5vi98CV+cbiSCs6+seywwaDeHoc64ZjvGOF /UW9GRmBJZbLY/0jUkj7lnrliWZCe/460qYO1tP2fMZ0rTMucpgED3PK5idBIjtyv4 JGhcG/GPH4HCg== From: guoren@kernel.org To: anup@brainfault.org, paul.walmsley@sifive.com, palmer@dabbelt.com, conor.dooley@microchip.com, heiko@sntech.de, philipp.tomsich@vrull.eu, alex@ghiti.fr, hch@lst.de, ajones@ventanamicro.com, gary@garyguo.net, jszhang@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Guo Ren , Guo Ren , Anup Patel , Palmer Dabbelt Subject: [PATCH V3] riscv: asid: Fixup stale TLB entry cause application crash Date: Fri, 11 Nov 2022 02:59:02 -0500 Message-Id: <20221111075902.798571-1-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Guo Ren After use_asid_allocator is enabled, the userspace application will crash by stale TLB entries. Because only using cpumask_clear_cpu without local_flush_tlb_all couldn't guarantee CPU's TLB entries were fresh. Then set_mm_asid would cause the user space application to get a stale value by stale TLB entry, but set_mm_noasid is okay. Here is the symptom of the bug: unhandled signal 11 code 0x1 (coredump) 0x0000003fd6d22524 <+4>: auipc s0,0x70 0x0000003fd6d22528 <+8>: ld s0,-148(s0) # 0x3fd6d92490 => 0x0000003fd6d2252c <+12>: ld a5,0(s0) (gdb) i r s0 s0 0x8082ed1cc3198b21 0x8082ed1cc3198b21 (gdb) x /2x 0x3fd6d92490 0x3fd6d92490: 0xd80ac8a8 0x0000003f The core dump file shows that register s0 is wrong, but the value in memory is correct. Because 'ld s0, -148(s0)' used a stale mapping entry in TLB and got a wrong result from an incorrect physical address. When the task ran on CPU0, which loaded/speculative-loaded the value of address(0x3fd6d92490), then the first version of the mapping entry was PTWed into CPU0's TLB. When the task switched from CPU0 to CPU1 (No local_tlb_flush_all here by asid), it happened to write a value on the address (0x3fd6d92490). It caused do_page_fault -> wp_page_copy -> ptep_clear_flush -> ptep_get_and_clear & flush_tlb_page. The flush_tlb_page used mm_cpumask(mm) to determine which CPUs need TLB flush, but CPU0 had cleared the CPU0's mm_cpumask in the previous switch_mm. So we only flushed the CPU1 TLB and set the second version mapping of the PTE. When the task switched from CPU1 to CPU0 again, CPU0 still used a stale TLB mapping entry which contained a wrong target physical address. It raised a bug when the task happened to read that value. CPU0 CPU1 - switch 'task' in - read addr (Fill stale mapping entry into TLB) - switch 'task' out (no tlb_flush) - switch 'task' in (no tlb_flush) - write addr cause pagefault do_page_fault() (change to new addr mapping) wp_page_copy() ptep_clear_flush() ptep_get_and_clear() & flush_tlb_page() write new value into addr - switch 'task' out (no tlb_flush) - switch 'task' in (no tlb_flush) - read addr again (Use stale mapping entry in TLB) get wrong value from old phyical addr, BUG! The solution is to keep all CPUs' footmarks of cpumask(mm) in switch_mm, which could guarantee to invalidate all stale TLB entries during TLB flush. Fixes: 65d4b9c53017 ("RISC-V: Implement ASID allocator") Signed-off-by: Guo Ren Signed-off-by: Guo Ren Cc: Anup Patel Cc: Palmer Dabbelt --- Changes in v3: - Move set/clear cpumask(mm) into set_mm (Make code more pretty with Andrew's advice) - Optimize comment description Changes in v2: - Fixup nommu compile problem (Thx Conor, Also Reported-by: kernel test robot ) - Keep cpumask_clear_cpu for noasid --- arch/riscv/mm/context.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 7acbfbd14557..0f784e3d307b 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -205,12 +205,24 @@ static void set_mm_noasid(struct mm_struct *mm) local_flush_tlb_all(); } -static inline void set_mm(struct mm_struct *mm, unsigned int cpu) +static inline void set_mm(struct mm_struct *prev, + struct mm_struct *next, unsigned int cpu) { - if (static_branch_unlikely(&use_asid_allocator)) - set_mm_asid(mm, cpu); - else - set_mm_noasid(mm); + /* + * The mm_cpumask indicates which harts' TLBs contain the virtual + * address mapping of the mm. Compared to noasid, using asid + * can't guarantee that stale TLB entries are invalidated because + * the asid mechanism wouldn't flush TLB for every switch_mm for + * performance. So when using asid, keep all CPUs footmarks in + * cpumask() until mm reset. + */ + cpumask_set_cpu(cpu, mm_cpumask(next)); + if (static_branch_unlikely(&use_asid_allocator)) { + set_mm_asid(next, cpu); + } else { + cpumask_clear_cpu(cpu, mm_cpumask(prev)); + set_mm_noasid(next); + } } static int __init asids_init(void) @@ -264,7 +276,8 @@ static int __init asids_init(void) } early_initcall(asids_init); #else -static inline void set_mm(struct mm_struct *mm, unsigned int cpu) +static inline void set_mm(struct mm_struct *prev, + struct mm_struct *next, unsigned int cpu) { /* Nothing to do here when there is no MMU */ } @@ -317,10 +330,7 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next, */ cpu = smp_processor_id(); - cpumask_clear_cpu(cpu, mm_cpumask(prev)); - cpumask_set_cpu(cpu, mm_cpumask(next)); - - set_mm(next, cpu); + set_mm(prev, next, cpu); flush_icache_deferred(next, cpu); } -- 2.36.1