Received: by 2002:a05:6a10:af89:0:0:0:0 with SMTP id iu9csp1528219pxb; Sat, 29 Jan 2022 08:34:04 -0800 (PST) X-Google-Smtp-Source: ABdhPJwUibrK1SL1E9+7MMilpgvB0Rqy5HfvBbRQHvCRplhVb/y9KkCv6Dc8fppR1jzBTfSV+lNB X-Received: by 2002:a05:6402:1617:: with SMTP id f23mr13502285edv.252.1643474043807; Sat, 29 Jan 2022 08:34:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1643474043; cv=none; d=google.com; s=arc-20160816; b=nqI4tl57l/qsViJQezFUagPV0v6r6RA79w30AUc6eV2Bu8myMCsQKvBu9zmZIJAvqs Iac2Qwp0KffuzMmKoFKY0qtRxVg1blwIFWmEOp5n/8DU5jdnbm5JelMnnMRL+Td57qml 1LJ21WSw77qwVmg2H0d2z8NL5m7wF5JbomMOkmTVtAqf02AOcU1vGx3BL3MFXuNeijkU bo6BrFDhgrid8aoGJ/sSpqjYyjG+XHBJ4dI600SXxnBnT7O60tgZ6ZOnYROEL2GQFtI9 1Dw7lmqinkgFsBU6nC710hX/28aHDDfYf87VeEDauEItuicPqRMFMpwTCh1f7ZS8PfD6 Qx4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ANfiyTNuLH3K34lM7cYA66sM88G94rOT/TdSDWy1dfM=; b=QtDDn66kSw8URNWFy/8ARrAbOhtUkoByhLD4D8OnqntJtV3+lGGrGGcYDHLTdCJKTd JQVsi6L9AWIs51daYfcXP6YHCZU0eHjAmf//COsL+/4fSB4WWdN7wjJDA5GBBfjvfj3G 9BstNbCjtVW3i949HUcXyDEqpBVuw9OyG3D2IFhkUhhGkyTHCgNcIrvcxCfX6C7Oiz2D 9HDATEICrnYZkAHBr9JkIDir+MHyshywH0CeQoq8QGXRniPwtKorOJtNU8a5rVtdbWCZ lyey+bvOkhuHb7EQ2DIEZ5QXKwyYu7lAFNNMevrw3KiulMRZZZ3W+ntUJ/nBVDn8BMkH xWWQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ventanamicro.com header.s=google header.b=Crn4O48S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id gt12si5313538ejc.886.2022.01.29.08.33.39; Sat, 29 Jan 2022 08:34:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@ventanamicro.com header.s=google header.b=Crn4O48S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346093AbiA1F0U (ORCPT + 99 others); Fri, 28 Jan 2022 00:26:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346165AbiA1F0N (ORCPT ); Fri, 28 Jan 2022 00:26:13 -0500 Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F472C06173B for ; Thu, 27 Jan 2022 21:26:13 -0800 (PST) Received: by mail-pj1-x1031.google.com with SMTP id g11-20020a17090a7d0b00b001b2c12c7273so6581863pjl.0 for ; Thu, 27 Jan 2022 21:26:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ventanamicro.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ANfiyTNuLH3K34lM7cYA66sM88G94rOT/TdSDWy1dfM=; b=Crn4O48Svbt1cMbrPm7OHVBzSfhmNPVV3P249sWbI5HBVMyMc+pvPClrZ0o6EjfLph HUldc8hL9NTHUzUGhir89D6lA/g3X0JRboyB2QPEmJXXJjNnwE3KStUUpC//gznpTwcg oGD6hC2x3PFQFC7p8cOTkHwT/vZKFLwq8pwaRJTxvL60g2DZ5vOzPw+ITsTMcbWdDyY7 Q9asyWMfNZrc0Ux2X2aXoNvTjtsvKorMS4h2JyVfke43yekRUsuQdvOsQdrZf95PK38j VhCr++061HQ7l0d+zLT7g+jGvWRmAXqo/CQqT/c8h/0I+SxpuVWsUfb4/p2o2VCIgqjN 61ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ANfiyTNuLH3K34lM7cYA66sM88G94rOT/TdSDWy1dfM=; b=utcx6UVHcmIG6Xa1aAuNNRFaMk3CyFdpex4ROYetAa4332eaNKi2SFujbq99hkmOQJ 3S5UwpdO1sWR6iuovDeDksjLym39y3F09lG3TrLCDyj2e7gyr8g3i39lPNVNbRfteZ0I tMOkpZUzDjVE6y6Z0ffJ91fiAKuCxtHpnBk2I4h9XofP24BNa5cWJpQMmr8rzARA58OU SjRK8GRMACB6Y9Cm202LLsRvOVfd4+KsOXJP1KcbWroARL+JuNteDWM+TgpudRpnaVQP kOSHmEibVCwMy158kwB1jFxtIfmdEN86+W9zZOopOhjhVOp1yPCsYM0yll8QlvUpGCrt NlMg== X-Gm-Message-State: AOAM530REJe+730TpZarfqJjzSbw2pO8RJKvC2h4p7dSTEfqKG3xh4uv T34e85ML8tT/CJTt5z3qt6hmYA== X-Received: by 2002:a17:902:e0c9:: with SMTP id e9mr7177806pla.56.1643347573119; Thu, 27 Jan 2022 21:26:13 -0800 (PST) Received: from localhost.localdomain ([122.171.184.231]) by smtp.gmail.com with ESMTPSA id b20sm7731744pfv.134.2022.01.27.21.26.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jan 2022 21:26:10 -0800 (PST) From: Anup Patel To: Palmer Dabbelt , Paul Walmsley , Thomas Gleixner , Marc Zyngier , Daniel Lezcano Cc: Atish Patra , Alistair Francis , Anup Patel , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Anup Patel Subject: [PATCH v2 5/6] RISC-V: Use IPIs for remote TLB flush when possible Date: Fri, 28 Jan 2022 10:55:04 +0530 Message-Id: <20220128052505.859518-6-apatel@ventanamicro.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220128052505.859518-1-apatel@ventanamicro.com> References: <20220128052505.859518-1-apatel@ventanamicro.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org If IPI calls are injected using SBI IPI calls then remote TLB flush using SBI RFENCE calls is much faster because using IPIs for remote TLB flush would still endup as SBI IPI calls with extra processing on kernel side. It is now possible to have specialized hardware (such as RISC-V AIA and RISC-V ACLINT) which allows S-mode software to directly inject IPIs without any assistance from M-mode runtime firmware. This patch extends remote TLB flush functions to use IPIs whenever underlying IPI operations are suitable for remote FENCEs. Signed-off-by: Anup Patel --- arch/riscv/mm/tlbflush.c | 93 +++++++++++++++++++++++++++++++++------- 1 file changed, 78 insertions(+), 15 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 37ed760d007c..27a7db8eb2c4 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -23,14 +23,62 @@ static inline void local_flush_tlb_page_asid(unsigned long addr, : "memory"); } +static inline void local_flush_tlb_range(unsigned long start, + unsigned long size, unsigned long stride) +{ + if (size <= stride) + local_flush_tlb_page(start); + else + local_flush_tlb_all(); +} + +static inline void local_flush_tlb_range_asid(unsigned long start, + unsigned long size, unsigned long stride, unsigned long asid) +{ + if (size <= stride) + local_flush_tlb_page_asid(start, asid); + else + local_flush_tlb_all_asid(asid); +} + +static void __ipi_flush_tlb_all(void *info) +{ + local_flush_tlb_all(); +} + void flush_tlb_all(void) { - sbi_remote_sfence_vma(NULL, 0, -1); + if (riscv_use_ipi_for_rfence()) + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); + else + sbi_remote_sfence_vma(NULL, 0, -1); +} + +struct flush_tlb_range_data { + unsigned long asid; + unsigned long start; + unsigned long size; + unsigned long stride; +}; + +static void __ipi_flush_tlb_range_asid(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); +} + +static void __ipi_flush_tlb_range(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range(d->start, d->size, d->stride); } -static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, - unsigned long size, unsigned long stride) +static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, + unsigned long size, unsigned long stride) { + struct flush_tlb_range_data ftd; struct cpumask *cmask = mm_cpumask(mm); unsigned int cpuid; bool broadcast; @@ -45,19 +93,34 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, unsigned long asid = atomic_long_read(&mm->context.id); if (broadcast) { - sbi_remote_sfence_vma_asid(cmask, start, size, asid); - } else if (size <= stride) { - local_flush_tlb_page_asid(start, asid); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range_asid, + &ftd, 1); + } else + sbi_remote_sfence_vma_asid(cmask, + start, size, asid); } else { - local_flush_tlb_all_asid(asid); + local_flush_tlb_range_asid(start, size, stride, asid); } } else { if (broadcast) { - sbi_remote_sfence_vma(cmask, start, size); - } else if (size <= stride) { - local_flush_tlb_page(start); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = 0; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range, + &ftd, 1); + } else + sbi_remote_sfence_vma(cmask, start, size); } else { - local_flush_tlb_all(); + local_flush_tlb_range(start, size, stride); } } @@ -66,23 +129,23 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, void flush_tlb_mm(struct mm_struct *mm) { - __sbi_tlb_flush_range(mm, 0, -1, PAGE_SIZE); + __flush_tlb_range(mm, 0, -1, PAGE_SIZE); } void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { - __sbi_tlb_flush_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); } void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PMD_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PMD_SIZE); } #endif -- 2.25.1