Received: by 2002:a05:6a10:5bc5:0:0:0:0 with SMTP id os5csp873338pxb; Thu, 21 Oct 2021 11:05:48 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwnnQmPO6031fAg6poAok+3E4zESsyEFfiUtPt9M2TW6TRf1Fd3+9FQfDc4rvQZIWcK+VMw X-Received: by 2002:a17:90b:3ec6:: with SMTP id rm6mr8457563pjb.27.1634839548312; Thu, 21 Oct 2021 11:05:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1634839548; cv=none; d=google.com; s=arc-20160816; b=c3iq9YOn8jIjznz5IOpSyW2DqTXZlaR1FXko7PzY1sfNUjaNUgnXeXUSdd79IdwT2f ufcIgdxM6kpaekM1ewkweKQwebw/+eGnYFFfcDqXOftqHHeIWEvMRBQxCqXczx1yWpSf UH0/OCFpfnjfqYNrVEtGpMmuPiGDZ8NGUIS44OC6RvSEsw79bwYaOE4H/L4lDodyEd+c +A1ghUmFnAxvghJs8dK6xE17yLBi2pDz4OQN2O0IZSYjb7EzeJosRgNlyV+rvETtpi0+ jssggxZH/F2DpWPSa2xXI0+ZJvsEhp358mz8OopFyZrEpH/PpoooHqmlMBRZvP/GxGBU pj9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=26UDl2RXZ5EOKjKS8pS+RO7G+YIRKVwILwE+momL6+o=; b=mfozCtvlbgPeiEiTqAa2ARveMIWQ9UHdkeUwEzsC6GT+5BJpSNfWARiefpTrqYZtSN y+xg0iglD6w7cIEWYiOVrNKKazHLS1XjRCFfFX/Z9D3cJk7stDL59Cx5ePJgQOeHL0lO TihVtqPuXsYbUve6tBd0fzzHN44HdeOQUyrG3w01GgDGixFNPt1lPn3J1CBej9SmI4gp EUidDkGN0gHHlDsiho1n8rMpWB7qPT1Pgo6Gncfzr7tic+b/ESTHuniBvMeYCxFJb082 MkYKaVafKUPBg0CsT1McSkz1QAANGFEecqEaAA4+jCY3yu1RZA3l/meH+6mYsFmMVMps 9Qew== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id c21si7792755pls.149.2021.10.21.11.05.25; Thu, 21 Oct 2021 11:05:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232554AbhJUSF7 (ORCPT + 99 others); Thu, 21 Oct 2021 14:05:59 -0400 Received: from foss.arm.com ([217.140.110.172]:46118 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232575AbhJUSFm (ORCPT ); Thu, 21 Oct 2021 14:05:42 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 933421063; Thu, 21 Oct 2021 11:03:26 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B5A143F70D; Thu, 21 Oct 2021 11:03:23 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: aou@eecs.berkeley.edu, catalin.marinas@arm.com, deanbo422@gmail.com, green.hu@gmail.com, guoren@kernel.org, jonas@southpole.se, kernelfans@gmail.com, linux-arm-kernel@lists.infradead.org, linux@armlinux.org.uk, mark.rutland@arm.com, maz@kernel.org, nickhu@andestech.com, palmer@dabbelt.com, paulmck@kernel.org, paul.walmsley@sifive.com, peterz@infradead.org, shorne@gmail.com, stefan.kristiansson@saunalahti.fi, tglx@linutronix.de, torvalds@linux-foundation.org, tsbogend@alpha.franken.de, vgupta@kernel.org, will@kernel.org Subject: [PATCH 10/15] irq: arm64: perform irqentry in entry code Date: Thu, 21 Oct 2021 19:02:31 +0100 Message-Id: <20211021180236.37428-11-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20211021180236.37428-1-mark.rutland@arm.com> References: <20211021180236.37428-1-mark.rutland@arm.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation for removing HANDLE_DOMAIN_IRQ_IRQENTRY, have arch/arm64 perform all the irqentry accounting in its entry code. As arch/arm64 already performs portions of the irqentry logic in enter_from_kernel_mode() and exit_to_kernel_mode(), including rcu_irq_{enter,exit}(), the only additional calls that need to be made are to irq_{enter,exit}_rcu(). Removing the calls to rcu_irq_{enter,exit}() from handle_domain_irq() ensures that we inform RCU once per IRQ entry and will correctly identify quiescent periods. Since we should not call irq_{enter,exit}_rcu() when entering a pseudo-NMI, el1_interrupt() is reworked to have separate __el1_irq() and __el1_pnmi() paths for regular IRQ and psuedo-NMI entry, with irq_{enter,exit}_irq() only called for the former. In preparation for removing HANDLE_DOMAIN_IRQ, the irq regs are managed in do_interrupt_handler() for both regular IRQ and pseudo-NMI. This is currently redundant, but not harmful. For clarity the preemption logic is moved into __el1_irq(). We should never preempt within a pseudo-NMI, and arm64_enter_nmi() already enforces this by incrementing the preempt_count, but it's clearer if we never invoke the preemption logic when entering a pseudo-NMI. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Marc Zyngier Cc: Thomas Gleixner Cc: Will Deacon --- arch/arm64/Kconfig | 1 - arch/arm64/kernel/entry-common.c | 52 ++++++++++++++++++++++++---------------- 2 files changed, 31 insertions(+), 22 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 553239a5a5f7..5c7ae4c3954b 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -134,7 +134,6 @@ config ARM64 select GENERIC_GETTIMEOFDAY select GENERIC_VDSO_TIME_NS select HANDLE_DOMAIN_IRQ - select HANDLE_DOMAIN_IRQ_IRQENTRY select HARDIRQS_SW_RESEND select HAVE_MOVE_PMD select HAVE_MOVE_PUD diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 32f9796c4ffe..f7408edf8571 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -219,22 +220,6 @@ static void noinstr arm64_exit_el1_dbg(struct pt_regs *regs) lockdep_hardirqs_on(CALLER_ADDR0); } -static void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs) -{ - if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs)) - arm64_enter_nmi(regs); - else - enter_from_kernel_mode(regs); -} - -static void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs) -{ - if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs)) - arm64_exit_nmi(regs); - else - exit_to_kernel_mode(regs); -} - static void __sched arm64_preempt_schedule_irq(void) { lockdep_assert_irqs_disabled(); @@ -263,10 +248,14 @@ static void __sched arm64_preempt_schedule_irq(void) static void do_interrupt_handler(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { + struct pt_regs *old_regs = set_irq_regs(regs); + if (on_thread_stack()) call_on_irq_stack(regs, handler); else handler(regs); + + set_irq_regs(old_regs); } extern void (*handle_arch_irq)(struct pt_regs *); @@ -432,13 +421,22 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs) } } -static void noinstr el1_interrupt(struct pt_regs *regs, - void (*handler)(struct pt_regs *)) +static __always_inline void __el1_pnmi(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) { - write_sysreg(DAIF_PROCCTX_NOIRQ, daif); + arm64_enter_nmi(regs); + do_interrupt_handler(regs, handler); + arm64_exit_nmi(regs); +} + +static __always_inline void __el1_irq(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) +{ + enter_from_kernel_mode(regs); - enter_el1_irq_or_nmi(regs); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); /* * Note: thread_info::preempt_count includes both thread_info::count @@ -449,7 +447,17 @@ static void noinstr el1_interrupt(struct pt_regs *regs, READ_ONCE(current_thread_info()->preempt_count) == 0) arm64_preempt_schedule_irq(); - exit_el1_irq_or_nmi(regs); + exit_to_kernel_mode(regs); +} +static void noinstr el1_interrupt(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) +{ + write_sysreg(DAIF_PROCCTX_NOIRQ, daif); + + if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs)) + __el1_pnmi(regs, handler); + else + __el1_irq(regs, handler); } asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs) @@ -667,7 +675,9 @@ static void noinstr el0_interrupt(struct pt_regs *regs, if (regs->pc & BIT(55)) arm64_apply_bp_hardening(); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); exit_to_user_mode(regs); } -- 2.11.0