Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751597AbbLJKBt (ORCPT ); Thu, 10 Dec 2015 05:01:49 -0500 Received: from foss.arm.com ([217.140.101.70]:54148 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750865AbbLJKBp (ORCPT ); Thu, 10 Dec 2015 05:01:45 -0500 Message-ID: <56694C0E.5050707@arm.com> Date: Thu, 10 Dec 2015 09:55:26 +0000 From: Marc Zyngier Organization: ARM Ltd User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Icedove/31.7.0 MIME-Version: 1.0 To: Vineet Gupta , Daniel Lezcano , Jason Cooper CC: Thomas Gleixner , arcml , lkml , Peter Zijlstra Subject: Re: percpu irq APIs and perf References: In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3291 Lines: 85 Hi Vinnet, On 10/12/15 09:25, Vineet Gupta wrote: > Hi Marc / Daniel / Jason, > > I had a couple of questions about percpu irq API, hopefully you can help answer. > > On ARM, how do u handle requesting per cpu IRQs - specifically usage > of request_percpu_irq() / enable_percpu_irq() API. > It seems, for using them, we obviously need to explicitly set irq as > percpu and as a consequence explicitly enable autoen (since former > disables that). See arch/arc/kernel/irq.c: arc_request_percpu_irq() > called by ARC per cpu timer setup. Indeed. The interrupt controller code flags these interrupts as being per-cpu, and we do rely on each CPU performing an enable_percpu_irq(). So the way the whole thing flows is as such: - Interrupt controller (GIC) flags the PPIs (Private Peripheral Interrupt) as per-CPU (hwirq 16 to 31 are replicated per CPU) very early in the boot process - request_percpu_irq() only occurs once, usually on the boot CPU (but that's not a requirement) - each CPU executes enable_percpu_irq(), which touches per-CPU registers. This usually involves a CPU notifier to enable/disable the interrupt when hotplug is on. > if (!cpu) { > irq_set_percpu_devid() <--- disables AUTOEN > irq_modify_status(IRQ_NOAUTOEN) <-- to undo side-effect of above > request_percpu_irq > } > enable_percpu_irq > > I don't see pattern in general for drivers/clocksource/ and/or > arm_arch_timer.c for PPI case. You can have a look at arch/arm/smp/smp_twd.c which is probably less cryptic. > Further there is an ordering requirement as in request_percpu_irq() > needs to be called only for the first calling core, and > enable_percpu_irq() on each one. If enable is done ahead of request > it obviously fails. Yup. > For ARC I've historically used a wrapper arc_request_percpu_irq() > [pseudo code above] - which has an inherent assumption (now realize > fragile) that it will be called on core0 first thus guaranteeing the > ordering above. This is true for timer, IPI etc but not for other > late probed peripherals - specially perf. > > Infact ARC perf probe open codes on_each_cpu() to ensure irq request > is done locally first. > > But this all falls apart, when perf probe happens on coreX (not > core0), causing enable to be called ahead of request anyways. This is > what I'm running into now. > > I think the solution is to call request_percpu_irq() on whatever core > hits first and call enable_percpu_irq() from a cpu up notifier. But I > think the notifier won't run on boot cpu ? Or is there a better way > to clean up all this mess. I think that's pretty much it. See drivers/perf/arm_pmu.c::cpu_pmu_request_irq() for example. > FWIW, I see this issue on 3.18 kernel but not latest 4.4-rcX because > in 3.18 arc perf probe invariably happens on coreX (due to init task > migration right after clocksource switch - something which doesn't > happen on 4.4 likely due to recent timer core changes). Hope this helps, M. -- Jazz is not dead. It just smells funny... -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/