Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S964818AbbEHR71 (ORCPT ); Fri, 8 May 2015 13:59:27 -0400 Received: from mail-am1on0074.outbound.protection.outlook.com ([157.56.112.74]:7744 "EHLO emea01-am1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932501AbbEHR7S (ORCPT ); Fri, 8 May 2015 13:59:18 -0400 Authentication-Results: spf=fail (sender IP is 12.216.194.146) smtp.mailfrom=ezchip.com; ezchip.com; dkim=none (message not signed) header.d=none; From: Chris Metcalf To: Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , Peter Zijlstra , Andrew Morton , "Rik van Riel" , Tejun Heo , Thomas Gleixner , Frederic Weisbecker , "Paul E. McKenney" , Christoph Lameter , "Srivatsa S. Bhat" , , CC: Chris Metcalf Subject: [PATCH 6/6] nohz: add dataplane_debug boot flag Date: Fri, 8 May 2015 13:58:47 -0400 Message-ID: <1431107927-13998-7-git-send-email-cmetcalf@ezchip.com> X-Mailer: git-send-email 2.1.2 In-Reply-To: <1431107927-13998-1-git-send-email-cmetcalf@ezchip.com> References: <1431107927-13998-1-git-send-email-cmetcalf@ezchip.com> X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:12.216.194.146;CTRY:US;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(339900001)(189002)(199003)(50986999)(2950100001)(76176999)(92566002)(6806004)(106466001)(42186005)(47776003)(36756003)(48376002)(33646002)(50466002)(105606002)(5001770100001)(46102003)(2201001)(5001920100001)(107886002)(5001960100002)(575784001)(86362001)(19580395003)(62966003)(19580405001)(104016003)(229853001)(85426001)(77156002)(50226001)(189998001)(87936001)(921003)(4001430100001)(1121003);DIR:OUT;SFP:1101;SCL:1;SRVR:AM2PR02MB0420;H:ld-1.internal.tilera.com;FPR:;SPF:Fail;MLV:sfv;MX:1;A:1;LANG:en; MIME-Version: 1.0 Content-Type: text/plain X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AM2PR02MB0420; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(5005006)(3002001);SRVR:AM2PR02MB0420;BCL:0;PCL:0;RULEID:;SRVR:AM2PR02MB0420; X-Forefront-PRVS: 0570F1F193 X-OriginatorOrg: ezchip.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2015 17:59:13.6904 (UTC) X-MS-Exchange-CrossTenant-Id: 0fc16e0a-3cd3-4092-8b2f-0a42cff122c3 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=0fc16e0a-3cd3-4092-8b2f-0a42cff122c3;Ip=[12.216.194.146];Helo=[ld-1.internal.tilera.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM2PR02MB0420 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 6947 Lines: 197 This flag simplifies debugging of NO_HZ_FULL kernels when processes are running in PR_DATAPLANE_QUIESCE mode. Such processes should get no interrupts from the kernel, and if they do, when this boot flag is specified a kernel stack dump on the console is generated. It's possible to use ftrace to simply detect whether a dataplane core has unexpectedly entered the kernel. But what this boot flag does is allow the kernel to provide better diagnostics, e.g. by reporting in the IPI-generating code what remote core and context is preparing to deliver an interrupt to a dataplane core. It may be worth considering other ways to generate useful debugging output rather than console spew, but for now that is simple and direct. Signed-off-by: Chris Metcalf --- Documentation/kernel-parameters.txt | 6 ++++++ arch/tile/mm/homecache.c | 5 ++++- include/linux/tick.h | 2 ++ kernel/irq_work.c | 4 +++- kernel/sched/core.c | 18 ++++++++++++++++++ kernel/signal.c | 5 +++++ kernel/smp.c | 4 ++++ kernel/softirq.c | 1 + 8 files changed, 43 insertions(+), 2 deletions(-) diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index f6befa9855c1..5c5af5258e17 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -794,6 +794,12 @@ bytes respectively. Such letter suffixes can also be entirely omitted. dasd= [HW,NET] See header of drivers/s390/block/dasd_devmap.c. + dataplane_debug [KNL] + In kernels built with CONFIG_NO_HZ_FULL and booted + in nohz_full= mode, this setting will generate console + backtraces when the kernel is about to interrupt a + task that has requested PR_DATAPLANE_QUIESCE. + db9.dev[2|3]= [HW,JOY] Multisystem joystick support via parallel port (one device per port) Format: , diff --git a/arch/tile/mm/homecache.c b/arch/tile/mm/homecache.c index 40ca30a9fee3..dd5ec7eca9a8 100644 --- a/arch/tile/mm/homecache.c +++ b/arch/tile/mm/homecache.c @@ -31,6 +31,7 @@ #include #include #include +#include #include #include @@ -83,8 +84,10 @@ static void hv_flush_update(const struct cpumask *cache_cpumask, * Don't bother to update atomically; losing a count * here is not that critical. */ - for_each_cpu(cpu, &mask) + for_each_cpu(cpu, &mask) { ++per_cpu(irq_stat, cpu).irq_hv_flush_count; + tick_nohz_dataplane_debug(cpu); + } } /* diff --git a/include/linux/tick.h b/include/linux/tick.h index d191cda9b71a..4610cdf0f972 100644 --- a/include/linux/tick.h +++ b/include/linux/tick.h @@ -147,6 +147,7 @@ extern void tick_nohz_full_kick_cpu(int cpu); extern void tick_nohz_full_kick_all(void); extern void __tick_nohz_task_switch(struct task_struct *tsk); extern void tick_nohz_dataplane_enter(void); +extern void tick_nohz_dataplane_debug(int cpu); #else static inline bool tick_nohz_full_enabled(void) { return false; } static inline bool tick_nohz_full_cpu(int cpu) { return false; } @@ -157,6 +158,7 @@ static inline void tick_nohz_full_kick_all(void) { } static inline void __tick_nohz_task_switch(struct task_struct *tsk) { } static inline bool tick_nohz_is_dataplane(void) { return false; } static inline void tick_nohz_dataplane_enter(void) { } +static inline void tick_nohz_dataplane_debug(int cpu) { } #endif static inline bool is_housekeeping_cpu(int cpu) diff --git a/kernel/irq_work.c b/kernel/irq_work.c index cbf9fb899d92..0adc53c4e899 100644 --- a/kernel/irq_work.c +++ b/kernel/irq_work.c @@ -75,8 +75,10 @@ bool irq_work_queue_on(struct irq_work *work, int cpu) if (!irq_work_claim(work)) return false; - if (llist_add(&work->llnode, &per_cpu(raised_list, cpu))) + if (llist_add(&work->llnode, &per_cpu(raised_list, cpu))) { + tick_nohz_dataplane_debug(cpu); arch_send_call_function_single_ipi(cpu); + } return true; } diff --git a/kernel/sched/core.c b/kernel/sched/core.c index f9123a82cbb6..202fab0c41cb 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -719,6 +719,24 @@ bool sched_can_stop_tick(void) return true; } + +/* Enable debugging of any interrupts of dataplane cores. */ +static int dataplane_debug; +static int __init dataplane_debug_func(char *str) +{ + dataplane_debug = true; + return 1; +} +__setup("dataplane_debug", dataplane_debug_func); + +void tick_nohz_dataplane_debug(int cpu) +{ + if (dataplane_debug && tick_nohz_full_cpu(cpu) && + (cpu_curr(cpu)->dataplane_flags & PR_DATAPLANE_QUIESCE)) { + pr_err("Interrupt detected for dataplane cpu %d\n", cpu); + dump_stack(); + } +} #endif /* CONFIG_NO_HZ_FULL */ void sched_avg_update(struct rq *rq) diff --git a/kernel/signal.c b/kernel/signal.c index d51c5ddd855c..ebc552cafff5 100644 --- a/kernel/signal.c +++ b/kernel/signal.c @@ -689,6 +689,11 @@ int dequeue_signal(struct task_struct *tsk, sigset_t *mask, siginfo_t *info) */ void signal_wake_up_state(struct task_struct *t, unsigned int state) { +#ifdef CONFIG_NO_HZ_FULL + /* If the task is being killed, don't complain about dataplane. */ + if (state & TASK_WAKEKILL) + t->dataplane_flags = 0; +#endif set_tsk_thread_flag(t, TIF_SIGPENDING); /* * TASK_WAKEKILL also means wake it up in the stopped/traced/killable diff --git a/kernel/smp.c b/kernel/smp.c index 07854477c164..9518fc80321b 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -14,6 +14,7 @@ #include #include #include +#include #include "smpboot.h" @@ -178,6 +179,7 @@ static int generic_exec_single(int cpu, struct call_single_data *csd, * locking and barrier primitives. Generic code isn't really * equipped to do the right thing... */ + tick_nohz_dataplane_debug(cpu); if (llist_add(&csd->llist, &per_cpu(call_single_queue, cpu))) arch_send_call_function_single_ipi(cpu); @@ -457,6 +459,8 @@ void smp_call_function_many(const struct cpumask *mask, } /* Send a message to all CPUs in the map */ + for_each_cpu(cpu, cfd->cpumask) + tick_nohz_dataplane_debug(cpu); arch_send_call_function_ipi_mask(cfd->cpumask); if (wait) { diff --git a/kernel/softirq.c b/kernel/softirq.c index bc9406337f82..eeacabf08ca6 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -394,6 +394,7 @@ void irq_exit(void) WARN_ON_ONCE(!irqs_disabled()); #endif + tick_nohz_dataplane_debug(smp_processor_id()); account_irq_exit_time(current); preempt_count_sub(HARDIRQ_OFFSET); if (!in_interrupt() && local_softirq_pending()) -- 2.1.2 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/