Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754045AbdHIQnA (ORCPT ); Wed, 9 Aug 2017 12:43:00 -0400 Received: from merlin.infradead.org ([205.233.59.134]:55522 "EHLO merlin.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753710AbdHIQm5 (ORCPT ); Wed, 9 Aug 2017 12:42:57 -0400 Date: Wed, 9 Aug 2017 18:42:50 +0200 From: Peter Zijlstra To: Yafang Shao Cc: mingo@redhat.com, gregkh@linuxfoundation.org, jslaby@suse.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH v3] scheduler: enhancement to show_state_filter and SysRq Message-ID: <20170809164250.aza4csoc2jcwkvx7@hirez.programming.kicks-ass.net> References: <1502274688-2508-1-git-send-email-laoar.shao@gmail.com> <20170809074337.awcwac6u2cxr5h2m@hirez.programming.kicks-ass.net> <20170809090925.lgrfa2clxz7afo3b@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20170609 (1.8.3) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2493 Lines: 54 On Wed, Aug 09, 2017 at 05:26:14PM +0800, Yafang Shao wrote: > 2017-08-09 17:09 GMT+08:00 Peter Zijlstra : > > On Wed, Aug 09, 2017 at 04:01:49PM +0800, Yafang Shao wrote: > >> 2017-08-09 15:43 GMT+08:00 Peter Zijlstra : > >> > On Wed, Aug 09, 2017 at 06:31:28PM +0800, Yafang Shao wrote: > >> >> Sometimes we want to get tasks in TASK_RUNNING sepcifically, > >> >> instead of dump all tasks. > >> >> > >> >> For example, when the loadavg are high, we want to dump > >> >> tasks in TASK_RUNNING and TASK_UNINTERRUPTIBLE, which contribute > >> >> to system load. But mostly there're lots of tasks in Sleep state, > >> >> which occupies almost all of the kernel log buffer, even overflows > >> >> it, that causes the useful messages get lost. Although we can > >> >> enlarge the kernel log buffer, but that's not a good idea. > >> > > >> > That's what you have serial consoles for... > >> > > >> mostly we don't even have one console because we alwayas login the > >> servers via ssh. And manage the servers with console is not so convenient. > > > > I find IPMI SOL very useful. Serial console (esp. earlyprintk) keeps on > > working long after most other things have died. > > > > In any case, you can easily dump the printk output into your ssh session > > if you want, use something like: > > > > cat /dev/kmsg | tee logfile & echo t > /proc/sysrq-trigger > > that's what I'm doing it currently :) > Then I thought deeply why not do it more smartly? > Introducing a new key(here I just modified the key 'w') only dump > tasks in running and blocked should be more smarter. Since you're strictly ssh based, you could maybe do a sysctl that allows changing the 'default' filter of sysctl-t, dunno if that makes sense though. Also, since you're not actually debugging a dead machine, maybe you can do a custom kernel module / systemtap / ebpf thing that collects precisely the information you want. sysrq is typically a last ditch debug mostly dead machine thing, which you're very much not having. > > > > I really see no problem here. Then you can run a bit of awk or whatever > > your favourite tool is to filter out the stuff you don't want. > > > Another question, if we could filter with scritpts in userland, why > did we introduced the key 'w' to dump only blocked state > as we already have a key 't' to dump all tasks ? No idea that is long before my time, I expect because 'w' (blocked) is typically a small number of tasks.