Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754014AbdFSSwz (ORCPT ); Mon, 19 Jun 2017 14:52:55 -0400 Received: from mail.kernel.org ([198.145.29.99]:48418 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752204AbdFSSwx (ORCPT ); Mon, 19 Jun 2017 14:52:53 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 521FA239D4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=goodmis.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=rostedt@goodmis.org Date: Mon, 19 Jun 2017 14:52:45 -0400 From: Steven Rostedt To: Will Hawkins Cc: LKML Subject: Re: Ftrace vs perf user page fault statistics differences Message-ID: <20170619145245.6966c134@gandalf.local.home> In-Reply-To: References: <20170612212005.1bf8c44d@gandalf.local.home> <20170613090649.2a5edb6a@gandalf.local.home> <20170613160434.605fdef2@gandalf.local.home> <20170613221316.64d46ea5@gandalf.local.home> <20170614160120.577a6385@gandalf.local.home> X-Mailer: Claws Mail 3.14.0 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2383 Lines: 71 On Mon, 19 Jun 2017 13:13:27 -0400 Will Hawkins wrote: > On Wed, Jun 14, 2017 at 4:01 PM, Steven Rostedt wrote: > > On Wed, 14 Jun 2017 13:47:17 -0400 > > Will Hawkins wrote: > > > > > >> > That's how trace-cmd parses it. > >> > >> In the kernel version that I am running (again, pretty old) I do not > >> have this file. I do, however, have > > > > It may be due to the kernel version. It gets the functions > > from /proc/kallsyms. That could have have an issue. Although, I have > > this too: > > > > # grep per_cpu_start /proc/kallsyms > > 0000000000000000 A __per_cpu_start > > Here's my result of a similar command: > > # cat /proc/kallsyms | grep per_cpu_start > 0000000000000000 D __per_cpu_start Right, that it doesn't have the latest code to say to ignore it in your kernel. > > There's only the difference between (what I think is) the flag value > there in the second column. > > > > > But I don't see it being converted in my report. > > > > Hmm, it's not saved. Interesting: > > > > trace-cmd report -f > > > > to see the list of saved functions. I need to figure out why it does > > for you, but not for me. > > sudo ./trace-cmd report -f | grep per_cpu_start > 0000000000000000 __per_cpu_start Yep, it took it. It only saves the address an the name. > > Ah, I bet it's a change in the kernel. A recent update to trace-cmd was > > to ignore functions in kallsyms of type "A" (which you can see is what > > I have above). > > And my output (above) seems to show that mine shows the per_cpu_start > as a D instead of an A. Perhaps that is why we are seeing differences > in our report? It's still curious that it would match 0x000..0 with > something that is clearly, well, not 0x00...0. Because function ips are seldom at the start of the function (well, other than fentry code), it grabs the first function with the address before the ip. Which would be 0x0000. > > Let me know if there is a spot in the code where it decides whether to > symbolize. Or, I can work backward from the recent change you > mentioned above to find that spot if you will tell me the hash of that > commit. If you run with trace-cmd report -O offset, you should see the offset off of functions. Not sure it works with function graph tracer, but should at least work with function tracing. -- Steve