Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752163AbdFNUBY (ORCPT ); Wed, 14 Jun 2017 16:01:24 -0400 Received: from mail.kernel.org ([198.145.29.99]:39666 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751760AbdFNUBX (ORCPT ); Wed, 14 Jun 2017 16:01:23 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 09E9F22CB4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=goodmis.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=rostedt@goodmis.org Date: Wed, 14 Jun 2017 16:01:20 -0400 From: Steven Rostedt To: Will Hawkins Cc: LKML , Namhyung Kim Subject: Re: Ftrace vs perf user page fault statistics differences Message-ID: <20170614160120.577a6385@gandalf.local.home> In-Reply-To: References: <20170612212005.1bf8c44d@gandalf.local.home> <20170613090649.2a5edb6a@gandalf.local.home> <20170613160434.605fdef2@gandalf.local.home> <20170613221316.64d46ea5@gandalf.local.home> X-Mailer: Claws Mail 3.14.0 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2302 Lines: 71 On Wed, 14 Jun 2017 13:47:17 -0400 Will Hawkins wrote: > > That's how trace-cmd parses it. > > In the kernel version that I am running (again, pretty old) I do not > have this file. I do, however, have It may be due to the kernel version. It gets the functions from /proc/kallsyms. That could have have an issue. Although, I have this too: # grep per_cpu_start /proc/kallsyms 0000000000000000 A __per_cpu_start But I don't see it being converted in my report. Hmm, it's not saved. Interesting: trace-cmd report -f to see the list of saved functions. I need to figure out why it does for you, but not for me. > > /sys/kernel/debug/tracing/events/exceptions/page_fault_user/format > > and the contents are: > > name: page_fault_user > ID: 79 > format: > field:unsigned short common_type; offset:0; size:2; signed:0; > field:unsigned char common_flags; offset:2; size:1; signed:0; > field:unsigned char common_preempt_count; offset:3; size:1;signed:0; > field:int common_pid; offset:4; size:4; signed:1; > > field:unsigned long address; offset:8; size:8; signed:0; > field:unsigned long ip; offset:16; size:8; signed:0; > field:unsigned long error_code; offset:24; size:8; signed:0; > > print fmt: "address=%pf ip=%pf error_code=0x%lx", (void > *)REC->address, (void *)REC->ip, REC->error_code > > Again, this looks like exactly what I would expect since address has > the cr2 value in that function. Plus, we know that the raw value is > correct. I suppose that the "symbolification" of that value is done in > trace-cmd, right? So, perhaps that is where I should start looking for > the problem? > > I definitely want to follow up on this and help where I can. That > said, I think I am satisfied with "our" (really, your) answer to the > original problem. It has something to do with the reading of the function names in /proc/kallsyms. Ah, I bet it's a change in the kernel. A recent update to trace-cmd was to ignore functions in kallsyms of type "A" (which you can see is what I have above). What do you have when you do: sudo grep per_cpu_start /proc/kallsyms May or may not need sudo depending on if the kernel lets non root have access to kallsyms. -- Steve