Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753235AbdFMBUe (ORCPT ); Mon, 12 Jun 2017 21:20:34 -0400 Received: from mail.kernel.org ([198.145.29.99]:48252 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752558AbdFMBUd (ORCPT ); Mon, 12 Jun 2017 21:20:33 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 493C5219A8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=goodmis.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=rostedt@goodmis.org Date: Mon, 12 Jun 2017 21:20:05 -0400 From: Steven Rostedt To: Will Hawkins Cc: LKML Subject: Re: Ftrace vs perf user page fault statistics differences Message-ID: <20170612212005.1bf8c44d@gandalf.local.home> In-Reply-To: References: X-Mailer: Claws Mail 3.14.0 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3918 Lines: 116 On Mon, 12 Jun 2017 20:20:42 -0400 Will Hawkins wrote: > Dear Mr. Rostedt and Kernel community, > > I hope that this is an appropriate place to ask this question, Please > forgive me for wasting your time if it is not. I've searched for > answers to this question on LKML and the "net" before asking. I > apologize if I already missed the question and answer somewhere else. No no, this is the correct forum. > > I was doing a gut check of my understanding of how the Kernel pages in > a binary program for execution. I created a very small program that is > two pages. The entry point to the program is on page a and the code > simply jumps to the second page (b) for its termination. > > I compiled the program and dropped the kernel's memory caches [1]. > Then I ran the program under perf: > > perf record --call-graph fp -e page-faults ../one_page_play/page Can you supply this program. So I can see exactly what it does? > > I looked at the results: > > perf report > > and the results were as I expected. There were two page faults for > loading the code into memory and a page fault to > copy_user_enhanced_fast_string invoked by execve's implementation when > loading the binary. What does perf script show you? > > I decided to run the application under ftrace just for fun. I wanted > an excuse to learn more about it and this seemed like the perfect > chance. I used the incredible trace-cmd suite for the actual > incantation of ftrace. I won't include the actual incantations here > because I used many of them while digging around. what events did you enable? > > The results are both expected and unexpected. I see output like this: > > Event: page_fault_user:0x4000e0 > > which indicates that there is a page fault at the program's entry > point (and matches what I saw with the perf output). I have another > similar entry that confirms the other expected page fault when loading > the second page of the test application. > > However, I also see entries like this: > > Event: page_fault_user:0x7f4f590934c4 (1) This could very well be the dynamic linker. > > The addresses of the faults I see that match that pattern are not > loaded into the application binary. What I discovered as I > investigated, is that those page faults seem to occur when the kernel > is attempting to record the output of stack traces, etc. Are you sure? What does ldd give you of your program? > > After thinking through this, I came up with the following hypothesis > which is the crux of this email: > > Ftrace's act of recording the traces that I requested to its ring > buffer generated page faults of their own. These page faults are > generated on behalf of the traced program and get reported in the > results. There are no page faults that happen by the ftrace ring buffer that would be associated with the program. > > If that is correct/reasonable, it explains the differences between > what perf is reporting and what ftrace is reporting and I am happy. > If, however, that is a bogus conclusion, please help me understand > what is going on. I'll need to know more about exactly what you are doing to help out. > > I know that everyone who is on this email is incredibly busy and has > much to do. I hope that I've included enough information to make it > possible for you experts to advise, but not included too much to waste > your time. > > If you have the time or interest in answering, I would love to hear > your responses. Please CC me directly on all responses. No worries about Ccing you directly. LKML gets over 600 emails a day. Nobody reads it all. If someone isn't Cc'd directly, they will most likely never respond. > > Thanks again for your time! No problem. -- Steve > > Will > > [1] I used echo 3 > /proc/sys/vm/drop_caches to accomplish this and > issued it between every run. It may have been overkill, but I did it > anyway.