Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932481AbbLBJ4W (ORCPT ); Wed, 2 Dec 2015 04:56:22 -0500 Received: from mail-wm0-f47.google.com ([74.125.82.47]:37470 "EHLO mail-wm0-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932334AbbLBJz7 (ORCPT ); Wed, 2 Dec 2015 04:55:59 -0500 Date: Wed, 2 Dec 2015 10:55:54 +0100 From: Ingo Molnar To: Peter Zijlstra Cc: "Wangnan (F)" , Jiri Olsa , Arnaldo Carvalho de Melo , David Ahern , Milian Wolff , linux-kernel@vger.kernel.org, pi3orama , lizefan 00213767 Subject: Re: [BUG REPORT] perf tools: x86_64: Broken calllchain when sampling taken at 'callq' instruction Message-ID: <20151202095554.GA17324@gmail.com> References: <20151119063709.GA14852@gmail.com> <564D6FF9.3030105@huawei.com> <20151119102300.GA2830@gmail.com> <20151119112315.GL3816@twins.programming.kicks-ass.net> <20151127083811.GA26257@gmail.com> <20151130092843.GF17308@twins.programming.kicks-ass.net> <20151201072826.GB28270@gmail.com> <20151201083845.GI3816@twins.programming.kicks-ass.net> <20151201161125.GA2441@gmail.com> <20151201172136.GP3816@twins.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151201172136.GP3816@twins.programming.kicks-ass.net> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3013 Lines: 66 * Peter Zijlstra wrote: > On Tue, Dec 01, 2015 at 05:11:25PM +0100, Ingo Molnar wrote: > > > > * Peter Zijlstra wrote: > > > > > > so I think the problem here is that the RSP does not match up to the RIP. We > > > > can either pass along the original RIP+RSP, or the fixed up one - but what we > > > > do currently is that we pass along only half of it - which corrupts dwarf > > > > unwinding state that doesn't tolerate such errors. > > > > > > Still not sure what that gets you. Then you get a sample at a known wrong > > > location, why would you want that? > > > > Well, we'd at least get a valid call trace - which the 'mixed' one isn't? I.e. > > this only matters with --call-graph. > > > > But yeah, with my suggestion we'd essentially fall back from cycles:pp to > > cycles:p, ideally we'd want to have real_rsp. Does the hardware provide that? > > No, no real_rsp. > > > User-space cannot compute that reliably I think, what if the 'real' instruction > > was manipulating RSP in more complex ways than doing a CALL? > > I'm not really too aware of these asm details :/ So the way I understand it is that dwarf unwind relies on having consistent pairs of {RIP,RSP}, to be able to look at the precise instruction, look up the debuginfo what the function-internal stack offset at that instruction is (as there's no RBP frame information available so the stack offset varies). That method really relies on having precise {RIP,RSP} pairs - it may otherwise lose its notion of where the next return address on the stack is, and the backtrace goes off into la-la-land. I suppose such broken callchains were how Wangnan noticed the problem? ( As a comparison, RBP based backtraces only rely on having an exact RBP available as a starting point, that will define the next backtrace link and the next RBP value. That is why we can do the backtrace walk in the kernel. ) There are other ways a dwarf unwind can fail: for example if there's no dwarf debuginfo available for the binary in question, or if the stack snapshot we did is too small to find the next link in the backtrace. But having an off-by-few RIP is enough to throw the backtrace off at the very first step. > Jiri, what is in PERF_SAMPLE_REGS_USER.IP ? from a quick reading that is > whatever is returned from task_pt_regs(current), not the perf_data.regs.ip field > which contains the corrected IP. > > Should the uwinder then not use PERF_SAMPLE_REGS_USER.{IP,SP} for a consistent > unwind? Yeah, if we have the real RIP belonging to that RSP value, then all should be golden. That way the unwinder can find the next link. (If all the other conditions for a successful backtrace are also there.) Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/