Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp551209yba; Fri, 5 Apr 2019 11:56:18 -0700 (PDT) X-Google-Smtp-Source: APXvYqxdKH9eWJ92dYMp9vVef+P+NMazRVnk2c50mT7uFR2qa0QhU8QMEkHOy7HDn+Xk0DgZFm/y X-Received: by 2002:a17:902:6b03:: with SMTP id o3mr14752213plk.226.1554490578657; Fri, 05 Apr 2019 11:56:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554490578; cv=none; d=google.com; s=arc-20160816; b=uZ/2r5b1EQaJQipgZVoK6QNbtuC3cWwfgaXy2wrn+H317DmkL8/05AXiam6p04BNp5 NeIOsLE4JnRsFU7gsca4iLRCXC+n50bBDO1Z1kwne7u5z2FetwdNXtTQUExYmTzdNcUn Ku6lFoeOHqne4Gl8Sfp1+h/g8OBLM3O3xBJ7gmFxLI71yCuszgL5wZnBODqFVS7V4Xht 1dhbuYDpaIpc6w35Qk1PTHhi2M2A+weTa8E5kppKht0kjmdoH51xmWw7oEXZa7WuWpYU 8CtJmZi/xF/Fb9RgQYJyNTAu51VcOTAGSCIgN1PL+PYnREYD7u3mbUwkakdkYpRRulB6 ZdMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version; bh=Zcz/sKmP2HSiW6fy+3CoqixG4dXseL+/zdfyDvrPkn4=; b=Hf0Fz04yIIMeiQxOP0awwpsMgdOxJhZdugRC5DM7MCdk5pjVV37XGLySZrQWm+UPeh 0/659KavpccL9OS5TOaZozqGn+UchZS927n0I2z5JdF3To4tqiPY8jRE3itw2ISsKw9s 8dMOkxai11TY74wQfnMRldBKcyhvrLzuEfK2xgv1fJf0fT1l1esaJt/+alyqZhtQMcaj yIxFBxmTbuYAzAsxyAq7yiAvKwn3bAjFYkkDU2L3UoeIl06MYQg+oloPFHXP6S6AkP9h vfRcZ6XwRaodex7yITQc1P0pmtZxCo50UKi6+g5QBOPWIQHGXGtABN0aghGgHDPcSElA DSnQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g10si15395853pgq.440.2019.04.05.11.56.02; Fri, 05 Apr 2019 11:56:18 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731670AbfDESz1 (ORCPT + 99 others); Fri, 5 Apr 2019 14:55:27 -0400 Received: from mail-it1-f194.google.com ([209.85.166.194]:36252 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731183AbfDESz1 (ORCPT ); Fri, 5 Apr 2019 14:55:27 -0400 Received: by mail-it1-f194.google.com with SMTP id y10so11267466itc.1 for ; Fri, 05 Apr 2019 11:55:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Zcz/sKmP2HSiW6fy+3CoqixG4dXseL+/zdfyDvrPkn4=; b=TvFnNYR57uUrTpLJiOP6LgdDt1h8F5U9n2tc4IMxbeXEKg3Ea/IZv7YN4p5NfDGPqP n5b5TTikcENzawRXnRIJHzrJ5isVsKFE6jjOmvpV39ubUZztfM7ttQJffTrExfLSvabl QsO5WRlHbPh9xbbr+H+2kaivftaAbnbWKiCDBKNO/5DtJocG7yHr2S/xu3jHfLiBdS2I JHulcg7X4EDesD6ckftewsLha6SraVx9gR26wyki24C9PgQsJr2fHI2vmwyeb65Joev3 tybDba5jV6ylx+dPE96XqPtpONHK8fFiwIH09pdIcN+svZzNqidrOPAWrrZVGskOo/XS M4cQ== X-Gm-Message-State: APjAAAX7QJD0ylv9C8LEv16MH9NZ5jQTHm6+H5BJo7oMPeELxzqxTDr6 9VPfxprSNKdNg2i2jdY1PrXe0HhzCE6Za1HCpGr2lQ== X-Received: by 2002:a24:65c8:: with SMTP id u191mr10745619itb.137.1554490525905; Fri, 05 Apr 2019 11:55:25 -0700 (PDT) MIME-Version: 1.0 References: <20190404172545.20928-1-kasong@redhat.com> <20190405140929.pycfea7drnpb2sug@treble> <20190405165715.fpgh4ggkmnqdtfwm@treble> <20190405172659.4hph4rocqgofkb73@treble> In-Reply-To: <20190405172659.4hph4rocqgofkb73@treble> From: Kairui Song Date: Sat, 6 Apr 2019 02:55:14 +0800 Message-ID: Subject: Re: [RFC PATCH] perf/x86: make perf callchain work without CONFIG_FRAME_POINTER To: Josh Poimboeuf Cc: Linux Kernel Mailing List , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Alexander Shishkin , Jiri Olsa , Namhyung Kim , Thomas Gleixner , Borislav Petkov , "H. Peter Anvin" , Dave Young Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, Apr 6, 2019 at 1:27 AM Josh Poimboeuf wrote: > > On Sat, Apr 06, 2019 at 01:05:55AM +0800, Kairui Song wrote: > > On Sat, Apr 6, 2019 at 12:57 AM Josh Poimboeuf wrote: > > > > > > On Fri, Apr 05, 2019 at 11:13:02PM +0800, Kairui Song wrote: > > > > Hi Josh, thanks for the review, I tried again, using latest upstream > > > > kernel commit ea2cec24c8d429ee6f99040e4eb6c7ad627fe777: > > > > # uname -a > > > > Linux localhost.localdomain 5.1.0-rc3+ #29 SMP Fri Apr 5 22:53:05 CST > > > > 2019 x86_64 x86_64 x86_64 GNU/Linux > > > > > > > > Having following config: > > > > > CONFIG_UNWINDER_ORC=y > > > > > # CONFIG_UNWINDER_FRAME_POINTER is not set > > > > and CONFIG_FRAME_POINTER is off too. > > > > > > > > Then record something with perf (also latest upstream version): > > > > ./perf record -g -e kmem:* -c 1 > > > > > > > > Interrupt it, then view the output: > > > > perf script | less > > > > > > > > Then I notice the stacktrace in kernle is incomplete like following. > > > > Did I miss anything? > > > > -------------- > > > > lvmetad 617 [000] 55.600786: kmem:kfree: > > > > call_site=ffffffffb219e269 ptr=(nil) > > > > ffffffffb22b2d1c kfree+0x11c (/lib/modules/5.1.0-rc3+/build/vmlinux) > > > > 7fba7e58fd0f __select+0x5f (/usr/lib64/libc-2.28.so) > > > > > > > > kworker/u2:5-rp 171 [000] 55.628529: > > > > kmem:kmem_cache_alloc: call_site=ffffffffb20e963d > > > > ptr=0xffffa07f39c581e0 bytes_req=80 bytes_alloc=80 > > > > gfp_flags=GFP_ATOMIC > > > > ffffffffb22b0dec kmem_cache_alloc+0x13c > > > > (/lib/modules/5.1.0-rc3+/build/vmlinux) > > > > ------------- > > > > > > > > And for the patch, I debugged the problem, and found how it happend: > > > > The reason is that we use following code for fetching the registers on > > > > a trace point: > > > > ...snip... > > > > #define perf_arch_fetch_caller_regs(regs, __ip) { \ > > > > (regs)->ip = (__ip); \ > > > > (regs)->bp = caller_frame_pointer(); \ > > > > (regs)->cs = __KERNEL_CS; > > > > ...snip... > > > > > > Thanks, I was able to recreate. It only happens when unwinding from a > > > tracepoint. I haven't investigated yet, but > > > perf_arch_fetch_caller_regs() looks highly suspect, since it's doing > > > (regs)->bp = caller_frame_pointer(), even for ORC. > > > > > > My only explanation for how your patch works is that RBP just happens to > > > point to somewhere higher on the stack, causing the unwinder to start at > > > a semi-random location. I suspect the real "fix" is that you're no > > > longer passing the regs to unwind_start(). > > > > > > > Yes that's right. Simply not passing regs to unwind_start will let the > > unwind start from the perf sample handling functions, and introduce a > > lot of "noise", so I let it skipped the frames until it reached the > > frame of the trace point. The regs->bp should still points to the > > stack base of the function which get called in the tracepoint that > > trigger perf sample, so let unwinder skip all the frames above it made > > it work. > > Ah, now I think I understand, thanks. perf_arch_fetch_caller_regs() > puts it in regs->bp, and then perf_callchain_kernel() reads that value > to tell the unwinder where to start dumping the stack trace. I guess > that explains why your patch works, though it still seems very odd that > perf_arch_fetch_caller_regs() is using regs->bp to store the frame > address. Maybe regs->sp would be more appropriate. > > -- > Josh Right, thanks for the comment. And after second thought there are some other issues here in the patch indeed, it still won't fix the problem when used with ebpf and tracepoint, I made some mistake about handling the callchain with different ways, will rethink about this and post an update later. -- Best Regards, Kairui Song