Received: by 2002:a05:6a10:8c0a:0:0:0:0 with SMTP id go10csp1093734pxb; Thu, 4 Mar 2021 03:08:42 -0800 (PST) X-Google-Smtp-Source: ABdhPJzwS1uu60aIC6WMMSYR5djpf4xx+RVossWahHd74JfypMlWzNT8mFvx7U/1fd417WKwpup3 X-Received: by 2002:a17:906:86c6:: with SMTP id j6mr3437256ejy.197.1614856122454; Thu, 04 Mar 2021 03:08:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1614856122; cv=none; d=google.com; s=arc-20160816; b=JKv7+l12aFXapRERgxuzsSVTmKzcPz2P/7jOKknorHya4uDVl4sEv5BTUr1vIiJ+cr bwphf0LDPJZ6hjF+YGc9viDNvJIiqDrI5V6PpThGT5fVKZi+9zU9599uJllgUVNZsudI Q3Tf0wYC4bot4JLnQctRE3rQdpq9EDUMNlMLH1/EU/fZqB3zug7J8WHmNZ6z2luKA4Vp rRbLzoX0r8xOHtPacjtBC3AMeLtjcgwvwYKAxrB7FhTnGzi+0B0S4qBc1CX/MRyxIA4A cUFPBoAnxa/RliKnz2j6V492Xec4D3rGlL0ICIA8fjgz3IjfAtqgtTDwMmMBhla2wMAe UoxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:subject:cc:to:from:date; bh=GtlSJfloTLKYh4WdiMEpya4+bYK9U0r9+wWilWIWgus=; b=0Vok2dnGVYfVE8tDbgiaJYBHHU8c7s04JZ5MR54jzYE4zTKoGdIT0u40qpnVR0gOu1 OkfDI7g95Tm6s+V9YbgSx7NflWnPmMGf+iSj4LfAyJKZ/l4wxw0tUmhfHOetL79PMS+s gLNsDN0JIOpxFIYofSwTwqi6k5CfSgEO7/fDcfFb+uX8yBlLtNgaFDvIeZkFRIqtnqGc hZRxg1UjA459boKOCs2ArV1afRyp78pop8hll7GwQJ+8BBHg4XWycD5FFFQNajcQ4uPA 55xzw3HMgaiO+SLhAn3p1JrX/qREVYWm6bEcsylWockdXqgFmfVWsfxtNv7vuLbx34eO B6Aw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id c6si6520231edr.602.2021.03.04.03.08.19; Thu, 04 Mar 2021 03:08:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1386147AbhCCRqr (ORCPT + 99 others); Wed, 3 Mar 2021 12:46:47 -0500 Received: from mail.kernel.org ([198.145.29.99]:50620 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242986AbhCCOhu (ORCPT ); Wed, 3 Mar 2021 09:37:50 -0500 Received: from gandalf.local.home (cpe-66-24-58-225.stny.res.rr.com [66.24.58.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1E13964EE1; Wed, 3 Mar 2021 14:26:07 +0000 (UTC) Date: Wed, 3 Mar 2021 09:26:04 -0500 From: Steven Rostedt To: Masami Hiramatsu Cc: "Daniel Xu" , linux-kernel@vger.kernel.org, "bpf@vger.kernel.org" , kuba@kernel.org Subject: Re: Broken kretprobe stack traces Message-ID: <20210303092604.59aea82c@gandalf.local.home> In-Reply-To: <20210303134828.39922eb167524bc7206c7880@kernel.org> References: <1fed0793-391c-4c68-8d19-6dcd9017271d@www.fastmail.com> <20210303134828.39922eb167524bc7206c7880@kernel.org> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 3 Mar 2021 13:48:28 +0900 Masami Hiramatsu wrote: > > > > > I think (can't prove) this used to work: Would be good to find out if it did. > > I'm not sure the bpftrace had correctly handled it or not. > > > > > # bpftrace -e 'kretprobe:__tcp_retransmit_skb { @[kstack()] = count() }' > > Attaching 1 probe... > > ^C > > > > @[ > > kretprobe_trampoline+0 > > ]: 1 > > Would you know how the bpftrace stacktracer rewinds the stack entries? > FYI, ftrace does it in trace_seq_print_sym()@kernel/trace/trace_output.c > The difference between trace events and normal function tracing stack traces is that it keeps its original return address. But kretprobes (and function graph tracing, and some bpf trampolines too) modify the return pointer, and that could possibly cause havoc with the stack trace. -- Steve