Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 22 Jun 2020 10:34:14 +0900
From:   Masami Hiramatsu <mhiramat@kernel.org>
To:     Ming Lei <ming.lei@redhat.com>
Cc:     Steven Rostedt <rostedt@goodmis.org>,
        Ming Lei <tom.leiming@gmail.com>,
        "Naveen N. Rao" <naveen.n.rao@linux.ibm.com>,
        Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        "David S. Miller" <davem@davemloft.net>,
        linux-block <linux-block@vger.kernel.org>
Subject: Re: kprobe: __blkdev_put probe is missed
Message-Id: <20200622103414.af303c4d4b0dad1c9d7262a3@kernel.org>
In-Reply-To: <20200622002753.GC670933@T590>
References: <20200618125438.GA191266@T590>
        <20200618225602.3f2cca3f0ed48427fc0a483b@kernel.org>
        <20200618231901.GA196099@T590>
        <20200619141239.56f6dda0976453b790190ff7@kernel.org>
        <20200619072859.GA205278@T590>
        <20200619081954.3d72a252@oasis.local.home>
        <20200619133240.GA351476@T590>
        <20200620003509.9521053fbd384f4f5d23408f@kernel.org>
        <20200619232820.GE353853@T590>
        <20200620103747.fb83f804083ef9956740acee@kernel.org>
        <20200622002753.GC670933@T590>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, 22 Jun 2020 08:27:53 +0800
Ming Lei <ming.lei@redhat.com> wrote:

> I mean it isn't from user's viewpoint, and the binary code is usually a
> black box for final kprobe user.
> 
> IMO, all your and Steven's input are just from kprobe/trace developer's viewpoint.
> Can you think about the issue from kprobe real/final user?
> 
> Trace is very useful tools to observe system internal, and people often
> relies on trace to understand system. However, missed probe often causes
> trouble for us to understand the system correctly.

Agreed. However, since kprobes related tracing tools are layered
to provide different features (e.g. kprobes abstructs sw breakpoint,
ftrace kprobe-events provides a minimum CUI, and perf-probe provides
binary analysis, etc.), this issue should be solved by user-level
binary analysis layer. (it is not good idea to analyze the optimized
code in kernel)


> > > 2) from implementation view, I understand exception should be trapped
> > > on the entry of __blkdev_put(), looks it isn't done.
> > 
> > No, it is correctly trapped the function entry address. The problem is
> > that the gcc optimized the nested function call into jump to the
> > beginning of function body (skip prologue).
> > 
> > Usually, a function is compiled as below
> > 
> > func()     (1) the entry address (func:)
> > {          (2) the function prologue (setup stackframe)  
> >   int a    (3) the beginning of function body 
> >    ...
> >   func()   (4) the nested function call
> > 
> > And in this case, the gcc optimized (4) into jump to (3) instead of
> > actual function call instruction. Thus, for the nested case (1) and
> > (2) are skipped.
> >  IOW, the code flow becomes
> >   (1)->(2)->(3)->(4)->(3)
> >  instead of 
> >   (1)->(2)->(3)->(4)->(1)->(2)->(3)
> > 
> > In this case, if we put a probe on (1) or (2), those are disappeared
> > in the nested call. Thus if you put a probe on (3) ('perf probe __blkdev_put:2')
> > you'll see the event twice.
> 
> Thanks for your explanation.
> 
> Can you kprobe guys improve the implementation for covering this case?
> For example, put probe on 3) in case the above situation is recognized.

OK, let me try to fix this in perf-probe since that is the simplest
binary analysis part in user-space.

Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>