Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 28 Jan 2021 00:24:52 +0900
From:   Masami Hiramatsu <mhiramat@kernel.org>
To:     Masami Hiramatsu <mhiramat@kernel.org>
Cc:     Nikolay Borisov <nborisov@suse.com>,
        Peter Zijlstra <peterz@infradead.org>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: kprobes broken since 0d00449c7a28
 ("x86: Replace ist_enter() with nmi_enter()")
Message-Id: <20210128002452.a79714c236b69ab9acfa986c@kernel.org>
In-Reply-To: <20210128001353.66e7171b395473ef992d6991@kernel.org>
References: <25cd2608-03c2-94b8-7760-9de9935fde64@suse.com>
        <20210128001353.66e7171b395473ef992d6991@kernel.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Thu, 28 Jan 2021 00:13:53 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:

> Hi Nikolay,
> 
> On Wed, 27 Jan 2021 15:43:29 +0200
> Nikolay Borisov <nborisov@suse.com> wrote:
> 
> > Hello,
> > 
> > I'm currently seeing latest Linus' master being somewhat broken w.r.t
> > krpobes. In particular I have the following test-case:
> > 
> > #!/bin/bash
> > 
> > mkfs.btrfs -f /dev/vdc &> /dev/null
> > mount /dev/vdc /media/scratch/
> > 
> > bpftrace -e 'kprobe:btrfs_sync_file {printf("kprobe: %s\n", kstack());}'
> > &>bpf-output &
> > bpf_trace_pid=$!
> > 
> > # force btrfs_sync_file to be called
> > sleep 2
> > xfs_io -f -c "pwrite 0 4m" -c "fsync" /media/scratch/file5
> > 
> > kill $bpf_trace_pid
> > sleep 1
> > 
> > grep -q kprobe bpf-output
> > retval=$?
> > rm -f bpf-output
> > umount /media/scratch
> > 
> > exit $retval
> > 
> > It traces btrfs_sync_file which is called when fsync is executed on a
> > btrfs file, however I don't see the stacktrace being printed i.e the
> > kprobe doesn't fire at all. The following alternative program:
> > 
> > bpftrace -e 'tracepoint:btrfs:btrfs_sync_file {printf("tracepoint:
> > %s\n", kstack());} kprobe:btrfs_sync_file {printf("kprobe: %s\n",
> > kstack());}'
> > 
> > only prints the stack from the tracepoint and not from the kprobe, given
> > that the tracepoint is called from the btrfs_sync_file function.
> 
> Thank you for reporting!
> 
> If you don't mind, could you confirm it with ftrace (tracefs)?
> bpftrace etc. involves too many things. It is better to test with
> simpler way to test it.
> I'm not familer with the bpftrace, but I think you can check it with
> 
> # cd /sys/kernel/debug/tracing
> # echo p:myevent btrfs_sync_file >> kprobe_events
> # echo stacktrace > events/kprobes/myevent/trigger
>  (or echo 1 > options/stacktrace , if trigger file doesn't exist)

Of course, also you have to enable the event.
 # echo 1 > events/kprobes/myevent/enable

And check the results

 # cat trace


> Could you also share your kernel config, so that we can reproduce it?

Thank you,

> 
> > 
> > I started bisecting this and arrived at the following commit:
> > 
> > 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")
> > 
> > FWIW the following series is applied on the kernel I was testing:
> > https://lore.kernel.org/lkml/159870598914.1229682.15230803449082078353.stgit@devnote2/
> > 
> > but it's still broken.
> 
> Peter, would you have any idea?
> 
> Thank you,
> 
> -- 
> Masami Hiramatsu <mhiramat@kernel.org>


-- 
Masami Hiramatsu <mhiramat@kernel.org>