Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Subject: Re: Linux-next POWER9 NULL pointer NIP since 1st Apr.
From:   Qian Cai <cai@lca.pw>
In-Reply-To: <20200409101413.35d9c72d@gandalf.local.home>
Date:   Fri, 10 Apr 2020 15:20:49 -0400
Cc:     Michael Ellerman <mpe@ellerman.id.au>,
        Nicholas Piggin <npiggin@gmail.com>,
        linuxppc-dev <linuxppc-dev@lists.ozlabs.org>,
        LKML <linux-kernel@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <06A2EA93-B730-4DB1-819F-D27E7032F0B3@lca.pw>
References: <15AC5B0E-A221-4B8C-9039-FA96B8EF7C88@lca.pw>
 <87eeszlb6u.fsf@mpe.ellerman.id.au>
 <0675B22E-8F32-432C-9378-FDE159DD1729@lca.pw>
 <20200407093054.3eb23e45@gandalf.local.home>
 <EA9F9A54-87BC-477A-BE8A-7D53F80C5223@lca.pw>
 <20200409101413.35d9c72d@gandalf.local.home>
To:     Steven Rostedt <rostedt@goodmis.org>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On Apr 9, 2020, at 10:14 AM, Steven Rostedt <rostedt@goodmis.org> =
wrote:
>=20
> On Thu, 9 Apr 2020 06:06:35 -0400
> Qian Cai <cai@lca.pw> wrote:
>=20
>>>> I=E2=80=99ll go to bisect some more but it is going to take a =
while.
>>>>=20
>>>> $ git log --oneline 4c205c84e249..8e99cf91b99b
>>>> 8e99cf91b99b tracing: Do not allocate buffer in =
trace_find_next_entry() in atomic
>>>> 2ab2a0924b99 tracing: Add documentation on set_ftrace_notrace_pid =
and set_event_notrace_pid
>>>> ebed9628f5c2 selftests/ftrace: Add test to test new =
set_event_notrace_pid file
>>>> ed8839e072b8 selftests/ftrace: Add test to test new =
set_ftrace_notrace_pid file
>>>> 276836260301 tracing: Create set_event_notrace_pid to not trace =
tasks =20
>>>=20
>>>> b3b1e6ededa4 ftrace: Create set_ftrace_notrace_pid to not trace =
tasks
>>>> 717e3f5ebc82 ftrace: Make function trace pid filtering a bit more =
exact =20
>>>=20
>>> If it is affecting function tracing, it is probably one of the above =
two
>>> commits. =20
>>=20
>> OK, it was narrowed down to one of those messed with mcount here,
>=20
> Thing is, nothing here touches mcount.

Yes, you are right. I went back to test the commit just before the =
5.7-trace merge request,
I did reproduce there. The thing is that this bastard could take more =
6-hour to happen,
so my previous attempt did not wait long enough. Back to the square =
one...

>=20
>>=20
>> 8e99cf91b99b tracing: Do not allocate buffer in =
trace_find_next_entry() in atomic
>=20
> Touches reading the trace buffer.
>=20
>> 2ab2a0924b99 tracing: Add documentation on set_ftrace_notrace_pid and =
set_event_notrace_pid
>=20
> Documentation.
>=20
>> 6a13a0d7b4d1 ftrace/kprobe: Show the maxactive number on =
kprobe_events
>=20
> kprobe output.
>=20
>> c9b7a4a72ff6 ring-buffer/tracing: Have iterator acknowledge dropped =
events
>=20
> Reading the buffer.
>=20
>> 06e0a548bad0 tracing: Do not disable tracing when reading the trace =
file
>=20
> Reading the buffer.
>=20
>> 1039221cc278 ring-buffer: Do not disable recording when there is an =
iterator
>=20
> Reading the buffer.
>=20
>> 07b8b10ec94f ring-buffer: Make resize disable per cpu buffer instead =
of total buffer
>=20
> Resizing the buffer.
>=20
>> 153368ce1bd0 ring-buffer: Optimize rb_iter_head_event()
>=20
> Reading the buffer.
>=20
>> ff84c50cfb4b ring-buffer: Do not die if rb_iter_peek() fails more =
than thrice
>=20
> Reading the buffer.
>=20
>> 785888c544e0 ring-buffer: Have rb_iter_head_event() handle concurrent =
writer
>=20
> Reading the buffer.
>=20
>> 28e3fc56a471 ring-buffer: Add page_stamp to iterator for =
synchronization
>=20
> Reading the buffer.
>=20
>> bc1a72afdc4a ring-buffer: Rename ring_buffer_read() to =
read_buffer_iter_advance()
>=20
> Reading the buffer.
>=20
>> ead6ecfddea5 ring-buffer: Have ring_buffer_empty() not depend on =
tracing stopped
>=20
> Reading the buffer.
>=20
>> ff895103a84a tracing: Save off entry when peeking at next entry
>=20
> Reading the buffer.
>=20
>> bf2cbe044da2 tracing: Use address-of operator on section symbols
>=20
> Affects trace_printk()
>=20
>> bbd9d05618a6 gpu/trace: add a gpu total memory usage tracepoint
>=20
> New tracepoint infrastructure (just new trace events for gpu)
>=20
>> 89b74cac7834 tools/bootconfig: Show line and column in parse error
>=20
> Extended command line boot config.
>=20
>> 306b69dce926 bootconfig: Support O=3D<builddir> option
>=20
> Extended command line boot config
>=20
>> 5412e0b763e0 tracing: Remove unused TRACE_BUFFER bits
>=20
> Removed unused enums.
>=20
>> b396bfdebffc tracing: Have hwlat ts be first instance and record =
count of instances
>=20
> Affects only the hard ware latency detector (most likely not even
> configured in the kernel).
>=20
> So I don't understand how any of the above commits can cause a =
problem.
>=20
> -- Steve