Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 20 Jul 2023 06:40:28 -0400
From:   Steven Rostedt <rostedt@goodmis.org>
To:     Tony Luck <tony.luck@intel.com>
Cc:     Aristeu Rozanski <aris@ruivo.org>, linux-kernel@vger.kernel.org
Subject: Re: rasdaemon broke between v6.0 and v6.3?
Message-ID: <20230720064028.1aeb3c18@gandalf.local.home>
In-Reply-To: <ZLhzWoyRCWN0FmqE@agluck-desk3>
References: <ZLhzWoyRCWN0FmqE@agluck-desk3>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8BIT
Precedence: bulk

On Wed, 19 Jul 2023 16:35:54 -0700
Tony Luck <tony.luck@intel.com> wrote:

> [resend as plain text - sorry for the earlier HTML]
> 
> An internal team is seeing tests that worked on v6.0 fail on v6.3. The problem is that
> rasdaemon isn’t waking up to process the “mce_record” trace events.
> 
> Manually checking for them works:
> 
> root@R-4251:/sys/kernel/debug/tracing>systemctl stop rasdaemon
> root@R-4251:/sys/kernel/debug/tracing>
> root@R-4251:/sys/kernel/debug/tracing>
> root@R-4251:/sys/kernel/debug/tracing>echo 1 > events/mce/mce_record/enable
> root@R-4251:/sys/kernel/debug/tracing>
> root@R-4251:/sys/kernel/debug/tracing>cat trace_pipe
>            <...>-235     [000] .....   596.892583: mce_record: CPU: 0, MCGc/s: f000c15/0, MC13: 8c00004200800090, IPID: 0000000000000000, ADDR/MISC/SYND: 0000000123450000/08000a80c2982086/0000000000000000, RIP: 00:<0000000000000000>, TSC: 14120b051a1, PROCESSOR: 0:c06f1, TIME: 1689802780, SOCKET: 0, APIC: 0
>      kworker/0:2-235     [000] .....   597.204343: mce_record: CPU: 0, MCGc/s: f000c15/0, MC255: 9c0000000000009f, IPID: 0000000000000000, ADDR/MISC/SYND: 0000000123450000/000000000000008c/0000000000000000, RIP: 00:<0000000000000000>, TSC: 0, PROCESSOR: 0:c06f1, TIME: 1689802781, SOCKET: 0, APIC: 0
> 
> So their tests are injecting errors, and the trace event is firing.
> 
> Is there some updated version of rasdaemon needed?
> 
> Some kernel CONFIG option problem?
> 

A bug was fixed that I think affected rasdaemon.

commit 3e46d910d8acf94e5360126593b68bf4fee4c4a1
Author: Shiju Jose <shiju.jose@huawei.com>
Date:   Thu Feb 2 18:23:09 2023 +0000

    tracing: Fix poll() and select() do not work on per_cpu trace_pipe and trace_pipe_raw

Make sure /sys/kernel/tracing/buffer_percent = 0

-- Steve