Received-SPF: pass (google.com: domain of linux-kernel+bounces-86620-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Thu, 29 Feb 2024 12:19:40 +0100
From: Petr Mladek <pmladek@suse.com>
To: "John B. Wyatt IV" <jwyatt@redhat.com>
Cc: John Ogness <john.ogness@linutronix.de>,
	Clark Williams <williams@redhat.com>, jlelli@redhat.com,
	Derek Barbosa <debarbos@redhat.com>,
	"John B. Wyatt IV" <sageofredondo@gmail.com>,
	LKML <linux-kernel@vger.kernel.org>,
	linux-rt-users <linux-rt-users@vger.kernel.org>
Subject: Re: NMI Reported with console_blast.sh
Message-ID: <ZeBoTNe2cjb4BeXb@alley>
References: <ZcQjxa4UA6hzXHnU@thinkpad2021>
 <87v86yc88b.fsf@jogness.linutronix.de>
 <ZcaQI8l1dcBx2feC@thinkpad2021>
 <87a5o8j9gp.fsf@jogness.linutronix.de>
 <ZcqMOKtHsZ9qnxNg@thinkpad2021>
 <ZdbZz-v_lug9_t6C@thinkpad2021>
 <ZeBnUCk598gttpds@alley>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZeBnUCk598gttpds@alley>

On Thu 2024-02-29 12:15:30, Petr Mladek wrote:
> On Thu 2024-02-22 00:21:19, John B. Wyatt IV wrote:
> > On Mon, Feb 12, 2024 at 04:23:04PM -0500, John B. Wyatt IV wrote:
> > > 
> > > Red Hat's conservers are having an issue with the machine I was testing
> > > on. It may take me a while to get back to you with more test results.
> > > 
> > 
> > Found a work-around with conserver. I was able to follow up on the printk-caller
> > info you requested.
> > 
> > I found 2 additional NMIs for a total of 3. Number 2 is very
> > large-please feel free to let me know what specific information you
> > wanted if it was unnecessary.

[...]

> > Compared to the two NMIs with throughput-performance (no preemption)
> > 
> >   <NMI>
> >   cpus=0
> >    .runnable_avg                  : 3072
> >  kthread (kernel/kthread.c:388) 
> >    .util_est_enqueued             : 0
> >   stack:0     pid:1733  tgid:1733  ppid:2      flags:0x00004000
> >    .min_vruntime                  : 2084315.290254
> >    .removed.load_avg              : 0
> >    .avg_vruntime                  : 2084315.290254
> >  console_blast.s  3497     34770.405603 N     34773.405603         3.000000     34764.898340         4   120 
> >    .util_avg                      : 1024
> >    .util_avg                      : 1024
> 
> It looks like messages from more (many) CPUs are mixed. I guess that they
> are printed by print_cfs_rq(). But the order looks random.
> 
> Also I wonder why it is printed from NMI context. Maybe, it is from
> some perf event, similar to hardlockup detector?

I have realized that we most likely see only small part of the mixed
output. I wonder if it is because it is printed from the emergency
context. Here the messages are flushed when leaving the context
and many might be lost.

Best Regards,
Petr