Received-SPF: pass (google.com: domain of linux-kernel+bounces-52683-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Mon, 5 Feb 2024 14:07:33 +0100
From: Petr Mladek <pmladek@suse.com>
To: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Tejun Heo <tj@kernel.org>,
	Jonas Oberhauser <jonas.oberhauser@huaweicloud.com>,
	Lai Jiangshan <jiangshanlai@gmail.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	John Ogness <john.ogness@linutronix.de>,
	Sergey Senozhatsky <senozhatsky@chromium.org>,
	Stephen Rothwell <sfr@canb.auug.org.au>,
	linux-kernel@vger.kernel.org, rcu@vger.kernel.org
Subject: Re: [BUG] workqueues and printk not playing nice since next-20240130
Message-ID: <ZcDdlRzpofn2cAuO@alley>
References: <410d6a87-bf34-457e-b714-1e6149d48532@paulmck-laptop>
 <c6ce8816-c4ff-4668-8cbb-88285330057d@huaweicloud.com>
 <25fd8537-5a27-4b62-9bf9-1ee7ca59b5b8@paulmck-laptop>
 <Zb0evDquygkDI_8P@slm.duckdns.org>
 <c5d5ad66-da86-447b-8014-820d2c67382d@paulmck-laptop>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <c5d5ad66-da86-447b-8014-820d2c67382d@paulmck-laptop>

On Fri 2024-02-02 09:40:05, Paul E. McKenney wrote:
> On Fri, Feb 02, 2024 at 06:56:28AM -1000, Tejun Heo wrote:
> > Hello,
> > 
> > On Fri, Feb 02, 2024 at 08:35:51AM -0800, Paul E. McKenney wrote:
> > > Good point, and if this sort of thing happens frequently, perhaps there
> > > should be an easy way of doing this.  One crude hack that might come
> > > pretty close would be to redefine the barrier() macro to be smp_mb().
> > > 
> > > But as noted earlier, -ENOREPRODUCE on today's -next.  I will try the
> > > next several -next releases.  But if they all get -ENOREPRODUCE, I owe
> > > everyone on CC an apology for having sent this report out before trying
> > > next-20240202.  :-/
> > 
> > I think I saw that problem too but could reproduce it with or without the
> > workqueue changes, so I did the lazy thing "oh well, somebody is gonna fix
> > that" and just tested as-is. It's a bit worrying that ppl don't seem to
> > already know what the culprit is. Hmm... I can't reproduce it anymore
> > either.
> 
> Glad that it is not just me!  I think...  ;-)
> 
> > So, there is some chance that this may really be a subtle breakage. If you
> > ever see it happening again, triggering sysrq-t and capturing the dmesg
> > output (network should still work fine, so these shouldn't be too difficult)
> > may help. sysrq-t has workqueue state dump at the end which should clearly
> > indicate if anything is stalled in workqueue.
> 
> Good point, if it does recur, I could try it on bare metal.

Please, me, John, and Sergey know if anyone see this again. I do not
feel comfortable when there is problem which might make consoles calm.

Well, there is no queued printk change. Also I do not see anything
obvious in the changes added to 6.7 which might cause this behavior.

> > That said, another data point. In my test setup, I use the earlyprintk boot
> > option which enables console output way before than workqueue becomes
> > operational, so having on console output at all is highly unlikely to be
> > indicative of workqueue problem. My memory is hazy but it seems like I can
> > no longer reproduce the problem on the same git commit. Maybe it was a
> > problem on the qemu side?
> 
> It might have been a qemu issue, but I am using the same qemu.

In theory, it might also be caused by a change in the serial console
driver. But I do not see anything obvious there either.

Best Regards,
Petr