Received-SPF: pass (google.com: domain of linux-kernel+bounces-50185-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Sender: Tejun Heo <htejun@gmail.com>
Date: Fri, 2 Feb 2024 06:56:28 -1000
From: Tejun Heo <tj@kernel.org>
To: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Jonas Oberhauser <jonas.oberhauser@huaweicloud.com>,
	Lai Jiangshan <jiangshanlai@gmail.com>,
	Petr Mladek <pmladek@suse.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	John Ogness <john.ogness@linutronix.de>,
	Sergey Senozhatsky <senozhatsky@chromium.org>,
	Stephen Rothwell <sfr@canb.auug.org.au>,
	linux-kernel@vger.kernel.org, rcu@vger.kernel.org
Subject: Re: [BUG] workqueues and printk not playing nice since next-20240130
Message-ID: <Zb0evDquygkDI_8P@slm.duckdns.org>
References: <410d6a87-bf34-457e-b714-1e6149d48532@paulmck-laptop>
 <c6ce8816-c4ff-4668-8cbb-88285330057d@huaweicloud.com>
 <25fd8537-5a27-4b62-9bf9-1ee7ca59b5b8@paulmck-laptop>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <25fd8537-5a27-4b62-9bf9-1ee7ca59b5b8@paulmck-laptop>

Hello,

On Fri, Feb 02, 2024 at 08:35:51AM -0800, Paul E. McKenney wrote:
> Good point, and if this sort of thing happens frequently, perhaps there
> should be an easy way of doing this.  One crude hack that might come
> pretty close would be to redefine the barrier() macro to be smp_mb().
> 
> But as noted earlier, -ENOREPRODUCE on today's -next.  I will try the
> next several -next releases.  But if they all get -ENOREPRODUCE, I owe
> everyone on CC an apology for having sent this report out before trying
> next-20240202.  :-/

I think I saw that problem too but could reproduce it with or without the
workqueue changes, so I did the lazy thing "oh well, somebody is gonna fix
that" and just tested as-is. It's a bit worrying that ppl don't seem to
already know what the culprit is. Hmm... I can't reproduce it anymore
either.

So, there is some chance that this may really be a subtle breakage. If you
ever see it happening again, triggering sysrq-t and capturing the dmesg
output (network should still work fine, so these shouldn't be too difficult)
may help. sysrq-t has workqueue state dump at the end which should clearly
indicate if anything is stalled in workqueue.

That said, another data point. In my test setup, I use the earlyprintk boot
option which enables console output way before than workqueue becomes
operational, so having on console output at all is highly unlikely to be
indicative of workqueue problem. My memory is hazy but it seems like I can
no longer reproduce the problem on the same git commit. Maybe it was a
problem on the qemu side?

Thanks.

-- 
tejun