Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Feedback-ID: iad51458e:Fastmail
Date:   Mon, 15 Aug 2022 11:58:52 -0700
From:   Boqun Feng <boqun.feng@gmail.com>
To:     Hector Martin <marcan@marcan.st>
Cc:     Will Deacon <will@kernel.org>,
        Linux ARM <linux-arm-kernel@lists.infradead.org>,
        Greg KH <gregkh@linuxfoundation.org>, jirislaby@kernel.org,
        Marc Zyngier <maz@kernel.org>,
        Mark Rutland <mark.rutland@arm.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Catalin Marinas <catalin.marinas@arm.com>,
        Asahi Linux <asahi@lists.linux.dev>,
        Oliver Neukum <oneukum@suse.com>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: Debugging a TTY race condition on M1 (memory ordering dragons)
Message-ID: <YvqXbApYbGGEqQl+@boqun-archlinux>
References: <6c089268-4f2c-9fdf-7bcb-107b611fbc21@marcan.st>
 <20220815134711.GA10374@willie-the-truck>
 <63cd54a8-3c48-d1b9-406a-c521bd02ee4a@marcan.st>
 <YvqKtJn5eBsDJXBI@boqun-archlinux>
 <8ccfdd2d-ef77-4586-e50c-985e1d13726a@marcan.st>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <8ccfdd2d-ef77-4586-e50c-985e1d13726a@marcan.st>
Precedence: bulk

On Tue, Aug 16, 2022 at 03:26:21AM +0900, Hector Martin wrote:
> On 16/08/2022 03.04, Boqun Feng wrote:
> > On Tue, Aug 16, 2022 at 01:01:17AM +0900, Hector Martin wrote:
> > Hmm.. but doesn't your (and Will's) finding actually show why
> > queue_work() only guarantee ordering if queuing succeeds? In other
> > words, if you want extra ordering, use smp_mb() before queue_work()
> > like:
> > 
> > 	smp_mb();	// pairs with smp_mb() in set_work_pool_and_clear_pending()
> > 	queue_work();	// if queue_work() return false, it means the work
> > 			// is pending, and someone will eventually clear
> > 		      	// the pending bit, with the smp_mb() above it's
> > 			// guaranteed that work function will see the
> > 			// memory accesses above.
> > 
> > Of course, I shall defer this to workqueue folks. Just saying that it
> > may not be broken. We have a few similar guarantees, for example,
> > wake_up_process() only provides ordering if it really wakes up a
> > process.
> 
> Technically yes, but that doesn't actually make a lot of sense, and in
> fact the comments inside the workqueue code imply that it does actually
> provide order even in the failure case (and there are other barriers to
> try to make that happen, just not enough). Note that the ordering
> documentation was added post-facto, and I don't think the person who
> wrote it necessarily considered whether it *actually* provides
> guarantees in the failure case, and whether it should.
> 
> wake_up_process() is different because it doesn't actually guarantee
> anything if the process is already awake. However, under this
> definition, queue_work() guarantees that *some* work execution will
> observe every preceding write before queue_work(), regardless of the
> current state, and that is a very useful property. That is something
> that wake_up_process() semantics can't do.
> 
> Without this guarantee, basically every queue_work() user that's using
> some kind of producer/consumer pattern would need the explicit barrier.
> I imagine that pattern is very common.
> 

I agree this is handy, but an unconditional full barrier may be costy to
some users, and probably unnecessary if the users periodically queue
the work. In that case, some successful enqueue will eventually make all
memory accesses observable. Also if workqueue users use their own
locking in work function, then the barrier is also unnecessary.

The document part of course needs some help to clear things up. But I'm
not sure "strengthen"ing the ordering guarantee of queue_work() is a
good idea. Maybe a dedicated API, like:

// More work is needed for the @work, it has the same semantics as
// queue_work() if the @work is not pending. If the @work is pending,
// this ensures the work function observes all memory access before
// this.
void queue_more_work(struct work_struct *work)
{
	smp_mb();
	queue_work(work);
}

Regards,
Boqun

> - Hector