Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 23 Apr 2019 06:30:10 -0700
From:   "Paul E. McKenney" <paulmck@linux.ibm.com>
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     Nicholas Piggin <npiggin@gmail.com>,
        LKMM Maintainers -- Akira Yokosawa <akiyks@gmail.com>,
        Andrea Parri <andrea.parri@amarulasolutions.com>,
        Boqun Feng <boqun.feng@gmail.com>,
        David Howells <dhowells@redhat.com>,
        Daniel Lustig <dlustig@nvidia.com>,
        Jade Alglave <j.alglave@ucl.ac.uk>,
        Kernel development list <linux-kernel@vger.kernel.org>,
        Luc Maranget <luc.maranget@inria.fr>,
        Alan Stern <stern@rowland.harvard.edu>,
        Will Deacon <will.deacon@arm.com>
Subject: Re: [PATCH] Documentation: atomic_t.txt: Explain ordering provided
 by smp_mb__{before,after}_atomic()
Reply-To: paulmck@linux.ibm.com
References: <Pine.LNX.4.44L0.1904191312200.1406-100000@iolanthe.rowland.org>
 <20190419180017.GP4038@hirez.programming.kicks-ass.net>
 <20190419182620.GF14111@linux.ibm.com>
 <1555719429.t9n8gkf70y.astroid@bobo.none>
 <20190420085440.GK14111@linux.ibm.com>
 <20190423123209.GR4038@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20190423123209.GR4038@hirez.programming.kicks-ass.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
Message-Id: <20190423133010.GK3923@linux.ibm.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Apr 23, 2019 at 02:32:09PM +0200, Peter Zijlstra wrote:
> On Sat, Apr 20, 2019 at 01:54:40AM -0700, Paul E. McKenney wrote:
> > 	And atomic_set(): set_preempt_state().	This fails
> > 	on x86, s390, and TSO friends, does it not?  Or is
> > 	this ARM-only?	Still, why not just smp_mb() before and
> > 	after?	Same issue in __kernfs_new_node(), bio_cnt_set(),
> > 	sbitmap_queue_update_wake_batch(), 
> > 
> > 	Ditto for atomic64_set() in __ceph_dir_set_complete().
> > 
> > 	Ditto for atomic_read() in rvt_qp_is_avail().  This function
> > 	has a couple of other oddly placed smp_mb__before_atomic().
> 
> That are just straight up bugs. The atomic_t.txt file clearly specifies
> the barriers only apply to RmW ops and both _set() and _read() are
> specified to not be a RmW.

Agreed.  The "Ditto" covers my atomic_set() consternation.  ;-)

> > 	And atomic_cmpxchg(): msc_buffer_alloc().  This instance
> > 	of smp_mb__before_atomic() can be removed unless I am missing
> > 	something subtle.  Ditto for kvm_vcpu_exiting_guest_mode(),
> > 	pv_kick_node(), __sbq_wake_up(), 
> 
> Note that pv_kick_node() uses cmpxchg_relaxed(), which does not
> otherwise imply barriers.

Good point, my eyes must have been going funny.

> > 	And lock acquisition??? acm_read_bulk_callback().
> 
> I think it goes with the set_bit() earlier, but what do I know.

Quite possibly!  In that case it should be smp_mb__after_atomic(),
and it would be nice if it immediately followed the set_bit().

> > 	In nfnl_acct_fill_info(), a smp_mb__before_atomic() after
> > 	a atomic64_xchg()???  Also before a clear_bit(), but the
> > 	clear_bit() is inside an "if".
> 
> Since it is _before, I'm thinking the pairing was intended with the
> clear_bit(), and yes, then I would expect the smp_mb__before_atomic() to
> be part of that same branch.

It is quite possible that this one is a leftover, where the atomic
operation was removed but the smp_mb__{before,after}_atomic() lived on.
I had one of those in RCU, which now has a patch in -rcu.

> > 	There are a few cases that would see added overhead.  For example,
> > 	svc_get_next_xprt() has the following:
> > 
> > 		smp_mb__before_atomic();
> > 		clear_bit(SP_CONGESTED, &pool->sp_flags);
> > 		clear_bit(RQ_BUSY, &rqstp->rq_flags);
> > 		smp_mb__after_atomic();
> > 
> > 	And xs_sock_reset_connection_flags() has this:
> > 
> > 		smp_mb__before_atomic();
> > 		clear_bit(XPRT_CLOSE_WAIT, &xprt->state);
> > 		clear_bit(XPRT_CLOSING, &xprt->state);
> > 		xs_sock_reset_state_flags(xprt);  /* Also a clear_bit(). */
> > 		smp_mb__after_atomic();
> > 
> > 	Yeah, there are more than a few misuses, aren't there?  :-/
> > 	A coccinelle script seems in order.  In 0day test robot.
> 
> If we can get it to flag the right patterns, then yes that might be
> useful regardless of the issue at hand, people seem to get this one
> wrong a lot.

To be fair, the odd-looking ones are maybe 5% of the total.  Still too
many wrong, but the vast majority look OK.

> > 	But there are a number of helper functions whose purpose
> > 	seems to be to wrap an atomic in smp_mb__before_atomic() and
> > 	smp_mb__after_atomic(), so some of the atomic_xxx_mb() functions
> > 	might be a good idea just for improved readability.
> 
> Are there really sites where _mb() makes sense? The above is just a lot
> of buggy code.

There are a great many that look like this:

	smp_mb__before_atomic();
	clear_bit(NFSD4_CLIENT_UPCALL_LOCK, &clp->cl_flags);
	smp_mb__after_atomic();

Replacing these three lines with this would not be a bad thing:

	clear_bit_mb(NFSD4_CLIENT_UPCALL_LOCK, &clp->cl_flags);

							Thanx, Paul