Received-SPF: pass (google.com: domain of linux-kernel+bounces-13230-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: David Laight <David.Laight@ACULAB.COM>
To: "'linux-kernel@vger.kernel.org'" <linux-kernel@vger.kernel.org>,
	"'peterz@infradead.org'" <peterz@infradead.org>, "'longman@redhat.com'"
	<longman@redhat.com>
CC: "'mingo@redhat.com'" <mingo@redhat.com>, "'will@kernel.org'"
	<will@kernel.org>, "'boqun.feng@gmail.com'" <boqun.feng@gmail.com>, "'Linus
 Torvalds'" <torvalds@linux-foundation.org>, "'xinhui.pan@linux.vnet.ibm.com'"
	<xinhui.pan@linux.vnet.ibm.com>,
	"'virtualization@lists.linux-foundation.org'"
	<virtualization@lists.linux-foundation.org>, 'Zeng Heng'
	<zengheng4@huawei.com>
Subject: RE: [PATCH next 2/5] locking/osq_lock: Avoid dirtying the local cpu's
 'node' in the osq_lock() fast path.
Thread-Topic: [PATCH next 2/5] locking/osq_lock: Avoid dirtying the local
 cpu's 'node' in the osq_lock() fast path.
Thread-Index: Ado6mUd9raebt4vmQzObd4YJJW0D/g==
Date: Fri, 29 Dec 2023 20:54:47 +0000
Message-ID: <6eacbdbd1e0f489783439af512dc8325@AcuMS.aculab.com>
References: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com>
In-Reply-To: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com>
Accept-Language: en-GB, en-US
Precedence: bulk
MIME-Version: 1.0
Content-Language: en-US
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

osq_lock() starts by setting node->next to NULL and node->locked to 0.
Careful analysis shows that node->next is always NULL on entry.

node->locked is set non-zero by another cpu to force a wakeup.
This can only happen after the 'prev->next =3D node' assignment,
so locked can be set to zero just before that (along with the assignment
to node->prev).

Only initialise node->cpu once, after that use its value instead
of smp_processor_id() - which is probably a real function call.

Should reduce cache-line bouncing a little.

Signed-off-by: David Laight <david.laight@aculab.com>
---
 kernel/locking/osq_lock.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c
index d414eef4bec6..55f5db896c02 100644
--- a/kernel/locking/osq_lock.c
+++ b/kernel/locking/osq_lock.c
@@ -51,7 +51,7 @@ osq_wait_next(struct optimistic_spin_queue *lock,
 =09      struct optimistic_spin_node *prev)
 {
 =09struct optimistic_spin_node *next =3D NULL;
-=09int curr =3D encode_cpu(smp_processor_id());
+=09int curr =3D node->cpu;
 =09int old;
=20
 =09/*
@@ -98,12 +98,10 @@ bool osq_lock(struct optimistic_spin_queue *lock)
 {
 =09struct optimistic_spin_node *node =3D this_cpu_ptr(&osq_node);
 =09struct optimistic_spin_node *prev, *next;
-=09int curr =3D encode_cpu(smp_processor_id());
 =09int old;
=20
-=09node->locked =3D 0;
-=09node->next =3D NULL;
-=09node->cpu =3D curr;
+=09if (unlikely(node->cpu =3D=3D OSQ_UNLOCKED_VAL))
+=09=09node->cpu =3D encode_cpu(smp_processor_id());
=20
 =09/*
 =09 * We need both ACQUIRE (pairs with corresponding RELEASE in
@@ -111,12 +109,13 @@ bool osq_lock(struct optimistic_spin_queue *lock)
 =09 * the node fields we just initialised) semantics when updating
 =09 * the lock tail.
 =09 */
-=09old =3D atomic_xchg(&lock->tail, curr);
+=09old =3D atomic_xchg(&lock->tail, node->cpu);
 =09if (old =3D=3D OSQ_UNLOCKED_VAL)
 =09=09return true;
=20
 =09prev =3D decode_cpu(old);
 =09node->prev =3D prev;
+=09node->locked =3D 0;
=20
 =09/*
 =09 * osq_lock()=09=09=09unqueue
@@ -214,7 +213,7 @@ bool osq_lock(struct optimistic_spin_queue *lock)
 void osq_unlock(struct optimistic_spin_queue *lock)
 {
 =09struct optimistic_spin_node *node, *next;
-=09int curr =3D encode_cpu(smp_processor_id());
+=09int curr =3D raw_cpu_read(osq_node.cpu);
=20
 =09/*
 =09 * Fast path for the uncontended case.
--=20
2.17.1

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1=
PT, UK
Registration No: 1397386 (Wales)