Received: by 2002:a05:7412:b995:b0:f9:9502:5bb8 with SMTP id it21csp5716752rdb; Sun, 31 Dec 2023 13:54:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IHy1kAUH57ZSfDm8ZveT68IjW9Djf6eUSRy1RcDpD1eujgIVlTrg52UXfk98LbQVeRCNIyx X-Received: by 2002:a17:902:f688:b0:1d3:d0ff:f49c with SMTP id l8-20020a170902f68800b001d3d0fff49cmr18172404plg.21.1704059676281; Sun, 31 Dec 2023 13:54:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704059676; cv=none; d=google.com; s=arc-20160816; b=cXSdpz7G+x9siv6TE/UOKx+Au6JFDS8Y97u6C4EdyAnzgxdVYeIIrdstdzq31njY1w PvRFN4ZFWAG99o/b0ERcs8cuFvB6tlJoFlDbLHkeclkDjK1aDYsWJFrgoCmSx+J2j6jA RRZTPsruE5GP1cC/QE7YL7V5w5ROwUF7A0PElCv42p5giH/izr7sS0yxDZpzsBOCzqyl 0bFEkvPQln4RMmz6CGOZVc6+4zrseYZ00BhkU90i2pE1P5VTzEkBMkvL99pU3Mv7pWc5 ZpfC1fAIWOXvu0CkRIgVqaSL54pKY/TMx+qawX0CmD40uK2LnPsgGZpBuzCEEMspDZh7 xycA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:accept-language :in-reply-to:references:message-id:date:thread-index:thread-topic :subject:cc:to:from; bh=PvJPYprWlcHYeWl0A5QK2ZiBfCNNlHFwxz5z+BrFJ5Q=; fh=rK8F4zm15J6Jc7pqIJYnBw4d8zHrqB7RdoX7pamyocA=; b=qfupsGng+YG036TH/GST5JRAGQzIyPrkcXqriA6iZtkBQD8E8wCm3AdpNVWNeQeniO bCtn5PckxvxYu+b4HHYwc0Od/HLr5CSmUQ0GQabvEtEIs7LcCzwr45XpsPa9n9IQHvTW 5SUW+mK0hkC3mfMU8ZmleL3gAXBMFMQ9JdCnJvR61iaRA0KAetgWa9Y32kLH49+pm64r rDPifnRjaRvxi0eVKxM+mkKsCb+FOOK+ZpsGABqOr0izDhXs4dH+blBk5wxGDRHSXzmq YbtNQsB2eReLcK6hMPJ9ZV1CXpjW9vG2M+W9n4cEoymziw8TaP59JJpN2OAA+3IRog0R bBPA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-13792-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-13792-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=aculab.com Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id e3-20020a17090301c300b001d4396bec33si14324087plh.469.2023.12.31.13.54.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Dec 2023 13:54:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-13792-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-13792-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-13792-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=aculab.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id EFE212826E9 for ; Sun, 31 Dec 2023 21:54:35 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 02EFDE558; Sun, 31 Dec 2023 21:54:30 +0000 (UTC) X-Original-To: linux-kernel@vger.kernel.org Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.85.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABEC9DF42 for ; Sun, 31 Dec 2023 21:54:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-254-BKaLleaNN2m97ul2XwiUmA-1; Sun, 31 Dec 2023 21:54:24 +0000 X-MC-Unique: BKaLleaNN2m97ul2XwiUmA-1 Received: from AcuMS.Aculab.com (10.202.163.6) by AcuMS.aculab.com (10.202.163.6) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Sun, 31 Dec 2023 21:54:02 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Sun, 31 Dec 2023 21:54:02 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: [PATCH next v2 3/5] locking/osq_lock: Use node->prev_cpu instead of saving node->prev. Thread-Topic: [PATCH next v2 3/5] locking/osq_lock: Use node->prev_cpu instead of saving node->prev. Thread-Index: Ado8M93l5L/nOR+URXCdpqS8EUheiQ== Date: Sun, 31 Dec 2023 21:54:02 +0000 Message-ID: <7906aaa73f93493c873e6286c1f96645@AcuMS.aculab.com> References: <2b4e8a5816a742d2bd23fdbaa8498e80@AcuMS.aculab.com> In-Reply-To: <2b4e8a5816a742d2bd23fdbaa8498e80@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable node->prev is only used to update 'prev' in the unlikely case of concurrent unqueues. This can be replaced by a check for node->prev_cpu changing and then calling decode_cpu() to get the changed 'prev' pointer. node->cpu (or more particularly) prev->cpu is only used for the osq_wait_next() call in the unqueue path. Normally this is exactly the value that the initial xchg() read from lock->tail (used to obtain 'prev'), but can get updated by concurrent unqueues. Both the 'prev' and 'cpu' members of optimistic_spin_node are now unused and can be deleted. Signed-off-by: David Laight --- kernel/locking/osq_lock.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index eb8a6dfdb79d..27324b509f68 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -13,9 +13,8 @@ */ =20 struct optimistic_spin_node { -=09struct optimistic_spin_node *next, *prev; +=09struct optimistic_spin_node *next; =09int locked; /* 1 if lock acquired */ -=09int cpu; /* encoded CPU # + 1 value */ =09int prev_cpu; /* encoded CPU # + 1 value */ }; =20 @@ -91,10 +90,9 @@ bool osq_lock(struct optimistic_spin_queue *lock) =09struct optimistic_spin_node *node =3D this_cpu_ptr(&osq_node); =09struct optimistic_spin_node *prev, *next; =09int curr =3D encode_cpu(smp_processor_id()); -=09int old; +=09int prev_cpu; =20 =09node->next =3D NULL; -=09node->cpu =3D curr; =20 =09/* =09 * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -102,13 +100,12 @@ bool osq_lock(struct optimistic_spin_queue *lock) =09 * the node fields we just initialised) semantics when updating =09 * the lock tail. =09 */ -=09old =3D atomic_xchg(&lock->tail, curr); -=09if (old =3D=3D OSQ_UNLOCKED_VAL) +=09prev_cpu =3D atomic_xchg(&lock->tail, curr); +=09if (prev_cpu =3D=3D OSQ_UNLOCKED_VAL) =09=09return true; =20 -=09node->prev_cpu =3D old; -=09prev =3D decode_cpu(old); -=09node->prev =3D prev; +=09node->prev_cpu =3D prev_cpu; +=09prev =3D decode_cpu(prev_cpu); =09node->locked =3D 0; =20 =09/* @@ -174,9 +171,16 @@ bool osq_lock(struct optimistic_spin_queue *lock) =20 =09=09/* =09=09 * Or we race against a concurrent unqueue()'s step-B, in which -=09=09 * case its step-C will write us a new @node->prev pointer. +=09=09 * case its step-C will write us a new @node->prev_cpu value. =09=09 */ -=09=09prev =3D READ_ONCE(node->prev); +=09=09{ +=09=09=09int new_prev_cpu =3D READ_ONCE(node->prev_cpu); + +=09=09=09if (new_prev_cpu =3D=3D prev_cpu) +=09=09=09=09continue; +=09=09=09prev_cpu =3D new_prev_cpu; +=09=09=09prev =3D decode_cpu(prev_cpu); +=09=09} =09} =20 =09/* @@ -186,7 +190,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) =09 * back to @prev. =09 */ =20 -=09next =3D osq_wait_next(lock, node, prev->cpu); +=09next =3D osq_wait_next(lock, node, prev_cpu); =09if (!next) =09=09return false; =20 @@ -198,8 +202,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) =09 * it will wait in Step-A. =09 */ =20 -=09WRITE_ONCE(next->prev_cpu, prev->cpu); -=09WRITE_ONCE(next->prev, prev); +=09WRITE_ONCE(next->prev_cpu, prev_cpu); =09WRITE_ONCE(prev->next, next); =20 =09return false; --=20 2.17.1 - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1= PT, UK Registration No: 1397386 (Wales)