Received: by 2002:a05:7412:b995:b0:f9:9502:5bb8 with SMTP id it21csp5716453rdb; Sun, 31 Dec 2023 13:53:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IFIL9FqNLEFkBh79v8FwbEALpzyIoK6kQYVv4hGBWrMoNCt9UoFs846w4HTRE4aoqaBTE/R X-Received: by 2002:a05:6e02:188f:b0:360:fd1:b30b with SMTP id o15-20020a056e02188f00b003600fd1b30bmr13724798ilu.5.1704059605367; Sun, 31 Dec 2023 13:53:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704059605; cv=none; d=google.com; s=arc-20160816; b=GCGTjjhzX6qmuXtNFil6GWzXRNBEmX4niS9p4Y7Xd5DA0P/yEODAxhtn0ZMPR8xmDh tGqJSP5NwR4G+pgzhf/kZiV5ldafMCvrhFcRlK27AGS9bgo6qBqD2Q2slP4bTWr5RmvX rZNLWcTqC/U2K5xrP59XBlMl96OJzB8n56GruQkVb+mmHiIJ0LcS39KirepPWZUs+GRH JE7ZBMbLFuNCxbcvcJQMcvxJbt0iQV9HO/eCEAPcwvQr1pxA43DWtXvTyjfkhOpwO+Dd 8NXEHPTReR/aVU1/0rxdHmdkhVPOP0Vh3HzcaG/tfumQiCnlTUUfRoVA2Di4tr2oD3zd 9oNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:accept-language :in-reply-to:references:message-id:date:thread-index:thread-topic :subject:cc:to:from; bh=R5ZtJV4gA2k0+zc+IY7TvZeXeKlgbidQgEzg/e/GAkE=; fh=rK8F4zm15J6Jc7pqIJYnBw4d8zHrqB7RdoX7pamyocA=; b=Ecm8r1hXUBtY4g5sp0wabTxBtzpCygt2j/olrXzMdB48X8XZhuZg9zcxR+fgqz8FEN fakKTbkS/7W2zFnG0unCin7zM8HqL9X/L47scp3JkzV5qPAeWEcKVRLFmoa6NLxZIHim exRmy/hLMltxPKTBeu9vBg/4fN2kTu2TbhtdXkIxoqOp2qWXT56TAT6byHz1Qw8G/x4D R4jFhCTNks2TPMJQdW4+tsa+z0VB4L2GYafe9+eaNEHAbySMltFLxIdVd+CycXbsQTRh m7DPegE9yyBCTsS89pBdVKoF533pzhRsIm/hmfqBaSnSRxSCA8mdQPoKuKXn+nNzWRN8 iNVA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-13791-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-13791-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=aculab.com Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id jd11-20020a170903260b00b001d4bd1f8647si60290plb.416.2023.12.31.13.53.25 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Dec 2023 13:53:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-13791-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-13791-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-13791-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=aculab.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 0AA242826D9 for ; Sun, 31 Dec 2023 21:53:25 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8527CC8EE; Sun, 31 Dec 2023 21:53:19 +0000 (UTC) X-Original-To: linux-kernel@vger.kernel.org Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.86.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5A62CC8CA for ; Sun, 31 Dec 2023 21:53:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-29-UXCaayHdPeixgRLFCuK6JQ-1; Sun, 31 Dec 2023 21:53:13 +0000 X-MC-Unique: UXCaayHdPeixgRLFCuK6JQ-1 Received: from AcuMS.Aculab.com (10.202.163.6) by AcuMS.aculab.com (10.202.163.6) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Sun, 31 Dec 2023 21:52:51 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Sun, 31 Dec 2023 21:52:51 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: [PATCH next v2 2/5] locking/osq_lock: Optimise the vcpu_is_preempted() check. Thread-Topic: [PATCH next v2 2/5] locking/osq_lock: Optimise the vcpu_is_preempted() check. Thread-Index: Ado8M69TGuBFtEJaSr+qwMS7CJLhkw== Date: Sun, 31 Dec 2023 21:52:51 +0000 Message-ID: <3a9d1782cd50436c99ced8c10175bae6@AcuMS.aculab.com> References: <2b4e8a5816a742d2bd23fdbaa8498e80@AcuMS.aculab.com> In-Reply-To: <2b4e8a5816a742d2bd23fdbaa8498e80@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable The vcpu_is_preempted() test stops osq_lock() spinning if a virtual cpu is no longer running. Although patched out for bare-metal the code still needs the cpu number. Reading this from 'prev->cpu' is a pretty much guaranteed have a cache miss when osq_unlock() is waking up the next cpu. Instead save 'prev->cpu' in 'node->prev_cpu' and use that value instead. Update in the osq_lock() 'unqueue' path when 'node->prev' is changed. This is simpler than checking for 'node->prev' changing and caching 'prev->cpu'. Signed-off-by: David Laight --- kernel/locking/osq_lock.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index e0bc74d85a76..eb8a6dfdb79d 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -14,8 +14,9 @@ =20 struct optimistic_spin_node { =09struct optimistic_spin_node *next, *prev; -=09int locked; /* 1 if lock acquired */ -=09int cpu; /* encoded CPU # + 1 value */ +=09int locked; /* 1 if lock acquired */ +=09int cpu; /* encoded CPU # + 1 value */ +=09int prev_cpu; /* encoded CPU # + 1 value */ }; =20 static DEFINE_PER_CPU_SHARED_ALIGNED(struct optimistic_spin_node, osq_node= ); @@ -29,11 +30,6 @@ static inline int encode_cpu(int cpu_nr) =09return cpu_nr + 1; } =20 -static inline int node_cpu(struct optimistic_spin_node *node) -{ -=09return node->cpu - 1; -} - static inline struct optimistic_spin_node *decode_cpu(int encoded_cpu_val) { =09int cpu_nr =3D encoded_cpu_val - 1; @@ -110,9 +106,10 @@ bool osq_lock(struct optimistic_spin_queue *lock) =09if (old =3D=3D OSQ_UNLOCKED_VAL) =09=09return true; =20 -=09node->locked =3D 0; +=09node->prev_cpu =3D old; =09prev =3D decode_cpu(old); =09node->prev =3D prev; +=09node->locked =3D 0; =20 =09/* =09 * osq_lock()=09=09=09unqueue @@ -144,7 +141,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) =09 * polling, be careful. =09 */ =09if (smp_cond_load_relaxed(&node->locked, VAL || need_resched() || -=09=09=09=09 vcpu_is_preempted(node_cpu(node->prev)))) +=09=09=09=09 vcpu_is_preempted(READ_ONCE(node->prev_cpu) - 1))) =09=09return true; =20 =09/* unqueue */ @@ -201,6 +198,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) =09 * it will wait in Step-A. =09 */ =20 +=09WRITE_ONCE(next->prev_cpu, prev->cpu); =09WRITE_ONCE(next->prev, prev); =09WRITE_ONCE(prev->next, next); =20 --=20 2.17.1 - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1= PT, UK Registration No: 1397386 (Wales)