Received: by 2002:a05:6512:2355:0:0:0:0 with SMTP id p21csp5521190lfu; Mon, 28 Mar 2022 15:54:37 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxrZVqyLr3MySblq8Ej62Ppm8WkTv8+sJAPlCuoJ4QixIkrugzYWuXgEB6St/kQDAy5bsqr X-Received: by 2002:a17:90b:1b0c:b0:1c7:2f63:dc7f with SMTP id nu12-20020a17090b1b0c00b001c72f63dc7fmr1346914pjb.233.1648508076901; Mon, 28 Mar 2022 15:54:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1648508076; cv=none; d=google.com; s=arc-20160816; b=aIDjErKl+ge6mtj73eBkGENTm8x5mIDzEzw9rFzYGgO3KCMLA/Pi5o3Qdu8rSPJW4L Jfa95RGCDmSzC7CdT7d1xuueOkyzPcHW8bSGtG6KKXstXDwEdcGNbc4yFAXNQoVSpS/e yOHi4dYiP57wxlhwCMuCp2MVjIrOUf5wQaH9JFoYbhZkLhCD4nxD6wtRAzeS0gL3eizJ eLoZimUsi6UVaDaPVe7/BCpUTWc2a328oj8Uw/Hcrw5OUxBxtQyfgEz7MVGW1neKpiHF x8AKhU8f5UiabT11LdDlTh9A9Y4xN4hm5UOJ6Jes3QeCiEFUmGz+Us/uC20thzzAE1hn ln4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=95nKNgevFsf3OTaWIgnK0HfP9o1yVst5IbB2ri3UMto=; b=aq2zjSBZ2wZF6SxMiwClrc4ngj9qcP/w1aHvuOZfUSWN4i0CaCjtRf8Czhb6+/HYpX aY7iiHozeSjgtPygAkAs+YnNoPqaplHkO2hcHGYJIftbpo0awxWS6AKeuJaBhweKfS2g Xx6YZGspPoGBhafFvb0pDf/5y/aBC26GfVCkerZ+0HDq7y2kKkphjpDuzkrDgaw9DRh3 x2Ls/DQofXhEPBPYzdjAsLzhRHHZw4ChBaLlNQ6qMT9AgFl6yKi9bw27WO3h2UAiVDxH iETaQgH9KCWodnhZWsySLicpaeDfQ9jQEcv4snqhh1BCzjbRBBBoBDqcoPjEJ8TpYkCU KF3w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ZQN4dBWo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [2620:137:e000::1:18]) by mx.google.com with ESMTPS id x7-20020a170902b40700b00153b4bc4360si14287520plr.483.2022.03.28.15.54.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Mar 2022 15:54:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ZQN4dBWo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 800BB2816AF; Mon, 28 Mar 2022 14:59:26 -0700 (PDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240649AbiC1LW3 (ORCPT + 99 others); Mon, 28 Mar 2022 07:22:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240771AbiC1LVd (ORCPT ); Mon, 28 Mar 2022 07:21:33 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF0225642F; Mon, 28 Mar 2022 04:19:03 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 59472B80E01; Mon, 28 Mar 2022 11:19:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E8ABEC340EC; Mon, 28 Mar 2022 11:18:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1648466341; bh=ie0Slw/MMRYXG+sa3rLyt1veHee/wbNfzo6+Xc7M9fQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZQN4dBWoNkdJz1WnkIK6iAhQOOSgjw1Q3ep0cOv89lp6hwWJ74VuCLg5nryxaOaRR VNSAa36vWrULD22m8w81RulDMEP28osRKxVjknPVaIIE1j4sxoDK8W1J9VKtfIzRjO cZAX9i+dKVjLznJDOA+lbXIy3O6hV6zXFw5mf/4UcHfeyOAV+Z5TV63u8ihYREDN0I 76NeeeAx9VuoTDqoccoOVhpTMHKSJYLGtj98rvDo8nWPjL4w+7ozxycQ4bs3Lz2jFU etPk7cT26wtrU1OgwIaIwqw7OQpvP91YDgPWfrJh/2nyHfDti4L4K+G205iMgTIogs zBJmJ752qFe7A== From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: "Jason A. Donenfeld" , Sebastian Andrzej Siewior , Dominik Brodowski , Eric Biggers , Andy Lutomirski , =?UTF-8?q?Jonathan=20Neusch=C3=A4fer?= , Sasha Levin , tytso@mit.edu Subject: [PATCH AUTOSEL 5.17 17/43] random: remove batched entropy locking Date: Mon, 28 Mar 2022 07:18:01 -0400 Message-Id: <20220328111828.1554086-17-sashal@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220328111828.1554086-1-sashal@kernel.org> References: <20220328111828.1554086-1-sashal@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-stable: review X-Patchwork-Hint: Ignore Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: "Jason A. Donenfeld" [ Upstream commit 77760fd7f7ae3dfd03668204e708d1568d75447d ] Rather than use spinlocks to protect batched entropy, we can instead disable interrupts locally, since we're dealing with per-cpu data, and manage resets with a basic generation counter. At the same time, we can't quite do this on PREEMPT_RT, where we still want spinlocks-as- mutexes semantics. So we use a local_lock_t, which provides the right behavior for each. Because this is a per-cpu lock, that generation counter is still doing the necessary CPU-to-CPU communication. This should improve performance a bit. It will also fix the linked splat that Jonathan received with a PROVE_RAW_LOCK_NESTING=y. Reviewed-by: Sebastian Andrzej Siewior Reviewed-by: Dominik Brodowski Reviewed-by: Eric Biggers Suggested-by: Andy Lutomirski Reported-by: Jonathan Neuschäfer Tested-by: Jonathan Neuschäfer Link: https://lore.kernel.org/lkml/YfMa0QgsjCVdRAvJ@latitude/ Signed-off-by: Jason A. Donenfeld Signed-off-by: Sasha Levin --- drivers/char/random.c | 55 ++++++++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 27 deletions(-) diff --git a/drivers/char/random.c b/drivers/char/random.c index 882f78829a24..34ee34b30993 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -1876,13 +1876,16 @@ static int __init random_sysctls_init(void) device_initcall(random_sysctls_init); #endif /* CONFIG_SYSCTL */ +static atomic_t batch_generation = ATOMIC_INIT(0); + struct batched_entropy { union { u64 entropy_u64[CHACHA_BLOCK_SIZE / sizeof(u64)]; u32 entropy_u32[CHACHA_BLOCK_SIZE / sizeof(u32)]; }; + local_lock_t lock; unsigned int position; - spinlock_t batch_lock; + int generation; }; /* @@ -1894,7 +1897,7 @@ struct batched_entropy { * point prior. */ static DEFINE_PER_CPU(struct batched_entropy, batched_entropy_u64) = { - .batch_lock = __SPIN_LOCK_UNLOCKED(batched_entropy_u64.lock), + .lock = INIT_LOCAL_LOCK(batched_entropy_u64.lock) }; u64 get_random_u64(void) @@ -1903,67 +1906,65 @@ u64 get_random_u64(void) unsigned long flags; struct batched_entropy *batch; static void *previous; + int next_gen; warn_unseeded_randomness(&previous); + local_lock_irqsave(&batched_entropy_u64.lock, flags); batch = raw_cpu_ptr(&batched_entropy_u64); - spin_lock_irqsave(&batch->batch_lock, flags); - if (batch->position % ARRAY_SIZE(batch->entropy_u64) == 0) { + + next_gen = atomic_read(&batch_generation); + if (batch->position % ARRAY_SIZE(batch->entropy_u64) == 0 || + next_gen != batch->generation) { extract_crng((u8 *)batch->entropy_u64); batch->position = 0; + batch->generation = next_gen; } + ret = batch->entropy_u64[batch->position++]; - spin_unlock_irqrestore(&batch->batch_lock, flags); + local_unlock_irqrestore(&batched_entropy_u64.lock, flags); return ret; } EXPORT_SYMBOL(get_random_u64); static DEFINE_PER_CPU(struct batched_entropy, batched_entropy_u32) = { - .batch_lock = __SPIN_LOCK_UNLOCKED(batched_entropy_u32.lock), + .lock = INIT_LOCAL_LOCK(batched_entropy_u32.lock) }; + u32 get_random_u32(void) { u32 ret; unsigned long flags; struct batched_entropy *batch; static void *previous; + int next_gen; warn_unseeded_randomness(&previous); + local_lock_irqsave(&batched_entropy_u32.lock, flags); batch = raw_cpu_ptr(&batched_entropy_u32); - spin_lock_irqsave(&batch->batch_lock, flags); - if (batch->position % ARRAY_SIZE(batch->entropy_u32) == 0) { + + next_gen = atomic_read(&batch_generation); + if (batch->position % ARRAY_SIZE(batch->entropy_u32) == 0 || + next_gen != batch->generation) { extract_crng((u8 *)batch->entropy_u32); batch->position = 0; + batch->generation = next_gen; } + ret = batch->entropy_u32[batch->position++]; - spin_unlock_irqrestore(&batch->batch_lock, flags); + local_unlock_irqrestore(&batched_entropy_u32.lock, flags); return ret; } EXPORT_SYMBOL(get_random_u32); /* It's important to invalidate all potential batched entropy that might * be stored before the crng is initialized, which we can do lazily by - * simply resetting the counter to zero so that it's re-extracted on the - * next usage. */ + * bumping the generation counter. + */ static void invalidate_batched_entropy(void) { - int cpu; - unsigned long flags; - - for_each_possible_cpu(cpu) { - struct batched_entropy *batched_entropy; - - batched_entropy = per_cpu_ptr(&batched_entropy_u32, cpu); - spin_lock_irqsave(&batched_entropy->batch_lock, flags); - batched_entropy->position = 0; - spin_unlock(&batched_entropy->batch_lock); - - batched_entropy = per_cpu_ptr(&batched_entropy_u64, cpu); - spin_lock(&batched_entropy->batch_lock); - batched_entropy->position = 0; - spin_unlock_irqrestore(&batched_entropy->batch_lock, flags); - } + atomic_inc(&batch_generation); } /** -- 2.34.1