Received: by 2002:ab2:6a05:0:b0:1f8:1780:a4ed with SMTP id w5csp604281lqo; Fri, 10 May 2024 09:07:53 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCWdbG6unAODCJjJFLNmkzmiZ2worWHee8s1SPYYJlzminBw0f/skMsyxb6vCAVMS/Wm0guuZzZPw5oqFnO0+5jb7j8DEg1n1sFyYEiDgg== X-Google-Smtp-Source: AGHT+IE270kancgvVSYOscVcBtKFZ2DQ2kB+wXhUm2JCivfTxj3l+JwlqOG8vBKpLWVT6mvO7589 X-Received: by 2002:a05:6a20:9150:b0:1af:ad46:cd4d with SMTP id adf61e73a8af0-1afde10f38fmr3686148637.32.1715357272953; Fri, 10 May 2024 09:07:52 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1715357272; cv=pass; d=google.com; s=arc-20160816; b=WlNT550nGoWLFVJ13LsfiqpwiZv9r2epyH4AFMAEIL8bpqwk1KzNOrNnFG1l56nhLR o3ctsKr81t8uXhUs21h+dA5ZO7Cnbf5J0LMZWZl+sGieIbVF80mtQnQOWsfhqTDERJ5n OXNnLCLc6ZXpZWvkuBi9vvNKiA9cyqb8lCwcVMihYYv9Makq9o2TVqjN3A+6rsElhCEA GdnJdF1a7wdYTcQ5aHCKEDTc13GOSDNnSvLq/19pMzR/pdT4tsPvqQ0tWfBXAzWbm+j6 CVzzE9/b6HW4DtMYMCKJhVes6N+fd8SRoJu4sxYUDCywNTLyLcpBdHogBQaIYYm5vFo5 Ghmg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-disposition:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=cTuMa2wmFqTn5YhI8kAoEmvUP34ataIz3Y56fKRuABU=; fh=i3zj+TVP45mbl1FpMSUeRdsu3rZlSuNlro4iHe5kow0=; b=dKHHKApm4qPY819OHL5ba7Y8VJ843Swx9DsrKjr2k7mT6OyMXYs94J2ECS+vQ/RBub 0j2L6hnVXaA2FMAjX/et1ygU1YQilyyAcGaElCywNbubgB5U5grl3xOJSS+mFDPFWm9R tiYjsJHAHIWpNqDA2N2PPeo4EbjtcA8I+R55DNyC/FQgtk6Q3zSfMoxmzwF8lE4XeRc5 KpQ1cywFed9muEtgoMm3/LkpUAOsIdPP9Sc3CXVe9LSm67BrZPrucbVsYPzZvlZVhdDF UnhnjbiIwmMMbc9ZprjualKZ/aIyJqHFOCBCc2tV2WAHgzswc7vbBaRDiZgyArhjuCIS JDSQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=L3zljmeo; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-175994-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-175994-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id 41be03b00d2f7-63412c369bdsi3748487a12.744.2024.05.10.09.07.52 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 May 2024 09:07:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-175994-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=L3zljmeo; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-175994-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-175994-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 066E4287B89 for ; Fri, 10 May 2024 16:07:32 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DB0DB172BAA; Fri, 10 May 2024 16:07:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="L3zljmeo" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DC8286AC9 for ; Fri, 10 May 2024 16:07:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715357240; cv=none; b=euRa32OrDSBfzgRieFJnWygGf+0tcTUYXRYB0e708TqrNbotiQhHdKMw4ZnU0k2IJgwgb3rHMFbfhLcVTpixuORmYfqMCdfYvec8sK+AVoOxgLWde+5kUgbqMU1AsrqIV+KV0ZKRdJ/wyfCwfZh0hf/WHGnfwTcGViVHOzhcUbI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715357240; c=relaxed/simple; bh=DGr9dznRCmLYTBRCLo5qomacRQrNmaUbOsxrcWje88U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type:Content-Disposition; b=IwpJ1GlhM1rgjM0Z8Yhj1uxfvePv7/VmJxjwfd6hgGSKh+tdTU/WniW1yHz1Kn5ytekmyF07iZ8s2brKRImgFTeMgMlz+D8cH2vcgWZLHq6Zlq0GO7GSooDLJrnOtbSf2xJS9Ce16s+84Y9PGHpnmPN/Sp3Q5TA+xcUm1TE3hk8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=L3zljmeo; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1715357236; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cTuMa2wmFqTn5YhI8kAoEmvUP34ataIz3Y56fKRuABU=; b=L3zljmeoJPC89/XpvdvyiILDulvmysGZx2Bz17bbdQnh88blP5OVszVyBqpjy/DsYo4Mt/ 2hZ/2GZwFeXZyios+NTqZHrcvC+7K4CIBuDbyDzJDdSI5EmgxqhMqkhJxVpiSLUsQTl9fX Y+Q7q9d36c0MqGs6qqo5iT0JrwFwDyU= Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-425-au_BnVB_Nxi-dYEDP7BqAw-1; Fri, 10 May 2024 12:07:05 -0400 X-MC-Unique: au_BnVB_Nxi-dYEDP7BqAw-1 Received: by mail-pf1-f198.google.com with SMTP id d2e1a72fcca58-6f468ae6a5eso2071736b3a.2 for ; Fri, 10 May 2024 09:07:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715357224; x=1715962024; h=content-transfer-encoding:content-disposition:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cTuMa2wmFqTn5YhI8kAoEmvUP34ataIz3Y56fKRuABU=; b=utZvt+F5m0KzZkVjud+z7oyh2yBCsLkBSwTW99zrZDCM5+E1Ug9n/tFXBs6b05VTMz j0OHB9BYHQ6ZBlz+eSOlFlo2gYycvqrbADhSetWu8iaFicU4Nz5jcTA7vmb1f4XWY+QV CfAFgUvgXVdAJV0F2SU8GtoCJqrbIrgzxMtkbi1sEYIcCoqeAEK9P+XVvc4900TYWOLZ SsB6XYzSBMZ3Mji83knTFzHD4rA7oWLw9YDtPOu8FdXt2xSk1WeVNj4O9KgFtirZ42g/ dRUkebz/Hu/0wleVDx6VKzGitWW38Gx6I+CL7714Lo0mS+BWttZAG0pzvQoIkJg5X6zg t//g== X-Forwarded-Encrypted: i=1; AJvYcCXEoygu7YsRiIkhkXk7MeJPFOFyXt1PaFTkiyRiogBadX0bWzvNciDvZm9/FDwMjA+6OB/Iz5ztG1wx0hDBlc4HNRm6V3o5AumNiHZ8 X-Gm-Message-State: AOJu0YzS9boZm7fXTQcIy84uwUoJV9DqFlYkfevRuoWzYQRIQHJaEY5z SwrwCiDftVtrNWIcfv7wt/P9xQ68K+uNSp4HruuQuPKuKvHT95qQPx8+p2KkWvfU7k55dldiV8m eJn5LxFO4yFG+cOQTeWc8GzeIp55T252BAW0mnVdU7FhrDFcZVppxkeLCKH7W6g== X-Received: by 2002:a05:6a20:da83:b0:1aa:41e4:f1b4 with SMTP id adf61e73a8af0-1afde1b7e10mr3796576637.44.1715357224006; Fri, 10 May 2024 09:07:04 -0700 (PDT) X-Received: by 2002:a05:6a20:da83:b0:1aa:41e4:f1b4 with SMTP id adf61e73a8af0-1afde1b7e10mr3796526637.44.1715357223509; Fri, 10 May 2024 09:07:03 -0700 (PDT) Received: from localhost.localdomain ([2804:1b3:a800:8d87:eac1:dae4:8dd4:fe50]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-6f4d2a86fb3sm3099598b3a.86.2024.05.10.09.06.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 May 2024 09:07:02 -0700 (PDT) From: Leonardo Bras To: "Paul E. McKenney" Cc: Leonardo Bras , Sean Christopherson , Paolo Bonzini , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Marcelo Tosatti , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org Subject: Re: [RFC PATCH v1 0/2] Avoid rcu_core() if CPU just left guest vcpu Date: Fri, 10 May 2024 13:06:40 -0300 Message-ID: X-Mailer: git-send-email 2.45.0 In-Reply-To: <09a8f4f6-a692-4586-bb68-b0a524b7a5d8@paulmck-laptop> References: <5fd66909-1250-4a91-aa71-93cb36ed4ad5@paulmck-laptop> <09a8f4f6-a692-4586-bb68-b0a524b7a5d8@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: 8bit On Thu, May 09, 2024 at 04:45:53PM -0700, Paul E. McKenney wrote: > On Thu, May 09, 2024 at 07:14:18AM -0300, Leonardo Bras wrote: > > On Thu, May 09, 2024 at 05:16:57AM -0300, Leonardo Bras wrote: > > [ . . . ] > > > > Here I suppose something like this can take care of not needing to convert > > > ms -> jiffies every rcu_pending(): > > > > > > + nocb_patience_delay = msecs_to_jiffies(nocb_patience_delay); > > > > > > > Uh, there is more to it, actually. We need to make sure the user > > understands that we are rounding-down the value to multiple of a jiffy > > period, so it's not a surprise if the delay value is not exactly the same > > as the passed on kernel cmdline. > > > > So something like bellow diff should be ok, as this behavior is explained > > in the docs, and pr_info() will print the effective value. > > > > What do you think? > > Good point, and I have taken your advice on making the documentation > say what it does. Thanks :) > > > Thanks! > > Leo > > > > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt > > index 0a3b0fd1910e..9a50be9fd9eb 100644 > > --- a/Documentation/admin-guide/kernel-parameters.txt > > +++ b/Documentation/admin-guide/kernel-parameters.txt > > @@ -4974,20 +4974,28 @@ > > otherwise be caused by callback floods through > > use of the ->nocb_bypass list. However, in the > > common non-flooded case, RCU queues directly to > > the main ->cblist in order to avoid the extra > > overhead of the ->nocb_bypass list and its lock. > > But if there are too many callbacks queued during > > a single jiffy, RCU pre-queues the callbacks into > > the ->nocb_bypass queue. The definition of "too > > many" is supplied by this kernel boot parameter. > > > > + rcutree.nocb_patience_delay= [KNL] > > + On callback-offloaded (rcu_nocbs) CPUs, avoid > > + disturbing RCU unless the grace period has > > + reached the specified age in milliseconds. > > + Defaults to zero. Large values will be capped > > + at five seconds. Values rounded-down to a multiple > > + of a jiffy period. > > + > > rcutree.qhimark= [KNL] > > Set threshold of queued RCU callbacks beyond which > > batch limiting is disabled. > > > > rcutree.qlowmark= [KNL] > > Set threshold of queued RCU callbacks below which > > batch limiting is re-enabled. > > > > rcutree.qovld= [KNL] > > Set threshold of queued RCU callbacks beyond which > > diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h > > index fcf2b4aa3441..62ede401420f 100644 > > --- a/kernel/rcu/tree.h > > +++ b/kernel/rcu/tree.h > > @@ -512,20 +512,21 @@ do { \ > > local_irq_save(flags); \ > > if (rcu_segcblist_is_offloaded(&(rdp)->cblist)) \ > > raw_spin_lock(&(rdp)->nocb_lock); \ > > } while (0) > > #else /* #ifdef CONFIG_RCU_NOCB_CPU */ > > #define rcu_nocb_lock_irqsave(rdp, flags) local_irq_save(flags) > > #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */ > > > > static void rcu_bind_gp_kthread(void); > > static bool rcu_nohz_full_cpu(void); > > +static bool rcu_on_patience_delay(void); > > I don't think we need an access function, but will check below. > > > /* Forward declarations for tree_stall.h */ > > static void record_gp_stall_check_time(void); > > static void rcu_iw_handler(struct irq_work *iwp); > > static void check_cpu_stall(struct rcu_data *rdp); > > static void rcu_check_gp_start_stall(struct rcu_node *rnp, struct rcu_data *rdp, > > const unsigned long gpssdelay); > > > > /* Forward declarations for tree_exp.h. */ > > static void sync_rcu_do_polled_gp(struct work_struct *wp); > > diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h > > index 340bbefe5f65..639243b0410f 100644 > > --- a/kernel/rcu/tree_plugin.h > > +++ b/kernel/rcu/tree_plugin.h > > @@ -5,20 +5,21 @@ > > * or preemptible semantics. > > * > > * Copyright Red Hat, 2009 > > * Copyright IBM Corporation, 2009 > > * > > * Author: Ingo Molnar > > * Paul E. McKenney > > */ > > > > #include "../locking/rtmutex_common.h" > > +#include > > This is already pulled in by the enclosing tree.c file, so it should not > be necessary to include it again. Even better :) > (Or did you get a build failure when > leaving this out?) I didn't, it's just that my editor complained the symbols were not getting properly resolved, so I included it and it was fixed. But since clangd is know to make some mistakes, I should have compile-test'd before adding it. > > > static bool rcu_rdp_is_offloaded(struct rcu_data *rdp) > > { > > /* > > * In order to read the offloaded state of an rdp in a safe > > * and stable way and prevent from its value to be changed > > * under us, we must either hold the barrier mutex, the cpu > > * hotplug lock (read or write) or the nocb lock. Local > > * non-preemptible reads are also safe. NOCB kthreads and > > * timers have their own means of synchronization against the > > @@ -86,20 +87,33 @@ static void __init rcu_bootup_announce_oddness(void) > > if (rcu_kick_kthreads) > > pr_info("\tKick kthreads if too-long grace period.\n"); > > if (IS_ENABLED(CONFIG_DEBUG_OBJECTS_RCU_HEAD)) > > pr_info("\tRCU callback double-/use-after-free debug is enabled.\n"); > > if (gp_preinit_delay) > > pr_info("\tRCU debug GP pre-init slowdown %d jiffies.\n", gp_preinit_delay); > > if (gp_init_delay) > > pr_info("\tRCU debug GP init slowdown %d jiffies.\n", gp_init_delay); > > if (gp_cleanup_delay) > > pr_info("\tRCU debug GP cleanup slowdown %d jiffies.\n", gp_cleanup_delay); > > + if (nocb_patience_delay < 0) { > > + pr_info("\tRCU NOCB CPU patience negative (%d), resetting to zero.\n", > > + nocb_patience_delay); > > + nocb_patience_delay = 0; > > + } else if (nocb_patience_delay > 5 * MSEC_PER_SEC) { > > + pr_info("\tRCU NOCB CPU patience too large (%d), resetting to %ld.\n", > > + nocb_patience_delay, 5 * MSEC_PER_SEC); > > + nocb_patience_delay = msecs_to_jiffies(5 * MSEC_PER_SEC); > > + } else if (nocb_patience_delay) { > > + nocb_patience_delay = msecs_to_jiffies(nocb_patience_delay); > > + pr_info("\tRCU NOCB CPU patience set to %d milliseconds.\n", > > + jiffies_to_msecs(nocb_patience_delay); > > + } > > I just did this here at the end: > > nocb_patience_delay_jiffies = msecs_to_jiffies(nocb_patience_delay); > > Ah, you are wanting to print out the milliseconds after the rounding > to jiffies. That's right, just to make sure the user gets the effective patience time, instead of the before-rounding one, which was on input. > > I am going to hold off on that for the moment, but I hear your request > and I have not yet said "no". ;-) Sure :) It's just something I think it's nice to have (as a user). > > > if (!use_softirq) > > pr_info("\tRCU_SOFTIRQ processing moved to rcuc kthreads.\n"); > > if (IS_ENABLED(CONFIG_RCU_EQS_DEBUG)) > > pr_info("\tRCU debug extended QS entry/exit.\n"); > > rcupdate_announce_bootup_oddness(); > > } > > > > #ifdef CONFIG_PREEMPT_RCU > > > > static void rcu_report_exp_rnp(struct rcu_node *rnp, bool wake); > > @@ -1260,10 +1274,29 @@ static bool rcu_nohz_full_cpu(void) > > > > /* > > * Bind the RCU grace-period kthreads to the housekeeping CPU. > > */ > > static void rcu_bind_gp_kthread(void) > > { > > if (!tick_nohz_full_enabled()) > > return; > > housekeeping_affine(current, HK_TYPE_RCU); > > } > > + > > +/* > > + * Is this CPU a NO_HZ_FULL CPU that should ignore RCU if the time since the > > + * start of current grace period is smaller than nocb_patience_delay ? > > + * > > + * This code relies on the fact that all NO_HZ_FULL CPUs are also > > + * RCU_NOCB_CPU CPUs. > > + */ > > +static bool rcu_on_patience_delay(void) > > +{ > > +#ifdef CONFIG_NO_HZ_FULL > > You lost me on this one. Why do we need the #ifdef instead of > IS_ENABLED()? Also, please note that rcu_nohz_full_cpu() is already a > compile-time @false in CONFIG_NO_HZ_FULL=n kernels. You are right. rcu_nohz_full_cpu() has a high chance of being inlined on if ((...) && rcu_nohz_full_cpu()) And since it returns false, this whole statement will be compiled out, and the new function will not exist in CONFIG_NO_HZ_FULL=n, so there is no need to test it. > > > + if (!nocb_patience_delay) > > + return false; > > We get this automatically with the comparison below, right? Right > If so, we > are not gaining much by creating the helper function. Or am I missing > some trick here? Well, it's a fastpath. Up to here, we just need to read nocb_patience_delay{,_jiffies} from memory. If we don't include the fastpath we have to read jiffies and rcu_state.gp_start, which can take extra time: up to 2 cache misses. I thought it could be relevant, as we reduce the overhead of the new parameter when it's disabled (patience=0). Do you think that could be relevant? Thanks! Leo > > Thanx, Paul > > > + if (time_before(jiffies, READ_ONCE(rcu_state.gp_start) + nocb_patience_delay)) > > + return true; > > +#endif /* #ifdef CONFIG_NO_HZ_FULL */ > > + return false; > > +} > > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c > > index 7560e204198b..7a2d94370ab4 100644 > > --- a/kernel/rcu/tree.c > > +++ b/kernel/rcu/tree.c > > @@ -169,20 +169,22 @@ static int kthread_prio = IS_ENABLED(CONFIG_RCU_BOOST) ? 1 : 0; > > module_param(kthread_prio, int, 0444); > > > > /* Delay in jiffies for grace-period initialization delays, debug only. */ > > > > static int gp_preinit_delay; > > module_param(gp_preinit_delay, int, 0444); > > static int gp_init_delay; > > module_param(gp_init_delay, int, 0444); > > static int gp_cleanup_delay; > > module_param(gp_cleanup_delay, int, 0444); > > +static int nocb_patience_delay; > > +module_param(nocb_patience_delay, int, 0444); > > > > // Add delay to rcu_read_unlock() for strict grace periods. > > static int rcu_unlock_delay; > > #ifdef CONFIG_RCU_STRICT_GRACE_PERIOD > > module_param(rcu_unlock_delay, int, 0444); > > #endif > > > > /* > > * This rcu parameter is runtime-read-only. It reflects > > * a minimum allowed number of objects which can be cached > > @@ -4340,25 +4342,27 @@ static int rcu_pending(int user) > > lockdep_assert_irqs_disabled(); > > > > /* Check for CPU stalls, if enabled. */ > > check_cpu_stall(rdp); > > > > /* Does this CPU need a deferred NOCB wakeup? */ > > if (rcu_nocb_need_deferred_wakeup(rdp, RCU_NOCB_WAKE)) > > return 1; > > > > /* Is this a nohz_full CPU in userspace or idle? (Ignore RCU if so.) */ > > - if ((user || rcu_is_cpu_rrupt_from_idle()) && rcu_nohz_full_cpu()) > > + gp_in_progress = rcu_gp_in_progress(); > > + if ((user || rcu_is_cpu_rrupt_from_idle() || > > + (gp_in_progress && rcu_on_patience_delay())) && > > + rcu_nohz_full_cpu()) > > return 0; > > > > /* Is the RCU core waiting for a quiescent state from this CPU? */ > > - gp_in_progress = rcu_gp_in_progress(); > > if (rdp->core_needs_qs && !rdp->cpu_no_qs.b.norm && gp_in_progress) > > return 1; > > > > /* Does this CPU have callbacks ready to invoke? */ > > if (!rcu_rdp_is_offloaded(rdp) && > > rcu_segcblist_ready_cbs(&rdp->cblist)) > > return 1; > > > > /* Has RCU gone idle with this CPU needing another grace period? */ > > if (!gp_in_progress && rcu_segcblist_is_enabled(&rdp->cblist) && > > > > > > >