Received-SPF: pass (google.com: domain of linux-kernel+bounces-191563-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
From: Ankur Arora <ankur.a.arora@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: tglx@linutronix.de, peterz@infradead.org, torvalds@linux-foundation.org,
        paulmck@kernel.org, rostedt@goodmis.org, mark.rutland@arm.com,
        juri.lelli@redhat.com, joel@joelfernandes.org, raghavendra.kt@amd.com,
        sshegde@linux.ibm.com, boris.ostrovsky@oracle.com,
        konrad.wilk@oracle.com, Ankur Arora <ankur.a.arora@oracle.com>
Subject: [PATCH v2 18/35] rcu: force context-switch for PREEMPT_RCU=n, PREEMPT_COUNT=y
Date: Mon, 27 May 2024 17:35:04 -0700
Message-Id: <20240528003521.979836-19-ankur.a.arora@oracle.com>
In-Reply-To: <20240528003521.979836-1-ankur.a.arora@oracle.com>
References: <20240528003521.979836-1-ankur.a.arora@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?KiVuz7KGfvQUoCLbBsB0rfyrJ4n66D8JuYqSbs2+g7tIxpUb8u5AybILOEqo?=
 =?us-ascii?Q?fDRbNMXswwHHJVb7HMnDXxtyzUrQfazZS95vU6eLG2Kt/Ss7zJ9bAxgUGwnF?=
 =?us-ascii?Q?wAX0YjqhCOUmG2YRQvUeMyswug6NHdeuvme1CzpDQC2kGzNrXNUn4AqFIH1l?=
 =?us-ascii?Q?Bje2OXkE+mFq2f/L2KURcZUy4zFugFjdk9SlLdIgffL184GgLgfLsXDae14N?=
 =?us-ascii?Q?TUQneRdGjf8bfbgFl+k7mU0a0H5PC3LvaCoaDB+zWLqOPREld7hwojXUcSbU?=
 =?us-ascii?Q?1jMsz3C/ynP8QccvRiJaqf18Oz9+/a3d3CemZO4qveSctW6QpTvhFl8gL7Tm?=
 =?us-ascii?Q?UmIk2QZNgtFtaospN9zsiEYmI2lvxo8CQv2qFvAHR2CyCoRQELwtdiJ93rrX?=
 =?us-ascii?Q?Sx/zSEI9eTq+EFXrjVRgaNkoeBA8jAv28pBhqeOyXt+hPdGUegITtqkjCpPw?=
 =?us-ascii?Q?61tQkqWnsuHyHX/xPJHSwV7YI0eJbEjhfr5Q3BqoTXfZ0i3UN5cq/j5eFsw1?=
 =?us-ascii?Q?gcBXLhkofRHCF0J6oqE+6AMDYr3aWvZdpkRz448xd3GXxrqY2+fcl9eBgIxO?=
 =?us-ascii?Q?vGfVZ+U/398ZxqaTzsiiwkZ9TSm8kUI9hZAUQjQkJq+We8Ves6C1pKlEPlZj?=
 =?us-ascii?Q?VYURCdpUfICy5R/NNpKirIWDUJYnEwN885ZWzA6ZLSjRN8kVpxFPdvm6EMg5?=
 =?us-ascii?Q?dNmCpYKo3Envrc1GwtnRUeYqt4+6IkKvXqQsjN7DDu3Yo6fKZHAxfeOSoxzr?=
 =?us-ascii?Q?pvGB1xcEzs2pg3l9FbngwQfm0zW+az5yCIY/mGbYG5XZbO8Kx/aFJ6BOgRDi?=
 =?us-ascii?Q?30jNOFNCGCrJOgYw9yPy1BkGxBMaEcg6Jv+yY6p1lDl4Y47QdfMCkG/gYg20?=
 =?us-ascii?Q?MHoe9QihEg12K9NAwL2AkNFjloWMIcikjVOuLmx6RiFgRDnaoMl87zZcmhnV?=
 =?us-ascii?Q?5UzyPEh9vUvRq1bwMShAWN+T2ErvOCX7EOPpQFQNM1XxEcHbCKqn86feraoj?=
 =?us-ascii?Q?FOEQdLyKLps8+2+MAZIFDCuWorQv7ZG7bj/o+0bUjIL5l1RH2vtY6xmvyV4K?=
 =?us-ascii?Q?wUWX6W4+kBV58Zp22uY9eBqMTlETtGkm+snLUCCxBLQvOqwTxoLpgyz445IX?=
 =?us-ascii?Q?JIj36g2xvuhUBkCRCiuj+cfmXS5cx9EIKyQ6nqwqvcOvUu7t0rU5nQveKKkz?=
 =?us-ascii?Q?WYu4MAFddhUmp03LoC5H2dvYUaGTdLABO82v3MsJGXJRRM5RqABmzvKu3FtT?=
 =?us-ascii?Q?Q2b9qdVOTQKenwUsduBJMdnRud343PcJqsb82B+GElYCXLTZI7xZsBU8VgI+?=
 =?us-ascii?Q?Obm0ptCju0dioAxxVTgrjl50ZOBndcZ38gbTyt7mjBZlJAq97XuPgBQ0wpw+?=
 =?us-ascii?Q?owkKZXBb7q07mRR4gBtke67HC8HYjOmie7ifuB6fXEFyfRz9E3yR/gO+yy32?=
 =?us-ascii?Q?cZYWVwvPd76JCwrPMZpEEb1k6mBGWXRHcvHkOw1afLmDk92qOvGd3avKzICX?=
 =?us-ascii?Q?fKqcQtxiYaiM6cg5ph0Xxg0l7gKpyUsE8L7t+AEQn4JjMEoWrv5QN5DWgFI/?=
 =?us-ascii?Q?UhCyBXy1OZDjRmLZFFQPuBVLnXsNBtUSjAm+o+LPdZ/p8f8lJ7vmYeeLxp9N?=
 =?us-ascii?Q?Kw=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	We+a8q3ivFvmEwqR3YL2vdVgY3cWdrY5nuppjzYkBH3xhvU9GNsxTCh3effJmzHa5XmTalTeCq/nJoWayZVfElnDEUr0HAqf9Zn2MoGefZiH/wyV55qXRy0etbt1R+nMrGKu5LJJgnArCrkVpRk5PqD+f+CRdtpp0pyx9P4Pb7IzRpMVWHff0aPOnc4YWOVWTJivBOEhKEZrvJeNLubDrllFkXWxwi3q/CQF7TRu7idEk0iBOCbE4Sp8t/9uV2n8V+1kCivp5LkvUusdPYVsCOYxwl8NbAOBEqilUaSIpADKpMoe7jQ2X0fRd0hYgOt+q65nIdHr62AUoMrML19zmTJK5LzNMwTWThOubFGCmiOKsEIjHL5k0sGsqaZpG3v6G7c852gjpWBpxAyeYRkeE6hdavCkqB46q/a8gF/LtzoQTGYE/5NVBFCPi+mcV2q22qXi5cyUa93Ld3zbzrt6psHjL2+7ttCb6OPKQUW1qiFKZMUbOH8AFgYDtSJc2H3yKsVwxgqQs3cMNjAPe2CylHYrVL/kIQ28wzhK+Lt2F/KmMtR2TAVRRwJ3G48DUVdXRS9hYXoz6jq5Uv4vYRLlbOJx+c2n8IcBKRXHmdE40UI=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: edd9f82c-e770-4e54-8c11-08dc7eae333b
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2024 00:36:22.9397
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8/GshhxQKvofIIM806fk5D4fg8ABsj2wdBp1mA49EiV5zTDCADI/6dySgIaNsYW2NpAeP6qhMSqOWFlsO+j895ZE8n6B1COnXx2Tr6AwUWU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB7177
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16
 definitions=2024-05-27_06,2024-05-27_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 spamscore=0
 phishscore=0 bulkscore=0 malwarescore=0 adultscore=0 mlxscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2405010000 definitions=main-2405280003
X-Proofpoint-GUID: OUn2i8iFRic1sbCoL4Wg1V7zMjXTNq-q
X-Proofpoint-ORIG-GUID: OUn2i8iFRic1sbCoL4Wg1V7zMjXTNq-q

With (PREEMPT_RCU=n, PREEMPT_COUNT=y), rcu_flavor_sched_clock_irq()
registers urgently needed quiescent states when preempt_count() is
available and no task or softirq is in a non-preemptible section.

This, however, does nothing for long running loops where preemption
is only temporarily enabled, since the tick is unlikely to neatly fall
in the preemptible() section.

Handle that by forcing a context-switch when we require a quiescent
state urgently but are holding a preempt_count().

Cc: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 kernel/rcu/tree.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index d9642dd06c25..3a0e1d0b939c 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2286,8 +2286,17 @@ void rcu_sched_clock_irq(int user)
 	raw_cpu_inc(rcu_data.ticks_this_gp);
 	/* The load-acquire pairs with the store-release setting to true. */
 	if (smp_load_acquire(this_cpu_ptr(&rcu_data.rcu_urgent_qs))) {
-		/* Idle and userspace execution already are quiescent states. */
-		if (!rcu_is_cpu_rrupt_from_idle() && !user) {
+		/*
+		 * Idle and userspace execution already are quiescent states.
+		 * If, however, we came here from a nested interrupt in the
+		 * kernel, or if we have PREEMPT_RCU=n but are holding a
+		 * preempt_count() (say, with CONFIG_PREEMPT_AUTO=y), then
+		 * force a context switch.
+		 */
+		if ((!rcu_is_cpu_rrupt_from_idle() && !user) ||
+		     ((!IS_ENABLED(CONFIG_PREEMPT_RCU) &&
+		       IS_ENABLED(CONFIG_PREEMPT_COUNT)) &&
+		     (preempt_count() & (PREEMPT_MASK | SOFTIRQ_MASK)))) {
 			set_tsk_need_resched(current);
 			set_preempt_need_resched();
 		}
-- 
2.31.1