From:   "Marciniszyn, Mike" <mike.marciniszyn@cornelisnetworks.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Thomas Gleixner <tglx@linutronix.de>
CC:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "Dalessandro, Dennis" <dennis.dalessandro@cornelisnetworks.com>
Subject: Issue with rcu_read_lock and CONFIG_PREEMPT_RCU
Thread-Topic: Issue with rcu_read_lock and CONFIG_PREEMPT_RCU
Thread-Index: AdfsJq8wCEInTfQFSzSfnl9OoxKArg==
Date:   Wed, 8 Dec 2021 11:35:20 +0000
Message-ID: <CH0PR01MB71536FB1BD5ECF16E65CB3BFF26F9@CH0PR01MB7153.prod.exchangelabs.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?RB1AlR/D0leYGG2f7sbivA1aJTx741I5rs4L+1QNwfVlaq73BhxiSDsuJnEK?=
 =?us-ascii?Q?/NQyUOjFGEyoKWGF+5xPtq2wIKJdKGJGtX7KI180OdP97mRuON73oYg8iJIQ?=
 =?us-ascii?Q?paRXqZga+mynNxvCKcZ+4ntItrovyXBlRrX4vXBbfSB3i5uZGH9SXFdUGsPS?=
 =?us-ascii?Q?mS+1uZGY4R2GDd6BvCxpkKRW4XHHXyLk0jkQuH7ZcayVCeN09FQiYMCNIMkT?=
 =?us-ascii?Q?RFIpVLkdk1XHECWh+e0IVPmOJPPAxORDJMeN2GiJZPmRBq3QxFYNz+AePxJo?=
 =?us-ascii?Q?g73++0Dq0JXiHwBbdDzAMaUt0f9jfMPnkVWxhKOuoxpkitcBBmF5ifD76883?=
 =?us-ascii?Q?f6HuAz4LN931sUGaaVDrlCsJmQaZGDTtjD85YDV19SnrVolGzhkgI0oxdt5Z?=
 =?us-ascii?Q?gc463ORfOxgDh0qe4wrOguX4bTanyllsE/z71mghpArnseH0yTS/wAth71+c?=
 =?us-ascii?Q?ABCZF9dqD7wWcDKfvaOx4VCewMErnFxcUCu9yDg/DIek7ewMz9gFZvZgOAyN?=
 =?us-ascii?Q?gSeDAuNbmAttoKiKN653WEEcVBJs9h/LW6sK4lAEsJejCf2fEZnil4x/fPCY?=
 =?us-ascii?Q?qhKxjeONtBX+XXLRuhFn7ymjJZcqSPvQtcKvD0kmgJYlgE/JxIU0h30svAHi?=
 =?us-ascii?Q?P/fmgbcOdv0pAtce3CGCmLSXOVnAbcX/9uPOsl9wz+bF7x3PZDu5ljbdImRU?=
 =?us-ascii?Q?TlmZgmHCWiSOd4RD0NgY2BUZ+v1SETXH+5kjOqIutheZ+f327+pGosVugl+e?=
 =?us-ascii?Q?s6A+HhLg9F3PjZyHMsHE/4Ym5j4HE4XkmLfodNSp1Tdjg5+3Giu59hj5gspD?=
 =?us-ascii?Q?7Q1M9/d5IcLq1fVs/OYnfatYTcte5FNzjB/fkmP3nQ6z69TMdY6m2zXQtLIU?=
 =?us-ascii?Q?Z+geY6/UbklooLRRVy+p2Th0ZHqwdZLcmxCSCa6U6zLXAuMFXYLOXGbrVeqw?=
 =?us-ascii?Q?RuyxUoC8ek/EDDz1U/MbTApmPky5wVEZIoduGSGSTRL7jx/f5wW/9FB50jhF?=
 =?us-ascii?Q?AGm6pqKVbiiuan+BSQMFq4YVpaBE+9zEe8wLmJllWA1sWkzar90SxLoMwwMc?=
 =?us-ascii?Q?xxxhXsuJAOmrUeHKuxLtm7EjJCzKx+0qDy6Gk0LmdNNuT+EkuXQ76WIsVbru?=
 =?us-ascii?Q?bgcdrn1jn8E9p7Ykf5YiSleXsk7ItPq1EurYRzlDSjXEoREcsyo9E7f8OISq?=
 =?us-ascii?Q?V0xgTSN8tOxcka/1WcE1hCBixX+oxLmdxGpMOS7ddgebzM336rDSkLO+J0zt?=
 =?us-ascii?Q?+hVcljXrDrQW2QFXX65QJFAoPJ8TcwnfKysADbc+LxGrbVitHF+fCyW0O4Fm?=
 =?us-ascii?Q?zRy/N3KxTVSvmELKqE2MCDdJTH/XAB/6bDKSjFJIBbHvxxdp7jWthnp67v3u?=
 =?us-ascii?Q?xdpdXj2EgJo9C+mX8Lkqff4nydYQws/VN4EjosqqffgcnIYQjTEtjWY5MKjO?=
 =?us-ascii?Q?d7a3SJt2MdobZkVklPqeBFN6KmfogYlJFs7xmnvqkmZVHkh7rfdFV34J3Uu8?=
 =?us-ascii?Q?8STG/xVJ+PYftX3e9HcTg3tsOtS9dHiNd9IB65WY+8R93xElMBiIna5pOEv7?=
 =?us-ascii?Q?FCNBSDQGO6dff4h/K//fLS0YySs7YbrV524Tm9S5Mxsp/DTG3ppi54ElqAbD?=
 =?us-ascii?Q?TbvhPaZA49HuRxC1IhxoUQOy85mx/EXAnJ2kznZgfpGuV4Nhw/Gjw4ccdzqZ?=
 =?us-ascii?Q?yrP2qOHrzsGPQH1Fyd9D74H4SI1vkJ1PEv8qXzz0u3Q2ewoK?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: cornelisnetworks.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB7153.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9251cd3b-d6c9-46c8-a5dd-08d9ba3ed130
X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Dec 2021 11:35:20.7138
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4dbdb7da-74ee-4b45-8747-ef5ce5ebe68a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 8YeN4WiZcSXSh5QiTbFVrnDti1cUvsssmwVYBxudxkfppsEAMUUWmMU+ff3OQ1qxBvsZj7V6FcoCsnDe06DBRfCz8Gg4JN3yPmIr4s6qll6u8kHtRboURF1BkxBfFsvS
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR01MB6987
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

As part of testing the 5.16 rc series we noticed a new BUG message originat=
ing from check_preemption_disabled().

We submitted a patch to move a call to smp_processor_id() into an rcu criti=
cal section within the same function.

See https://lore.kernel.org/linux-rdma/20211129191958.101968.87329.stgit@aw=
fm-01.cornelisnetworks.com/T/#u.

Much to my surprise, additional testing still sees the BUG!

Additional testing has shown that an explicit preempt_disable()/preempt_ena=
ble() silences the warning when placed around the RCU critical section.

The RCU config is:

#
# RCU Subsystem
#
CONFIG_TREE_RCU=3Dy
CONFIG_PREEMPT_RCU=3Dy
# CONFIG_RCU_EXPERT is not set
CONFIG_SRCU=3Dy
CONFIG_TREE_SRCU=3Dy
CONFIG_TASKS_RCU_GENERIC=3Dy
CONFIG_TASKS_RCU=3Dy
CONFIG_TASKS_RUDE_RCU=3Dy
CONFIG_TASKS_TRACE_RCU=3Dy
CONFIG_RCU_STALL_COMMON=3Dy
CONFIG_RCU_NEED_SEGCBLIST=3Dy
CONFIG_RCU_NOCB_CPU=3Dy
# end of RCU Subsystem


It looks like there is a difference between the checking in check_preemptio=
n_disabled() and the implicit preemption disabling in __rcu_read_lock().

The implicit disable looks like:

static void rcu_preempt_read_enter(void)
{
        WRITE_ONCE(current->rcu_read_lock_nesting, READ_ONCE(current->rcu_r=
ead_lock_nesting) + 1);
}

The checking code uses the x86 define preempt_count():

static __always_inline void __preempt_count_add(int val)
{
        raw_cpu_add_4(__preempt_count, val);
}

An explicit disable uses this x86 code:

static __always_inline void __preempt_count_add(int val)
{
        raw_cpu_add_4(__preempt_count, val);
}

The difference seems to be the use of __preempt_count vs. rcu_read_lock_nes=
ting.

This can't be good...

Mike