Received-SPF: pass (google.com: domain of linux-kernel+bounces-67671-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
References: <20240213055554.1802415-1-ankur.a.arora@oracle.com>
 <a7e785f8-d4c3-4b1b-9abe-36ac0b971e44@paulmck-laptop>
 <87le7mpjpr.fsf@oracle.com>
 <4e070ae0-29dc-41ee-aee6-0d3670304825@paulmck-laptop>
 <0d4a4eec-ce91-48da-91b6-1708a97edaeb@paulmck-laptop>
User-agent: mu4e 1.4.10; emacs 27.2
From: Ankur Arora <ankur.a.arora@oracle.com>
To: paulmck@kernel.org
Cc: Ankur Arora <ankur.a.arora@oracle.com>, linux-kernel@vger.kernel.org,
        tglx@linutronix.de, peterz@infradead.org,
        torvalds@linux-foundation.org, akpm@linux-foundation.org,
        luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com,
        hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com,
        vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de,
        jpoimboe@kernel.org, mark.rutland@arm.com, jgross@suse.com,
        andrew.cooper3@citrix.com, bristot@kernel.org,
        mathieu.desnoyers@efficios.com, glaubitz@physik.fu-berlin.de,
        anton.ivanov@cambridgegreys.com, mattst88@gmail.com,
        krypton@ulrich-teichert.org, rostedt@goodmis.org,
        David.Laight@aculab.com, richard@nod.at, jon.grimm@amd.com,
        bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com
Subject: Re: [PATCH 00/30] PREEMPT_AUTO: support lazy rescheduling
In-reply-to: <0d4a4eec-ce91-48da-91b6-1708a97edaeb@paulmck-laptop>
Date: Thu, 15 Feb 2024 12:53:13 -0800
Message-ID: <878r3lmoue.fsf@oracle.com>
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?w/2HO0xGooI8MCDGVmKrQPfPDCUFup3pRDM7ViFUeD1aEgOI50ucSu2yLNdr?=
 =?us-ascii?Q?KN4gEZdtauwPhkawUGEpZRIiRNTTXjqSII7Kd4Kl8qiF7jZBN4UMG8PIFshz?=
 =?us-ascii?Q?E5OgHTT1pbK4IL33NUOMiEk5vPPaUzODy0+zGDSypPqZVRP9WbL7AEukMftC?=
 =?us-ascii?Q?wT0jIYsiFrMasCLcWNF0QR8MemTIzmiqElRgBVAkl0wjaflCYSu0R5OY4NR+?=
 =?us-ascii?Q?md9U7LtBSCbMngjabngPHr/JyzERH/Edt8MjY3/TnLblvokiPaL/7LPCEZ1D?=
 =?us-ascii?Q?qTqBCYwLl6mxx072ItMmlWLT+T4PsTKDwnWlLRJnisuE1pPNehdpEifhF8HL?=
 =?us-ascii?Q?h5GEozbPGNCZrA+v8kEDPnEAZ/Bvhf9BVpKdi31VbThjcKkRr2y/JZSwmIRP?=
 =?us-ascii?Q?rsDXkv4oNW7oXr39pFhBjwet+nsRbSykX0QaEv/lcBqk0C0xMaKuo5Ijg6sH?=
 =?us-ascii?Q?QcLYEi3DGhx3z8VWDDT+eviHds4VSY7BbH9COXD3CR2VtpE8M+ayTGlXtzMO?=
 =?us-ascii?Q?fmHb7Cjaay6vfVTwn4gaz3bNNchBndFkvEs7vqzGfOFGKnU0kYjtozezpzC2?=
 =?us-ascii?Q?v+ac6v4WiTJMF7b9FE1WVx86fb0ynsCBD5h6rt8eQUMpj0tRWotvwFtYHdo0?=
 =?us-ascii?Q?Qg+EUymStCEc8CVisGVF36JvpcQZ1oiZExqFpq0gU/z41tyV5Vg3gUslzM20?=
 =?us-ascii?Q?OGUCnzkbA1z4IN8BnD/rthVPavJxG3ygi9TrjAhfoLAqXI3B3qWWgdNfd4H7?=
 =?us-ascii?Q?8BmwnyifceWDP16j4TTQ3UZVXgcsMxX3aRt4nJYzf9MiMhTKXSPvpEdcv6tE?=
 =?us-ascii?Q?3ln/sqod38hNbR37FwArbH3vtxuIlIyrzupMDO6vVNizh4eb1D1sOc0zOlTo?=
 =?us-ascii?Q?heAI4K0fWqJVV5PeeDcXlJifrBqeG18+4iArBts7nrrMZB3CSPuPoU6cilCF?=
 =?us-ascii?Q?gGTkLO5t9UPtkZUH0qO8rKDuCTZBWj4Ahzc9jloyhO8pIElIHtmXovTF0qov?=
 =?us-ascii?Q?NivxvaqFXklLu8ncUfVxjv+NFaLbXYtJQFOtWkLOl1VHcNCi9e0MyCyje4oe?=
 =?us-ascii?Q?IkPEFMdmxQs0lSmv/TgMG7nozGU3dkvEohSfHrymCnOZJL2LgHheNVLj7B0i?=
 =?us-ascii?Q?wIo61PuKwjg6J7DSgtTKEasOHLtLxS2FHyrAFIVuEsmv6TxCI7ieG+K+IrNw?=
 =?us-ascii?Q?rttmB6JqTOCUk2usO0W+bsVaqZ4YleqkXDA+cSaU6RcQK5spPVmgH3sI3/7J?=
 =?us-ascii?Q?+nf+KNPhYLhWDmxXd/FtKjdTRzo314H1xWfFLGCaXmaLqWtLnUMjNUp+L81d?=
 =?us-ascii?Q?Iqfyb4z+FXVPHd2rZJuHQN2UvKt9c2fB/wtOfTE2YA7lruMJLELbxqSBQj3C?=
 =?us-ascii?Q?0V5w72ugY1MQwXtJS2fn/p0Rj/1a4c5VU24mW0dp5aWeb6dFqsZOvwhi+vkb?=
 =?us-ascii?Q?5o6Qw/H04CyayGHU8F7eWaG7Tjzr68MVjO3Fd6ZWHW06tACiW6XZvq7p7cm7?=
 =?us-ascii?Q?i0kWQAbAERmAQCWPrSeU7aoYZ7aHesc7jy+B//TAkL7QV/aY5Qw75GtwgAj3?=
 =?us-ascii?Q?UwTbDei9cE91P0krXvDLgRWyMDtmR2R+cRSNrVUCR4/gUGcoJ/mI8FcAcnNx?=
 =?us-ascii?Q?FQ=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	sMPhDGAbpxK9oD42wrKnFTijKa7duKMN4L6Twu4QDTXORI1lC1kp3+fZY5mVlMoBgygAfYYhgoIJev8QWrTzl6EsKKPoLvn9siQ8hoQ/J5Z8RP0jFF8mSI8oSc3RV+WbVyfZbOirRw8Mi552nXwvx0FDf6cDMB6EnxaIX21ZQCWPsFC+4BQY22FzIw1Cv83J+6SjI0CYUJEynKgBjQ43Ip8v6qBTlEmV09dASI0ey8wRH37lmgQR+dvBS1P/OYkZM6qf1JrH1C188xmOXYM1IBDkmHgMyuXQxLpHVzlqyR22hm9+gnor7mGbyOuXUVNnSPRm8bNrALt/lGS2GvYV6rMy3O9Uk1Ue8mji40BJGCIy52jkGxH6dKacywqy0VTZUPIxe3sre878u+pa+u16m1VoO0Va9azVEu57fymkodPJe+EMv39ATlvYwwhw/nZFG0DAc6L6aXfnHI7LC79FHelOeVqHPj+boBpq4DQv//3Nj7mX/9MqZFYxxs1x+BLqTcvjD6wBnHy19047XOO0F9q1OnBfC7nlSA4K54wh0/ngzwNeOqUBGCEskIPAZSpsZYQ0daIgTvpA31i1oNIzLmF8NLgEK52pVbD9YGi/RBE=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9550f981-dae7-45c7-5436-08dc2e6815a0
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2024 20:52:55.6439
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: T+u/U+LEh+ihxgMmOPNnlnSIuGKOYbDiRZSyyUK/Jb2U0sFoc9XQdPpLYO2bOQGRj/oRvLME0kzxgTzUVWqo0CxHp2ER4fxiU0jI9+LB/JY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR10MB7330
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-15_19,2024-02-14_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0
 mlxlogscore=786 bulkscore=0 phishscore=0 mlxscore=0 suspectscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402150167
X-Proofpoint-GUID: fPDpOkHbTfyGxoQ05rQOhnC2Ox6T2n4x
X-Proofpoint-ORIG-GUID: fPDpOkHbTfyGxoQ05rQOhnC2Ox6T2n4x


Paul E. McKenney <paulmck@kernel.org> writes:

> On Wed, Feb 14, 2024 at 07:45:18PM -0800, Paul E. McKenney wrote:
>> On Wed, Feb 14, 2024 at 06:03:28PM -0800, Ankur Arora wrote:
>> >
>> > Paul E. McKenney <paulmck@kernel.org> writes:
>> >
>> > > On Mon, Feb 12, 2024 at 09:55:24PM -0800, Ankur Arora wrote:
>> > >> Hi,
>> > >>
>> > >> This series adds a new scheduling model PREEMPT_AUTO, which like
>> > >> PREEMPT_DYNAMIC allows dynamic switching between a none/voluntary/full
>> > >> preemption model. However, unlike PREEMPT_DYNAMIC, it doesn't depend
>> > >> on explicit preemption points for the voluntary models.
>> > >>
>> > >> The series is based on Thomas' original proposal which he outlined
>> > >> in [1], [2] and in his PoC [3].
>> > >>
>> > >> An earlier RFC version is at [4].
>> > >
>> > > This uncovered a couple of latent bugs in RCU due to its having been
>> > > a good long time since anyone built a !SMP preemptible kernel with
>> > > non-preemptible RCU.  I have a couple of fixes queued on -rcu [1], most
>> > > likely for the merge window after next, but let me know if you need
>> > > them sooner.
>> >
>> > Thanks. As you can probably tell, I skipped out on !SMP in my testing.
>> > But, the attached diff should tide me over until the fixes are in.
>>
>> That was indeed my guess.  ;-)
>>
>> > > I am also seeing OOM conditions during rcutorture testing of callback
>> > > flooding, but I am still looking into this.
>> >
>> > That's on the PREEMPT_AUTO && PREEMPT_VOLUNTARY configuration?
>>
>> On two of the PREEMPT_AUTO && PREEMPT_NONE configurations, but only on
>> two of them thus far.  I am running a longer test to see if this might
>> be just luck.  If not, I look to see what rcutorture scenarios TREE10
>> and TRACE01 have in common.
>
> And still TRACE01 and TREE10 are hitting OOMs, still not seeing what
> sets them apart.  I also hit a grace-period hang in TREE04, which does
> CONFIG_PREEMPT_VOLUNTARY=y along with CONFIG_PREEMPT_AUTO=y.  Something
> to dig into more.
>
> I am also getting these from builds that enable KASAN:
>
> vmlinux.o: warning: objtool: mwait_idle+0x13: call to tif_resched.constprop.0() leaves .noinstr.text section
> vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0x36: call to tif_resched.constprop.0() leaves .noinstr.text section
> vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x18: call to tif_resched.constprop.0() leaves .noinstr.text section
> vmlinux.o: warning: objtool: acpi_safe_halt+0x0: call to tif_resched.constprop.0() leaves .noinstr.text section
> vmlinux.o: warning: objtool: poll_idle+0x33: call to tif_resched.constprop.0() leaves .noinstr.text section
> vmlinux.o: warning: objtool: default_enter_idle+0x18: call to tif_resched.constprop.0() leaves .noinstr.text section

Thanks Paul. Yeah, with KASAN, tif_resched() transforms into this out of
line function:

ffffffff810fec20 <tif_resched.constprop.0>:
ffffffff810fec20:       e8 5b c6 20 00          call   ffffffff8130b280 <__sanitizer_cov_trace_pc>
ffffffff810fec25:       b8 03 00 00 00          mov    $0x3,%eax
ffffffff810fec2a:       e9 71 56 61 01          jmp    ffffffff827142a0 <__x86_return_thunk>
ffffffff810fec2f:       90                      nop

> Does tif_resched() need to be marked noinstr or some such?

Builds fine with Thomas' suggested fix.

--------
diff --git a/include/linux/thread_info.h b/include/linux/thread_info.h
index 8752dbc2dac7..0810ddeb365d 100644
--- a/include/linux/thread_info.h
+++ b/include/linux/thread_info.h
@@ -81,12 +81,12 @@ typedef enum {
  * reduce to the same value (TIF_NEED_RESCHED) leaving any scheduling behaviour
  * unchanged.
  */
-static inline int tif_resched(resched_t rs)
+static __always_inline inline int tif_resched(resched_t rs)
 {
        return TIF_NEED_RESCHED + rs * TIF_NEED_RESCHED_LAZY_OFFSET;
 }

-static inline int _tif_resched(resched_t rs)
+static __always_inline inline int _tif_resched(resched_t rs)
 {
        return 1 << tif_resched(rs);
 }