Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
References: <20231107215742.363031-1-ankur.a.arora@oracle.com>
 <20231107215742.363031-42-ankur.a.arora@oracle.com>
 <20231108093645.GL8262@noisy.programming.kicks-ass.net>
 <87fs1gk0bm.fsf@oracle.com>
User-agent: mu4e 1.4.10; emacs 27.2
From:   Ankur Arora <ankur.a.arora@oracle.com>
To:     Ankur Arora <ankur.a.arora@oracle.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        linux-kernel@vger.kernel.org, tglx@linutronix.de,
        torvalds@linux-foundation.org, paulmck@kernel.org,
        linux-mm@kvack.org, x86@kernel.org, akpm@linux-foundation.org,
        luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com,
        hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com,
        vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de,
        jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com,
        boris.ostrovsky@oracle.com, konrad.wilk@oracle.com,
        jgross@suse.com, andrew.cooper3@citrix.com, mingo@kernel.org,
        bristot@kernel.org, mathieu.desnoyers@efficios.com,
        geert@linux-m68k.org, glaubitz@physik.fu-berlin.de,
        anton.ivanov@cambridgegreys.com, mattst88@gmail.com,
        krypton@ulrich-teichert.org, rostedt@goodmis.org,
        David.Laight@aculab.com, richard@nod.at, mjguzik@gmail.com
Subject: Re: [RFC PATCH 41/86] sched: handle resched policy in resched_curr()
In-reply-to: <87fs1gk0bm.fsf@oracle.com>
Date:   Mon, 20 Nov 2023 22:31:27 -0800
Message-ID: <87wmuby5vk.fsf@oracle.com>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ptfskRLv1WBRDprd56L5YXzRtlxceFNB7HpgySpL0NoTsGtVvyVRTbkUg9hu?=
 =?us-ascii?Q?ojcK/Y5UIToAQLOmFp9iZslSkRxwwMZ7A+y6aKTJfCmz+kPadSV3JK/zHYZu?=
 =?us-ascii?Q?Vr5mpcoiw93d3kUZFMXY/i72FS68TVoIH1nvg7HczmsXctjDYfxmtcJI6481?=
 =?us-ascii?Q?NZl+CsuAF2G+ygUkWTy5Aykxh57yIlieD7khguYG5EM1o9d9qSqwJtwzwH8k?=
 =?us-ascii?Q?yBPono7NrNfuCiWRJ724LvsXrdr0oNRctr+RV2e3xN5bcPGm0XU4K9LzpOvL?=
 =?us-ascii?Q?PCkCFYG2/D8FuUnW8Hh9E4ZxDkKnlgfx0sx8KLL8AgPfH0EF/g5ltQoDhAV+?=
 =?us-ascii?Q?w3zS7xwmwNa+I+Jnbr8nH+5/3RfddYmaE9gXFhwPbdCgyikbejfS3Ygn6gTL?=
 =?us-ascii?Q?db8hbe855f9//k4HGlwlXZVwprwhJkY1XdNrFTPrm3vNhfWEcRdQu+Bh7i4N?=
 =?us-ascii?Q?zz8CHQ/CPiJA8bImfVUUX6f6ZTPrSbIFAtitn1WRTPoJ0cxr5bL89NLWxLR4?=
 =?us-ascii?Q?9XMw/ca6e3Foq9Uc+3vpTrWhNCHUrFhqWmVByLknPBdlscRdBdVWHshZm8et?=
 =?us-ascii?Q?+jd9Tmj76K2CeYLZGA6muIqjkQ3OaOn4LQ1lKFliRRgBUoINMJ6LsAJeBZyT?=
 =?us-ascii?Q?t4X934xMcDTxapd7RxK53q40gkuccQZcrc8X62sSNPCRNfnfgVACpCKwq5R0?=
 =?us-ascii?Q?qMwc3x493TAiIkKCrXovUBfkUTHZiqErzpSiQsVIOw0cUwmPP4PV3JKmW8uA?=
 =?us-ascii?Q?X/+FIf9LXgCg0Kle7Ppn/3e+0SaQOmuYiqW+MBS0T1cbNBYs3DVdYsWtKWYN?=
 =?us-ascii?Q?eWAhOcc+C5I2MnnqW5B7iNL6HE9NQhPnetlgiDc5UQaXRo3LKsa7sa1lqOqK?=
 =?us-ascii?Q?wCijcns3iUnYYeEYTlrmtWYgaRBSpDIinBnYb3CVLcvBECh7wV0CrOo6GFqk?=
 =?us-ascii?Q?t0JYvaprpzknq0HHdYOjZZ54JyMp8hqYVyvdgKCUVByCYHZGzMEjeYc9fHpJ?=
 =?us-ascii?Q?drJjTgWcrRR2ysHMlnQrtNVazPEXl4DRSSqECSt9fWp98sfw2DLgKcZPek7+?=
 =?us-ascii?Q?gkNnDlZIpYUzP3K/Cya38kVBK8oIqFImE5epp3sqdvJblNOaJAK/vOayYrIs?=
 =?us-ascii?Q?9K20pFG5AisNPNSB/PDq4/mcASvlxiJoIOx+zx6KVdlBh2vsurRhhM3Vd8kB?=
 =?us-ascii?Q?0o3d+xKQdksMBJAcaxZ8Y6cKrD2hkH5HvunsNC4EFZKyzf0tZr3Paiaz/hZ4?=
 =?us-ascii?Q?ORm4cwFe234unrJYLPNBn2Bdta/PySYV7jPUsv6Eyu9zCMkV99Cc9g6TvEEP?=
 =?us-ascii?Q?cBb1m/otPgoXINnLTawFQgNUj6RfIBpQPP34kUyeoshgm1B20ARl0PftCOBt?=
 =?us-ascii?Q?er3TrjEu2fJPZCNpHGYBRTHokA+zrj0ZCr5SkY0+Lavfrzj/rySUS2wwWa6z?=
 =?us-ascii?Q?IVNm1W6I5y84atHX+Q38+6t/NRXBSy2CvSccn+DAub+MyoyCpu66dB7XovJU?=
 =?us-ascii?Q?Yp7ct7WKqBJW3RUAafWNuk6pyCfYlfmWeiqvonWEjk2JnLVf/OVjs5bASxHm?=
 =?us-ascii?Q?hlEEkhVfW0zthQIlVorQ+NTyjNGVfp8bGPwfAAGJRJZ1RS/CVd0phSJ4RNl1?=
 =?us-ascii?Q?RQ=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?unZPiBf3b5p4+srdW2YqXMTSvufgJKoVN3oimT66Ghg3Tl6MPm8zSC1RwL5Y?=
 =?us-ascii?Q?fIDmFwfd5fnGvqT1O5u0/4qShmSvhTRTzjV8KZNVZXbvqAM+I6X2UNCEp7ZE?=
 =?us-ascii?Q?YRlUjFK7RqrrsA6XMrxTcCEYGVoj7TjreHmlBdok0Jz+zN8owlSVbcpYj6wp?=
 =?us-ascii?Q?qTOudA2Dgo32H7jVeI9YFXZ04on+vQS5txhKbxg8lOrQ7I+dhtejWGmbz42A?=
 =?us-ascii?Q?fqeRNAkrstvsMqSCkBFlJRY87aYgMlJp8wft8hAo7Z37D0P6afSdNQ/u+kUR?=
 =?us-ascii?Q?DMnp+BCFEj4T0DZqAl0TWQZv2ymoiYpUKHt4qVte6j71g+hdLf/UB29Vi9L8?=
 =?us-ascii?Q?w+TXIyD+qtNQncywBU7YzJMYOgADdTb7l+yPQvxRa37q6aqvUfV0FvwNrA6I?=
 =?us-ascii?Q?2KFR946FQ9n/4ya+T9K4iPU4wRRqOTrRxT6drFJlqyoI5aMYkD7V2wczmIfe?=
 =?us-ascii?Q?UMKpJg8B8ie/eU8ZKjun4wWS7uDBg3B0jsAcNR34PtIUSF7SBAwL6H6GfZcS?=
 =?us-ascii?Q?jHmU/25JpRGVyU3kred550X8ffBpZmzl2zlZ0ePZXVU/6PcYobeMbs5nYuYv?=
 =?us-ascii?Q?cJyc1oXFyFFPhuJ4gFbYMnt9DKRqOasRukVqDJbE7uKhdY6S9zjqHSv2sHvV?=
 =?us-ascii?Q?xmY5DnToiYmJQXqwlNeMHmBd0uF12Ln4sl0t2UN/67+uy4q/UrGwPP9t75cj?=
 =?us-ascii?Q?LiRFQRxbKdinB6Vfvy2c/z72S1hSjTE/zKLjXxkuXTmXNCj6lu84CxVG7aPA?=
 =?us-ascii?Q?EyT7x+UhkpQifK2LjHpuU5M8eiLieTgZx2NsYcb6G2KlrML5ZxCa7zgRK185?=
 =?us-ascii?Q?IuGKJQH8yzImyCQw5ajBukQUHa8kyTIRnMkxcO98VchYzpLWyhPmjvtI38Pz?=
 =?us-ascii?Q?e5vJGbFNheFH3UQbtowdyQdUN/tf4YzpIaZA0h7ZpKQSCBCOcVz5lNojqLV2?=
 =?us-ascii?Q?zgCbrswJjGxbykmtTe8P1Bfzj+OG7jQReY5TB+roU01CGz+T0sPLoP37Eau1?=
 =?us-ascii?Q?pE+EyHzRBNJRwPgr921LgepZAYjHurKiQrm2y7R7iHApf18uof+R/3BWp99V?=
 =?us-ascii?Q?LD3msSqR5oi6hbva64fWsY3+BZeURGS75d9ZJs3xzjmUybJhotPeERDvc+Yk?=
 =?us-ascii?Q?ilGI4qsXoCnyzH3IhNTf9OvZ5lMKk8q+9sS5LjONXXelXs9EytNKfgtLRmq3?=
 =?us-ascii?Q?jeOdWz5oG7RQSIfYhSHHfItcMiV5X58gSkLbAyVVA++fqmK4igTfg/IMKI0O?=
 =?us-ascii?Q?lL3YbZcEBiXwM7a6o53cJv+Ez59Mk/DpLM17UgFCNe59K61O26e7etzuJ8K7?=
 =?us-ascii?Q?lo//s1DeREJbeYYIT8mXmGAWOv4fPY+k+aWfNJ8ArmR+9tHe448/3rVW12+g?=
 =?us-ascii?Q?SdWUOMADL5f3CQAR047v6wF+9ebbrURa0RaPskEPDwKi3loogXy38PDg04BB?=
 =?us-ascii?Q?rIbBLg70uUxGDVHUyQs+2YryiTBSS7Tvf3Pkl2kIH/nUswf4IyR+RhNJSoeB?=
 =?us-ascii?Q?KjX+Ent4LK5KbM3NjrnDmzhCgKm13o/PhKyvA0TDBAOlcJAeavFEhdC3gQ?=
 =?us-ascii?Q?=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d09d981b-74c8-4491-c26b-08dbea5b7f34
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Nov 2023 06:31:30.1981
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: qEErQQObK6ogDONA9rgH1dX8591aZYVWoSbdoTyBkbS6g0NWj3EiDOc6q/rwnVoqn8utg5xpaYh/aVFrSdn5zUirWRaWWJlw6yNgl7XvaIA=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB6733
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.987,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-11-21_03,2023-11-20_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 malwarescore=0
 spamscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311060000
 definitions=main-2311210049
X-Proofpoint-GUID: -peT4AevBI0UX2xUPi-VduhMyyHHELeN
X-Proofpoint-ORIG-GUID: -peT4AevBI0UX2xUPi-VduhMyyHHELeN
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 20 Nov 2023 22:32:53 -0800 (PST)


Ankur Arora <ankur.a.arora@oracle.com> writes:

> Peter Zijlstra <peterz@infradead.org> writes:
>
>> On Tue, Nov 07, 2023 at 01:57:27PM -0800, Ankur Arora wrote:
>>
>>> +	 * We might race with the target CPU while checking its ct_state:
>>> +	 *
>>> +	 * 1. The task might have just entered the kernel, but has not yet
>>> +	 * called user_exit(). We will see stale state (CONTEXT_USER) and
>>> +	 * send an unnecessary resched-IPI.
>>> +	 *
>>> +	 * 2. The user task is through with exit_to_user_mode_loop() but has
>>> +	 * not yet called user_enter().
>>> +	 *
>>> +	 * We'll see the thread's state as CONTEXT_KERNEL and will try to
>>> +	 * schedule it lazily. There's obviously nothing that will handle
>>> +	 * this need-resched bit until the thread enters the kernel next.
>>> +	 *
>>> +	 * The scheduler will still do tick accounting, but a potentially
>>> +	 * higher priority task waited to be scheduled for a user tick,
>>> +	 * instead of execution time in the kernel.
>>> +	 */
>>> +	context = ct_state_cpu(cpu_of(rq));
>>> +	if ((context == CONTEXT_USER) ||
>>> +	    (context == CONTEXT_GUEST)) {
>>> +
>>> +		rs = RESCHED_eager;
>>> +		goto resched;
>>> +	}
>>
>> Like said, this simply cannot be. You must not rely on the remote CPU
>> being in some state or not. Also, it's racy, you could observe USER and
>> then it enters KERNEL.
>
> Or worse. We might observe KERNEL and it enters USER.
>
> I think we would be fine if we observe USER: it would be upgrade
> to RESCHED_eager and send an unnecessary IPI.
>
> But if we observe KERNEL and it enters USER, then we will have
> set the need-resched-lazy bit which the thread might not see
> (it might have left exit_to_user_mode_loop()) until the next
> entry to the kernel.
>
> But, yes I would like to avoid the ct_state as well. But
> need-resched-lazy only makes sense when the task on the runqueue
> is executing in the kernel...

So, I discussed this with Thomas offlist, and he pointed out that I'm
overengineering this.

If we decide to wake up a remote rq lazily with (!sched_feat(TTWU_QUEUE)),
and if the target is running in user space, then the resched would
happen when the process enters kernel mode.

That's somewhat similar to how in this preemption model we let a task
run for upto one extra tick while in kernel mode. So I'll drop this and
allow the same behaviour in userspace instead of solving it in
unnecessarily complicated ways.

--
ankur