Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
Date:   Tue, 10 Oct 2023 10:08:31 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Phil Auld <pauld@redhat.com>
CC:     Biju Das <biju.das.jz@bp.renesas.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Mike Galbraith <efault@gmx.de>,
        Marek Szyprowski <m.szyprowski@samsung.com>,
        "bsegall@google.com" <bsegall@google.com>,
        "bristot@redhat.com" <bristot@redhat.com>,
        "chris.hyser@oracle.com" <chris.hyser@oracle.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "dietmar.eggemann@arm.com" <dietmar.eggemann@arm.com>,
        "gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
        "joel@joelfernandes.org" <joel@joelfernandes.org>,
        "joshdon@google.com" <joshdon@google.com>,
        "juri.lelli@redhat.com" <juri.lelli@redhat.com>,
        "kprateek.nayak@amd.com" <kprateek.nayak@amd.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "mingo@kernel.org" <mingo@kernel.org>,
        "patrick.bellasi@matbug.net" <patrick.bellasi@matbug.net>,
        Pavel Machek <pavel@ucw.cz>, "pjt@google.com" <pjt@google.com>,
        "qperret@google.com" <qperret@google.com>,
        "qyousef@layalina.io" <qyousef@layalina.io>,
        "rostedt@goodmis.org" <rostedt@goodmis.org>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "tim.c.chen@linux.intel.com" <tim.c.chen@linux.intel.com>,
        "timj@gnu.org" <timj@gnu.org>,
        "vincent.guittot@linaro.org" <vincent.guittot@linaro.org>,
        "youssefesmat@chromium.org" <youssefesmat@chromium.org>,
        "mgorman@suse.de" <mgorman@suse.de>,
        "linux-renesas-soc@vger.kernel.org" 
        <linux-renesas-soc@vger.kernel.org>
Subject: Re: [PATCH] sched/fair: fix pick_eevdf to always find the correct se
Message-ID: <ZSSyH6pCu+v10xsB@chenyu5-mobl2.ccr.corp.intel.com>
References: <553e2ee4-ab3a-4635-a74f-0ba4cc03f3f9@samsung.com>
 <867f5121d7d010cacf938c293f862b0cea560ec2.camel@gmx.de>
 <20231006140042.GG36277@noisy.programming.kicks-ass.net>
 <e2645a659b6fc1c592e2f2dc45273c361a2d2f47.camel@gmx.de>
 <20231006155501.GH36277@noisy.programming.kicks-ass.net>
 <20231006192445.GE743@noisy.programming.kicks-ass.net>
 <ZSCoLllO/cYnnIMi@chenyu5-mobl2.ccr.corp.intel.com>
 <OS0PR01MB59222333D50ADBAD1786DA9A86C8A@OS0PR01MB5922.jpnprd01.prod.outlook.com>
 <ZSD9wsBPUfhTULf7@chenyu5-mobl2.ccr.corp.intel.com>
 <20231009142740.GA437913@lorien.usersys.redhat.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20231009142740.GA437913@lorien.usersys.redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?MFQV5dkvSZIu9wo5nNUoa5tm7m+0+k14Hfv3o+71Y3RXl4bXPYqYkFwolJCJ?=
 =?us-ascii?Q?f52TQ9MM+eglEeiMZkLmovRfup2U3pWEzD8fvfhJtGaGhk1M9P93xjKhOovP?=
 =?us-ascii?Q?eTb91sqwYKJ+05LeFU2F8FViCzxGZXbQLhMzP49IBkAi7BdAnHQabH3ZAiY/?=
 =?us-ascii?Q?2VScJElgK6QvwwsS3OtZSU0vQyMjo8acshx0BxpOJ6FILB9xgyg3GQ2fNwwg?=
 =?us-ascii?Q?Stt3TZtx7WWeuiW/4QHUz435IjoUFhwFD/8+FG3hfU0Xi0x+YbfpLlvg16Tg?=
 =?us-ascii?Q?WrFAOU/KpepEiBxizxQUpRms0vRjDBmOiVqz5MTR2E0ApTpI2FKnlipE7/N4?=
 =?us-ascii?Q?QEXfqDDbgA56ktBBdy4WcYCunUvMPiU45rC8EXaCIsMVIEDpYD+ENYXnv+Oe?=
 =?us-ascii?Q?88eDd0t8sdGhfxpiJY7E9CvSjcMZtHDjrUt3O2KhpswR5Inx5Dql6LTI0rQf?=
 =?us-ascii?Q?9fc/zVQOiweoUhLCZ3xVSTFAYI1sxBYLjZIPE7UbrPxyoHGaPH1fenPeF9r3?=
 =?us-ascii?Q?SVNcZNCzEzkj0CRyF/ut1S7kIrWWcRjZt+29M3OgyMzN3+a19qoShu7OBsb+?=
 =?us-ascii?Q?jo0yUJVvpvEyN3RWJnXBo8k2oZczlIzLyf81Rh8P1ECI/vBtDyVl5d9N9jkz?=
 =?us-ascii?Q?1xzDJp5od+8YzYXSXN6ULpBvG+xPW5+y16qvj98ogBjo1xNOlWorVHIO598I?=
 =?us-ascii?Q?039rOIaRPWgvse61kWhuYC87i4+utGe7ZYM8HXEQUgdzXsEI/9MT46XXUEsw?=
 =?us-ascii?Q?ZRGiqiiFeVysauCdjIDMjj1nonmteWSlZAByA6ZHaqWSUh53rqB0ftMFoAXp?=
 =?us-ascii?Q?eAGggn3ibel1Uv6w1yUHSzol4QqjLAugEtudBxxNAUVolkhh0HMjtf5D5tQV?=
 =?us-ascii?Q?ctnbIQ7GMqwDrNKU7RP1KfIpvOUIfa8psN7iMOt5YHJzuqTEJcjuL6CUrxk0?=
 =?us-ascii?Q?jT8Et+rAbSQpye3k9ZVzHGhrKaZrlE+2hjn8vwqueBUJHwPelIxc0kgo1A8f?=
 =?us-ascii?Q?9u/uYMSRfB1yWnDtjpeP1huUfl+UcYoHvaTreYSYCCovgEmuUwqNghGXNp08?=
 =?us-ascii?Q?CogwFE9NcGskAcKDoGfO+KR3pmdGXErkbi9jBiGREi20592n2m9/mcLpuQ8O?=
 =?us-ascii?Q?hO1AXokENqo7qkZp7qYs8M+C81jNyYyfN28+yfGeBD40BW56BKKMpg9AU3aB?=
 =?us-ascii?Q?AksNel6fVwR0RPz5UCFmI0CMowSm6lp78PeVLpZ2gj/yeQILqeTlyrb3EZAv?=
 =?us-ascii?Q?6SCajr2jWnMWojPf4nZQSuVpWTfLmbBIc0sikl3mJECBVoDih3OSOB2mf0/T?=
 =?us-ascii?Q?ELE1OsPhiE9dnzx2OW7d3FOARN71dkwkX6p3lFz4rFwJqhKttzSIRb0KtMjF?=
 =?us-ascii?Q?CTWVvDVzFG5eP5T4MLDCigpjc6Q0u2Z46/SydTjwkX8ca2IjkoZvi7+ZZf30?=
 =?us-ascii?Q?/C5PVmCwQghDzGtr8yopODS+Yu9bP+zp4Xxt0tcA9rR8S3sjwxrvXTcxTw7+?=
 =?us-ascii?Q?so5/gw/kfb1dgrgSw533X2c0++wus4bbEcHbsv6zsmTuJ5MW6z79ut9TspSZ?=
 =?us-ascii?Q?ZKa62FaM6Mrw1lgqjgVCdRSAyo6XGaLHCQhRUqwQ?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 59d93cc5-3476-4c91-e385-08dbc935d810
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2023 02:08:50.0187
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: AB/W2D5FrGx7qLvDEIOAd5YPk2wgBRywDLsHaX/Y9amyGewwJcvYqrdn8Oyqt9Kxlfo0JIolWekZXdHx7nKnCg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB8231
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 09 Oct 2023 19:09:07 -0700 (PDT)

Hi Phil,

On 2023-10-09 at 10:27:40 -0400, Phil Auld wrote:
> On Sat, Oct 07, 2023 at 02:42:10PM +0800 Chen Yu wrote:
> > Hi Biju,
> > 
> > On 2023-10-07 at 06:26:05 +0000, Biju Das wrote:
> > > Hi Chen Yu,
> > > 
> > > > Subject: Re: [PATCH] sched/fair: fix pick_eevdf to always find the correct
> > > > se
> > > > 
> > > > On 2023-10-06 at 21:24:45 +0200, Peter Zijlstra wrote:
> > > > > On Fri, Oct 06, 2023 at 05:55:01PM +0200, Peter Zijlstra wrote:
> > > > >
> > > > > > And yeah, min_deadline is hosed somehow.
> > > > > >
> > > > > > migration/28-185     [028] d..2.    70.264274: validate_cfs_rq: --- /
> > > > > > migration/28-185     [028] d..2.    70.264277: __print_se:
> > > > ffff88845cf48080 w: 1024 ve: -58857638 lag: 870381 vd: -55861854 vmd: -
> > > > 66302085 E (11372/tr)
> > > > > > migration/28-185     [028] d..2.    70.264280: __print_se:
> > > > ffff88810d165800 w: 25 ve: -80323686 lag: 22336429 vd: -41496434 vmd: -
> > > > 66302085 E (-1//autogroup-31)
> > > > > > migration/28-185     [028] d..2.    70.264282: __print_se:
> > > > ffff888108379000 w: 25 ve: 0 lag: -57987257 vd: 114632828 vmd: 114632828 N
> > > > (-1//autogroup-33)
> > > > > > migration/28-185     [028] d..2.    70.264283: validate_cfs_rq:
> > > > min_deadline: -55861854 avg_vruntime: -62278313462 / 1074 = -57987256
> > > > > >
> > > > > > I need to go make dinner (kids hungry), but I'll see if I can figure
> > > > > > out how this happens...
> > > > >
> > > > > *sigh*, does the below help?
> > > > >
> > > > > ---
> > > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index
> > > > > 04fbcbda97d5..6a670f119efa 100644
> > > > > --- a/kernel/sched/fair.c
> > > > > +++ b/kernel/sched/fair.c
> > > > > @@ -3632,6 +3747,7 @@ static void reweight_entity(struct cfs_rq *cfs_rq,
> > > > struct sched_entity *se,
> > > > >  		 */
> > > > >  		deadline = div_s64(deadline * old_weight, weight);
> > > > >  		se->deadline = se->vruntime + deadline;
> > > > > +		min_deadline_cb_propagate(&se->run_node, NULL);
> > > > >  	}
> > > > >
> > > > >  #ifdef CONFIG_SMP
> > > > 
> > > > Is it because without this patch, the se->deadline is not always synced
> > > > with se->min_deadline, then in pick_eevdf() the following condition could
> > > > not be met, thus we get a NULL candidate and has to pick the leftmost one?
> > > > 	if (se->deadline == se->min_deadline)
> > > > 
> > > > Regarding the circular locking warning triggered by printk, does it mean we
> > > > should not get a NULL candidate from __pick_eevdf() in theory? And besides,
> > > > we should not use printk with rq lock hold?
> > > 
> > > Is it not a useful error log? At least from the initial report Marek Szyprowski doesn't see "EEVDF scheduling fail, picking leftmost" but seen only warning triggered by this in the logs.
> > > 
> > 
> > Yes, it is a useful log. I was trying to figure out the safe scenario to use
> > printk.
> >
> 
> You should not use printk with a rq lock held as some console implementations
> (framebuffer etc) call schedule_work() which loops back into the scheduler
> and bad things happen.
> 
> Some places in the scheduler use printk_deferred() when needed. 
> 
> I think this will be better when the RT printk stuff is fully merged. 
>

I see, got it! Thanks for the education.

thanks,
Chenyu