Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 19 May 2023 12:18:04 +0200
From:   luca abeni <luca.abeni@santannapisa.it>
To:     Vineeth Remanan Pillai <vineeth@bitbyteword.org>
Cc:     Juri Lelli <juri.lelli@redhat.com>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        Joel Fernandes <joel@joelfernandes.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        Valentin Schneider <vschneid@redhat.com>,
        Jonathan Corbet <corbet@lwn.net>, linux-kernel@vger.kernel.org,
        linux-doc@vger.kernel.org
Subject: Re: [PATCH v3 2/5] sched/deadline: Fix reclaim inaccuracy with SMP
Message-ID: <20230519121804.6c85a3ed@luca64>
In-Reply-To: <20230519115621.2b3f75e2@luca64>
References: <20230515025716.316888-1-vineeth@bitbyteword.org>
        <20230515025716.316888-3-vineeth@bitbyteword.org>
        <20230515100616.33ba5dd9@luca64>
        <CAO7JXPgq8V5yHM6F2+iXf4XJ9cyT30Hn4ot5b2k7srjsaPc3JQ@mail.gmail.com>
        <20230516093729.0771938c@luca64>
        <CAO7JXPh5uLV4QjAEi6bJXfAGSsZ=XsnCyzrvKS8m35BGbRPYJw@mail.gmail.com>
        <20230519115621.2b3f75e2@luca64>
Organization: Scuola Superiore S. Anna
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?QJypVBtA/5CpZCgiKyPMYR25Ii0ROZL7wYxclYUX8ZVfmhkGpE6TRiK2579A?=
 =?us-ascii?Q?U8Kc2KjQ9F/q8UZWHOzSQtYJNLxmxx4EsCrh7cX4jZeQvPMZPXcvT8ItqyTb?=
 =?us-ascii?Q?uHnqHVtvmYXuWROM8Oy6UQ4557M5xBsD6JaDGxLAA6gm42GPY03l0UGRcSdD?=
 =?us-ascii?Q?wIBex7ovHwD54O/5Gx1rqEwdg66efq1oAJ3jVqR9ifcBODt7c7LT8lzDx4aA?=
 =?us-ascii?Q?9MtfXoVzyVBzf1PZf7UiibhqxRvzlAuDf4SNsQzMwYSAxNEhG6y6SAfVg22R?=
 =?us-ascii?Q?20DVQv2J9CeFiVbQo+HyaTiuVwFKfsZMOkrAYgxQ9L237iUYqULvBcZFYKcf?=
 =?us-ascii?Q?rSOhRpeCWwlL8KHSEoygsGHAPVAzt58o0ji4zaF1eZawXIlBuE067ssRidh5?=
 =?us-ascii?Q?/Z7UTYSQJpmySpiSZ6QAcS59ghFo6CBFcp9EskSdKWGDLMsfbm6Rlz0r/PQM?=
 =?us-ascii?Q?0dRqyz7DZQu8s8N6Rzfba+pl5+T4Z1o3L17mdom57Mgvtz/wll5P5dogEJuz?=
 =?us-ascii?Q?jQ5RvIsUqs6mSk2441AqOmYiXuUFikDdlcMzUKcgofOdZkUcyaHZT5luzqYa?=
 =?us-ascii?Q?uG6DWCfIR/mQ9ehEOQwcAOa4SaeZ1/BS8CNqcw4E5MyDUFjSPIiUhHAPE6Ts?=
 =?us-ascii?Q?X4oWSUPpAWInuCqnpDOjRS+Yy4APj6r17KIkk9EWsej0ha5brP0iGu9s00x2?=
 =?us-ascii?Q?CqdsnGRmPGRn8gSogX1dCK5Dxvt+qUQA1rVz0h753ry/d6YQ8YgrQWrCH1iy?=
 =?us-ascii?Q?kFK5lc+5hbjXnP3HsV/NP0ObnsdTe3e3vsqet4i0uB3kbtm2ekOv2+nvd3p4?=
 =?us-ascii?Q?yi9HG0+xxEXeU2xfcpkG5wbz/JMhQlR94l110ybXu2SpYfhFkZ6cJ8ujTkVj?=
 =?us-ascii?Q?RCJZ/AgxsYx51pC0o8X5mkjrcSNb44KKMRz65LLP3CBMYGkBB9n5tihNIQFS?=
 =?us-ascii?Q?WIQ55pRBpTy3/pCLLyob215he6xN68uzXZKCKCX6JO9ibL5J68m1g1yWoMhj?=
 =?us-ascii?Q?wCeNnOmF22BKAtFK6y/UcFcVAATL54yo8zC9UU0IJS5eFlNmNucKj7VoWxcQ?=
 =?us-ascii?Q?o2F4tx2yTXgwnp6vozx4IIFBOjQr7fUF7iR24rHEBSeGH4FWIIPWbkhtiWNN?=
 =?us-ascii?Q?w1LefFodk2S9FVYKiM57zgCsbT9vz45el5sXCImVwUQ/occeKRTdQt9p/JS5?=
 =?us-ascii?Q?lCl72pmUhoZwZ5E7c10vhQGOV0BrmgKqHDn/IbkLcRE2J7z6yFJW0Mq/OFpl?=
 =?us-ascii?Q?b6ht2BOA4nraKyt8a6ZB3SwsVQDHw6eHqnhlpVScXFqsn79oEsTcRLusivCg?=
 =?us-ascii?Q?NDuedjYQKMYUJTK4ChFT44NWktZJRR3iAMY/xWgEek1MxZ0aBqOTuyggudaX?=
 =?us-ascii?Q?WDSZqyIJ392J+QOccHKfstTQUFVpVJaxoozmwtTkHbkQDdSKQmoCUkyve7zE?=
 =?us-ascii?Q?G3lSfgcDRE0rBbLq1kSO909vBQM3wekm2Moi3LQRuK89bY4ChzL/THmgHFCN?=
 =?us-ascii?Q?ZMyd5L9wzFivpdlsxD9R1gPRbeztC0o6xf1gMO8mc+lQjB3Bdhf2nNdtYK3E?=
 =?us-ascii?Q?XCghh94ZYXXrZ2UN5ylY+kPXWXMG5r4oT7hC/CYgXkbm+xA/ZLAsUJ8DX6qc?=
 =?us-ascii?Q?mw=3D=3D?=
X-OriginatorOrg: santannapisa.it
X-MS-Exchange-CrossTenant-Network-Message-Id: d8258d4d-b7b1-4e0d-7afb-08db58525704
X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB2880.eurprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 May 2023 10:18:07.5035
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: d97360e3-138d-4b5f-956f-a646c364a01e
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: m+TmyVDzdJEznWobY89nOY6iMRmLZn6euTQ+CzcUOyX8GLy0YH2HQfUzos5ZRu5SZ8mnv87C7ToHkr76pSgvnurZkcVH3z4Iq73Elicded4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS4PR03MB8697
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, 19 May 2023 11:56:21 +0200
luca abeni <luca.abeni@santannapisa.it> wrote:

> Hi,
> 
> sorry for returning on this discussion, but there is something I still
> do not understand:
> 
> On Tue, 16 May 2023 11:08:18 -0400
> Vineeth Remanan Pillai <vineeth@bitbyteword.org> wrote:
> [...]  
> > I had tested this and it was reclaiming much less compared to the
> > first one. I had 3 tasks with reservation (3,100) and 3 cpus.  
> 
> So, just to confirm: here you have only 3 SCHED_DEADLINE tasks,
> scheduled on a root domain containing only 3 CPUs (dl_bw_cpus() return
> 3)... Right?
> So, the utilization of each task is 3/100 = 0.03 and Uextra is
> 1 - (0.03 * 3) / 3 = 0.97.

OK, sorry again... I found my error immediately after sending the email.
Uextra is computed as "Umax - ...", not "1 - ...".
So, I now understand where the 35% comes from.

I now _suspect_ the correct equation should be
	dq = -(max{u_i / Umax, (Umax - Uinact - Uextra)}) * dt
but I want to test it before wasting your time again; I'll write more
after performing some more tests.


			Luca

> And since all the tasks are always active, Uinact = 0...
> Is this understanding right?
> 
> If so:
> > With dq = -(max{u_i, (Umax - Uinact - Uextra)} / Umax) * dt (1)
> > TID[636]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 95.08
> > TID[635]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 95.07
> > TID[637]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 95.06
> > 
> > With dq = -(max{u_i, (1 - Uinact - Uextra)} / Umax) * dt (2)
> > TID[601]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 35.65
> > TID[600]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 35.65
> > TID[602]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 35.65  
> 
> Here, we should have
> 	dq = -(max{0.03, (1 - 0 - 0.97)} / Umax) * dt
> 	   = -(0.03 / Umax) * dt
> which reclaims up to Umax... So, the utilization should be 95%
> Since you measured 35.65%, it means that (1-Uextra) is much larger
> than 0.97... So, maybe you found some bug in the Uextra computation?
> 
> Can you try printing the extra_bw value, to check what happened?
> 
> 
> 
> 			Thanks,
> 				Luca
> 
> > 
> > As the task bandwidth goes higher, equation (2) reclaims more, but
> > equation (2) is a constant of 95% as long as number of tasks less
> > than cpus. If the number of tasks is more than cpus, eq (2) fares
> > better in reclaiming than eq (1)
> > 
> > eq (1) with 5 tasks (3,100)
> > TID[627]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 28.64
> > TID[626]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 28.64
> > TID[629]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 28.62
> > TID[628]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 29.00
> > TID[630]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 28.99
> > 
> > Here top shows 3 cpus in the range ~45 to 50% util
> > 
> > eq (2) with 5 tasks (3,100)
> > TID[667]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 57.20
> > TID[670]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 57.79
> > TID[668]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 57.11
> > TID[666]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 56.34
> > TID[669]: RECLAIM=1, (r=3ms, d=100ms, p=100ms), Util: 55.82
> > 
> > And here top shows all 3 cpus with 95% util
> >   
> > > I'll write more about this later... And thanks for coping with all
> > > my comments!
> > >    
> > Thanks :-)
> > 
> > Vineeth  
>