Received-SPF: pass (google.com: domain of linux-kernel+bounces-152152-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <6b2487403dafb09fbdfb0075123fc3fda8ab7636.camel@gmx.de>
Subject: Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
From: Mike Galbraith <efault@gmx.de>
To: Peter Zijlstra <peterz@infradead.org>, mingo@redhat.com, 
 juri.lelli@redhat.com, vincent.guittot@linaro.org,
 dietmar.eggemann@arm.com,  rostedt@goodmis.org, bsegall@google.com,
 mgorman@suse.de, bristot@redhat.com,  vschneid@redhat.com,
 linux-kernel@vger.kernel.org
Cc: "kprateek.nayak" <kprateek.nayak@amd.com>, tglx@linutronix.de
Date: Sat, 20 Apr 2024 07:57:21 +0200
In-Reply-To: <20240405110010.631664251@infradead.org>
References: <20240405102754.435410987@infradead.org>
	 <20240405110010.631664251@infradead.org>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.42.4 
Precedence: bulk
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
UI-OutboundReport: notjunk:1;M01:P0:toGnqYiqh2M=;UdCW03HA+SW2hkNU6yJbikJDfKd
 R0llb77tczDkenXy5gSHIJgf54UaIlQA7Q2U1oqb5+mZwLyPzVyMh0jR5vlSCl9dHRenN2b9a
 qUm471hqTyU7sB9wktILlNV1Z6TsxLcNDURxOuHYjhxXOtFeK+BOkc0hrNjt4xRMDwYYmJ2O3
 audpWF7tOYHqr81MBb6OpHdRJbAQ86+9Vy2uvS9OfVmLhIh9Vl/j/cOevpp7k6VthsEdOuVSP
 f1HPnb4KYtNj5ByKFtZti96Ts5qNWp2/nRB/LYBotcgv0BmmmEebGpx0OEd8srUzO0Yo856Hs
 f4JFGqza/rvSdewk0/ore5IoyZHDkbqY5pmGio9lUoJm+LPHW8/pe/RlQJXsVojzLRsslpTam
 ZN5Rc0+w/8cv8gRFo60vkVQKOAMN/51HLj2Bu/f8wzX93mc+tpcvOSpczuuyQfkroP/eQnqUe
 rxKwiHXnnq4cNObOPxvnDw4BlinTiSeuv2IeRWbkn2Bx2V5V/aXSV8j5amhaT2zJdVVxkSwHG
 mInMSstIyAPD4dpnrd44FEiivfGewgY2dYaBIHY4ztMUOiIwpRq3icGMeww+HzQ1C9NU5BKse
 RXFPDlVkdW2WOEQMFPcrZCv4Ef2im5vofLNCvL5aQ8yDBMGOWSa6F1bhz5Hm9lE/CdgOvhvvi
 8OJ47GAB4FnJ6I1/sLLSyDT6xCMxt/Q3bnuagdM5Z7MTHKuy21oEJ9yMKsR/Vh77yFUrLvjVn
 aYZEvOSbrqAs/phrmOyoe53RK/BxTCkg1/kLg2svn6g54xKMJT+n+AbZWolFFmdFGa/eosTTL
 g5VBEoCbTaRgKK9VIfqJad7E/8/T2xd6VEUEjucD3cvok=

(removes apparently busted bytedance.com address and retries xmit)

Greetings!

With this version, the better CPU distribution (for tbench synchronous
net-blasting) closed the CFS vs EEVDF throughput deficit.  I verified
both by rolling the previous version forward and back-porting to 6.1
where I've got CFS and EEVDF to re-compare, now with both dequeue delay
patch versions.

As usual, there will be winners and losers, but (modulo dead buglet) it
looks kinda promising to me.

Distribution of single pinned buddy pair measured in master:
DELAY_DEQUEUE
 -------------------------------------------------------------------------=
---------------------------------
  Task                  |   Runtime ms  | Switches | Avg delay ms    | Max=
 delay ms    | Sum delay ms     |
 -------------------------------------------------------------------------=
---------------------------------
  tbench:(2)            |   6277.099 ms |  1597104 | avg:   0.003 ms | max=
:   0.129 ms | sum: 4334.723 ms |
  tbench_srv:(2)        |   5724.971 ms |  1682629 | avg:   0.001 ms | max=
:   0.083 ms | sum: 2076.616 ms |
 -------------------------------------------------------------------------=
---------------------------------
  TOTAL:                |  12021.128 ms |  3280275 |                 |    =
    1.729 ms |      6425.483 ms |
 -------------------------------------------------------------------------=
---------------------------------
client/server CPU distribution ~52%/48%

NO_DELAY_DEQUEUE
 -------------------------------------------------------------------------=
---------------------------------
  Task                  |   Runtime ms  | Switches | Avg delay ms    | Max=
 delay ms    | Sum delay ms     |
 -------------------------------------------------------------------------=
---------------------------------
  tbench:(2)            |   6724.774 ms |  1546761 | avg:   0.002 ms | max=
:   0.409 ms | sum: 2443.549 ms |
  tbench_srv:(2)        |   5275.329 ms |  1571688 | avg:   0.002 ms | max=
:   0.086 ms | sum: 2734.151 ms |
 -------------------------------------------------------------------------=
---------------------------------
  TOTAL:                |  12019.641 ms |  3119000 |                 |    =
 9996.367 ms |     15187.833 ms |
 -------------------------------------------------------------------------=
---------------------------------
client/server CPU distribution ~56%/44%

Note switches and delay sum.  For tbench, they translate directly to
throughput.  The other shoe lands with async CPU hog net-blasters, for
those, scheduler cycles tends to be wasted cycles.

	-Mike