Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 29 Mar 2023 21:54:55 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Dietmar Eggemann <dietmar.eggemann@arm.com>
CC:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        "Ben Segall" <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        "Daniel Bristot de Oliveira" <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        "Tim Chen" <tim.c.chen@intel.com>,
        Nitin Tekchandani <nitin.tekchandani@intel.com>,
        Waiman Long <longman@redhat.com>,
        Yu Chen <yu.c.chen@intel.com>, <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH] sched/fair: Make tg->load_avg per node
Message-ID: <20230329135455.GA108864@ziqianlu-desk2>
References: <20230327053955.GA570404@ziqianlu-desk2>
 <943d44f7-1fa9-8545-dc1d-890e4dd20854@arm.com>
 <20230328125624.GA6130@ziqianlu-desk2>
 <76939bf6-1d9d-5f8f-f15c-f03b2322d684@arm.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <76939bf6-1d9d-5f8f-f15c-f03b2322d684@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?LlkE0AiuBsli7yyl/QLQFl7umekEAeXQlb4gq1k0qT5k/MqZxEptcSa72rkm?=
 =?us-ascii?Q?CSeZ8ee32E/vC+9KH/jCyxrHR7FwY1elfdJ6fPv4xWtEpZsgslLcuY2BVSQD?=
 =?us-ascii?Q?ELygKMCWA3yTE+vwwOrEkrUkvfEIMaafQvLFwtKeYs8jSc3gd0c3mNLLBp2V?=
 =?us-ascii?Q?/c3CldHR9+JbuJJJlT9UIEVbl96KIXNweEmJiGcTqTqssX6zYhhphJa7W80V?=
 =?us-ascii?Q?6k2oBK0UGocJ9vyn3uvYKheF8aRkLFIp01VQKM88TaFnke9Uu0/2enORqSRS?=
 =?us-ascii?Q?CNuNJTRiRMZuPJakowGlrUTaMHR/aJbMYDsWaHPzas7ct6L6CW621PLbz0Qt?=
 =?us-ascii?Q?TVDfpo+H2Y+VmQqNVe2oWFAEkMSvC1QoFPT/i5nycq3/gVMBbatNBC3jJ9Ih?=
 =?us-ascii?Q?ruZEFWLQX4lhHuxccCvlzBCBw1SR9w64TEgWgl0cikCoOyMcJX7brq+efOWb?=
 =?us-ascii?Q?eSKarMjHVB67woNIF+3xpRtOTTOmz8XLmwt8aYTQ6K9k5IkwGd4RLLNE5iq+?=
 =?us-ascii?Q?vJrlPJvxGKDVodVmPQTape6HNlXv/yHf05w0QO1wZgLxS8hR8RaCGtJpMeh0?=
 =?us-ascii?Q?ZV7E5Q8Za6oZT17FJv1PQa0qZcYrRZx/RCkYqCher4/3FAqTxYJ/ou8a1mNi?=
 =?us-ascii?Q?nPMOhqGRpfVQ5q7Rn2jpASSRX09405KLBlYH/K5HOSp6KRZJp/yXnMC1osNU?=
 =?us-ascii?Q?3Y5kBCdHI0WztF8Pzuq0QwplYqdpNhLAAU/ZzrxWte2/M4ofcBmmGEui6A4v?=
 =?us-ascii?Q?Y5vy2b2pkugiHW3vk1j0TWWvolEn1YXJcGliZGUZGfg6QEgvyqp4HJt5M05d?=
 =?us-ascii?Q?mMyyLmwQq0NfNHyE2L3Cxlh5pKYnUu/NJlpzEeivBbsx0+fy+n2tQTZtqacn?=
 =?us-ascii?Q?po0mBLhzQ40ppkrQvFCM7FYHGmdVAovyOaonUbL81RFsZE0j4UOfJa+NhRmv?=
 =?us-ascii?Q?l1S49mpI8FoS+FcldOsQdmkJ356EC6TBXlg/gtuKty3zDrwB/ce+dsuGtpai?=
 =?us-ascii?Q?GhsWndRNJedNZhl1dDCfzvayYcFP9iw5VPdvNvzHJOTgrPNqrutSGazeAprD?=
 =?us-ascii?Q?yJOIrTeYqhC3V5sFAz9yRvfvf2plSxc6NWknbU7fUYmJPq+pNYBdFac86qRO?=
 =?us-ascii?Q?3Te8rj7Wa4Exp21GgmuXn0L0mxpzjdlEfJ6b/kQEe47tGU3DB6/PYx27A1xD?=
 =?us-ascii?Q?uIz8gYFHp+H59vJhd2HJ21E/TXJgRIPaGggGZjZAOVf1JYpk+FeUdaj0rc2U?=
 =?us-ascii?Q?hnWER5/cbmZnDLEAt6/FXLzKgC9EnUu5INm/YNDK/uJwDrS+Micnq06Xl7pw?=
 =?us-ascii?Q?SEsBby4E1e+oTN14Ww1qxz7XBjkTgPIKBkAX1+1eRG7yXRsQ7Jpz1QNvomHo?=
 =?us-ascii?Q?ZODCGN0NCJdEaza/KSE+lcYvoIkI7+XAjJtaHAkJr3ZoKHddTLl5qbs7ri9C?=
 =?us-ascii?Q?xogoF96CEq+Td/ZBVA2A+ePDPEOCgMHPmg5ef7lYiAlcRCV7hoLs/HVTUwbO?=
 =?us-ascii?Q?0Y4WHCKswKz7C30YY8j1sG8BbxzfJaEoNQSM65W7bBBsbW33H4KfhoYympTt?=
 =?us-ascii?Q?OT0H4CJ14f9tEzMi1Q90N+GBdzcC9Ks3kNeXPx35?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 131123ae-8635-4a65-2bd7-08db305d3444
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Mar 2023 13:55:07.3304
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YsIusLU0zKXhxho0BMHcI23F4PSQyQlrZcozsS/nLmMp+AMOUj8+l+een1uU/GxotzOWBoMJHZaSQOXTUz1dEQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR11MB6103
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Mar 29, 2023 at 02:36:44PM +0200, Dietmar Eggemann wrote:
> On 28/03/2023 14:56, Aaron Lu wrote:
> > Hi Dietmar,
> > 
> > Thanks for taking a look.
> > 
> > On Tue, Mar 28, 2023 at 02:09:39PM +0200, Dietmar Eggemann wrote:
> >> On 27/03/2023 07:39, Aaron Lu wrote:
> 
> [...]
> 
> > Did you test with a v6.3-rc based kernel?
> > I encountered another problem on those kernels and had to temporarily use
> > a v6.2 based kernel, maybe you have to do the same:
> > https://lore.kernel.org/lkml/20230327080502.GA570847@ziqianlu-desk2/
> 
> No, I'm also on v6.2.
> 
> >> Is your postgres/sysbench running in a cgroup with cpu controller
> >> attached? Mine isn't.
> > 
> > Yes, I had postgres and sysbench running in the same cgroup with cpu
> > controller enabled. docker created the cgroup directory under
> > /sys/fs/cgroup/system.slice/docker-XXX and cgroup.controllers has cpu
> > there.
> 
> I'm running postgresql service directly on the machine. I boot now with
> 'cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1' so I can add the
> cpu controller to:
> 
>   system.slice/system-postgresql.slice/postgresql@11-main.service
> 
> where the 96 postgres threads run and to
> 
>   user.slice/user-1005.slice/session-4.scope
> 
> where the 96 sysbench threads run.
> 
> Checked with systemd-cgls and `cat /sys/kernel/debug/sched/debug` that
> those threads are really running there.
> 
> Still not seeing `update_load_avg` or `update_cfs_group` in perf report,
> only some very low values for `update_blocked_averages`.
> 
> Also added CFS BW throttling to both cgroups. No change.
> 
> Then I moved session-4.scope's shell into `postgresql@11-main.service`
> so that `postgres` and `sysbench` threads run in the same cgroup.
> 
> Didn't change much.
> 
> >> Maybe I'm doing something else differently?
> > 
> > Maybe, you didn't mention how you started postgres, if you start it from
> > the same session as sysbench and if autogroup is enabled, then all those
> > tasks would be in the same autogroup taskgroup then it should have the
> > same effect as my setup.
> 
> This should be now close to my setup running `postgres` and `sysbench`
> in `postgresql@11-main.service`.

Yes.

> 
> > Anyway, you can try the following steps to see if you can reproduce this
> > problem on your Arm64 server:
> > 
> > 1 docker pull postgres
> > 2 sudo docker run --rm --name postgres-instance -e POSTGRES_PASSWORD=mypass -e POSTGRES_USER=sbtest -d postgres -c shared_buffers=80MB -c max_connections=250
> > 3 go inside the container
> >   sudo docker exec -it $the_just_started_container_id bash
> > 4 install sysbench inside container
> >   apt update and apt install sysbench
> > 5 prepare
> >   root@container:/# sysbench --db-driver=pgsql --pgsql-user=sbtest --pgsql_password=mypass --pgsql-db=sbtest --pgsql-port=5432 --tables=16 --table-size=10000 --threads=224 --time=60 --report-interval=2 /usr/share/sysbench/oltp_read_only.lua prepare
> > 6 run
> >   root@container:/# sysbench --db-driver=pgsql --pgsql-user=sbtest --pgsql_password=mypass --pgsql-db=sbtest --pgsql-port=5432 --tables=16 --table-size=10000 --threads=224 --time=60 --report-interval=2 /usr/share/sysbench/oltp_read_only.lua run
> 
> I would have to find time to learn how to set up docker on my machine
> ... But I use very similar values for the setup and sysbench test.

Agree. And docker just made running this workload easier but since you
already grouped all tasks in the same taskgroup, there is no need to
mess with docker.

> 
> > Note that I used 224 threads where this problem is visible. I also tried
> > 96 and update_cfs_group() and update_load_avg() cost about 1% cycles then.
> 
> True, I was hopping to see at least the 1% ;-)

One more question: when you do 'perf report', did you use
--sort=dso,symbol to aggregate different paths of the same target? Maybe
you have already done this, just want to confirm :-)

And not sure if you did the profile on different nodes? I normally chose
4 cpus of each node and do 'perf record -C' with them, to get an idea
of how different node behaves and also to reduce the record size.
Normally, when tg is allocated on node 0, then node 1's profile would
show higher cycles for update_cfs_group() and update_load_avg().

Another thing worth mentioning about this workload is, it has a lot of
wakeups and migrations during the initial 2 minutes or so and a lot of
migrations is the reason of increased cost of update_cfs_group() and
update_load_avg(). On my side, with sysbench's nr_thread=224, the
wakeups and migration numbers during a 5s window are(recorded after
about 1 minute the workload is started):
@migrations[1]: 1821379
@migrations[0]: 4482989
@wakeups[1]: 3036473
@wakeups[0]: 6504496

The above number is derived from below script:
kretfunc:select_task_rq_fair
{
	@wakeups[numaid] = count();
	if (args->p->thread_info.cpu != retval) {
		@migrations[numaid] = count();
	}
}

interval:s:5
{
	exit();
}

And during this time window, node1's profile shows update_cfs_group()'s
cycle percent is 12.45% and update_load_avg() is 7.99%.

I guess your setup may have a much lower migration number?