Received-SPF: pass (google.com: domain of linux-kernel+bounces-101213-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
From: Thomas Gleixner <tglx@linutronix.de>
To: Frederic Weisbecker <frederic@kernel.org>, kernel test robot
 <oliver.sang@intel.com>
Cc: Anna-Maria Behnsen <anna-maria@linutronix.de>, oe-lkp@lists.linux.dev,
 lkp@intel.com, linux-kernel@vger.kernel.org, x86@kernel.org,
 ying.huang@intel.com, feng.tang@intel.com, fengwei.yin@intel.com
Subject: Re: [tip:timers/core] [timers]  7ee9887703:
  netperf.Throughput_Mbps -1.2% regression
In-Reply-To: <ZfDr6qOaMVnyoB8W@pavilion.home>
References: <202403011511.24defbbd-oliver.sang@intel.com>
 <ZfDr6qOaMVnyoB8W@pavilion.home>
Date: Wed, 13 Mar 2024 09:25:51 +0100
Message-ID: <87y1amo7w0.ffs@tglx>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On Wed, Mar 13 2024 at 00:57, Frederic Weisbecker wrote:
> So I can reproduce. And after hours staring at traces I haven't really fo=
und
> the real cause of this. 1% difference is not always easy to track down.
> But here are some sort of conclusion so far:
>
> _ There is an increase of ksoftirqd use (+13%) but if I boot with threadi=
rqs
>   before and after the patch (which means that ksoftirqd is used all the =
time
>   for softirq handling) I still see the performance regression. So this
>   shouldn't play a role here.
>
> _ I suspected that timer migrators handling big queues of timers on behal=
f of
>   idle CPUs would delay NET_RX softirqs but it doesn't seem to be the cas=
e. I
>   don't see TIMER vector delaying NET_RX vector after the hierarchical pu=
ll
>   model, quite the opposite actually, they are less delayed overall.
>
> _ I suspected that timer migrators handling big queues would add scheduli=
ng
>   latency. But it doesn't seem to be the case. Quite the opposite again,
>   surprisingly.
>
> _ I have observed that, in average, timers execute later with the hierarc=
hical
>   pull model. The following delta:
>        time of callback execution - bucket_expiry
>   is 3 times higher with the hierarchical pull model. Whether that plays =
a role
>   is unclear. It might still be interesting to investigate.
>
> _ The initial perf profile seem to suggest a big increase of task migrati=
on. Is
>   it the result of ping-pong wakeup? Does that play a role?

Migration is not cheap. The interesting question is whether this is
caused by remote timer expiry.

Looking at the perf data there are significant changes vs. idle too:

    perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_=
enter.cpuidle_idle_call.do_idle
	 36.91 =C2=B1  2%     -12.6       24.32 =C2=B1 10%     -12.3       24.63 =
=C2=B1  5%=20

That indicates that cpuidle is spending less time in idle polling, which
means that wakeup latency increases. That obviously might be a result of
the timer migration properties.

Do you have traces (before and after) handy to share?

Thanks,

        tglx