Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 22 Oct 2019 10:02:23 +0200
From:   Jiri Olsa <jolsa@redhat.com>
To:     Andi Kleen <andi@firstfloor.org>
Cc:     acme@kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org,
        eranian@google.com, kan.liang@linux.intel.com, peterz@infradead.org
Subject: Re: Optimize perf stat for large number of events/cpus v2
Message-ID: <20191022080223.GC28177@krava>
References: <20191020175202.32456-1-andi@firstfloor.org>
MIME-Version: 1.0
In-Reply-To: <20191020175202.32456-1-andi@firstfloor.org>
User-Agent: Mutt/1.12.1 (2019-06-15)
Content-Type: text/plain; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Sun, Oct 20, 2019 at 10:51:53AM -0700, Andi Kleen wrote:
> [The earlier v1 version had a lot of conflicts against some
> recent libperf changes in tip/perf/core. Resolve that and
> also fix some minor issues.]
>=20
> This patch kit optimizes perf stat for a large number of events=20
> on systems with many CPUs and PMUs.
>=20
> Some profiling shows that the most overhead is doing IPIs to
> all the target CPUs. We can optimize this by using sched_setaffinity
> to set the affinity to a target CPU once and then doing
> the perf operation for all events on that CPU. This requires
> some restructuring, but cuts the set up time quite a bit.
>=20
> In theory we could go further by parallelizing these setups
> too, but that would be much more complicated and for now just batching it
> per CPU seems to be sufficient. At some point with many more cores=20
> parallelization or a better bulk perf setup API might be needed though.
>=20
> In addition perf does a lot of redundant /sys accesses with
> many PMUs, which can be also expensve. This is also optimized.
>=20
> On a large test case (>700 events with many weak groups) on a 94 CPU
> system I go from
>=20
> real=090m8.607s
> user=090m0.550s
> sys=090m8.041s
>=20
> to=20
>=20
> real=090m3.269s
> user=090m0.760s
> sys=090m1.694s
>=20
> so shaving ~6 seconds of system time, at slightly more cost
> in perf stat itself. On a 4 socket system with the savings
> are more dramatic:
>=20
> real=090m15.641s
> user=090m0.873s
> sys=090m14.729s
>=20
> to=20
>=20
> real=090m4.493s
> user=090m1.578s
> sys=090m2.444s
>=20
> so 11s difference in the user visible set up time.
>=20
> Also available in=20
>=20
> git://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-misc perf/stat-sca=
le-4
>=20
> v1: Initial post.
> v2: Rebase. Fix some minor issues.

looks really helpful, I ack-ed 1st 2 patches,
I'll need more time for the rest

thanks,
jirka