Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 1 Jul 2022 12:34:09 -0700
From:   Fangrui Song <maskray@google.com>
To:     Bill Wendling <morbo@google.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        "Jose E. Marchesi" <jemarch@gnu.org>,
        Ruud van der Pas <ruud.vanderpas@oracle.com>,
        Nick Desaulniers <ndesaulniers@google.com>,
        Sami Tolvanen <samitolvanen@google.com>,
        Vladimir Mezentsev <vladimir.mezentsev@oracle.com>,
        clang-built-linux <llvm@lists.linux.dev>,
        LKML <linux-kernel@vger.kernel.org>, Yonghong Song <yhs@fb.com>,
        Wenlei He <wenlei@fb.com>, Hongtao Yu <hoy@fb.com>,
        Ingo Molnar <mingo@kernel.org>,
        linux-toolchains <linux-toolchains@vger.kernel.org>,
        elena.zannoni@oracle.com
Subject: Re: plumbers session on profiling?
Message-ID: <20220701193409.p4ejod7olx7ngl5m@google.com>
References: <CAKwvOdnsZekEM77axBf67MDqQVP0n6PTKH=njSyPSWTNiWAOiA@mail.gmail.com>
 <87mtf7z0rt.fsf@gnu.org>
 <6F9E9D93-3913-4022-9384-D809C8EF7715@oracle.com>
 <CAKwvOdm=_YqBpuBzouqoWHYNe6MMUE10vqF0PUkU=hcOj+UqrQ@mail.gmail.com>
 <B0A01DE7-1B50-479A-92DF-DAFAB3F06E0F@oracle.com>
 <878rpgpvfj.fsf@gnu.org>
 <Yr638aOIaaEBPICy@worktop.programming.kicks-ass.net>
 <CAGG=3QWbW-Dang49Jx3fyNExWtL8syuMkMJmcPHA7J25cHQ0zw@mail.gmail.com>
 <Yr7fMre18pUMz9rA@worktop.programming.kicks-ass.net>
 <CAGG=3QUA8BnhN9QaJao6nH6EyNks0yWO+cWWyvE0pJbOUm+cMw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <CAGG=3QUA8BnhN9QaJao6nH6EyNks0yWO+cWWyvE0pJbOUm+cMw@mail.gmail.com>
Precedence: bulk

On 2022-07-01, Bill Wendling wrote:
>On Fri, Jul 1, 2022 at 4:49 AM Peter Zijlstra <peterz@infradead.org> wrote:
>> On Fri, Jul 01, 2022 at 03:17:54AM -0700, Bill Wendling wrote:
>> > On Fri, Jul 1, 2022 at 2:02 AM Peter Zijlstra <peterz@infradead.org> wrote:
>> > >
>> > > On Tue, Jun 28, 2022 at 07:08:48PM +0200, Jose E. Marchesi wrote:
>> > > >
>> > > > [Added linux-toolchains@vger in CC]
>> > > >
>> > > > It would be interesting to have some discussion in the Toolchains track
>> > > > on building the kernel with PGO/FDO.  I have seen a raise on interest on
>> > > > the topic in several companies, but it would make very little sense if
>> > > > no kernel hacker is interested in participating... anybody?
>> > >
>> > > I know there's been a lot of work in this area, but none of it seems to
>> > > have trickled down to be easy enough for me to use it.
>> >
>> > We use an instrumented kernel to collect the data we need. It gives us
>> > the best payoff, because the profiling data is more fine-grained and
>> > accurate. (PGO does much more than make inlining decisions.)
>> >
>> > If I recall correctly, you previously suggested using sampling data.
>> > (Correct?) Is there a document or article that outlines that process?
>>
>> IIRC Google has LBR sample driven PGO somewhere as well. ISTR that being
>> the whole motivation for that gruesome Zen3 BRS hack.
>>
>> Google got me this: https://research.google.com/pubs/archive/45290.pdf
>>

I very support that the mainline kernel adds instrumentation based PGO
but I vaguely recall that it was NAKed by Linus (because he thought
sample based is better).

>Right. However, there's a chicken-and-egg issue with AutoFDO for the
>production kernel. We can't release a kernel that hasn't been compiled
>with PGO/FDO. We could only release it in a test environment, in which
>case we could use AutoFDO. However, the document says that AutoFDO
>only reaches ~90% of FDO. They list some reasons for this, but
>nonetheless I suspect that the delta would be too severe for us to
>release the kernel.
>
>As for LBR, that will work with Intel/AMD, but I thought that LBR
>doesn't exist for Arm processors (my knowledge could be out of date on
>this).

Some folks try using Embedded Trace Macrocells.
I am not at all familiar with it but it seems that retrieving profiles
is not easy. The needed efforts seem even higher than using
instrumentation based PGO.

Instrumentation based PGO has the nice property that it works with all
architectures (the compiler supports) and does not rely on hardware
support. In addition, it collects indirect call targets and string
operation sizes which are very difficult/impossible for sample based
PGO.

>What would make PGO (sample-based or instrumented) easy enough for you
>to use? What're the key elements missing?
>
>-bw
>