Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <1569899984-16272-1-git-send-email-laoar.shao@gmail.com> <20191001144524.GB3321@techsingularity.net>
In-Reply-To: <20191001144524.GB3321@techsingularity.net>
From:   Yafang Shao <laoar.shao@gmail.com>
Date:   Wed, 2 Oct 2019 12:59:14 +0800
Message-ID: <CALOAHbCCGhxtnpsc7par1K0dDO4HANh6W_X4_5fTDDKwpPkkWw@mail.gmail.com>
Subject: Re: [PATCH v2] perf script python: integrate page reclaim analyze script
To:     Mel Gorman <mgorman@techsingularity.net>
Cc:     tonyj@suse.com, acme@kernel.org,
        Peter Zijlstra <peterz@infradead.org>, mingo@redhat.com,
        alexander.shishkin@linux.intel.com, jolsa@redhat.com,
        namhyung@kernel.org, Andrew Morton <akpm@linux-foundation.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Linux MM <linux-mm@kvack.org>, Tony Jones <tonyj@suse.de>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Oct 1, 2019 at 10:45 PM Mel Gorman <mgorman@techsingularity.net> wrote:
>
> On Mon, Sep 30, 2019 at 11:19:44PM -0400, Yafang Shao wrote:
> > A new perf script page-reclaim is introduced in this patch. This new script
> > is used to report the page reclaim details. The possible usage of this
> > script is as bellow,
> > - identify latency spike caused by direct reclaim
> > - whehter the latency spike is relevant with pageout
> > - why is page reclaim requested, i.e. whether it is because of memory
> >   fragmentation
> > - page reclaim efficiency
> > etc
> > In the future we may also enhance it to analyze the memcg reclaim.
> >
>
> Hi,
>
> I ended up not reviewing this patch in detail simply because I would
> approach the same class of problem in an entirely different way today.
> There is value in accumulating the stats in a report like this;
>
> >     $ perf script report page-reclaim
> >     Direct reclaims: 4924
> >     Direct latency (ms)        total         max         avg         min
> >                         177823.211    6378.977      36.114       0.051
> >     Direct file reclaimed 22920
> >     Direct file scanned 28306
> >     Direct file sync write I/O 0
> >     Direct file async write I/O 0
> >     Direct anon reclaimed 212567
> >     Direct anon scanned 1446854
> >     Direct anon sync write I/O 0
> >     Direct anon async write I/O 278325
> >     Direct order      0     1     3
> >                  4870    23    31
> >     Wake kswapd requests 716
> >     Wake order      0     1
> >                 715     1
> >
> >     Kswapd reclaims: 9
>
> However, the basic option I would prefer is having the raw latency
> information for Direct latency that can be externally parsed by R or any
> other statistical method. The reason why is because knowing the max latency
> is not enough, I'd want to know the spread of latencies and whether they
> were clustered at a point of time or spread out over long periods of
> time. I would then build the higher-level reports on top if necessary.
>
> Today, I would also have considered getting the latency figures using eBPF
> or systemtap instead although having perf do it may be useful too. That's
> not universally popular though so at minimum I would have;
>

eBPF requires newer kernel, while there're still lots of servers
running with old kernels.
The systemtap is not convenient as it requires many debug packages,
and it is still not stable enough to run on the product environment,
for example, the systemtap deamon may exit without uninstalling the
systemtap kernel module.

> perf script record page-reclaim -- capture all page-reclaim tracepoints
> perf script report page-reclaim -- For reclaim entry/exit, merge the two
>         tracepoints into one that reports latency. Dump the rest out
>         verbatim
>
> For latencies, I would externally post-process them until such time as I
> found a common class of bug that needed a high-level report and then
> build the perf script support for it.
>

This seem like a good suggestion.
I will try to think about it.

> Please note that I did not spot anything wrong with your script, it's
> just that I would not use it myself in its current format for debugging
> a reclaim-related problem.
>
> --
> Mel Gorman
> SUSE Labs