Received-SPF: pass (google.com: domain of linux-kernel+bounces-103377-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Precedence: bulk
MIME-Version: 1.0
References: <20240308074921.45752-1-ioworker0@gmail.com> <ef409d5e-5652-4fff-933c-49bda6d75018@redhat.com>
 <CAK1f24k_+qAqxKGMpKziouuds=PQ6kfKyQ8D3SYEyW7cQOAJWw@mail.gmail.com>
 <75630ba6-79b6-4105-b614-29cfb0331084@redhat.com> <CAK1f24=vU5kEuJC6x099JGFD7z6FK5q+o1to7QY8Q12jNQzr_g@mail.gmail.com>
In-Reply-To: <CAK1f24=vU5kEuJC6x099JGFD7z6FK5q+o1to7QY8Q12jNQzr_g@mail.gmail.com>
From: Lance Yang <ioworker0@gmail.com>
Date: Thu, 14 Mar 2024 22:19:29 +0800
Message-ID: <CAK1f24ktQMYogUETyu04KahC1YAdrY1XwCNNrYUQXN4tSEPKsQ@mail.gmail.com>
Subject: Re: [PATCH 1/1] mm/khugepaged: reduce process visible downtime by
 pre-zeroing hugepage
To: David Hildenbrand <david@redhat.com>
Cc: akpm@linux-foundation.org, mhocko@suse.com, zokeefe@google.com, 
	shy828301@gmail.com, xiehuan09@gmail.com, songmuchun@bytedance.com, 
	minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, 
	libang.li@antgroup.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Another thought suggested by Bang Li is that we record which pte is none
in hpage_collapse_scan_pmd. Then, before acquiring the mmap_lock (write mod=
e),
we will pre-zero pages as needed.

What do you think?

Thanks,
Lance

On Tue, Mar 12, 2024 at 9:55=E2=80=AFPM Lance Yang <ioworker0@gmail.com> wr=
ote:
>
> On Tue, Mar 12, 2024 at 9:19=E2=80=AFPM David Hildenbrand <david@redhat.c=
om> wrote:
> >
> > On 12.03.24 14:09, Lance Yang wrote:
> > > Hey David,
> > >
> > > Thanks for taking time to review!
> > >
> > > On Tue, Mar 12, 2024 at 12:19=E2=80=AFAM David Hildenbrand <david@red=
hat.com> wrote:
> > >>
> > >> On 08.03.24 08:49, Lance Yang wrote:
> > >>> The patch reduces the process visible downtime during hugepage
> > >>> collapse. This is achieved by pre-zeroing the hugepage before
> > >>> acquiring mmap_lock(write mode) if nr_pte_none >=3D 256, without
> > >>> affecting the efficiency of khugepaged.
> > >>>
> > >>> On an Intel Core i5 CPU, the process visible downtime during
> > >>> hugepage collapse is as follows:
> > >>>
> > >>> | nr_ptes_none  | w/o __GFP_ZERO | w/ __GFP_ZERO  |  Change |
> > >>> --------------------------------------------------=E2=80=94--------=
--
> > >>> |      511      |     233us      |      95us      |  -59.21%|
> > >>> |      384      |     376us      |     219us      |  -41.20%|
> > >>> |      256      |     421us      |     323us      |  -23.28%|
> > >>> |      128      |     523us      |     507us      |   -3.06%|
> > >>>
> > >>> Of course, alloc_charge_hpage() will take longer to run with
> > >>> the __GFP_ZERO flag.
> > >>>
> > >>> |       Func           | w/o __GFP_ZERO | w/ __GFP_ZERO |
> > >>> |----------------------|----------------|---------------|
> > >>> | alloc_charge_hpage   |      198us     |      295us    |
> > >>>
> > >>> But it's not a big deal because it doesn't impact the total
> > >>> time spent by khugepaged in collapsing a hugepage. In fact,
> > >>> it would decrease.
> > >>
> > >> It does look sane to me and not overly complicated.
> > >>
> > >> But, it's an optimization really only when we have quite a bunch of
> > >> pte_none(), possibly repeatedly so that it really makes a difference=
.
> > >>
> > >> Usually, when we repeatedly collapse that many pte_none() we're just
> > >> wasting a lot of memory and should re-evaluate life choices :)
> > >
> > > Agreed! It seems that the default value of max_pte_none may be set to=
o
> > > high, which could result in the memory wastage issue we're discussing=
.
> >
> > IIRC, some companies disable it completely (set to 0) because of that.
> >
> > >
> > >>
> > >> So my question is: do we really care about it that much that we care=
 to
> > >> optimize?
> > >
> > > IMO, although it may not be our main concern, reducing the impact of
> > > khugepaged on the process remains crucial. I think that users also pr=
efer
> > > minimal interference from khugepaged.
> >
> > The problem I am having with this is that for the *common* case where w=
e
> > have a small number of pte_none(), we cannot really optimize because we
> > have to perform the copy.
> >
> > So this feels like we're rather optimizing a corner case, and I am not
> > so sure if that is really worth it.
> >
> > Other thoughts?
>
> Another thought is to introduce khugepaged/alloc_zeroed_hpage for THP
> sysfs settings. This would enable users to decide whether to avoid unnece=
ssary
> copies when nr_ptes_none > 0.
>
> >
> > --
> > Cheers,
> >
> > David / dhildenb
> >