Received-SPF: pass (google.com: domain of linux-kernel+bounces-210652-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Precedence: bulk
MIME-Version: 1.0
References: <20240304081348.197341-1-21cnbao@gmail.com> <20240304081348.197341-6-21cnbao@gmail.com>
 <c9702789-5772-4750-a609-e44a5bbd8468@arm.com> <CANzGp4+p3xSo9uX2i7K2bSZ3VKEQQChAVzdmBD3O2qXq_cE2yA@mail.gmail.com>
 <emvsj7wfy24dzr6uxyac2qotp7nsdi7hnesihaldkvgo3mfzrf@u7fafr7mc3e7>
 <CAGsJ_4zTpcBj_0uC9v4YOHihx-vEek+Y6rr=M1noijwbhfBw7A@mail.gmail.com> <ly745k53gpkef6ktaoilbib4bzrwyuobli7adlylk5yf24ddhk@l4x2swggwm3f>
In-Reply-To: <ly745k53gpkef6ktaoilbib4bzrwyuobli7adlylk5yf24ddhk@l4x2swggwm3f>
From: Barry Song <21cnbao@gmail.com>
Date: Wed, 12 Jun 2024 10:13:06 +1200
Message-ID: <CAGsJ_4xVerq0bukCeZgXmjn2uBUviBEBjY6AWM4wm1M4D2N0og@mail.gmail.com>
Subject: Re: [RFC PATCH v3 5/5] mm: support large folios swapin as a whole
To: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Chuanhua Han <chuanhuahan@gmail.com>, Ryan Roberts <ryan.roberts@arm.com>, 
	akpm@linux-foundation.org, linux-mm@kvack.org, chengming.zhou@linux.dev, 
	chrisl@kernel.org, david@redhat.com, hannes@cmpxchg.org, kasong@tencent.com, 
	linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, 
	mhocko@suse.com, nphamcs@gmail.com, shy828301@gmail.com, steven.price@arm.com, 
	surenb@google.com, wangkefeng.wang@huawei.com, willy@infradead.org, 
	xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, 
	yuzhao@google.com, Chuanhua Han <hanchuanhua@oppo.com>, 
	Barry Song <v-songbaohua@oppo.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Wed, Jun 12, 2024 at 5:24=E2=80=AFAM Shakeel Butt <shakeel.butt@linux.de=
v> wrote:
>
> On Tue, Jun 11, 2024 at 12:23:41PM GMT, Barry Song wrote:
> > On Tue, Jun 11, 2024 at 8:43=E2=80=AFAM Shakeel Butt <shakeel.butt@linu=
x.dev> wrote:
> > >
> > > On Thu, Mar 14, 2024 at 08:56:17PM GMT, Chuanhua Han wrote:
> > > [...]
> > > > >
> > > > > So in the common case, swap-in will pull in the same size of foli=
o as was
> > > > > swapped-out. Is that definitely the right policy for all folio si=
zes? Certainly
> > > > > it makes sense for "small" large folios (e.g. up to 64K IMHO). Bu=
t I'm not sure
> > > > > it makes sense for 2M THP; As the size increases the chances of a=
ctually needing
> > > > > all of the folio reduces so chances are we are wasting IO. There =
are similar
> > > > > arguments for CoW, where we currently copy 1 page per fault - it =
probably makes
> > > > > sense to copy the whole folio up to a certain size.
> > > > For 2M THP, IO overhead may not necessarily be large? :)
> > > > 1.If 2M THP are continuously stored in the swap device, the IO
> > > > overhead may not be very large (such as submitting bio with one
> > > > bio_vec at a time).
> > > > 2.If the process really needs this 2M data, one page-fault may perf=
orm
> > > > much better than multiple.
> > > > 3.For swap devices like zram,using 2M THP might also improve
> > > > decompression efficiency.
> > > >
> > >
> > > Sorry for late response, do we have any performance data backing the
> > > above claims particularly for zswap/swap-on-zram cases?
> >
> > no need to say sorry. You are always welcome to give comments.
> >
> > this, combining with zram modification, not only improves compression
> > ratio but also reduces CPU time significantly. you may find some data
> > here[1].
> >
> > granularity   orig_data_size   compr_data_size   time(us)
> > 4KiB-zstd      1048576000       246876055        50259962
> > 64KiB-zstd     1048576000       199763892        18330605
> >
> > On mobile devices, We tested the performance of swapin by running
> > 100 iterations of swapping in 100MB of data ,and the results were
> > as follows.the swapin speed increased by about 45%.
> >
> >                 time consumption of swapin(ms)
> > lz4 4k                  45274
> > lz4 64k                 22942
> >
> > zstdn 4k                85035
> > zstdn 64k               46558
>
> Thanks for the response. Above numbers are actually very fascinating and
> counter intuitive (at least to me). Do you also have numbers for 2MiB
> THP? I am assuming 64k is the right balance between too small or too
> large. Did you experiment on server machines as well?

I don=E2=80=99t possess data on 2MiB, and regrettably, I lack a server mach=
ine
for testing. However, I believe that this type of higher compression ratio
and lower CPU consumption generally holds true for generic anonymous
memory.

64KB is a right balance. But nothing can stop THP from using 64KB to
swapin, compression and decompression. as you can see from the
zram/zsmalloc series,  we actually have a configuration
CONFIG_ZSMALLOC_MULTI_PAGES_ORDER

The default value is 4.

That means a 2MB THP can be compressed/decompressed as 32 * 64KB.
If we use 64KB as the swapin granularity, we still have the balance and
all the benefits if 2MB is a too large swap-in granularity which might caus=
e
memory waste.

>
> >
> > [1] https://lore.kernel.org/linux-mm/20240327214816.31191-1-21cnbao@gma=
il.com/
> >

Thanks
Barry