Date:   Sun, 26 Feb 2023 12:50:45 +0900
From:   Sergey Senozhatsky <senozhatsky@chromium.org>
To:     Minchan Kim <minchan@kernel.org>
Cc:     Sergey Senozhatsky <senozhatsky@chromium.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Yosry Ahmed <yosryahmed@google.com>,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org
Subject: Re: [PATCHv2 0/6] zsmalloc: fine-grained fullness and new compaction
 algorithm
Message-ID: <Y/rXFc4dI+190uJa@google.com>
References: <20230223030451.543162-1-senozhatsky@chromium.org>
 <Y/f8esHswgLtjkee@google.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y/f8esHswgLtjkee@google.com>
Precedence: bulk

On (23/02/23 15:53), Minchan Kim wrote:
> > TEST
> > ====
> > 
> > It's very challenging to reliably test this series. I ended up
> > developing my own synthetic test that has 100% reproducibility.
> > The test generates significan fragmentation (for each size class)
> > and then performs compaction for each class individually and tracks
> > the number of memcpy() in zs_object_copy(), so that we can compare
> > the amount work compaction does on per-class basis.
> > 
> > Total amount of work (zram mm_stat objs_moved)
> > ----------------------------------------------
> > 
> > Old fullness grouping, old compaction algorithm:
> > 323977 memcpy() in zs_object_copy().
> > 
> > Old fullness grouping, new compaction algorithm:
> > 262944 memcpy() in zs_object_copy().
> > 
> > New fullness grouping, new compaction algorithm:
> > 213978 memcpy() in zs_object_copy().
> > 
> > 
> > Per-class compaction memcpy() comparison (T-test)
> 
> Just curiosity: What's the T-test?

T-test is a statistical method used to compare the means
of two independent groups or samples and determine if the
difference between them is statistically significant.

> > x Old fullness grouping, old compaction algorithm
> > + Old fullness grouping, new compaction algorithm
> > 
> >     N           Min           Max        Median           Avg        Stddev
> > x 140           349          3513          2461     2314.1214     806.03271
> > + 140           289          2778          2006     1878.1714     641.02073
> > Difference at 95.0% confidence
> > 	-435.95 +/- 170.595
> > 	-18.8387% +/- 7.37193%
> > 	(Student's t, pooled s = 728.216)
> > 
> > 
> > x Old fullness grouping, old compaction algorithm
> > + New fullness grouping, new compaction algorithm
> > 
> >     N           Min           Max        Median           Avg        Stddev
> > x 140           349          3513          2461     2314.1214     806.03271
> > + 140           226          2279          1644     1528.4143     524.85268
> > Difference at 95.0% confidence
> > 	-785.707 +/- 159.331
> > 	-33.9527% +/- 6.88516%
> > 	(Student's t, pooled s = 680.132)
> 
> What's the different with result above? Did you just run two times and
> shows they are consistent or this is new result based on different
> testing?

The test is exactly the same, it is designed to have 0 variability, it
creates exactly same fragmentation during each run, so we always compare
apples to apples. What is being changed (and hence tested) are fullness
grouping and compaction algorithm.

The first one tests the effect of new compaction algorithm alone:
old fullness grouping and old compaction algorithm VS old fullness
grouping and new compaction algorithm. The data show that with
sufficient level of confidence (95%) we can claim that new compaction
does make a statstically significant improvement and reduce the number
of memcpy() calls (by 18.3% in this particular case).

The second one tests the effect of new fullness grouping and new
compaction algorithm. The data show that with sufficient level of
confidence we can claim that new fullness grouping and new compaction
do make a statstically significant improvement and reduce the number
of memcpy() calls (by 33.9% in this particular case).