Sender: Minchan Kim <minchan.kim@gmail.com>
Date:   Tue, 28 Feb 2023 14:17:55 -0800
From:   Minchan Kim <minchan@kernel.org>
To:     Sergey Senozhatsky <senozhatsky@chromium.org>
Cc:     Andrew Morton <akpm@linux-foundation.org>,
        Yosry Ahmed <yosryahmed@google.com>,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org
Subject: Re: [PATCHv2 0/6] zsmalloc: fine-grained fullness and new compaction
 algorithm
Message-ID: <Y/59k9T3akHc4JZW@google.com>
References: <20230223030451.543162-1-senozhatsky@chromium.org>
 <Y/f8esHswgLtjkee@google.com>
 <Y/rXFc4dI+190uJa@google.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y/rXFc4dI+190uJa@google.com>
Precedence: bulk

On Sun, Feb 26, 2023 at 12:50:45PM +0900, Sergey Senozhatsky wrote:
> On (23/02/23 15:53), Minchan Kim wrote:
> > > TEST
> > > ====
> > > 
> > > It's very challenging to reliably test this series. I ended up
> > > developing my own synthetic test that has 100% reproducibility.
> > > The test generates significan fragmentation (for each size class)
> > > and then performs compaction for each class individually and tracks
> > > the number of memcpy() in zs_object_copy(), so that we can compare
> > > the amount work compaction does on per-class basis.
> > > 
> > > Total amount of work (zram mm_stat objs_moved)
> > > ----------------------------------------------
> > > 
> > > Old fullness grouping, old compaction algorithm:
> > > 323977 memcpy() in zs_object_copy().
> > > 
> > > Old fullness grouping, new compaction algorithm:
> > > 262944 memcpy() in zs_object_copy().
> > > 
> > > New fullness grouping, new compaction algorithm:
> > > 213978 memcpy() in zs_object_copy().
> > > 
> > > 
> > > Per-class compaction memcpy() comparison (T-test)
> > 
> > Just curiosity: What's the T-test?
> 
> T-test is a statistical method used to compare the means
> of two independent groups or samples and determine if the
> difference between them is statistically significant.
> 
> > > x Old fullness grouping, old compaction algorithm
> > > + Old fullness grouping, new compaction algorithm
> > > 
> > >     N           Min           Max        Median           Avg        Stddev
> > > x 140           349          3513          2461     2314.1214     806.03271
> > > + 140           289          2778          2006     1878.1714     641.02073
> > > Difference at 95.0% confidence
> > > 	-435.95 +/- 170.595
> > > 	-18.8387% +/- 7.37193%
> > > 	(Student's t, pooled s = 728.216)
> > > 
> > > 
> > > x Old fullness grouping, old compaction algorithm
> > > + New fullness grouping, new compaction algorithm
> > > 
> > >     N           Min           Max        Median           Avg        Stddev
> > > x 140           349          3513          2461     2314.1214     806.03271
> > > + 140           226          2279          1644     1528.4143     524.85268
> > > Difference at 95.0% confidence
> > > 	-785.707 +/- 159.331
> > > 	-33.9527% +/- 6.88516%
> > > 	(Student's t, pooled s = 680.132)
> > 
> > What's the different with result above? Did you just run two times and
> > shows they are consistent or this is new result based on different
> > testing?
> 
> The test is exactly the same, it is designed to have 0 variability, it
> creates exactly same fragmentation during each run, so we always compare
> apples to apples. What is being changed (and hence tested) are fullness
> grouping and compaction algorithm.
> 
> The first one tests the effect of new compaction algorithm alone:
> old fullness grouping and old compaction algorithm VS old fullness
> grouping and new compaction algorithm. The data show that with
> sufficient level of confidence (95%) we can claim that new compaction
> does make a statstically significant improvement and reduce the number
> of memcpy() calls (by 18.3% in this particular case).
> 
> The second one tests the effect of new fullness grouping and new
> compaction algorithm. The data show that with sufficient level of
> confidence we can claim that new fullness grouping and new compaction
> do make a statstically significant improvement and reduce the number
> of memcpy() calls (by 33.9% in this particular case).

Thanks for the explanation, Sergey.

Please include the testing result data in the description of the patch
you made significant change to achieve it as well as cover letter.

Otherwise, zsmalloc-remove-insert_zspage-inuse-optimization.patch
has every data now but that patch didn't make such an improvement.