Received-SPF: pass (google.com: domain of linux-kernel+bounces-89989-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Mon, 4 Mar 2024 11:51:14 +0900
From: Byungchul Park <byungchul@sk.com>
To: "Huang, Ying" <ying.huang@intel.com>
Cc: David Hildenbrand <david@redhat.com>, linux-kernel@vger.kernel.org,
	linux-mm@kvack.org, kernel_team@skhynix.com,
	akpm@linux-foundation.org, vernhao@tencent.com,
	mgorman@techsingularity.net, hughd@google.com, willy@infradead.org,
	peterz@infradead.org, luto@kernel.org, tglx@linutronix.de,
	mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com,
	rjgolo@gmail.com
Subject: Re: [RESEND PATCH v8 0/8] Reduce TLB flushes by 94% by improving
 folio migration
Message-ID: <20240304025114.GB13332@system.software.com>
References: <20240226030613.22366-1-byungchul@sk.com>
 <20240229092810.GC64252@system.software.com>
 <54053f0d-024b-4064-8d82-235cc71b61f8@redhat.com>
 <87wmqmbxko.fsf@yhuang6-desk2.ccr.corp.intel.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87wmqmbxko.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Mutt/1.9.4 (2018-02-28)

On Fri, Mar 01, 2024 at 08:33:11AM +0800, Huang, Ying wrote:
> David Hildenbrand <david@redhat.com> writes:
> 
> > On 29.02.24 10:28, Byungchul Park wrote:
> >> On Mon, Feb 26, 2024 at 12:06:05PM +0900, Byungchul Park wrote:
> >>> Hi everyone,
> >>>
> >>> While I'm working with a tiered memory system e.g. CXL memory, I have
> >>> been facing migration overhead esp. TLB shootdown on promotion or
> >>> demotion between different tiers. Yeah.. most TLB shootdowns on
> >>> migration through hinting fault can be avoided thanks to Huang Ying's
> >>> work, commit 4d4b6d66db ("mm,unmap: avoid flushing TLB in batch if PTE
> >>> is inaccessible"). See the following link:
> >>>
> >>> https://lore.kernel.org/lkml/20231115025755.GA29979@system.software.com/
> >>>
> >>> However, it's only for ones using hinting fault. I thought it'd be much
> >>> better if we have a general mechanism to reduce the number of TLB
> >>> flushes and TLB misses, that we can ultimately apply to any type of
> >>> migration, I tried it only for tiering for now tho.
> >>>
> >>> I'm suggesting a mechanism called MIGRC that stands for 'Migration Read
> >>> Copy', to reduce TLB flushes by keeping source and destination of folios
> >>> participated in the migrations until all TLB flushes required are done,
> >>> only if those folios are not mapped with write permission PTE entries.
> >>>
> >>> To achieve that:
> >>>
> >>>     1. For the folios that map only to non-writable TLB entries, prevent
> >>>        TLB flush at migration by keeping both source and destination
> >>>        folios, which will be handled later at a better time.
> >>>
> >>>     2. When any non-writable TLB entry changes to writable e.g. through
> >>>        fault handler, give up migrc mechanism so as to perform TLB flush
> >>>        required right away.
> >>>
> >>> I observed a big improvement of TLB flushes # and TLB misses # at the
> >>> following evaluation using XSBench like:
> >>>
> >>>     1. itlb flush was reduced by 93.9%.
> >>>     2. dtlb thread was reduced by 43.5%.
> >>>     3. stlb flush was reduced by 24.9%.
> >> Hi guys,
> >
> > Hi,
> >
> >> The TLB flush reduction is 25% ~ 94%, IMO, it's unbelievable.
> >
> > Can't we find at least one benchmark that shows an actual improvement
> > on some system?
> >
> > Staring at the number TLB flushes is nice, but if it does not affect
> > actual performance of at least one benchmark why do we even care?
> >
> > "12 files changed, 597 insertions(+), 59 deletions(-)"
> >
> > is not negligible and needs proper review.
> 
> And, the TLB flush is reduced at cost of memory wastage.  The old pages
> could have been freed.  That may cause regression for some workloads.

You seem to understand the key of migrc(migation read copy) :) Yeah, the
most important thing to deal with is to remove the 'memory wastage'. The
pages deferred to free for the optimization can be freed anytime when
it's needed though TLB flush required that would've been already flushed
unless migrc mechanism.

So memory wastage can be totally removed if resolving some technical
issues that might need your help :)

	Byungchul

> > That review needs motivation. The current numbers do not seem to be
> > motivating enough :)
> 
> --
> Best Regards,
> Huang, Ying