Received-SPF: pass (google.com: domain of linux-kernel+bounces-87738-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
From: "Huang, Ying" <ying.huang@intel.com>
To: David Hildenbrand <david@redhat.com>
Cc: Byungchul Park <byungchul@sk.com>,  linux-kernel@vger.kernel.org,
  linux-mm@kvack.org,  kernel_team@skhynix.com,  akpm@linux-foundation.org,
  vernhao@tencent.com,  mgorman@techsingularity.net,  hughd@google.com,
  willy@infradead.org,  peterz@infradead.org,  luto@kernel.org,
  tglx@linutronix.de,  mingo@redhat.com,  bp@alien8.de,
  dave.hansen@linux.intel.com,  rjgolo@gmail.com
Subject: Re: [RESEND PATCH v8 0/8] Reduce TLB flushes by 94% by improving
 folio migration
In-Reply-To: <54053f0d-024b-4064-8d82-235cc71b61f8@redhat.com> (David
	Hildenbrand's message of "Thu, 29 Feb 2024 10:33:44 +0100")
References: <20240226030613.22366-1-byungchul@sk.com>
	<20240229092810.GC64252@system.software.com>
	<54053f0d-024b-4064-8d82-235cc71b61f8@redhat.com>
Date: Fri, 01 Mar 2024 08:33:11 +0800
Message-ID: <87wmqmbxko.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13)
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=ascii

David Hildenbrand <david@redhat.com> writes:

> On 29.02.24 10:28, Byungchul Park wrote:
>> On Mon, Feb 26, 2024 at 12:06:05PM +0900, Byungchul Park wrote:
>>> Hi everyone,
>>>
>>> While I'm working with a tiered memory system e.g. CXL memory, I have
>>> been facing migration overhead esp. TLB shootdown on promotion or
>>> demotion between different tiers. Yeah.. most TLB shootdowns on
>>> migration through hinting fault can be avoided thanks to Huang Ying's
>>> work, commit 4d4b6d66db ("mm,unmap: avoid flushing TLB in batch if PTE
>>> is inaccessible"). See the following link:
>>>
>>> https://lore.kernel.org/lkml/20231115025755.GA29979@system.software.com/
>>>
>>> However, it's only for ones using hinting fault. I thought it'd be much
>>> better if we have a general mechanism to reduce the number of TLB
>>> flushes and TLB misses, that we can ultimately apply to any type of
>>> migration, I tried it only for tiering for now tho.
>>>
>>> I'm suggesting a mechanism called MIGRC that stands for 'Migration Read
>>> Copy', to reduce TLB flushes by keeping source and destination of folios
>>> participated in the migrations until all TLB flushes required are done,
>>> only if those folios are not mapped with write permission PTE entries.
>>>
>>> To achieve that:
>>>
>>>     1. For the folios that map only to non-writable TLB entries, prevent
>>>        TLB flush at migration by keeping both source and destination
>>>        folios, which will be handled later at a better time.
>>>
>>>     2. When any non-writable TLB entry changes to writable e.g. through
>>>        fault handler, give up migrc mechanism so as to perform TLB flush
>>>        required right away.
>>>
>>> I observed a big improvement of TLB flushes # and TLB misses # at the
>>> following evaluation using XSBench like:
>>>
>>>     1. itlb flush was reduced by 93.9%.
>>>     2. dtlb thread was reduced by 43.5%.
>>>     3. stlb flush was reduced by 24.9%.
>> Hi guys,
>
> Hi,
>
>> The TLB flush reduction is 25% ~ 94%, IMO, it's unbelievable.
>
> Can't we find at least one benchmark that shows an actual improvement
> on some system?
>
> Staring at the number TLB flushes is nice, but if it does not affect
> actual performance of at least one benchmark why do we even care?
>
> "12 files changed, 597 insertions(+), 59 deletions(-)"
>
> is not negligible and needs proper review.

And, the TLB flush is reduced at cost of memory wastage.  The old pages
could have been freed.  That may cause regression for some workloads.

> That review needs motivation. The current numbers do not seem to be
> motivating enough :)

--
Best Regards,
Huang, Ying