Received-SPF: pass (google.com: domain of linux-kernel+bounces-185868-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
From: "Huang, Ying" <ying.huang@intel.com>
To: Byungchul Park <byungchul@sk.com>
Cc: <linux-kernel@vger.kernel.org>,  <linux-mm@kvack.org>,
  <kernel_team@skhynix.com>,  <akpm@linux-foundation.org>,
  <vernhao@tencent.com>,  <mgorman@techsingularity.net>,
  <hughd@google.com>,  <willy@infradead.org>,  <david@redhat.com>,
  <peterz@infradead.org>,  <luto@kernel.org>,  <tglx@linutronix.de>,
  <mingo@redhat.com>,  <bp@alien8.de>,  <dave.hansen@linux.intel.com>,
  <rjgolo@gmail.com>
Subject: Re: [PATCH v10 00/12] LUF(Lazy Unmap Flush) reducing tlb numbers
 over 90%
In-Reply-To: <20240522021616.GA34580@system.software.com> (Byungchul Park's
	message of "Wed, 22 May 2024 11:16:16 +0900")
References: <20240510065206.76078-1-byungchul@sk.com>
	<87eda8g6q2.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<20240513014428.GB38851@system.software.com>
	<20240522021616.GA34580@system.software.com>
Date: Wed, 22 May 2024 15:38:04 +0800
Message-ID: <87h6eqb8kj.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13)
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=ascii

Hi, Byungchul,

Byungchul Park <byungchul@sk.com> writes:

> On Mon, May 13, 2024 at 10:44:29AM +0900, Byungchul Park wrote:
>> On Sat, May 11, 2024 at 03:15:01PM +0800, Huang, Ying wrote:
>> > Byungchul Park <byungchul@sk.com> writes:
>> > 
>> > > Hi everyone,
>> > >
>> > > While I'm working with a tiered memory system e.g. CXL memory, I have
>> > > been facing migration overhead esp. tlb shootdown on promotion or
>> > > demotion between different tiers.  Yeah..  most tlb shootdowns on
>> > > migration through hinting fault can be avoided thanks to Huang Ying's
>> > > work, commit 4d4b6d66db ("mm,unmap: avoid flushing tlb in batch if PTE
>> > > is inaccessible").  See the following link for more information:
>> > >
>> > > https://lore.kernel.org/lkml/20231115025755.GA29979@system.software.com/
>> > 
>> > And, I still have interest of the performance impact of commit
>> > 7e12beb8ca2a ("migrate_pages: batch flushing TLB").  In the email above,
>> > you said that the performance of v6.5-rc5 + 7e12beb8ca2a reverted has
>> > better performance than v6.5-rc5.  Can you provide more details?  For
>> > example, the number of TLB flushing IPI for two kernels?
>> 
>> Okay.  I will test and share the result with what you asked me now once
>> I get available for the test.
>
> I should admit that the test using qemu is so unstable.  While using
> qemu for the test, kernel with 7e12beb8ca2a applied gave better results
> sometimes and worse ones sometimes.  I should've used a bare metal from
> the beginning.  Sorry for making you confused with the unstable result.
>
> Since I thought you asked me for the test with the same environment in
> the link above, I used qemu to reproduce the similar result but changed
> the number of threads for the test from 16 to 14 to get rid of noise
> that might be introduced by other than the intended test just in case.
>
> As expected, the stats are better with your work:
>
>    ------------------------------------------
>    v6.6-rc5 with 7e12beb8ca2a commit reverted
>    ------------------------------------------
>
>    1) from output of XSBench
>
>    Threads:     14              
>    Runtime:     1127.043 seconds
>    Lookups:     1,700,000,000   
>    Lookups/s:   1,508,371       
>
>    2) from /proc/vmstat
>
>    numa_hit 15580171                      
>    numa_miss 1034233                      
>    numa_foreign 1034233                   
>    numa_interleave 773                    
>    numa_local 7927442                     
>    numa_other 8686962                     
>    numa_pte_updates 24068923              
>    numa_hint_faults 24061125              
>    numa_hint_faults_local 0               
>    numa_pages_migrated 7426480            
>    pgmigrate_success 15407375             
>    pgmigrate_fail 1849                    
>    compact_migrate_scanned 4445414        
>    compact_daemon_migrate_scanned 4445414 
>    pgdemote_kswapd 7651061                
>    pgdemote_direct 0                      
>    nr_tlb_remote_flush 8080092            
>    nr_tlb_remote_flush_received 109915713 
>    nr_tlb_local_flush_all 53800           
>    nr_tlb_local_flush_one 770466                                                   
>    
>    3) from /proc/interrupts
>
>    TLB: 8022927    7840769     123588    7837008    7835967    7839837
>    	7838332    7839886    7837610    7837221    7834524     407260
>    	7430090    7835696    7839081    7712568    TLB shootdowns  
>    
>    4) from 'perf stat -a'
>
>    222371217		itlb.itlb_flush      
>    919832520		tlb_flush.dtlb_thread
>    372223809		tlb_flush.stlb_any   
>    120210808042		dTLB-load-misses     
>    979352769		dTLB-store-misses    
>    3650767665		iTLB-load-misses     
>
>    -----------------------------------------
>    v6.6-rc5 with 7e12beb8ca2a commit applied
>    -----------------------------------------
>
>    1) from output of XSBench
>
>    Threads:     14
>    Runtime:     1105.521 seconds
>    Lookups:     1,700,000,000
>    Lookups/s:   1,537,737
>
>    2) from /proc/vmstat
>
>    numa_hit 24148399
>    numa_miss 797483
>    numa_foreign 797483
>    numa_interleave 772
>    numa_local 12214575
>    numa_other 12731307
>    numa_pte_updates 24250278
>    numa_hint_faults 24199756
>    numa_hint_faults_local 0
>    numa_pages_migrated 11476195
>    pgmigrate_success 23634639
>    pgmigrate_fail 1391
>    compact_migrate_scanned 3760803
>    compact_daemon_migrate_scanned 3760803
>    pgdemote_kswapd 11932217
>    pgdemote_direct 0
>    nr_tlb_remote_flush 2151945
>    nr_tlb_remote_flush_received 29672808
>    nr_tlb_local_flush_all 124006
>    nr_tlb_local_flush_one 741165
>    
>    3) from /proc/interrupts
>
>    TLB: 2130784    2120142    2117571     844962    2071766     114675
>    	2117258    2119596    2116816    1205446    2119176    2119209
>    	2116792    2118763    2118773    2117762    TLB shootdowns
>
>    4) from 'perf stat -a'
>
>    60851902		itlb.itlb_flush
>    334068491		tlb_flush.dtlb_thread
>    223732916		tlb_flush.stlb_any
>    120207083382		dTLB-load-misses
>    446823059		dTLB-store-misses
>    1926669373		iTLB-load-misses
>

Thanks a lot for test results!

From your test results, the TLB shootdown IPI can be reduced effectively
with commit 7e12beb8ca2a.  So that the benchmark score improved a
little.

And, your changes will reduce the TLB shootdown IPI further, right?  Do
you have the number?

--
Best Regards,
Huang, Ying