Received-SPF: pass (google.com: domain of linux-kernel+bounces-185686-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Wed, 22 May 2024 11:16:16 +0900
From: Byungchul Park <byungchul@sk.com>
To: "Huang, Ying" <ying.huang@intel.com>
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
	kernel_team@skhynix.com, akpm@linux-foundation.org,
	vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com,
	willy@infradead.org, david@redhat.com, peterz@infradead.org,
	luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de,
	dave.hansen@linux.intel.com, rjgolo@gmail.com
Subject: Re: [PATCH v10 00/12] LUF(Lazy Unmap Flush) reducing tlb numbers
 over 90%
Message-ID: <20240522021616.GA34580@system.software.com>
References: <20240510065206.76078-1-byungchul@sk.com>
 <87eda8g6q2.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <20240513014428.GB38851@system.software.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240513014428.GB38851@system.software.com>
User-Agent: Mutt/1.9.4 (2018-02-28)

On Mon, May 13, 2024 at 10:44:29AM +0900, Byungchul Park wrote:
> On Sat, May 11, 2024 at 03:15:01PM +0800, Huang, Ying wrote:
> > Byungchul Park <byungchul@sk.com> writes:
> > 
> > > Hi everyone,
> > >
> > > While I'm working with a tiered memory system e.g. CXL memory, I have
> > > been facing migration overhead esp. tlb shootdown on promotion or
> > > demotion between different tiers.  Yeah..  most tlb shootdowns on
> > > migration through hinting fault can be avoided thanks to Huang Ying's
> > > work, commit 4d4b6d66db ("mm,unmap: avoid flushing tlb in batch if PTE
> > > is inaccessible").  See the following link for more information:
> > >
> > > https://lore.kernel.org/lkml/20231115025755.GA29979@system.software.com/
> > 
> > And, I still have interest of the performance impact of commit
> > 7e12beb8ca2a ("migrate_pages: batch flushing TLB").  In the email above,
> > you said that the performance of v6.5-rc5 + 7e12beb8ca2a reverted has
> > better performance than v6.5-rc5.  Can you provide more details?  For
> > example, the number of TLB flushing IPI for two kernels?
> 
> Okay.  I will test and share the result with what you asked me now once
> I get available for the test.

I should admit that the test using qemu is so unstable.  While using
qemu for the test, kernel with 7e12beb8ca2a applied gave better results
sometimes and worse ones sometimes.  I should've used a bare metal from
the beginning.  Sorry for making you confused with the unstable result.

Since I thought you asked me for the test with the same environment in
the link above, I used qemu to reproduce the similar result but changed
the number of threads for the test from 16 to 14 to get rid of noise
that might be introduced by other than the intended test just in case.

As expected, the stats are better with your work:

   ------------------------------------------
   v6.6-rc5 with 7e12beb8ca2a commit reverted
   ------------------------------------------

   1) from output of XSBench

   Threads:     14              
   Runtime:     1127.043 seconds
   Lookups:     1,700,000,000   
   Lookups/s:   1,508,371       

   2) from /proc/vmstat

   numa_hit 15580171                      
   numa_miss 1034233                      
   numa_foreign 1034233                   
   numa_interleave 773                    
   numa_local 7927442                     
   numa_other 8686962                     
   numa_pte_updates 24068923              
   numa_hint_faults 24061125              
   numa_hint_faults_local 0               
   numa_pages_migrated 7426480            
   pgmigrate_success 15407375             
   pgmigrate_fail 1849                    
   compact_migrate_scanned 4445414        
   compact_daemon_migrate_scanned 4445414 
   pgdemote_kswapd 7651061                
   pgdemote_direct 0                      
   nr_tlb_remote_flush 8080092            
   nr_tlb_remote_flush_received 109915713 
   nr_tlb_local_flush_all 53800           
   nr_tlb_local_flush_one 770466                                                   
   
   3) from /proc/interrupts

   TLB: 8022927    7840769     123588    7837008    7835967    7839837
   	7838332    7839886    7837610    7837221    7834524     407260
   	7430090    7835696    7839081    7712568    TLB shootdowns  
   
   4) from 'perf stat -a'

   222371217		itlb.itlb_flush      
   919832520		tlb_flush.dtlb_thread
   372223809		tlb_flush.stlb_any   
   120210808042		dTLB-load-misses     
   979352769		dTLB-store-misses    
   3650767665		iTLB-load-misses     

   -----------------------------------------
   v6.6-rc5 with 7e12beb8ca2a commit applied
   -----------------------------------------

   1) from output of XSBench

   Threads:     14
   Runtime:     1105.521 seconds
   Lookups:     1,700,000,000
   Lookups/s:   1,537,737

   2) from /proc/vmstat

   numa_hit 24148399
   numa_miss 797483
   numa_foreign 797483
   numa_interleave 772
   numa_local 12214575
   numa_other 12731307
   numa_pte_updates 24250278
   numa_hint_faults 24199756
   numa_hint_faults_local 0
   numa_pages_migrated 11476195
   pgmigrate_success 23634639
   pgmigrate_fail 1391
   compact_migrate_scanned 3760803
   compact_daemon_migrate_scanned 3760803
   pgdemote_kswapd 11932217
   pgdemote_direct 0
   nr_tlb_remote_flush 2151945
   nr_tlb_remote_flush_received 29672808
   nr_tlb_local_flush_all 124006
   nr_tlb_local_flush_one 741165
   
   3) from /proc/interrupts

   TLB: 2130784    2120142    2117571     844962    2071766     114675
   	2117258    2119596    2116816    1205446    2119176    2119209
   	2116792    2118763    2118773    2117762    TLB shootdowns

   4) from 'perf stat -a'

   60851902		itlb.itlb_flush
   334068491		tlb_flush.dtlb_thread
   223732916		tlb_flush.stlb_any
   120207083382		dTLB-load-misses
   446823059		dTLB-store-misses
   1926669373		iTLB-load-misses

---

	Byungchul