Received-SPF: pass (google.com: domain of linux-kernel+bounces-186054-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Wed, 22 May 2024 19:27:44 +0900
From: Byungchul Park <byungchul@sk.com>
To: "Huang, Ying" <ying.huang@intel.com>
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
	kernel_team@skhynix.com, akpm@linux-foundation.org,
	vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com,
	willy@infradead.org, david@redhat.com, peterz@infradead.org,
	luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de,
	dave.hansen@linux.intel.com, rjgolo@gmail.com
Subject: Re: [PATCH v10 00/12] LUF(Lazy Unmap Flush) reducing tlb numbers
 over 90%
Message-ID: <20240522102743.GA29930@system.software.com>
References: <20240510065206.76078-1-byungchul@sk.com>
 <87eda8g6q2.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <20240513014428.GB38851@system.software.com>
 <20240522021616.GA34580@system.software.com>
 <87h6eqb8kj.fsf@yhuang6-desk2.ccr.corp.intel.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87h6eqb8kj.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Mutt/1.9.4 (2018-02-28)

On Wed, May 22, 2024 at 03:38:04PM +0800, Huang, Ying wrote:
> Hi, Byungchul,
> 
> Byungchul Park <byungchul@sk.com> writes:
> 
> > On Mon, May 13, 2024 at 10:44:29AM +0900, Byungchul Park wrote:
> >> On Sat, May 11, 2024 at 03:15:01PM +0800, Huang, Ying wrote:
> >> > Byungchul Park <byungchul@sk.com> writes:
> >> > 
> >> > > Hi everyone,
> >> > >
> >> > > While I'm working with a tiered memory system e.g. CXL memory, I have
> >> > > been facing migration overhead esp. tlb shootdown on promotion or
> >> > > demotion between different tiers.  Yeah..  most tlb shootdowns on
> >> > > migration through hinting fault can be avoided thanks to Huang Ying's
> >> > > work, commit 4d4b6d66db ("mm,unmap: avoid flushing tlb in batch if PTE
> >> > > is inaccessible").  See the following link for more information:
> >> > >
> >> > > https://lore.kernel.org/lkml/20231115025755.GA29979@system.software.com/
> >> > 
> >> > And, I still have interest of the performance impact of commit
> >> > 7e12beb8ca2a ("migrate_pages: batch flushing TLB").  In the email above,
> >> > you said that the performance of v6.5-rc5 + 7e12beb8ca2a reverted has
> >> > better performance than v6.5-rc5.  Can you provide more details?  For
> >> > example, the number of TLB flushing IPI for two kernels?
> >> 
> >> Okay.  I will test and share the result with what you asked me now once
> >> I get available for the test.
> >
> > I should admit that the test using qemu is so unstable.  While using
> > qemu for the test, kernel with 7e12beb8ca2a applied gave better results
> > sometimes and worse ones sometimes.  I should've used a bare metal from
> > the beginning.  Sorry for making you confused with the unstable result.
> >
> > Since I thought you asked me for the test with the same environment in
> > the link above, I used qemu to reproduce the similar result but changed
> > the number of threads for the test from 16 to 14 to get rid of noise
> > that might be introduced by other than the intended test just in case.
> >
> > As expected, the stats are better with your work:
> >
> >    ------------------------------------------
> >    v6.6-rc5 with 7e12beb8ca2a commit reverted
> >    ------------------------------------------
> >
> >    1) from output of XSBench
> >
> >    Threads:     14              
> >    Runtime:     1127.043 seconds
> >    Lookups:     1,700,000,000   
> >    Lookups/s:   1,508,371       
> >
> >    2) from /proc/vmstat
> >
> >    numa_hit 15580171                      
> >    numa_miss 1034233                      
> >    numa_foreign 1034233                   
> >    numa_interleave 773                    
> >    numa_local 7927442                     
> >    numa_other 8686962                     
> >    numa_pte_updates 24068923              
> >    numa_hint_faults 24061125              
> >    numa_hint_faults_local 0               
> >    numa_pages_migrated 7426480            
> >    pgmigrate_success 15407375             
> >    pgmigrate_fail 1849                    
> >    compact_migrate_scanned 4445414        
> >    compact_daemon_migrate_scanned 4445414 
> >    pgdemote_kswapd 7651061                
> >    pgdemote_direct 0                      
> >    nr_tlb_remote_flush 8080092            
> >    nr_tlb_remote_flush_received 109915713 
> >    nr_tlb_local_flush_all 53800           
> >    nr_tlb_local_flush_one 770466                                                   
> >    
> >    3) from /proc/interrupts
> >
> >    TLB: 8022927    7840769     123588    7837008    7835967    7839837
> >    	7838332    7839886    7837610    7837221    7834524     407260
> >    	7430090    7835696    7839081    7712568    TLB shootdowns  
> >    
> >    4) from 'perf stat -a'
> >
> >    222371217		itlb.itlb_flush      
> >    919832520		tlb_flush.dtlb_thread
> >    372223809		tlb_flush.stlb_any   
> >    120210808042		dTLB-load-misses     
> >    979352769		dTLB-store-misses    
> >    3650767665		iTLB-load-misses     
> >
> >    -----------------------------------------
> >    v6.6-rc5 with 7e12beb8ca2a commit applied
> >    -----------------------------------------
> >
> >    1) from output of XSBench
> >
> >    Threads:     14
> >    Runtime:     1105.521 seconds
> >    Lookups:     1,700,000,000
> >    Lookups/s:   1,537,737
> >
> >    2) from /proc/vmstat
> >
> >    numa_hit 24148399
> >    numa_miss 797483
> >    numa_foreign 797483
> >    numa_interleave 772
> >    numa_local 12214575
> >    numa_other 12731307
> >    numa_pte_updates 24250278
> >    numa_hint_faults 24199756
> >    numa_hint_faults_local 0
> >    numa_pages_migrated 11476195
> >    pgmigrate_success 23634639
> >    pgmigrate_fail 1391
> >    compact_migrate_scanned 3760803
> >    compact_daemon_migrate_scanned 3760803
> >    pgdemote_kswapd 11932217
> >    pgdemote_direct 0
> >    nr_tlb_remote_flush 2151945
> >    nr_tlb_remote_flush_received 29672808
> >    nr_tlb_local_flush_all 124006
> >    nr_tlb_local_flush_one 741165
> >    
> >    3) from /proc/interrupts
> >
> >    TLB: 2130784    2120142    2117571     844962    2071766     114675
> >    	2117258    2119596    2116816    1205446    2119176    2119209
> >    	2116792    2118763    2118773    2117762    TLB shootdowns
> >
> >    4) from 'perf stat -a'
> >
> >    60851902		itlb.itlb_flush
> >    334068491		tlb_flush.dtlb_thread
> >    223732916		tlb_flush.stlb_any
> >    120207083382		dTLB-load-misses
> >    446823059		dTLB-store-misses
> >    1926669373		iTLB-load-misses
> >
> 
> Thanks a lot for test results!
> 
> >From your test results, the TLB shootdown IPI can be reduced effectively
> with commit 7e12beb8ca2a.  So that the benchmark score improved a
> little.
> 
> And, your changes will reduce the TLB shootdown IPI further, right?  Do

Yes, right. LUF(Lazy Unmap Flush) reduces TLB shootdown IPI further.

> you have the number?

You can find the number obtained from llama.cpp in this cover letter:

   https://lore.kernel.org/lkml/20240520021734.21527-1-byungchul@sk.com/

If you meant the number from the same test above, XSBench + qemu, I will
re-test with mm-unstable branch of mm tree and share the result shortly.

	Byungchul