Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 21 Aug 2023 10:28:05 +0900
From:   Byungchul Park <byungchul@sk.com>
To:     "Huang, Ying" <ying.huang@intel.com>
Cc:     linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        kernel_team@skhynix.com, akpm@linux-foundation.org,
        namit@vmware.com, xhao@linux.alibaba.com,
        mgorman@techsingularity.net, hughd@google.com, willy@infradead.org,
        david@redhat.com, peterz@infradead.org, luto@kernel.org,
        dave.hansen@linux.intel.com
Subject: Re: [RFC 2/2] mm: Defer TLB flush by keeping both src and dst folios
 at migration
Message-ID: <20230821012804.GA43847@system.software.com>
References: <20230804061850.21498-1-byungchul@sk.com>
 <20230804061850.21498-3-byungchul@sk.com>
 <877cpx9jsx.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <20230816001307.GA44941@system.software.com>
 <87r0o37qcn.fsf@yhuang6-desk2.ccr.corp.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87r0o37qcn.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Precedence: bulk

On Wed, Aug 16, 2023 at 09:01:12AM +0800, Huang, Ying wrote:
> Byungchul Park <byungchul@sk.com> writes:
> 
> > On Tue, Aug 15, 2023 at 09:27:26AM +0800, Huang, Ying wrote:
> >> Byungchul Park <byungchul@sk.com> writes:
> >> 
> >> > Implementation of CONFIG_MIGRC that stands for 'Migration Read Copy'.
> >> >
> >> > We always face the migration overhead at either promotion or demotion,
> >> > while working with tiered memory e.g. CXL memory and found out TLB
> >> > shootdown is a quite big one that is needed to get rid of if possible.
> >> >
> >> > Fortunately, TLB flush can be defered or even skipped if both source and
> >> > destination of folios during migration are kept until all TLB flushes
> >> > required will have been done, of course, only if the target PTE entries
> >> > have read only permission, more precisely speaking, don't have write
> >> > permission. Otherwise, no doubt the folio might get messed up.
> >> >
> >> > To achieve that:
> >> >
> >> >    1. For the folios that have only non-writable TLB entries, prevent
> >> >       TLB flush by keeping both source and destination of folios during
> >> >       migration, which will be handled later at a better time.
> >> >
> >> >    2. When any non-writable TLB entry changes to writable e.g. through
> >> >       fault handler, give up CONFIG_MIGRC mechanism so as to perform
> >> >       TLB flush required right away.
> >> >
> >> >    3. TLB flushes can be skipped if all TLB flushes required to free the
> >> >       duplicated folios have been done by any reason, which doesn't have
> >> >       to be done from migrations.
> >> >
> >> >    4. Adjust watermark check routine, __zone_watermark_ok(), with the
> >> >       number of duplicated folios because those folios can be freed
> >> >       and obtained right away through appropreate TLB flushes.
> >> >
> >> >    5. Perform TLB flushes and free the duplicated folios pending the
> >> >       flushes if page allocation routine is in trouble due to memory
> >> >       pressure, even more aggresively for high order allocation.
> >> 
> >> Is the optimization restricted for page migration only?  Can it be used
> >> for other places?  Like page reclaiming?
> >
> > Just to make sure, are you talking about the (5) description? For now,
> > it's performed at the beginning of __alloc_pages_slowpath(), say, before
> > page recaiming. Do you think it'd be meaningful to perform it during page
> > reclaiming? Or do you mean something else?
> 
> Not for (5).  TLB needs to be flushed during page reclaiming too.  Can
> similar method be used to reduce TLB flushing there too?

If you were talking about unmapping for swap while reclaiming, then I
think yes. The case can also take benefit from CONFIG_MIGRC.

	Byungchul