Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Fri, 18 Sep 2020 07:45:13 +0200
From:   Michal Hocko <mhocko@suse.com>
To:     Vijay Balakrishna <vijayb@linux.microsoft.com>
Cc:     Andrew Morton <akpm@linux-foundation.org>,
        "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
        Oleg Nesterov <oleg@redhat.com>,
        Song Liu <songliubraving@fb.com>,
        Andrea Arcangeli <aarcange@redhat.com>,
        Pavel Tatashin <pasha.tatashin@soleen.com>,
        Allen Pais <apais@microsoft.com>, linux-kernel@vger.kernel.org,
        linux-mm@kvack.org
Subject: Re: [[PATCH]] mm: khugepaged: recalculate min_free_kbytes after
 memory hotplug as expected by khugepaged
Message-ID: <20200918054513.GA28827@dhcp22.suse.cz>
References: <1599770859-14826-1-git-send-email-vijayb@linux.microsoft.com>
 <20200914143312.GU16999@dhcp22.suse.cz>
 <c6fcc196-ce7f-1f48-e9bd-c18448272df1@linux.microsoft.com>
 <20200915081832.GA4649@dhcp22.suse.cz>
 <53dd1e2c-f07e-ee5b-51a1-0ef8adb53926@linux.microsoft.com>
 <20200916065306.GB18998@dhcp22.suse.cz>
 <32b73685-48f2-b6dd-f000-8ea52cfee70a@linux.microsoft.com>
 <20200917121213.GC29887@dhcp22.suse.cz>
 <7eddcc58-f65f-0be9-60e8-2de013365909@linux.microsoft.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <7eddcc58-f65f-0be9-60e8-2de013365909@linux.microsoft.com>
Precedence: bulk

On Thu 17-09-20 11:03:56, Vijay Balakrishna wrote:
[...]
> > > The auto tuned value is incorrect post hotplug memory operation, in our use
> > > case memoy hot add occurs very early during boot.
> > Define incorrect. What are the actual values? Have you tried to increase
> > the value manually after the hotplug?
> 
> In our case SoC with 8GB memory, system tuned min_free_kbytes
> - first to 22528
> - we perform memory hot add very early in boot

What was the original and after-the-hotplug size of memory and layout?
I suspect that all the hotplugged memory is in Movable zone, right?

> - now min_free_kbytes is 8703
> 
> Before looking at code, first I manually restored min_free_kbytes soon after
> boot, reran stress and didn't notice symptoms I mentioned in change log.

This is really surprising and I strongly suspect that an earlier reclaim
just changed the timing enough so that workload has spread the memory
prpessure over a longer time and that might have been enough to recycle
some of the unreclaimable memory due to its natural life time. But this
is a pure speculation. Much more data would be needed to analyze this.

In any case your stress test is oveprovisioning your Normal zone and
increased min_free_kbytes just papers over the sizing problem.
-- 
Michal Hocko
SUSE Labs