Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   =?iso-2022-jp?B?SE9SSUdVQ0hJIE5BT1lBKBskQktZOH0hIUQ+TGkbKEIp?= 
        <naoya.horiguchi@nec.com>
To:     Qian Cai <cai@lca.pw>
CC:     "nao.horiguchi@gmail.com" <nao.horiguchi@gmail.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "mhocko@kernel.org" <mhocko@kernel.org>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "mike.kravetz@oracle.com" <mike.kravetz@oracle.com>,
        "osalvador@suse.de" <osalvador@suse.de>,
        "tony.luck@intel.com" <tony.luck@intel.com>,
        "david@redhat.com" <david@redhat.com>,
        "aneesh.kumar@linux.vnet.ibm.com" <aneesh.kumar@linux.vnet.ibm.com>,
        "zeil@yandex-team.ru" <zeil@yandex-team.ru>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "catalin.marinas@arm.com" <catalin.marinas@arm.com>,
        "will@kernel.org" <will@kernel.org>
Subject: Re: [PATCH v6 00/12] HWPOISON: soft offline rework
Thread-Topic: [PATCH v6 00/12] HWPOISON: soft offline rework
Thread-Index: AQHWbyom92sGRlXQfk+BXW7m7bWNQ6kyO+2A
Date:   Tue, 11 Aug 2020 03:11:40 +0000
Message-ID: <20200811031139.GA7145@hori.linux.bs1.fc.nec.co.jp>
References: <20200806184923.7007-1-nao.horiguchi@gmail.com>
 <20200810152254.GC5307@lca.pw>
In-Reply-To: <20200810152254.GC5307@lca.pw>
Accept-Language: ja-JP, en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-2022-jp"
Content-ID: <0E671F1123F4F14E8A06168884936EE9@jpnprd01.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 958eb7c0-e420-4f57-39a5-08d83da44457
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Aug 2020 03:11:40.2583
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: e67df547-9d0d-4f4d-9161-51c6ed1f7d11
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Ld/99Nys2kf0faOYCY+FZpXBcYFuACUADmsDrbBI7S1zV3v28EEqn3QZKx083vZpyvxrp1+JuFonL37VfITfGA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYBPR01MB5486
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Aug 10, 2020 at 11:22:55AM -0400, Qian Cai wrote:
> On Thu, Aug 06, 2020 at 06:49:11PM +0000, nao.horiguchi@gmail.com wrote:
> > Hi,
> >=20
> > This patchset is the latest version of soft offline rework patchset
> > targetted for v5.9.
> >=20
> > Since v5, I dropped some patches which tweak refcount handling in
> > madvise_inject_error() to avoid the "unknown refcount page" error.
> > I don't confirm the fix (that didn't reproduce with v5 in my environmen=
t),
> > but this change surely call soft_offline_page() after holding refcount,
> > so the error should not happen any more.
>=20
> With this patchset, arm64 is still suffering from premature 512M-size hug=
epages
> allocation failures.
>=20
> # git clone https://gitlab.com/cailca/linux-mm
> # cd linux-mm; make
> # ./random 1
> - start: migrate_huge_offline
> - use NUMA nodes 0,1.
> - mmap and free 2147483648 bytes hugepages on node 0
> - mmap and free 2147483648 bytes hugepages on node 1
> madvise: Cannot allocate memory
>=20
> [  292.456538][ T3685] soft offline: 0x8a000: hugepage isolation failed: =
0, page count 2, type 7ffff80001000e (referenced|uptodate|dirty|head)
> [  292.469113][ T3685] Soft offlining pfn 0x8c000 at process virtual addr=
ess 0xffff60000000
> [  292.983855][ T3685] Soft offlining pfn 0x88000 at process virtual addr=
ess 0xffff40000000
> [  293.271369][ T3685] Soft offlining pfn 0x8a000 at process virtual addr=
ess 0xffff60000000
> [  293.834030][ T3685] Soft offlining pfn 0xa000 at process virtual addre=
ss 0xffff40000000
> [  293.851378][ T3685] soft offline: 0xa000: hugepage migration failed -1=
2, type 7ffff80001000e (referenced|uptodate|dirty|head)
>=20
> The fresh-booted system still had 40G+ memory free before running the tes=
t.

As I commented over v5, this failure is expected and it doesn't mean kernel
issue.  Once we successfully soft offline a hugepage, the memory range
covering the hugepage will never participate in hugepage because one of the
subpages is removed from buddy.  So if you iterate soft offlining hugepages=
,
all memory range are "holed" finally, and no hugepage will be available in
the system.

Please fix your test program to properly determine nubmer of loop (NR_LOOP)
so that you can assume that you can always allocate hugepage during testing=
.
For example, if you can use 40G memory and hugepage size is 512MB, NR_LOOP
should not be larger than 80.

>=20
> Reverting the following commits allowed the test to run succesfully over =
and over again.
>=20
> "mm, hwpoison: remove recalculating hpage"
> "mm,hwpoison-inject: don't pin for hwpoison_filter"
> "mm,hwpoison: Un-export get_hwpoison_page and make it static"
> "mm,hwpoison: kill put_hwpoison_page"
> "mm,hwpoison: unify THP handling for hard and soft offline"
> "mm,hwpoison: rework soft offline for free pages"
> "mm,hwpoison: rework soft offline for in-use pages"
> "mm,hwpoison: refactor soft_offline_huge_page and __soft_offline_page"

I'm still not sure why the test succeeded by reverting these because
current mainline kernel provides similar mechanism to prevent reuse of
soft offlined page. So this success seems to me something suspicious.

To investigate more, I want to have additional info about the page states
of the relevant pages after soft offlining.  Could you collect it by the
following steps?

  - modify random.c not to run hotplug_memory() in migrate_huge_hotplug_mem=
ory(),
  - compile it and run "./random 1" once,
  - to collect page state with hwpoisoned pages, run "./page-types -Nlr -b =
hwpoison",
    where page-types is available under tools/vm in kernel source tree.
  - choose a few pfns of soft offlined pages from kernel message
    "Soft offlining pfn ...", and run "./page-types -Nlr -a <pfn>".

Thanks,
Naoya Horiguchi

>=20
> i.e., it is not enough to only revert,
>=20
> mm,hwpoison: double-check page count in __get_any_page()
> mm,hwpoison: introduce MF_MSG_UNSPLIT_THP
> mm,hwpoison: return 0 if the page is already poisoned in soft-offline
> =