Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   =?iso-2022-jp?B?SE9SSUdVQ0hJIE5BT1lBKBskQktZOH0hIUQ+TGkbKEIp?= 
        <naoya.horiguchi@nec.com>
To:     Qian Cai <cai@lca.pw>
CC:     "nao.horiguchi@gmail.com" <nao.horiguchi@gmail.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "mhocko@kernel.org" <mhocko@kernel.org>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "mike.kravetz@oracle.com" <mike.kravetz@oracle.com>,
        "osalvador@suse.de" <osalvador@suse.de>,
        "tony.luck@intel.com" <tony.luck@intel.com>,
        "david@redhat.com" <david@redhat.com>,
        "aneesh.kumar@linux.vnet.ibm.com" <aneesh.kumar@linux.vnet.ibm.com>,
        "zeil@yandex-team.ru" <zeil@yandex-team.ru>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v5 00/16] HWPOISON: soft offline rework
Thread-Topic: [PATCH v5 00/16] HWPOISON: soft offline rework
Thread-Index: AQHWaclRY/IcAl5ZPUiTqlijcX3D9KknJkIAgAAJNgCAAs+EgA==
Date:   Wed, 5 Aug 2020 20:44:58 +0000
Message-ID: <20200805204457.GB16406@hori.linux.bs1.fc.nec.co.jp>
References: <20200731122112.11263-1-nao.horiguchi@gmail.com>
 <20200803190709.GB8894@lca.pw>
 <20200804011644.GA25028@hori.linux.bs1.fc.nec.co.jp>
 <20200804014942.GC8894@lca.pw>
In-Reply-To: <20200804014942.GC8894@lca.pw>
Accept-Language: ja-JP, en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-2022-jp"
Content-ID: <574DB1C497E1CD45980F6F9A27505DBE@jpnprd01.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: d2208103-ccf7-4bf0-f14e-08d839806abb
X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Aug 2020 20:44:58.0934
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: e67df547-9d0d-4f4d-9161-51c6ed1f7d11
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: hPmUzlNvr1xjrhvCIfl7C7quouhJGEd+K49dOXZB5UBdR+/OIDdcXx8B+oKPiDzk6n3rr9Kc38zsgcdWPXCxYQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TY1PR01MB1610
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Aug 03, 2020 at 09:49:42PM -0400, Qian Cai wrote:
> On Tue, Aug 04, 2020 at 01:16:45AM +0000, HORIGUCHI NAOYA(=1B$BKY8}=1B(B =
=1B$BD>Li=1B(B) wrote:
> > On Mon, Aug 03, 2020 at 03:07:09PM -0400, Qian Cai wrote:
> > > On Fri, Jul 31, 2020 at 12:20:56PM +0000, nao.horiguchi@gmail.com wro=
te:
> > > > This patchset is the latest version of soft offline rework patchset
> > > > targetted for v5.9.
> > > >=20
> > > > Main focus of this series is to stabilize soft offline.  Historical=
ly soft
> > > > offlined pages have suffered from racy conditions because PageHWPoi=
son is
> > > > used to a little too aggressively, which (directly or indirectly) i=
nvades
> > > > other mm code which cares little about hwpoison.  This results in u=
nexpected
> > > > behavior or kernel panic, which is very far from soft offline's "do=
 not
> > > > disturb userspace or other kernel component" policy.
> > > >=20
> > > > Main point of this change set is to contain target page "via buddy =
allocator",
> > > > where we first free the target page as we do for normal pages, and =
remove
> > > > from buddy only when we confirm that it reaches free list. There is=
 surely
> > > > race window of page allocation, but that's fine because someone rea=
lly want
> > > > that page and the page is still working, so soft offline can happil=
y give up.
> > > >=20
> > > > v4 from Oscar tries to handle the race around reallocation, but tha=
t part
> > > > seems still work in progress, so I decide to separate it for change=
s into
> > > > v5.9.  Thank you for your contribution, Oscar.
> > > >=20
> > > > The issue reported by Qian Cai is fixed by patch 16/16.
> > > >=20
> > > > This patchset is based on v5.8-rc7-mmotm-2020-07-27-18-18, but I ap=
plied
> > > > this series after reverting previous version.
> > > > Maybe https://github.com/Naoya-Horiguchi/linux/commits/soft-offline=
-rework.v5
> > > > shows what I did more precisely.
> > > >=20
> > > > Any other comment/suggestion/help would be appreciated.
> > >=20
> > > There is another issue with this patchset (with and without the patch=
 [1]).
> > >=20
> > > [1] https://lore.kernel.org/lkml/20200803133657.GA13307@hori.linux.bs=
1.fc.nec.co.jp/
> > >=20
> > > Arm64 using 512M-size hugepages starts to fail allocations prematurel=
y.
> > >=20
> > > # ./random 1
> > > - start: migrate_huge_offline
> > > - use NUMA nodes 0,1.
> > > - mmap and free 2147483648 bytes hugepages on node 0
> > > - mmap and free 2147483648 bytes hugepages on node 1
> > > madvise: Cannot allocate memory
> > >=20
> > > [  284.388061][ T3706] soft offline: 0x956000: hugepage isolation fai=
led: 0, page count 2, type 17ffff80001000e (referenced|uptodate|dirty|head)
> > > [  284.400777][ T3706] Soft offlining pfn 0x8e000 at process virtual =
address 0xffff80000000
> > > [  284.893412][ T3706] Soft offlining pfn 0x8a000 at process virtual =
address 0xffff60000000
> > > [  284.901539][ T3706] soft offline: 0x8a000: hugepage isolation fail=
ed: 0, page count 2, type 7ffff80001000e (referenced|uptodate|dirty|head)
> > > [  284.914129][ T3706] Soft offlining pfn 0x8c000 at process virtual =
address 0xffff80000000
> > > [  285.433497][ T3706] Soft offlining pfn 0x88000 at process virtual =
address 0xffff60000000
> > > [  285.720377][ T3706] Soft offlining pfn 0x8a000 at process virtual =
address 0xffff80000000
> > > [  286.281620][ T3706] Soft offlining pfn 0xa000 at process virtual a=
ddress 0xffff60000000
> > > [  286.290065][ T3706] soft offline: 0xa000: hugepage migration faile=
d -12, type 7ffff80001000e (referenced|uptodate|dirty|head)
> >=20
> > I think that this is due to the lack of contiguous memory.
> > This test program iterates soft offlining many times for hugepages,
> > so finally one page in every 512MB will be removed from buddy, then we
> > can't allocate hugepage any more even if we have enough free pages.
> > This is not good for heavy hugepage users, but that should be intended.
> >=20
> > It seems that random.c calls madvise(MADV_SOFT_OFFLINE) for 2 hugepages=
,
> > and iterates it 1000 (=3D=3DNR_LOOP) times, so if the system doesn't ha=
ve
> > enough memory to cover the range of 2000 hugepages (1000GB in the Arm64
> > system), this ENOMEM should reproduce as expected.
>=20
> Well, each iteration will mmap/munmap, so there should be no leaking.=20
>=20
> https://gitlab.com/cailca/linux-mm/-/blob/master/random.c#L376
>=20
> It also seem to me madvise(MADV_SOFT_OFFLINE) does start to fragment memo=
ry
> somehow, because after this "madvise: Cannot allocate memory" happened, I
> immediately checked /proc/meminfo and then found no hugepage usage at all=
.
>=20
> >=20
> > >=20
> > > Reverting this patchset and its dependency patchset [2] (reverting th=
e
> > > dependency alone did not help) fixed it,
> >=20
> > But it's still not clear to me why this was not visible before this
> > patchset, so I need more check for it.

I've reproduced ENOMEM with v5.8 (without this patchset) simply by using
VM with small memory (4GB). So this specific error seems not to be caused
by this series.

Thanks,
Naoya Horiguchi=