Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Subject: Re: memory offline infinite loop after soft offline
From:   Qian Cai <cai@lca.pw>
In-Reply-To: <20191021031641.GA8007@hori.linux.bs1.fc.nec.co.jp>
Date:   Thu, 14 May 2020 22:46:33 -0400
Cc:     Michal Hocko <mhocko@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        David Hildenbrand <david@redhat.com>,
        Mike Kravetz <mike.kravetz@oracle.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <DE0721DF-9E5C-4719-B382-01A4A74C04AD@lca.pw>
References: <20191018063222.GA15406@hori.linux.bs1.fc.nec.co.jp>
 <64DC81FB-C1D2-44F2-981F-C6F766124B91@lca.pw>
 <20191021031641.GA8007@hori.linux.bs1.fc.nec.co.jp>
To:     Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>,
        Oscar Salvador <osalvador@suse.de>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On Oct 20, 2019, at 11:16 PM, Naoya Horiguchi =
<n-horiguchi@ah.jp.nec.com> wrote:
>=20
> On Fri, Oct 18, 2019 at 07:56:09AM -0400, Qian Cai wrote:
>>=20
>>=20
>>    On Oct 18, 2019, at 2:35 AM, Naoya Horiguchi =
<n-horiguchi@ah.jp.nec.com>
>>    wrote:
>>=20
>>=20
>>    You're right, then I don't see how this happens. If the error =
hugepage was
>>    isolated without having PG_hwpoison set, it's unexpected and =
problematic.
>>    I'm testing myself with v5.4-rc2 (simply ran move_pages12 and did =
hotremove
>>    /hotadd)
>>    but don't reproduce the issue yet.  Do we need specific kernel =
version/
>>    config
>>    to trigger this?
>>=20
>>=20
>> This is reproducible on linux-next with the config. Not sure if it is
>> reproducible on x86.
>>=20
>> =
https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config
>>=20
>> and kernel cmdline if that matters
>>=20
>> page_poison=3Don page_owner=3Don numa_balancing=3Denable \
>> systemd.unified_cgroup_hierarchy=3D1 debug_guardpage_minorder=3D1 \
>> page_alloc.shuffle=3D1
>=20
> Thanks for the info.
>=20
>>=20
>> BTW, where does the code set PG_hwpoison for the head page?
>=20
> Precisely speaking, soft offline only sets PG_hwpoison after the =
target
> hugepage is successfully dissolved (then it's not a hugepage any =
more),
> so PG_hwpoison is set on the raw page in =
set_hwpoison_free_buddy_page().
>=20
> In move_pages12 case, madvise(MADV_SOFT_OFFLINE) is called for the =
range
> of 2 hugepages, so the expected result is that page offset 0 and 512
> are marked as PG_hwpoison after injection.
>=20
> Looking at your dump_page() output, the end_pfn is page offset 1
> ("page:c00c000800458040" is likely to point to pfn 0x11601.)
> The page belongs to high order buddy free page, but doesn't have
> PageBuddy nor PageHWPoison because it was not the head page or
> the raw error page.
>=20
>> Unfortunately, this does not solve the problem. It looks to me that =
in           =20
>> soft_offline_huge_page(), set_hwpoison_free_buddy_page() will only =
set           =20
>> PG_hwpoison for buddy pages, so the even the compound_head() has no =
PG_hwpoison  =20
>> set.                                                                  =
           =20
>=20
> Your analysis is totally correct, and this behavior will be fixed by
> the change (https://lkml.org/lkml/2019/10/17/551) in Oscar's rework.
> The raw error page will be taken off from buddy system and the other
> subpages are properly split into lower orderer pages (we'll properly
> manage PageBuddy flags). So all possible cases would be covered by
> branches in __test_page_isolated_in_pageblock.

Naoya, Oscar, it looks like this series was stuck.

https://lkml.org/lkml/2019/10/17/551

I can still reproduce this issue as today. Maybe it is best we could =
post a single patch (which one?) to fix the loop first?