Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   =?iso-2022-jp?B?SE9SSUdVQ0hJIE5BT1lBKBskQktZOH0hIUQ+TGkbKEIp?= 
        <naoya.horiguchi@nec.com>
To:     Aristeu Rozanski <aris@ruivo.org>,
        Oscar Salvador <osalvador@suse.de>
CC:     "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "mhocko@kernel.org" <mhocko@kernel.org>,
        "tony.luck@intel.com" <tony.luck@intel.com>,
        "cai@lca.pw" <cai@lca.pw>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>
Subject: Re: [PATCH v3 0/5] HWpoison: further fixes and cleanups
Thread-Topic: [PATCH v3 0/5] HWpoison: further fixes and cleanups
Thread-Index: AQHWioASOS4BIdKZp02dPkadmQVT+qlqN5kAgAERxoA=
Date:   Wed, 16 Sep 2020 13:42:15 +0000
Message-ID: <20200916134215.GA30407@hori.linux.bs1.fc.nec.co.jp>
References: <20200914101559.17103-1-osalvador@suse.de>
 <20200915212222.GA18315@cathedrallabs.org>
In-Reply-To: <20200915212222.GA18315@cathedrallabs.org>
Accept-Language: ja-JP, en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-2022-jp"
Content-ID: <32DC52BA5F0AE043838FFB17157C4154@jpnprd01.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 8ca54f40-7164-4b1f-4fb2-08d85a4652fa
X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Sep 2020 13:42:15.8502
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: e67df547-9d0d-4f4d-9161-51c6ed1f7d11
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: zmlBNMMaFybitvickbNTaZP5hs3AykdBRmkTOaUO345upzLeaP5BTuGEqpzZ8s3Jj7z1TqwxEQJ8IK0NLQKhRQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYAPR01MB4766
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Sep 15, 2020 at 05:22:22PM -0400, Aristeu Rozanski wrote:
> Hi Oscar, Naoya,
>=20
> On Mon, Sep 14, 2020 at 12:15:54PM +0200, Oscar Salvador wrote:
> > The important bit of this patchset is patch#1, which is a fix to take o=
ff
> > HWPoison pages off a buddy freelist since it can lead us to having HWPo=
ison
> > pages back in the game without no one noticing it.
> > So fix it (we did that already for soft_offline_page [1]).
> >=20
> > The other patches are clean-ups and not that important, so if anything,
> > consider patch#1 for inclusion.
> >=20
> > [1] https://patchwork.kernel.org/cover/11704083/
>=20
> I found something strange with your and Naoya's hwpoison rework. We have =
a
> customer with a testcase that basically does:
>=20
> 	p1 =3D mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYM=
OUS, -1, 0);
> 	p2 =3D mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYM=
OUS, -1, 0);
>=20
> 	madvise(p1, size, MADV_MERGEABLE);
> 	madvise(p2, size, MADV_MERGEABLE);
>=20
> 	memset(p1, 'a', size);
> 	memset(p2, 'a', size);
>=20
> 	madvise(p1, size, MADV_SOFT_OFFLINE);
>=20
> 	madvise(p1, size, MADV_UNMERGEABLE);
> 	madvise(p2, size, MADV_UNMERGEABLE);
>=20
>=20
> where size is about 200,000 pages. It works on a x86_64 box (with and wit=
hout the
> hwpoison rework). On ppc64 boxes (tested 3 different ones with at least 2=
50GB memory)
> it fails to take a page off the buddy list (page_handle_poison()/take_pag=
e_off_buddy())
> (madvise MADV_SOFT_OFFLINE returns -EBUSY). Without the hwpoison rework t=
he test passes.

I reproduced the similar -EBUSY with small average x86 VM, where it seems t=
o me
a race between page_take_off_buddy() and page allocation.  Oscar's debug pa=
tch
shows the following kernel messages:

    [  627.357009] Soft offlining pfn 0x235018 at process virtual address 0=
x7fd112140000
    [  627.358747] __get_any_page: 0x235018 free buddy page
    [  627.359875] page:00000000038b52c9 refcount:0 mapcount:-128 mapping:0=
000000000000000 index:0x1 pfn:0x235018
    [  627.362002] flags: 0x57ffe000000000()
    [  627.362841] raw: 0057ffe000000000 fffff84648d12688 ffff955abffd1dd0 =
0000000000000000
    [  627.364555] raw: 0000000000000001 0000000000000000 00000000ffffff7f =
0000000000000000
    [  627.366258] page dumped because: page_handle_poison
    [  627.367357] page->mem_cgroup:ffff9559b6912000
    [  627.368342] page_handle_poison: hugepage_or_freepage failed\xb8n
    [  627.368344] soft_offline_free_page: page_handle_poison -EBUSY
    [  627.370901] page:00000000038b52c9 refcount:6 mapcount:3 mapping:0000=
00001226bf89 index:0x2710 pfn:0x235018
    [  627.373048] aops:ext4_da_aops ino:c63f3 dentry name:"system.journal"
    [  627.374526] flags: 0x57ffe00000201c(uptodate|dirty|lru|private)
    [  627.375865] raw: 0057ffe00000201c fffff84648d300c8 ffff955ab8c3f020 =
ffff955aba5f4ee0
    [  627.377586] raw: 0000000000002710 ffff9559b811fc98 0000000500000002 =
ffff9559b6912000
    [  627.379308] page dumped because: soft_offline_free_page
    [  627.380480] page->mem_cgroup:ffff9559b6912000

    CPU 0                                CPU 1

    get_any_page // returns 0 (free buddy path)
      soft_offline_free_page
                                         the page is allocated
        page_handle_poison -> fail
          return -EBUSY

I'm still not sure why this issue is invisible before rework patch,
but setting migrate type to MIGRATE_ISOLATE during offlining could affect
the behavior sensitively.

Thanks,
Naoya Horiguchi=