Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From:   David Hildenbrand <david@redhat.com>
Mime-Version: 1.0 (1.0)
Subject: Re: Pinning ZONE_MOVABLE pages
Date:   Fri, 20 Nov 2020 22:34:49 +0100
Message-Id: <9C8F0E5D-2D42-4BA5-A5B5-9E049E1BE862@redhat.com>
References: <20201120211707.GC4327@casper.infradead.org>
Cc:     David Hildenbrand <david@redhat.com>,
        Pavel Tatashin <pasha.tatashin@soleen.com>,
        linux-mm <linux-mm@kvack.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Vlastimil Babka <vbabka@suse.cz>,
        LKML <linux-kernel@vger.kernel.org>,
        Michal Hocko <mhocko@suse.com>,
        Oscar Salvador <osalvador@suse.de>,
        Dan Williams <dan.j.williams@intel.com>,
        Sasha Levin <sashal@kernel.org>,
        Tyler Hicks <tyhicks@linux.microsoft.com>,
        Joonsoo Kim <iamjoonsoo.kim@lge.com>, sthemmin@microsoft.com
In-Reply-To: <20201120211707.GC4327@casper.infradead.org>
To:     Matthew Wilcox <willy@infradead.org>
Precedence: bulk


> Am 20.11.2020 um 22:17 schrieb Matthew Wilcox <willy@infradead.org>:
>=20
> =EF=BB=BFOn Fri, Nov 20, 2020 at 09:59:24PM +0100, David Hildenbrand wrote=
:
>>=20
>>>> Am 20.11.2020 um 21:28 schrieb Pavel Tatashin <pasha.tatashin@soleen.co=
m>:
>>>=20
>>> =EF=BB=BFRecently, I encountered a hang that is happening during memory h=
ot
>>> remove operation. It turns out that the hang is caused by pinned user
>>> pages in ZONE_MOVABLE.
>>>=20
>>> Kernel expects that all pages in ZONE_MOVABLE can be migrated, but
>>> this is not the case if a user applications such as through dpdk
>>> libraries pinned them via vfio dma map. Kernel keeps trying to
>>> hot-remove them, but refcnt never gets to zero, so we are looping
>>> until the hardware watchdog kicks in.
>>>=20
>>> We cannot do dma unmaps before hot-remove, because hot-remove is a
>>> slow operation, and we have thousands for network flows handled by
>>> dpdk that we just cannot suspend for the duration of hot-remove
>>> operation.
>>>=20
>>=20
>> Hi!
>>=20
>> It=E2=80=98s a known problem also for VMs using vfio. I thought about thi=
s some while ago an came to the same conclusion: before performing long-term=
 pinnings, we have to migrate pages off the movable zone. After that, it=E2=80=
=98s too late.
>=20
> We can't, though.  VMs using vfio pin their entire address space (right?)
> so we end up with basically all of the !MOVABLE memory used for VMs and
> the MOVABLE memory goes unused (I'm thinking about the case of a machine
> which only hosts VMs and has nothing else to do with its memory).  In
> that case, the sysadmin is going to reconfigure ZONE_MOVABLE away, and
> now we just don't have any ZONE_MOVABLE.  So what's the point?

When the guest is using an vIOMMU, it will only pin what=E2=80=98s currently=
 mapped by the guest into the vIOMMU. Otherwise: yes.

If you assume all memory will be used for VMs with vfio, then yes: no ZONE_M=
OVABLE, no memory hotunplug. If its=E2=80=98s only some VMs, it=E2=80=98s a d=
ifferent story.

>=20
> ZONE_MOVABLE can also be pinned by mlock() and other such system calls.

Mlocked pages can be migrated, no? They are simply not swappable iirc.

> The kernel needs to understand that ZONE_MOVABLE memory may not actually
> be movable, and skip the unmovable stuff.
>=20

Then you don=E2=80=98t have unplug guarantees. Memory unplug broken by desig=
n. Then there is no point in optimizing that case at all and tell customers =E2=
=80=9Evfio and memory hotunplug is incompatible=E2=80=9C. The only ugly thin=
g is the endless loop.=