Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CA+CK2bBffHBxjmb9jmSKacm0fJMinyt3Nhk8Nx6iudcQSj80_w@mail.gmail.com>
 <d668b0f2-2644-0f5e-a8c1-a6b8f515e9ab@suse.cz>
In-Reply-To: <d668b0f2-2644-0f5e-a8c1-a6b8f515e9ab@suse.cz>
From:   Pavel Tatashin <pasha.tatashin@soleen.com>
Date:   Mon, 23 Nov 2020 11:31:59 -0500
Message-ID: <CA+CK2bBuEhH7cSEZUKTYE_g9mw_rwEG-v1Jk4BL6WuLWK824Aw@mail.gmail.com>
Subject: Re: Pinning ZONE_MOVABLE pages
To:     Vlastimil Babka <vbabka@suse.cz>
Cc:     linux-mm <linux-mm@kvack.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Michal Hocko <mhocko@suse.com>,
        David Hildenbrand <david@redhat.com>,
        Oscar Salvador <osalvador@suse.de>,
        Dan Williams <dan.j.williams@intel.com>,
        Sasha Levin <sashal@kernel.org>,
        Tyler Hicks <tyhicks@linux.microsoft.com>,
        Joonsoo Kim <iamjoonsoo.kim@lge.com>, sthemmin@microsoft.com,
        John Hubbard <jhubbard@nvidia.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

> Makes sense, as this means no userspace change.
>
> > 2. Add an internal move_pages_zone() similar to move_pages() syscall
> > but instead of migrating to a different NUMA node, migrate pages from
> > ZONE_MOVABLE to another zone.
> > Call move_pages_zone() on demand prior to pinning pages from
> > vfio_pin_map_dma() for instance.
>
> As others already said, migrating away before the longterm pin should be
> the solution. IIRC it was one of the goals of long term pinning api
> proposed long time ago by Peter Ziljstra I think? The implementation
> that was merged relatively recently doesn't do that (yet?) for all
> movable pages, just CMA, but it could.

From what I can tell, CMA is not solving exactly this problem. It
migrates pages from CMA before pinning, but it migrates them to
ZONE_MOVABLE. Also, we still need to take care of the fault scenario.

>
> > 3. Perhaps, it also makes sense to add madvise() flag, to allocate
> > pages from non-movable zone. When a user application knows that it
> > will do DMA mapping, and pin pages for a long time, the memory that it
> > allocates should never be migrated or hot-removed, so make sure that
> > it comes from the appropriate place.
> > The benefit of adding madvise() flag is that we won't have to deal
> > with slow page migration during pin time, but the disadvantage is that
> > we would need to change the user interface.
>
> It's best if we avoid involving userspace until it's shown that's it's
> insufficient.

Agree.

Thank you,
Pasha