Date:   Tue, 23 Nov 2021 19:59:53 -0400
From:   Jason Gunthorpe <jgg@ziepe.ca>
To:     Vlastimil Babka <vbabka@suse.cz>
Cc:     David Hildenbrand <david@redhat.com>, Jens Axboe <axboe@kernel.dk>,
        Andrew Dona-Couch <andrew@donacou.ch>,
        Andrew Morton <akpm@linux-foundation.org>,
        Drew DeVault <sir@cmpwn.com>,
        Ammar Faizi <ammarfaizi2@gnuweeb.org>,
        linux-kernel@vger.kernel.org, linux-api@vger.kernel.org,
        io_uring Mailing List <io-uring@vger.kernel.org>,
        Pavel Begunkov <asml.silence@gmail.com>, linux-mm@kvack.org
Subject: Re: [PATCH] Increase default MLOCK_LIMIT to 8 MiB
Message-ID: <20211123235953.GF5112@ziepe.ca>
References: <8f219a64-a39f-45f0-a7ad-708a33888a3b@www.fastmail.com>
 <333cb52b-5b02-648e-af7a-090e23261801@redhat.com>
 <ca96bb88-295c-ccad-ed2f-abc585cb4904@kernel.dk>
 <5f998bb7-7b5d-9253-2337-b1d9ea59c796@redhat.com>
 <20211123132523.GA5112@ziepe.ca>
 <10ccf01b-f13a-d626-beba-cbee70770cf1@redhat.com>
 <20211123140709.GB5112@ziepe.ca>
 <e4d7d211-5d62-df89-8f94-e49385286f1f@redhat.com>
 <20211123170056.GC5112@ziepe.ca>
 <dd92a69a-6d09-93a1-4f50-5020f5cc59d0@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <dd92a69a-6d09-93a1-4f50-5020f5cc59d0@suse.cz>
Precedence: bulk

On Tue, Nov 23, 2021 at 11:04:04PM +0100, Vlastimil Babka wrote:
> On 11/23/21 18:00, Jason Gunthorpe wrote:
> > 
> >> believe what you say and I trust your experience :) So could as well be
> >> that on such a "special" (or not so special) systems there should be a
> >> way to restrict it to privileged users only.
> > 
> > At this point RDMA is about as "special" as people running large
> > ZONE_MOVABLE systems, and the two are going to start colliding
> > heavily. The RDMA VFIO migration driver should be merged soon which
> > makes VMs using this stuff finally practical.
> 
> How does that work, I see the word migration, so does it cause pages to

Sorry I mean what is often called "VM live migration". Typically that
cannot be done if a PCI device is assigned to the VM as suspending and
the migrating a PCI device to another server is complicated. With
forthcoming hardware mlx5 can do this and thus the entire RDMA stack
becomes practically usable and performant within a VM.

> be migrated out of ZONE_MOVABLE before they are pinned?

GUP already does this automatically for FOLL_LONGTERM.

> Similarly for io-uring we could be migrating pages to be pinned so that
> the end up consolidated close together, and prevent pathologic
> situations like in David's reproducer. 

It is an interesting idea to have GUP do some kind of THP preserving
migration.

Jason