Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <93f2614e-4521-8bc8-2eca-e7ad03e7e399@redhat.com>
Date:   Wed, 12 Apr 2023 10:17:18 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.1
Content-Language: en-US
To:     "Teterevkov, Ivan" <Ivan.Teterevkov@amd.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "jhubbard@nvidia.com" <jhubbard@nvidia.com>,
        "jack@suse.cz" <jack@suse.cz>,
        "rppt@linux.ibm.com" <rppt@linux.ibm.com>,
        "jglisse@redhat.com" <jglisse@redhat.com>,
        "ira.weiny@intel.com" <ira.weiny@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>,
        Christoph Hellwig <hch@lst.de>,
        Matthew Wilcox <willy@infradead.org>
References: <PH0PR12MB5606D4611050BC8B1CC430FEF09A9@PH0PR12MB5606.namprd12.prod.outlook.com>
From:   David Hildenbrand <david@redhat.com>
Organization: Red Hat
Subject: Re: find_get_page() VS pin_user_pages()
In-Reply-To: <PH0PR12MB5606D4611050BC8B1CC430FEF09A9@PH0PR12MB5606.namprd12.prod.outlook.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 11.04.23 21:43, Teterevkov, Ivan wrote:
> Hello folks,
> 
> I work with an application which aims to share memory in the userspace and
> interact with the NIC DMA. The memory allocation workflow begins in the
> userspace, which creates a new file backed by 2MiB hugepages with
> memfd_create(MFD_HUGETLB, MFD_HUGE_2MB) and fallocate(). Then the userspace
> makes an IOCTL to the kernel module with the file descriptor and size so that
> the kernel module can get the struct page with find_get_page(). Then the kernel
> module calls dma_map_single(page_address(page)) for NIC, which concludes the
> datapath. The allocated memory may (significantly) outlive the originating
> userspace application. The hugepages stay mapped with NIC, and the kernel
> module wants to continue using them and map to other applications that come and
> go with vm_mmap().
> 
> I am studying the pin_user_pages*() family of functions, and I wonder if the
> outlined workflow requires it. The hugepages do not page out, but they can move
> as they may be allocated with GFP_HIGHUSER_MOVABLE. However, find_get_page()
> must increment the page reference counter without mapping and prevent it from
> moving. In particular, https://docs.kernel.org/mm/page_migration.html:

I suspect that find_get_page() is not the kind of interface you want to 
use for the purpose you describe. find_get_page() is a wrapper around 
pagecache_get_page() and seems more like a helper for implementing an fs 
(looking at the users and the fact that it only considers pages that are 
in the pagecache).

Instead, you might want to mmap the memfd and pass the user space 
address range to the ioctl. There, you'd call pin_user_pages_*().

In general, for long-term pinning a page (possibly keeping the page 
pinned forever, controlled by user space, which seems to be what you are 
doing) you want so use pin_user_pages() with FOLL_LONGTERM. That will 
try migrating the page off of e.g., ZONE_MOVABLE or MIGRATE_CMA, where 
movability has to be guaranteed.


But I am no fs expert, so I'll cc some people that might know better if 
this would be an abuse of find_get_page().


-- 
Thanks,

David / dhildenb