Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 12.22.5.234 as permitted sender) receiver=protection.outlook.com;
 client-ip=12.22.5.234; helo=mail.nvidia.com;
From:   Alistair Popple <apopple@nvidia.com>
To:     <akpm@linux-foundation.org>, <linux-mm@kvack.org>,
        David Hildenbrand <david@redhat.com>
CC:     <Felix.Kuehling@amd.com>, <rcampbell@nvidia.com>,
        <linux-ext4@vger.kernel.org>, <linux-xfs@vger.kernel.org>,
        <amd-gfx@lists.freedesktop.org>, <dri-devel@lists.freedesktop.org>,
        <hch@lst.de>, <jgg@nvidia.com>, <jglisse@redhat.com>,
        <willy@infradead.org>, <alex.sierra@amd.com>, <jhubbard@nvidia.com>
Subject: Re: [PATCH v2 2/3] mm/gup.c: Migrate device coherent pages when pinning instead of failing
Date:   Fri, 11 Feb 2022 10:41:05 +1100
Message-ID: <5251686.PpEh1BJ82l@nvdebian>
In-Reply-To: <fb557284-bcab-6d95-ac60-acd7459e9e80@redhat.com>
References: <cover.0d3c846b1c6c294e055ff7ebe221fab9964c1436.1644207242.git-series.apopple@nvidia.com> <1894939.704c7Wv018@nvdebian> <fb557284-bcab-6d95-ac60-acd7459e9e80@redhat.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="us-ascii"
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Feb 2022 23:41:17.8172
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 16a4bd5f-504c-415c-b966-08d9eceed59d
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[12.22.5.234];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT066.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2908
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

On Thursday, 10 February 2022 10:47:35 PM AEDT David Hildenbrand wrote:
> On 10.02.22 12:39, Alistair Popple wrote:
> > On Thursday, 10 February 2022 9:53:38 PM AEDT David Hildenbrand wrote:
> >> On 07.02.22 05:26, Alistair Popple wrote:
> >>> Currently any attempts to pin a device coherent page will fail. This is
> >>> because device coherent pages need to be managed by a device driver, and
> >>> pinning them would prevent a driver from migrating them off the device.
> >>>
> >>> However this is no reason to fail pinning of these pages. These are
> >>> coherent and accessible from the CPU so can be migrated just like
> >>> pinning ZONE_MOVABLE pages. So instead of failing all attempts to pin
> >>> them first try migrating them out of ZONE_DEVICE.
> >>>
> >>> Signed-off-by: Alistair Popple <apopple@nvidia.com>
> >>> Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
> >>> ---
> >>>
> >>> Changes for v2:
> >>>
> >>>  - Added Felix's Acked-by
> >>>  - Fixed missing check for dpage == NULL
> >>>
> >>>  mm/gup.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++++++------
> >>>  1 file changed, 95 insertions(+), 10 deletions(-)
> >>>
> >>> diff --git a/mm/gup.c b/mm/gup.c
> >>> index 56d9577..5e826db 100644
> >>> --- a/mm/gup.c
> >>> +++ b/mm/gup.c
> >>> @@ -1861,6 +1861,60 @@ struct page *get_dump_page(unsigned long addr)
> >>>  
> >>>  #ifdef CONFIG_MIGRATION
> >>>  /*
> >>> + * Migrates a device coherent page back to normal memory. Caller should have a
> >>> + * reference on page which will be copied to the new page if migration is
> >>> + * successful or dropped on failure.
> >>> + */
> >>> +static struct page *migrate_device_page(struct page *page,
> >>> +					unsigned int gup_flags)
> >>> +{
> >>> +	struct page *dpage;
> >>> +	struct migrate_vma args;
> >>> +	unsigned long src_pfn, dst_pfn = 0;
> >>> +
> >>> +	lock_page(page);
> >>> +	src_pfn = migrate_pfn(page_to_pfn(page)) | MIGRATE_PFN_MIGRATE;
> >>> +	args.src = &src_pfn;
> >>> +	args.dst = &dst_pfn;
> >>> +	args.cpages = 1;
> >>> +	args.npages = 1;
> >>> +	args.vma = NULL;
> >>> +	migrate_vma_setup(&args);
> >>> +	if (!(src_pfn & MIGRATE_PFN_MIGRATE))
> >>> +		return NULL;
> >>> +
> >>> +	dpage = alloc_pages(GFP_USER | __GFP_NOWARN, 0);
> >>> +
> >>> +	/*
> >>> +	 * get/pin the new page now so we don't have to retry gup after
> >>> +	 * migrating. We already have a reference so this should never fail.
> >>> +	 */
> >>> +	if (dpage && WARN_ON_ONCE(!try_grab_page(dpage, gup_flags))) {
> >>> +		__free_pages(dpage, 0);
> >>> +		dpage = NULL;
> >>> +	}
> >>> +
> >>> +	if (dpage) {
> >>> +		lock_page(dpage);
> >>> +		dst_pfn = migrate_pfn(page_to_pfn(dpage));
> >>> +	}
> >>> +
> >>> +	migrate_vma_pages(&args);
> >>> +	if (src_pfn & MIGRATE_PFN_MIGRATE)
> >>> +		copy_highpage(dpage, page);
> >>> +	migrate_vma_finalize(&args);
> >>> +	if (dpage && !(src_pfn & MIGRATE_PFN_MIGRATE)) {
> >>> +		if (gup_flags & FOLL_PIN)
> >>> +			unpin_user_page(dpage);
> >>> +		else
> >>> +			put_page(dpage);
> >>> +		dpage = NULL;
> >>> +	}
> >>> +
> >>> +	return dpage;
> >>> +}
> >>> +
> >>> +/*
> >>>   * Check whether all pages are pinnable, if so return number of pages.  If some
> >>>   * pages are not pinnable, migrate them, and unpin all pages. Return zero if
> >>>   * pages were migrated, or if some pages were not successfully isolated.
> >>> @@ -1888,15 +1942,40 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages,
> >>>  			continue;
> >>>  		prev_head = head;
> >>>  		/*
> >>> -		 * If we get a movable page, since we are going to be pinning
> >>> -		 * these entries, try to move them out if possible.
> >>> +		 * Device coherent pages are managed by a driver and should not
> >>> +		 * be pinned indefinitely as it prevents the driver moving the
> >>> +		 * page. So when trying to pin with FOLL_LONGTERM instead try
> >>> +		 * migrating page out of device memory.
> >>>  		 */
> >>>  		if (is_dev_private_or_coherent_page(head)) {
> >>> +			/*
> >>> +			 * device private pages will get faulted in during gup
> >>> +			 * so it shouldn't be possible to see one here.
> >>> +			 */
> >>>  			WARN_ON_ONCE(is_device_private_page(head));
> >>> -			ret = -EFAULT;
> >>> -			goto unpin_pages;
> >>> +			WARN_ON_ONCE(PageCompound(head));
> >>> +
> >>> +			/*
> >>> +			 * migration will fail if the page is pinned, so convert
> >>> +			 * the pin on the source page to a normal reference.
> >>> +			 */
> >>> +			if (gup_flags & FOLL_PIN) {
> >>> +				get_page(head);
> >>> +				unpin_user_page(head);
> >>> +			}
> >>> +
> >>> +			pages[i] = migrate_device_page(head, gup_flags);
> >>
> >> For ordinary migrate_pages(), we'll unpin all pages and return 0 so the
> >> caller will retry pinning by walking the page tables again.
> >>
> >> Why can't we apply the same mechanism here? This "let's avoid another
> >> walk" looks unnecessary complicated to me, but I might be wrong.
> > 
> > There's no reason we couldn't. I figured we have the page in the right spot
> > anyway so it was easy to do, and looking at this rebased on top of Christoph's
> > ZONE_DEVICE refcount simplification I'm not sure it would be any simpler
> > anyway.
> > 
> > It would remove the call to try_grab_page(), but we'd still have to return an
> > error on migration failures whilst also ensuring we putback any non-device
> > pages that may have been isolated. I might have overlooked something though,
> > so certainly happy for suggestions.
> 
> Staring at the code, I was wondering if we could either
> 
> * build a second list of device coherent pages to migrate and call a
>   migrate_device_pages() bulk function
> * simply use movable_page_list() and teach migrate_pages() how to handle
>   them.

I did consider that approach. The problem is zone device pages are not LRU
pages. In particular page->lru is not available to add the page to a list, and
as an external API and internally migrate_pages() relies heavily on moving
pages between lists.

> I'd really appreciate as little special casing as possible for the ever
> growing list of new DEVICE types all over the place. E.g., just staring
> at fork even before the new device coherent made my head spin.

That's fair. We could pull the checks for device pages out into a self
contained function (eg. check_and_migrate_device_pages()) called before
check_and_migrate_movable_pages(). The down side of that is we'd always have an
extra loop over all the pages just to scan for device pages, but perhaps that's
not a concern?