Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Alistair Popple <apopple@nvidia.com>
To:     Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org
Cc:     Alistair Popple <apopple@nvidia.com>, linux-kernel@vger.kernel.org,
        amd-gfx@lists.freedesktop.org, nouveau@lists.freedesktop.org,
        dri-devel@lists.freedesktop.org, Lyude Paul <lyude@redhat.com>,
        Ben Skeggs <bskeggs@redhat.com>,
        Ralph Campbell <rcampbell@nvidia.com>,
        John Hubbard <jhubbard@nvidia.com>
Subject: [PATCH v2 7/8] nouveau/dmem: Evict device private memory during release
Date:   Wed, 28 Sep 2022 22:01:21 +1000
Message-Id: <66277601fb8fda9af408b33da9887192bf895bda.1664366292.git-series.apopple@nvidia.com>
In-Reply-To: <cover.60659b549d8509ddecafad4f498ee7f03bb23c69.1664366292.git-series.apopple@nvidia.com>
References: <cover.60659b549d8509ddecafad4f498ee7f03bb23c69.1664366292.git-series.apopple@nvidia.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?kmL+har2Xav8wqyZC/khsNbgUUdiQTqjj3Ujj/A/L8kYrzCsbGEzrMRmTEGl?=
 =?us-ascii?Q?7CGnCrgRN5CqnlKhHkX2squoLt6xI+RbMCTWzNYclaic8r6bCzcxA02ydZKO?=
 =?us-ascii?Q?/lNDfCz7LwL3Rh/PRcKHLU8CBrlVLpqId2aX8l31d9SABVI3FQS0xKexhIjk?=
 =?us-ascii?Q?yOs0HLFJ5CeKBiHbusn6JhlhxFCs9iIlmddBVeZ/3jaakyvzsBFb2PTDj7tW?=
 =?us-ascii?Q?FAkoJfEQgW2vYGhCBD2DPK9vsjAr9FvXiPvB2+jIwXuvr3adszFjj9XrlByS?=
 =?us-ascii?Q?gso2+FSm6sUACa11NrnpQkrXFzlZ1+AESqwAC72MRHHtCOsO7vMRf/BOOpy7?=
 =?us-ascii?Q?hiX6P/aG84N7uEDirE2Vk+CNy/W4vAyVIkb4CV+qiG5qjEER4eqObWruE3IO?=
 =?us-ascii?Q?SB35yxhwgYmqDc7ov6g7hWNDGgkaCOAssnhVwJy5+dQdCUwVNpcwFhu7XuRM?=
 =?us-ascii?Q?ak4GybSPo7QSz+GXoJS10HyNeLE1HtQ2rATtneEjELeWWKgs/DZNBpziuDhq?=
 =?us-ascii?Q?wnlvFH0oNwGI24KJLVZQ7d536psyqOwUVHtUqI1s/lQVIUyC89HZNDu1m4iN?=
 =?us-ascii?Q?P2XmwWfMzNiYzCkO7FD/CPgYIKD7Y8w7CAaV5+Ruo0Vqtf7IQep2ibDR3isF?=
 =?us-ascii?Q?qY7gR/H8V0Y3QAylBfXuP+Z2D3NPsM7nVkcMotQVnoFNGfo9bgbEpeHn37Tt?=
 =?us-ascii?Q?gcHo30RGfhhK8AjwB9WRvAU60PLhmpkLJ4zeY5IwS8+JAJShaxoG14R7y7lN?=
 =?us-ascii?Q?ctjZJshcPX6OHoAbxDus9VJrUF5zT6QdnJ5MxONdn++ZsMHq16ZNNoouhHYz?=
 =?us-ascii?Q?zbfTq6kvoyN453x2vnb517Zgpg/37x3qkR62vcvCvyYtee/q1DWj7xECvxiB?=
 =?us-ascii?Q?NUm+m900forIPOJWhvQGPFDPPxiA0z1ORxe7JavTE2fLF/uV4bPDIIX9tGMb?=
 =?us-ascii?Q?+yAUyzJJeRqPyCzIlvyBv+qOijLZ6TEdiutrMPnknggh+AS8PDj2CjKJU5Hj?=
 =?us-ascii?Q?62i7H/g5eyUPA+x+4nIJAGV0SuQHaoWp2rWNYVk9Z/Y0OEIch4Rfcce5bjo1?=
 =?us-ascii?Q?LMfFateG+g/dNAcJgC1aW2LACPeXmmmzuYL7a5ma7UnDGBDGgbLrx/Ptamgt?=
 =?us-ascii?Q?MiF7ar6uwxrVAVKbwefJ0D7LWPAycpdQCj2kL018gb94Km//K5acQq2eEj2n?=
 =?us-ascii?Q?ZNfPiP5wG6guFGDZy9eSjQoIEaGXGgSPXDc0Hpp6s/jZ+y/bB3Rq04my3/qB?=
 =?us-ascii?Q?KQnDFJPQVf4xnLYBmRcam4pa/A1PwAMtS5FbZ/8VbpZS5zAj/HuTiBmlggO5?=
 =?us-ascii?Q?l0EJKDlgjktG/yBfukAX7cNjNT4/tpq/ndoEgCnWh5GSGXheu44Vs6UOD7fX?=
 =?us-ascii?Q?0fklz4rTIUeH+zjmaiSoRHrfWmU4v8iB/cJt8D+OJASR0voYZv/IRz6fQI45?=
 =?us-ascii?Q?yj2lhxiDjmCcflz6PpVlvEjlv8MkCNp72gNOpYtIoCQo2acndOwgWO1kbE1i?=
 =?us-ascii?Q?ELzm3m26hLPVnDuQnC4Sn9wLQn/4yUATcD4eqiKTQZ4XIlGfITG4Xp5DIif2?=
 =?us-ascii?Q?gqsXpexaW9tR1aMJzAP0JLwb/iZDWbiVgrFN2krR?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: eb695943-c5a0-4f81-6e03-08daa1494e55
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2022 12:02:24.6971
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1UZhIYBuFJ4j59BfUjyLjNKXg21Ejm1SGHcqPwPTPiwHkzt5AoMoYxIsqkW++igFxNlsGlBd4Jl8rztmFxtfCw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7604
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

When the module is unloaded or a GPU is unbound from the module it is
possible for device private pages to still be mapped in currently
running processes. This can lead to a hangs and RCU stall warnings when
unbinding the device as memunmap_pages() will wait in an uninterruptible
state until all device pages have been freed which may never happen.

Fix this by migrating device mappings back to normal CPU memory prior to
freeing the GPU memory chunks and associated device private pages.

Signed-off-by: Alistair Popple <apopple@nvidia.com>
Cc: Lyude Paul <lyude@redhat.com>
Cc: Ben Skeggs <bskeggs@redhat.com>
Cc: Ralph Campbell <rcampbell@nvidia.com>
Cc: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/drm/nouveau/nouveau_dmem.c | 48 +++++++++++++++++++++++++++-
 1 file changed, 48 insertions(+)

diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouveau/nouveau_dmem.c
index 65f51fb..5fe2091 100644
--- a/drivers/gpu/drm/nouveau/nouveau_dmem.c
+++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c
@@ -367,6 +367,52 @@ nouveau_dmem_suspend(struct nouveau_drm *drm)
 	mutex_unlock(&drm->dmem->mutex);
 }
 
+/*
+ * Evict all pages mapping a chunk.
+ */
+static void
+nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk)
+{
+	unsigned long i, npages = range_len(&chunk->pagemap.range) >> PAGE_SHIFT;
+	unsigned long *src_pfns, *dst_pfns;
+	dma_addr_t *dma_addrs;
+	struct nouveau_fence *fence;
+
+	src_pfns = kcalloc(npages, sizeof(*src_pfns), GFP_KERNEL);
+	dst_pfns = kcalloc(npages, sizeof(*dst_pfns), GFP_KERNEL);
+	dma_addrs = kcalloc(npages, sizeof(*dma_addrs), GFP_KERNEL);
+
+	migrate_device_range(src_pfns, chunk->pagemap.range.start >> PAGE_SHIFT,
+			npages);
+
+	for (i = 0; i < npages; i++) {
+		if (src_pfns[i] & MIGRATE_PFN_MIGRATE) {
+			struct page *dpage;
+
+			/*
+			 * _GFP_NOFAIL because the GPU is going away and there
+			 * is nothing sensible we can do if we can't copy the
+			 * data back.
+			 */
+			dpage = alloc_page(GFP_HIGHUSER | __GFP_NOFAIL);
+			dst_pfns[i] = migrate_pfn(page_to_pfn(dpage));
+			nouveau_dmem_copy_one(chunk->drm,
+					migrate_pfn_to_page(src_pfns[i]), dpage,
+					&dma_addrs[i]);
+		}
+	}
+
+	nouveau_fence_new(chunk->drm->dmem->migrate.chan, false, &fence);
+	migrate_device_pages(src_pfns, dst_pfns, npages);
+	nouveau_dmem_fence_done(&fence);
+	migrate_device_finalize(src_pfns, dst_pfns, npages);
+	kfree(src_pfns);
+	kfree(dst_pfns);
+	for (i = 0; i < npages; i++)
+		dma_unmap_page(chunk->drm->dev->dev, dma_addrs[i], PAGE_SIZE, DMA_BIDIRECTIONAL);
+	kfree(dma_addrs);
+}
+
 void
 nouveau_dmem_fini(struct nouveau_drm *drm)
 {
@@ -378,8 +424,10 @@ nouveau_dmem_fini(struct nouveau_drm *drm)
 	mutex_lock(&drm->dmem->mutex);
 
 	list_for_each_entry_safe(chunk, tmp, &drm->dmem->chunks, list) {
+		nouveau_dmem_evict_chunk(chunk);
 		nouveau_bo_unpin(chunk->bo);
 		nouveau_bo_ref(NULL, &chunk->bo);
+		WARN_ON(chunk->callocated);
 		list_del(&chunk->list);
 		memunmap_pages(&chunk->pagemap);
 		release_mem_region(chunk->pagemap.range.start,
-- 
git-series 0.9.1