Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
References: <cover.f15b25597fc3afd45b144df863eeca3b2c13f9f4.1664171943.git-series.apopple@nvidia.com>
 <ea208905d853a0fdc277c2b5e74742593e53f767.1664171943.git-series.apopple@nvidia.com>
 <d839ead12d782a184ca104d6b5f62184c0f178dd.camel@redhat.com>
User-agent: mu4e 1.6.9; emacs 27.1
From:   Alistair Popple <apopple@nvidia.com>
To:     Lyude Paul <lyude@redhat.com>
Cc:     linux-mm@kvack.org, Andrew Morton <akpm@linux-foundation.org>,
        Michael Ellerman <mpe@ellerman.id.au>,
        Nicholas Piggin <npiggin@gmail.com>,
        Felix Kuehling <Felix.Kuehling@amd.com>,
        Alex Deucher <alexander.deucher@amd.com>,
        Christian =?utf-8?Q?K=C3=B6nig?= <christian.koenig@amd.com>,
        "Pan, Xinhui" <Xinhui.Pan@amd.com>,
        David Airlie <airlied@linux.ie>,
        Daniel Vetter <daniel@ffwll.ch>,
        Ben Skeggs <bskeggs@redhat.com>,
        Karol Herbst <kherbst@redhat.com>,
        Ralph Campbell <rcampbell@nvidia.com>,
        "Matthew Wilcox (Oracle)" <willy@infradead.org>,
        Alex Sierra <alex.sierra@amd.com>,
        John Hubbard <jhubbard@nvidia.com>,
        linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org,
        amd-gfx@lists.freedesktop.org, nouveau@lists.freedesktop.org,
        dri-devel@lists.freedesktop.org, Jason Gunthorpe <jgg@nvidia.com>,
        Dan Williams <dan.j.williams@intel.com>
Subject: Re: [PATCH 5/7] nouveau/dmem: Refactor nouveau_dmem_fault_copy_one()
Date:   Wed, 28 Sep 2022 21:30:18 +1000
In-reply-to: <d839ead12d782a184ca104d6b5f62184c0f178dd.camel@redhat.com>
Message-ID: <87a66jpweq.fsf@nvdebian.thelocal>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?NIc0Poe3/aWkjXaUJSxUZAbRLdTB5s+3kNF7FiSP2KI/cWrbdRf2ZT6dVPhd?=
 =?us-ascii?Q?Lb+5ZScGZqf20xbWIM7q2bzn7lkSugNMLw32S06e7wsaZTOSf3Jvzl4+/Iai?=
 =?us-ascii?Q?22uDJigA0iova9qInRvC4P2rCa82hgjxGz1J/gDzcJSLJ6lkfA5193JpI7Bf?=
 =?us-ascii?Q?xT03bvJLrtPbUYdwDUTO6qlrV+GgzP/2TXiA81+E2OYjsnhS5CPZHviKdBZb?=
 =?us-ascii?Q?mIv11qbMYfUri7Wx31Nm3R9mJJj1BfUR+B6RyNg7D08biZePffoOIFvtgRIY?=
 =?us-ascii?Q?mZLQAx+vNDHdHS5GO89J6BXIw7fbUNidiChIVc/DF9AgFxL6impqd5QFPCwn?=
 =?us-ascii?Q?Q1YHRtRgg6J6dAYKHhk/FW7Ntp+Gbk91D6ok+suk8bZ6qTE59v9TURDKTbis?=
 =?us-ascii?Q?LXtR/c9uYUHq1GZes7SaevdOM4dBhPP1SorhWx/mryLgIov5xuvF0+LThkw8?=
 =?us-ascii?Q?jwvBQzMIPC+bBZJ6dZMEA7fv2OVjKrjVcWaw6Hv6RbOEtxUZrXLxj0uKkRq3?=
 =?us-ascii?Q?4APZzOpUT8A77hqH9fcMjQvpETdVurZVOkyShMs93zj3z0SNzQX9Kx+mCdeO?=
 =?us-ascii?Q?okeTR+NAA03lAYgZ8lMZa+1g8BhkTL8ZElw0QKd7k2d11nGazWEYtYOOwmcr?=
 =?us-ascii?Q?Hlfl34Gtrm8Pt8o1F4cVio3+po9DlfZDSaIVXvmBs+HtzHDhooSfgLMWBM7V?=
 =?us-ascii?Q?Edmi8MTuYL+YGTP1KadOsXAut23aU0iFWtlbBZgDCvC9bYs3givn+tJEuuWr?=
 =?us-ascii?Q?UqQCEUDMKKjRot467RvISlML9rgCjiDL17NyUgBESgPbAFmDKRnEsw46dcFK?=
 =?us-ascii?Q?Lop6gk0arQfgmanVL65N+P8dI8Cd4N52qa/qqzQmflPbuOjpgr1ASsJUoe3k?=
 =?us-ascii?Q?pMaM2b2j1ybCIN+xQgC4UtSpKQtm4zRUvvh9A+EDnknn+JZam9ZdJVJaoVyc?=
 =?us-ascii?Q?1eXt6j9/dymYLJ/CtmInL+P/O7+ejeZPUDG1AqwXtchp/Vi86zJMuWneyuZS?=
 =?us-ascii?Q?bAjjdaPd5ehNjyvGF6dU1hKT5Uk6so4iy0Kny31doXjWOpsAJechdGIC6WsO?=
 =?us-ascii?Q?QjRWGavYqOM6QplEJUwqoitqANHy6q1HfY/rV3IlvvOgMRR//Xqwjqb5wuiI?=
 =?us-ascii?Q?Pyp5ebGln2nDNhwNc+NgQ9W5zC/F7x1+yyjMZt8G5wp6Vr7uYmgAG2VWs/Rk?=
 =?us-ascii?Q?EaTiYAvcntZeDs7d7X6yyhp3hfy+7J+MxL+MlXm7W/jjcq88si2P2Wu9UKnk?=
 =?us-ascii?Q?OTaGoJsLgcr+90avC6kp6l+sX0Mx2N9Ss7rIKJNKBZElWZtqDuJ0+bpbmQzR?=
 =?us-ascii?Q?yGEh2o+N2yfjz21kEwYNFCfu7VVMdP4cyL+Kna10SKI4Tg8qZNOMdedJJ65v?=
 =?us-ascii?Q?3WSjz0FHsra2+L+jZymqs7+7NwHQiTRYxaxmaIQYHIw896DfWURgjsV2jlDW?=
 =?us-ascii?Q?EhRIUvkyAhOGwOGEY+rNd78mvjYDdEG46jT0ijTS3B10Rjj3sLRgBwNElWlR?=
 =?us-ascii?Q?gNpP83f0QnjexGCbvcvw6RvNQwx137EDUz5k0pXZF3iKyWP6jEuXGwKYg7JQ?=
 =?us-ascii?Q?MY7Bx62eEu9TksSKHh0eGW4Gpx9lBrRcG9Wna0cB?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 78eef31f-0652-49bf-9ee5-08daa1451157
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2022 11:32:04.6647
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: NyO3P1hBMXZujEc4N5yYHFkfYuzOE9I0U2QkUZ3yN0afLoYqoa89V1KCqCC9Kz8p0nhFBK9n9Pmp+ilO64M/kQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6469
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


Lyude Paul <lyude@redhat.com> writes:

> On Mon, 2022-09-26 at 16:03 +1000, Alistair Popple wrote:
>> nouveau_dmem_fault_copy_one() is used during handling of CPU faults via
>> the migrate_to_ram() callback and is used to copy data from GPU to CPU
>> memory. It is currently specific to fault handling, however a future
>> patch implementing eviction of data during teardown needs similar
>> functionality.
>>
>> Refactor out the core functionality so that it is not specific to fault
>> handling.
>>
>> Signed-off-by: Alistair Popple <apopple@nvidia.com>
>> ---
>>  drivers/gpu/drm/nouveau/nouveau_dmem.c | 59 +++++++++++++--------------
>>  1 file changed, 29 insertions(+), 30 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouveau/nouveau_dmem.c
>> index f9234ed..66ebbd4 100644
>> --- a/drivers/gpu/drm/nouveau/nouveau_dmem.c
>> +++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c
>> @@ -139,44 +139,25 @@ static void nouveau_dmem_fence_done(struct nouveau_fence **fence)
>>  	}
>>  }
>>
>> -static vm_fault_t nouveau_dmem_fault_copy_one(struct nouveau_drm *drm,
>> -		struct vm_fault *vmf, struct migrate_vma *args,
>> -		dma_addr_t *dma_addr)
>> +static int nouveau_dmem_copy_one(struct nouveau_drm *drm, struct page *spage,
>> +				struct page *dpage, dma_addr_t *dma_addr)
>>  {
>>  	struct device *dev = drm->dev->dev;
>> -	struct page *dpage, *spage;
>> -	struct nouveau_svmm *svmm;
>> -
>> -	spage = migrate_pfn_to_page(args->src[0]);
>> -	if (!spage || !(args->src[0] & MIGRATE_PFN_MIGRATE))
>> -		return 0;
>>
>> -	dpage = alloc_page_vma(GFP_HIGHUSER, vmf->vma, vmf->address);
>> -	if (!dpage)
>> -		return VM_FAULT_SIGBUS;
>>  	lock_page(dpage);
>>
>>  	*dma_addr = dma_map_page(dev, dpage, 0, PAGE_SIZE, DMA_BIDIRECTIONAL);
>>  	if (dma_mapping_error(dev, *dma_addr))
>> -		goto error_free_page;
>> +		return -EIO;
>>
>> -	svmm = spage->zone_device_data;
>> -	mutex_lock(&svmm->mutex);
>> -	nouveau_svmm_invalidate(svmm, args->start, args->end);
>>  	if (drm->dmem->migrate.copy_func(drm, 1, NOUVEAU_APER_HOST, *dma_addr,
>> -			NOUVEAU_APER_VRAM, nouveau_dmem_page_addr(spage)))
>> -		goto error_dma_unmap;
>> -	mutex_unlock(&svmm->mutex);
>> +					 NOUVEAU_APER_VRAM,
>> +					 nouveau_dmem_page_addr(spage))) {
>> +		dma_unmap_page(dev, *dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL);
>> +		return -EIO;
>> +	}
>
> Feel free to just align this with the starting (, as long as it doesn't go
> above 100 characters it doesn't really matter imho and would look nicer that
> way.
>
> Otherwise:
>
> Reviewed-by: Lyude Paul <lyude@redhat.com>

Thanks! I'm not sure I precisely understood your alignment comment above
but feel free to let me know if I got it wrong in v2.

> Will look at the other patch in a moment
>
>>
>> -	args->dst[0] = migrate_pfn(page_to_pfn(dpage));
>>  	return 0;
>> -
>> -error_dma_unmap:
>> -	mutex_unlock(&svmm->mutex);
>> -	dma_unmap_page(dev, *dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL);
>> -error_free_page:
>> -	__free_page(dpage);
>> -	return VM_FAULT_SIGBUS;
>>  }
>>
>>  static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
>> @@ -184,9 +165,11 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
>>  	struct nouveau_drm *drm = page_to_drm(vmf->page);
>>  	struct nouveau_dmem *dmem = drm->dmem;
>>  	struct nouveau_fence *fence;
>> +	struct nouveau_svmm *svmm;
>> +	struct page *spage, *dpage;
>>  	unsigned long src = 0, dst = 0;
>>  	dma_addr_t dma_addr = 0;
>> -	vm_fault_t ret;
>> +	vm_fault_t ret = 0;
>>  	struct migrate_vma args = {
>>  		.vma		= vmf->vma,
>>  		.start		= vmf->address,
>> @@ -207,9 +190,25 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
>>  	if (!args.cpages)
>>  		return 0;
>>
>> -	ret = nouveau_dmem_fault_copy_one(drm, vmf, &args, &dma_addr);
>> -	if (ret || dst == 0)
>> +	spage = migrate_pfn_to_page(src);
>> +	if (!spage || !(src & MIGRATE_PFN_MIGRATE))
>> +		goto done;
>> +
>> +	dpage = alloc_page_vma(GFP_HIGHUSER, vmf->vma, vmf->address);
>> +	if (!dpage)
>> +		goto done;
>> +
>> +	dst = migrate_pfn(page_to_pfn(dpage));
>> +
>> +	svmm = spage->zone_device_data;
>> +	mutex_lock(&svmm->mutex);
>> +	nouveau_svmm_invalidate(svmm, args.start, args.end);
>> +	ret = nouveau_dmem_copy_one(drm, spage, dpage, &dma_addr);
>> +	mutex_unlock(&svmm->mutex);
>> +	if (ret) {
>> +		ret = VM_FAULT_SIGBUS;
>>  		goto done;
>> +	}
>>
>>  	nouveau_fence_new(dmem->migrate.chan, false, &fence);
>>  	migrate_vma_pages(&args);