Received-SPF: pass (google.com: domain of linux-ext4+bounces-2054-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
References: <cover.fe275e9819458a4bbb9451b888cafb88af8867d4.1712796818.git-series.apopple@nvidia.com>
 <a443974e64917824e078485d4e755ef04c89d73f.1712796818.git-series.apopple@nvidia.com>
 <20240411122927.GR5383@nvidia.com>
User-agent: mu4e 1.10.8; emacs 29.1
From: Alistair Popple <apopple@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: linux-mm@kvack.org, david@fromorbit.com, dan.j.williams@intel.com,
 jhubbard@nvidia.com, rcampbell@nvidia.com, willy@infradead.org,
 linux-fsdevel@vger.kernel.org, jack@suse.cz, djwong@kernel.org,
 hch@lst.de, david@redhat.com, ruansy.fnst@fujitsu.com,
 nvdimm@lists.linux.dev, linux-xfs@vger.kernel.org,
 linux-ext4@vger.kernel.org, jglisse@redhat.com
Subject: Re: [RFC 03/10] pci/p2pdma: Don't initialise page refcount to one
Date: Fri, 12 Apr 2024 15:40:47 +1000
In-reply-to: <20240411122927.GR5383@nvidia.com>
Message-ID: <87bk6f5dwz.fsf@nvdebian.thelocal>
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?mFWiZ24zrOKbBifa3PzVXdiDQX1j+1flEcLKityiyBFJQbq5yzj1Ln4b00uK?=
 =?us-ascii?Q?dkXykBK5yyqnYDQBt563tRB/PdlFnqkcOmIthGmTI7z8cWfZ0LIkrA6uoFUW?=
 =?us-ascii?Q?N1ONhd0713vJhAPYXe4WS4Qj4vGnbx0ruYs+4VXHTRcXt6+bG85diQ4cUhxI?=
 =?us-ascii?Q?fwBkJHnkJoG76Iy8+dwOieiFiXnDFUvKm0FW/OpSBhNTAO6OM1XIHmyL0BPb?=
 =?us-ascii?Q?WoS1dNw+sCGWeIqT3Ap7LspXdcBfvTOmhPukzN0caU9ZDZys0RdSJ2g+7QsY?=
 =?us-ascii?Q?NdIU/tCt6OVs7ZLlNDurBujfLxaklQF6tPzibZOGLvTVF4Awmin2Nl6gKz4v?=
 =?us-ascii?Q?KH4jBsrGMK3Aktqq+OCP7H7N7n8Tn1Z40czBqZhhgZE+6Bbjy6zCQuXf2GmL?=
 =?us-ascii?Q?lLm8gahAY5ZlarSiMQ8O076HX3RjoqYaVCLp+QQe021bG77T3ZpJF1XrLuiE?=
 =?us-ascii?Q?ApWfFROAcYcKxhonrGRgXtorz7BQpAM7AZu/ucpkEy1ZNupd093pLPwDFMb1?=
 =?us-ascii?Q?pmjENx9DpNuAD6asFyozlnDvwdTIczYnSB6/eJGaGyRqkjNpvGVlXmSjuKF9?=
 =?us-ascii?Q?mWtNNXF79t8JtUVIqQ9gVmIVpU39fE5ojCSekau1bQU59Oi71Fj4nR+4MLRP?=
 =?us-ascii?Q?djdyKDqtH7sdX5WfoIw6RGt6pW7J107GdoG9h7aH1br6JrCgDmS11ovn+vtK?=
 =?us-ascii?Q?lRYzoNeGmTpHyImCdj8Oflcs6+mb0IgBhcL5stpayHNt1mtUmlxLjYXKOgJu?=
 =?us-ascii?Q?rojPuAaDSWU2VqaD4WX979eh/KuU7kxhqCrtSSvoQ7mHZqmf8lQJwwe4E1FP?=
 =?us-ascii?Q?jW+gUaBvlKFJBZlbdd1ozrZdvqCzCjrIO3k9VI5liNIEgtkd38uMCWbR3IrY?=
 =?us-ascii?Q?4WUpp2OMj1EZvmOLowcsgtcG7VsPe6pL/EqLNL/AgeELBJ8cKuJknl+moRCc?=
 =?us-ascii?Q?5/MOgtXFC1QeXohObXXqNtGcU9TnZKhIi0ybR4X+9eHcT7osKGEmdpMuneq5?=
 =?us-ascii?Q?l7mY3WL2z6ZIH9P2uLKmBLyxwzDHsTay7XicyCuTf8TMy8gCNDGFSP1E5zqj?=
 =?us-ascii?Q?Imls32dKH2kwnYo7DFw0oaprFdzhiBtQa08yLbSA7rm7Wg4kVxmB8vWWen3f?=
 =?us-ascii?Q?hgRoNWOq2gvjn8Wf0Rg8WcRFengXqPACvmaw7tY26KDEsczztHwL0b65FIvs?=
 =?us-ascii?Q?HbPeuSG1qBRmi9jO0RLHxfRcB7cVGlv7WgH3miAwHvPSYoVf5LyrJzDbISha?=
 =?us-ascii?Q?GwkS+aKyKVQ6JuwrQXTFoxTlGKzo8zYXYruZ9PAw581yG1pHaLwrWb9BFx9p?=
 =?us-ascii?Q?1elQPoscWsTrV8/HUS3ERUqq41XUcVm48ZKuV6UzzqFe76e7ApMLN2CRLtD0?=
 =?us-ascii?Q?YUdLV/1YM+NANCSEdyWuj/UCSYYIPVvEUjoNRopMrJQGHG+WKKteNqjtNBB4?=
 =?us-ascii?Q?anVJ2DCVaUcXXY8fzIMeFNRu1A+QyOnpdY02Jhlw5q9q1d+CPhhiv9/BM68s?=
 =?us-ascii?Q?pOcFae3PlqX3UIccWYFLMcttJDE/5V8/4Z5d+eK3skEKVCR2a81HPfqFGxy4?=
 =?us-ascii?Q?k9uDfdosk7Ei5JfIEcfu9NzS2KzcUX/L5HKCpOgO?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4c8d20f4-0595-4c62-9774-08dc5ab443ce
X-MS-Exchange-CrossTenant-AuthSource: CY8PR12MB7705.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Apr 2024 05:49:05.9507
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: r90Wb9JB7+6ntVHpm/lXW0SYn6KbDGLr70g4YPWw58R/RIzo/ONtBrJr6PKicLiaUPpZDk80n5eTifdlEh/mwQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8694


Jason Gunthorpe <jgg@nvidia.com> writes:

> On Thu, Apr 11, 2024 at 10:57:24AM +1000, Alistair Popple wrote:
>> The reference counts for ZONE_DEVICE private pages should be
>> initialised by the driver when the page is actually allocated by the
>> driver allocator, not when they are first created. This is currently
>> the case for MEMORY_DEVICE_PRIVATE and MEMORY_DEVICE_COHERENT pages
>> but not MEMORY_DEVICE_PCI_P2PDMA pages so fix that up.
>> 
>> Signed-off-by: Alistair Popple <apopple@nvidia.com>
>> ---
>>  drivers/pci/p2pdma.c | 2 ++
>>  mm/memremap.c        | 8 ++++----
>>  mm/mm_init.c         | 4 +++-
>>  3 files changed, 9 insertions(+), 5 deletions(-)
>> 
>> diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
>> index fa7370f..ab7ef18 100644
>> --- a/drivers/pci/p2pdma.c
>> +++ b/drivers/pci/p2pdma.c
>> @@ -128,6 +128,8 @@ static int p2pmem_alloc_mmap(struct file *filp, struct kobject *kobj,
>>  		goto out;
>>  	}
>>  
>> +	get_page(virt_to_page(kaddr));
>> +
>
> Should this be 
>
>  set_page_count(page, 1)
>
> If the refcount is already known to be 0 ?

Yeah, that would avoid the obvious warning that calling get_page there
will generate. My test setup for p2pdma is pretty clunky, so haven't run
it a while. Not sure if there are any good qemu based tests for this.

>> @@ -508,15 +508,15 @@ void free_zone_device_page(struct page *page)
>>  	page->mapping = NULL;
>>  	page->pgmap->ops->page_free(page);
>>  
>> -	if (page->pgmap->type != MEMORY_DEVICE_PRIVATE &&
>> -	    page->pgmap->type != MEMORY_DEVICE_COHERENT)
>> +	if (page->pgmap->type == MEMORY_DEVICE_PRIVATE ||
>> +	    page->pgmap->type == MEMORY_DEVICE_COHERENT)
>> +		put_dev_pagemap(page->pgmap);
>
> Not related, but we should really be getting rid of this devmap
> refcount traffic too, IMHO..

Absolutely. I think there's a bunch of clean ups for this in mm/gup.c
that could be done as well. I plan on doing that as a follow up to this
series. We pretty much don't use that for device private/coherent pages
anyway.

> If an implementation wants this then it should hook the page
> free/alloc callbacks and do this, not put it in the core code.
>
> Jason