Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Date:   Tue, 15 Feb 2022 17:47:49 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Felix Kuehling <felix.kuehling@amd.com>
Cc:     Christoph Hellwig <hch@lst.de>,
        David Hildenbrand <david@redhat.com>,
        Alex Sierra <alex.sierra@amd.com>, akpm@linux-foundation.org,
        linux-mm@kvack.org, rcampbell@nvidia.com,
        linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org,
        amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
        jglisse@redhat.com, apopple@nvidia.com, willy@infradead.org
Subject: Re: [PATCH v6 01/10] mm: add zone device coherent type memory support
Message-ID: <20220215214749.GA4160@nvidia.com>
References: <20220201154901.7921-1-alex.sierra@amd.com>
 <20220201154901.7921-2-alex.sierra@amd.com>
 <beb38138-2266-1ff8-cc82-8fe914bed862@redhat.com>
 <f2af73c1-396b-168f-7f86-eb10b3b68a26@redhat.com>
 <a24d82d9-daf9-fa1a-8b9d-5db7fe10655e@amd.com>
 <078dd84e-ebbc-5c89-0407-f5ecc2ca3ebf@redhat.com>
 <20220215144524.GR4160@nvidia.com>
 <20220215183209.GA24409@lst.de>
 <20220215194107.GZ4160@nvidia.com>
 <ac3d5157-9251-f9fb-a973-f268ce58b4e0@amd.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ac3d5157-9251-f9fb-a973-f268ce58b4e0@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?rjpNj1cq1+6ZHekg0bJvkKUDj7Nq3SUkHwbV+z/Xnry+C775vbj+vGQ16HKU?=
 =?us-ascii?Q?2FcF97X6HPqRxTlhgpVi8LVaUgz4mYB86zPMLrcGAQ4E0Gn2xttdkr4LaUvj?=
 =?us-ascii?Q?F19MSde4Si6KeRHJ+CN6v5JX+0piNtN8yhLExJ72RoqS6aL+G7qFHMplNmm0?=
 =?us-ascii?Q?6eAhWvFiLLZDGy0bYySaL7V3urC1OCk8u92rmPgXeZThu96CvFS0xPtFAlCI?=
 =?us-ascii?Q?zEkCupY7wmY3Y4wGaXlXiM8q3gg7JXKa3+7bkybx7oAcvbnd5CegiRUfEt+Y?=
 =?us-ascii?Q?LfQe4dxo6PuYEm1GxOv4NeOPG7FLobz57nW/THbr4ykc74oVqek23w/n/9Ye?=
 =?us-ascii?Q?9Wkwdkt5Ne87TCxLOL62YwFkACUEbNyE8z+gNYp+7OusM7f2EgK7QByqJlZg?=
 =?us-ascii?Q?Hc7hYU/FDn92szsb/lKcGgVC2lLHLZ8ttJyQfLcXXr/09+Jc3yYTB0PQptez?=
 =?us-ascii?Q?YOj78uu2O/wekvUP9UgTkLuvT65UnkGne1PIXsBHR/2geNg0P80QvFMij29I?=
 =?us-ascii?Q?mBOIKfxnKTntuiIOi0+RuVfOZajrtmaBAh4qkHrNOvbkwyIT3xj3rvO4vTsV?=
 =?us-ascii?Q?yxWAKiXZQjrBoEdt/0NgC+3/oSk2UoQGyl2usESjDh6yrbMFMQb2iW5wcv2h?=
 =?us-ascii?Q?V3cBP/1jpiycp3saN7MAoIj2ClQEDjnpk8nXr9t1ydzZ28XmAaMNPn1NUC8O?=
 =?us-ascii?Q?qTb4hZdGpzyix6Z/REtZLMMYm4ihhT/xH29iww6Fqu7VEBAJVhvVHKBqOx8Y?=
 =?us-ascii?Q?W9CeNZiCdaYUpfyDwpmpfCwRLJH756ODOI9Zd46BY+U5FKLWHBRE0ePuDZyl?=
 =?us-ascii?Q?E958EV+VznWzK0VrMWnlZufnYLrDwtbjp+oDHXld1Yvh6g4koJiqWBB+3k4d?=
 =?us-ascii?Q?a6Kmeytv1idpZZcuCymeJr/MNvCY13h2SiL5en1S0KyuAoZ413YPHYWiRkZ+?=
 =?us-ascii?Q?nJbmWVq+s7zpos/Jc0CLLCtheQXr7YmXjFwr7mCpY0hsWdxy4ZEhT+yk3269?=
 =?us-ascii?Q?RpfyUnyOlDofLO4OSmaW4zqvhB0bqcXsiYeKxxvmJw2T27qOEZ3T/4HcbLMo?=
 =?us-ascii?Q?edQRqIeB1xdh18KFS/XyXaPNr93WJpYHxB1BTupL/RjertdCwRN7X8PYaKVE?=
 =?us-ascii?Q?ZWNi4AHl87UsZorUGJw35CVe7gtGuggcufo9DNfRDhX5oi1a7ntbz8IOIbfN?=
 =?us-ascii?Q?bjBzti5CEylZtXSlI1QwN2xBHqaWFC99Csp6dIc3ps9i76Ik25ze0XQ3fQ7g?=
 =?us-ascii?Q?K1QwZ3VCg46urj+jfIkI3OIc/fRHqkmqNKrUxbcvXvP8qz/2pjUdvjuOOtDz?=
 =?us-ascii?Q?PHzk4uG52ONp6aKW9HLEByOx3Mfibm2Qaqt7BdyVwzuyDBF0Jw0vHayKbTQL?=
 =?us-ascii?Q?T89SPicSz6M14SUjTMkW/J7jaHNOUVYAPaNevXiYPc5JoX2xNcyi1DttsjG2?=
 =?us-ascii?Q?56NQhO6xeD/EcVF9H3UgOwGnFjYbIad5dBVIgcqBLy2BejyaSMXaq/45ThQ3?=
 =?us-ascii?Q?xYD93KeJU/e0do352kxWPiSpyqXDufBYaeowbSSlKKXpVc+FpofXV3TmXD/k?=
 =?us-ascii?Q?70rNCF3oHpe1tIh8MHw=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: baa9f4d1-c3ba-475f-a94a-08d9f0ccd02b
X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4181.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2022 21:47:50.8351
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: IhoZ4gE7jeglok5iYA+44CwEXi2iabe4Qob6thHMAiA/XS7CjxBgKl0/qaM2oGIu
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5217
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

On Tue, Feb 15, 2022 at 04:35:56PM -0500, Felix Kuehling wrote:
> 
> On 2022-02-15 14:41, Jason Gunthorpe wrote:
> > On Tue, Feb 15, 2022 at 07:32:09PM +0100, Christoph Hellwig wrote:
> > > On Tue, Feb 15, 2022 at 10:45:24AM -0400, Jason Gunthorpe wrote:
> > > > > Do you know if DEVICE_GENERIC pages would end up as PageAnon()? My
> > > > > assumption was that they would be part of a special mapping.
> > > > We need to stop using the special PTEs and VMAs for things that have a
> > > > struct page. This is a mistake DAX created that must be undone.
> > > Yes, we'll get to it.  Maybe we can do it for the non-DAX devmap
> > > ptes first given that DAX is more complicated.
> > Probably, I think we can check the page->pgmap type to tell the
> > difference.
> > 
> > I'm not sure how the DEVICE_GENERIC can work without this, as DAX was
> > made safe by using the unmap_mapping_range(), which won't work
> > here. Is there some other trick being used to keep track of references
> > inside the AMD driver?
> 
> Not sure I'm following all the discussion about VMAs and DAX. So I may be
> answering the wrong question: We treat each ZONE_DEVICE page as a reference
> to the BO (buffer object) that backs the page. We increment the BO refcount
> for each page we migrate into it. In the dev_pagemap_ops.page_free callback
> we drop that reference. Once all pages backed by a BO are freed, the BO
> refcount reaches 0 [*] and we can free the BO allocation.

Userspace does
 1) mmap(MAP_PRIVATE) to allocate anon memory
 2) something to trigger migration to install a ZONE_DEVICE page
 3) munmap()

Who decrements the refcout on the munmap?

When a ZONE_DEVICE page is installed in the PTE is supposed to be
marked as pte_devmap and that disables all the normal page refcounting
during munmap().

fsdax makes this work by working the refcounts backwards, the page is
refcounted while it exists in the driver, when the driver decides to
remove it then unmap_mapping_range() is called to purge it from all
PTEs and then refcount is decrd. munmap/fork/etc don't change the
refcount.

Jason