Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Alistair Popple <apopple@nvidia.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Felix Kuehling <felix.kuehling@amd.com>,
        Christoph Hellwig <hch@lst.de>,
        David Hildenbrand <david@redhat.com>,
        Alex Sierra <alex.sierra@amd.com>, akpm@linux-foundation.org,
        linux-mm@kvack.org, rcampbell@nvidia.com,
        linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org,
        amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
        jglisse@redhat.com, willy@infradead.org
Subject: Re: [PATCH v6 01/10] mm: add zone device coherent type memory support
Date:   Wed, 16 Feb 2022 12:23:44 +1100
References: <20220201154901.7921-1-alex.sierra@amd.com>
 <20220201154901.7921-2-alex.sierra@amd.com>
 <beb38138-2266-1ff8-cc82-8fe914bed862@redhat.com>
 <f2af73c1-396b-168f-7f86-eb10b3b68a26@redhat.com>
 <a24d82d9-daf9-fa1a-8b9d-5db7fe10655e@amd.com>
 <078dd84e-ebbc-5c89-0407-f5ecc2ca3ebf@redhat.com>
 <20220215144524.GR4160@nvidia.com> <20220215183209.GA24409@lst.de>
 <20220215194107.GZ4160@nvidia.com>
 <ac3d5157-9251-f9fb-a973-f268ce58b4e0@amd.com>
 <20220215214749.GA4160@nvidia.com>
User-agent: mu4e 1.6.9; emacs 27.1
In-reply-to: <20220215214749.GA4160@nvidia.com>
Message-ID: <877d9vd10u.fsf@nvdebian.thelocal>
Content-Type: multipart/mixed; boundary="=-=-="
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?aufCifvl2+1r8IBD8fm9vthpTUWN3JCDT20Cc/U1pdmwDSnNpe4r25Jzfl2G?=
 =?us-ascii?Q?Gw7U7tPRDDNWbvCEbg71wTbufG3OdONTWkYMcS2xlca6IjTIkxPoxtHpmKja?=
 =?us-ascii?Q?l3n2qPfSF6R0hBgfWL4aNEHY5Juos0JkuT34T79jTatHTgwjVLBvwiN+wgMH?=
 =?us-ascii?Q?HOJPuT+n8tNJHk9zfD+PThjAflGw3163Xv6bNOCRZNiJB/zPYk0DQzINoQX6?=
 =?us-ascii?Q?Daf+C+EbSGQsgvxyfy2uQV2uGz0D6dBYOM0X2RM1QXlT2pqGfxXxbTIjk3ZT?=
 =?us-ascii?Q?q7yeUGgBTaFu3UVKLAK/pCPTmX4eUl6p6lfsGlzRY03+wxmk7s/0VXoeqA82?=
 =?us-ascii?Q?CCJaIdJlFy6t2Z7La1lRIOnlbbH+hAB0KM+UBeS0Nvrj15+c1F7p0pn04Lzd?=
 =?us-ascii?Q?brhFC+yTSMrLmbYLoShZaQHAkPEJurA2Ujvx1W4B+TLCY62DBTwIfWjatrJ+?=
 =?us-ascii?Q?V6EUkVwZQJYL4UmE/IekNtnPrXafXsgjo5q5Gm9E1bVxqzk7oYyql1atRcIQ?=
 =?us-ascii?Q?GEFVmC4dgRSIwzmd3e2oR8L4GAVB/+3lLh6rQlelPgnMAsYgMA0PHjdQUvXg?=
 =?us-ascii?Q?U2uCot1xWvUOKC/inu2FAQjS+tdCczXPLSmGNb68PYdXaS26g+iN/2x6niwC?=
 =?us-ascii?Q?AciCBXQ5HEtgm955PZ+QYFKyOUgodyZS8FM1qBLNnpnyCmhJSJ91h8MEMcwC?=
 =?us-ascii?Q?Zn3WilXdnTUMKvilGQJ54mkfShdPNw2F3UoIlNmTWk6nk5wHewaN9ofn1o1Y?=
 =?us-ascii?Q?2pg1DaTIEIXVWN9AqOBXfnCGxjDbHUD0BIQQqmg/20i/gJhmPEagcTo71kSH?=
 =?us-ascii?Q?4yw32yvs5iFxN0rm/X/NgcvregdIluOkvMbT2OT+7lM7sFg9KBG0tsRH2FQJ?=
 =?us-ascii?Q?oqMg+9ePZ/0W8mA1ExzDgErYTRW5yjdDs91N/+bid2W067/+YFij+qnTDVI3?=
 =?us-ascii?Q?UyptFz48Byzz8aQG08/LJXFHiWI/Xpd9AwX0HJd2mUR4+DsS28JItFO6X+TY?=
 =?us-ascii?Q?k3F7yJqem0A5aiVl0jFtpoXh1iOxddqu1yC6pQ5xZH+76xxwyiaMMyxc8e+w?=
 =?us-ascii?Q?rIitmjfJTDy2vRpS4mUsu/fBguZcsdeF5r2YiLQgSvjftXmUc8SzjxxdwIGI?=
 =?us-ascii?Q?PGUuAEgpmwYvOgLEiW3i28LDi5Aq1dCYsV6Y94SV7TziSph37TzQShO8D1ZR?=
 =?us-ascii?Q?0Eb2HB1qSyTayhoK91rem+uWWvCd7bqVn9ufoMiytr1IVpp/vWOH4GMSza0j?=
 =?us-ascii?Q?iCxMbvEtL+ndWQoKTIB00+kDp1mHxjfhKF/liApheW+o/4tHbBai5Pa7alN2?=
 =?us-ascii?Q?iKDrcYl9VQZqkyLPCy4xOjS6lS2Ira8C3FKos3F6UtXKU0pQPgZce62t1D6/?=
 =?us-ascii?Q?0WarzuJiLeZmLEkBTZLRaaAu2uaaZVyeNxL6d+POjAUmyY0BontaXB7LavEE?=
 =?us-ascii?Q?b3XM58ecDtpaWVEx09fUNrdVjx5hpUvWTY33xyqAXGvDgEawtC3jRxMPKR9f?=
 =?us-ascii?Q?E+iPU9fG8z+VEe0EbzrYU2iKmTryrKPkzqq26nN9IBI/5gkX3PBl7mTynKbo?=
 =?us-ascii?Q?QPOfm1NNwBJcq7o4CVj5wDW93wazj3hecZgNtxD9olwKweYICwEyNvekVgv+?=
 =?us-ascii?Q?RxxqEbE7AdPioTqKTVmFYnc=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3b735cc9-17a6-47ae-d826-08d9f0f002d1
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2022 01:59:48.1947
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: XPGkitRuRn1JGng18z90KblluL4BsROevpxI8jcs+OQMjwFRkfIwIYJy1M8R2Ng50h2C7WS/KUJMHrCqZMSr4A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3268
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

--=-=-=
Content-Type: text/plain
Content-Disposition: inline

Jason Gunthorpe <jgg@nvidia.com> writes:

> On Tue, Feb 15, 2022 at 04:35:56PM -0500, Felix Kuehling wrote:
>>
>> On 2022-02-15 14:41, Jason Gunthorpe wrote:
>> > On Tue, Feb 15, 2022 at 07:32:09PM +0100, Christoph Hellwig wrote:
>> > > On Tue, Feb 15, 2022 at 10:45:24AM -0400, Jason Gunthorpe wrote:
>> > > > > Do you know if DEVICE_GENERIC pages would end up as PageAnon()? My
>> > > > > assumption was that they would be part of a special mapping.
>> > > > We need to stop using the special PTEs and VMAs for things that have a
>> > > > struct page. This is a mistake DAX created that must be undone.
>> > > Yes, we'll get to it.  Maybe we can do it for the non-DAX devmap
>> > > ptes first given that DAX is more complicated.
>> > Probably, I think we can check the page->pgmap type to tell the
>> > difference.
>> >
>> > I'm not sure how the DEVICE_GENERIC can work without this, as DAX was
>> > made safe by using the unmap_mapping_range(), which won't work
>> > here. Is there some other trick being used to keep track of references
>> > inside the AMD driver?
>>
>> Not sure I'm following all the discussion about VMAs and DAX. So I may be
>> answering the wrong question: We treat each ZONE_DEVICE page as a reference
>> to the BO (buffer object) that backs the page. We increment the BO refcount
>> for each page we migrate into it. In the dev_pagemap_ops.page_free callback
>> we drop that reference. Once all pages backed by a BO are freed, the BO
>> refcount reaches 0 [*] and we can free the BO allocation.
>
> Userspace does
>  1) mmap(MAP_PRIVATE) to allocate anon memory
>  2) something to trigger migration to install a ZONE_DEVICE page
>  3) munmap()
>
> Who decrements the refcout on the munmap?
>
> When a ZONE_DEVICE page is installed in the PTE is supposed to be
> marked as pte_devmap and that disables all the normal page refcounting
> during munmap().

Device private and device coherent pages are not marked with pte_devmap and they
are backed by a struct page. The only way of inserting them is via migrate_vma.
The refcount is decremented in zap_pte_range() on munmap() with special handling
for device private pages. Looking at it again though I wonder if there is any
special treatment required in zap_pte_range() for device coherent pages given
they count as present pages.

> fsdax makes this work by working the refcounts backwards, the page is
> refcounted while it exists in the driver, when the driver decides to
> remove it then unmap_mapping_range() is called to purge it from all
> PTEs and then refcount is decrd. munmap/fork/etc don't change the
> refcount.

The equivalent here is for drivers to use migrate_vma to migrate the pages back
from device memory to CPU memory. In this case the refcounting is (mostly)
handled by migration code which decrements the refcount on the original source
device page during the migration.

- Alistair

> Jason

--=-=-=--