Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <ac3d5157-9251-f9fb-a973-f268ce58b4e0@amd.com>
Date:   Tue, 15 Feb 2022 16:35:56 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Subject: Re: [PATCH v6 01/10] mm: add zone device coherent type memory support
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>, Christoph Hellwig <hch@lst.de>
Cc:     David Hildenbrand <david@redhat.com>,
        Alex Sierra <alex.sierra@amd.com>, akpm@linux-foundation.org,
        linux-mm@kvack.org, rcampbell@nvidia.com,
        linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org,
        amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
        jglisse@redhat.com, apopple@nvidia.com, willy@infradead.org
References: <20220201154901.7921-1-alex.sierra@amd.com>
 <20220201154901.7921-2-alex.sierra@amd.com>
 <beb38138-2266-1ff8-cc82-8fe914bed862@redhat.com>
 <f2af73c1-396b-168f-7f86-eb10b3b68a26@redhat.com>
 <a24d82d9-daf9-fa1a-8b9d-5db7fe10655e@amd.com>
 <078dd84e-ebbc-5c89-0407-f5ecc2ca3ebf@redhat.com>
 <20220215144524.GR4160@nvidia.com> <20220215183209.GA24409@lst.de>
 <20220215194107.GZ4160@nvidia.com>
From:   Felix Kuehling <felix.kuehling@amd.com>
Organization: AMD Inc.
In-Reply-To: <20220215194107.GZ4160@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RXc3b2ZCWEtrTG1uM3J1Q0ZhL2sxWjNsK1kyc3YyREd3K1ZhZlh1ZHd2RDVs?=
 =?utf-8?B?akZBU1ZLME50b3lySE80akpuNzVTNGNLWTFVUldmL20rZ2Y0Q2VWM1lpb2hm?=
 =?utf-8?B?T0JOalg3R1plNEM2WHdNTVFqc0FjbzVNODNwcHVDcjNDTjhlZC9TWmxRZVJm?=
 =?utf-8?B?UC9XMjhzUzE1bDlkbnhnTi9hYmlLejc1L1BzWlNodHJKS3REbFY1U05vdVRT?=
 =?utf-8?B?bVE5NXRQREJ0RmxVL3IrUEZiTkVFVytLWjhNaTBkNHh0aGhGSzIzRWMzZmE5?=
 =?utf-8?B?YXRaSG01aHF5d1VSRldNT0pKNEJhQXZkSmlkSlR1SXlMWUltQTQyOHJ3UUhC?=
 =?utf-8?B?RVFNZVlydUlrZzN3SDlRK0ErMmdDQTFlNHhzWTFWeE5QUlRnQ2pzalpweXR6?=
 =?utf-8?B?NWw3cVJidHRTMm1RSFBWdmlCRTU5L0NJVEZDV08vdW00TmxNeEh0QVlRMCtl?=
 =?utf-8?B?UUJ3RHN4cmlyMFJWL0xWem1PcFc1M1EyUjhEQ1lZWFM0VHNzV2p2S3dGajJS?=
 =?utf-8?B?dXJYb1p3ZG52QnVqNllwNnR6TGYyOGtveWdoL2NHQUNrM0VScllGeVNCeWtp?=
 =?utf-8?B?NWtnZzlGWEdjQmxRb3ZaeVVtcW55cUxTNHB3K3haODVqZXFiN21DUXJoblNy?=
 =?utf-8?B?SytyWjV4OHFoS1NHRFFPU21rN09uNGRoS3RwY04rNEpmUkNnK0Qxam1YWkl0?=
 =?utf-8?B?eVc3WFNUV0NIdGFGT0xXNDk2aUwyNGZPODZxY21HNi9BQlB2WlZBNVh2Njc5?=
 =?utf-8?B?Q2RLR3hDVXFRR3hwTnJiN0VIenZEVHZWaVhUWm5md1BXVXQyZkExbkplTFFl?=
 =?utf-8?B?TnJDazNTQ1NwRStrRXE0VDJtRE05MFp0Y3ZLTmVmR2huSEdZVjlFWHhBWFlY?=
 =?utf-8?B?OGN3TVR1UEtSNnEyTmJHSEI4NGJWSVRSNTJjVFRSNjh5cEZFZzl4aDZod2tX?=
 =?utf-8?B?WWVnSjBXM0haVTQ4UXRuSTlIS2c0eGprVFNQSC9GbDFsTFcya1ZCbkZxUmZy?=
 =?utf-8?B?bnI4TDhzT0FML1FERHozMngrcjl0TVV0bCtLZEhWdktodEkxZG1GR3NKdnRu?=
 =?utf-8?B?cCtZMXFlN05nVytaV2R1Vzl3Ri9WV2ljVktUbDIvSmFIaTByeWgyaGtFRnVV?=
 =?utf-8?B?YnJKMWZhK09yZEE0MXBJdkhsVXhYL1dIdXlMdGoreFJZYmhTZ0lEa2VySHI4?=
 =?utf-8?B?dUVKUkIyUmxETVN4MGJGWGdEZjRDQ2Z0QWlySEZPaUhKQ0pyN25XK1c5cmwx?=
 =?utf-8?B?d0c2eVp4YS84cmFrN0lLZjNyUVJOeHZrUzZsaXU3UFBqRExWUGhkRWk1U2c1?=
 =?utf-8?B?bWpZQnM4M0pGYTNTckxhNFlnZXpiVXpQeExmMkJRaDQ4dDEzOTdnR1B1Y2U4?=
 =?utf-8?B?ekx0d1FLSU0xV0tiWWEvZ1RLd1ozSUJHM3paZmt6dmNNTEZYaVYrWmQvWjNX?=
 =?utf-8?B?U2lzMGJsLzNYY0FHSnl6bUJIRHFpWDlhZHRzMUQva1NkREdNbUpic3BIWUkz?=
 =?utf-8?B?MEo2WHlGWUVDbGdVQVpRZkhqY1l2UzRML1o1Nm5LYlU5b2VhYzBFMXJzQW5X?=
 =?utf-8?B?djVDTzhqSjkyeTVpRlNSSCtqTnEvVElKVUYrK2NXWG16Yk5vSzlyNzBqL3pF?=
 =?utf-8?B?OFFKbnM4Snl0Y2RRYmMyMzF6MHlDM0ROMjFSblFBN2gzRWJWNk9hclkySjk2?=
 =?utf-8?B?NkpVZ3JsS1VlL3JsRnYzamxySHFxR0VQM09DMXo1SGVoUlBwOTBRUTBPMVJ6?=
 =?utf-8?B?V0djc29WdklWaUZWSG5uNEJmb1JUVjR3MXRJbnhoS3FCc1QxMXhBc0Z3Tk5U?=
 =?utf-8?B?alp0YXpObUUzODdpcVl3aVQvbUhBcHRQcC9XUFpWTGhiTkpQRm1yWWs3RVc5?=
 =?utf-8?B?RTRoUEQ5dGY0M0M2ZGpNUXVrbXF4ZDZxckxpS0FnWjN5TkJzNytFV3pNMFFR?=
 =?utf-8?B?SG5kV1lmbUtTbkNibU92Q3BoTUFIWWkzcTh0dXFncHNkclFVeHQ4dC9raXNV?=
 =?utf-8?B?YWw0S2NRTXVwVlZoK2YwTnJmQzRFYldESG9vbEQ4RlViNnFPYzB5dEpnTDhj?=
 =?utf-8?B?YURLbVdNMDJGNDRTS3JaMm43WWFKRHQyZCtpaTlGeFJrRVVxTk1qVUlYdVFZ?=
 =?utf-8?B?emlGNXJFa3hVUzl0Q29GOVFmVXh4ZVhFZU9pOEs2TmJBM0luekNUV3k5YVFD?=
 =?utf-8?Q?e/7Aup75XACJHvQgQSDtxgA=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 36f2833e-8ef4-49ae-6de9-08d9f0cb280d
X-MS-Exchange-CrossTenant-AuthSource: BN9PR12MB5115.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2022 21:35:59.3756
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: nhLLSIIkx4Ifdm1aq9fc7uIMEEHcuKGrkFhfSHdyKOXP+piLVeUWoXXH6zldy8k7ynEk4OPCBAn32DtMse+Ufg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5367
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org


On 2022-02-15 14:41, Jason Gunthorpe wrote:
> On Tue, Feb 15, 2022 at 07:32:09PM +0100, Christoph Hellwig wrote:
>> On Tue, Feb 15, 2022 at 10:45:24AM -0400, Jason Gunthorpe wrote:
>>>> Do you know if DEVICE_GENERIC pages would end up as PageAnon()? My
>>>> assumption was that they would be part of a special mapping.
>>> We need to stop using the special PTEs and VMAs for things that have a
>>> struct page. This is a mistake DAX created that must be undone.
>> Yes, we'll get to it.  Maybe we can do it for the non-DAX devmap
>> ptes first given that DAX is more complicated.
> Probably, I think we can check the page->pgmap type to tell the
> difference.
>
> I'm not sure how the DEVICE_GENERIC can work without this, as DAX was
> made safe by using the unmap_mapping_range(), which won't work
> here. Is there some other trick being used to keep track of references
> inside the AMD driver?

Not sure I'm following all the discussion about VMAs and DAX. So I may 
be answering the wrong question: We treat each ZONE_DEVICE page as a 
reference to the BO (buffer object) that backs the page. We increment 
the BO refcount for each page we migrate into it. In the 
dev_pagemap_ops.page_free callback we drop that reference. Once all 
pages backed by a BO are freed, the BO refcount reaches 0 [*] and we can 
free the BO allocation.

Regards,
   Felix


[*] That's a somewhat simplified view. There may be other references to 
the BO, which allows us to reuse the same BO for the same virtual 
address range.

>
> Jason