Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <7ab81ae3-d2c1-d633-5050-3562e9923771@amd.com>
Date:   Tue, 15 Feb 2022 13:52:35 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Subject: Re: [PATCH v6 01/10] mm: add zone device coherent type memory support
Content-Language: en-US
To:     David Hildenbrand <david@redhat.com>,
        Jason Gunthorpe <jgg@nvidia.com>
Cc:     Alex Sierra <alex.sierra@amd.com>, akpm@linux-foundation.org,
        linux-mm@kvack.org, rcampbell@nvidia.com,
        linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org,
        amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
        hch@lst.de, jglisse@redhat.com, apopple@nvidia.com,
        willy@infradead.org
References: <20220201154901.7921-1-alex.sierra@amd.com>
 <20220201154901.7921-2-alex.sierra@amd.com>
 <beb38138-2266-1ff8-cc82-8fe914bed862@redhat.com>
 <20220211164537.GO4160@nvidia.com>
 <6a8df47e-96d0-ffaf-247a-acc504e2532b@redhat.com>
 <20220211165624.GP4160@nvidia.com>
 <7b830dc4-37bc-fb7b-c094-16595bd2a128@redhat.com>
From:   Felix Kuehling <felix.kuehling@amd.com>
Organization: AMD Inc.
In-Reply-To: <7b830dc4-37bc-fb7b-c094-16595bd2a128@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SUlONXlTTWllc0ZQd0NOODU1Y3ZYM1lOcTBISURrMkRNekJqZDUvR1JwUTZ0?=
 =?utf-8?B?U0Y2L1VKMm1HZjlWdXFXd1Z5WHZSQjk1MWZmVStTb0JqVE5XRS9xUzY5NUx2?=
 =?utf-8?B?RFhUZjdYektnZEJmVG1DeHlrRnh3SHBaREE5SUZwN2Y0cHpaTVNpbkZxaDMr?=
 =?utf-8?B?YnROb1FWR0dQWDkwRlJHdGlhMWdzMFVRTkJSRzFMTTF2UTRRY0NRVUxSRWhj?=
 =?utf-8?B?bGdkOHdlV25zVjc5dDJ6T3d5Ukt0aDZHZm8yWFJUMzlYUHRkbk1IN2d1MER4?=
 =?utf-8?B?RW1jR2dJRE8vZE9hUDlZeWhjdE9nT0JMRk5VZnh2cEFBSFpNcEpXLzBHcEJr?=
 =?utf-8?B?WTZSRU9rb0lWMmNBeWJFRXFUSWFpRFFnSGxINkx1eEZGbkxPRlhBZ3VVRW1L?=
 =?utf-8?B?NUFmZGRKT1RpWWtYSW9NU0lyVHhoSUNvaFpmeXBTSjlXeDJIZ0JPOCtHS0Mr?=
 =?utf-8?B?TWZmQjJOc1l6Z0xmOC9tMXlYOXM4a0xGVFpDazJPMnd2cDNNQmxJaWdESDhh?=
 =?utf-8?B?cForYTZYYjFHSFBZUTRWcDZmYVo5RGhtYTZsT0ZiYUlDZFBiUlU4YjRZQmZC?=
 =?utf-8?B?ajdBeXB2L0VBc3l0SUNZbkQrZmp0NTY3MkJ0aUlMMmk2a2ppeGFZeUoxMDlT?=
 =?utf-8?B?MDBUMnBnVm9kUkxzZGZWZm5yM2VsRUZHZFhiNGpjdi90bFRQaXZKNmFsanhS?=
 =?utf-8?B?OHhKbTlzK2N5WkpsUE5ReEo4QjA5WXN3dmd2T1FwN2ZaVTFUZGJRbWdsczFi?=
 =?utf-8?B?RzBzdGlOelYza05PNko0Tmh4VU1oRXQxczc3ajAvV1lBUTFIUFRrTmQ4Z1F6?=
 =?utf-8?B?cllqcWlKSW1FcHUzaHVaRE41dWJ2SzRRQW1pUG1QNGlRUVM2Vk5vMmI0eVBV?=
 =?utf-8?B?ekhpRHovd0RtRnE4U0pXTEVRbmZFZkh4MWdHdlRIUFA2cDRnZlM0a1ZBMnlK?=
 =?utf-8?B?MWtIamVTQWUySHZoWXAwNjN2TjhSM2t4cDVPMFZwQ3RsdzRLR3BnNjBvLzlo?=
 =?utf-8?B?WUVpSEtnWmsyUnRUUnZSZTBSZ1lXVEJOR00zbi9UeXdyNXpSLzdGOVQ3cTBF?=
 =?utf-8?B?dHp1d3UwMHVDYWk5WSsvdmFWeFFLMHZiSFZOLzFlZzJmK01jMnNzY1g1Y2ZF?=
 =?utf-8?B?T0t3U2NyZGJzQkNMdG43eStVV1M5c015MUZHaTlYYkozRDF0eUljUW9TVnNm?=
 =?utf-8?B?ajhIYUwvdHB2TUhYKy9rNWF0L3RrcXM0SnZUNEZFLzlyVS9yQ3VlTHd6UnZR?=
 =?utf-8?B?bVgvRERJb0d6WUNRTnJGWnQwRTBYWUxMYURFRmtvREEwWEVudGxpYXcraEdU?=
 =?utf-8?B?cjBhUGk5c0JXRU9lWGM2d08rYlNCcEVXelJhdHhWNXoyUTRjNHVadmVNTDJN?=
 =?utf-8?B?R3ZaRTRwdG5BMlNTenIyWVNpRE9FbW1PUzlGT05nNHUvUk1UMDVjUFNraENo?=
 =?utf-8?B?TDlKUFRUc0lNNW1kUG1JQkRqbUhWWjRsNWxzd2lteFd5QVlCeHErQXNlemF4?=
 =?utf-8?B?VHVSL2tCL01qRUhsR01aMUpMeWJ3ZmJYWkMvRi9NRXg3NkZDaStleG9SQkMz?=
 =?utf-8?B?RGRKZHROcG1hdUhycnk0d083RWU2Z3FKOG1yc24wcWFvMHlHZEtLMjR3NXhr?=
 =?utf-8?B?cEplWUNTMEhmWTUxNWxlMHNCT2VFQVJpbXU4b0JWeWZJd0VwYVY3VlRLUmtG?=
 =?utf-8?B?b2IzU01UbitwcSt0dW9SaUJuNzd5UjIxc0dhZjZ1YnRnVGp6SjBtb2JsRFJK?=
 =?utf-8?B?MHdyQVdPSDkyZnJJVVp1cEJycEl1RlJ0V2pSYm44ZnNSeERINDh3Z3VtQll6?=
 =?utf-8?B?R2ZreGczQmdNRGEzUE9CbXROZThNSk9JY2c4YzRNVC9kS3FlZytXL3ZVZ0lG?=
 =?utf-8?B?YWNuMnFnN1llbmI3dEwrWjg0cWpuTXIrbld2eUtMMkNoeUZwWCtIdEFyQWZI?=
 =?utf-8?B?RFplQXRlYTBZWkgyQkF0Rkg2bVMzcTNPRmZ5SXNUMXlOazVDZ3dPSkNoU2RW?=
 =?utf-8?B?K1Bua0ZINXpzK3RraS9pRmozL0ppWU93cWs5R3F4SWpJM014eXRNRW9TZHdn?=
 =?utf-8?B?b3JKdGkra2IwQzNESC9nUnl5MjA0Tm9NaFh0ZFk4U1VSSk9BemR4RmNPSitK?=
 =?utf-8?B?UVhmbzJwQy81T0xkWmpWNmxUK3FhTldPZWJhVjRtdFhhV0UyZ21pTkRIWmJK?=
 =?utf-8?Q?h6onjWFy2dIQBZZnM8DtVpU=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8ac95cb4-537b-4735-4ca9-08d9f0b45601
X-MS-Exchange-CrossTenant-AuthSource: BN9PR12MB5115.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2022 18:52:37.9215
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pQarD/fMTx0oySYgRCBq+ErUopWxrLb+2PvxeJMzSgF1hbuhc3xYGVZOdAdQmazXtWvQQitFkztXd8UgZcdAQw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4910
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org


On 2022-02-15 07:15, David Hildenbrand wrote:
> On 11.02.22 17:56, Jason Gunthorpe wrote:
>> On Fri, Feb 11, 2022 at 05:49:08PM +0100, David Hildenbrand wrote:
>>> On 11.02.22 17:45, Jason Gunthorpe wrote:
>>>> On Fri, Feb 11, 2022 at 05:15:25PM +0100, David Hildenbrand wrote:
>>>>
>>>>> ... I'm pretty sure we cannot FOLL_PIN DEVICE_PRIVATE pages
>>>> Currently the only way to get a DEVICE_PRIVATE page out of the page
>>>> tables is via hmm_range_fault() and that doesn't manipulate any ref
>>>> counts.
>>> Thanks for clarifying Jason! ... and AFAIU, device exclusive entries are
>>> essentially just pointers at ordinary PageAnon() pages. So with DEVICE
>>> COHERENT we'll have the first PageAnon() ZONE_DEVICE pages mapped as
>>> present in the page tables where GUP could FOLL_PIN them.
>> This is my understanding
>>
>> Though you probably understand what PageAnon means alot better than I
>> do.. I wonder if it really makes sense to talk about that together
>> with ZONE_DEVICE which has alot in common with filesystem originated
>> pages too.
> For me, PageAnon() means that modifications are visible only to the
> modifying process. On actual CoW, the underlying page will get replaced
> -- in the world of DEVICE_COHERENT that would mean that once you write
> to a DEVICE_COHERENT you could suddenly have a !DEVICE_COHERENT page.
>
> PageAnon() pages don't have a mapping, thus they can only be found in
> MAP_ANON VMAs or in MAP_SHARED VMAs with MAP_PRIVATE. They can only be
> found via a page table, and not looked up via the page cache (excluding
> the swap cache).
>
> So if we have PageAnon() pages on ZONE_DEVICE, they generally have the
> exact same semantics as !ZONE_DEVICE pages, but the way they "appear" in
> the page tables the allocation/freeing path differs -- I guess :)
>
> ... and as we want pinning semantics to be different we have to touch GUP.
>
>> I'm not sure what AMDs plan is here, is there an expecation that a GPU
>> driver will somehow stuff these pages into an existing anonymous
>> memory VMA or do they always come from a driver originated VMA?
> My understanding is that a driver can just decide to replace "ordinary"
> PageAnon() pages e.g., in a MAP_ANON VMA by these pages. Hopefully AMD
> can clarify.

Yes. DEVICE_COHERENT pages are very similar to DEVICE_PRIVATE. They are 
in normal anonymous VMAs (e.g. the application called mmap(..., 
MAP_ANONYMOUS, ...)). You get DEVICE_PRIVATE/COHERENT pages as a result 
of the driver migrating normal anonymous pages into device memory. The 
main difference is, that DEVICE_PRIVATE pages aren't host accessible, 
while DEVICE_COHERENT pages are host-accessible and can be mapped in the 
CPU page table or DMA-mapped by peer devices. That's why GUP needs to 
deal with them.

Regards,
   Felix


>
>