Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <a24d82d9-daf9-fa1a-8b9d-5db7fe10655e@amd.com>
Date:   Fri, 11 Feb 2022 12:07:11 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Subject: Re: [PATCH v6 01/10] mm: add zone device coherent type memory support
Content-Language: en-US
To:     David Hildenbrand <david@redhat.com>,
        Alex Sierra <alex.sierra@amd.com>, akpm@linux-foundation.org,
        linux-mm@kvack.org, rcampbell@nvidia.com,
        linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org
Cc:     amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
        hch@lst.de, jgg@nvidia.com, jglisse@redhat.com, apopple@nvidia.com,
        willy@infradead.org
References: <20220201154901.7921-1-alex.sierra@amd.com>
 <20220201154901.7921-2-alex.sierra@amd.com>
 <beb38138-2266-1ff8-cc82-8fe914bed862@redhat.com>
 <f2af73c1-396b-168f-7f86-eb10b3b68a26@redhat.com>
From:   Felix Kuehling <felix.kuehling@amd.com>
In-Reply-To: <f2af73c1-396b-168f-7f86-eb10b3b68a26@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dHdKdjRFVkYyTnY2R2ZvVWwzMTJiOGZyZlo2RjNWako3VWlpV3VRdWhVK2Zj?=
 =?utf-8?B?WFYwdWNrbDk4U3ArSlAwVWV5QUxoUmhBNFluem9uYXpZS1ZVc0JXQkRLbUdw?=
 =?utf-8?B?YnF6SGlXbVBSMmRUcnFyQ0I1dGRycm5PbGdXRFVGc0FnaUlqOTZlN2NsYXky?=
 =?utf-8?B?TlpVK3djL0R6ME5HYmxYQVlBMC9KeVdFcTAvVmpQQmNBUEZHNE9FNDlmWTlJ?=
 =?utf-8?B?K0E4cWRMczBrSmlrTGFkeXZzY3VibG5MWGZkMmltNkVDWkpZdVFKcEVMUS9G?=
 =?utf-8?B?cjQza2t4SlhweXV1dTh2Zi8rMnU0bVBZM2xRcW5ESUhybGQva1ZrVmF4YzNN?=
 =?utf-8?B?UnFlS3NlSFVldURSamR0a1djME9QdTFhL2UwdVlKMnp5QllZcUo2ZEh1YWNW?=
 =?utf-8?B?RlNyNjRpbkJickJZRmhoTmpEZlJ5SXRxckNkbDU5QWxQRXZabENYVnBFampu?=
 =?utf-8?B?bWRZWnoxdWw5T0ZKZW11ZDkvdEVxYkEyem5sZzgvR1JLaXl6ZTEyMkpFTjZG?=
 =?utf-8?B?Q09sM0NqWHZXY3NRRkJ3RTI3eWJ0MjR2cDZKRFh0eDgxRHRtVmVDNmtKeU1L?=
 =?utf-8?B?REVqRUdUa2xGWDQ5YVpjRnZONGpQaGorb1ExdjlhMlVIWCtaaTF2QXdmNXQr?=
 =?utf-8?B?NEVvR0k1MlVIQStsZCt3cm83alNoWGNaRWdsQUxVdmV6dTdXYXJmTlJTMFY0?=
 =?utf-8?B?MGVtV040bzJ2REJQU0hTdDJWN2pMV3BmbnRxRGdTZWFDd0pWVGkzbWNWWldB?=
 =?utf-8?B?cEQ4a2tXeUFnbS9rOU9yZk1kTHRyRk92K1dnYm9DTUVpNlhDYm9JaEltVGRQ?=
 =?utf-8?B?eHpsM3hGeVdMa0FEWC9wTjRHNWVQVmw3Y21ZVVUwMnRBaUhnb2txelJiNCs0?=
 =?utf-8?B?KzlYYzdXYlFTWnVSamJpSVUvSE40Q2Z6ZnIwRlRLQTdyYzNINEkzZk1ncWlv?=
 =?utf-8?B?UkNUb243VTBWbzA4MUxUR1krTEU0YlZjVmxWelFuOVlNb3cwS0Z6LzBuQVdq?=
 =?utf-8?B?L0dwWGQrVzB5VmFHbDEyUzErVFZ4aU9nVGI0Ylp3OTRSQWh6ZW1yZkJiczAw?=
 =?utf-8?B?a21iQ2pUb085UUVZNlJQQktzOXQ3S0d2NVVINkJNRTJ5OFB4UVJrVnIycDdp?=
 =?utf-8?B?b21sVFZjTDVFRlVTSytQV21RVUpqajNCYlhKWkZnZldJMUxaOTY2YU5kZzMz?=
 =?utf-8?B?cDJkSDAxSWd1dGNObXdlZjVDTXFKbFBtRmtwR1VQeXpyM20wL3RKRHdFck9j?=
 =?utf-8?B?R0c0NGxyNW5GMEdmdlRkaXRuck5tZmF3cldwUXkyYzhnQmZ6NVhhb0ZxSHRM?=
 =?utf-8?B?bkE1SDcrM2s3K1dxUDBYWTF0RWQyTnRnN1hBdElMcHdXUDRDeHpPK1lGTmE1?=
 =?utf-8?B?THlBV2ZlR1oxVkNCbVRtRG1MN3ZWeC9vcHl4SVhZbnVMV3FLTVpwRlFKZ05C?=
 =?utf-8?B?eElpY3R0c2UvNE1aNE1SeWt4THlmVlU1QUMwek11d1lPdkx6NFYrTDZGYy9k?=
 =?utf-8?B?QTJsK1E4UlAyZW5qTUN5bWkxNWpvNzVNd3FmOXkvblo0cGd5M0krNjBUYWlz?=
 =?utf-8?B?UmFWN1dFOFd1V1JVL09zUWN6eWhwSWZpc0grSWNkNGtOVjJnNE9kbnVhMUti?=
 =?utf-8?B?WjI4RnE5RHViMmQ0UGRMSUdKS29yZ200VXh4UlFHUitXMms0bklnRGVJNDJr?=
 =?utf-8?B?MlNtbERLTE1hRWZob0YvbmtwTFBMT3lEYWYybFJDUHl3QXhrSUtwR2RGd004?=
 =?utf-8?B?Vzh1YUdMOGlwNHQ0akVOcUhHNjNET3Uwd2hsOXhoTWtFWFlWSVM1S0c1azdp?=
 =?utf-8?B?R1FEVTRtZWR4RytTVVNRdjZQWlFGRTRmZzV3Z1E3dUtMWFVJZXpVSUlPRUlq?=
 =?utf-8?B?OGpGUzVBR1dvZkU2dGxmQXMvMXFDMVc0ZEVsM1NUd3ppdVRnUkUzRXl0dHpx?=
 =?utf-8?B?aE43SDFaNlZ5ZHdBd0F2WUpuYUsvS3NON2s0WG5pZWM4WmtwQnM3SmY1bk9S?=
 =?utf-8?B?alhhK0xzWHd4MFRFMnY4ejJMVUZ6UDVmQytTOUVMN0pnVy9yN3dKSHpZbEpU?=
 =?utf-8?B?RE55MnpkUkVGNXZkR0pDVmdxRlJzR2xTUU04SkRYRGdKcUJNQUM5TTZQZm5U?=
 =?utf-8?B?empneGxieGQ1Nkh0SC9RUmkzRklCOGpqdXNSTVVjQTd1V0hpSnM4TitqUFgr?=
 =?utf-8?Q?fQEqHi9zliDap43iP9X5btI=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5b65f081-ee5b-49bf-546e-08d9ed80f2a8
X-MS-Exchange-CrossTenant-AuthSource: BN9PR12MB5115.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Feb 2022 17:07:13.3444
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: uKbA+uuO0Vvo3mqirdvZIipYvNqwJxlwnWZXPzzlghUj1nZWX72Vqk/fQ35SUd7yHstsoudwIQhRK2F/HX4yQg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1925
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org


Am 2022-02-11 um 11:39 schrieb David Hildenbrand:
> On 11.02.22 17:15, David Hildenbrand wrote:
>> On 01.02.22 16:48, Alex Sierra wrote:
>>> Device memory that is cache coherent from device and CPU point of view.
>>> This is used on platforms that have an advanced system bus (like CAPI
>>> or CXL). Any page of a process can be migrated to such memory. However,
>>> no one should be allowed to pin such memory so that it can always be
>>> evicted.
>>>
>>> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
>>> Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
>>> Reviewed-by: Alistair Popple <apopple@nvidia.com>
>> So, I'm currently messing with PageAnon() pages and CoW semantics ...
>> all these PageAnon() ZONE_DEVICE variants don't necessarily make my life
>> easier but I'm not sure yet if they make my life harder. I hope you can
>> help me understand some of that stuff.
>>
>> 1) What are expected CoW semantics for DEVICE_COHERENT?
>>
>> I assume we'll share them just like other PageAnon() pages during fork()
>> readable, and the first sharer writing to them receives an "ordinary"
>> !ZONE_DEVICE copy.
>>
>> So this would be just like DEVICE_EXCLUSIVE CoW handling I assume, just
>> that we don't have to go through the loop of restoring a device
>> exclusive entry?
>>
>> 2) How are these pages freed to clear/invalidate PageAnon() ?
>>
>> I assume for PageAnon() ZONE_DEVICE pages we'll always for via
>> free_devmap_managed_page(), correct?
>>
>>
>> 3) FOLL_PIN
>>
>> While you write "no one should be allowed to pin such memory", patch #2
>> only blocks FOLL_LONGTERM. So I assume we allow ordinary FOLL_PIN and
>> you might want to be a bit more precise?
>>
>>
>> ... I'm pretty sure we cannot FOLL_PIN DEVICE_PRIVATE pages, but can we
>> FILL_PIN DEVICE_EXCLUSIVE pages? I strongly assume so?
>>
>>
>> Thanks for any information.
>>
> (digging a bit more, I realized that device exclusive pages are not
> actually/necessarily ZONE_DEVICE pages -- so I assume DEVICE_COHERENT
> will be the actual first PageAnon() ZONE_DEVICE pages that can be
> present in a page table.)

I think DEVICE_GENERIC pages can also be mapped in the page table. In 
fact, the first version of our patches attempted to add migration 
support to DEVICE_GENERIC. But we were convinced to create a new 
ZONE_DEVICE page type for our use case instead.

Regards,
   Felix


>