Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <41aecd10-9956-0752-2838-34c97834f0c8@amd.com>
Date:   Thu, 22 Jun 2023 15:54:51 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.0
Subject: Re: [PATCH drm-next v5 03/14] drm: manager to keep track of GPUs VA
 mappings
Content-Language: en-US
To:     Danilo Krummrich <dakr@redhat.com>, airlied@gmail.com,
        daniel@ffwll.ch, tzimmermann@suse.de, mripard@kernel.org,
        corbet@lwn.net, bskeggs@redhat.com, Liam.Howlett@oracle.com,
        matthew.brost@intel.com, boris.brezillon@collabora.com,
        alexdeucher@gmail.com, ogabbay@kernel.org, bagasdotme@gmail.com,
        willy@infradead.org, jason@jlekstrand.net
Cc:     dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org,
        linux-doc@vger.kernel.org, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org,
        Donald Robson <donald.robson@imgtec.com>,
        Dave Airlie <airlied@redhat.com>
References: <20230620004217.4700-1-dakr@redhat.com>
 <20230620004217.4700-4-dakr@redhat.com>
 <cf6846ea-5bd0-0b41-b7e6-901c70701751@amd.com>
 <a8edf75b-e0f7-a6c7-7d29-f0d39923549b@redhat.com>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
In-Reply-To: <a8edf75b-e0f7-a6c7-7d29-f0d39923549b@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?c01ZVjNLd0o2Z3N3UnRha3BnV2VJM2krY2ZyQ2d6TkxTS3hLR2p0S1Q4eDNm?=
 =?utf-8?B?NmVZWUtLQTc5WHEzeE5wQnFnSVJIOGNxTnJ3QVFvY1laNS9iSFI4d0RoVlFh?=
 =?utf-8?B?RFRKSERGTGcxRWdsNnV6M1ZLWnVKaVRYUnJwTUJsUENna2ZqNnNscTVwd2Zt?=
 =?utf-8?B?UmxweVdQKytLWmdLK0JkQnVBTk9RZEY4dU5lTGVPL0NLU2tUT05HVEwzK1dr?=
 =?utf-8?B?WDBYNHJFb251N2pzRld0UjE0Ry8xM3c4VHhsOW44K2xldHJ4L2dvWU9BNEZW?=
 =?utf-8?B?amlwNUNNMXhETGJwMXhndHJEN2xIZ0NaMUc4cjYvYnQ2eWViT2t4WHo0RHJ1?=
 =?utf-8?B?dFd0eFBYNzRnTW5ZM1FDZzgyRWhMekFpTlViaVlSTDdKZmNRZUd2dVY0bkhr?=
 =?utf-8?B?V2VqOG1IT2I5bnBBbnZJRzdQSTdPamJzU1FGY3FtZmtNVEFoaXZGTGdtNnJE?=
 =?utf-8?B?dmFVWDQ4TnFVMTFHVmRkQ2FZU2RWNENCMkpuQTMvKyszYlJldTF5V1VEOXBu?=
 =?utf-8?B?WW93THVEbkZKbE1xLzdvRCswNEJlYktaZGh6N0JZMGxKV1pMRmRKb3RYdlFK?=
 =?utf-8?B?bW94WTJnUWRpUDVJTWFvWDJzZlBnSmMwRWRreXAvUmJMcEs2bWtLUXRzU3lG?=
 =?utf-8?B?KzhXclRTakN3Z1R1cDdaUTdPUE1JV2gvaDM1TUhkSlhYbTJDOGxPNFpoQW11?=
 =?utf-8?B?Sm9vbEtEYk4wL1hQYm15Vi9KNUhJdlVITjVXakUzWHNzS09FQXZhZjA4Z1Jq?=
 =?utf-8?B?ZXk0STE0TjdTTUJyTXRCeGVmdDRoOG5DcWFMcWU3RnM4eGt6c3hSNkdoalBo?=
 =?utf-8?B?dlZ4dk05U0o5RHUrSEpVQ1hlSlR2MDRBalVIS2VhbENZQUowKy9ScWNtSVVX?=
 =?utf-8?B?MWhGVVFpdlJUOUs4WUJML1hnY1RKRWM2YWFBVElyVkxIQldFb1VRclVrcVNk?=
 =?utf-8?B?OTNuK2FHdjRMQ3Vobit1bG9iTitCbmNzWTRxeSt3VXJTdTF3VWVOY0VjNjgr?=
 =?utf-8?B?ak0yVExYSGhKYmpoZk9GNDhuUUtGNzNZNEFJN1E3NUk1MGdydkpSRGJodDdo?=
 =?utf-8?B?ZEJLSzRobnVSTm5Fc0hRdzYxRWQ4L3doVlhmb3VDN1BSbzZLSXBoenk5eitY?=
 =?utf-8?B?VDNvNC8wV0JQcExjUUdLSHdxaWdlRXVLWG9oKzNlUUNBQ1plNDJDL24yelZF?=
 =?utf-8?B?OEVaZE9XWDRUeGdyRXBCRUJiNFdQV0ZhREJSZUhKWG1NVXViM0h0VkcxTng3?=
 =?utf-8?B?ZHpJUnVLQVR6S01TN0lSUTBNckFDbDlya3lBOHZrczVPWFBJWjByOG5zUGdM?=
 =?utf-8?B?MWx3QzdsSE9xMlpFckVSVVhaYmNxMk9HcmlEbVRKb3ZSUHhjbEQzQmtMeFdJ?=
 =?utf-8?B?dklEYmxQZis1clgyVDhvaW1HQWJuTVNuZlAyUTI3clZ1RUFVS0FFSW5FeUtN?=
 =?utf-8?B?eC9WbWZtTDN1K3g2VEhWUFhtV3ZTcEgrNlJlUHd2Ny94WWs5TWRnYmQwWUs2?=
 =?utf-8?B?M3I3US8xSHdhTHd6RVpZbE0xWFoxcjFKY3hoL2NjdjRsdEorZ3J6ZlV2S2U4?=
 =?utf-8?B?TDFrcjJMWUZJTUlFN2hVenAvWW9GZ1ZDN1UxUWxncWpuT0Jta1VjOGpMbCto?=
 =?utf-8?B?S3JhVTNWV1dXQzBoL0JDWEgvY0V6bHIwMmhSNFNJN3JxUFBEWnhsdWltZHJm?=
 =?utf-8?B?N0xQVDFnTmMyV2RPbytjbkQzeG52R09yNE11TWs5aTZCSmpmVHcwZ0ttcVJ5?=
 =?utf-8?B?OVlsTFhNd3QwOXpQdWxBNUdBeXBGQ25NNzFwUGtkU2lNMjlIN1N0T3RIMjVI?=
 =?utf-8?B?VUd3SngwRUFSV3A3TlhyNjUvQ2dqUHNvWnJHN3B2L2dHZmtPS0N4VVJxRFJ6?=
 =?utf-8?B?K0laTWY4V2tkMHlqK1BCc1FkZGJrV3dsRTNsU1RmaWNacG1WV0I4amd6QjdD?=
 =?utf-8?B?VXpMYVo4TTdvTFpBbGVDZTBJNjh4LzZ3NDJCaXMrQmpucEI3RDI1Rjd1dnlF?=
 =?utf-8?B?dGJteUNiVWJDOVpYUlNkU3RZSVdzbjRZN09PQkpramVLUEtacTIyQTkzc1N0?=
 =?utf-8?B?eUJPTi9maVQzaVZsSVpndkw0M3prMHRSVHNVZ3FQVWlFUStDMWVvVVRQZlNK?=
 =?utf-8?B?Wmc4R3ZKam9qNXl5eTJkQk0zTERwbngzVXk3aDdnN3pQQUZ2NzlJT05TdHNl?=
 =?utf-8?Q?aFfIkRH4QLvlgmEbPGDZYBomMM7hNGTaO2n3h6fBumRY?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 118ca507-aea3-4868-9e17-08db73284348
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3587.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jun 2023 13:54:57.1208
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: WV9aN4OUL8VmVyvbtRTs5CD04l85uWRAt46eWxkdCkejs8zcG4Esa95Y+aIGnkqY
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6683
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Am 20.06.23 um 14:23 schrieb Danilo Krummrich:
> Hi Christian,
>
> On 6/20/23 08:45, Christian König wrote:
>> Hi Danilo,
>>
>> sorry for the delayed reply. I've trying to dig myself out of a hole 
>> at the moment.
>
> No worries, thank you for taking a look anyway!
>
>>
>> Am 20.06.23 um 02:42 schrieb Danilo Krummrich:
>>> [SNIP]
>>> diff --git a/include/drm/drm_gem.h b/include/drm/drm_gem.h
>>> index bbc721870c13..5ec8148a30ee 100644
>>> --- a/include/drm/drm_gem.h
>>> +++ b/include/drm/drm_gem.h
>>> @@ -36,6 +36,8 @@
>>>   #include <linux/kref.h>
>>>   #include <linux/dma-resv.h>
>>> +#include <linux/list.h>
>>> +#include <linux/mutex.h>
>>>   #include <drm/drm_vma_manager.h>
>>> @@ -379,6 +381,18 @@ struct drm_gem_object {
>>>        */
>>>       struct dma_resv _resv;
>>> +    /**
>>> +     * @gpuva:
>>> +     *
>>> +     * Provides the list of GPU VAs attached to this GEM object.
>>> +     *
>>> +     * Drivers should lock list accesses with the GEMs &dma_resv lock
>>> +     * (&drm_gem_object.resv).
>>> +     */
>>> +    struct {
>>> +        struct list_head list;
>>> +    } gpuva;
>>> +
>>>       /**
>>>        * @funcs:
>>>        *
>>
>> I'm pretty sure that it's not a good idea to attach this directly to 
>> the GEM object.
>
> Why do you think so? IMHO having a common way to connect mappings to 
> their backing buffers is a good thing, since every driver needs this 
> connection anyway.
>
> E.g. when a BO gets evicted, drivers can just iterate the list of 
> mappings and, as the circumstances require, invalidate the 
> corresponding mappings or to unmap all existing mappings of a given 
> buffer.
>
> What would be the advantage to let every driver implement a driver 
> specific way of keeping this connection?

Flexibility. For example on amdgpu the mappings of a BO are groups by VM 
address spaces.

E.g. the BO points to multiple bo_vm structures which in turn have lists 
of their mappings.

Additional to that there is a state maschine associated with the 
mappings, e.g. if the page tables are up to date or need to be updated 
etc....

> Do you see cases where this kind of connection between mappings and 
> backing buffers wouldn't be good enough? If so, which cases do you 
> have in mind? Maybe we can cover them in a common way as well?

Yeah, we have tons of cases like that. But I have no idea how to 
generalize them.

>
>>
>> As you wrote in the commit message it's highly driver specific what 
>> to map and where to map it.
>
> In the end the common case should be that in a VA space at least every 
> mapping being backed by a BO is represented by a struct drm_gpuva.

Oh, no! We also have mappings not backed by a BO at all! For example for 
partial resident textures or data routing to internal hw etc...

You can't be sure that a mapping is backed by a BO at all.

>
>>
>> Instead I suggest to have a separate structure for mappings in a VA 
>> space which driver can then add to their GEM objects or whatever they 
>> want to map into their VMs.
>
> Which kind of separate structure for mappings? Another one analogous 
> to struct drm_gpuva?

Well similar to what amdgpu uses BO -> one structure for each 
combination of BO and VM -> mappings inside that VM

Regards,
Christian.

>
> - Danilo
>
>>
>> Regards,
>> Christian.
>>
>>
>