Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <923e914f-d367-2f74-9774-f0024d946bdd@amd.com>
Date:   Fri, 23 Jun 2023 17:34:19 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.0
Subject: Re: [PATCH drm-next v5 03/14] drm: manager to keep track of GPUs VA
 mappings
Content-Language: en-US
To:     Danilo Krummrich <dakr@redhat.com>, airlied@gmail.com,
        daniel@ffwll.ch, tzimmermann@suse.de, mripard@kernel.org,
        corbet@lwn.net, bskeggs@redhat.com, Liam.Howlett@oracle.com,
        matthew.brost@intel.com, boris.brezillon@collabora.com,
        alexdeucher@gmail.com, ogabbay@kernel.org, bagasdotme@gmail.com,
        willy@infradead.org, jason@jlekstrand.net
Cc:     dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org,
        linux-doc@vger.kernel.org, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org,
        Donald Robson <donald.robson@imgtec.com>,
        Dave Airlie <airlied@redhat.com>
References: <20230620004217.4700-1-dakr@redhat.com>
 <20230620004217.4700-4-dakr@redhat.com>
 <cf6846ea-5bd0-0b41-b7e6-901c70701751@amd.com>
 <a8edf75b-e0f7-a6c7-7d29-f0d39923549b@redhat.com>
 <41aecd10-9956-0752-2838-34c97834f0c8@amd.com>
 <bcde7ea3-fbab-3a18-e810-64b6589ddb18@redhat.com>
 <86ef9898-c4b6-f4c0-7ad3-3ffe5358365a@amd.com>
 <c1f05169-dec0-22ee-52fa-c8070678394e@redhat.com>
 <2f502150-c1f8-615c-66d9-c3fb59b8c409@redhat.com>
 <4a52ac7c-19ba-8906-5902-fbf75673bf59@amd.com>
 <d18a4ea5-4e8e-be69-84c3-ca658f5cfd24@redhat.com>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
In-Reply-To: <d18a4ea5-4e8e-be69-84c3-ca658f5cfd24@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?LzQxaUx3bHBEd2V1eFRZR004cFpiLzZXN3k2MExVZk1ueWdLWDZkZktIVnkv?=
 =?utf-8?B?YXErL1dLU2FMQ210WFBqakE3TU5LMUZIZ3k0VCtMdVlvOWtiOEFTSUVTL091?=
 =?utf-8?B?cWlDby8vSVFPWGl6TGczTzZ6YWFCK2ZUUzNrSjBlc1poN0ZBQlBNL2gwL0Y5?=
 =?utf-8?B?OEptN0xnZitVWW1Ud2ZUU1YvN1N3WEphVFd5bkhCVmRzSWdlL3ZzY0E4eTZt?=
 =?utf-8?B?aWxKV2hSaGp3NFdLYmptMzAzOHUwRVk1bHBpTGIrbXIzZHFlUjQ3SUozZWM4?=
 =?utf-8?B?ZWg2eW1samh6OUZBaFQ4Z0twaDdRd0NpTFEwTWgra0VYeHpiNlJndnQrdWhZ?=
 =?utf-8?B?UlY5TkJOcTA4ZUVGbGRQNEdaUTdSQm0wWUt5N25VK2lnckt3V2hFVXpabENL?=
 =?utf-8?B?dldpb2RwZ3dQeGdtNlZBN3FJNXJITmx2a21WL1d4c1Q0cW9KOHNzSEUza0po?=
 =?utf-8?B?dXBhNzFaT2xoTzNSS2hZMUNmVGtCVmZXc0RUc2NhSDZDNWF2VEtCUFpibG9x?=
 =?utf-8?B?RFRpWEZMSXpKRWlLNHRpWURkUFFJcWdWQWcwcUJ0TWY0V0NpQjJtNWlFMkRv?=
 =?utf-8?B?YjhEdE5LaFRaaXA1YkVTM1k5YmZzZGJrZndEV2ZkNjA3aGdUUTQ5VWhRWWlr?=
 =?utf-8?B?alhIT2h5OXY1MjZkcldadnpkL1pzY1VWdUladFA5cjFGSVE0T0VRZXMyMmZP?=
 =?utf-8?B?a1JHZTBPdW16c0o5UU1NcTZNNmlTTFp6cHBhUlFZbG1oWkYrZTVJOUJhd3RH?=
 =?utf-8?B?ZFFLdkF0T3YvbFdxK3VVZlc1ZWVNSitlMVNBd3h3MnprOFZ2ajcwRHlxQVYv?=
 =?utf-8?B?UnhqRGFwSWZ3MElrai9GN2ZsMEhpWDQ3d25oaG83WG4vWEpNNS9ENmtab2cy?=
 =?utf-8?B?THZFRE93TC9wZ1k4OGJQWGc3TFhKMlgyNVgvRUtSc0FTa1JYSEtLdTVpSThy?=
 =?utf-8?B?V2lXN1dqeVRpd1hOTzRtUFdiVC82amV3ckg1REZRcVhWTGtqVFBVZ3dvSnhy?=
 =?utf-8?B?aUx3WVppcDBUWURTV2wyU216aVozZlJCWk9mUGlrVU5VNTQ3bEcxaVpHcUNQ?=
 =?utf-8?B?Ky9BY0Y5cUMvK3pNYXpGcFBCZkw0WXh5dGcySzM0RjVQMmh1WExIYm5oYUVS?=
 =?utf-8?B?bjdqdldXaUdJK2xaQVVJNHh0bHZqT3dmdFRzRDhkSmZrTE5TNTNoYnovRTl0?=
 =?utf-8?B?OXVZbW9TZUs5a0JheE9iTU9FbjhWUEpPeUxmQ05aQ1Z5L3ZsT2FZMUJHdGpD?=
 =?utf-8?B?NjJHY28vMllBbER2Z2tqRGZFdUJBS3UrU1ptSU1XdEYyMy9IRlc2SENMbUxw?=
 =?utf-8?B?bGZDV3pLdFV6YXYrdzF3YS94VnhKTnBVZVRiZXgwNkhPWUZtWVdWQ0pMenAr?=
 =?utf-8?B?R3FzVUQydGtvYW9VZnhVbTdZSGROSDA3NENQZEYrSi9mNVVOL0NsWGRPYU1J?=
 =?utf-8?B?VVZuN0pVUnorWkVtKzg2TUNFMGlmVmFlWVZqT1F2ZTNzV2xWb3NuMk1BU2d3?=
 =?utf-8?B?SHo5WjY5bkZic3VXdE9Xc2xLZGpENS90SXVmc05EdFJsaVA2eFBBNWlOMnhF?=
 =?utf-8?B?SndhMU1pRHNqNnE0VVBzUmNEVjZHQzE3dXQ1aGNSY1pOVGRyek80Nlp2NFVr?=
 =?utf-8?B?Ym9Vd0hyQmczbU5zL2UxSkdlZE9CWkR1cjBRdzhnZGxjZno5Zy9qUFQvRlVF?=
 =?utf-8?B?Zi92VjJ5TXNzYTI5Y3pBcmFjbzhmRUx1YWgxN3lsWll4QXhaZ3hhbkhha2Fo?=
 =?utf-8?B?Q3BQYVhZQ1FwMXlHWHovRFZqSVFUTkdrNGRVQlpaa2RuQnhqbFlUT1VGdDBw?=
 =?utf-8?B?bDdDbUdkRDBIcTROTnBJUmc2djNUclQ1dGt0ZGxYR2tLKy80emtqUExxTjJy?=
 =?utf-8?B?VTRlekwxM3oxd3gvMVdEbWZBWmFuRU5mclNUZDNYeUVRSkkzM1cyZ0dWZHh5?=
 =?utf-8?B?WnRxNlVvMEN1Y05SM2ljSEtiMlhYTkpTT3FnT0xMQm5nZUVTVzdiTHk4MUVX?=
 =?utf-8?B?cysyM21uc3hUeDhjZVdvenFDUEQwblAwbXZMQkx5Q1BYeU9jaDV4M25WNGd6?=
 =?utf-8?B?ZnBZeWh1YkVLb1B4MWQ2UFYwNndFR0w3bGZqRTJ4c2tRclJ1VUNmUEc1SEFu?=
 =?utf-8?B?RHlJbCtOOXdMNHhjMnZyZzNXZjlGQlFIQnBGWWcxazR2dDVCNzd1cVNQVWF5?=
 =?utf-8?Q?mPza21kACHVwc8JV9WPoQJJ1oXgkKQeGkZ3MhRM4Rh7d?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 41e5811d-6aff-490a-260a-08db73ff53c0
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3587.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jun 2023 15:34:26.4166
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: cHnF45eJ+195HB/5TVixGA9/cfo9lydwsW7mIQ6gLWVWsqYq2nkY0Hg6EB40qwkj
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB9176
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Am 23.06.23 um 15:55 schrieb Danilo Krummrich:
> [SNIP]
>>>>> How do you efficiently find only the mappings of a BO in one VM?
>>>>
>>>> Actually, I think this case should even be more efficient than with 
>>>> a BO having a list of GPUVAs (or mappings):
>>>
>>> *than with a BO having a list of VMs:
>>>
>>>>
>>>> Having a list of GPUVAs per GEM, each GPUVA has a pointer to it's 
>>>> VM. Hence, you'd only need to iterate the list of mappings for a 
>>>> given BO and check the mappings VM pointer.
>>
>> Yeah, and that is extremely time consuming if you have tons of 
>> mappings in different VMs.
>>
>>>>
>>>> Having a list of VMs per BO, you'd have to iterate the whole VM to 
>>>> find the mappings having a pointer to the given BO, right?
>>
>> No, you don't seem to understand what I'm suggesting.
>>
>> Currently you have a list of mappings attached to the BO, so when you 
>> need to make sure that a specific BO is up to date in a specific VM 
>> you either need to iterate over the VM or the BO. Neither of that is 
>> a good idea.
>>
>> What you need is a representation of the data used for each BO+VM 
>> combination. In other words another indirection which allows you to 
>> handle all the mappings of a BO inside a VM at once.
>
> Ok, after having a quick look at amdgpu, I can see what you mean.
>
> The missing piece for me was that the BO+VM abstraction itself keeps a 
> list of mappings for this specific BO and VM.
>
> Just to make it obvious for other people following the discussion, let 
> me quickly sketch up how this approach would look like for the GPUVA 
> manager:
>
> 1. We would need a new structure to represent the BO+VM combination, 
> something like:
>
>     struct drm_gpuva_mgr_gem {
>             struct drm_gpuva_manager *mgr;
>         struct drm_gem_object *obj;
>         struct list_head gpuva_list;
>     };
>
> with a less horrible name, hopefully.
>
> 2. Create an instance of struct drm_gpuva_mgr_gem once a GEM becomes 
> associated with a GPUVA manager (VM) and attach it to the GEMs, as by 
> now, "gpuva" list.
>
> In amdgpu, for example, this seems to be the case once a GEM object is 
> opened, since there is one VM per file_priv.
>
> However, for other drivers this could be different, hence drivers 
> would need to take care about this.

Yes, exactly that.

>
>
> 3. Attach GPUVAs to the new gpuva_list of the corresponding instance of
> struct drm_gpuva_mgr_gem.
>
> 4. Drivers would need to clean up the instance of struct 
> drm_gpuva_mgr_gem, once the GEM is not associated with the GPUVA 
> manager anymore.
>
> As pointed out by Christian, this would optimize the "get all mappings 
> backed by a specific BO from a given VM" use case.
>
> The question for me is, do other drivers than amdgpu commonly need this?

I have no idea.

>
> And what does amdgpu need this for? Maybe amdgpu does something we're 
> not doing (yet)?

Basically when we do a CS we need to make sure that the VM used by this 
CS is up to date. For this we walk over the relocation list of BOs and 
check the status of each BO+VM structure.

This is done because we don't want to update all VMs at the same time, 
but rather just those who needs the update.

>
> Christian - I know you didn't ask for "do it the way amdgpu does", 
> instead you voted for keeping it entirely driver specific. But I think 
> everyone is pretty close and I'm still optimistic that we could just 
> generalize this.

Well, you should *not* necessarily do it like amdgpu does! Basically the 
implementation in amdgpu was driven by requirements, e.g. we need that, 
let's do it like this.

It's perfectly possible that other requirements (e.g. focus on Vulkan) 
lead to a completely different implementation.

It's just that ideally I would like to have an implementation where I 
can apply at least the basics to amdgpu as well.

Regards,
Christian.

>
> - Danilo
>
>>
>>>>
>>>> I'd think that a single VM potentially has more mapping entries 
>>>> than a single BO was mapped in multiple VMs.
>>>>
>>>> Another case to consider is the case I originally had in mind 
>>>> choosing this relationship: finding all mappings for a given BO, 
>>>> which I guess all drivers need to do in order to invalidate 
>>>> mappings on BO eviction.
>>>>
>>>> Having a list of VMs per BO, wouldn't you need to iterate all of 
>>>> the VMs entirely?
>>
>> No, see how amdgpu works.
>>
>> Regards,
>> Christian.
>>
>