Received-SPF: pass (google.com: domain of linux-kernel+bounces-94265-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <32f94f5c-88c6-4e18-8ac3-ff1b80cbd5d0@amd.com>
Date: Wed, 6 Mar 2024 21:56:48 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] drm/amdgpu: cache in more vm fault information
Content-Language: en-US
To: =?UTF-8?Q?Christian_K=C3=B6nig?= <christian.koenig@amd.com>,
 =?UTF-8?Q?Christian_K=C3=B6nig?= <ckoenig.leichtzumerken@gmail.com>,
 Alex Deucher <alexdeucher@gmail.com>
Cc: Sunil Khatri <sunil.khatri@amd.com>,
 Alex Deucher <alexander.deucher@amd.com>,
 Shashank Sharma <shashank.sharma@amd.com>, amd-gfx@lists.freedesktop.org,
 Pan@rtg-sunil-navi33.amd.com, Xinhui <Xinhui.Pan@amd.com>,
 dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org,
 Mukul Joshi <mukul.joshi@amd.com>,
 Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
References: <20240306090408.3453152-1-sunil.khatri@amd.com>
 <2f792620-fd8a-412e-9130-e276ba36d5a0@amd.com>
 <5e2899cd-75b4-4ddd-97ff-4e10a2e67fbb@amd.com>
 <66815303-bd9c-4dfc-ae1a-bbdc5d1bb47c@amd.com>
 <17e12147-79dd-44ba-b8ae-b96fb72dcfbd@amd.com>
 <CADnq5_OkeH1x4YgSv6uw0HLb5c-5NOXnzQPJHsDvb=NmEePB-A@mail.gmail.com>
 <e5781df5-5244-465e-b986-c1802e1262db@gmail.com>
 <0df75ff4-ece5-4eaa-93bd-6f03ec31ecfa@amd.com>
 <bfaaad63-a5d7-4ceb-8e1c-d541f76f4037@amd.com>
 <852e4f0e-c743-44c2-a2bb-59f0e8e25e1b@amd.com>
 <0be0df75-9794-4b7a-a975-a5ea86b7d3f3@amd.com>
From: "Khatri, Sunil" <sukhatri@amd.com>
In-Reply-To: <0be0df75-9794-4b7a-a975-a5ea86b7d3f3@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?NWM4anNJUG1aeXlvNHZMSkh2bUcxcFJqdVJ5Rmlxa3MwL3F3Wm1CUU94Szdz?=
 =?utf-8?B?WXg5UVRjMytTczBlZFhtd0FURjM1TDJZeFJ0YjJ3WHlRU2VHU01IZ3NUbyt6?=
 =?utf-8?B?R3BwcHRxd0R1S0x6NGpOT0IxMGVxK3B1enNPYlpyUFNpbm81Y1FaYVJpMTdW?=
 =?utf-8?B?Zk4vQ1VIRmt6M1kxaHZ5VWsyUExnQ215R0puR3FzMVV2N2paNUV0VkxmNFRr?=
 =?utf-8?B?ZFhwSFlSY3BrS3cvbEgxYWt1eit3UG5rWnltMk1Zcmo2eXNHOS9TNjc4ZlZO?=
 =?utf-8?B?NGZiNExoWEV2SDBhSVNweUZRMGx1MS8xcWxEWm1VVFNiLytWSmR2MkhlSTY5?=
 =?utf-8?B?MlRjZXhaSjBaajNTSUhzdmNOUmRGRG8ycGV4Q0Q4b1dYOG41ckJoMHhOS1Y1?=
 =?utf-8?B?T3EwOHZIM2JCNlBsVElVdWFjc1AyQUthaDJJTDI2N1dpbHZEU1FnTVRnVG81?=
 =?utf-8?B?Q1g2QXBnek9raHJ5N2lySkN3UjRleE41T2w4UkthTm5aanp4dmhuZkhwQlR2?=
 =?utf-8?B?R0k1NEx1K0xwOGNiVFNUdWVxQzJxNXNaNlM4OEZwbzBLbm5YYTA0dStmWVZn?=
 =?utf-8?B?aHNGVTdFandxSXlxMWdrZG1JNlJkSWdpcmlFQStORi9xeDhlRDJ6ajBRdVhm?=
 =?utf-8?B?Nnp4UElFVUVKYTd3NmJIRjIyeklpa3lRUnVVL2NJY3c3Sjh1VlQ4NWdNZi8v?=
 =?utf-8?B?UjVEVFlXOXYzdG1BSG13SjRIaGxiNWM4SmJieHBuNXFOZlpGQjlwOHFRU0Ju?=
 =?utf-8?B?aktISm5JaGt5SnhFQjk1Q0JZQThiS1Z5dURDY25ITFVMbm53dXNhWnFQcWZv?=
 =?utf-8?B?OVhCd3c3SnJkcDdUV0RKQVlYWVYzRW5wdzJxd1B1dEx5MFRUTkRuckVxcmRO?=
 =?utf-8?B?dlVjOXhFVWhHQU9JMG9pck93U0NYU3dUdlhraG13eTRlVVVoRWNoR3hwWlJK?=
 =?utf-8?B?dDNBd0p1ejM4czdRbGRUWnVHSjE3T1dMWk5GQTg5b0M5NnVBWTJ1emZIL0pI?=
 =?utf-8?B?ZkxsL3YxY1FYK3ZKR2Rzc0RsVEczWitZM3BTdUNNQmZNdkZsUFI3OHVRU1Jq?=
 =?utf-8?B?TFJVV1FhZXFMaUR3dkZPQk90cmdYa2MweE5PZjEyTTdqU3FFeFBUbXJ2bWdE?=
 =?utf-8?B?SW9NTzkyM1JlL3BtcFBkL2pQQ2VpcS9wajd6TGNscTJsRVNDZzJMbE4wWGtu?=
 =?utf-8?B?SmZ0MU5wem00Q2FIOVdja2lFdEtaWGhKeWtubHYzL1lWaG05VUx5WjVzc29t?=
 =?utf-8?B?NjZ2bzhKVmxmRmphSURuTmV2VnZ2ZzVWd29aZW9aNjQ5REdxUEZlbmxHWjNo?=
 =?utf-8?B?cGFnaHEvZWxGZ2pBdzBURnJHQ256SnFJYURIUkdZK1Z5WTZuUVNEMzNlbnVU?=
 =?utf-8?B?UFlmYUxYRlRTcGYybEtyTGNmcHBQc2ZhRXNHWWxlL0lSZkMvUXd2VkVJY0g5?=
 =?utf-8?B?TWtXb2w5MG9zbU9XTUdEd0FNdUtjQjB2ak1SWVlHZ2VCODU3ZnlHVGt2T2Fn?=
 =?utf-8?B?WEwyUUNONnVvYnF5dlkrMzlXRjhKWHcrTzlxNFRSc1p5R0ZsVGFMSUVaVzhM?=
 =?utf-8?B?OWQ3Rlp1KzNRaDhPVU1veGJrMnJPRTBRN2crUitka09VanRmbmZ3LzJWKzZD?=
 =?utf-8?B?OFJMY1JESDY5MGZtamVUWlVrZmMzbnZLMEFuU1VGVGN1dCtZRGxER0VuMXc3?=
 =?utf-8?B?NGdpS0UvV1RXQVZ2RS8rTjJWaWJlYWVZcElTZHQxSTVUbUFET3ZrT0hlWW5J?=
 =?utf-8?B?bklsUktxeE00TmRmOFJweVV0Q3VsNHkzQWdJdkI2ZEZpRjRJVEZnaC84UG9r?=
 =?utf-8?B?YUJVaXR5NDU1eWc1Z0hqRFlrUGhoTytxVGtiVFdvVWx2MUlQWklvWlpJUmNB?=
 =?utf-8?B?SE1UYk9mT25hT1J6alR1OEhacFBCSndVN3dGVGZXci80N0g0d2ZWeFMydVIw?=
 =?utf-8?B?dlhPM1JwL2htbFhPdFFhMEFlNzRnWjZ2QnNCdFdPUVliWXF3RTlsakxsOXZl?=
 =?utf-8?B?Qy91Ymk1bWpCbXFETjBNdlJBcHBLNERUVTFhQUh2cGwxSG5lMFlmMU1Sd2tQ?=
 =?utf-8?B?em40ZjdERlViOHk1cTN5cUpYeTRtOHJlQXJKaFV4RkVidUlGMVNYcEpQaWJv?=
 =?utf-8?Q?dP0wBvpmGW1EHu+LWGV42dZxI?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 46167744-8c53-47fc-7100-08dc3dfa3d9e
X-MS-Exchange-CrossTenant-AuthSource: PH7PR12MB5596.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Mar 2024 16:26:56.6117
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: NVOZrSwOkp0Eeu/8rw1W4ONAXwxejH5r7RPBz1kRwyqOqECO4iMIjJ4DB0zN2qyjzYs//stJv7TX5MEYWPUxoA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6174


On 3/6/2024 9:49 PM, Christian König wrote:
> Am 06.03.24 um 17:06 schrieb Khatri, Sunil:
>>
>> On 3/6/2024 9:07 PM, Christian König wrote:
>>> Am 06.03.24 um 16:13 schrieb Khatri, Sunil:
>>>>
>>>> On 3/6/2024 8:34 PM, Christian König wrote:
>>>>> Am 06.03.24 um 15:29 schrieb Alex Deucher:
>>>>>> On Wed, Mar 6, 2024 at 8:04 AM Khatri, Sunil <sukhatri@amd.com> 
>>>>>> wrote:
>>>>>>>
>>>>>>> On 3/6/2024 6:12 PM, Christian König wrote:
>>>>>>>> Am 06.03.24 um 11:40 schrieb Khatri, Sunil:
>>>>>>>>> On 3/6/2024 3:37 PM, Christian König wrote:
>>>>>>>>>> Am 06.03.24 um 10:04 schrieb Sunil Khatri:
>>>>>>>>>>> When an  page fault interrupt is raised there
>>>>>>>>>>> is a lot more information that is useful for
>>>>>>>>>>> developers to analyse the pagefault.
>>>>>>>>>> Well actually those information are not that interesting because
>>>>>>>>>> they are hw generation specific.
>>>>>>>>>>
>>>>>>>>>> You should probably rather use the decoded strings here, e.g. 
>>>>>>>>>> hub,
>>>>>>>>>> client, xcc_id, node_id etc...
>>>>>>>>>>
>>>>>>>>>> See gmc_v9_0_process_interrupt() an example.
>>>>>>>>>> I saw this v9 does provide more information than what v10 and 
>>>>>>>>>> v11
>>>>>>>>>> provide like node_id and fault from which die but thats again 
>>>>>>>>>> very
>>>>>>>>>> specific to IP_VERSION(9, 4, 3)) i dont know why thats 
>>>>>>>>>> information
>>>>>>>>>> is not there in v10 and v11.
>>>>>>>>> I agree to your point but, as of now during a pagefault we are
>>>>>>>>> dumping this information which is useful like which client
>>>>>>>>> has generated an interrupt and for which src and other 
>>>>>>>>> information
>>>>>>>>> like address. So i think to provide the similar information in 
>>>>>>>>> the
>>>>>>>>> devcoredump.
>>>>>>>>>
>>>>>>>>> Currently we do not have all this information from either job 
>>>>>>>>> or vm
>>>>>>>>> being derived from the job during a reset. We surely could add 
>>>>>>>>> more
>>>>>>>>> relevant information later on as per request but this 
>>>>>>>>> information is
>>>>>>>>> useful as
>>>>>>>>> eventually its developers only who would use the dump file 
>>>>>>>>> provided
>>>>>>>>> by customer to debug.
>>>>>>>>>
>>>>>>>>> Below is the information that i dump in devcore and i feel 
>>>>>>>>> that is
>>>>>>>>> good information but new information could be added which 
>>>>>>>>> could be
>>>>>>>>> picked later.
>>>>>>>>>
>>>>>>>>>> Page fault information
>>>>>>>>>> [gfxhub] page fault (src_id:0 ring:24 vmid:3 pasid:32773)
>>>>>>>>>> in page starting at address 0x0000000000000000 from client 
>>>>>>>>>> 0x1b (UTCL2)
>>>>>>>> This is a perfect example what I mean. You record in the patch 
>>>>>>>> is the
>>>>>>>> client_id, but this is is basically meaningless unless you have 
>>>>>>>> access
>>>>>>>> to the AMD internal hw documentation.
>>>>>>>>
>>>>>>>> What you really need is the client in decoded form, in this case
>>>>>>>> UTCL2. You can keep the client_id additionally, but the decoded 
>>>>>>>> client
>>>>>>>> string is mandatory to have I think.
>>>>>>>>
>>>>>>>> Sure i am capturing that information as i am trying to minimise 
>>>>>>>> the
>>>>>>>> memory interaction to minimum as we are still in interrupt context
>>>>>>>> here that why i recorded the integer information compared to 
>>>>>>>> decoding
>>>>>>> and writing strings there itself but to postpone till we dump.
>>>>>>>
>>>>>>> Like decoding to the gfxhub/mmhub based on vmhub/vmid_src and 
>>>>>>> client
>>>>>>> string from client id. So are we good to go with the information 
>>>>>>> with
>>>>>>> the above information of sharing details in devcoredump using the
>>>>>>> additional information from pagefault cached.
>>>>>> I think amdgpu_vm_fault_info() has everything you need already 
>>>>>> (vmhub,
>>>>>> status, and addr).  client_id and src_id are just tokens in the
>>>>>> interrupt cookie so we know which IP to route the interrupt to. We
>>>>>> know what they will be because otherwise we'd be in the interrupt
>>>>>> handler for a different IP.  I don't think ring_id has any useful
>>>>>> information in this context and vmid and pasid are probably not too
>>>>>> useful because they are just tokens to associate the fault with a
>>>>>> process.  It would be better to have the process name.
>>>>
>>>> Just to share context here Alex, i am preparing this for 
>>>> devcoredump, my intention was to replicate the information which in 
>>>> KMD we are sharing in Dmesg for page faults. If assuming we do not 
>>>> add client id specially we would not be able to share enough 
>>>> information in devcoredump.
>>>> It would be just address and hub(gfxhub/mmhub) and i think that is 
>>>> partial information as src id and client id and ip block shares 
>>>> good information.
>>>>
>>>> For process related information we are capturing that information 
>>>> part of dump from existing functionality.
>>>> **** AMDGPU Device Coredump ****
>>>> version: 1
>>>> kernel: 6.7.0-amd-staging-drm-next
>>>> module: amdgpu
>>>> time: 45.084775181
>>>> process_name: soft_recovery_p PID: 1780
>>>>
>>>> Ring timed out details
>>>> IP Type: 0 Ring Name: gfx_0.0.0
>>>>
>>>> Page fault information
>>>> [gfxhub] page fault (src_id:0 ring:24 vmid:3 pasid:32773)
>>>> in page starting at address 0x0000000000000000 from client 0x1b 
>>>> (UTCL2)
>>>> VRAM is lost due to GPU reset!
>>>>
>>>> Regards
>>>> Sunil
>>>>
>>>>>
>>>>> The decoded client name would be really useful I think since the 
>>>>> fault handled is a catch all and handles a whole bunch of 
>>>>> different clients.
>>>>>
>>>>> But that should be ideally passed in as const string instead of 
>>>>> the hw generation specific client_id.
>>>>>
>>>>> As long as it's only a pointer we also don't run into the trouble 
>>>>> that we need to allocate memory for it.
>>>>
>>>> I agree but i prefer adding the client id and decoding it in 
>>>> devcorecump using soc15_ih_clientid_name[fault_info->client_id]) is 
>>>> better else we have to do an sprintf this string to fault_info in 
>>>> irq context which is writing more bytes to memory i guess compared 
>>>> to an integer:)
>>>
>>> Well I totally agree that we shouldn't fiddle to much in the 
>>> interrupt handler, but exactly what you suggest here won't work.
>>>
>>> The client_id is hw generation specific, so the only one who has 
>>> that is the hw generation specific fault handler. Just compare the 
>>> defines here:
>>>
>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c#L83 
>>>
>>>
>>> and here:
>>>
>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c#L38 
>>>
>>>
>> Got your point. Let me see but this is a lot of work in irq context. 
>> Either we can drop totally the client id thing as alex is suggesting 
>> here as its always be same client and src id or let me come up with a 
>> patch and see if its acceptable.
>
> Wait a second, I now realized that you are mixing something up here. 
> As Alex said the src_id and client_id in the IV are always the same, 
> e.g. the VMC or the UTCL2.
>
> This is the client_id which send the IV to IH so that the IH can write 
> it into the ring buffer and we end up in the fault handler.
>
> But additional to that we also have a client_id inside the fault and 
> that is the value printed in the logs. This is the client which caused 
> the fault inside the VMC or UTCL2.
>
Yes the value remains the same irrespective of the family. Client always 
will be VMC/UTCL2 so i think as Alex suggested we can drop this 
information or just add a hardcoded string for information purposes only.
>>
>> Also as Alex pointed we need to decode from status register which 
>> kind of page fault it is (permission, read, write etc) this all is 
>> again family specific and thats all in IRQ context. Not feeling good 
>> about it but let me try to share all that in a new patch.
>
> Yeah, but that is all hw specific. I'm not sure how best to put it 
> into a devcoredump.
>
> Maybe just record the 32bit value and re-design the GMC code to have 
> that decoded into a string for both the system log and the devcoredump.
>
> Alex suggested a good way to just share the value of status register 
> and add family information and let developer use the family/asic id to 
> check the register value and decode it manually.
>
Regards

Sunil.

>
>
>>
>> Regards
>> Sunil.
>>
>>> Regards,
>>> Christian.
>>>
>>>>
>>>> We can argue on values like pasid and vmid and ring id to be taken 
>>>> off if they are totally not useful.
>>>>
>>>> Regards
>>>> Sunil
>>>>
>>>>>
>>>>> Christian.
>>>>>
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>>> regards
>>>>>>> sunil
>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Christian.
>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Sunil Khatri
>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Christian.
>>>>>>>>>>
>>>>>>>>>>> Add all such information in the last cached
>>>>>>>>>>> pagefault from an interrupt handler.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Sunil Khatri <sunil.khatri@amd.com>
>>>>>>>>>>> ---
>>>>>>>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 9 +++++++--
>>>>>>>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 7 ++++++-
>>>>>>>>>>>    drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 2 +-
>>>>>>>>>>>    drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c | 2 +-
>>>>>>>>>>>    drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c  | 2 +-
>>>>>>>>>>>    drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c  | 2 +-
>>>>>>>>>>>    drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c  | 2 +-
>>>>>>>>>>>    7 files changed, 18 insertions(+), 8 deletions(-)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>>>>>>>>> index 4299ce386322..b77e8e28769d 100644
>>>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>>>>>>>>> @@ -2905,7 +2905,7 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>>>>>>>>> amdgpu_vm *vm, struct seq_file *m)
>>>>>>>>>>>     * Cache the fault info for later use by userspace in 
>>>>>>>>>>> debugging.
>>>>>>>>>>>     */
>>>>>>>>>>>    void amdgpu_vm_update_fault_cache(struct amdgpu_device 
>>>>>>>>>>> *adev,
>>>>>>>>>>> -                  unsigned int pasid,
>>>>>>>>>>> +                  struct amdgpu_iv_entry *entry,
>>>>>>>>>>>                      uint64_t addr,
>>>>>>>>>>>                      uint32_t status,
>>>>>>>>>>>                      unsigned int vmhub)
>>>>>>>>>>> @@ -2915,7 +2915,7 @@ void amdgpu_vm_update_fault_cache(struct
>>>>>>>>>>> amdgpu_device *adev,
>>>>>>>>>>> xa_lock_irqsave(&adev->vm_manager.pasids, flags);
>>>>>>>>>>>    -    vm = xa_load(&adev->vm_manager.pasids, pasid);
>>>>>>>>>>> +    vm = xa_load(&adev->vm_manager.pasids, entry->pasid);
>>>>>>>>>>>        /* Don't update the fault cache if status is 0.  In 
>>>>>>>>>>> the multiple
>>>>>>>>>>>         * fault case, subsequent faults will return a 0 
>>>>>>>>>>> status which is
>>>>>>>>>>>         * useless for userspace and replaces the useful fault
>>>>>>>>>>> status, so
>>>>>>>>>>> @@ -2924,6 +2924,11 @@ void amdgpu_vm_update_fault_cache(struct
>>>>>>>>>>> amdgpu_device *adev,
>>>>>>>>>>>        if (vm && status) {
>>>>>>>>>>>            vm->fault_info.addr = addr;
>>>>>>>>>>>            vm->fault_info.status = status;
>>>>>>>>>>> +        vm->fault_info.client_id = entry->client_id;
>>>>>>>>>>> +        vm->fault_info.src_id = entry->src_id;
>>>>>>>>>>> +        vm->fault_info.vmid = entry->vmid;
>>>>>>>>>>> +        vm->fault_info.pasid = entry->pasid;
>>>>>>>>>>> +        vm->fault_info.ring_id = entry->ring_id;
>>>>>>>>>>>            if (AMDGPU_IS_GFXHUB(vmhub)) {
>>>>>>>>>>>                vm->fault_info.vmhub = AMDGPU_VMHUB_TYPE_GFX;
>>>>>>>>>>>                vm->fault_info.vmhub |=
>>>>>>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>>>>>>>> index 047ec1930d12..c7782a89bdb5 100644
>>>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>>>>>>>> @@ -286,6 +286,11 @@ struct amdgpu_vm_fault_info {
>>>>>>>>>>>        uint32_t    status;
>>>>>>>>>>>        /* which vmhub? gfxhub, mmhub, etc. */
>>>>>>>>>>>        unsigned int    vmhub;
>>>>>>>>>>> +    unsigned int    client_id;
>>>>>>>>>>> +    unsigned int    src_id;
>>>>>>>>>>> +    unsigned int    ring_id;
>>>>>>>>>>> +    unsigned int    pasid;
>>>>>>>>>>> +    unsigned int    vmid;
>>>>>>>>>>>    };
>>>>>>>>>>>      struct amdgpu_vm {
>>>>>>>>>>> @@ -605,7 +610,7 @@ static inline void
>>>>>>>>>>> amdgpu_vm_eviction_unlock(struct amdgpu_vm *vm)
>>>>>>>>>>>    }
>>>>>>>>>>>      void amdgpu_vm_update_fault_cache(struct amdgpu_device 
>>>>>>>>>>> *adev,
>>>>>>>>>>> -                  unsigned int pasid,
>>>>>>>>>>> +                  struct amdgpu_iv_entry *entry,
>>>>>>>>>>>                      uint64_t addr,
>>>>>>>>>>>                      uint32_t status,
>>>>>>>>>>>                      unsigned int vmhub);
>>>>>>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
>>>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
>>>>>>>>>>> index d933e19e0cf5..6b177ce8db0e 100644
>>>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
>>>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
>>>>>>>>>>> @@ -150,7 +150,7 @@ static int 
>>>>>>>>>>> gmc_v10_0_process_interrupt(struct
>>>>>>>>>>> amdgpu_device *adev,
>>>>>>>>>>>            status = RREG32(hub->vm_l2_pro_fault_status);
>>>>>>>>>>> WREG32_P(hub->vm_l2_pro_fault_cntl, 1, ~1);
>>>>>>>>>>>    -        amdgpu_vm_update_fault_cache(adev, entry->pasid, 
>>>>>>>>>>> addr,
>>>>>>>>>>> status,
>>>>>>>>>>> +        amdgpu_vm_update_fault_cache(adev, entry, addr, 
>>>>>>>>>>> status,
>>>>>>>>>>>                             entry->vmid_src ? 
>>>>>>>>>>> AMDGPU_MMHUB0(0) :
>>>>>>>>>>> AMDGPU_GFXHUB(0));
>>>>>>>>>>>        }
>>>>>>>>>>>    diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
>>>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
>>>>>>>>>>> index 527dc917e049..bcf254856a3e 100644
>>>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
>>>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
>>>>>>>>>>> @@ -121,7 +121,7 @@ static int 
>>>>>>>>>>> gmc_v11_0_process_interrupt(struct
>>>>>>>>>>> amdgpu_device *adev,
>>>>>>>>>>>            status = RREG32(hub->vm_l2_pro_fault_status);
>>>>>>>>>>> WREG32_P(hub->vm_l2_pro_fault_cntl, 1, ~1);
>>>>>>>>>>>    -        amdgpu_vm_update_fault_cache(adev, entry->pasid, 
>>>>>>>>>>> addr,
>>>>>>>>>>> status,
>>>>>>>>>>> +        amdgpu_vm_update_fault_cache(adev, entry, addr, 
>>>>>>>>>>> status,
>>>>>>>>>>>                             entry->vmid_src ? 
>>>>>>>>>>> AMDGPU_MMHUB0(0) :
>>>>>>>>>>> AMDGPU_GFXHUB(0));
>>>>>>>>>>>        }
>>>>>>>>>>>    diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
>>>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
>>>>>>>>>>> index 3da7b6a2b00d..e9517ebbe1fd 100644
>>>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
>>>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
>>>>>>>>>>> @@ -1270,7 +1270,7 @@ static int 
>>>>>>>>>>> gmc_v7_0_process_interrupt(struct
>>>>>>>>>>> amdgpu_device *adev,
>>>>>>>>>>>        if (!addr && !status)
>>>>>>>>>>>            return 0;
>>>>>>>>>>>    -    amdgpu_vm_update_fault_cache(adev, entry->pasid,
>>>>>>>>>>> +    amdgpu_vm_update_fault_cache(adev, entry,
>>>>>>>>>>>                         ((u64)addr) << AMDGPU_GPU_PAGE_SHIFT,
>>>>>>>>>>> status, AMDGPU_GFXHUB(0));
>>>>>>>>>>>          if (amdgpu_vm_fault_stop == 
>>>>>>>>>>> AMDGPU_VM_FAULT_STOP_FIRST)
>>>>>>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
>>>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
>>>>>>>>>>> index d20e5f20ee31..a271bf832312 100644
>>>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
>>>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
>>>>>>>>>>> @@ -1438,7 +1438,7 @@ static int 
>>>>>>>>>>> gmc_v8_0_process_interrupt(struct
>>>>>>>>>>> amdgpu_device *adev,
>>>>>>>>>>>        if (!addr && !status)
>>>>>>>>>>>            return 0;
>>>>>>>>>>>    -    amdgpu_vm_update_fault_cache(adev, entry->pasid,
>>>>>>>>>>> +    amdgpu_vm_update_fault_cache(adev, entry,
>>>>>>>>>>>                         ((u64)addr) << AMDGPU_GPU_PAGE_SHIFT,
>>>>>>>>>>> status, AMDGPU_GFXHUB(0));
>>>>>>>>>>>          if (amdgpu_vm_fault_stop == 
>>>>>>>>>>> AMDGPU_VM_FAULT_STOP_FIRST)
>>>>>>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>>>>> index 47b63a4ce68b..dc9fb1fb9540 100644
>>>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>>>>> @@ -666,7 +666,7 @@ static int 
>>>>>>>>>>> gmc_v9_0_process_interrupt(struct
>>>>>>>>>>> amdgpu_device *adev,
>>>>>>>>>>>        rw = REG_GET_FIELD(status, 
>>>>>>>>>>> VM_L2_PROTECTION_FAULT_STATUS, RW);
>>>>>>>>>>>        WREG32_P(hub->vm_l2_pro_fault_cntl, 1, ~1);
>>>>>>>>>>>    -    amdgpu_vm_update_fault_cache(adev, entry->pasid, addr,
>>>>>>>>>>> status, vmhub);
>>>>>>>>>>> +    amdgpu_vm_update_fault_cache(adev, entry, addr, status, 
>>>>>>>>>>> vmhub);
>>>>>>>>>>>          dev_err(adev->dev,
>>>>>>>>>>> "VM_L2_PROTECTION_FAULT_STATUS:0x%08X\n",
>>>>>
>>>
>