Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <1a620e7c-5b71-3d16-001a-0d79b292aca7@amd.com>
Date:   Fri, 25 Nov 2022 11:42:44 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.4.1
Subject: Re: [PATCH] drm/amdgpu: add mb for si
Content-Language: en-US
To:     "Quan, Evan" <Evan.Quan@amd.com>,
        =?UTF-8?B?5p2O55yf6IO9?= <lizhenneng@kylinos.cn>,
        =?UTF-8?Q?Michel_D=c3=a4nzer?= <michel.daenzer@mailbox.org>,
        "Koenig, Christian" <Christian.Koenig@amd.com>,
        "Deucher, Alexander" <Alexander.Deucher@amd.com>
Cc:     "amd-gfx@lists.freedesktop.org" <amd-gfx@lists.freedesktop.org>,
        "Pan, Xinhui" <Xinhui.Pan@amd.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "dri-devel@lists.freedesktop.org" <dri-devel@lists.freedesktop.org>
References: <20221118074810.380368-1-lizhenneng@kylinos.cn>
 <ecd9d251-8941-b2db-71b2-e4ac06f860a3@amd.com>
 <800a1207-8ff6-4cfa-60f3-6ff456874890@mailbox.org>
 <4adc40d1-e775-c480-bb35-23fe9c63e05e@kylinos.cn>
 <DM6PR12MB26196C35AC2F28FD50ECCAF0E40F9@DM6PR12MB2619.namprd12.prod.outlook.com>
 <4ac1b046-b3bc-6090-f03a-eb6352f52a5a@amd.com>
 <71571bdc-8310-502f-77b5-954f5efbff91@amd.com>
 <DM6PR12MB261978DE4F13F3DB548509C3E40E9@DM6PR12MB2619.namprd12.prod.outlook.com>
From:   "Lazar, Lijo" <lijo.lazar@amd.com>
In-Reply-To: <DM6PR12MB261978DE4F13F3DB548509C3E40E9@DM6PR12MB2619.namprd12.prod.outlook.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NXAzTFJhNGZheHludW4vUVZyUTZ5Nm9yc0dMbzBhNU84YVF5dW95UVN0VWlo?=
 =?utf-8?B?MEl1VXZUcFFMYzVJR2Y5aVBwSHJVT1FpQmUydnpQTmFNem9oSzQ4K2xudXlD?=
 =?utf-8?B?QVo1azIvUkZSZHlxZG5raVMyMFpSU090c3RnZUsyWTQ3RFYvQy9VenZuUk1C?=
 =?utf-8?B?RVpMa0svMWN4RXRyaUN6UWl6ZmtQNjdkNnF0QzMzb2JiaENjcWZ0VG11ZUF6?=
 =?utf-8?B?bE45ckNxKzRDbDc5VXQ2WnhLS2NzSmZoTTJQa0JNbjd1SWFJUHpWK2gwWS9y?=
 =?utf-8?B?dU12Mk55ZDEwWE9Oa1ZGdVlkclo3WTF1VjBwR3hmZmprTVkyTVkxU0E1SXdo?=
 =?utf-8?B?SmJhNkpBWGJpRTJZK3JhQkRzNk5GYWF0c0MxOWZNZkZRU0VJT0R1SUhGZk1k?=
 =?utf-8?B?a3JZeFg5b1ZPZWlwR3IveUYvTlBGVXNZYjAxWExDNVMxU2haQmtEa2UzNzB5?=
 =?utf-8?B?eTA3QWZZUGxicUVMWEZSbjgrSjNCVEJ0S0ZJdkh2Z0NZQy9SVVVHaUh2YitF?=
 =?utf-8?B?ZmlRMFlSVlpXRGh4M3dqbFphQUVSMzg0S0txZDY2RWxWS2FvWDBsdmNjU3Uv?=
 =?utf-8?B?SWhKTkluMWliamp3b2tQUUpwVXFGaGQ0ZHliSSs0QWIwUk9DVk9PWXd3QjJ1?=
 =?utf-8?B?VjlNSUdMWlZkQ2JjRXB3UFg0QVJIYks5dkcwUE9MQVdoWWpoMHVFTDBmYWg0?=
 =?utf-8?B?R3dPcmN2Y0dCT25YOEpsTmV2MVNiNDNIcUNRZUVLM25DdjlZR3dPZDBiTzRP?=
 =?utf-8?B?WW4yMzlPYmNDczk2bFJ1bE5ZUWxEcHFiUGF3b2xlOWJhc0JMdGdlRXlQQ29n?=
 =?utf-8?B?ZGR5YWhJa0lBZ0Q2a3IrSlczU2xWeFdDZHpHVk1keE9yKzZkbEZvdVpwNytB?=
 =?utf-8?B?aUhJUWFMTXpLcms0b2NSY3N5cDVscmMvOHNlaXhNTlRqYXA5dWE1a2xydmpP?=
 =?utf-8?B?M2ZMdi9TalZCa1JjeXk3RVVmanl5WG0vWnFYVjl5NlVaS0VkUjlrbG9FRW9C?=
 =?utf-8?B?YitjTE53VXlKZ0VnOG9EQnBhWGVnM1l1L0JlR0R6V2ppaEVnSkkxVkZxbjZV?=
 =?utf-8?B?M0d4M1BaVEd1OXdKL3U1Y0E0SmpKQnRHSXJrSk1HNzFtZFlLdTJBd3oyQkl1?=
 =?utf-8?B?ZGpSUkZpRUJLVjN6ZnBXZ3JGeVFNV25LbHpXWlNOejRwemVnN1ZoVGlDZmNy?=
 =?utf-8?B?SXZRMEpYdTZqK3IvMkN5VkcyamxZOHo5VEMyQ0YxNHlaSVVtS2ZIRVZORU9P?=
 =?utf-8?B?YmZ5S05aSWkrbStxWUFWZW5VMzRGQkVuTFEwN1lJWnFYVExaVEdoalFQQklt?=
 =?utf-8?B?Y0J2R2lMNVNRblhwTGtTQW5DUzFHQlRGQXdMbjN3ckZzSE1SRml3d24xMGxW?=
 =?utf-8?B?TUZHTE1XMzJqbVkvaFp1YlBjMEs3Nm50SmpZT2JNbWJhd3kwcElRb0syRzh3?=
 =?utf-8?B?OUROdStSUUppTTVOYzAwd20yd25qbGhHbm0rL0VCaG9mYzJEQkN5NXZSN2ti?=
 =?utf-8?B?aFRFS1JGM05zL1FIZ0liMXhGd0MrcU1HSEEwUnh5anZCWTJrTDBkc2cyRWFa?=
 =?utf-8?B?Qy9SejZOV1k4TXdTZHhhVFhnellsQ05udVFMSEExcTFHbXdYZmROQmRrbHFZ?=
 =?utf-8?B?TEN6TWlBTUJSTHVrOUlLYTA4cTgrdVhoT1lISTN1V3Ezd0crUFJLOWtPTk51?=
 =?utf-8?B?Q2thOE4xdWxpeUhqNkEzdi9CMWlCbXFmZTh5NmVTN3AzNkVOWkI1L0lLMmtY?=
 =?utf-8?B?eTZqN1RDbWQ3Q2lyTVhzKy9aQ0RELzY3VlFuSExxVUZrNlRtNjJKdDNtTnZj?=
 =?utf-8?B?dkdqUG11Ulc5NWFWTUFvcGxUWWREekd2dEVEclJyNHZFVG50cGhnZkYzaEEv?=
 =?utf-8?B?NVd6S1BRVTJpWG1uNDRWVlhtaWRQTTNVY1h3eHpjTTJ1RlVIVDVCSFE0VVFk?=
 =?utf-8?B?TjR4TGp3cmpKVklRV3lHSUVUVkQyVmY2UW1jVVAwQ3VXbnc5TFJXaDRBRVpF?=
 =?utf-8?B?Tld0R0pLcFM1Q3ZFMGNJc3ozcDBiRUpsaUJ4b2ZkendyL1lOTjl2VkZSc0kw?=
 =?utf-8?B?c1NCbGVueEpUR1h3RjJQYmt3eE5nVGh0TUVudVppNGNCSGNJaXVoNUlrWG1R?=
 =?utf-8?Q?LRRD6tGaqUUiabVPfOb9xtl/O?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 861fae6d-132e-4a9a-2ca3-08daceac1861
X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB4621.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Nov 2022 06:12:56.8658
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: i+dqE16UpV2vM8ljDSlbNWwQo8A0p24Yruc9bqWRGwW6v7DE+16szIthNeivaVo7
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7335
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 11/25/2022 7:43 AM, Quan, Evan wrote:
> [AMD Official Use Only - General]
>
>
>
>> -----Original Message-----
>> From: Lazar, Lijo <Lijo.Lazar@amd.com>
>> Sent: Thursday, November 24, 2022 6:49 PM
>> To: Quan, Evan <Evan.Quan@amd.com>; 李真能 <lizhenneng@kylinos.cn>;
>> Michel Dänzer <michel.daenzer@mailbox.org>; Koenig, Christian
>> <Christian.Koenig@amd.com>; Deucher, Alexander
>> <Alexander.Deucher@amd.com>
>> Cc: amd-gfx@lists.freedesktop.org; Pan, Xinhui <Xinhui.Pan@amd.com>;
>> linux-kernel@vger.kernel.org; dri-devel@lists.freedesktop.org
>> Subject: Re: [PATCH] drm/amdgpu: add mb for si
>>
>>
>>
>> On 11/24/2022 4:11 PM, Lazar, Lijo wrote:
>>>
>>> On 11/24/2022 3:34 PM, Quan, Evan wrote:
>>>> [AMD Official Use Only - General]
>>>>
>>>> Could the attached patch help?
>>>>
>>>> Evan
>>>>> -----Original Message-----
>>>>> From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf
>> Of ???
>>>>> Sent: Friday, November 18, 2022 5:25 PM
>>>>> To: Michel Dänzer <michel.daenzer@mailbox.org>; Koenig, Christian
>>>>> <Christian.Koenig@amd.com>; Deucher, Alexander
>>>>> <Alexander.Deucher@amd.com>
>>>>> Cc: amd-gfx@lists.freedesktop.org; Pan, Xinhui <Xinhui.Pan@amd.com>;
>>>>> linux-kernel@vger.kernel.org; dri-devel@lists.freedesktop.org
>>>>> Subject: Re: [PATCH] drm/amdgpu: add mb for si
>>>>>
>>>>>
>>>>> 在 2022/11/18 17:18, Michel Dänzer 写道:
>>>>>> On 11/18/22 09:01, Christian König wrote:
>>>>>>> Am 18.11.22 um 08:48 schrieb Zhenneng Li:
>>>>>>>> During reboot test on arm64 platform, it may failure on boot, so
>>>>>>>> add this mb in smc.
>>>>>>>>
>>>>>>>> The error message are as follows:
>>>>>>>> [    6.996395][ 7] [  T295] [drm:amdgpu_device_ip_late_init
>>>>>>>> [amdgpu]] *ERROR*
>>>>>>>>                    late_init of IP block <si_dpm> failed -22 [
>>>>>>>> 7.006919][ 7] [  T295] amdgpu 0000:04:00.0:
>>> The issue is happening in late_init() which eventually does
>>>
>>>       ret = si_thermal_enable_alert(adev, false);
>>>
>>> Just before this, si_thermal_start_thermal_controller is called in
>>> hw_init and that enables thermal alert.
>>>
>>> Maybe the issue is with enable/disable of thermal alerts in quick
>>> succession. Adding a delay inside si_thermal_start_thermal_controller
>>> might help.
>>>
>> On a second look, temperature range is already set as part of
>> si_thermal_start_thermal_controller in hw_init
>> https://elixir.bootlin.com/linux/v6.1-
>> rc6/source/drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c#L6780
>>
>> There is no need to set it again here -
>>
>> https://elixir.bootlin.com/linux/v6.1-
>> rc6/source/drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c#L7635
>>
>> I think it is safe to remove the call from late_init altogether. Alex/Evan?
>>
> [Quan, Evan] Yes, it makes sense to me. But I'm not sure whether that’s related with the issue here.
> Since per my understandings, if the issue is caused by double calling of thermal_alert enablement, it will fail every time.
> That cannot explain why adding some delays or a mb() calling can help.

The side effect of the patch is just some random delay introduced for 
every SMC message

The issue happens in late_init(). Between late_init() and dpm 
enablement, there are many smc messages sent which don't have this 
issue. So I think the issue is not with FW not running.

Thus the only case I see is enable/disable of thermal alert in random 
succession.

Thanks,

Lijo

> BR
> Evan
>> Thanks,
>> Lijo
>>
>>> Thanks,
>>> Lijo
>>>
>>>>>>>> amdgpu_device_ip_late_init failed [    7.014224][ 7] [  T295] amdgpu
>>>>>>>> 0000:04:00.0: Fatal error during GPU init
>>>>>>> Memory barries are not supposed to be sprinkled around like this,
>> you
>>>>> need to give a detailed explanation why this is necessary.
>>>>>>> Regards,
>>>>>>> Christian.
>>>>>>>
>>>>>>>> Signed-off-by: Zhenneng Li <lizhenneng@kylinos.cn>
>>>>>>>> ---
>>>>>>>>      drivers/gpu/drm/amd/pm/legacy-dpm/si_smc.c | 2 ++
>>>>>>>>      1 file changed, 2 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/drivers/gpu/drm/amd/pm/legacy-dpm/si_smc.c
>>>>>>>> b/drivers/gpu/drm/amd/pm/legacy-dpm/si_smc.c
>>>>>>>> index 8f994ffa9cd1..c7656f22278d 100644
>>>>>>>> --- a/drivers/gpu/drm/amd/pm/legacy-dpm/si_smc.c
>>>>>>>> +++ b/drivers/gpu/drm/amd/pm/legacy-dpm/si_smc.c
>>>>>>>> @@ -155,6 +155,8 @@ bool amdgpu_si_is_smc_running(struct
>>>>>>>> amdgpu_device *adev)
>>>>>>>>          u32 rst = RREG32_SMC(SMC_SYSCON_RESET_CNTL);
>>>>>>>>          u32 clk = RREG32_SMC(SMC_SYSCON_CLOCK_CNTL_0);
>>>>>>>>      +    mb();
>>>>>>>> +
>>>>>>>>          if (!(rst & RST_REG) && !(clk & CK_DISABLE))
>>>>>>>>              return true;
>>>>>> In particular, it makes no sense in this specific place, since it
>>>>>> cannot directly
>>>>> affect the values of rst & clk.
>>>>>
>>>>> I thinks so too.
>>>>>
>>>>> But when I do reboot test using nine desktop machines,  there maybe
>>>>> report
>>>>> this error on one or two machines after Hundreds of times or
>>>>> Thousands of
>>>>> times reboot test, at the beginning, I use msleep() instead of mb(),
>>>>> these
>>>>> two methods are all works, but I don't know what is the root case.
>>>>>
>>>>> I use this method on other verdor's oland card, this error message are
>>>>> reported again.
>>>>>
>>>>> What could be the root reason?
>>>>>
>>>>> test environmen:
>>>>>
>>>>> graphics card: OLAND 0x1002:0x6611 0x1642:0x1869 0x87
>>>>>
>>>>> driver: amdgpu
>>>>>
>>>>> os: ubuntu 2004
>>>>>
>>>>> platform: arm64
>>>>>
>>>>> kernel: 5.4.18
>>>>>