Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <d6f082eb-8948-8dde-6813-371cf6b1c7a3@amd.com>
Date:   Fri, 7 Oct 2022 08:11:26 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.11.0
Subject: Re: [git pull] drm for 6.1-rc1
Content-Language: en-US
To:     Dave Airlie <airlied@gmail.com>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Arvind.Yadav@amd.com
Cc:     Alex Deucher <alexdeucher@gmail.com>,
        Alex Deucher <alexander.deucher@amd.com>,
        Daniel Vetter <daniel.vetter@ffwll.ch>,
        LKML <linux-kernel@vger.kernel.org>,
        dri-devel <dri-devel@lists.freedesktop.org>
References: <CAPM=9tzs4n8dDQ_XVVPS_5jrBgsNkhDQvf-B_XmUg+EG_M2i4Q@mail.gmail.com>
 <CAHk-=whUp5Ufur6Bmv=H-rDDTNJJ-KVqOKkL+5FmR01jp0dbcA@mail.gmail.com>
 <CADnq5_Of-8ZyBxee0fZ=0x-eV-2NX_+e9sd-9nmuHdLugSHp2g@mail.gmail.com>
 <CAHk-=wi43xD06UgO2McDT3R=ze_aHgOGjcDOoggSwmQRv2kA+A@mail.gmail.com>
 <CADnq5_N0Ef+1VUoDLdpHfJXqZFuPYbx5Lq+94NWciHcQC+VrMg@mail.gmail.com>
 <CAPM=9tyAOnzwXyyPuoceZ1mimAkzxR-63YmKfVtcQX=swywNvg@mail.gmail.com>
 <CAHk-=wgghR4N-4XWjoK18NDkvjBL7i00ab8+otQg955pNGG_dQ@mail.gmail.com>
 <CAPM=9txE+0EH2Tv_0toDD52j0JO7iDZoJap6qmvMAnRaDRwJNg@mail.gmail.com>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
In-Reply-To: <CAPM=9txE+0EH2Tv_0toDD52j0JO7iDZoJap6qmvMAnRaDRwJNg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZHJ0elp0Yy84eEVRWWNIWTN5TERrSDFLb3A5VWZudHdyekF4TGlKdURuclRo?=
 =?utf-8?B?bTFJenFIYjI5MVJqWExkQ1ZTZzdVRnFQRVQ3RCtiZFcvWmNTdEp3c2Vydi9R?=
 =?utf-8?B?RjlydWhyM3NyMGhETE5Ua2o4S2l2Umtick83V21GdFdwVTNRbmM0RGwwblVN?=
 =?utf-8?B?ZjE2YnBpWWJLZ2huL2dIMWplQ3AwY2xTRlc1N3NRcW51Y0s5Y05pQWdydlU0?=
 =?utf-8?B?SlJhWFJJZEVscnhEMUFTTGlxaS9iS0xTSUE0SHJFVm5La0tMTjRjNDBpdDlE?=
 =?utf-8?B?VC9RM3d4anpCTGM3VmpCZlJGbWQ2ZjBLSzI0d1lzc0FhemdSWE9NZ2d3ZmUv?=
 =?utf-8?B?bDVsYTMvd0t4QjRudDljeGFXUVZFKzU0eWtwMURNVFZVMGVPakJWMnBGaXRU?=
 =?utf-8?B?K1NURE10cWsyR1N5TnovdW42RHdMOWZrcWtzZFdadFlJRHI2SVIwdWNZd3hJ?=
 =?utf-8?B?LzRGNWt4VFJqOEZhZnB5eHlQRmxpLzFZUnVMUFJLeHZNcWtobUluelorZ1F2?=
 =?utf-8?B?cDNqb0RjdGVncE80eWEwektML3M0dDVpUXZ5akN1cVdGNFZ1djYwMFpmeUVO?=
 =?utf-8?B?bXJCUWViYUl2eXptRkRRY3AxSDY4L0lqNnI0dHJMZzE0SDFQMkpvVGtnWXp3?=
 =?utf-8?B?Qno0T2lBQklBTTh4QzdBU21KREZ2YWt0QmxTc3V5RkF2bVVjMGVlWm9qUXFt?=
 =?utf-8?B?MUpVZmJmU1JxdG9DVS9MRDdSOHRxZnF2M3JZTm1GUG8yTStiU0xCeVg0aTBt?=
 =?utf-8?B?TlBwMFhDbkZkek9iRkFjeVg4TVJ3c1dpNWNTdllOMDEyN09jZGVVVDdhbkRC?=
 =?utf-8?B?R3Y4TlFEbW1uMlVMM2REeXlscHhTMGlkS2drSzVNQjJHMVovd3kzdGs0TGIy?=
 =?utf-8?B?THZSNmlNaU9sUW9JVm9sR2xRWm5vZ2VjdkhXWjBvOTc4RFRXOEorcVJVWWlx?=
 =?utf-8?B?clhWKzIvWWgxWVFidTBCM3pJZkYwRmQweG1aekxtRUJGL1VGR3Bpc3BIb3JO?=
 =?utf-8?B?MXlOem96aE1jVk9JSXV4ajIvK1ZFMDZ0bjJQY2JNS3pta2dFbzUxRVBHT3Rj?=
 =?utf-8?B?VC9scnJ3K0RuNmNFVGpLK2ZMYm5Pd09tVUwvc1c2Q1JnVFBXWk5Bbk1GdmpV?=
 =?utf-8?B?RUtoOUVCbWJLQkhSSXFzdGVwZFBQT0RXSkVCRHhXQndzVmVydGtBckR4b3d3?=
 =?utf-8?B?eHcwOTBoS3R3aUh6ZU9pRW1PbXBXc1RUSm1VZW1PSVRldTZ0cE1aK1BwSGVB?=
 =?utf-8?B?aVl4SHhCKzNxdzl2eWNqT0c3TWNEdGxoNEtrU2ZnUEswb08yanpWeC91N1dx?=
 =?utf-8?B?bVRlR1FyN3djNmRHTDBOUjdSWkloaU0rRW9vaUFZc2NkVmxIT1pBYndseU9G?=
 =?utf-8?B?elR0WkUyM0JRWGIwcGFxUXJlNzM3a0ZmNDFqT05RWGphTTQxdFJpUFVhUCsz?=
 =?utf-8?B?NGF4SUtBSzc0WkwzNXB1NVZzRHNHSWVOeUE4aXdMWWhkSnBjOHh3WEsxeDhm?=
 =?utf-8?B?MHJyY3ljM2FEYThXKzdTbUc4ekkwaGMrUEhMbXcyWVB2NFZpWWEzTkVqQ1RQ?=
 =?utf-8?B?YzdZaU1kbWw5YVRhRUhGYndyTjhhYTE5aXNwd0ZiRGE0M2I3QjV3aTFYZTdB?=
 =?utf-8?B?TCtycXF6T3lIVE40VHNRTHBzTjNxK0dMbWt1WHAxaFg1UVd5eC9JNW0wd0xH?=
 =?utf-8?B?dWI1cVlKYVdPK09VOHRDSmFVdkF2Q1NPY3o1TmI1VG9TODlVL2hNdFBJblg3?=
 =?utf-8?B?R2o4QTJGeU52aXkzSU5DRzA2SVZtM0s1ajhycXR6WGd2d2ZzRWxJYklDa1Jl?=
 =?utf-8?B?VkpEM1Rnc2h5RkRXODFzeXJNQXIzZzluS0FFcVVXV242anJKMmpKTW9hNmtm?=
 =?utf-8?B?YTRuckI5ck1XVStNaDFFSEc0M04wOEYvZlc4TXNYSzhIZUFFaVVhQ0VjVSs2?=
 =?utf-8?B?ZGY3MjhVanJKZWlkUkJRVncyeU81OTBQdW5uMWJDaGhUVHhkQUlCTlpxOXFz?=
 =?utf-8?B?VGNKVEZwdWRlZFBDRDdiNEIzUDgwZWxuYXQ1dFRjdWdkUnVEV1F2L2NMWWFX?=
 =?utf-8?B?YUNWa0pTbjFqbDl4d2ZZb0tyRTZWOHJ2c3dxWStKUlhBQkpOYUlxS0NadXNZ?=
 =?utf-8?B?U0hnM1kzY2RRY0QwRUZMSGVmckVlTGJkOVZscFVSdW1QRlAvMGRlUHF0R3lr?=
 =?utf-8?Q?rfc9HnKHCcRROvxHUJGoEctKa7xkZoCBbuoAlokRm+mv?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8e688b5f-e2df-40be-b43c-08daa82ac6dc
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3587.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2022 06:11:30.7916
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FblHLhk1PMpd/4HYR8m3jWsO0BOeb+0UlQxCCmllruT/+Ej4THwfLK76Q+KT0CIc
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7626
X-Spam-Status: No, score=-3.5 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Am 07.10.22 um 04:45 schrieb Dave Airlie:
> On Fri, 7 Oct 2022 at 09:45, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
>> On Thu, Oct 6, 2022 at 1:25 PM Dave Airlie <airlied@gmail.com> wrote:
>>>
>>> [ 1234.778760] BUG: kernel NULL pointer dereference, address: 0000000000000088
>>> [ 1234.778813] RIP: 0010:drm_sched_job_done.isra.0+0xc/0x140 [gpu_sched]
>> As far as I can tell, that's the line
>>
>>          struct drm_gpu_scheduler *sched = s_fence->sched;
>>
>> where 's_fence' is NULL. The code is
>>
>>     0: 0f 1f 44 00 00        nopl   0x0(%rax,%rax,1)
>>     5: 41 54                push   %r12
>>     7: 55                    push   %rbp
>>     8: 53                    push   %rbx
>>     9: 48 89 fb              mov    %rdi,%rbx
>>     c:* 48 8b af 88 00 00 00 mov    0x88(%rdi),%rbp <-- trapping instruction
>>    13: f0 ff 8d f0 00 00 00 lock decl 0xf0(%rbp)
>>    1a: 48 8b 85 80 01 00 00 mov    0x180(%rbp),%rax
>>
>> and that next 'lock decl' instruction would have been the
>>
>>          atomic_dec(&sched->hw_rq_count);
>>
>> at the top of drm_sched_job_done().
>>
>> Now, as to *why* you'd have a NULL s_fence, it would seem that
>> drm_sched_job_cleanup() was called with an active job. Looking at that
>> code, it does
>>
>>          if (kref_read(&job->s_fence->finished.refcount)) {
>>                  /* drm_sched_job_arm() has been called */
>>                  dma_fence_put(&job->s_fence->finished);
>>          ...
>>
>> but then it does
>>
>>          job->s_fence = NULL;
>>
>> anyway, despite the job still being active. The logic of that kind of
>> "fake refcount" escapes me. The above looks fundamentally racy, not to
>> say pointless and wrong (a refcount is a _count_, not a flag, so there
>> could be multiple references to it, what says that you can just
>> decrement one of them and say "I'm done").
>>
>> Now, _why_ any of that happens, I have no idea. I'm just looking at
>> the immediate "that pointer is NULL" thing, and reacting to what looks
>> like a completely bogus refcount pattern.
>>
>> But that odd refcount pattern isn't new, so it's presumably some user
>> on the amd gpu side that changed.
>>
>> The problem hasn't happened again for me, but that's not saying a lot,
>> since it was very random to begin with.
> I chased down the culprit to a drm sched patch, I'll send you a pull
> with a revert in it.
>
> commit e4dc45b1848bc6bcac31eb1b4ccdd7f6718b3c86
> Author: Arvind Yadav <Arvind.Yadav@amd.com>
> Date:   Wed Sep 14 22:13:20 2022 +0530
>
>      drm/sched: Use parent fence instead of finished
>
>      Using the parent fence instead of the finished fence
>      to get the job status. This change is to avoid GPU
>      scheduler timeout error which can cause GPU reset.
>
>      Signed-off-by: Arvind Yadav <Arvind.Yadav@amd.com>
>      Reviewed-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
>      Link: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.freedesktop.org%2Fpatch%2Fmsgid%2F20220914164321.2156-6-Arvind.Yadav%40amd.com&amp;data=05%7C01%7Cchristian.koenig%40amd.com%7C516db37183e84489e1aa08daa80e087e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638007075495101336%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=JWT8R205jIPQu87K7a1T0UJ0iKNO8smHhosijAA0%2BNk%3D&amp;reserved=0
>      Signed-off-by: Christian König <christian.koenig@amd.com>
>
> I'll let Arvind and Christian maybe work out what is going wrong there.

That's a known issue Arvind is already investigating for a while.

Any idea how you triggered it on boot? We have only be able to trigger 
it very sporadic.

Reverting the patch for now sounds like a good idea to me, it's only a 
cleanup anyway.

Thanks,
Christian.

>
> Dave.
>
>>                   Linus