Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH 2/3] drm/scheduler: Don't call wait_event_killable for
 signaled process.
To:     christian.koenig@amd.com, Oleg Nesterov <oleg@redhat.com>
Cc:     David.Panariti@amd.com, linux-kernel@vger.kernel.org,
        amd-gfx@lists.freedesktop.org,
        "Eric W. Biederman" <ebiederm@xmission.com>,
        Alexander.Deucher@amd.com, akpm@linux-foundation.org
References: <87muxsbmkp.fsf@xmission.com>
 <8840ac96-50c4-f94d-eb7c-f007940163f3@amd.com> <877eowa5qh.fsf@xmission.com>
 <20180425135552.GD7592@redhat.com>
 <f45c7017-c75a-7f0c-b6d5-4d79600cdd33@amd.com>
 <20180425171757.GA10441@redhat.com>
 <c39cb438-1e8a-4f2c-50f6-c6e2a144487b@amd.com> <874ljyu98e.fsf@xmission.com>
 <c3c9787d-b279-8169-43d1-74eeb666ffbd@gmail.com>
 <bceb1a1b-c453-782d-5a7d-40fa2f22c813@amd.com>
 <20180430160006.GB10583@redhat.com>
 <e5b0221d-84ba-10ff-4a58-4fa27c99650f@amd.com>
 <bb224134-7ccb-cc87-9a71-3ef1743eb074@gmail.com>
From:   Andrey Grodzovsky <Andrey.Grodzovsky@amd.com>
Message-ID: <79b2ce10-2cd7-b6f2-551e-0b4ae21072af@amd.com>
Date:   Mon, 30 Apr 2018 15:28:42 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.4.0
MIME-Version: 1.0
In-Reply-To: <bb224134-7ccb-cc87-9a71-3ef1743eb074@gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Received-SPF: None (protection.outlook.com: amd.com does not designate
 permitted sender hosts)
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2018 19:28:48.3859 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 17fb842f-b758-480f-2212-08d5aed09946
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR12MB0313
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 04/30/2018 02:29 PM, Christian König wrote:
> Am 30.04.2018 um 18:10 schrieb Andrey Grodzovsky:
>>
>>
>> On 04/30/2018 12:00 PM, Oleg Nesterov wrote:
>>> On 04/30, Andrey Grodzovsky wrote:
>>>> What about changing PF_SIGNALED to PF_EXITING in
>>>> drm_sched_entity_do_release
>>>>
>>>> -       if ((current->flags & PF_SIGNALED) && current->exit_code == 
>>>> SIGKILL)
>>>> +      if ((current->flags & PF_EXITING) && current->exit_code == 
>>>> SIGKILL)
>>> let me repeat, please don't use task->exit_code. And in fact this 
>>> check is racy
>>>
>>> But this doesn't matter. Say, we can trivially add 
>>> SIGNAL_GROUP_KILLED_BY_SIGKILL,
>>> or do something else,
>>
>>
>> Can you explain where is the race and what is a possible alternative 
>> then ?
>
> The race is that the release doesn't necessarily comes from the 
> process/context which used the fd.
>
> E.g. it is just called when the last reference count goes away, but 
> that can be anywhere not related to the original process using it, 
> e.g. in a kernel thread or a debugger etc...

I still don't see how it is a problem, if release comes from another 
task, then our process  (let's say Firefox who received SIGKILL) won't 
even get here since fput will not call .release so it will die instantly,
the last process who holds the reference (let's say the debugger) when 
finish will just go to wait_event_timeout and wait for SW queue to be 
empty from jobs (if any). So all the jobs will have their chance to get 
to HW anyway.

>
> The approach with the flush is indeed a really nice idea and I bite 
> myself to not had that previously as well.

Regarding your request from another email to investigate more on .flush

Looked at the code and did some reading -

 From LDD3
"The flush operation is invoked when a process closes its copy of a file 
descriptor for a device; it should execute (and wait for) any 
outstanding operations on the device"

 From printing back trace from dummy .flush hook in our driver -

Normal exit (process terminates on it's own)

[  295.586130 <    0.000006>]  dump_stack+0x5c/0x78
[  295.586273 <    0.000143>]  my_flush+0xa/0x10 [amdgpu]
[  295.586283 <    0.000010>]  filp_close+0x4a/0x90
[  295.586288 <    0.000005>]  SyS_close+0x2d/0x60
[  295.586295 <    0.000003>]  do_syscall_64+0xee/0x270

Exit triggered by fatal signal (not handled  signal, including SIGKILL)

[  356.551456 <    0.000008>]  dump_stack+0x5c/0x78
[  356.551592 <    0.000136>]  my_flush+0xa/0x10 [amdgpu]
[  356.551597 <    0.000005>]  filp_close+0x4a/0x90
[  356.551605 <    0.000008>]  put_files_struct+0xaf/0x120
[  356.551615 <    0.000010>]  do_exit+0x468/0x1280
[  356.551669 <    0.000009>]  do_group_exit+0x89/0x140
[  356.551679 <    0.000010>]  get_signal+0x375/0x8f0
[  356.551696 <    0.000017>]  do_signal+0x79/0xaa0
[  356.551756 <    0.000014>]  exit_to_usermode_loop+0x83/0xd0
[  356.551764 <    0.000008>]  do_syscall_64+0x244/0x270

So as it was said here before, it will be called for every process 
closing his FD to the file.

But again, I don't quire see yet what we earn by using .flush, is it 
that you force every process holding reference to DRM file not
die until all jobs are submitted to HW (as long as the process not being 
killed by  a signal) ?

Andrey

>
> Christian.

The idea here is that any task still referencing this file and putting 
down the reference and is not
exiting due to SIGKILL will just have to go through the  slow path - 
wait for jobs completion on GPU (with some TO).
>
>>
>>>   but I fail to understand what are you trying to do. Suppose
>>> that the check above is correct in that it is true iff the task is 
>>> exiting and
>>> it was killed by SIGKILL. What about the "else" branch which does
>>>
>>>     r = wait_event_killable(sched->job_scheduled, ...)
>>>
>>> ?
>>>
>>> Once again, fatal_signal_pending() (or even signal_pending()) is not 
>>> well defined
>>> after the exiting task passes exit_signals().
>>>
>>> So wait_event_killable() can fail because fatal_signal_pending() is 
>>> true; and this
>>> can happen even if it was not killed.
>>>
>>> Or it can block and SIGKILL won't be able to wake it up.
>>>
>>>> If SIGINT was sent then it's SIGINT,
>>> Yes, but see above. in this case fatal_signal_pending() will be 
>>> likely true so
>>> wait_event_killable() will fail unless condition is already true.
>>
>> My bad, I didn't show the full intended fix, it was just a snippet to 
>> address the differentiation between exiting
>> do to SIGKILL and any other exit, I also intended to change 
>> wait_event_killable to wait_event_timeout.
>>
>> Andrey
>>
>>>
>>> Oleg.
>>>
>>
>> _______________________________________________
>> amd-gfx mailing list
>> amd-gfx@lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>