Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH] sched/deadline: Unthrottle PI boosted threads while
 enqueuing
To:     Juri Lelli <juri.lelli@redhat.com>
Cc:     Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Mark Simmons <msimmons@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        linux-kernel@vger.kernel.org
References: <5076e003450835ec74e6fa5917d02c4fa41687e6.1600170294.git.bristot@redhat.com>
 <20200918060026.GC261845@localhost.localdomain>
From:   Daniel Bristot de Oliveira <bristot@redhat.com>
Message-ID: <f4316f5a-eccd-ec96-3f7b-85900f5f3124@redhat.com>
Date:   Fri, 2 Oct 2020 17:57:52 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.11.0
MIME-Version: 1.0
In-Reply-To: <20200918060026.GC261845@localhost.localdomain>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 9/18/20 8:00 AM, Juri Lelli wrote:
> Hi Daniel,
> 
> On 16/09/20 09:06, Daniel Bristot de Oliveira wrote:
>> stress-ng has a test (stress-ng --cyclic) that creates a set of threads
>> under SCHED_DEADLINE with the following parameters:
>>
>>     dl_runtime   =  10000 (10 us)
>>     dl_deadline  = 100000 (100 us)
>>     dl_period    = 100000 (100 us)
>>
>> These parameters are very aggressive. When using a system without HRTICK
>> set, these threads can easily execute longer than the dl_runtime because
>> the throttling happens with 1/HZ resolution.
>>
>> During the main part of the test, the system works just fine because
>> the workload does not try to run over the 10 us. The problem happens at
>> the end of the test, on the exit() path. During exit(), the threads need
>> to do some cleanups that require real-time mutex locks, mainly those
>> related to memory management, resulting in this scenario:
>>
>> Note: locks are rt_mutexes...
>>  ------------------------------------------------------------------------
>>     TASK A:		TASK B:				TASK C:
>>     activation
>> 							activation
>> 			activation
>>
>>     lock(a): OK!	lock(b): OK!
>>     			<overrun runtime>
>>     			lock(a)
>>     			-> block (task A owns it)
>> 			  -> self notice/set throttled
>>  +--<			  -> arm replenished timer
>>  |    			switch-out
>>  |    							lock(b)
>>  |    							-> <C prio > B prio>
>>  |    							-> boost TASK B
>>  |  unlock(a)						switch-out
>>  |  -> handle lock a to B
>>  |    -> wakeup(B)
>>  |      -> B is throttled:
>>  |        -> do not enqueue
>>  |     switch-out
>>  |
>>  |
>>  +---------------------> replenishment timer
>> 			-> TASK B is boosted:
>> 			  -> do not enqueue
>>  ------------------------------------------------------------------------
>>
>> BOOM: TASK B is runnable but !enqueued, holding TASK C: the system
>> crashes with hung task C.
>>
>> This problem is avoided by removing the throttle state from the boosted
>> thread while boosting it (by TASK A in the example above), allowing it to
>> be queued and run boosted.
>>
>> The next replenishment will take care of the runtime overrun, pushing
>> the deadline further away. See the "while (dl_se->runtime <= 0)" on
>> replenish_dl_entity() for more information.
>>
>> Signed-off-by: Daniel Bristot de Oliveira <bristot@redhat.com>
>> Reported-by: Mark Simmons <msimmons@redhat.com>
>> Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
>> Tested-by: Mark Simmons <msimmons@redhat.com>
>> Cc: Ingo Molnar <mingo@redhat.com>
>> Cc: Peter Zijlstra <peterz@infradead.org>
>> Cc: Juri Lelli <juri.lelli@redhat.com>
>> Cc: Vincent Guittot <vincent.guittot@linaro.org>
>> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
>> Cc: Steven Rostedt <rostedt@goodmis.org>
>> Cc: Ben Segall <bsegall@google.com>
>> Cc: Mel Gorman <mgorman@suse.de>
>> Cc: Daniel Bristot de Oliveira <bristot@redhat.com>
>> Cc: linux-kernel@vger.kernel.org
>>
>> ---
> 
> Thanks for this fix.
> 
> Acked-by: Juri Lelli <juri.lelli@redhat.com>

This is a gentle ping... [we are facing this bug in practice :-(].

-- Daniel

> Best,
> Juri
>