Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Message-ID: <9581428d-13ba-114c-7594-c2357a8ccf11@bytedance.com>
Date:   Tue, 12 Sep 2023 10:35:46 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.15.0
Subject: Re: [External] Re: [PATCH 0/2] Fix nohz_full vs rt bandwidth
To:     Phil Auld <pauld@redhat.com>
Cc:     mingo@redhat.com, peterz@infradead.org, mingo@kernel.org,
        juri.lelli@redhat.com, vincent.guittot@linaro.org,
        dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com,
        mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com,
        linux-kernel@vger.kernel.org
References: <20230821094927.51079-1-jiahao.os@bytedance.com>
 <1823b6fd-037f-38dc-2d33-0879d77768c3@bytedance.com>
 <20230907141703.GA441901@lorien.usersys.redhat.com>
 <bd786914-4718-7a35-fcc1-9cdee58d2f06@bytedance.com>
 <20230908124501.GA471894@lorien.usersys.redhat.com>
 <a353150b-1382-9e76-e14f-4170c956b8ce@bytedance.com>
 <20230911132535.GA24480@lorien.usersys.redhat.com>
From:   Hao Jia <jiahao.os@bytedance.com>
In-Reply-To: <20230911132535.GA24480@lorien.usersys.redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 2023/9/11 Phil Auld wrote:
> 
> Hi Hao,
> 
> On Mon, Sep 11, 2023 at 11:39:02AM +0800 Hao Jia wrote:
>> On 2023/9/8 Phil Auld wrote:
>>> On Fri, Sep 08, 2023 at 10:57:26AM +0800 Hao Jia wrote:
>>>> On 2023/9/7 Phil Auld wrote:
>>>>> Hi Hao,
> 
> ...
> 
>>>>>
>>>>> Are you actually hitting this in the real world?
>>>>>
>>>>> We, for example, no longer enable RT_GROUP_SCHED so this is a non-issue
>>>>> for our use cases.  I'd recommend considering that. (Does it even
>>>>> work with cgroup2?)
>>>>>
>>>>
>>>> Yes, it has always been there. Regardless of whether RT_GROUP_SCHED is
>>>> enabled or not, rt bandwidth is always enabled. If RT_GROUP_SCHED is not
>>>> enabled, all rt tasks in the system are a group, and rt_runtime is 950000,
>>>> and rt_period is 1000000.So rt bandwidth is always enabled by default.
>>>
>>> Sure, there is that. But I think Daniel is actively trying to remove it.
>>>
>>
>> Thank you for your reply. Maybe I'm missing something. Can you give me some
>> links to discussions about it?
>>
> 
> Sure, try this one:
>        https://lore.kernel.org/lkml/cover.1693510979.git.bristot@kernel.org/
> 

Thanks for the information you shared.

> 
>>> Also I'm not sure you answered my question. Are you actually hitting this
>>> in the real world?  I'd be tempted to think this is a mis-configuration or
>>> mis-use of RT.  Plus you can disable that throttling and use stalld to catch
>>> cases where the rt task goes out of control.
>>>
>>
>>> Are you actually hitting this in the real world?
>>
>> I tested on my machine using default settings (rt_runtime is 950000, and
>> rt_period is 1000000.). The rt task is supposed to be throttled after
>> running for 0.95 seconds, but due to the influence of NO_HZ_FULL, it may be
>> throttled after running for about 1.4 seconds. This will only cause the
>> rt_bandwidth throttle to be delayed, but no warning will be triggered.
> 
> Yes, you can hit this in testing.  I'm asking if it's causing your real-world
> applicaton issues or is this just a theoretical problem you can contrive a
> test for?  Are you actually hitting this when running your workload?
>  From what you are showing (a test setup) I'm guessing no.
> 

Yes, I don't see this issue in our production environment. The number of 
rt tasks is very small in our production environment, and their running 
time is very short, so the rt_bandwidth throttle will not be triggered 
unless the rt task goes out of control.

Thanks,
Hao

>>
>>
>>> Plus you can disable that throttling and use stalld to catch cases where
>> the rt task goes out of control.
>>
>> IIRC, if we disable rt_bandwidth. The rt task is always running, which may
>> cause cfs task starvation and hung_task warnning. This may be the reason why
>> rt_bandwidth is enabled by default (rt_runtime is 950000, and rt_period is
>> 1000000).
> 
> That's what stalld is for.  Some rt applications don't like giving up 5% of
> the cpu time when they don't really need to.
> 
> 
> Cheers,
> Phil
> 
>