Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Song Liu <songliubraving@fb.com>
To:     Vincent Guittot <vincent.guittot@linaro.org>,
        Morten Rasmussen <morten.rasmussen@arm.com>
CC:     linux-kernel <linux-kernel@vger.kernel.org>,
        "cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "peterz@infradead.org" <peterz@infradead.org>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        Kernel Team <Kernel-team@fb.com>
Subject: Re: [PATCH 0/7] introduce cpu.headroom knob to cpu controller
Thread-Topic: [PATCH 0/7] introduce cpu.headroom knob to cpu controller
Thread-Index: AQHU7lSD68FtcB4UGUOYupgC9AfmOKY1TPCAgACBxACACo6VgIAIioaAgAkx7AA=
Date:   Sun, 28 Apr 2019 19:47:24 +0000
Message-ID: <F0A127DD-F9B6-4FBE-B9AD-8E8B00A7D676@fb.com>
References: <20190408214539.2705660-1-songliubraving@fb.com>
 <20190410115907.GE19434@e105550-lin.cambridge.arm.com>
 <A2E9A149-9EAA-478D-A096-1D4D4BA442B3@fb.com>
 <CAKfTPtAFB3gSZYJN1BcjU_XoY=Pfu2xtea+2MEw7FkVc3mwTSA@mail.gmail.com>
 <E97E73F4-CE8C-4CD7-B6B6-5F63A4E881B1@fb.com>
In-Reply-To: <E97E73F4-CE8C-4CD7-B6B6-5F63A4E881B1@fb.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: fb.com does not designate
 permitted sender hosts)
Content-Type: text/plain; charset="us-ascii"
Content-ID: <E274646D4D5DC84AB9DB7A9F308DA3DF@namprd15.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: c0ebbb4f-9fd8-4965-8fcc-08d6cc12560d
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Apr 2019 19:47:24.3880
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR15MB1527
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-28_13:,,
 signatures=0
X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501
 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0
 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0
 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx
 scancount=1 engine=8.0.1-1810050000 definitions=main-1904280144
X-FB-Internal: deliver
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Morten and Vincent,

> On Apr 22, 2019, at 6:22 PM, Song Liu <songliubraving@fb.com> wrote:
>=20
> Hi Vincent,
>=20
>> On Apr 17, 2019, at 5:56 AM, Vincent Guittot <vincent.guittot@linaro.org=
> wrote:
>>=20
>> On Wed, 10 Apr 2019 at 21:43, Song Liu <songliubraving@fb.com> wrote:
>>>=20
>>> Hi Morten,
>>>=20
>>>> On Apr 10, 2019, at 4:59 AM, Morten Rasmussen <morten.rasmussen@arm.co=
m> wrote:
>>>>=20
>>=20
>>>>=20
>>>> The bit that isn't clear to me, is _why_ adding idle cycles helps your
>>>> workload. I'm not convinced that adding headroom gives any latency
>>>> improvements beyond watering down the impact of your side jobs. AFAIK,
>>>=20
>>> We think the latency improvements actually come from watering down the
>>> impact of side jobs. It is not just statistically improving average
>>> latency numbers, but also reduces resource contention caused by the sid=
e
>>> workload. I don't know whether it is from reducing contention of ALUs,
>>> memory bandwidth, CPU caches, or something else, but we saw reduced
>>> latencies when headroom is used.
>>>=20
>>>> the throttling mechanism effectively removes the throttled tasks from
>>>> the schedule according to a specific duty cycle. When the side job is
>>>> not throttled the main workload is experiencing the same latency issue=
s
>>>> as before, but by dynamically tuning the side job throttling you can
>>>> achieve a better average latency. Am I missing something?
>>>>=20
>>>> Have you looked at your distribution of main job latency and tried to
>>>> compare with when throttling is active/not active?
>>>=20
>>> cfs_bandwidth adjusts allowed runtime for each task_group each period
>>> (configurable, 100ms by default). cpu.headroom logic applies gentle
>>> throttling, so that the side workload gets some runtime in every period=
.
>>> Therefore, if we look at time window equal to or bigger than 100ms, we
>>> don't really see "throttling active time" vs. "throttling inactive time=
".
>>>=20
>>>>=20
>>>> I'm wondering if the headroom solution is really the right solution fo=
r
>>>> your use-case or if what you are really after is something which is
>>>> lower priority than just setting the weight to 1. Something that
>>>=20
>>> The experiments show that, cpu.weight does proper work for priority: th=
e
>>> main workload gets priority to use the CPU; while the side workload onl=
y
>>> fill the idle CPU. However, this is not sufficient, as the side workloa=
d
>>> creates big enough contention to impact the main workload.
>>>=20
>>>> (nearly) always gets pre-empted by your main job (SCHED_BATCH and
>>>> SCHED_IDLE might not be enough). If your main job consist
>>>> of lots of relatively short wake-ups things like the min_granularity
>>>> could have significant latency impact.
>>>=20
>>> cpu.headroom gives benefits in addition to optimizations in pre-empt
>>> side. By maintaining some idle time, fewer pre-empt actions are
>>> necessary, thus the main workload will get better latency.
>>=20
>> I agree with Morten's proposal, SCHED_IDLE should help your latency
>> problem because side job will be directly preempted unlike normal cfs
>> task even lowest priority.
>> In addition to min_granularity, sched_period also has an impact on the
>> time that a task has to wait before preempting the running task. Also,
>> some sched_feature like GENTLE_FAIR_SLEEPERS can also impact the
>> latency of a task.
>>=20
>> It would be nice to know if the latency problem comes from contention
>> on cache resources or if it's mainly because you main load waits
>> before running on a CPU
>>=20
>> Regards,
>> Vincent
>=20
> Thanks for these suggestions. Here are some more tests to show the impact=
=20
> of scheduler knobs and cpu.headroom.
>=20
> side-load | cpu.headroom | side/cpu.weight | min_gran | cpu-idle | main/l=
atency
> -------------------------------------------------------------------------=
-------
>  none    |      0       |     n/a         |    1 ms  |  45.20%  |   1.00
> ffmpeg   |      0       |      1          |   10 ms  |   3.38%  |   1.46
> ffmpeg   |      0       |   SCHED_IDLE    |    1 ms  |   5.69%  |   1.42
> ffmpeg   |    20%       |   SCHED_IDLE    |    1 ms  |  19.00%  |   1.13
> ffmpeg   |    30%       |   SCHED_IDLE    |    1 ms  |  27.60%  |   1.08
>=20
> In all these cases, the main workload is loaded with same level of=20
> traffic (request per second). Main workload latency numbers are normalize=
d=20
> based on the baseline (first row).=20
>=20
> For the baseline, the main workload runs without any side workload, the=20
> system has about 45.20% idle CPU.=20
>=20
> The next two rows compare the impact of scheduling knobs cpu.weight and=20
> sched_min_granularity. With cpu.weight of 1 and min_granularity of 10ms,=
=20
> we see a latency of 1.46; with SCHED_IDLE and min_granularity of 1ms, we=
=20
> see a latency of 1.42. So SCHED_IDLE and min_granularity help protecting=
=20
> the main workload. However, it is not sufficient, as the latency overhead=
=20
> is high (>40%).=20
>=20
> The last two rows show the benefit of cpu.headroom. With 20% headroom,=20
> the latency is 1.13; while with 30% headroom, the latency is 1.08.=20
>=20
> We can also see a clear correlation between latency and global idle CPU:=
=20
> more idle CPU yields better lower latency.=20
>=20
> Over all, these results show that cpu.headroom provides effective=20
> mechanism to control the latency impact of side workloads. Other knobs=20
> could also help the latency, but they are not as effective and flexible=20
> as cpu.headroom.=20
>=20
> Does this analysis address your concern?=20
>=20
> Thanks,
> Song
>=20

Could you please share your comments and suggestions on this work? Did
the results address your questions/concerns?=20

Thanks again,
Song=20

>>=20
>>>=20
>>> Thanks,
>>> Song
>>>=20
>>>>=20
>>>> Morten