Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Wed, 20 Nov 2019 09:50:24 +0100
From:   Juri Lelli <juri.lelli@redhat.com>
To:     Philipp Stanner <stanner@posteo.de>
Cc:     linux-kernel@vger.kernel.org, Hagen Pfeifer <hagen@jauu.net>,
        mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org,
        dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com,
        mgorman@suse.de
Subject: Re: SCHED_DEADLINE with CPU affinity
Message-ID: <20191120085024.GB23227@localhost.localdomain>
References: <1574202052.1931.17.camel@posteo.de>
MIME-Version: 1.0
In-Reply-To: <1574202052.1931.17.camel@posteo.de>
User-Agent: Mutt/1.11.3 (2019-02-01)
Content-Type: text/plain; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi Philipp,

On 19/11/19 23:20, Philipp Stanner wrote:
> Hey folks,
> (please put me in CC when answering, I'm not subscribed)
>=20
> I'm currently working student in the embedded industry. We have a device =
where
> we need to be able to process network data within a certain deadline. At =
the
> same time, safety is a primary requirement; that's why we construct every=
thing
> fully redundant. Meaning: We have two network interfaces, each IRQ then b=
ound
> to one CPU core and spawn a container (systemd-nspawn, cgroups based) whi=
ch in
> turn is bound to the corresponding CPU (CPU affinity masked).
>=20
> =A0=A0=A0=A0=A0=A0=A0=A0Container0=A0=A0=A0=A0=A0=A0=A0Container1
> =A0=A0=A0-----------------=A0=A0-----------------
> =A0=A0=A0|=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0|=A0=A0|=A0=A0=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0|
> =A0=A0=A0|=A0=A0=A0=A0Proc. A=A0=A0=A0=A0|=A0=A0|=A0=A0=A0Proc. A'=A0=A0=
=A0=A0|
> =A0=A0=A0|=A0=A0=A0=A0Proc. B=A0=A0=A0=A0|=A0=A0|=A0=A0=A0Proc. B'=A0=A0=
=A0=A0|
> =A0=A0=A0|=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0|=A0=A0|=A0=A0=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0|
> =A0=A0=A0-----------------=A0=A0-----------------
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0^=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0^
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0|=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0|
> =A0=A0=A0=A0=A0=A0=A0=A0CPU 0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0CP=
U 1
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0|=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0|
> =A0=A0=A0=A0=A0=A0=A0IRQ eth0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0IRQ eth1
>=20
>=20
> Within each container several processes are started. Ranging from systemd
> (SCHED_OTHER) till two (soft) real-time critical processes: which we want=
 to
> execute via SCHED_DEADLINE.
>=20
> Now, I've worked through the manpage describing scheduling policies, and =
it
> seems that our scenario is forbidden my the kernel.=A0=A0I've done some t=
ests with
> the syscalls sched_setattr and sched_setaffinity, trying to activate
> SCHED_DEADLINE while also binding to a certain core.=A0=A0It fails with E=
INVAL or
> EINBUSY, depending on the order of the syscalls.
>=20
> I've read that the kernel accomplishes plausibility checks when you ask f=
or a

Yeah, admission control.

> new deadline task to be scheduled, and I assume this check is what preven=
ts us
> from implementing our intended architecture.
>=20
> Now, the questions we're having are:
>=20
> =A0=A0=A01. Why does the kernel do this, what is the problem with schedul=
ing with
> =A0=A0=A0=A0=A0=A0SCHED_DEADLINE on a certain core? In contrast, how is i=
t handled when
> =A0=A0=A0=A0=A0=A0you have single core systems etc.? Why this artificial =
limitation?

Please have also a look (you only mentioned manpage so, in case you
missed it) at

https://elixir.bootlin.com/linux/latest/source/Documentation/scheduler/sche=
d-deadline.rst#L667

and the document in general should hopefully give you the answer about
why we need admission control and current limitations regarding
affinities.

> =A0=A0=A02. How can we possibly implement this? We don't want to use SCHE=
D_FIFO,
> =A0=A0=A0=A0=A0=A0because out-of-control tasks would freeze the entire co=
ntainer.

I experimented myself a bit with this kind of setup in the past and I
think I made it work by pre-configuring exclusive cpusets (similarly as
what detailed in the doc above) and then starting containers inside such
exclusive sets with podman run --cgroup-parent option.

I don't have proper instructions yet for how to do this (plan to put
them together soon-ish), but please see if you can make it work with
this hint.

Best,

Juri