Hi,
I have created an experimental multimedia scheduling class (SCHED_MM)
for use with multithreaded non-hardcore-realtime multimedia
applications, such as ones based on gstreamer. These typically don't
follow the determinism rules of well behaved SCHED_FIFO applications.
However, these usually handle tasks like VoIP call audio processing,
where somewhat lower scheduling latency is needed to obtain good user
experience. Usually these are not very CPU-heavy and are mostly IO-bound
processes. Thus, something between normal SCHED_OTHER and SCHED_FIFO is
needed in a way where different threads of execution can set different
kinds of scheduling parameters.
I have attached an initial version of my experimental patch for comments...
Best regards,
- Jussi Laako
On Mon, 2008-12-29 at 20:40 +0200, Jussi Laako wrote:
> Hi,
>
> I have created an experimental multimedia scheduling class (SCHED_MM)
> for use with multithreaded non-hardcore-realtime multimedia
> applications, such as ones based on gstreamer. These typically don't
> follow the determinism rules of well behaved SCHED_FIFO applications.
> However, these usually handle tasks like VoIP call audio processing,
> where somewhat lower scheduling latency is needed to obtain good user
> experience. Usually these are not very CPU-heavy and are mostly IO-bound
> processes. Thus, something between normal SCHED_OTHER and SCHED_FIFO is
> needed in a way where different threads of execution can set different
> kinds of scheduling parameters.
>
> I have attached an initial version of my experimental patch for comments...
Sorry, I don't think this is a viable solution for anything.
This is typically the domain of soft-realtime, and as such would need a
realtime scheduling class -- using a proportional class like you did
just doesn't make sense for these tasks.
Eventually we'd hope to provide an sporadic task EDF based scheduler
with hard and soft realtime capabilities, but until that time, FIFO and
RR are the classes to use for anything realtime.
I'm not sure why you think SCHED_FIFO isn't good enough for your
applications, could you elaborate on that?
Peter Zijlstra wrote:
> This is typically the domain of soft-realtime, and as such would need a
> realtime scheduling class -- using a proportional class like you did
> just doesn't make sense for these tasks.
Yes, this is for a soft-realtime usage. Some of the tasks are
CPU-intensive while being realtime'ish, like video codecs and some audio
processing tasks. These audio processing tasks can have some amount of
buffering. The idea behind this patch is to make these tasks overlap
with the normal tasks while giving a slightly more responsive scheduling
behavior and to favor these multimedia tasks over others.
Another option would be to create another scheduler which could overlap
with the normal one, in a way FIFO/RR overlap. After inspecting current
scheduler code I thought that this kind of modification and use of the
same task tree would be feasible, instead of having it's own run queue.
> Eventually we'd hope to provide an sporadic task EDF based scheduler
> with hard and soft realtime capabilities, but until that time, FIFO and
> RR are the classes to use for anything realtime.
Yes, this is also what I thought first and I still believe this would
also be a good addition. After a bit more thinking I concluded that it
might be a bit too hard-realtime'ish for many of the tasks. It would
become rather close to running FIFO with flat priority?
...and these tasks are not sporadic, but strictly periodic...
> I'm not sure why you think SCHED_FIFO isn't good enough for your
> applications, could you elaborate on that?
Actually the biggest problem is that many of the developers are not
comfortable with using SCHED_FIFO for the purpose... :)
FIFO is unsuitable for video codecs, RR would be better there, but still
it would starve rest of the system too much in some cases (it can lose
some frames to keep rest of the system responsive).
Some multimedia software is just not implemented in a way that it would
be safe to run these at RT-priority (having various too
nondeterministic code paths).
- Jussi
Jussi Laako wrote:
> Yes, this is for a soft-realtime usage. Some of the tasks are
> CPU-intensive while being realtime'ish, like video codecs and some audio
> processing tasks. These audio processing tasks can have some amount of
> buffering. The idea behind this patch is to make these tasks overlap
> with the normal tasks while giving a slightly more responsive scheduling
> behavior and to favor these multimedia tasks over others.
I've been running a kernel with the scheduler patch on my x86-64 desktop
since Christmas.
Here are some test results... Workload in these tests is mostly starting
a browser and rendering a set of reasonably heavy pages which creates
CPU load spikes. BKL is a bit of a problem too...
I think this is reasonably good result.
Results for the deskop (x86-64), voluntary preempt enabled (as the
proprietary ATI display driver behaves badly with full preempt):
SCHED_OTHER:
maximum latency: 45098.2 µs
average latency: 366.2 µs
missed timer events: 16
SCHED_MM:
maximum latency: 10051.2 µs
average latency: 44.7 µs
missed timer events: 0
SCHED_FIFO:
maximum latency: 720.3 µs
average latency: 23.1 µs
missed timer events: 0
Results for an OMAP3-based platform, preempt enabled:
SCHED_OTHER:
maximum latency: 44.7 ms
average latency: 1.8 ms
missed timer events: 227
SCHED_MM:
maximum latency: 19.2 ms
average latency: 1.1 ms
missed timer events: 23
SCHED_FIFO:
maximum latency: 10.7 ms
average latency: 0.7 ms
missed timer events: 1
Best regards,
- Jussi Laako
Right, so I totally lost this thread in my inbox -- sorry for that.
On Tue, 2008-12-30 at 10:39 +0200, Jussi Laako wrote:
> > Eventually we'd hope to provide an sporadic task EDF based scheduler
> > with hard and soft realtime capabilities, but until that time, FIFO and
> > RR are the classes to use for anything realtime.
>
> Yes, this is also what I thought first and I still believe this would
> also be a good addition. After a bit more thinking I concluded that it
> might be a bit too hard-realtime'ish for many of the tasks. It would
> become rather close to running FIFO with flat priority?
Not quite so, you can avoid starvation with deadline schedulers, just
limit their budget.
> ....and these tasks are not sporadic, but strictly periodic...
>
> > I'm not sure why you think SCHED_FIFO isn't good enough for your
> > applications, could you elaborate on that?
>
> Actually the biggest problem is that many of the developers are not
> comfortable with using SCHED_FIFO for the purpose... :)
Well, that's not my problem is it ;-), just batter them with a
clue-stick, no need to fudge the kernel for that.
> FIFO is unsuitable for video codecs, RR would be better there, but still
> it would starve rest of the system too much in some cases (it can lose
> some frames to keep rest of the system responsive).
Right, which is where deadline scheduling would be nice. Once you start
running into the budget throttle you know you've got to start dropping
frames in order to keep up.
The proposal is for it to start sending SIGXCPU once it starts
throttling tasks in order to notify them of missed deadlines etc.
> Some multimedia software is just not implemented in a way that it would
> be safe to run these at RT-priority (having various too
> nondeterministic code paths).
Like said, deadline schedulers can help here. You can even dynamically
adjust the parameters -- eg. fall back to half frame rate but double
budget or something.
Peter Zijlstra wrote:
>
> Well, that's not my problem is it ;-), just batter them with a
> clue-stick, no need to fudge the kernel for that.
Sure, been doing that already... :)
There's just sort of a huge gap between rt-schedulers and the normal
scheduler.
> Right, which is where deadline scheduling would be nice. Once you start
> running into the budget throttle you know you've got to start dropping
> frames in order to keep up.
>
> The proposal is for it to start sending SIGXCPU once it starts
> throttling tasks in order to notify them of missed deadlines etc.
For sure this is nice for certain tasks. I'm not entirely convinced if
the average media player or Flash-plugin would or should start using these.
I believe both approaches could co-exist to address different needs.
SCHED_MM is for the average software and is in a sense similar to
SCHED_BATCH and SCHED_IDLE, just targeting different use cases.
Modification is reasonably small and shouldn't have any impact when not
used. Deadline scheduler would probably belong to the
SCHED_RR/SCHED_FIFO family?
> Like said, deadline schedulers can help here. You can even dynamically
> adjust the parameters -- eg. fall back to half frame rate but double
> budget or something.
Managing the budget in an average system will be painful. In a
completely embedded environment this is of course rather straightforward.
Jussi Laako wrote:
> Peter Zijlstra wrote:
>> Well, that's not my problem is it ;-), just batter them with a
>> clue-stick, no need to fudge the kernel for that.
>
> Sure, been doing that already... :)
> There's just sort of a huge gap between rt-schedulers and the normal
> scheduler.
>
>> Right, which is where deadline scheduling would be nice. Once you start
>> running into the budget throttle you know you've got to start dropping
>> frames in order to keep up.
>>
>> The proposal is for it to start sending SIGXCPU once it starts
>> throttling tasks in order to notify them of missed deadlines etc.
>
> For sure this is nice for certain tasks. I'm not entirely convinced if
> the average media player or Flash-plugin would or should start using these.
>
There is never a need for media players to use this.
Media players have time stamps on the displayed frames.
If the timestamp on a frame indicates it has taken too long to decode
it, the media player just skips the frame until it reaches frames that
have non-expired time stamps. No need for any kernel help here.
James Courtier-Dutton wrote:
>> For sure this is nice for certain tasks. I'm not entirely convinced if
>> the average media player or Flash-plugin would or should start using these.
>
> There is never a need for media players to use this.
> Media players have time stamps on the displayed frames.
> If the timestamp on a frame indicates it has taken too long to decode
> it, the media player just skips the frame until it reaches frames that
> have non-expired time stamps. No need for any kernel help here.
This is completely irrelevant. These media players still play audio and
sync video to audio. Many of these players are not programmed in a way
that it would be safe to run these on SCHED_FIFO. Or the environment
these are running in is not safe enough. But still smooth video and
audio playback is needed, even in cases when locate database is being
rebuilt in the background and possibly other CPU and IO intensive tasks
are running. Any skipped frames make the video playback look jumpy, if
frames are lost, it should be single frame periodically, not burst of
frames at once...
Good everyday normal example is HD video played from Youtube or similar
site using Flash plugin inside browser. There can be various background
tasks running at the same time, but the video playback should still be
smooth. One may want to run thread doing video decoding at significantly
lower priority than audio decoding thread in order to maintain overall
system responsiveness in cases of high CPU load from the video decoding
part. While the audio thread shouldn't starve or miss it's deadline.
On Mon, 2009-01-26 at 01:09 +0200, Jussi Laako wrote:
> James Courtier-Dutton wrote:
> >> For sure this is nice for certain tasks. I'm not entirely convinced if
> >> the average media player or Flash-plugin would or should start using these.
> >
> > There is never a need for media players to use this.
> > Media players have time stamps on the displayed frames.
> > If the timestamp on a frame indicates it has taken too long to decode
> > it, the media player just skips the frame until it reaches frames that
> > have non-expired time stamps. No need for any kernel help here.
>
> This is completely irrelevant. These media players still play audio and
> sync video to audio. Many of these players are not programmed in a way
> that it would be safe to run these on SCHED_FIFO. Or the environment
> these are running in is not safe enough. But still smooth video and
> audio playback is needed, even in cases when locate database is being
> rebuilt in the background and possibly other CPU and IO intensive tasks
> are running. Any skipped frames make the video playback look jumpy, if
> frames are lost, it should be single frame periodically, not burst of
> frames at once...
>
> Good everyday normal example is HD video played from Youtube or similar
> site using Flash plugin inside browser. There can be various background
> tasks running at the same time, but the video playback should still be
> smooth. One may want to run thread doing video decoding at significantly
> lower priority than audio decoding thread in order to maintain overall
> system responsiveness in cases of high CPU load from the video decoding
> part. While the audio thread shouldn't starve or miss it's deadline.
Right, and I think the solution to this problem is twofold, 1)
application writers should start writing (soft) realtime applications if
they want (soft) realtime behaviour -- there's just no way around that.
And 2), the kernel can help by providing a deadline based scheduler,
which should make the above easier and less likely to mess up the rest
of the system. ie. a deadline scheduled application will not exceed its
allotted budget, unlike a FIFO scheduled app.