2005-01-18 13:03:07

by Con Kolivas

[permalink] [raw]
Subject: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

This patch for 2.6.11-rc1 provides a method of providing real time
scheduling to unprivileged users which increasingly is desired for
multimedia workloads.

It does this by adding a new scheduling class called SCHED_ISO or
Isochronous scheduling which means "same time" scheduling. This class
does not require superuser privileges and is starvation free. The
scheduling class no. 4 was chosen since there are quite a few userspace
applications already supporting 3 and 4 for SCHED_BATCH and SCHED_ISO
respectively on non-mainline kernels. As a way of immediately providing
support for current userspace apps, any unprivileged user starting an
application requesting SCHED_RR or SCHED_FIFO will be demoted to
SCHED_ISO. This may or may not be worth removing later.

The SCHED_ISO class runs as SCHED_RR effectively at a priority just
above all SCHED_NORMAL tasks and below all true real time tasks. Once a
cpu usage limit is exceeded by tasks of this class (per cpu), SCHED_ISO
tasks will then run as SCHED_NORMAL until the cpu usage drops to 90% of
the cpu limit.

By default the cpu limit is set to 70% which literature suggests should
provide good real time behaviour for most applications without gross
unfairness. This cpu limit is calculated as a decaying average over 5
seconds. These limits are configurable with the tunables
/proc/sys/kernel/iso_cpu
/proc/sys/kernel/iso_period

iso_cpu can be set to 100 which would allow all unprivileged users
access to unrestricted SCHED_RR behaviour. OSX provides a similar class
to SCHED_ISO and uses 90% as its cpu limit.

The sysrq-n combination which converts all user real-time tasks to
SCHED_NORMAL also will affect SCHED_ISO tasks.

Currently the round robin interval is set to 10ms which is a cache
friendly timeslice. It may be worth making this configurable or smaller,
and it would also be possible to implement SCHED_ISO of a FIFO nature as
well.

For testing, the userspace tool schedtool available here:
http://freequaos.host.sk/schedtool/
can be used as a wrapper to start SCHED_ISO tasks
schedtool -I -e xmms
for example

Patch also available here:
http://ck.kolivas.org/patches/SCHED_ISO/

Comments and testing welcome.

Signed-off-by: Con Kolivas <[email protected]>


Attachments:
2.6.11-rc1-iso-0501182249.diff (15.85 kB)
signature.asc (256.00 B)
OpenPGP digital signature
Download all attachments

2005-01-18 14:54:03

by Con Kolivas

[permalink] [raw]
Subject: Re: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Con Kolivas wrote:
> This patch for 2.6.11-rc1 provides a method of providing real time
> scheduling to unprivileged users which increasingly is desired for
> multimedia workloads.

I should have mentioned. Many thanks to Alex Nyberg for generous
debugging help.

Cheers,
Con


Attachments:
signature.asc (256.00 B)
OpenPGP digital signature

2005-01-18 15:47:41

by Cal

[permalink] [raw]
Subject: Re: [ck] [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Con Kolivas wrote:
> Comments and testing welcome.

There's a collection of test summaries from jack_test3.2 runs at
<http://www.graggrag.com/ck-tests/ck-tests-0501182249.txt>

Tests were run with iso_cpu at 70, 90, 99, 100, each test was run twice.
The discrepancies between consecutive runs (with same parameters) is
puzzling. Also recorded were tests with SCHED_FIFO and SCHED_RR.

Before drawing any hardball conclusions, verification of the results
would be nice. At first glance, it does seem that we still have that
fateful gap between "harm minimisation" (policy) and "zero tolerance"
(audio reality requirement).

cheers, Cal

2005-01-18 15:55:55

by Con Kolivas

[permalink] [raw]
Subject: Re: [ck] [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Cal wrote:
> Con Kolivas wrote:
>
>> Comments and testing welcome.
>
>
> There's a collection of test summaries from jack_test3.2 runs at
> <http://www.graggrag.com/ck-tests/ck-tests-0501182249.txt>
>
> Tests were run with iso_cpu at 70, 90, 99, 100, each test was run twice.
> The discrepancies between consecutive runs (with same parameters) is
> puzzling. Also recorded were tests with SCHED_FIFO and SCHED_RR.
>
> Before drawing any hardball conclusions, verification of the results
> would be nice. At first glance, it does seem that we still have that
> fateful gap between "harm minimisation" (policy) and "zero tolerance"
> (audio reality requirement).

Thanks.

SCHED_ISO
/proc/sys/kernel/iso_cpu . . .: 70
/proc/sys/kernel/iso_period . : 5
XRUN Count . . . . . . . . . : 110

vs

SCHED_FIFO
XRUN Count . . . . . . . . . : 114
XRUN Count . . . . . . . . . : 187

vs

SCHED_RR
XRUN Count . . . . . . . . . : 0
XRUN Count . . . . . . . . . : 0

Something funny going on here... You had more xruns with SCHED_FIFO than
the default SCHED_ISO settings, and had none with SCHED_RR. Even in the
absence of the SCHED_ISO results, the other results dont make a lot of
sense.

Con

P.S. If you're running on SMP it may be worth booting on UP or using cpu
affinity (schedtool -a 0x1 will bind you to 1st cpu only) and see what
effect that is having. There are some interesting things that can
adversely affect latency on SMP.


Attachments:
signature.asc (256.00 B)
OpenPGP digital signature

2005-01-18 16:16:39

by Jack O'Quin

[permalink] [raw]
Subject: Re: [ck] [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Cal <[email protected]> writes:

> There's a collection of test summaries from jack_test3.2 runs at
> <http://www.graggrag.com/ck-tests/ck-tests-0501182249.txt>
>
> Tests were run with iso_cpu at 70, 90, 99, 100, each test was run
> twice. The discrepancies between consecutive runs (with same
> parameters) is puzzling. Also recorded were tests with SCHED_FIFO and
> SCHED_RR.

It's probably suffering from some of the same problems of thread
granularity we saw running nice --20. It looks like you used
schedtool to start jackd. IIUC, that will cause all jackd processes
to run in the specified scheduling class. JACK is carefully written
not to do that. Did you also use schedtool to start all the clients?

I think your puzzling discrepancies are probably due to interference
from non-realtime JACK threads running at elevated priority.

> Before drawing any hardball conclusions, verification of the results
> would be nice. At first glance, it does seem that we still have that
> fateful gap between "harm minimisation" (policy) and "zero tolerance"
> (audio reality requirement).

I still have not found time to run Con's latest version. I've been
too busy trying to hack JACK into working with nice(-20), which has
turned out to be quite difficult. The interfaces we need don't work,
and the ones that do are not what we need. :-(

I believe Con's latest sched patch should work fine with the normal
jack_test3.2 using the -R option of jackd. That should produce more
consistent results.
--
joq

2005-01-18 16:22:03

by Jack O'Quin

[permalink] [raw]
Subject: Re: [ck] [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Con Kolivas <[email protected]> writes:

> Cal wrote:
>
> SCHED_ISO
> /proc/sys/kernel/iso_cpu . . .: 70
> /proc/sys/kernel/iso_period . : 5
> XRUN Count . . . . . . . . . : 110
>
> vs
>
> SCHED_FIFO
> XRUN Count . . . . . . . . . : 114
> XRUN Count . . . . . . . . . : 187
>
> vs
>
> SCHED_RR
> XRUN Count . . . . . . . . . : 0
> XRUN Count . . . . . . . . . : 0
>
> Something funny going on here... You had more xruns with SCHED_FIFO
> than the default SCHED_ISO settings, and had none with SCHED_RR. Even
> in the absence of the SCHED_ISO results, the other results dont make a
> lot of sense.

Actually it makes perfect sense. Running non-realtime JACK threads
SCHED_FIFO will do the most harm. The others less.

I predict that using normal jackd -R (without schedtool) will produce
the same results running SCHED_FIFO and SCHED_ISO (within the normal
variance).

I think schedtool is too blunt and instrument for making these
measurements.
--
joq

2005-01-19 02:02:08

by Lee Revell

[permalink] [raw]
Subject: Re: [ck] [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

On Tue, 2005-01-18 at 10:17 -0600, Jack O'Quin wrote:
> Cal <[email protected]> writes:
>
> > There's a collection of test summaries from jack_test3.2 runs at
> > <http://www.graggrag.com/ck-tests/ck-tests-0501182249.txt>
> >
> > Tests were run with iso_cpu at 70, 90, 99, 100, each test was run
> > twice. The discrepancies between consecutive runs (with same
> > parameters) is puzzling. Also recorded were tests with SCHED_FIFO and
> > SCHED_RR.
>
> It's probably suffering from some of the same problems of thread
> granularity we saw running nice --20. It looks like you used
> schedtool to start jackd. IIUC, that will cause all jackd processes
> to run in the specified scheduling class. JACK is carefully written
> not to do that. Did you also use schedtool to start all the clients?
>
> I think your puzzling discrepancies are probably due to interference
> from non-realtime JACK threads running at elevated priority.

Isn't this going to be a showstopper? If I understand the scheduler
correctly, a nice -20 task is not guaranteed to preempt a nice -19 task,
if the scheduler decides that one is more CPU bound than the other and
lowers its dynamic priority. The design of JACK, however, requires the
higher priority threads to *always* preempt the lower ones.

Lee

2005-01-19 02:07:26

by Con Kolivas

[permalink] [raw]
Subject: Re: [ck] [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Lee Revell wrote:
> On Tue, 2005-01-18 at 10:17 -0600, Jack O'Quin wrote:
>
>>Cal <[email protected]> writes:
>>
>>
>>>There's a collection of test summaries from jack_test3.2 runs at
>>><http://www.graggrag.com/ck-tests/ck-tests-0501182249.txt>
>>>
>>>Tests were run with iso_cpu at 70, 90, 99, 100, each test was run
>>>twice. The discrepancies between consecutive runs (with same
>>>parameters) is puzzling. Also recorded were tests with SCHED_FIFO and
>>>SCHED_RR.
>>
>>It's probably suffering from some of the same problems of thread
>>granularity we saw running nice --20. It looks like you used
>>schedtool to start jackd. IIUC, that will cause all jackd processes
>>to run in the specified scheduling class. JACK is carefully written
>>not to do that. Did you also use schedtool to start all the clients?
>>
>>I think your puzzling discrepancies are probably due to interference
>>from non-realtime JACK threads running at elevated priority.
>
>
> Isn't this going to be a showstopper? If I understand the scheduler
> correctly, a nice -20 task is not guaranteed to preempt a nice -19 task,
> if the scheduler decides that one is more CPU bound than the other and
> lowers its dynamic priority. The design of JACK, however, requires the
> higher priority threads to *always* preempt the lower ones.

The point was the application was started in a manner which would not
make best use of this policy. The way it was started put everything
under the same policy, and had equal performance with doing the same
thing as SCHED_FIFO. So if it's a showstopper for SCHED_ISO then it is a
showstopper for SCHED_FIFO. Which is, of course, not the case. The test
needs to be performed again with the rt threads running SCHED_ISO, which
Jack has pointed out is trivial. Nice -n -20 on the other hand will
suffer from this problem even if only the real time thread was run at -20.

Cheers,
Con

2005-01-19 05:29:18

by utz

[permalink] [raw]
Subject: Re: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Hi Con

I just played with your SCHED_ISO patch and found a simple way to crash
my machine.

I'm running this as unprivileged user with SCHED_ISO:

main ()
{
while(1) {
sched_yield();
}
}

The system hangs about 3s and then reboots itself.
2.6.11-rc1 + 2.6.11-rc1-iso-0501182249 on an UP Athlon XP.

With real SCHED_FIFO it just lockup the system.

utz


2005-01-19 05:30:53

by Con Kolivas

[permalink] [raw]
Subject: Re: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

utz wrote:
> Hi Con
>
> I just played with your SCHED_ISO patch and found a simple way to crash
> my machine.
>
> I'm running this as unprivileged user with SCHED_ISO:
>
> main ()
> {
> while(1) {
> sched_yield();
> }
> }
>
> The system hangs about 3s and then reboots itself.
> 2.6.11-rc1 + 2.6.11-rc1-iso-0501182249 on an UP Athlon XP.
>
> With real SCHED_FIFO it just lockup the system.

Thanks I'll look into it.

Cheers,
Con

2005-01-19 06:53:17

by Jack O'Quin

[permalink] [raw]
Subject: Re: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Con Kolivas <[email protected]> writes:

> This patch for 2.6.11-rc1 provides a method of providing real time
> scheduling to unprivileged users which increasingly is desired for
> multimedia workloads.

I ran some jack_test3.2 runs with this, using all the default
settings. The results of three runs differ quite significantly for no
obvious reason. I can't figure out why the DSP load should vary so
much.

These may be bogus results. It looks like a libjack bug sometimes
causes clients to crash when deactivating. I will investigate more
tomorrow, and come up with a fix.

For comparison, I also made a couple of runs using the realtime-lsm to
grant SCHED_FIFO privileges. There was some variablility, but nowhere
near as much (and no crashes). I used schedtool to verify that the
jackd threads actually have the expected scheduler type.

============================================
Unprivileged, realtime threads are SCHED_ISO
============================================

*** Terminated Tue Jan 18 23:54:55 CST 2005 ***
************* SUMMARY RESULT ****************
Total seconds ran . . . . . . : 300
Number of clients . . . . . . : 20
Ports per client . . . . . . : 4
Frames per buffer . . . . . . : 64
*********************************************
Timeout Count . . . . . . . . :( 3) ( 14) ( 2)
XRUN Count . . . . . . . . . : 10 42 3
Delay Count (>spare time) . . : 1 0 0
Delay Count (>1000 usecs) . . : 0 0 0
Delay Maximum . . . . . . . . : 307419 usecs 6492 usecs 19339 usecs
Cycle Maximum . . . . . . . . : 858 usecs 866 usecs 860 usecs
Average DSP Load. . . . . . . : 37.3 % 14.5 % 37.7 %
Average CPU System Load . . . : 10.2 % 4.5 % 10.0 %
Average CPU User Load . . . . : 26.6 % 11.4 % 23.8 %
Average CPU Nice Load . . . . : 0.0 % 0.0 % 0.0 %
Average CPU I/O Wait Load . . : 2.0 % 0.7 % 0.2 %
Average CPU IRQ Load . . . . : 0.8 % 0.7 % 0.7 %
Average CPU Soft-IRQ Load . . : 0.0 % 0.0 % 0.0 %
Average Interrupt Rate . . . : 1730.3 /sec 1695.5 /sec 1694.8 /sec
Average Context-Switch Rate . : 11523.1 /sec 6151.1 /sec 11672.2 /sec
*********************************************


==================================================
With CAP_SYS_NICE, realtime threads are SCHED_FIFO
==================================================

*** Terminated Tue Jan 18 23:41:42 CST 2005 ***
************* SUMMARY RESULT ****************
Total seconds ran . . . . . . : 300
Number of clients . . . . . . : 20
Ports per client . . . . . . : 4
Frames per buffer . . . . . . : 64
*********************************************
Timeout Count . . . . . . . . :( 0) ( 0)
XRUN Count . . . . . . . . . : 0 0
Delay Count (>spare time) . . : 0 0
Delay Count (>1000 usecs) . . : 0 0
Delay Maximum . . . . . . . . : 331 usecs 201 usecs
Cycle Maximum . . . . . . . . : 882 usecs 1017 usecs
Average DSP Load. . . . . . . : 40.7 % 41.7 %
Average CPU System Load . . . : 11.1 % 10.9 %
Average CPU User Load . . . . : 26.7 % 27.7 %
Average CPU Nice Load . . . . : 0.0 % 0.0 %
Average CPU I/O Wait Load . . : 0.6 % 1.0 %
Average CPU IRQ Load . . . . : 0.7 % 0.7 %
Average CPU Soft-IRQ Load . . : 0.0 % 0.0 %
Average Interrupt Rate . . . : 1708.0 /sec 1697.1 /sec
Average Context-Switch Rate . : 13297.0 /sec 13314.8 /sec
*********************************************

--
joq

2005-01-19 08:43:26

by Con Kolivas

[permalink] [raw]
Subject: Re: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Jack O'Quin wrote:
> Con Kolivas <[email protected]> writes:
>
>
>>This patch for 2.6.11-rc1 provides a method of providing real time
>>scheduling to unprivileged users which increasingly is desired for
>>multimedia workloads.
>
>
> I ran some jack_test3.2 runs with this, using all the default
> settings. The results of three runs differ quite significantly for no
> obvious reason. I can't figure out why the DSP load should vary so
> much.
>
> These may be bogus results. It looks like a libjack bug sometimes
> causes clients to crash when deactivating. I will investigate more
> tomorrow, and come up with a fix.
>
> For comparison, I also made a couple of runs using the realtime-lsm to
> grant SCHED_FIFO privileges. There was some variablility, but nowhere
> near as much (and no crashes). I used schedtool to verify that the
> jackd threads actually have the expected scheduler type.

Thanks for those. If you don't know what to make of the dsp variation
and the crashing then I'm not sure what I should make of it either. It's
highly likely that my code still needs fixing to ensure it behaves as
expected. Already one bug has been picked up in testing with respect to
yield() so there may be others. By design, if you set iso_cpu to 100 it
should be as good as SCHED_RR. If not, then the implementation is still
buggy.

Cheers,
Con


Attachments:
signature.asc (256.00 B)
OpenPGP digital signature

2005-01-19 09:38:31

by Con Kolivas

[permalink] [raw]
Subject: Re: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Jack O'Quin wrote:
> Con Kolivas <[email protected]> writes:
>
>
>>This patch for 2.6.11-rc1 provides a method of providing real time
>>scheduling to unprivileged users which increasingly is desired for
>>multimedia workloads.
>
>
> I ran some jack_test3.2 runs with this, using all the default
> settings. The results of three runs differ quite significantly for no
> obvious reason. I can't figure out why the DSP load should vary so
> much.

I installed a fresh jack installation and got the test suite. I tried
running the test suite and found it only ran to completion if I changed
the run time right down to 30 seconds from 300. Otherwise it bombed out
almost instantly at the default of 300. I don't know if that helps you
debug the problem or not but it might be worth mentioning.

As for my own results I gave it a run on the weak SCHED_ISO
implementation in 2.6.10-ck5 (P4HT 3.06):

SCHED_NORMAL:
*********************************************
Timeout Count . . . . . . . . :( 0)
XRUN Count . . . . . . . . . : 74
Delay Count (>spare time) . . : 0
Delay Count (>1000 usecs) . . : 0
Delay Maximum . . . . . . . . : 0 usecs
Cycle Maximum . . . . . . . . : 1046 usecs
Average DSP Load. . . . . . . : 18.0 %
Average CPU System Load . . . : 2.5 %
Average CPU User Load . . . . : 7.8 %
Average CPU Nice Load . . . . : 0.1 %
Average CPU I/O Wait Load . . : 0.1 %
Average CPU IRQ Load . . . . : 0.1 %
Average CPU Soft-IRQ Load . . : 0.0 %
Average Interrupt Rate . . . : 1776.0 /sec
Average Context-Switch Rate . : 10290.4 /sec
*********************************************

SCHED_NORMAL nice -n -20:
*********************************************
Timeout Count . . . . . . . . :( 0)
XRUN Count . . . . . . . . . : 266
Delay Count (>spare time) . . : 0
Delay Count (>1000 usecs) . . : 0
Delay Maximum . . . . . . . . : 0 usecs
Cycle Maximum . . . . . . . . : 2239 usecs
Average DSP Load. . . . . . . : 28.6 %
Average CPU System Load . . . : 2.9 %
Average CPU User Load . . . . : 10.2 %
Average CPU Nice Load . . . . : 0.0 %
Average CPU I/O Wait Load . . : 1.0 %
Average CPU IRQ Load . . . . : 0.2 %
Average CPU Soft-IRQ Load . . : 0.1 %
Average Interrupt Rate . . . : 2049.7 /sec
Average Context-Switch Rate . : 10145.1 /sec
*********************************************

SCHED_ISO:
*********************************************
Timeout Count . . . . . . . . :( 0)
XRUN Count . . . . . . . . . : 1
Delay Count (>spare time) . . : 0
Delay Count (>1000 usecs) . . : 0
Delay Maximum . . . . . . . . : 0 usecs
Cycle Maximum . . . . . . . . : 687 usecs
Average DSP Load. . . . . . . : 19.9 %
Average CPU System Load . . . : 2.6 %
Average CPU User Load . . . . : 10.3 %
Average CPU Nice Load . . . . : 0.0 %
Average CPU I/O Wait Load . . : 0.0 %
Average CPU IRQ Load . . . . : 0.2 %
Average CPU Soft-IRQ Load . . : 0.3 %
Average Interrupt Rate . . . : 2166.2 /sec
Average Context-Switch Rate . : 10117.3 /sec
*********************************************

SCHED_FIFO:
*********************************************
Timeout Count . . . . . . . . :( 0)
XRUN Count . . . . . . . . . : 2
Delay Count (>spare time) . . : 0
Delay Count (>1000 usecs) . . : 0
Delay Maximum . . . . . . . . : 0 usecs
Cycle Maximum . . . . . . . . : 544 usecs
Average DSP Load. . . . . . . : 19.5 %
Average CPU System Load . . . : 3.1 %
Average CPU User Load . . . . : 12.6 %
Average CPU Nice Load . . . . : 0.0 %
Average CPU I/O Wait Load . . : 0.0 %
Average CPU IRQ Load . . . . : 1.0 %
Average CPU Soft-IRQ Load . . : 1.1 %
Average Interrupt Rate . . . : 5018.4 /sec
Average Context-Switch Rate . : 10902.5 /sec
*********************************************


It occasionally would segfault on client exit as well (as you've already
mentioned). I think we're still in the dark here to be honest.

Con


Attachments:
signature.asc (256.00 B)
OpenPGP digital signature

2005-01-19 14:02:34

by Con Kolivas

[permalink] [raw]
Subject: Re: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Just as a headsup there is a new cleaned up patch here:
http://ck.kolivas.org/patches/SCHED_ISO/2.6.11-rc1-iso-0501192240.diff

The yield bug is still eluding my capture but appears to be related to
the array switch of yielding and rescheduling as ISO after the fact.
Still working on it.

Cheers,
Con


Attachments:
signature.asc (256.00 B)
OpenPGP digital signature

2005-01-19 14:26:19

by Jack O'Quin

[permalink] [raw]
Subject: Re: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Con Kolivas <[email protected]> writes:

> Jack O'Quin wrote:
>> Con Kolivas <[email protected]> writes:
>>
>>>This patch for 2.6.11-rc1 provides a method of providing real time
>>>scheduling to unprivileged users which increasingly is desired for
>>>multimedia workloads.
>> I ran some jack_test3.2 runs with this, using all the default
>> settings. The results of three runs differ quite significantly for no
>> obvious reason. I can't figure out why the DSP load should vary so
>> much. These may be bogus results. It looks like a libjack bug
>> sometimes
>> causes clients to crash when deactivating. I will investigate more
>> tomorrow, and come up with a fix.
>> For comparison, I also made a couple of runs using the realtime-lsm
>> to
>> grant SCHED_FIFO privileges. There was some variablility, but nowhere
>> near as much (and no crashes). I used schedtool to verify that the
>> jackd threads actually have the expected scheduler type.
>
> Thanks for those. If you don't know what to make of the dsp variation
> and the crashing then I'm not sure what I should make of it
> either. It's highly likely that my code still needs fixing to ensure
> it behaves as expected. Already one bug has been picked up in testing
> with respect to yield() so there may be others. By design, if you set
> iso_cpu to 100 it should be as good as SCHED_RR. If not, then the
> implementation is still buggy.

I fixed that bug in libjack, eliminating the crashes on disconnect.

They must have been perturbing the numbers quite a bit. Here are
three more runs (all SCHED_ISO). The results are much more
consistent. The only significant anomaly I see now is that small
Delay Max in the first run.

*** Terminated Wed Jan 19 01:04:55 CST 2005 ***
************* SUMMARY RESULT ****************
Total seconds ran . . . . . . : 300
Number of clients . . . . . . : 20
Ports per client . . . . . . : 4
Frames per buffer . . . . . . : 64
*********************************************
Timeout Count . . . . . . . . :( 1) ( 5) ( 3)
XRUN Count . . . . . . . . . : 2 16 15
Delay Count (>spare time) . . : 0 0 0
Delay Count (>1000 usecs) . . : 0 0 0
Delay Maximum . . . . . . . . : 11932 usecs 101053 usecs 98719 usecs
Cycle Maximum . . . . . . . . : 868 usecs 1099 usecs 887 usecs
Average DSP Load. . . . . . . : 37.9 % 33.6 % 36.0 %
Average CPU System Load . . . : 10.2 % 9.0 % 9.9 %
Average CPU User Load . . . . : 24.5 % 22.7 % 23.0 %
Average CPU Nice Load . . . . : 0.0 % 0.0 % 0.0 %
Average CPU I/O Wait Load . . : 0.6 % 3.4 % 0.4 %
Average CPU IRQ Load . . . . : 0.7 % 0.7 % 0.7 %
Average CPU Soft-IRQ Load . . : 0.0 % 0.0 % 0.0 %
Average Interrupt Rate . . . : 1688.1 /sec 1696.1 /sec 1685.2 /sec
Average Context-Switch Rate . : 11727.9 /sec 10642.4 /sec 11568.3 /sec
*********************************************

I should probably experiment with higher thresholds on the SCHED_ISO class.
--
joq

2005-01-20 00:14:23

by Con Kolivas

[permalink] [raw]
Subject: Re: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Jack O'Quin wrote:
> Try again with JACK 0.99.48. It's in CVS now, but you probably need
> this tarball to get around the dreaded SourceForge anon CVS lag...
>
> http://www.joq.us/jack/tarballs/jack-audio-connection-kit-0.99.48.tar.gz

Thanks it finally ran to completion. By the way the patch you sent with
the test suite did not apply so I had to do it manually (booraroom..)

> The results I get with these versions are a lot more stable. But,
> there are still some puzzles about the scheduling. Do you distinguish
> different SCHED_ISO priorities?

It was not clear whether that would be required or not. This is why
testing is so critical because if you are having latency issues it
wouldn't be a problem of getting cpu time since your cpu usage is well
below the 70% limit.

>
> JACK runs with three different SCHED_FIFO priorities:
>
> (1) The main jackd audio thread uses the -P parameter. The JACK
> default is 10, but Rui's script sets it with -P60. I don't think
> the absolute value matters, but the value relative to the other JACK
> realtime threads probably does.
>
> (2) The clients' process threads run at a priority one less (59).
>
> (3) The watchdog timer thread runs at a priority 10 greater (70).
>
> (4) LinuxThreads creates a manager thread in each process running
> one level higher than the highest user realtime thread priority.

Since I (finally) have it running at this end at last I'll do some
benchmarking of my own to see how (lack of) priorities affects
SCHED_ISO. If it is inadequate, it wont be too difficult to add them to
the design. The problem with priorities is that once you go over the cpu
limit everyone suffers equally; but that's a failsafe that you shouldn't
actually hit in normal usage so it probably doesn't matter... Hmm come
to think of it, it probably _is_ a good idea to implement priority
support afterall.

Cheers,
Con

2005-01-20 01:19:57

by Jack O'Quin

[permalink] [raw]
Subject: Re: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

Con Kolivas <[email protected]> writes:

> Jack O'Quin wrote:
>> Try again with JACK 0.99.48. It's in CVS now, but you probably need
>> this tarball to get around the dreaded SourceForge anon CVS lag...
>>
>> http://www.joq.us/jack/tarballs/jack-audio-connection-kit-0.99.48.tar.gz
>
> Thanks it finally ran to completion. By the way the patch you sent
> with the test suite did not apply so I had to do it manually
> (booraroom..)

Oops! Sorry. I generated those by hand using some rather crude
`diff -u .... >> xxx.diff' commands.

We should just add Rui's latest version to JACK CVS.

> Since I (finally) have it running at this end at last I'll do some
> benchmarking of my own to see how (lack of) priorities affects
> SCHED_ISO. If it is inadequate, it wont be too difficult to add them
> to the design. The problem with priorities is that once you go over
> the cpu limit everyone suffers equally; but that's a failsafe that you
> shouldn't actually hit in normal usage so it probably doesn't
> matter...

I'd be surprised if we're hitting it in this test. AFAICT, our
"Average DSP Load" should approximate your CPU limit. That's running
in the 30% to 40% range. Is there any way to verify this? Is your
running average readable in /proc/sys/kernel somewhere?

We do need to test that the system degrades gracefully when the CPU is
overloaded. That *will* happen (the dreaded P4 float denormal
problems quickly come to mind). At some point, the user should be
allowed to choose how much CPU to consume, possibly shutting down
plugins as needed.

> Hmm come to think of it, it probably _is_ a good idea to implement
> priority support afterall.

Hard to say for sure without trying it. These threads are dealing
with a realtime cycle that is smaller than normal scheduler time
slices. Getting all the work done is important. But, getting it done
in the right order might make a difference for important statistics
like Max Delay.
--
joq