On Tue, 2004-08-31 at 23:53, Paul E. McKenney wrote:
> On Mon, Aug 30, 2004 at 11:22:49PM -0400, Jim Houston wrote:
> > On Mon, 2004-08-30 at 14:52, Paul E. McKenney wrote:
> > > How does the rest of the kernel work with all interrupts to
> > > a particular CPU shut off? For example, how do you timeslice?
> >
> > It's a balancing act. In some cases we just document the
> > missing functionality. If the local timer is disabled on a cpu,
> > all processes are SCHED_FIFO. In the case of Posix timers, we
> > move timers to honor the procesor shielding an the process affinity.
>
> I have to ask... When you say that you move the timers, you mean that
> non-realtime CPU 1 managers timers for realtime CPU 0, so that CPU 1
> is (effectively) taking CPU 0's timer interrupts?
Hi Paul,
That is part of the idea. There are lots of timers which we don't
expect to have realtime behavior.
There are also services like Posix timers and nanosleep() where we want
very predictable behavior. If a process does a nanosleep(), we queue
that timer on the local cpu. If process affinity is changed, we will
move the timer to a cpu where the process is allowed to run.
We have separate queues for high resolution timers. If the local queue
is empty, we shutdown the timer.
Jim Houston - Concurrent Computer Corp.
On Wed, Sep 01, 2004 at 09:02:00AM -0400, Jim Houston wrote:
> On Tue, 2004-08-31 at 23:53, Paul E. McKenney wrote:
> > On Mon, Aug 30, 2004 at 11:22:49PM -0400, Jim Houston wrote:
> > > On Mon, 2004-08-30 at 14:52, Paul E. McKenney wrote:
> > > > How does the rest of the kernel work with all interrupts to
> > > > a particular CPU shut off? For example, how do you timeslice?
> > >
> > > It's a balancing act. In some cases we just document the
> > > missing functionality. If the local timer is disabled on a cpu,
> > > all processes are SCHED_FIFO. In the case of Posix timers, we
> > > move timers to honor the procesor shielding an the process affinity.
> >
> > I have to ask... When you say that you move the timers, you mean that
> > non-realtime CPU 1 managers timers for realtime CPU 0, so that CPU 1
> > is (effectively) taking CPU 0's timer interrupts?
>
> Hi Paul,
>
> That is part of the idea. There are lots of timers which we don't
> expect to have realtime behavior.
>
> There are also services like Posix timers and nanosleep() where we want
> very predictable behavior. If a process does a nanosleep(), we queue
> that timer on the local cpu. If process affinity is changed, we will
> move the timer to a cpu where the process is allowed to run.
>
> We have separate queues for high resolution timers. If the local queue
> is empty, we shutdown the timer.
Hello, Jim,
How do you mark a given CPU as being in realtime mode? Or is the
timer-shutdown decision based on the presence of a realtime process
runnable on the given CPU or some such?
Still trying to figure out a way to make this work without adding
overhead to rcu_read_unlock()...
Thanx, Paul
On Thu, 2004-09-02 at 12:38, Paul E. McKenney wrote:
> On Wed, Sep 01, 2004 at 09:02:00AM -0400, Jim Houston wrote:
> > On Tue, 2004-08-31 at 23:53, Paul E. McKenney wrote:
> > > On Mon, Aug 30, 2004 at 11:22:49PM -0400, Jim Houston wrote:
> > > > On Mon, 2004-08-30 at 14:52, Paul E. McKenney wrote:
> > > > > How does the rest of the kernel work with all interrupts to
> > > > > a particular CPU shut off? For example, how do you timeslice?
> > > >
> > > > It's a balancing act. In some cases we just document the
> > > > missing functionality. If the local timer is disabled on a cpu,
> > > > all processes are SCHED_FIFO. In the case of Posix timers, we
> > > > move timers to honor the procesor shielding an the process affinity.
> > >
> > > I have to ask... When you say that you move the timers, you mean that
> > > non-realtime CPU 1 managers timers for realtime CPU 0, so that CPU 1
> > > is (effectively) taking CPU 0's timer interrupts?
> >
> > Hi Paul,
> >
> > That is part of the idea. There are lots of timers which we don't
> > expect to have realtime behavior.
> >
> > There are also services like Posix timers and nanosleep() where we want
> > very predictable behavior. If a process does a nanosleep(), we queue
> > that timer on the local cpu. If process affinity is changed, we will
> > move the timer to a cpu where the process is allowed to run.
> >
> > We have separate queues for high resolution timers. If the local queue
> > is empty, we shutdown the timer.
>
> Hello, Jim,
>
> How do you mark a given CPU as being in realtime mode? Or is the
> timer-shutdown decision based on the presence of a realtime process
> runnable on the given CPU or some such?
>
> Still trying to figure out a way to make this work without adding
> overhead to rcu_read_unlock()...
Hi Paul
We add the following /proc files:
/proc/shield/irqs
Setting a bit limits the corresponding cpu to only handle
interrupts which are explicitly directed to that cpu.
/proc/shield/ltmrs
Setting a bit limits the use of local timers on the
corresponding cpu.
/proc/shield/procs
Setting a bit limits the cpu to only run processes which
have set affinity to that cpu.
When the user changes something we adjust irq routing and migrate
processes and timers appropriately.
Jim Houston - Concurrent Computer Corp.
On Thu, 2004-09-02 at 17:20, Manfred Spraul wrote:
> Jim Houston wrote:
>
> >We add the following /proc files:
> >/proc/shield/ltmrs
> > Setting a bit limits the use of local timers on the
> > corresponding cpu.
> >
> How do you handle schedule_delayed_work_on()?
> slab uses it to drain the per-cpu caches. It's not fatal if a cpu
> doesn't drain it's caches (just some wasted memory), but it should be
> documented.
Hi Manfred,
The timer shielding migrates most of the timers to non-shielded cpus
but does keep track of timers queued with add_timer_on(). These
are polled periodically from a non-shielded cpu, and an inter-processsor
interrupt is used to force the shielded cpu to handle their expiry.
Jim Houston - Concurrent Computer Corp.
Jim Houston wrote:
>We add the following /proc files:
>
>/proc/shield/irqs
> Setting a bit limits the corresponding cpu to only handle
> interrupts which are explicitly directed to that cpu.
>
>/proc/shield/ltmrs
> Setting a bit limits the use of local timers on the
> corresponding cpu.
>
>
>
How do you handle schedule_delayed_work_on()?
slab uses it to drain the per-cpu caches. It's not fatal if a cpu
doesn't drain it's caches (just some wasted memory), but it should be
documented.
--
Manfred