LinuxLists.cc - 2.6.18-rt1

2006-09-20 14:27:24

Subject: 2.6.18-rt1

I'm pleased to announce the 2.6.18-rt1 tree, which can be downloaded
from the usual place:

http://redhat.com/~mingo/realtime-preempt/

This port to the 2.6.18 codebase was very complex (given that 5 major
features moved from -rt into 2.6.18: rtmutex, pi-futex, genirq, gtod and
lockdep), so it took quite some time to finish, but in exchange it
includes nice new features and also lots of bugfixes.

In particular, a nasty softirq performance bug has been fixed, which
caused the "5x slowdown under TCP" bug reported to lkml - those TCP
performance figures are now on par with vanilla performance.

The biggest new features are:

- Dynticks (a tickless kernel). This is a new feature we implemented
ontop of hrtimers for the -hrt patchqueue and now we merged it to -rt
too. There's a new config option: CONFIG_NO_HZ, which if enabled,
produces a system with fewer timer interrupts. Currently i686 and
x86_64 are tested, but in general a hrtimers-ready platform needs
minimal changes to support dynticks too. (from Thomas Gleixner and
me, the x86_64 port is from Arjan van de Ven)

- Timer expiry statistics feature from Thomas Gleixner:
CONFIG_TIMER_STATS. Enable/disable via "echo 1[0] > /proc/tstats",
display via "cat /proc/stats".

This kernel feature tracks the starting site (and expiry function) of
expired timers (both timer_list timers and hrtimers), and the number
of times they expired. This is a nice tool for those who'd like to
minimize the amount of timer ticks on their battery-based systems.
Sample output:

Timerstats sample period: 3.888770 s
12, 0 swapper hrtimer_stop_sched_tick (hrtimer_sched_tick)
15, 1 swapper hcd_submit_urb (rh_timer_func)
4, 959 kedac schedule_timeout (process_timeout)
1, 0 swapper page_writeback_init (wb_timer_fn)
28, 0 swapper hrtimer_stop_sched_tick (hrtimer_sched_tick)
22, 2948 IRQ 4 tty_flip_buffer_push (delayed_work_timer_fn)
3, 3100 bash schedule_timeout (process_timeout)
1, 1 swapper queue_delayed_work_on (delayed_work_timer_fn)
1, 1 swapper queue_delayed_work_on (delayed_work_timer_fn)
1, 1 swapper neigh_table_init_no_netlink (neigh_periodic_timer)
1, 2292 ip __netdev_watchdog_up (dev_watchdog)
1, 23 events/1 do_cache_clean (delayed_work_timer_fn)
90 total events, 30.0 events/sec

- There's also a new runtime-configurable kernel option under
/proc/sys/kernel/timeout_granularity that allows the coarser expiry
of timer_list timers, and hence better batching of timer expiry.
(hrtimers are not affected) The value of this option gives a
'multiplier' to the standard HZ granularity.

- Lock validator ported to -rt: it now checks all the sleeping lock
variants in -rt too. This helps us catch not only deadlocks but also
raw-lock candidates, sooner than before.

- The latest High Resolution Timers queue from Thomas Gleixner with
initial support for ARM and PPC. (Kevin Hilman, Deepak Saksena,
Sergei Shtylyov)

- The latest GTOD (Generic Time Of Day) queue from John Stultz,
including NTP cleanups from Roman Zippel.

- The latest genirq queue: more irq-chip-ization of i686 and x86_64,
MSI cleanups from Eric W. Biederman et al.

- The latest preemptible-RCU queue from Paul E McKenney and Dipankar
Sarma.

- (assorted fixes and improvements.)

Right now i686 and x86_64 are boot-tested, and ARM is compile-tested.
The other architectures likely wont even build yet. This being the first
2.6.18 based release of -rt, some tester caution is called for.

to build a 2.6.18-rt1 tree, the following patches should be applied:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2
http://redhat.com/~mingo/realtime-preempt/patch-2.6.18-rt1

as usual, bugreports, fixes and suggestions are welcome,

Ingo

2006-09-20 16:51:05

by Gene Heskett

[permalink] [raw]

Subject: Re: 2.6.18-rt1

On Wednesday 20 September 2006 10:19, Ingo Molnar wrote:
>I'm pleased to announce the 2.6.18-rt1 tree, which can be downloaded
>from the usual place:
>
> http://redhat.com/~mingo/realtime-preempt/
>
>This port to the 2.6.18 codebase was very complex (given that 5 major
>features moved from -rt into 2.6.18: rtmutex, pi-futex, genirq, gtod and
>lockdep), so it took quite some time to finish, but in exchange it
>includes nice new features and also lots of bugfixes.
>
>In particular, a nasty softirq performance bug has been fixed, which
>caused the "5x slowdown under TCP" bug reported to lkml - those TCP
>performance figures are now on par with vanilla performance.
>
>The biggest new features are:
>
> - Dynticks (a tickless kernel). This is a new feature we implemented
> ontop of hrtimers for the -hrt patchqueue and now we merged it to -rt
> too. There's a new config option: CONFIG_NO_HZ, which if enabled,
> produces a system with fewer timer interrupts. Currently i686 and
> x86_64 are tested, but in general a hrtimers-ready platform needs
> minimal changes to support dynticks too. (from Thomas Gleixner and
> me, the x86_64 port is from Arjan van de Ven)
>
> - Timer expiry statistics feature from Thomas Gleixner:
> CONFIG_TIMER_STATS. Enable/disable via "echo 1[0] > /proc/tstats",
> display via "cat /proc/stats".
>
> This kernel feature tracks the starting site (and expiry function) of
> expired timers (both timer_list timers and hrtimers), and the number
> of times they expired. This is a nice tool for those who'd like to
> minimize the amount of timer ticks on their battery-based systems.
> Sample output:
>
> Timerstats sample period: 3.888770 s
> 12, 0 swapper hrtimer_stop_sched_tick
> (hrtimer_sched_tick) 15, 1 swapper hcd_submit_urb
> (rh_timer_func)
> 4, 959 kedac schedule_timeout (process_timeout)
> 1, 0 swapper page_writeback_init (wb_timer_fn)
> 28, 0 swapper hrtimer_stop_sched_tick
> (hrtimer_sched_tick) 22, 2948 IRQ 4 tty_flip_buffer_push
> (delayed_work_timer_fn) 3, 3100 bash schedule_timeout
> (process_timeout) 1, 1 swapper queue_delayed_work_on
> (delayed_work_timer_fn) 1, 1 swapper queue_delayed_work_on
> (delayed_work_timer_fn) 1, 1 swapper
> neigh_table_init_no_netlink (neigh_periodic_timer) 1, 2292 ip
> __netdev_watchdog_up (dev_watchdog) 1, 23 events/1
> do_cache_clean (delayed_work_timer_fn) 90 total events, 30.0 events/sec
>
> - There's also a new runtime-configurable kernel option under
> /proc/sys/kernel/timeout_granularity that allows the coarser expiry
> of timer_list timers, and hence better batching of timer expiry.
> (hrtimers are not affected) The value of this option gives a
> 'multiplier' to the standard HZ granularity.
>
> - Lock validator ported to -rt: it now checks all the sleeping lock
> variants in -rt too. This helps us catch not only deadlocks but also
> raw-lock candidates, sooner than before.
>
> - The latest High Resolution Timers queue from Thomas Gleixner with
> initial support for ARM and PPC. (Kevin Hilman, Deepak Saksena,
> Sergei Shtylyov)
>
> - The latest GTOD (Generic Time Of Day) queue from John Stultz,
> including NTP cleanups from Roman Zippel.
>
> - The latest genirq queue: more irq-chip-ization of i686 and x86_64,
> MSI cleanups from Eric W. Biederman et al.
>
> - The latest preemptible-RCU queue from Paul E McKenney and Dipankar
> Sarma.
>
> - (assorted fixes and improvements.)
>
>Right now i686 and x86_64 are boot-tested, and ARM is compile-tested.
>The other architectures likely wont even build yet. This being the first
>2.6.18 based release of -rt, some tester caution is called for.
>
>to build a 2.6.18-rt1 tree, the following patches should be applied:
>
> http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2
> http://redhat.com/~mingo/realtime-preempt/patch-2.6.18-rt1
>
>as usual, bugreports, fixes and suggestions are welcome,
>
> Ingo
>
I hadn't built one of these in quite a while, mainly because I could never
get tvtime to run while booted to them, some sort of a dma related failure
I think, so I rather dropped off the path here.

But the announcement was enough to make me try to build it, sort of the
ultimate "can an old fart build it test", but the make exits with:

LD .tmp_vmlinux1
kernel/built-in.o(.text+0x16f25): In function `hrtimer_start':
: undefined reference to `hrtimer_update_timer_prio'
make: *** [.tmp_vmlinux1] Error 1

about half way thru the normal time. config attached. I don't think hires
timers are enabled.

Comments?

--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2006 by Maurice Eugene Heskett, all rights reserved.

Attachments:

(No filename) (4.91 kB)
.config (40.10 kB)
Download all attachments

2006-09-20 17:01:46

* Daniel Walker <[email protected]> wrote:

> What about the !PREEMPT_RT case.. It still wouldn't disable interrupts
> in the below..
>
> - local_irq_save(flags);
> if (up->port.sysrq) {
> /* serial8250_handle_port() already took the lock */
> locked = 0;
> } else if (oops_in_progress) {
> - locked = spin_trylock(&up->port.lock);
> + locked = spin_trylock_irqsave(&up->port.lock, flags);
> } else
> - spin_lock(&up->port.lock);
> + spin_lock_irqsave(&up->port.lock, flags);

_irqsave disables interrupts, always did.

Ingo

2006-09-20 20:27:41

[permalink] [raw]

Subject: Re: 2.6.18-rt1

* Dipankar Sarma <[email protected]> wrote:

> Sorry, I should have published my investigations long ago. I tracked
> this down (atleast the crash in my machine) to NMI interference with
> rcu_read_lock()/rcu_read_unlock(). We use those APIs from NMI context
> as well
> (default_do_nmi()->notify_die()->atomic_notifier_call_chain()).
>
> Can you try with nmi_watchdog=0 in the kernel command line ?
>
> Paul has an NMI-safe patch for rcupreempt which I am adopting and
> testing at the moment. If this works well, I will publish a new
> patchset.

spent some good time debugging this 2 weeks ago and added the fix below
to rt5, but i forgot to do the symmetric fix for x86_64...

Ingo

----------->
arch/i386/kernel/traps.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

Index: linux/arch/i386/kernel/traps.c
===================================================================
--- linux.orig/arch/i386/kernel/traps.c
+++ linux/arch/i386/kernel/traps.c
@@ -716,9 +716,6 @@ static void default_do_nmi(struct pt_reg
reason = get_nmi_reason();

if (!(reason & 0xc0)) {
- if (notify_die(DIE_NMI_IPI, "nmi_ipi", regs, reason, 2, SIGINT)
- == NOTIFY_STOP)
- return;
#ifdef CONFIG_X86_LOCAL_APIC
/*
* Ok, so this is none of the documented NMI sources,
@@ -729,6 +726,9 @@ static void default_do_nmi(struct pt_reg
return;
}
#endif
+ if (notify_die(DIE_NMI_IPI, "nmi_ipi", regs, reason, 2, SIGINT)
+ == NOTIFY_STOP)
+ return;
unknown_nmi_error(reason, regs);
return;
}

2006-10-18 08:46:20

by Ingo Molnar

[permalink] [raw]

Subject: Re: 2.6.18-rt1

* Lee Revell <[email protected]> wrote:

> > Can you try whatever you were doing with nmi_watchdog=0 ? If it is
> > stable, then that would explain the problem. I believe Andi enabled
> > nmi watchdog on x86_64 by default recently, that might be why we are
> > seeing it now.
>
> Looks like that was the problem, the hard lockups are gone.

ok. Meanwhile i discovered that i fixed this bug on i686 but not on
x86_64. Could you try -rt6, does it work with the NMI watchdog
re-enabled?

Ingo