On Thu, 30 Jul 2009 16:20:26 -0700
Darren Hart <[email protected]> wrote:

> Thomas Gleixner wrote:
> > We are pleased to announce the next update to our new preempt-rt
> > series.
> >
> > - update to 2.6.31-rc4
> >
> > This is a major rework of the rt patch series. Thanks to Clark
> > Williams and John Kacur for providing the merge to 2.6.30 while I was
> > stabilizing .29-rt. While the 30-rt series looked quite stable, we
> > decided to skip 30-rt entirely to keep track with the ongoing mainline
> > development for various reaons. The .31-rt series is planned to be
> > stabilized as we have done with .29-rt.
>
> I hit this on boot on an IBM Thinkpad T60p (Intel Core Duo). Haven't
> had a chance to dig in yet, but wanted to post sooner rather than later.
>
> [ 4.764702] Freeing unused kernel memory: 552k freed
> [ 4.765205] Write protecting the kernel text: 4552k
> [ 4.765389] Write protecting the kernel read-only data: 1776k
> [ 4.766458] BUG: sleeping function called from invalid context at
> kernel/rtmutex.c:684
> [ 4.766596] in_atomic(): 1, irqs_disabled(): 0, pid: 103, name: init
> [ 4.766723] Pid: 103, comm: init Not tainted 2.6.31-rc4-rt1-dvh01 #1
> [ 4.766848] Call Trace:
> [ 4.766973] [<c012d881>] __might_sleep+0xe1/0x100
> [ 4.767099] [<c056ccba>] rt_spin_lock+0x2a/0x70
> [ 4.767224] [<c018243a>] res_counter_uncharge+0x2a/0x50
> [ 4.767350] [<c01e6e23>] __mem_cgroup_uncharge_common+0x93/0x190
> [ 4.767477] [<c01e6fb8>] mem_cgroup_uncharge_page+0x28/0x30
> [ 4.767604] [<c01d6b87>] page_remove_rmap+0x47/0x50
> [ 4.767728] [<c01cf2d9>] unmap_vmas+0x349/0x6b0
> [ 4.767852] [<c01d45e5>] exit_mmap+0xc5/0x1c0
> [ 4.767977] [<c037d634>] ? get_random_int+0xb4/0xe0
> [ 4.768114] [<c0140481>] mmput+0x51/0xc0
> [ 4.768237] [<c01ef651>] flush_old_exec+0x381/0x6c0
> [ 4.768361] [<c01ea0c6>] ? vfs_read+0x126/0x190
> [ 4.768484] [<c01eea3f>] ? kernel_read+0x3f/0x60
> [ 4.768609] [<c02228cb>] load_elf_binary+0x33b/0x18c0
> [ 4.768734] [<c012f06f>] ? __wake_up+0x3f/0x50
> [ 4.768859] [<c01ccfee>] ? kunmap_high+0x7e/0xb0
> [ 4.768983] [<c01ccede>] ? page_address+0x8e/0x90
> [ 4.769125] [<c01cd15b>] ? kmap_high+0x13b/0x4c0
> [ 4.769248] [<c01d1752>] ? __get_user_pages+0x112/0x3d0
> [ 4.769374] [<c012f06f>] ? __wake_up+0x3f/0x50
> [ 4.769498] [<c056cd9a>] ? __rt_spin_lock+0x2a/0x70
> [ 4.769623] [<c0222590>] ? load_elf_binary+0x0/0x18c0
> [ 4.769747] [<c01eedbd>] search_binary_handler+0xfd/0x300
> [ 4.769872] [<c01f0568>] do_execve+0x228/0x300
> [ 4.769996] [<c0319626>] ? strncpy_from_user+0x46/0x70
> [ 4.770121] [<c0101b16>] sys_execve+0x36/0x60
> [ 4.770244] [<c0103025>] syscall_call+0x7/0xb
>

Ok, here's a traceback I got running 2.6.31-rc4-rt1 on a Lenovo T60 that
had been running for a while:

BUG: scheduling while atomic: gkrellm/0x00000001/2483, CPU#0
Modules linked in: ati_remote pl2303 usbserial fuse i915 drm i2c_algo_bit sunrpc ipv6 cpufreq_ondemand acpi_cpufreq freq_table loop dm_multipath scsi_dh kvm_intel kvm uinput arc4 ecb iwl3945 btusb iwlcore snd_hda_codec_analog bluetooth snd_hda_intel snd_hda_codec snd_hwdep mac80211 snd_pcm thinkpad_acpi snd_timer hwmon snd cfg80211 soundcore video rfkill iTCO_wdt iTCO_vendor_support sr_mod joydev cdrom ata_generic yenta_socket snd_page_alloc pcspkr i2c_i801 i2c_core output e1000e ata_piix rsrc_nonstatic sg ahci libata sd_mod scsi_mod dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod uhci_hcd ohci_hcd ssb [last unloaded: microcode]
Pid: 2483, comm: gkrellm Not tainted 2.6.31-rc4-rt1 #31
Call Trace:
[<ffffffff8105742b>] __schedule_bug+0x97/0x9c
[<ffffffff8153856b>] __schedule+0xc7/0x112c
[<ffffffff8153c8c6>] ? _atomic_spin_unlock+0x46/0x62
[<ffffffff81539ad5>] schedule+0x1f/0x70
[<ffffffff8153b163>] rt_spin_lock_slowlock+0x313/0x475
[<ffffffff8153c1ed>] rt_spin_lock+0xd4/0xd9
[<ffffffff810c5d6b>] res_counter_uncharge+0x2e/0x53
[<ffffffff8115b1f5>] __mem_cgroup_uncharge_common+0x24e/0x376
[<ffffffff8115b3f7>] mem_cgroup_uncharge_page+0x52/0x54
[<ffffffff8113e9ca>] page_remove_rmap+0x5a/0x7d
[<ffffffff8112eb62>] unmap_vmas+0xb0b/0x111b
[<ffffffff81112ec0>] ? __alloc_pages_nodemask+0x1ea/0xa16
[<ffffffff81137f16>] unmap_region+0x148/0x294
[<ffffffff81139940>] do_munmap+0x35d/0x3e5
[<ffffffff81139a0e>] sys_munmap+0x46/0x5d
[<ffffffff8100cf72>] system_call_fastpath+0x16/0x1b

Steven seemed to think that we have a mismatched preempt_disable() and
preempt_enable() but I haven't had enough coffee to actually try
debugging it.

Clark

Attachments:

signature.asc (198.00 B)

2009-07-31 23:22:13

by Frederic Weisbecker

[permalink] [raw]

Subject: Re: [ANNOUNCE] 2.6.31-rc4-rt1

On Thu, Jul 30, 2009 at 02:48:04AM +0200, Thomas Gleixner wrote:
> We are pleased to announce the next update to our new preempt-rt
> series.
>
> - update to 2.6.31-rc4
>
> This is a major rework of the rt patch series. Thanks to Clark
> Williams and John Kacur for providing the merge to 2.6.30 while I was
> stabilizing .29-rt. While the 30-rt series looked quite stable, we
> decided to skip 30-rt entirely to keep track with the ongoing mainline
> development for various reaons. The .31-rt series is planned to be
> stabilized as we have done with .29-rt.
>
> The main changes in this release are:
>
> - interrupt threading
>
> interrupt threading is now a pure extension of the mainline
> threaded interrupt infrastructure. This reduced the patch size of
> the forced irq threading to mere
>
> 8 files changed, 178 insertions(+), 13 deletions(-)
>
> Another interesting detail is that the new forced threaded code
> uses per device threads instead of per interrupt line threads as
> we have done in the past. This was just a logical consequence of
> the per device thread (voluntary threading) infrastructure in
> mainline and allows us now to share an interrupt line between a
> hardirq based handler and a threaded handler device. One use case
> which comes to my mind is AT91 which shares the timer and the
> serial port interrupt; we now can solve that problem w/o nasty
> hacks by requesting a threaded handler for the serial port which
> shuts up the serial device interrupt in the hard interrupt handler
> part.
>
> - rework of the locking infrastructure
>
> Up to now the -rt patches changed the raw_spinlock_t to
> __raw_spinlock_t and added another two levels of underscores to
> many of the locking primitives. A compiler trick was used to chose
> the implementation for RT=y and RT=n compiles depending on the lock
> type in the lock definition.
>
> This is nasty as there is no destinction in the source code which
> kind of lock we are dealing with except if one looks up the lock
> definition/declaration. It definitely was a clever move in the
> first place to get things going, but aside of the underscore
> conflicts which were introduced by lockdep it was not longer
> acceptable to hide the fact that we are treating a lock
> differently. Same applies for the changes to (rw_)semaphores which
> used the compat_ trick for those ownerless anonymous semaphores
> which are taken in one context and released in another.
>
> The annotation of the code which uses those special treated locks
> has been long discussed and one of the proposed solutions was to
> change all spinlocks which are converted by -rt to sleeping
> spinlocks from spinlock_t to lock_t and have another set of
> lock/unlock/trylock functions for those. That is definitely the
> _preferred_ solution, but it's a massive and horribly intrusive
> change. Steven was working on it for some time, but it simply does
> not scale IMNSHO.
>
> I went the other way round. In -RT we have identified the locks
> which can _not_ be converted to sleeping locks and so I went there
> and converted them to atomic_spinlock_t and created a set of
> functions for them. I converted the already known locks to that
> type and fixed up all the functions (s/spin_*/atomic_spin_*/) which
> annotates the code and makes it clear what we are dealing with.
>
> [ I admit "atomic_spinlock_t" is a horrible name, but it's the best
> I came up with so far. If you have a better idea please feel free
> to add it to
>
> http://rt.wiki.kernel.org/index.php/Atomic_Spinlock
>
> instead of starting a bikeshed painting thread on the mailing
> lists about that name. Once we have something better it's just a
> sed script to fix it. ]
>
> For !RT the spin_* functions are mapped to atomic_spin_* via inline
> functions which do the type conversion. That has another nice side
> effect: some places in the kernel (mostly scheduler) use
> _raw_spin_* functions on locks to avoid the lockdep invocation in
> some places. With the type conversion a lock needs to be defined
> atomic_spinlock_t (or raw_spinlock_t) to have access to that
> _raw_spin_* functions. Using e.g. _raw_spin_lock() on a lock
> defined with spinlock_t/DEFINE_SPINLOCK will cause a compiler
> warning. I think that's a Good Thing.
>
> On RT the spin_* functions are mapped to the corresponding rt_lock
> functions with inlines as well. Very simple and much more
> understandable than the nifty PICK_OP magic with the underscore
> convolution. :)
>
> I did the same conversion for all (rw_)semaphores which are known
> from -rt to be ownerless anonymous semaphores; i.e. taken in one
> context and released in another. We renamed them to
> compat_(rw_)semaphores up to now and let the compiler pick the
> right function. Again here I went down the road and annotated the
> code for those with newly created anon_* and [read|write]_anon_*
> functions. In !RT the non annotated ones map to the anon_ functions
> and on RT we map them to the corresponging rt_* ones. This
> annotation should also be helpful to cover at least the non
> anonymous (rw_)semaphores via lockdep.
>
> Part of that semaphore rework is the RFC patch series I posted
> recently to get rid of the init_MUTEX[_LOCKED] irritation (minus
> the ones which turned out to be wrong)
>
> The spinlock and semaphore annotation work is separate now and can
> be found in the rt/atomic-lock and rt/semaphore branches of the
> -tip git repository, which leads me to the next important point:
>
> - start of gitification
>
> While reworking all of the above I went through the quilt queue and
> sorted out patches into different rt/ branches. If you clone the
> -tip git tree you'll find a bunch of branches starting with rt/.
> They contain various independent changes which are all part of the
> -rt patch. The combination of those branches can be found in the
> rt/base branch.
>
> I still have a leftover of ~140 patches (roughly 40% of the -rt
> queue) which I committed into the rt/rt-2.6.31-rc4 branch just as
> is simply because I ran out of time. My annual summer vacation
> (helping my wife to run the kitchen in the church community kids
> summer camp) is starting on friday.
>
> While the other rt/ branches are mostly bisectable the final one is
> not yet there. I restructured the patch queue in a logical way, but
> there is more work to be done to clean it up. So expect it to be
> replaced.
>
> Further plans:
>
> 1) We seriously want to tackle the elimination of the PREEMPT_RT
> annoyance #1, aka BKL. The Big Kernel Lock is still used in ~330
> files all across the kernel. A lot of work has been done already to
> push down the lock into the code which still thinks it needs to be
> protected by it. Some work has been done already in the (a little
> bit stale) kill-the-BKL and core/kill-the-BKL branches of the -tip
> git tree. If you want to help, please check those branches whether
> the code has been tackled already or not to avoid redundant
> work. If you decide to take care of one please note it on:
>
> http://rt.wiki.kernel.org/index.php/Big_Kernel_Lock

We (I) can't write to this file :)

However, I'll try to attack the remaining bkl that resides in tty...

Thanks,
Frederic.

>
> 2) I'm going on vacation for 10 days. Please send patches and
> bugreports^Wsuccess stories to the mailinglist as usual. There are
> folks looking out.
>
> Enough said. Get the code and have fun!
>
> Download locations:
>
> http://rt.et.redhat.com/download/
> http://www.kernel.org/pub/linux/kernel/projects/rt/
>
> Git:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git rt/rt-2.6.31-rc4-rt1
>
> Gitweb:
> http://git.kernel.org/?p=linux/kernel/git/tip/linux-2.6-tip.git;a=shortlog;h=rt/rt-2.6.31-rc4
>
> Information on the RT patch can be found at:
>
> http://rt.wiki.kernel.org/index.php/Main_Page
>
> to build the 2.6.31-rc4-rt1 tree, the following patches should be
> applied:
>
> http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.30.tar.bz2
> http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.31-rc4.bz2
> http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.31-rc4-rt1.bz2
>
> Thanks to Carsten Emde, Clark Williams and John Kacur who were testing
> my various steps to get the code into the shape where it is now.
>
> Enjoy !
>
> tglx
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2009-08-01 00:42:39

by Darren Hart

[permalink] [raw]

Subject: Re: [ANNOUNCE] 2.6.31-rc4-rt1

Frederic Weisbecker wrote:
> On Thu, Jul 30, 2009 at 02:48:04AM +0200, Thomas Gleixner wrote:

>> Further plans:
>>
>> 1) We seriously want to tackle the elimination of the PREEMPT_RT
>> annoyance #1, aka BKL. The Big Kernel Lock is still used in ~330
>> files all across the kernel. A lot of work has been done already to
>> push down the lock into the code which still thinks it needs to be
>> protected by it. Some work has been done already in the (a little
>> bit stale) kill-the-BKL and core/kill-the-BKL branches of the -tip
>> git tree. If you want to help, please check those branches whether
>> the code has been tackled already or not to avoid redundant
>> work. If you decide to take care of one please note it on:
>>
>> http://rt.wiki.kernel.org/index.php/Big_Kernel_Lock
>
>
> We (I) can't write to this file :)
>
> However, I'll try to attack the remaining bkl that resides in tty...

I took a look at the perms, then the history - looks like you sorted it out.

--
Darren Hart
IBM Linux Technology Center
Real-Time Linux Team

2009-08-01 01:20:08

by Frederic Weisbecker

[permalink] [raw]

Subject: Re: [ANNOUNCE] 2.6.31-rc4-rt1

On Fri, Jul 31, 2009 at 05:42:32PM -0700, Darren Hart wrote:
> Frederic Weisbecker wrote:
>> On Thu, Jul 30, 2009 at 02:48:04AM +0200, Thomas Gleixner wrote:
>
>>> Further plans:
>>>
>>> 1) We seriously want to tackle the elimination of the PREEMPT_RT
>>> annoyance #1, aka BKL. The Big Kernel Lock is still used in ~330
>>> files all across the kernel. A lot of work has been done already to
>>> push down the lock into the code which still thinks it needs to be
>>> protected by it. Some work has been done already in the (a little
>>> bit stale) kill-the-BKL and core/kill-the-BKL branches of the -tip
>>> git tree. If you want to help, please check those branches whether
>>> the code has been tackled already or not to avoid redundant
>>> work. If you decide to take care of one please note it on:
>>>
>>> http://rt.wiki.kernel.org/index.php/Big_Kernel_Lock
>>
>>
>> We (I) can't write to this file :)
>>
>> However, I'll try to attack the remaining bkl that resides in tty...
>
> I took a look at the perms, then the history - looks like you sorted it out.

Yeah, I just had to register. I'm an idiot.

Thanks!

> --
> Darren Hart
> IBM Linux Technology Center
> Real-Time Linux Team

2009-09-12 09:52:01

by Jan Engelhardt

[permalink] [raw]

Subject: Re: [ANNOUNCE] 2.6.29.6-rt23

On Tuesday 2009-08-11 12:52, Jan Engelhardt wrote:
>On Friday 2009-07-10 20:29, Thomas Gleixner wrote:
>>On Fri, 10 Jul 2009, Fernando Lopez-Lezcano wrote:
>>>
>>> I'm getting slightly weird behavior from rt23 in an intel quad core
>>> machine (my T61 laptop is fine), rt22 was fine. It is some timer issue,
>>> if I type continuously I get (every few seconds) repeated keys. Nothing
>>> in the logs, nothing in dmesg that is (significantly) different from an
>>> rt22 boot.
>
>I am getting the same issue, reliably reproducibly - though ironically
>only with the proprietary NVIDIA. Key repeats, screen going "No signal"
>for a sec, sometimes comes back, sometimes hangs the whole machine.

The problem is reproducible on 2.6.31-rc8-rt9. Keys repeat, and the
patch for 2.6.31-rt does not want to compile anymore.