As a value add to the 2.6.34 long term release, I'm happy to also
announce the availability of 2.6.34-RT.
You can find it in the v2.6.34-rt branch at:
git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
as a repository of patches. The v2.6.34-rt branch contains the latest
RT patches against the latest v2.6.34.8 kernel release. (The master
branch currently stops at v2.6.34 flat, i.e. 2.6.34.0 so to speak.)
I've also created over 400 known working RT enabled bisection points
between 33 and 34 that you can make use of for testing. The details
on how/why those exist follows - read on if it is of interest to you.
Note that all credit of the thinking and engineering of the RT stuff
lies with the original patch authors -- to be clear, I'm just doing
the grunt work of a carry forward here.
Have fun with it, maybe drop me a note if you find it useful etc.
Paul.
The carry forward process:
---------------------------
This warrants some notes, since (a) it has strong consequence on how
you can diagnose/bisect problems that were not in 33-rt but may exist
in 34-rt and (b) we've recently seen people who wonder how to do a
similar task and I think they *really* underestimate the complexities
and sheer effort of trying to do any sort of RT carry-forward.
For those who track these things, you will know that the most recent
release of RT was based on v2.6.33, and in turn, it was created by
merging forward the mainline kernel.org v2.6.33 content into the older
v2.6.31 RT tree. There was no rebase.
With that detail, some of you are probably now going "Aha, now I know
why he created that merge-free v2.6.33 based RT tree some weeks ago."
(see: http://www.spinics.net/lists/linux-rt-users/msg06310.html )
Step one was that I needed the RT commits rebased against 33 (and not 31)
as my starting point for rebasing RT onto v2.6.34 -- see the above
link for details on the significant work involved with that.
There are roughly 500 RT patches, and literally 10,000+ commits between
v2.6.33 and v2.6.34. So if one was to move things ahead all in one go,
there can be roughly 5 million things that can go wrong. Maybe some sharp
person can move those ahead all in one shot, and then figure out the
resulting inevitable runtime breakage, but that isn't me. After all, a
man has to know his limitations, and I'm pretty much just a patch monkey.
Knowing that the RT patches applied (and worked) on base X, but fail to
apply or fail to work on X+N (where N is small) can be very empowering
to a dumb person like me. It reduces the problem space *immensely* and
it lets you focus on what changed and caused the problem/conflict. With
that in mind, I saw no choice but to undertake what would have been the
equivalent of recreating history, had a continuous integration between
the two streams taken place during the 33 --> 34 dev cycle.
What this means in concrete terms, is incrementally applying, updating,
and testing the 500-odd RT patches at successive points along the 10000
commits found in the 33 --> 34 development cycle. But assuming you
agree with that logic, what does a person choose/use as increments?
At a 1st guess, a person might suggest choosing the rcN tags (where we
have N=1,2,... 7) as places for these incremental tests. The problem
is that the granularity is too coarse -- just for N=1 means you have
already jumped over 6000 commits ahead, and so you are still facing
being lost for the same reason 10000 commits left you lost before.
At the opposite end of the scale is a brute force approach, where a
person tries applying/updating/testing the RT patch on *each* of the
10,000 development commits. Sounds like it might work, but no. The
problem here is that the kernel development history is non-linear
in time. See the git FAQ on why bisects can drag you back in time, if
you've never personally encountered this yourself. A lot of people
are surprised by this when they first encounter it.
My approach was to adopt a semi-brute force approach. During the dev
cycle of any kernel, Linus merges the content from various subsystem
maintainers. Each of these merge points represents a point where
you are guaranteed to not be "rewound in time", should you choose it
as a bisect point. Using git, these points can easily be identified.
There happens to be ~400 of them, so the 10,000 development commits get
"digested" at an average rate of a humanly manageable number of about 25
commits each -- something that a stupid person like me has a chance to
be able to diagnose/debug without going insane. You will find that there
is an un-annotated tag for each of these merges in the patch repository.
You really should use these for bisecting your own problems/issues.
On each of these, I've done a patch test, a compile test (x86, x86-64,
ppc, arm) and a boot test (x86, x86-64, ppc-SMP) to ensure that I've not
done a colossal screw-up. I've probably still screwed *something* up,
but at least I've ensured some level of continuing sanity with these tests
being done across these integration points.
--
On 03/04/2011 02:24 PM, Paul Gortmaker wrote:
> As a value add to the 2.6.34 long term release, I'm happy to also
> announce the availability of 2.6.34-RT.
>
> You can find it in the v2.6.34-rt branch at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
>
> as a repository of patches. The v2.6.34-rt branch contains the latest
> RT patches against the latest v2.6.34.8 kernel release. (The master
> branch currently stops at v2.6.34 flat, i.e. 2.6.34.0 so to speak.)
>
> I've also created over 400 known working RT enabled bisection points
> between 33 and 34 that you can make use of for testing. The details
> on how/why those exist follows - read on if it is of interest to you.
>
> Note that all credit of the thinking and engineering of the RT stuff
> lies with the original patch authors -- to be clear, I'm just doing
> the grunt work of a carry forward here.
>
> Have fun with it, maybe drop me a note if you find it useful etc.
I tried compiling this, and it quickly failed. Ideas?
I did this on a Fedora 14 x86_64 system with gcc-4.5.1-4.fc14.x86_64.
Here is what I did:
1. Downloaded 2.6.34.8 tarball and extracted it
2. git clone -b v2.6.34-rt
git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
3. cat ../rt-patches/series | xargs -i cat ../rt-patches/{} | patch -p1
(Applied just fine)
4. Copied an 2.6.31-rt configuration to .config
5. Ran make oldconfig
6. make bzImage
CHK include/linux/version.h
CHK include/generated/utsrelease.h
HOSTCC scripts/basic/fixdep
HOSTCC scripts/basic/docproc
HOSTCC scripts/basic/hash
CC kernel/bounds.s
GEN include/generated/bounds.h
CC arch/x86/kernel/asm-offsets.s
GEN include/generated/asm-offsets.h
CALL scripts/checksyscalls.sh
HOSTCC scripts/genksyms/genksyms.o
HOSTCC scripts/genksyms/lex.o
HOSTCC scripts/genksyms/parse.o
HOSTLD scripts/genksyms/genksyms
CC scripts/mod/empty.o
HOSTCC scripts/mod/mk_elfconfig
MKELF scripts/mod/elfconfig.h
HOSTCC scripts/mod/file2alias.o
HOSTCC scripts/mod/modpost.o
HOSTCC scripts/mod/sumversion.o
HOSTLD scripts/mod/modpost
HOSTCC scripts/selinux/genheaders/genheaders
HOSTCC scripts/selinux/mdp/mdp
HOSTCC scripts/kallsyms
HOSTCC scripts/conmakehash
CC init/main.o
init/main.c: In function ?do_one_initcall?:
init/main.c:740:10: warning: ?calltime.tv64? may be used uninitialized
in this function
CHK include/generated/compile.h
CC init/version.o
CC init/do_mounts.o
In file included from include/linux/suspend.h:8:0,
from init/do_mounts.c:6:
include/linux/mm.h: In function ?pte_lock_init?:
include/linux/mm.h:1125:2: error: implicit declaration of function ?kmalloc?
include/linux/mm.h:1125:12: warning: assignment makes pointer from
integer without a cast
include/linux/mm.h: In function ?pte_lock_deinit?:
include/linux/mm.h:1137:2: error: implicit declaration of function ?kfree?
In file included from include/linux/security.h:36:0,
from init/do_mounts.c:8:
include/linux/slab.h: At top level:
include/linux/slab.h:144:6: warning: conflicting types for ?kfree?
include/linux/mm.h:1137:2: note: previous implicit declaration of
?kfree? was here
In file included from include/linux/slab.h:172:0,
from include/linux/security.h:36,
from init/do_mounts.c:8:
include/linux/slab_def.h:128:52: error: conflicting types for ?kmalloc?
include/linux/mm.h:1125:14: note: previous implicit declaration of
?kmalloc? was here
make[1]: *** [init/do_mounts.o] Error 1
make: *** [init] Error 2
On 03/04/2011 02:24 PM, Paul Gortmaker wrote:
> As a value add to the 2.6.34 long term release, I'm happy to also
> announce the availability of 2.6.34-RT.
>
> You can find it in the v2.6.34-rt branch at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
Great! Many thanks... Actually, impressive work.
I'm trying to integrate/build it into my Fedora based rt packages for
Planet CCRMA. Warning: I'm doing this on top of the 2.6.34.8-68 fc13
koji build, so rt is patched on top of the additional Fedora patches for
that build (with a few trivial tweaks to patch cleanly). So I don't know
if the following issues are due to that - I will investigate:
I found a couple of extra argument cases for __netif_tx_lock:
--- a/drivers/net/qlcnic/qlcnic_main.c 2010-05-16 17:17:36.000000000 -0400
+++ b/drivers/net/qlcnic/qlcnic_main.c 2011-03-06 20:08:16.000000000 -0500
@@ -1881,7 +1881,7 @@
smp_mb();
if (netif_queue_stopped(netdev) && netif_carrier_ok(netdev)) {
- __netif_tx_lock(tx_ring->txq, smp_processor_id());
+ __netif_tx_lock(tx_ring->txq);
if (qlcnic_tx_avail(tx_ring) > TX_STOP_THRESH) {
netif_wake_queue(netdev);
adapter->tx_timeo_cnt = 0;
--- a/drivers/net/bnx2x_main.c 2010-05-16 17:17:36.000000000 -0400
+++ b/drivers/net/bnx2x_main.c 2011-03-06 19:58:45.000000000 -0500
@@ -976,7 +976,7 @@
* stops the queue
*/
- __netif_tx_lock(txq, smp_processor_id());
+ __netif_tx_lock(txq);
if ((netif_tx_queue_stopped(txq)) &&
(bp->state == BNX2X_STATE_OPEN) &&
----
a missing include (incomplete type errors for semaphores):
--- a/drivers/net/mlx4/mlx4.h 2010-05-16 17:17:36.000000000 -0400
+++ b/drivers/net/mlx4/mlx4.h 2011-03-06 20:07:05.000000000 -0500
@@ -37,6 +37,7 @@
#ifndef MLX4_H
#define MLX4_H
+#include <linux/semaphore.h>
#include <linux/mutex.h>
#include <linux/radix-tree.h>
#include <linux/timer.h>
----
and this (found compilation fix here:
http://groups.google.com/group/linux.kernel/browse_thread/thread/c4f0fd1ca9298c45/18316184209e6e84?show_docid=18316184209e6e84&fwc=1):
--- a/drivers/net/wireless/ath/ar9170/usb.c 2011-03-06
19:13:12.000000000 -0500
+++ b/drivers/net/wireless/ath/ar9170/usb.c 2011-03-06
20:13:06.000000000 -0500
@@ -741,10 +741,10 @@
/* unbind anything failed */
if (parent)
- down(&parent->sem);
+ device_lock(parent);
device_release_driver(&aru->udev->dev);
if (parent)
- up(&parent->sem);
+ device_unlock(parent);
usb_put_dev(aru->udev);
}
----
I'll let you know how it goes (light testing probably tomorrow, still
building).
-- Fernando "patiently waiting for 2.6.37-rtx"
:-)
On Fri, Mar 4, 2011 at 10:05 PM, Nathan Grennan
<[email protected]> wrote:
> On 03/04/2011 02:24 PM, Paul Gortmaker wrote:
>>
>> As a value add to the 2.6.34 long term release, I'm happy to also
>> announce the availability of 2.6.34-RT.
>>
>> You can find it in the v2.6.34-rt branch at:
>>
>> ? git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
>>
>> as a repository of patches. ?The v2.6.34-rt branch contains the latest
>> RT patches against the latest v2.6.34.8 kernel release. (The master
>> branch currently stops at v2.6.34 flat, i.e. 2.6.34.0 so to speak.)
>>
>> I've also created over 400 known working RT enabled bisection points
>> between 33 and 34 that you can make use of for testing. ?The details
>> on how/why those exist follows - read on if it is of interest to you.
>>
>> Note that all credit of the thinking and engineering of the RT stuff
>> lies with the original patch authors -- to be clear, I'm just doing
>> the grunt work of a carry forward here.
>>
>> Have fun with it, maybe drop me a note if you find it useful etc.
>
> I tried compiling this, and it quickly failed. Ideas?
Yes, it is the header cleanup done in the 33-->34 dev cycle, seen
between the rev list ^9e74e7c b66696e that changes slab.h use.
If you look at 8a2032dba45e24 *in the patch repo* you will see where
I fixed it for the baseline RT-enabled defconfig. (Meaning I did a
make defconfig, then enabled the RT options). Your config must be
setting something different/extra.
If you do a git show on the above rev list and on the fix in the patch
repo, you will probably be able to tell what is going on. Also please
send me your config (off list) so I can ensure it compiles properly myself.
Thanks,
Paul.
>
> I did this on a Fedora 14 x86_64 system with gcc-4.5.1-4.fc14.x86_64.
>
>
> Here is what I did:
>
> 1. Downloaded 2.6.34.8 tarball and extracted it
> 2. git clone -b v2.6.34-rt
> git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
> 3. cat ../rt-patches/series | xargs -i cat ../rt-patches/{} | patch -p1
> (Applied just fine)
> 4. Copied an 2.6.31-rt configuration to .config
> 5. Ran make oldconfig
> 6. make bzImage
>
>
> CHK include/linux/version.h
> CHK include/generated/utsrelease.h
> HOSTCC scripts/basic/fixdep
> HOSTCC scripts/basic/docproc
> HOSTCC scripts/basic/hash
> CC kernel/bounds.s
> GEN include/generated/bounds.h
> CC arch/x86/kernel/asm-offsets.s
> GEN include/generated/asm-offsets.h
> CALL scripts/checksyscalls.sh
> HOSTCC scripts/genksyms/genksyms.o
> HOSTCC scripts/genksyms/lex.o
> HOSTCC scripts/genksyms/parse.o
> HOSTLD scripts/genksyms/genksyms
> CC scripts/mod/empty.o
> HOSTCC scripts/mod/mk_elfconfig
> MKELF scripts/mod/elfconfig.h
> HOSTCC scripts/mod/file2alias.o
> HOSTCC scripts/mod/modpost.o
> HOSTCC scripts/mod/sumversion.o
> HOSTLD scripts/mod/modpost
> HOSTCC scripts/selinux/genheaders/genheaders
> HOSTCC scripts/selinux/mdp/mdp
> HOSTCC scripts/kallsyms
> HOSTCC scripts/conmakehash
> CC init/main.o
> init/main.c: In function ?do_one_initcall?:
> init/main.c:740:10: warning: ?calltime.tv64? may be used uninitialized in
> this function
> CHK include/generated/compile.h
> CC init/version.o
> CC init/do_mounts.o
> In file included from include/linux/suspend.h:8:0,
> from init/do_mounts.c:6:
> include/linux/mm.h: In function ?pte_lock_init?:
> include/linux/mm.h:1125:2: error: implicit declaration of function ?kmalloc?
> include/linux/mm.h:1125:12: warning: assignment makes pointer from integer
> without a cast
> include/linux/mm.h: In function ?pte_lock_deinit?:
> include/linux/mm.h:1137:2: error: implicit declaration of function ?kfree?
> In file included from include/linux/security.h:36:0,
> from init/do_mounts.c:8:
> include/linux/slab.h: At top level:
> include/linux/slab.h:144:6: warning: conflicting types for ?kfree?
> include/linux/mm.h:1137:2: note: previous implicit declaration of ?kfree?
> was here
> In file included from include/linux/slab.h:172:0,
> from include/linux/security.h:36,
> from init/do_mounts.c:8:
> include/linux/slab_def.h:128:52: error: conflicting types for ?kmalloc?
> include/linux/mm.h:1125:14: note: previous implicit declaration of ?kmalloc?
> was here
> make[1]: *** [init/do_mounts.o] Error 1
> make: *** [init] Error 2
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to [email protected]
> More majordomo info at ?http://vger.kernel.org/majordomo-info.html
>
On Mon, Mar 7, 2011 at 1:51 AM, Fernando Lopez-Lezcano
<[email protected]> wrote:
> On 03/04/2011 02:24 PM, Paul Gortmaker wrote:
>>
>> As a value add to the 2.6.34 long term release, I'm happy to also
>> announce the availability of 2.6.34-RT.
>>
>> You can find it in the v2.6.34-rt branch at:
>>
>> ? git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
>
> Great! Many thanks... Actually, impressive work.
>
> I'm trying to integrate/build it into my Fedora based rt packages for Planet
> CCRMA. Warning: I'm doing this on top of the 2.6.34.8-68 fc13 koji build, so
> rt is patched on top of the additional Fedora patches for that build (with a
> few trivial tweaks to patch cleanly). So I don't know if the following
> issues are due to that - I will investigate:
Thanks - I'm not surprised to see some fallout in new/updated drivers,
since they change so much per release and I didn't attempt to build every
driver under the sun - my core focus was on getting it stable with the main
defconfigs I could easily test. I'll have a look and integrate accordingly.
P.
>
> I found a couple of extra argument cases for __netif_tx_lock:
>
> --- a/drivers/net/qlcnic/qlcnic_main.c ?2010-05-16 17:17:36.000000000 -0400
> +++ b/drivers/net/qlcnic/qlcnic_main.c ?2011-03-06 20:08:16.000000000 -0500
> @@ -1881,7 +1881,7 @@
> ? ? ? ? ? ? ? ?smp_mb();
>
> ? ? ? ? ? ? ? ?if (netif_queue_stopped(netdev) && netif_carrier_ok(netdev))
> {
> - ? ? ? ? ? ? ? ? ? ? ? __netif_tx_lock(tx_ring->txq, smp_processor_id());
> + ? ? ? ? ? ? ? ? ? ? ? __netif_tx_lock(tx_ring->txq);
> ? ? ? ? ? ? ? ? ? ? ? ?if (qlcnic_tx_avail(tx_ring) > TX_STOP_THRESH) {
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?netif_wake_queue(netdev);
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?adapter->tx_timeo_cnt = 0;
> --- a/drivers/net/bnx2x_main.c ?2010-05-16 17:17:36.000000000 -0400
> +++ b/drivers/net/bnx2x_main.c ?2011-03-06 19:58:45.000000000 -0500
> @@ -976,7 +976,7 @@
> ? ? ? ? ? ? ? ? * stops the queue
> ? ? ? ? ? ? ? ? */
>
> - ? ? ? ? ? ? ? __netif_tx_lock(txq, smp_processor_id());
> + ? ? ? ? ? ? ? __netif_tx_lock(txq);
>
> ? ? ? ? ? ? ? ?if ((netif_tx_queue_stopped(txq)) &&
> ? ? ? ? ? ? ? ? ? ?(bp->state == BNX2X_STATE_OPEN) &&
> ----
>
> a missing include (incomplete type errors for semaphores):
>
> --- a/drivers/net/mlx4/mlx4.h ? 2010-05-16 17:17:36.000000000 -0400
> +++ b/drivers/net/mlx4/mlx4.h ? 2011-03-06 20:07:05.000000000 -0500
> @@ -37,6 +37,7 @@
> ?#ifndef MLX4_H
> ?#define MLX4_H
>
> +#include <linux/semaphore.h>
> ?#include <linux/mutex.h>
> ?#include <linux/radix-tree.h>
> ?#include <linux/timer.h>
> ----
>
> and this (found compilation fix here:
> http://groups.google.com/group/linux.kernel/browse_thread/thread/c4f0fd1ca9298c45/18316184209e6e84?show_docid=18316184209e6e84&fwc=1):
>
> --- a/drivers/net/wireless/ath/ar9170/usb.c ? ? 2011-03-06
> 19:13:12.000000000 -0500
> +++ b/drivers/net/wireless/ath/ar9170/usb.c ? ? 2011-03-06
> 20:13:06.000000000 -0500
> @@ -741,10 +741,10 @@
>
> ? ? ? ?/* unbind anything failed */
> ? ? ? ?if (parent)
> - ? ? ? ? ? ? ? down(&parent->sem);
> + ? ? ? ? ? ? ? device_lock(parent);
> ? ? ? ?device_release_driver(&aru->udev->dev);
> ? ? ? ?if (parent)
> - ? ? ? ? ? ? ? up(&parent->sem);
> + ? ? ? ? ? ? ? device_unlock(parent);
>
> ? ? ? ?usb_put_dev(aru->udev);
> ?}
> ----
>
> I'll let you know how it goes (light testing probably tomorrow, still
> building).
>
> -- Fernando "patiently waiting for 2.6.37-rtx"
> :-)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at ?http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at ?http://www.tux.org/lkml/
>
On 03/06/2011 10:51 PM, Fernando Lopez-Lezcano wrote:
> On 03/04/2011 02:24 PM, Paul Gortmaker wrote:
>> As a value add to the 2.6.34 long term release, I'm happy to also
>> announce the availability of 2.6.34-RT.
>>
>> You can find it in the v2.6.34-rt branch at:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
>
> Great! Many thanks... Actually, impressive work.
>
> I'm trying to integrate/build it into my Fedora based rt packages for
> Planet CCRMA. Warning: I'm doing this on top of the 2.6.34.8-68 fc13
> koji build, so rt is patched on top of the additional Fedora patches for
> that build (with a few trivial tweaks to patch cleanly). So I don't know
> if the following issues are due to that...
Caveat above still applies...
So far mixed success:
- Fedora 14 desktop w/radeon graphics: OK
- Lenovo laptop, Fedora 13, Nvidia binary driver (barf!): OK
- EeePC netbook, Fedora 13, Intel Mobile 945GME graphics: X does not start
- Lenovo laptop, fedora 14, Intel 9xx graphics (at home): X does not start
These last two share Intel graphics and the problem seems to be the
same, something to do with dri. The standard Fedora kernel works fine so
it must be some rt interaction (perhaps with the added Fedora patches).
But it is a good start!
-- Fernando
On Friday 04 March 2011 23:24:15 Paul Gortmaker wrote:
> As a value add to the 2.6.34 long term release, I'm happy to also
> announce the availability of 2.6.34-RT.
>
> You can find it in the v2.6.34-rt branch at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
>
> as a repository of patches. The v2.6.34-rt branch contains the latest
> RT patches against the latest v2.6.34.8 kernel release. (The master
> branch currently stops at v2.6.34 flat, i.e. 2.6.34.0 so to speak.)
>
> I've also created over 400 known working RT enabled bisection points
> between 33 and 34 that you can make use of for testing. The details
> on how/why those exist follows - read on if it is of interest to you.
A great achievement, but I think your approach can be improved in
case you want to do it again for another kernel version.
The problem with your method is that you have more work that depends
on the amount of changes in upstream than the work that depends on
the size of the patch set.
Note that I'm not using the -rt patch set myself, but I have some
experience with managing other patch sets.
> For those who track these things, you will know that the most recent
> release of RT was based on v2.6.33, and in turn, it was created by
> merging forward the mainline kernel.org v2.6.33 content into the older
> v2.6.31 RT tree. There was no rebase.
I think 2.6.31 was done the right way, with a number of topic branches
that got merged into one release branch. This made it fairly easy to
come out with a quick-and-dirty forward port to 2.6.33, but from that
point on, it became much harder to port it again.
> There are roughly 500 RT patches, and literally 10,000+ commits between
> v2.6.33 and v2.6.34. So if one was to move things ahead all in one go,
> there can be roughly 5 million things that can go wrong. Maybe some sharp
> person can move those ahead all in one shot, and then figure out the
> resulting inevitable runtime breakage, but that isn't me. After all, a
> man has to know his limitations, and I'm pretty much just a patch monkey.
Doing it all in one shot is not really possible, I fully agree. However,
it's absolutely possible to port a small self-contained series of
interdependent patches over 10000 or more other changesets. The trick
is to split the 500 patches into e.g. 20 branches of less than 50 patches,
where each branch by itself adds one aspect of the full tree and can
be built and run independently of the other branches.
This is a lot of upfront work, but less work to keep it running if you
maintain the branches after the initial release (unlike the 2.6.33-rt
tree).
I've played around with your tree a bit and it seems fairly easy
to split into multiple series, e.g. one for each arch, one for
drivers, mm, rtmutex, mm using git-rebase and some heuristics.
Making sure that each of those branches actually makes sense
by itself may be a different story.
When you have these branches, you can merge them all into one release
branch, similar to how linux-next works, and then just merge updates
from the branches. To port the entire tree to a new upstream, you
look at each branch individually and do a merge, a rebase, a rewrite
or whatever is appropriate for that branch. Finally, you merge the
new branches into another release.
Arnd
On 11-03-07 03:01 PM, Arnd Bergmann wrote:
> On Friday 04 March 2011 23:24:15 Paul Gortmaker wrote:
>> As a value add to the 2.6.34 long term release, I'm happy to also
>> announce the availability of 2.6.34-RT.
>>
>> You can find it in the v2.6.34-rt branch at:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
>>
>> as a repository of patches. The v2.6.34-rt branch contains the latest
>> RT patches against the latest v2.6.34.8 kernel release. (The master
>> branch currently stops at v2.6.34 flat, i.e. 2.6.34.0 so to speak.)
>>
>> I've also created over 400 known working RT enabled bisection points
>> between 33 and 34 that you can make use of for testing. The details
>> on how/why those exist follows - read on if it is of interest to you.
>
> A great achievement, but I think your approach can be improved in
> case you want to do it again for another kernel version.
>
> The problem with your method is that you have more work that depends
> on the amount of changes in upstream than the work that depends on
> the size of the patch set.
You might think so, but that really isn't the case at all. The
work involved is dictated only by the upstream subset that have
an impact on the RT commits. If there was a stream of a thousand
contiguous upstream commits that did not break patch application,
or the run time boot test, I'd not have to lift a finger. Scripts
applied the commits and ensured a continuous sanity. If I was to
do it again, I'd employ the exact same tactic in a heartbeat.
>
> Note that I'm not using the -rt patch set myself, but I have some
> experience with managing other patch sets.
>
>> For those who track these things, you will know that the most recent
>> release of RT was based on v2.6.33, and in turn, it was created by
>> merging forward the mainline kernel.org v2.6.33 content into the older
>> v2.6.31 RT tree. There was no rebase.
>
> I think 2.6.31 was done the right way, with a number of topic branches
> that got merged into one release branch. This made it fairly easy to
> come out with a quick-and-dirty forward port to 2.6.33, but from that
> point on, it became much harder to port it again.
>
>> There are roughly 500 RT patches, and literally 10,000+ commits between
>> v2.6.33 and v2.6.34. So if one was to move things ahead all in one go,
>> there can be roughly 5 million things that can go wrong. Maybe some sharp
>> person can move those ahead all in one shot, and then figure out the
>> resulting inevitable runtime breakage, but that isn't me. After all, a
>> man has to know his limitations, and I'm pretty much just a patch monkey.
>
> Doing it all in one shot is not really possible, I fully agree. However,
> it's absolutely possible to port a small self-contained series of
> interdependent patches over 10000 or more other changesets. The trick
> is to split the 500 patches into e.g. 20 branches of less than 50 patches,
> where each branch by itself adds one aspect of the full tree and can
> be built and run independently of the other branches.
This could make sense if they were real independent functionality
boundaries that happened to give you this nice uniform division.
But it is unlikely in practice. And adding an artificial division
may not work if there are implicit dependencies between the patches
that a patch monkey like me might not see. Another added cost is that
I would now have 20 branches to individually compile and boot test.
Maybe I could get away with more sparse testing, but maybe not.
In any case, I really don't see what you are trying to propose here
as being really all that fundamentally different -- you want to move
a small set of patches over a bigger set of patches. So you can say
I'm moving a set of 20 odd patches, which were merged by Linus (and hence
already known to be treatable as a whole self contained unit) over a larger
set of some 500 RT patches. Does it matter which group you call the
big set and which one the small set?
>
> This is a lot of upfront work, but less work to keep it running if you
> maintain the branches after the initial release (unlike the 2.6.33-rt
> tree).
>
> I've played around with your tree a bit and it seems fairly easy
> to split into multiple series, e.g. one for each arch, one for
> drivers, mm, rtmutex, mm using git-rebase and some heuristics.
> Making sure that each of those branches actually makes sense
> by itself may be a different story.
>
> When you have these branches, you can merge them all into one release
> branch, similar to how linux-next works, and then just merge updates
> from the branches. To port the entire tree to a new upstream, you
> look at each branch individually and do a merge, a rebase, a rewrite
> or whatever is appropriate for that branch. Finally, you merge the
> new branches into another release.
If any knowledge is embedded in a merge commit, it makes it difficult to
extract a commit and deal with it in isolation, should it be something
that has a chance of being pushed upstream. A merge commit makes sense
when there is value in capturing that patches X-->Y were developed and
tested on baseline Z. On the other hand, there is value in showing instead
X'--->Y' which are meant to apply to a newer baseline Z'. For the RT
case, I think the latter is more valuable (at least to me) so I'll be
sticking to a linear merge-free patchset on anything RT related.
Thanks for the comments though -- at least I know someone was reading
the process I used and thinking about it.
Paul.
>
> Arnd
On 03/07/2011 12:00 PM, Fernando Lopez-Lezcano wrote:
> On 03/06/2011 10:51 PM, Fernando Lopez-Lezcano wrote:
>> On 03/04/2011 02:24 PM, Paul Gortmaker wrote:
>>> As a value add to the 2.6.34 long term release, I'm happy to also
>>> announce the availability of 2.6.34-RT.
>>>
>>> You can find it in the v2.6.34-rt branch at:
>>>
>>> git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
>>
>> Great! Many thanks... Actually, impressive work.
>>
>> I'm trying to integrate/build it into my Fedora based rt packages for
>> Planet CCRMA. Warning: I'm doing this on top of the 2.6.34.8-68 fc13
>> koji build, so rt is patched on top of the additional Fedora patches for
>> that build (with a few trivial tweaks to patch cleanly). So I don't know
>> if the following issues are due to that...
>
> Caveat above still applies...
...
> - EeePC netbook, Fedora 13, Intel Mobile 945GME graphics: X does not start
> - Lenovo laptop, fedora 14, Intel 9xx graphics (at home): X does not start
> These last two share Intel graphics and the problem seems to be the
> same, something to do with dri.
This is what I'm getting (dmesg|grep i915):
i915 0000:00:02.0 PCI INT A -> GSI 16 (level, low) -> IRQ 16
i915 0000:00:02.0 setting latency timer to 64
i915 0000:00:02.0 PCI INT A disabled
i915: probe of 0000:00:02.0 failed with error -5
Same kernel w/o rt...:
i915 0000:00:02.0 PCI INT A -> GSI 16 (level, low) -> IRQ 16
i915 0000:00:02.0 setting latency timer to 64
[drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
So, some problem with irq processing?
Where should I go look?
-- Fernando
On 11-03-07 03:41 PM, Fernando Lopez-Lezcano wrote:
> On 03/07/2011 12:00 PM, Fernando Lopez-Lezcano wrote:
>> On 03/06/2011 10:51 PM, Fernando Lopez-Lezcano wrote:
>>> On 03/04/2011 02:24 PM, Paul Gortmaker wrote:
>>>> As a value add to the 2.6.34 long term release, I'm happy to also
>>>> announce the availability of 2.6.34-RT.
>>>>
>>>> You can find it in the v2.6.34-rt branch at:
>>>>
>>>> git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
>>>
>>> Great! Many thanks... Actually, impressive work.
>>>
>>> I'm trying to integrate/build it into my Fedora based rt packages for
>>> Planet CCRMA. Warning: I'm doing this on top of the 2.6.34.8-68 fc13
>>> koji build, so rt is patched on top of the additional Fedora patches for
>>> that build (with a few trivial tweaks to patch cleanly). So I don't know
>>> if the following issues are due to that...
>>
>> Caveat above still applies...
> ...
>> - EeePC netbook, Fedora 13, Intel Mobile 945GME graphics: X does not start
>> - Lenovo laptop, fedora 14, Intel 9xx graphics (at home): X does not start
>> These last two share Intel graphics and the problem seems to be the
>> same, something to do with dri.
>
> This is what I'm getting (dmesg|grep i915):
>
> i915 0000:00:02.0 PCI INT A -> GSI 16 (level, low) -> IRQ 16
> i915 0000:00:02.0 setting latency timer to 64
> i915 0000:00:02.0 PCI INT A disabled
> i915: probe of 0000:00:02.0 failed with error -5
>
> Same kernel w/o rt...:
>
> i915 0000:00:02.0 PCI INT A -> GSI 16 (level, low) -> IRQ 16
> i915 0000:00:02.0 setting latency timer to 64
> [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
>
> So, some problem with irq processing?
> Where should I go look?
Try v2.6.33-rt -- see if it is a new regression, or was a problem that
already existed and has just been carried forward.
P.
> -- Fernando
On 03/07/2011 12:52 PM, Paul Gortmaker wrote:
> On 11-03-07 03:41 PM, Fernando Lopez-Lezcano wrote:
>> On 03/07/2011 12:00 PM, Fernando Lopez-Lezcano wrote:
>>> On 03/06/2011 10:51 PM, Fernando Lopez-Lezcano wrote:
>>>> On 03/04/2011 02:24 PM, Paul Gortmaker wrote:
>>>>> As a value add to the 2.6.34 long term release, I'm happy to also
>>>>> announce the availability of 2.6.34-RT.
>>>>>
>>>>> You can find it in the v2.6.34-rt branch at:
>>>>>
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
>>>>
>>>> Great! Many thanks... Actually, impressive work.
>>>>
>>>> I'm trying to integrate/build it into my Fedora based rt packages for
>>>> Planet CCRMA. Warning: I'm doing this on top of the 2.6.34.8-68 fc13
>>>> koji build, so rt is patched on top of the additional Fedora patches for
>>>> that build (with a few trivial tweaks to patch cleanly). So I don't know
>>>> if the following issues are due to that...
>>>
>>> Caveat above still applies...
>> ...
>>> - EeePC netbook, Fedora 13, Intel Mobile 945GME graphics: X does not start
>>> - Lenovo laptop, fedora 14, Intel 9xx graphics (at home): X does not start
>>> These last two share Intel graphics and the problem seems to be the
>>> same, something to do with dri.
>>
>> This is what I'm getting (dmesg|grep i915):
>>
>> i915 0000:00:02.0 PCI INT A -> GSI 16 (level, low) -> IRQ 16
>> i915 0000:00:02.0 setting latency timer to 64
>> i915 0000:00:02.0 PCI INT A disabled
>> i915: probe of 0000:00:02.0 failed with error -5
>>
>> Same kernel w/o rt...:
>>
>> i915 0000:00:02.0 PCI INT A -> GSI 16 (level, low) -> IRQ 16
>> i915 0000:00:02.0 setting latency timer to 64
>> [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
>>
>> So, some problem with irq processing?
>> Where should I go look?
>
> Try v2.6.33-rt -- see if it is a new regression, or was a problem that
> already existed and has just been carried forward.
2.6.33.x-rt30 works on all of the above so it seems this is new... (yuck :-)
(just in case, this is not pristine 2.6.34.8 + rt git)
-- Fernando
> Yes, it is the header cleanup done in the 33-->34 dev cycle, seen
> between the rev list ^9e74e7c b66696e that changes slab.h use.
>
> If you look at 8a2032dba45e24 *in the patch repo* you will see where
> I fixed it for the baseline RT-enabled defconfig. (Meaning I did a
> make defconfig, then enabled the RT options). Your config must be
> setting something different/extra.
>
> If you do a git show on the above rev list and on the fix in the patch
> repo, you will probably be able to tell what is going on. Also please
> send me your config (off list) so I can ensure it compiles properly myself.
>
I found the cause. The patching was incomplete. The correct patch
command is the one below, removing the comment lines.
cat ../rt-patches/series | grep -v '\#' | xargs -i cat ../rt-patches/{}
| patch -p1
On 11-03-07 04:36 PM, Nathan Grennan wrote:
>
>> Yes, it is the header cleanup done in the 33-->34 dev cycle, seen
>> between the rev list ^9e74e7c b66696e that changes slab.h use.
>>
>> If you look at 8a2032dba45e24 *in the patch repo* you will see where
>> I fixed it for the baseline RT-enabled defconfig. (Meaning I did a
>> make defconfig, then enabled the RT options). Your config must be
>> setting something different/extra.
>>
>> If you do a git show on the above rev list and on the fix in the patch
>> repo, you will probably be able to tell what is going on. Also please
>> send me your config (off list) so I can ensure it compiles properly myself.
>>
>
> I found the cause. The patching was incomplete. The correct patch
> command is the one below, removing the comment lines.
>
> cat ../rt-patches/series | grep -v '\#' | xargs -i cat ../rt-patches/{}
> | patch -p1
Folks should really use git, or quilt, or guilt, or git quilt-import, or
for that matter, *anything* that at least checks exit status.
I like the fact that "git am" is quite strict about what it accepts
and applies, so I'll typically wrap things around that core operation.
Here is a completely trivial example of using git am:
cat `grep ^[a-zA-Z0-9] series` > /tmp/foo.mbox
git am /tmp/foo.mbox
Paul.
On 03/07/2011 06:44 AM, Paul Gortmaker wrote:
> On Mon, Mar 7, 2011 at 1:51 AM, Fernando Lopez-Lezcano
> <[email protected]> wrote:
>> On 03/04/2011 02:24 PM, Paul Gortmaker wrote:
>>> As a value add to the 2.6.34 long term release, I'm happy to also
>>> announce the availability of 2.6.34-RT.
>>>
>>> You can find it in the v2.6.34-rt branch at:
>>>
>>> git://git.kernel.org/pub/scm/linux/kernel/git/paulg/rt-patches.git
>> Great! Many thanks... Actually, impressive work.
>>
>> I'm trying to integrate/build it into my Fedora based rt packages for Planet
>> CCRMA. Warning: I'm doing this on top of the 2.6.34.8-68 fc13 koji build, so
>> rt is patched on top of the additional Fedora patches for that build (with a
>> few trivial tweaks to patch cleanly). So I don't know if the following
>> issues are due to that - I will investigate:
> Thanks - I'm not surprised to see some fallout in new/updated drivers,
> since they change so much per release and I didn't attempt to build every
> driver under the sun - my core focus was on getting it stable with the main
> defconfigs I could easily test. I'll have a look and integrate accordingly.
>
> P.
drivers/net/bnx2x_main.c
drivers/staging/comedi/drivers/quatech_daqp_cs.c
These drivers didn't compile for me. bnx2x_main was an easy fix, but I
didn't bother with quatech_daqp_cs.
It's not working here. That has been the case with earlier versions of the
RT kernel I have tried aswell.
I am on a MSI7380 mainboard. Do you need more info?
http://www.paradoxuncreated.com/tmp/rterror.jpg
On 11-03-07 03:54 PM, Fernando Lopez-Lezcano wrote:
> On 03/07/2011 12:52 PM, Paul Gortmaker wrote:
>> On 11-03-07 03:41 PM, Fernando Lopez-Lezcano wrote:
[...]
>>> ...
>>>> - EeePC netbook, Fedora 13, Intel Mobile 945GME graphics: X does not start
>>>> - Lenovo laptop, fedora 14, Intel 9xx graphics (at home): X does not start
>>>> These last two share Intel graphics and the problem seems to be the
>>>> same, something to do with dri.
>>>
>>> This is what I'm getting (dmesg|grep i915):
>>>
>>> i915 0000:00:02.0 PCI INT A -> GSI 16 (level, low) -> IRQ 16
>>> i915 0000:00:02.0 setting latency timer to 64
>>> i915 0000:00:02.0 PCI INT A disabled
>>> i915: probe of 0000:00:02.0 failed with error -5
>>>
>>> Same kernel w/o rt...:
>>>
>>> i915 0000:00:02.0 PCI INT A -> GSI 16 (level, low) -> IRQ 16
>>> i915 0000:00:02.0 setting latency timer to 64
>>> [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
>>>
>>> So, some problem with irq processing?
>>> Where should I go look?
>>
>> Try v2.6.33-rt -- see if it is a new regression, or was a problem that
>> already existed and has just been carried forward.
>
> 2.6.33.x-rt30 works on all of the above so it seems this is new... (yuck :-)
> (just in case, this is not pristine 2.6.34.8 + rt git)
I'm pretty sure I know what this is. If you cherry pick the two
commits from tip, namely:
e89fab2bf44afccaa48ffb8ae33c03f4f3dcd208 and
cd4ac9b7a1c80cfcd9b8e76a1f6855a0329d982f
then I'll bet your i915 gets detected OK. I was forced to
drop these because they allow ATOMIC_IOMAP to be selected
for RT & HIGHMEM, which causes sporadic bootup Oops (approx
one in 10) for intel ata_piix. (This problem isn't new; it
is present in v2.6.33.7.2-rt30 by the way.)
But w/o ATOMIC_IOMAP, people on 32bit stuff end up trying to
grab a big chunk of address space that they can't afford, and
so the driver init fails.
The proper solution needs more thought, but in the meantime I'll
bet the above makes your i915 work again.
Related info to this can be found here:
http://lkml.indiana.edu/hypermail/linux/kernel/0811.0/00411.html
Tip repo, if you don't have it, is here:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git
Paul.
>
> -- Fernando
On Monday 07 March 2011, Paul Gortmaker wrote:
> On 11-03-07 03:01 PM, Arnd Bergmann wrote:
>
> > The problem with your method is that you have more work that depends
> > on the amount of changes in upstream than the work that depends on
> > the size of the patch set.
>
> You might think so, but that really isn't the case at all. The
> work involved is dictated only by the upstream subset that have
> an impact on the RT commits. If there was a stream of a thousand
> contiguous upstream commits that did not break patch application,
> or the run time boot test, I'd not have to lift a finger. Scripts
> applied the commits and ensured a continuous sanity. If I was to
> do it again, I'd employ the exact same tactic in a heartbeat.
Fair enough, everyone has their own ways of dealing with this, I just
wanted to make sure that you are aware that there are other options
that people have used successfully.
Obviously, the number of conflicts is about the same, independent
of the way you partition the work.
> This could make sense if they were real independent functionality
> boundaries that happened to give you this nice uniform division.
> But it is unlikely in practice.
You have obviously worked a lot with these patches, so you have
a better understanding than me of how interrelated the patches
actually are. From a brief look at the contents and from the fact
that there already were independent branches in the 2.6.31-rt
kernel, my guess was that it is very possible to do a reasonable
split. Someone would have to seriously try it in order to know
for sure.
> And adding an artificial division
> may not work if there are implicit dependencies between the patches
> that a patch monkey like me might not see. Another added cost is that
> I would now have 20 branches to individually compile and boot test.
> Maybe I could get away with more sparse testing, but maybe not.
>
> In any case, I really don't see what you are trying to propose here
> as being really all that fundamentally different -- you want to move
> a small set of patches over a bigger set of patches. So you can say
> I'm moving a set of 20 odd patches, which were merged by Linus (and hence
> already known to be treatable as a whole self contained unit) over a larger
> set of some 500 RT patches. Does it matter which group you call the
> big set and which one the small set?
The important difference IMHO is that when you have topic branches,
you can partition the work into hard and easy tasks at your own
choice, rather than having to go through all the merges that someone
else did in the same order:
* Half the branches will have little or no conflicts with the new
upstream version, so you immediately reduce the amount of dull
work by moving the branch up to the new upstream.
* Some particularly complex branches will have patches from only
one or two people who know that code really well, so you can ask
them to rebase the branch.
* A branch that has complex conflicts may turn out to be relevant
only for a small group of users, e.g. a specific architecture,
so you can defer rebasing it until someone else is interested
in doing the work.
* Some feature from one branch (e.g. BKL or raw spinlocks) may
have been solved upstream in a different way. When the branch
is self-contained, you can immediately drop the branch instead
of having to look through the 500 patches for possible
interactions with the new upstream code.
Another very significant difference is that having topic branches
makes it much easier to merge code upstream. There may be some
branches that are just in need of a bit care to be acceptable to
upstream, and other branches contain the code that would need
a full redesign from scratch before anyone could seriously submit
them. In both cases, isolating the changes in a branch helps you
make forward progress towards upstream.
Arnd
On 03/07/2011 04:52 PM, Paul Gortmaker wrote:
> On 11-03-07 03:54 PM, Fernando Lopez-Lezcano wrote:
>> On 03/07/2011 12:52 PM, Paul Gortmaker wrote:
>>> On 11-03-07 03:41 PM, Fernando Lopez-Lezcano wrote:
>
> [...]
>
>>>> ...
>>>>> - EeePC netbook, Fedora 13, Intel Mobile 945GME graphics: X does not start
>>>>> - Lenovo laptop, fedora 14, Intel 9xx graphics (at home): X does not start
>>>>> These last two share Intel graphics and the problem seems to be the
>>>>> same, something to do with dri.
>>>>
>>>> This is what I'm getting (dmesg|grep i915):
>>>>
>>>> i915 0000:00:02.0 PCI INT A -> GSI 16 (level, low) -> IRQ 16
>>>> i915 0000:00:02.0 setting latency timer to 64
>>>> i915 0000:00:02.0 PCI INT A disabled
>>>> i915: probe of 0000:00:02.0 failed with error -5
>>>>
>>>> Same kernel w/o rt...:
>>>>
>>>> i915 0000:00:02.0 PCI INT A -> GSI 16 (level, low) -> IRQ 16
>>>> i915 0000:00:02.0 setting latency timer to 64
>>>> [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
>>>>
>>>> So, some problem with irq processing?
>>>> Where should I go look?
>>>
>>> Try v2.6.33-rt -- see if it is a new regression, or was a problem that
>>> already existed and has just been carried forward.
>>
>> 2.6.33.x-rt30 works on all of the above so it seems this is new... (yuck :-)
>> (just in case, this is not pristine 2.6.34.8 + rt git)
>
> I'm pretty sure I know what this is. If you cherry pick the two
> commits from tip, namely:
>
> e89fab2bf44afccaa48ffb8ae33c03f4f3dcd208 and
> cd4ac9b7a1c80cfcd9b8e76a1f6855a0329d982f
>
> then I'll bet your i915 gets detected OK.
It does, thank you very much! I tested on the EeePc only but I suspect
the Lenovo will be happy as well (it is at home right now). I have not
hit any other problems so far with 2.6.34.8 + rtx (0?)...
-- Fernando
