Hi all,
I would like to write a short article about the new development
process discussed during last Linux Kernel Developers' Summit for my
local LUG.
Since I'm not able to find an accurate report of what has been
discussed during that meeting I try to summariza what is my
understanding of the current process:
The are two kind of releases, 2.6.x kerneles and 2.6.x.y
2.6.x.y are maintained by the "stable" team (stable at kernel dot
org), are released amost every week.
Here is the latest ( I hope ) announce I found in the archive
including an explanation of how stable development works:
Rules on what kind of patches are accepted, and what ones are not, into
the "-stable" tree:
- It must be obviously correct and tested.
- It can not bigger than 100 lines, with context.
- It must fix only one thing.
- It must fix a real bug that bothers people (not a, "This could be a
problem..." type thing.)
- It must fix a problem that causes a build error (but not for things
marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
security issue, or some "oh, that's not good" issue. In short,
something critical.
- No "theoretical race condition" issues, unless an explanation of how
the race can be exploited.
- It can not contain any "trivial" fixes in it (spelling changes,
whitespace cleanups, etc.)
- It must be accepted by the relevant subsystem maintainer.
- It must follow Documentation/SubmittingPatches rules.
Procedure for submitting patches to the -stable tree:
- Send the patch, after verifying that it follows the above rules, to
[email protected].
- The sender will receive an ack when the patch has been accepted into
the queue, or a nak if the patch is rejected. This response might
take a few days, according to the developer's schedules.
- If accepted, the patch will be added to the -stable queue, for review
by other developers.
- Security patches should not be sent to this alias, but instead to the
documented [email protected].
Review cycle:
- When the -stable maintainers decide for a review cycle, the patches
will be sent to the review committee, and the maintainer of the
affected area of the patch (unless the submitter is the maintainer of
the area) and CC: to the linux-kernel mailing list.
- The review committee has 48 hours in which to ack or nak the patch.
- If the patch is rejected by a member of the committee, or linux-kernel
members object to the patch by bringing up issues that the maintainer
and members did not realize, the patch will be dropped from the
queue.
- At the end of the review cycle, the acked patches will be added to
the latest -stable release, and a new -stable release will happen.
- Security patches will be accepted into the -stable tree directly from
the security kernel team, and not go through the normal review cycle.
Contact the kernel security team for more details on this procedure.
Review committe:
- This will be made up of a number of kernel developers who have
volunteered for this task, and a few that haven't.
2.6.x kernel are maintained by Linus Torvalds and Andrew Morton (both
are from OSDL) and the development is as follow:
- As soon a new kernel is released a two weeks windows is open, during
this period of time maintainers can submit big diffs to Linus (usually
the patched sitted in -mm kernels for a few weeks). Preferred way to
submit big changes is using GIT ( more information about GIT at
http://git.or.cz/ and
http://www.kernel.org/pub/software/scm/git/docs/tutorial.html).
- After two weeks a -rc1 kernel is released and now is possible to
push only patches that do not include new functionalities)
- Aftwer two weeks -rc2 is released
- Process continues until the kernel is considered "ready", the
process should last three months ( 4 kernels per yeard should be
released).
I'm sure I'm missing lot of details so I would really appreciate if
you could support me in writing this article that hopefully will be
usefull outside my LUG as well :-)
Thanks in advance.
--
Paolo
On 10/29/05, Paolo Ciarrocchi <[email protected]> wrote:
> 2.6.x kernel are maintained by Linus Torvalds and Andrew Morton (both
> are from OSDL) and the development is as follow:
Linus maintains the 2.6.x kernel, and Andrew maintains the "-mm" tree
(which is used as a testing ground for anything non-trivial before it is
fed into Linus' tree).
> - As soon a new kernel is released a two weeks windows is open, during
> this period of time maintainers can submit big diffs to Linus (usually
> the patched sitted in -mm kernels for a few weeks). Preferred way to
> submit big changes is using GIT ( more information about GIT at
> http://git.or.cz/ and
> http://www.kernel.org/pub/software/scm/git/docs/tutorial.html).
> - After two weeks a -rc1 kernel is released and now is possible to
> push only patches that do not include new functionalities)
Initially Linus said that he would only accept e-mail patches after
rc1 ... but common sense prevailed and he later clarified to say that
git merges could still be used, but only to include bug fixes.
Also note that a whole new driver (or filesystem) might be accepted
after -rc1 because there is no risk of causing regressions with
such a change.
> - After two weeks -rc2 is released
There isn't really a hard schedule for when -rcN for N>1 are released.
> - Process continues until the kernel is considered "ready", the
> process should last three months ( 4 kernels per yeard should be
> released).
IIRC the goal was to make a release in around 8 weeks (which would
be closer to six per year). But you have the important part, which is
that Linus doesn't make the release until it is "ready". 2.6.13 was
released on August 28th, and 2.6.14 on October 27th ... so we
appear to have hit the goal this time through.
-Tony
On Sat, Oct 29, 2005 at 11:57:30AM -0700, Tony Luck wrote:
> > - Process continues until the kernel is considered "ready", the
> > process should last three months ( 4 kernels per yeard should be
> > released).
> IIRC the goal was to make a release in around 8 weeks (which would
> be closer to six per year). But you have the important part, which is
> that Linus doesn't make the release until it is "ready". 2.6.13 was
> released on August 28th, and 2.6.14 on October 27th ... so we
> appear to have hit the goal this time through.
However, three months is _far_ too long. Look at what is happening
post-2.6.14? There's loads of really huge changes going in which
were backed up over the last two months.
Continuing like this will just push the release of each kernel further
and further out as there's more stuff merged during the mayhem than
can possibly be properly reviewed and tested.
Shorter release cycles means that there's less mayhem, which in turn
means more time to review.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
On Sat, 29 Oct 2005, Russell King wrote:
> On Sat, Oct 29, 2005 at 11:57:30AM -0700, Tony Luck wrote:
> > IIRC the goal was to make a release in around 8 weeks (which would
> > be closer to six per year). But you have the important part, which is
> > that Linus doesn't make the release until it is "ready". 2.6.13 was
> > released on August 28th, and 2.6.14 on October 27th ... so we
> > appear to have hit the goal this time through.
>
> However, three months is _far_ too long. Look at what is happening
> post-2.6.14? There's loads of really huge changes going in which
> were backed up over the last two months.
Yes. Three months would be too much. I considered even 2.6.14 to be
delayed by at least one week. So 8 weeks is certainly better than it used
to be, but I think 6 weeks should be the goal-post.
But to hit that, we'd might need to shrink the "merge window" from two
weeks to just one, otherwise there's not enough time to calm down. With
most of the real development happening during the previous calm-down
stage, that might not be impossible: we'd only have one week for merges,
but that isn't the week when development actually happens, so who cares?
Everything said, I think 2.6.13->14 worked well enough, even if it's hard
to say how well a process works after one release. Considering that 2.6.13
had the painful PCI changes (you may not have noticed too much, since they
were x86 only) and there were some potentially painful SCSI changes in the
.14 early merges, so it's not like 13->14 was an "easy" release - so the
process certainly _seems_ to be workable.
So I'm planning on continuing with it unchanged for now. Two-week merge
window until -rc1, and then another -rc kernel roughly every week until
release. With the goal being 6 weeks, and 8 weeks being ok.
I don't think anybody has been really unhappy with this approach? Hmm?
Linus
Hi Linus,
This is my first mail to the Kernel tree:-
> Everything said, I think 2.6.13->14 worked well enough, even if it's hard
> to say how well a process works after one release. Considering that 2.6.13
> had the painful PCI changes (you may not have noticed too much, since they
> were x86 only) and there were some potentially painful SCSI changes in the
> .14 early merges, so it's not like 13->14 was an "easy" release - so the
> process certainly _seems_ to be workable.
Will you please throw more light on the *painful* PCI & SCSI changees?
> I don't think anybody has been really unhappy with this approach? Hmm?
honestly, I did download 14 yesterday and made the kernel ready to
compile. today I've seen 14git1 patch! Am trying to understand why is
it this kind of kernel development model...
thanks,
Akula
Linus Torvalds <[email protected]> writes:
>
> But to hit that, we'd might need to shrink the "merge window" from two
> weeks to just one, otherwise there's not enough time to calm down. With
Please don't. Even the two weeks are too short IMHO, because it is
hard to digest so much code in such a short time and also it is not
always easy for maintainers to hit such short time windows for sending
patches.
> I don't think anybody has been really unhappy with this approach? Hmm?
The long freeze periods were nothing much happens are painful. It
would be better to have some more overlap of merging and stabilizing
(stable does that already kind of, but not enough)
-Andi
On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> Linus Torvalds <[email protected]> writes:
> > I don't think anybody has been really unhappy with this approach? Hmm?
>
> The long freeze periods were nothing much happens are painful. It
> would be better to have some more overlap of merging and stabilizing
> (stable does that already kind of, but not enough)
Violently agree. I find the long freeze periods painful and very very
very boring, to the point of looking for other stuff to do (such as
cleaning up bits of the kernel and queuing mega-patches for the next
round of merging.)
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
On Sun, Oct 30, 2005 at 02:14:39AM +0530, Akula2 wrote:
> Hi Linus,
>
> This is my first mail to the Kernel tree:-
>
> > Everything said, I think 2.6.13->14 worked well enough, even if it's hard
> > to say how well a process works after one release. Considering that 2.6.13
> > had the painful PCI changes (you may not have noticed too much, since they
> > were x86 only) and there were some potentially painful SCSI changes in the
> > .14 early merges, so it's not like 13->14 was an "easy" release - so the
> > process certainly _seems_ to be workable.
>
> Will you please throw more light on the *painful* PCI & SCSI changees?
If it worked for you, it wasn't painful :)
But for PCI, we changed the way we did resource allocation for devices,
which caused a lot of odd problems on some machines.
thanks,
greg k-h
On 10/29/05, Paolo Ciarrocchi <[email protected]> wrote:
> Hi all,
> I would like to write a short article about the new development
> process discussed during last Linux Kernel Developers' Summit for my
> local LUG.
>
> Since I'm not able to find an accurate report of what has been
> discussed during that meeting I try to summariza what is my
> understanding of the current process:
>
> The are two kind of releases, 2.6.x kerneles and 2.6.x.y
>
I recently wrote a document on the different kernel trees and how to
apply patches for them. In that document I also give a description of
the various trees.
Perhaps that document will be useful to you. You can find it in a
recent kernel source tree as Documentation/applying-patches.txt or you
can read it online via lrx at this URL :
http://sosdg.org/~coywolf/lxr/source/Documentation/applying-patches.txt
Good luck on your article. Once you are done with it I'd love to read
it and I guess a lot of other people would too, so please supply an
URL to it at that time :-)
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
On 10/29/05, Linus Torvalds <[email protected]> wrote:
> But to hit that, we'd might need to shrink the "merge window" from two
> weeks to just one, otherwise there's not enough time to calm down.
The problem with just one week to merge would be if other stuff
happened that ate up the merge period (e.g. this coming week I'm
travelling and will spend all day Tuesday and Wednesday in meetings).
It would be easier to manage if I could be sure which week was going
to be the merge window ... but I know that it is hard to predict when
a release will happen. Overall I'd be much happier sticking with a
two week window.
-Tony
Russell King <[email protected]> wrote:
>
> On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > Linus Torvalds <[email protected]> writes:
> > > I don't think anybody has been really unhappy with this approach? Hmm?
> >
> > The long freeze periods were nothing much happens are painful. It
> > would be better to have some more overlap of merging and stabilizing
> > (stable does that already kind of, but not enough)
>
> Violently agree. I find the long freeze periods painful and very very
> very boring, to the point of looking for other stuff to do (such as
> cleaning up bits of the kernel and queuing mega-patches for the next
> round of merging.)
The freezes are for fixing bugs, especially recent regressions. There's no
shortage of them, you know.
I you can think of a better way to get kernel developers off their butts
and actually fixing bugs, I'm all ears.
On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> Please don't. Even the two weeks are too short IMHO, because it is
> hard to digest so much code in such a short time and also it is not
> always easy for maintainers to hit such short time windows for sending
> patches.
>
> > I don't think anybody has been really unhappy with this approach? Hmm?
>
> The long freeze periods were nothing much happens are painful. It
> would be better to have some more overlap of merging and stabilizing
> (stable does that already kind of, but not enough)
I thought Andrew was accepting patches targeted at 2.6.n+1 into the
-mm tree during the freeze periods, yes? If so, why would it be a
case of "nothing much happens"? Nothing much might be happening in
Linus's git tree, but that doesn't that they can't be happening in
Andrew's -mm patchsets....
- Ted
On Sun, Oct 30, 2005 at 11:12:41AM -0800, Andrew Morton wrote:
> Russell King <[email protected]> wrote:
> >
> > On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > > Linus Torvalds <[email protected]> writes:
> > > > I don't think anybody has been really unhappy with this approach? Hmm?
> > >
> > > The long freeze periods were nothing much happens are painful. It
> > > would be better to have some more overlap of merging and stabilizing
> > > (stable does that already kind of, but not enough)
> >
> > Violently agree. I find the long freeze periods painful and very very
> > very boring, to the point of looking for other stuff to do (such as
> > cleaning up bits of the kernel and queuing mega-patches for the next
> > round of merging.)
>
> The freezes are for fixing bugs, especially recent regressions.
Given my stated low activity during the -rc periods, well, you draw
your conclusion from that.
> There's no shortage of them, you know.
Please let me know when there's something in my area regresses.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
Russell King <[email protected]> wrote:
>
> On Sun, Oct 30, 2005 at 11:12:41AM -0800, Andrew Morton wrote:
> > Russell King <[email protected]> wrote:
> > >
> > > On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > > > Linus Torvalds <[email protected]> writes:
> > > > > I don't think anybody has been really unhappy with this approach? Hmm?
> > > >
> > > > The long freeze periods were nothing much happens are painful. It
> > > > would be better to have some more overlap of merging and stabilizing
> > > > (stable does that already kind of, but not enough)
> > >
> > > Violently agree. I find the long freeze periods painful and very very
> > > very boring, to the point of looking for other stuff to do (such as
> > > cleaning up bits of the kernel and queuing mega-patches for the next
> > > round of merging.)
> >
> > The freezes are for fixing bugs, especially recent regressions.
>
> Given my stated low activity during the -rc periods, well, you draw
> your conclusion from that.
>
> > There's no shortage of them, you know.
>
> Please let me know when there's something in my area regresses.
>
a) you're sitting around feeling very very very bored while
b) the kernel is in long freeze due to the lack of kernel developer
attention to known bugs
The solution seems fairly obvious to me?
On Sun, Oct 30, 2005 at 02:31:03PM -0800, Andrew Morton wrote:
> Russell King <[email protected]> wrote:
> >
> > On Sun, Oct 30, 2005 at 11:12:41AM -0800, Andrew Morton wrote:
> > > Russell King <[email protected]> wrote:
> > > >
> > > > On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > > > > Linus Torvalds <[email protected]> writes:
> > > > > > I don't think anybody has been really unhappy with this approach? Hmm?
> > > > >
> > > > > The long freeze periods were nothing much happens are painful. It
> > > > > would be better to have some more overlap of merging and stabilizing
> > > > > (stable does that already kind of, but not enough)
> > > >
> > > > Violently agree. I find the long freeze periods painful and very very
> > > > very boring, to the point of looking for other stuff to do (such as
> > > > cleaning up bits of the kernel and queuing mega-patches for the next
> > > > round of merging.)
> > >
> > > The freezes are for fixing bugs, especially recent regressions.
> >
> > Given my stated low activity during the -rc periods, well, you draw
> > your conclusion from that.
> >
> > > There's no shortage of them, you know.
> >
> > Please let me know when there's something in my area regresses.
> >
>
> a) you're sitting around feeling very very very bored while
>
> b) the kernel is in long freeze due to the lack of kernel developer
> attention to known bugs
>
> The solution seems fairly obvious to me?
That's fine if you have the hardware to be able to debug these issues.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
Russell King <[email protected]> wrote:
>
> That's fine if you have the hardware to be able to debug these issues.
Most driver bugs cannot be reproduced by the developer. Almost by
definition: if the patch had caused problems on the developer's machine, he
wouldn't have shipped it.
This is why we have this wonderful group of long-suffering people who
download and test development kernels for us, and who take the time to
report defects.
Yes, it's painful to get into a long-range debugging session, sending debug
patches, twelve-hour turnaround, etc. But what alternative have we?
On Sun, Oct 30, 2005 at 02:55:33PM -0800, Andrew Morton wrote:
> Russell King <[email protected]> wrote:
> >
> > That's fine if you have the hardware to be able to debug these issues.
>
> Most driver bugs cannot be reproduced by the developer. Almost by
> definition: if the patch had caused problems on the developer's machine, he
> wouldn't have shipped it.
>
> This is why we have this wonderful group of long-suffering people who
> download and test development kernels for us, and who take the time to
> report defects.
>
> Yes, it's painful to get into a long-range debugging session, sending debug
> patches, twelve-hour turnaround, etc. But what alternative have we?
However, it helps if you have a grasp of technologies like ACPI,
IO-APICs, APICs, PCI IRQ routing for x86 problems. Since I don't
work in the x86 field, I have _zero_ knowledge of such things
because they just don't apply when working on ARM.
That makes debugging x86 problems a nightmare. Remember I gave
up with trying to sort out peoples PCMCIA problems?
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
On Sunday 30 October 2005 20:12, Andrew Morton wrote:
> The freezes are for fixing bugs, especially recent regressions. There's no
> shortage of them, you know.
>
> I you can think of a better way to get kernel developers off their butts
> and actually fixing bugs, I'm all ears.
The problem is that you usually cannot do proper bug fixing because
the release might be just around the corner, so you typically
chose the ugly workaround or revert, or just reject changes for bugs that a
are too risky or the impact too low because there is not enough time to
properly test anymore.
It might work better if we were told when the releases would actually
happen and you don't need to fear that this not quite tested everywhere
bugfix you're about to submit might make it into the gold kernel, breaking
the world for some subset of users.
-Andi
On Sunday 30 October 2005 22:32, Theodore Ts'o wrote:
> I thought Andrew was accepting patches targeted at 2.6.n+1 into the
> -mm tree during the freeze periods, yes? If so, why would it be a
> case of "nothing much happens"? Nothing much might be happening in
> Linus's git tree, but that doesn't that they can't be happening in
> Andrew's -mm patchsets....
The problem is that -mm* contains typically so many more or less
broken changes that any extensive work on there is futile
because you never know whose bugs you're debugging
(and if the patch that is broken will even make it anywhere)
In short mainline is frozen too long and -mm* is too unstable.
-Andi
On Mon, Oct 31, 2005 at 01:48:56AM +0100, Andi Kleen wrote:
> On Sunday 30 October 2005 20:12, Andrew Morton wrote:
>
> > The freezes are for fixing bugs, especially recent regressions. There's no
> > shortage of them, you know.
> >
> > I you can think of a better way to get kernel developers off their butts
> > and actually fixing bugs, I'm all ears.
>
> The problem is that you usually cannot do proper bug fixing because
> the release might be just around the corner, so you typically
> chose the ugly workaround or revert, or just reject changes for bugs that a
> are too risky or the impact too low because there is not enough time to
> properly test anymore.
>
> It might work better if we were told when the releases would actually
> happen and you don't need to fear that this not quite tested everywhere
> bugfix you're about to submit might make it into the gold kernel, breaking
> the world for some subset of users.
Indeed - a prime example is the bootmem initialisation problem. Had
I known on 1st October that the final release wouldn't have been for
a long time, I'd have applied that patch and you wouldn't have had
that problem with ARM relying on the bootmem initialisation order
clashing with your ia64 (or x86_64) swiotlb fix.
But stupid rmk - again I hasten to add - was suckered into this "-rc
is frozen" idea again and decided it wasn't appropriate to submit it.
In hind-sight, there would have been plenty of time to sort out any
issues.
Hell, we held up 2.6.14 for swiotlb _anyway_.
Time to start writing those lines...
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
On Mon, Oct 31, 2005 at 01:45:43AM +0100, Andi Kleen wrote:
> The problem is that -mm* contains typically so many more or less
> broken changes that any extensive work on there is futile
> because you never know whose bugs you're debugging
> (and if the patch that is broken will even make it anywhere)
>
> In short mainline is frozen too long and -mm* is too unstable.
Besides, -mm is changing so fscking fast that it doesn't build on a lot
of configs most of the time. And trying to keep track of it and at least
deal with build breakage at real time is, IME, hopeless.
Andi Kleen <[email protected]> wrote:
>
> On Sunday 30 October 2005 20:12, Andrew Morton wrote:
>
> > The freezes are for fixing bugs, especially recent regressions. There's no
> > shortage of them, you know.
> >
> > I you can think of a better way to get kernel developers off their butts
> > and actually fixing bugs, I'm all ears.
>
> The problem is that you usually cannot do proper bug fixing because
> the release might be just around the corner, so you typically
> chose the ugly workaround or revert, or just reject changes for bugs that a
> are too risky or the impact too low because there is not enough time to
> properly test anymore.
There is absolutely nothing preventing people from working on bugs at any
time! It's not as if a bugfix can ever come too early.
> It might work better if we were told when the releases would actually
> happen and you don't need to fear that this not quite tested everywhere
> bugfix you're about to submit might make it into the gold kernel, breaking
> the world for some subset of users.
Nobody knows when a kernel will be released, because it's released
according to perceived bug status, not according to a preconceived
timeline.
I just don't buy what you're saying. Unless the kernel is at -rc3 or -rc4,
we *know* the release is weeks away - it's always been thus.
Russell King <[email protected]> wrote:
>
> > It might work better if we were told when the releases would actually
> > happen and you don't need to fear that this not quite tested everywhere
> > bugfix you're about to submit might make it into the gold kernel, breaking
> > the world for some subset of users.
>
> Indeed - a prime example is the bootmem initialisation problem.
No, I'd say that was an *exception*, not a "prime example".
I've filed away 322 unresolved bug reports here. The great majority are
busted drivers on random hardware dating back as far as 2.6.11. Many of
them are regressions.
There is nothing stopping anyone from working with the originators to get
these things fixed up at any time.
Why is it necessary for me to chase maintainers to get their bugs fixed?
Why are maintainers working on new features when they have unresolved bugs?
Why is it so often me who has to do the followup for un-responded-to bug
reports against subsystems which I don't know anything about?
(Those are rhetorical questions, btw).
On Monday 31 October 2005 02:22, Andrew Morton wrote:
> There is nothing stopping anyone from working with the originators to get
> these things fixed up at any time.
>
> Why is it necessary for me to chase maintainers to get their bugs fixed?
>
> Why are maintainers working on new features when they have unresolved bugs?
Because zero bugs is just unrealistic and they would never get anything done
if that was the requirement?
(especially considering that a lot of the bugs at least on x86-64 are
hardware/firmware bugs of some sort, so often it is not really a linux
bug but just a missing ha^w^wwork^w^w^w^wfix for something else)
I agree regressions are a problem and need to be addressed, but handling all
non regressions on a non trivial platforms is just impossible IMHO...
Perhaps it would be nice to have better bug classification: e.g.
regression/new hardware/reported by more than one person etc. I think
with some prioritization like that it would be much easier to keep the bugs
under control.
Sometimes bugs are less important than others.
e.g. on x86-64 the bootmem issue was obscure at best, affecting
only a very small part of the user base. Sure the few people hit by it
will be annoyed, but trying to make everyone happy is impossible so
one has to try to just make the majority of users happy. So it was imho
ok to just revert the patch to fix ARM again and not breaking IA64 (I cannot
assess how many users were on ARM/IA64 affected) for the next release and
try to sort it out later.
Regressions are important, everything else has to be prioritized based on the
impact on the user base (and this doesn't necessarily mean the most noisy
part of the userbase)
Perhaps some people could volunteer to set some flags in bugzilla for obvious
things, like regression or new hardware or missing basic information or for
really old kernel and no report for a new one and that could be used to
filter the queries better. Should be an relatively easy task.
-Andi
Al, responding to Andi,
> > because you never know whose bugs you're debugging
> > (and if the patch that is broken will even make it anywhere)
> >
> > In short mainline is frozen too long and -mm* is too unstable.
>
> Besides, -mm is changing so fscking fast that it doesn't build on a lot
> of configs most of the time.
I think you are exagerating.
It builds on most configs most of the time in my experience. If I
haven't tried a crosstool rebuild of the several defconfig arch's in a
week, I might expect one of the less popular archs to drop out, usually
for something so easy even I can figure some sort of fix or workaround.
It's tough to get stable kernels if the only kernels people want to
test on are stable kernels. Linux is benefiting immensely from its
rapid evolution in various directions.
Granted - build and boot tested versions of *-mm, just a couple days
behind Andrew's patch set releases, would save people of dealing with
some of the simply stupid breakage.
Is there any corporate Linux supporter able to fund that? Clearing
off the simple breakage could help attract the more serious and
diverse testing of a wider group of users with various specialized
interests.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
On Sun, Oct 30, 2005 at 07:14:02PM -0800, Paul Jackson wrote:
> I think you are exagerating.
>
> It builds on most configs most of the time in my experience. If I
> haven't tried a crosstool rebuild of the several defconfig arch's in a
> week, I might expect one of the less popular archs to drop out, usually
> for something so easy even I can figure some sort of fix or workaround.
Try allmodconfig for a change... I'm doing that for mainline on a regular
basis and even that turns into considerable amount of time. I have tried
that for -mm and had to give up.
On Sunday 30 October 2005 18:45, Andi Kleen wrote:
> The problem is that -mm* contains typically so many more or less
> broken changes that any extensive work on there is futile
> because you never know whose bugs you're debugging
> (and if the patch that is broken will even make it anywhere)
>
> In short mainline is frozen too long and -mm* is too unstable.
Are you implying that if mainline wasn't frozen so much, it would still be
more stable than -mm?
Rob
On Sunday 30 October 2005 18:48, Andi Kleen wrote:
> The problem is that you usually cannot do proper bug fixing because
> the release might be just around the corner, so you typically
> chose the ugly workaround or revert, or just reject changes for bugs that a
> are too risky or the impact too low because there is not enough time to
> properly test anymore.
>
> It might work better if we were told when the releases would actually
> happen and you don't need to fear that this not quite tested everywhere
> bugfix you're about to submit might make it into the gold kernel, breaking
> the world for some subset of users.
Hence the -mm tree, which takes stuff that may still need to be debugged.
Except that it has this nasty habit of taking stuff which still needs to be
debugged from people _other_than_you_, which screws you up. You seem to want
a tree where the only stuff likely to break is your stuff, which is another
popular option: maintaining your own developer tree. Getting people to _use_
such a tree takes a bit of work, but that's not news to anybody.
Think about what you're asking for here. Imagine that other people _also_ get
what you're asking for, at the same time. Is it still what you want?
Right now patches go from developer tree, to -mm tree, to -linus tree, with a
larger audience each time. The _reason_ linus's tree has a larger audience
is exactly _because_ the patches in it have had more testing so it's less
likely to break. And the releases have a way larger audience than Linus's
-rc releases, and the distro kernels have a larger audience than that...
> -Andi
Rob
Andi Kleen <[email protected]> wrote:
>
> On Monday 31 October 2005 02:22, Andrew Morton wrote:
>
> > There is nothing stopping anyone from working with the originators to get
> > these things fixed up at any time.
> >
> > Why is it necessary for me to chase maintainers to get their bugs fixed?
> >
> > Why are maintainers working on new features when they have unresolved bugs?
>
> Because zero bugs is just unrealistic and they would never get anything done
> if that was the requirement?
>
> (especially considering that a lot of the bugs at least on x86-64 are
> hardware/firmware bugs of some sort, so often it is not really a linux
> bug but just a missing ha^w^wwork^w^w^w^wfix for something else)
>
> I agree regressions are a problem and need to be addressed, but handling all
> non regressions on a non trivial platforms is just impossible IMHO...
>
> Perhaps it would be nice to have better bug classification: e.g.
> regression/new hardware/reported by more than one person etc. I think
> with some prioritization like that it would be much easier to keep the bugs
> under control.
Well sure. But who does that? It should be the relevant maintainer, IMO.
If that's the way in which he chooses to work. Expecting some third party
to do this on a kenrel-wide basis won't fly.
> ...
> Sometimes bugs are less important than others.
I don't believe that what we're seeing is some prioritisation process. The
problem is that some bugs are *harder* than others, and people are just
ducking things which cannot be locally reproduced.
Rob Landley <[email protected]> wrote:
>
> You seem to want
> a tree where the only stuff likely to break is your stuff
That's what I was thinking ;)
The simple fact is that we have more developers doing more stuff faster
than they used to. All within a coupled system which has a lot of
interactions.
End result: yes, we do all need to spend more time looking at other
people's code and less time looking at our own. That's just life in a
large project.
I'm very careful to make sure that relevant developers are copied on
patches which go into -mm. In fact there's significantly better review
opportunity on patches which go developer->mm->Linus than there are on
patches which go developer->maintainer-git->Linus.
But the cc'ed people just _have_ to take time out to read the dang patch!
They almost always have multiple weeks in which to do this. But if they
just delete the thing while they work on their own stuff, well...
Al Viro wrote:
> Try allmodconfig for a change...
;). That's ok. What I do keeps me busy enough.
Still, it's too bad one of Linux's corporate friends can't find a way
to do this sort of thing, providing rapid turn around on build-blessed
*-mm versions.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
Al Viro <[email protected]> wrote:
>
> On Sun, Oct 30, 2005 at 07:14:02PM -0800, Paul Jackson wrote:
> > I think you are exagerating.
> >
> > It builds on most configs most of the time in my experience. If I
> > haven't tried a crosstool rebuild of the several defconfig arch's in a
> > week, I might expect one of the less popular archs to drop out, usually
> > for something so easy even I can figure some sort of fix or workaround.
>
> Try allmodconfig for a change... I'm doing that for mainline on a regular
> basis and even that turns into considerable amount of time. I have tried
> that for -mm and had to give up.
fud. Every -mm release is built with allmodconfig on x86 and on x86_64.
It's also cross-compiled on fat configs for alpha, ppc32, ppc64, sparc64,
arm and ia64. It's booted on x86, x86_64, ppc64 and ia64. Every release.
Yes, some tty drivers have been busted in recent releases, as described in
the release notices. That's why the tty patches are at the end of the -mm
series: so I can back them off for the allmodconfig builds.
I usually do allnoconfig and allyesconfig, too.
On Mon, 31 Oct 2005, Andi Kleen wrote:
> Perhaps some people could volunteer to set some flags in bugzilla for obvious
> things, like regression or new hardware or missing basic information or for
> really old kernel and no report for a new one and that could be used to
> filter the queries better. Should be an relatively easy task.
I don't think we need any special flags, we just need more people paying
attention to it. It doesn't take that much time to go through pending bugs
and trying to identify real ones, but the problem is that we need
knowledgeable people to do this.
On Sat, Oct 29, 2005 at 01:28:23PM -0700, Linus Torvalds wrote:
> So I'm planning on continuing with it unchanged for now. Two-week merge
> window until -rc1, and then another -rc kernel roughly every week until
> release. With the goal being 6 weeks, and 8 weeks being ok.
>
> I don't think anybody has been really unhappy with this approach? Hmm?
I believe it was a good experience. However, I still find it sad that
there are changes between the latest -rc and the final version. This
time, it seems it did not cause trouble, but in the past, it happened
several times, because any valid fix can have side effects.
I think that if you could announce what you intend to release with a
message like "I will make this -final tomorrow", there would be some
time to test builds in various configs, and check that no obvious bug
has been introduced.
Regards,
Willy
On Sun, Oct 30, 2005 at 11:22:34PM -0800, Andrew Morton wrote:
> Al Viro <[email protected]> wrote:
> >
> > On Sun, Oct 30, 2005 at 07:14:02PM -0800, Paul Jackson wrote:
> > > I think you are exagerating.
> > >
> > > It builds on most configs most of the time in my experience. If I
> > > haven't tried a crosstool rebuild of the several defconfig arch's in a
> > > week, I might expect one of the less popular archs to drop out, usually
> > > for something so easy even I can figure some sort of fix or workaround.
> >
> > Try allmodconfig for a change... I'm doing that for mainline on a regular
> > basis and even that turns into considerable amount of time. I have tried
> > that for -mm and had to give up.
>
> fud. Every -mm release is built with allmodconfig on x86 and on x86_64.
> It's also cross-compiled on fat configs for alpha, ppc32, ppc64, sparc64,
> arm and ia64. It's booted on x86, x86_64, ppc64 and ia64. Every release.
What fud? I stand by my claim - I have tried to do the same thing for
-mm and found that I didn't have guts for that; too much work. For mainline
I do cross-builds for allmodconfig on a *lot* more targets than what you've
mentioned and generally it stays within ~150-200Kb of patches, about half
of that being a fix for 8390 mess.
_IF_ somebody wants to do that for -mm, yell and you are more than welcome
to all infrastructure, except for the cycles on build box I'm using.
Incidentally, it is a box at work - my energy bill is high enough as it
is, without adding an 8-way 3GHz iamd64 to it...
The last time I've attempted that for -mm was this summer; right now mainline
is quite enough work, TYVM... Especially since I use the same tree as a
staging point for annotations and watch for build regressions both for gcc
and sparse.
Al Viro wrote:
> I do cross-builds for allmodconfig on a *lot* more targets than what you've
> mentioned
Apples to oranges, and half full versus half empty.
If Al is testing combinations that Andrew doesn't regularly test,
then Al might expect to find problems that Andrew didn't find.
Non-specific complaints that make it all sound too hard are
not particularly constructive. But then Al was trying for the
"particularly constructive" achievement award of the day ;).
I find that the defconfig crosstool builds of *-mm more or less always
work for the popular configurations. That meets my needs just fine.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
On Oct 30 2005, Andrew Morton wrote:
> But the cc'ed people just _have_ to take time out to read the dang
> patch! They almost always have multiple weeks in which to do this.
> But if they just delete the thing while they work on their own stuff,
> well...
Indeed. And there are the guinea pigs/lusers like me that are willing to
test and do some roundtrips of reports to get the kernel working better
(with the little that I can help). :-)
Now that I have backups working as I wanted, I surely intend to give the
-mm tree more tries.
Oh, and I would really appreciate if some options had at least some
description when I do a make menuconfig. Would identifying these options
or asking for clarification of others be of any interest?
I do think that they would help me when I try to compile the kernel for
some of my friends' computers.
Thanks, Rog?rio.
--
Rog?rio Brito : [email protected] : http://www.ime.usp.br/~rbrito
Homepage of the algorithms package : http://algorithms.berlios.de
Homepage on freshmeat: http://freshmeat.net/projects/algorithms/
Rog?rio Brito <[email protected]> wrote:
>
> Oh, and I would really appreciate if some options had at least some
> description when I do a make menuconfig. Would identifying these options
> or asking for clarification of others be of any interest?
Yes please - people often seem to omit the help text.
Better still is to send a patch which adds the help text, of course. If
it's a bit wrong, people will help sort that out.
Hi,
On Mon, 31 Oct 2005, Andi Kleen wrote:
> I agree regressions are a problem and need to be addressed, but handling all
> non regressions on a non trivial platforms is just impossible IMHO...
Here is a different kind of non trivial regression on a common platform:
$ ll -S /boot/ | grep ...
-rw-r--r-- 1 root root 1518317 2005-11-01 00:38 vmlinuz-2.6.14
-rw-r--r-- 1 root root 1506432 2005-08-30 00:36 vmlinuz-2.6.13
-rw-r--r-- 1 root root 1451154 2004-12-26 16:54 vmlinuz-2.6.10
-rw-r--r-- 1 root root 1432032 2005-06-26 03:50 vmlinuz-2.6.12
-rw-r--r-- 1 root root 1413888 2004-06-21 18:33 vmlinuz-2.6.7
-rw-r--r-- 1 root root 1394801 2004-05-17 22:14 vmlinuz-2.6.5
-rw-r--r-- 1 root root 1390233 2004-05-18 01:54 vmlinuz-2.6.6
-rw-r--r-- 1 root root 1315050 2003-09-12 22:10 vmlinuz-2.6.0-test5
-rw-r--r-- 1 root root 1280189 2003-06-06 01:00 vmlinuz-2.5.70
-rw-r--r-- 1 root root 918663 2002-11-11 22:42 vmlinuz-2.5.47
-rw-r--r-- 1 root root 887758 2004-02-19 01:24 vmlinuz-2.4.25
-rw-r--r-- 1 root root 883868 2003-12-20 21:02 vmlinuz-2.4.23
-rw-r--r-- 1 root root 875858 2003-08-26 00:39 vmlinuz-2.4.22
-rw-r--r-- 1 root root 859547 2003-02-06 22:35 vmlinuz-2.5.59
-rw-r--r-- 1 root root 855092 2003-06-14 02:29 vmlinuz-2.4.21
-rw-r--r-- 1 root root 703585 2002-12-19 11:58 vmlinuz-2.4.20
-rw-r--r-- 1 root root 700359 2002-08-03 19:57 vmlinuz-2.4.19
-rw-r--r-- 1 root root 690106 2001-01-07 23:38 vmlinuz-2.4.0
-rw-r--r-- 1 root root 678080 2002-03-02 20:48 vmlinuz-2.4.18
-rw-r--r-- 1 root root 641409 2001-02-23 02:53 vmlinuz-2.4.2
-rw-r--r-- 1 root root 640950 2001-01-31 01:30 vmlinuz-2.4.1
-rw-r--r-- 1 root root 635339 2001-12-22 03:16 vmlinuz-2.4.17
-rw-r--r-- 1 root root 632647 2001-12-04 00:37 vmlinuz-2.4.16
-rw-r--r-- 1 root root 629787 2001-11-07 22:00 vmlinuz-2.4.14
-rw-r--r-- 1 root root 629240 2001-08-11 18:16 vmlinuz-2.4.8
-rw-r--r-- 1 root root 628556 2001-07-21 17:06 vmlinuz-2.4.7
-rw-r--r-- 1 root root 628386 2001-09-24 00:02 vmlinuz-2.4.10
-rw-r--r-- 1 root root 627597 2001-10-20 15:14 vmlinuz-2.4.12
-rw-r--r-- 1 root root 626634 2001-07-11 20:22 vmlinuz-2.4.6
-rw-r--r-- 1 root root 623325 2001-05-28 01:57 vmlinuz-2.4.5
-rw-r--r-- 1 root root 622619 2001-05-01 16:20 vmlinuz-2.4.4
-rw-r--r-- 1 root root 611493 2001-04-02 20:44 vmlinuz-2.4.3
-rw-r--r-- 1 root root 497523 2001-03-30 18:56 vmlinuz-2.2.19
-rw-r--r-- 1 root root 495553 2001-03-25 17:24 vmlinuz-2.2.18
I'm a bit scared of this development and I'm afraid that merging all the
new features at high speed has become more important than code quality...
bye, Roman
Roman Zippel <[email protected]> wrote:
>
> Here is a different kind of non trivial regression on a common platform:
>
> $ ll -S /boot/ | grep ...
> -rw-r--r-- 1 root root 1518317 2005-11-01 00:38 vmlinuz-2.6.14
> -rw-r--r-- 1 root root 1506432 2005-08-30 00:36 vmlinuz-2.6.13
> -rw-r--r-- 1 root root 1451154 2004-12-26 16:54 vmlinuz-2.6.10
> -rw-r--r-- 1 root root 1432032 2005-06-26 03:50 vmlinuz-2.6.12
> -rw-r--r-- 1 root root 1413888 2004-06-21 18:33 vmlinuz-2.6.7
> -rw-r--r-- 1 root root 1394801 2004-05-17 22:14 vmlinuz-2.6.5
> -rw-r--r-- 1 root root 1390233 2004-05-18 01:54 vmlinuz-2.6.6
> -rw-r--r-- 1 root root 1315050 2003-09-12 22:10 vmlinuz-2.6.0-test5
> -rw-r--r-- 1 root root 1280189 2003-06-06 01:00 vmlinuz-2.5.70
> -rw-r--r-- 1 root root 918663 2002-11-11 22:42 vmlinuz-2.5.47
> -rw-r--r-- 1 root root 887758 2004-02-19 01:24 vmlinuz-2.4.25
> -rw-r--r-- 1 root root 883868 2003-12-20 21:02 vmlinuz-2.4.23
What does size(1) say?
Are you sure these kernels are feature-equivalent?
On Mon, 31 Oct 2005, Andrew Morton wrote:
>
> Are you sure these kernels are feature-equivalent?
They may not be feature-equivalent in reality, but it's hard to generate
something that has the features (or lack there-of) of old kernels these
days. Which is problematic.
But some of it is likely also compilers. gcc does insane padding in many
cases these days.
And a lot of it is us just being bloated. Argh.
Linus
Hi,
On Mon, 31 Oct 2005, Andrew Morton wrote:
> Roman Zippel <[email protected]> wrote:
> >
> > Here is a different kind of non trivial regression on a common platform:
> >
> > $ ll -S /boot/ | grep ...
> > -rw-r--r-- 1 root root 1518317 2005-11-01 00:38 vmlinuz-2.6.14
> > -rw-r--r-- 1 root root 1506432 2005-08-30 00:36 vmlinuz-2.6.13
> > -rw-r--r-- 1 root root 1451154 2004-12-26 16:54 vmlinuz-2.6.10
> > -rw-r--r-- 1 root root 1432032 2005-06-26 03:50 vmlinuz-2.6.12
> > -rw-r--r-- 1 root root 1413888 2004-06-21 18:33 vmlinuz-2.6.7
> > -rw-r--r-- 1 root root 1394801 2004-05-17 22:14 vmlinuz-2.6.5
> > -rw-r--r-- 1 root root 1390233 2004-05-18 01:54 vmlinuz-2.6.6
> > -rw-r--r-- 1 root root 1315050 2003-09-12 22:10 vmlinuz-2.6.0-test5
> > -rw-r--r-- 1 root root 1280189 2003-06-06 01:00 vmlinuz-2.5.70
> > -rw-r--r-- 1 root root 918663 2002-11-11 22:42 vmlinuz-2.5.47
> > -rw-r--r-- 1 root root 887758 2004-02-19 01:24 vmlinuz-2.4.25
> > -rw-r--r-- 1 root root 883868 2003-12-20 21:02 vmlinuz-2.4.23
>
> What does size(1) say?
This is already stripped and compressed, so it's not directly available.
> Are you sure these kernels are feature-equivalent?
Pretty much, on this machine I generally only include what I need, so only
a few drivers were added, I even have KALLSYMS disabled.
bye, Roman
Linus Torvalds <[email protected]> wrote:
>
>
>
> On Mon, 31 Oct 2005, Andrew Morton wrote:
> >
> > Are you sure these kernels are feature-equivalent?
>
> They may not be feature-equivalent in reality, but it's hard to generate
> something that has the features (or lack there-of) of old kernels these
> days. Which is problematic.
Probably.
> But some of it is likely also compilers. gcc does insane padding in many
> cases these days.
2.6.14 `make allnoconfig':
gcc-2.95.4:
bix:/usr/src/25> size vmlinux
text data bss dec hex filename
665502 152379 55120 873001 d5229 vmlinux
gcc version 4.1.0 20050513 (experimental):
bix:/usr/src/25> size vmlinux
text data bss dec hex filename
761415 151851 55280 968546 ec762 vmlinux
(There's a new reason for retaining gcc-2.95.x support)
(gcc-4.x can probably be tuned up with appropriate `-malign' options)
> And a lot of it is us just being bloated. Argh.
2.5.71, gcc-2.95.4:
bix:/usr/src/aa/linux-2.5.71> size vmlinux
text data bss dec hex filename
501892 54163 40420 596475 919fb vmlinux
yes, it got bigger. .data went through the roof - maybe inlined debug stuff?
2.6.8.1, gcc-2.95.4:
bix:/usr/src/aa/linux-2.6.8.1> size vmlinux
text data bss dec hex filename
605032 153817 58176 817025 c7781 vmlinux
It happened somewhere between 2.5.71 and 2.6.8.
2.4.x doesn't have allnoconfig, so no numbers for that.
On Monday, October 31, 2005 4:17 pm, Roman Zippel wrote:
> > Are you sure these kernels are feature-equivalent?
>
> Pretty much, on this machine I generally only include what I need, so
> only a few drivers were added, I even have KALLSYMS disabled.
Is it just one top level subsystem that's increasing in size faster than
the others? Last time I broke it down, networking (net/built-in.o) was
the biggest by far, and it does seem to add features at a fast rate (not
that I'm complaining!). On FC devel with the FC kernel config:
-rw-rw-r-- 1 jbarnes jbarnes 555088 Oct 31 16:33 net/built-in.o
(stripped)
Jesse
Andrew Morton wrote:
>
> 2.4.x doesn't have allnoconfig, so no numbers for that.
-rw-r--r-- 1 root root 7501 2005-11-01 10:57 config-2.4.31-none
-rw-r--r-- 1 root root 249263 2005-11-01 10:57 bzImage-2.4.31-none
-rw-r--r-- 1 root root 71548 2005-11-01 10:57 System.map-2.4.31-none
These things snuck in (everything on make menuconfig cleared):
grant@sempro:~/linux/linux-2.4.31-none$ grep = .config
CONFIG_X86=y
CONFIG_UID16=y
CONFIG_MPENTIUMIII=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_L1_CACHE_SHIFT=5
CONFIG_X86_HAS_TSC=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_PGE=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_F00F_WORKS_OK=y
CONFIG_NOHIGHMEM=y
CONFIG_X86_TSC=y
CONFIG_RAMFS=y
CONFIG_MSDOS_PARTITION=y
CONFIG_LOG_BUF_SHIFT=0
grant@sempro:~/linux/linux-2.4.31-none$
Cheers,
Grant.
On Mon, Oct 31, 2005 at 04:13:22PM -0800, Linus Torvalds wrote:
> On Mon, 31 Oct 2005, Andrew Morton wrote:
> > Are you sure these kernels are feature-equivalent?
>
> They may not be feature-equivalent in reality, but it's hard to generate
> something that has the features (or lack there-of) of old kernels these
> days. Which is problematic.
>
> But some of it is likely also compilers. gcc does insane padding in many
> cases these days.
>
> And a lot of it is us just being bloated. Argh.
Which is one of the reasons I've started working on fixing up the
platform device/driver stuff to conform to the "usual" method,
with the view to killing off _all_ the function pointers in
struct device_driver.
Most bus types wrap struct device_driver, and then provide their own
function pointers which pass their bus-type specific device structure.
This does two things: 1. it centralises the conversion from struct
device to struct whatever_device, and 2. improves typechecking.
However, once the use of the function pointers in struct device_driver
have been eliminated, we can be sure of reclaiming at least 20 bytes
per device driver, maybe more if GCC does insane padding.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
On Monday 31 October 2005 18:13, Linus Torvalds wrote:
> On Mon, 31 Oct 2005, Andrew Morton wrote:
> > Are you sure these kernels are feature-equivalent?
>
> They may not be feature-equivalent in reality, but it's hard to generate
> something that has the features (or lack there-of) of old kernels these
> days. Which is problematic.
>
> But some of it is likely also compilers. gcc does insane padding in many
> cases these days.
>
> And a lot of it is us just being bloated. Argh.
>
> Linus
Matt Mackall! Tiny tree! Yay rah cool!
http://selenic.com/tiny/2.6.14-tiny1-broken-out.tar.bz2
Rob
P.S. There's a reason I'm trying to make a real working development system
based on busybox and uclibc. I think things like live CDs should be using
that, not the GNU packages.
There seems to be a periodic trend, where ever few years open source programs
get feature-laden enough that somebody forks off (or starts over) a version
that has the sole virtue of being smaller and simpler. From glibc->uClibc,
gnome/kde->xfce, OpenSSH->dropbear, gnu->busybox... Of course mozilla had to
do this twice (Galleon, then Firefox) to get something remotely reasonable,
but oh well.
(And it'd be really NICE if tcc became a reasonable replacement for gcc.
Guess what the bloated memory-thrashing load that selectively triggers the
OOM killer (when swappiness=0 but not when swappiness=60) I reported earlier
is? Building gcc 4.0.2, genattrtab and compiling the resulting
insn-attrab.c. It won't run in "only" 128 megs of ram at the best of
times...)
Rob
On Mon, Oct 31, 2005 at 04:34:08PM -0800, Andrew Morton wrote:
> Linus Torvalds <[email protected]> wrote:
> >
> >
> >
> > On Mon, 31 Oct 2005, Andrew Morton wrote:
> > >
> > > Are you sure these kernels are feature-equivalent?
> >
> > They may not be feature-equivalent in reality, but it's hard to generate
> > something that has the features (or lack there-of) of old kernels these
> > days. Which is problematic.
>
> Probably.
>
> > But some of it is likely also compilers. gcc does insane padding in many
> > cases these days.
>
> 2.6.14 `make allnoconfig':
>
> gcc-2.95.4:
>
> bix:/usr/src/25> size vmlinux
> text data bss dec hex filename
> 665502 152379 55120 873001 d5229 vmlinux
>
> gcc version 4.1.0 20050513 (experimental):
>
> bix:/usr/src/25> size vmlinux
> text data bss dec hex filename
> 761415 151851 55280 968546 ec762 vmlinux
That's not a fair comparison.
If you do not tell gcc to optimize for size you can't expect it to
produce small code.
The following is with allnoconfig + CONFIG_EMBEDDED=y + intelligent
setting of the options shown by "make oldconfig" after setting
CONFIG_EMBEDDED=y (the third kernel is built without
-fno-unit-at-a-time):
text data bss dec hex filename
522545 77694 31256 631495 9a2c7 vmlinux-2.95
436878 76594 32248 545720 853b8 vmlinux-4.0
429420 76370 32184 537974 83576 vmlinux-4.0-unit-at-a-time
> (There's a new reason for retaining gcc-2.95.x support)
>...
There's a new reason for removing gcc-2.95.x support. ;-)
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
On Mon, Oct 31, 2005 at 04:13:22PM -0800, Linus Torvalds wrote:
>
>
> On Mon, 31 Oct 2005, Andrew Morton wrote:
> >
> > Are you sure these kernels are feature-equivalent?
>
> They may not be feature-equivalent in reality, but it's hard to generate
> something that has the features (or lack there-of) of old kernels these
> days. Which is problematic.
>...
There's also the question how to define feature-equivalent.
I want my computer to power off after "halt".
There's a significant difference between how much kernel code was
required for this feature for my old computer and what is required for
my new computer (in kernel 2.6.14):
text data bss dec hex filename
12721 2124 104 14949 3a65 arch/i386/kernel/apm.o
136789 5724 4788 147301 23f65 drivers/acpi/built-in.o
> Linus
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
On 1 Nov 2005, Andrew Morton stated:
> gcc-2.95.4:
>
> bix:/usr/src/25> size vmlinux
> text data bss dec hex filename
> 665502 152379 55120 873001 d5229 vmlinux
>
> gcc version 4.1.0 20050513 (experimental):
>
> bix:/usr/src/25> size vmlinux
> text data bss dec hex filename
> 761415 151851 55280 968546 ec762 vmlinux
>
> (There's a new reason for retaining gcc-2.95.x support)
What if you build with -Os? That tends to hold alignments down quite a
bit.
(I'll try it this evening...)
--
`"Gun-wielding recluse gunned down by local police" isn't the epitaph
I want. I am hoping for "Witnesses reported the sound up to two hundred
kilometers away" or "Last body part finally located".' --- James Nicoll
Andrew Morton wrote:
> Linus Torvalds <[email protected]> wrote:
>
>>
>>
>>On Mon, 31 Oct 2005, Andrew Morton wrote:
>>
>>>Are you sure these kernels are feature-equivalent?
>>
>>They may not be feature-equivalent in reality, but it's hard to generate
>>something that has the features (or lack there-of) of old kernels these
>>days. Which is problematic.
>
>
> Probably.
>
>
>>But some of it is likely also compilers. gcc does insane padding in many
>>cases these days.
>
>
> 2.6.14 `make allnoconfig':
>
> gcc-2.95.4:
>
> bix:/usr/src/25> size vmlinux
> text data bss dec hex filename
> 665502 152379 55120 873001 d5229 vmlinux
>
> gcc version 4.1.0 20050513 (experimental):
>
> bix:/usr/src/25> size vmlinux
> text data bss dec hex filename
> 761415 151851 55280 968546 ec762 vmlinux
Was this with -Os or ???
--
-bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
last possible moment - but no longer" -me
--Al Viro <[email protected]> wrote (on Monday, October 31, 2005 00:18:10 +0000):
> On Mon, Oct 31, 2005 at 01:45:43AM +0100, Andi Kleen wrote:
>> The problem is that -mm* contains typically so many more or less
>> broken changes that any extensive work on there is futile
>> because you never know whose bugs you're debugging
>> (and if the patch that is broken will even make it anywhere)
>>
>> In short mainline is frozen too long and -mm* is too unstable.
>
> Besides, -mm is changing so fscking fast that it doesn't build on a lot
> of configs most of the time. And trying to keep track of it and at least
> deal with build breakage at real time is, IME, hopeless.
To be fair, it's been a lot better the last month than it has the 2 before
that, when it was really pretty scary.
If it helps, we can try to build a wider range of configs to test. And I
can try to hook Andrew up with something he can do a quick compile-farm
test on before release (though I think he has his own battery of stuff).
M.
>> fud. Every -mm release is built with allmodconfig on x86 and on x86_64.
>> It's also cross-compiled on fat configs for alpha, ppc32, ppc64, sparc64,
>> arm and ia64. It's booted on x86, x86_64, ppc64 and ia64. Every release.
>
> What fud? I stand by my claim - I have tried to do the same thing for
> -mm and found that I didn't have guts for that; too much work. For mainline
> I do cross-builds for allmodconfig on a *lot* more targets than what you've
> mentioned and generally it stays within ~150-200Kb of patches, about half
> of that being a fix for 8390 mess.
>
> _IF_ somebody wants to do that for -mm, yell and you are more than welcome
> to all infrastructure, except for the cycles on build box I'm using.
> Incidentally, it is a box at work - my energy bill is high enough as it
> is, without adding an 8-way 3GHz iamd64 to it...
I'll do that if you want. I have a big lab full of largish boxes with
serious aircon, and IBM can afford the power bill. I'm assuming this
is a farm of cross-compilers that'll run on x86 (or x86_64)?
M.
> And a lot of it is us just being bloated. Argh.
Perhaps it's time to dust this patch off:
Reduce kernel text by moving many uses of GFP_KERNEL out of line, by
creating __kmem_cache_alloc_gfp_kernel(), __kmalloc_gfp_kernel() and
__kzalloc_gfp_kernel() wrapper functions. Then kmem_cache_alloc(),
kmalloc() and kzalloc() can be inline functions that just call their
_gfp_kernel version if their flags parameter is a compile-time
constant and is GFP_KERNEL, and call their ordinary version otherwise.
On an x86_64 allnoconfig kernel, I see the following sizes:
text data bss dec hex filename
795683 197356 116384 1109423 10edaf vmlinux-before
794894 197420 116384 1108698 10eada vmlinux-after
for a net savings of 789 bytes of text. More realistic configs will
save even more. With my usual config, the patched kernel boots and
runs fine.
Signed-off-by: Roland Dreier <[email protected]>
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 09b9aa6..618331d 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -61,7 +61,8 @@ extern kmem_cache_t *kmem_cache_create(c
void (*)(void *, kmem_cache_t *, unsigned long));
extern int kmem_cache_destroy(kmem_cache_t *);
extern int kmem_cache_shrink(kmem_cache_t *);
-extern void *kmem_cache_alloc(kmem_cache_t *, gfp_t);
+extern void *__kmem_cache_alloc(kmem_cache_t *, gfp_t);
+extern void *__kmem_cache_alloc_gfp_kernel(kmem_cache_t *);
extern void kmem_cache_free(kmem_cache_t *, void *);
extern unsigned int kmem_cache_size(kmem_cache_t *);
extern const char *kmem_cache_name(kmem_cache_t *);
@@ -75,6 +76,15 @@ struct cache_sizes {
};
extern struct cache_sizes malloc_sizes[];
extern void *__kmalloc(size_t, gfp_t);
+extern void *__kmalloc_gfp_kernel(size_t);
+
+static inline void *kmem_cache_alloc(kmem_cache_t *cache, gfp_t flags)
+{
+ if (__builtin_constant_p(flags) && flags == GFP_KERNEL)
+ return __kmem_cache_alloc_gfp_kernel(cache);
+
+ return __kmem_cache_alloc(cache, flags);
+}
static inline void *kmalloc(size_t size, gfp_t flags)
{
@@ -96,10 +106,22 @@ found:
malloc_sizes[i].cs_dmacachep :
malloc_sizes[i].cs_cachep, flags);
}
+
+ if (__builtin_constant_p(flags) && flags == GFP_KERNEL)
+ return __kmalloc_gfp_kernel(size);
return __kmalloc(size, flags);
}
-extern void *kzalloc(size_t, gfp_t);
+extern void *__kzalloc(size_t, gfp_t);
+extern void *__kzalloc_gfp_kernel(size_t);
+
+static inline void *kzalloc(size_t size, gfp_t flags)
+{
+ if (__builtin_constant_p(flags) && flags == GFP_KERNEL)
+ return __kzalloc_gfp_kernel(size);
+
+ return __kzalloc(size, flags);
+}
/**
* kcalloc - allocate memory for an array. The memory is set to zero.
diff --git a/mm/slab.c b/mm/slab.c
index 22bfb0b..2971e1c 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -2788,11 +2788,17 @@ static inline void __cache_free(kmem_cac
* Allocate an object from this cache. The flags are only relevant
* if the cache has no available objects.
*/
-void *kmem_cache_alloc(kmem_cache_t *cachep, gfp_t flags)
+void *__kmem_cache_alloc(kmem_cache_t *cachep, gfp_t flags)
{
return __cache_alloc(cachep, flags);
}
-EXPORT_SYMBOL(kmem_cache_alloc);
+EXPORT_SYMBOL(__kmem_cache_alloc);
+
+void *__kmem_cache_alloc_gfp_kernel(kmem_cache_t *cachep)
+{
+ return __cache_alloc(cachep, GFP_KERNEL);
+}
+EXPORT_SYMBOL(__kmem_cache_alloc_gfp_kernel);
/**
* kmem_ptr_validate - check if an untrusted pointer might
@@ -2925,6 +2931,12 @@ void *__kmalloc(size_t size, gfp_t flags
}
EXPORT_SYMBOL(__kmalloc);
+void *__kmalloc_gfp_kernel(size_t size)
+{
+ return __kmalloc(size, GFP_KERNEL);
+}
+EXPORT_SYMBOL(__kmalloc_gfp_kernel);
+
#ifdef CONFIG_SMP
/**
* __alloc_percpu - allocate one copy of the object for every present
@@ -2998,14 +3010,20 @@ EXPORT_SYMBOL(kmem_cache_free);
* @size: how many bytes of memory are required.
* @flags: the type of memory to allocate.
*/
-void *kzalloc(size_t size, gfp_t flags)
+void *__kzalloc(size_t size, gfp_t flags)
{
void *ret = kmalloc(size, flags);
if (ret)
memset(ret, 0, size);
return ret;
}
-EXPORT_SYMBOL(kzalloc);
+EXPORT_SYMBOL(__kzalloc);
+
+void *__kzalloc_gfp_kernel(size_t size)
+{
+ return __kzalloc(size, GFP_KERNEL);
+}
+EXPORT_SYMBOL(__kzalloc_gfp_kernel);
/**
* kfree - free previously allocated memory
> That's what I was thinking ;)
>
> The simple fact is that we have more developers doing more stuff faster
> than they used to. All within a coupled system which has a lot of
> interactions.
>
> End result: yes, we do all need to spend more time looking at other
> people's code and less time looking at our own. That's just life in a
> large project.
>
> I'm very careful to make sure that relevant developers are copied on
> patches which go into -mm. In fact there's significantly better review
> opportunity on patches which go developer->mm->Linus than there are on
> patches which go developer->maintainer-git->Linus.
Moreover, it's fairly easy to test stuff that's all in one place, in a
consistent format, with a simple linear stack of patches to sort through
to find culprits.
Plus you have a great tendency of dropping stuff like a stone when it's
broken, which helps a lot. Having some basic pre-mainline-merge testing
keeps the quality of mainline way up.
It'd help more if people focused more on testing their own shit before
submitting it than complaining about -mm. If it's the same people breaking
the tree all the time, I'm sure we can find a recycled set of stocks
somewhere.
M.
On Tue, 1 Nov 2005, Roland Dreier wrote:
>
> On an x86_64 allnoconfig kernel, I see the following sizes:
>
> text data bss dec hex filename
> 795683 197356 116384 1109423 10edaf vmlinux-before
> 794894 197420 116384 1108698 10eada vmlinux-after
I really don't see the point of shaving less than a kB with ugly calling
convention magic, when switching to -Os can save us much more, and when
the networking code is several hundred kB.
If we start doing size optimizations, we need to think big.
Linus
Linus> I really don't see the point of shaving less than a kB with
Linus> ugly calling convention magic, when switching to -Os can
Linus> save us much more, and when the networking code is several
Linus> hundred kB.
Fair enough, although to be fair the savings are still there even with
-Os enabled. With CONFIG_CC_OPTIMIZE_FOR_SIZE=y:
text data bss dec hex filename
682564 197172 116384 996120 f3318 vmlinux-before
681824 197244 116384 995452 f307c vmlinux-after
And with a realistic config rather than allnoconfig, you could easily
save as much as 3 or even 4 KB of text!
BTW, allnoconfig has CONFIG_NET=n, so we have to find someone else to
blame here.
Linus> If we start doing size optimizations, we need to think big.
No pun intended I'm sure...
Anyway, it would be great to find ways to make big improvements. But
I think the most realistic way to shrink the kernel is the same way it
grows in the first place -- one small piece at a time.
- R.
On Tue, 1 Nov 2005, Roland Dreier wrote:
>
> Anyway, it would be great to find ways to make big improvements. But
> I think the most realistic way to shrink the kernel is the same way it
> grows in the first place -- one small piece at a time.
No, I think that's a lost cause.
It doesn't grow by 700 bytes once in a while. It grows by much more, and
much more often. And we can't fight it that way, that's just not going to
work. Maybe have something that tracks individual object file sizes and
shames people into not growing them..
Linus
Linus> No, I think that's a lost cause.
Linus> It doesn't grow by 700 bytes once in a while. It grows by
Linus> much more, and much more often. And we can't fight it that
Linus> way, that's just not going to work. Maybe have something
Linus> that tracks individual object file sizes and shames people
Linus> into not growing them..
I think we're actually agreeing. My kmalloc/kzalloc patch is a cute
hack but magic tricks like that aren't going to shrink the kernel by
very much and it's probably not worth merging complications like that.
The way to a smaller kernel is for a lot of people to do a lot of hard
work and look at where we can trim fat.
For your last suggestion, maybe someone can automate running Andi's
bloat-o-meter? I think the hard part is maintaining comparable configs.
- R.
Am Mo 31.10.2005 05:52 schrieb Rob Landley <[email protected]>:
> On Sunday 30 October 2005 18:45, Andi Kleen wrote:
> > The problem is that -mm* contains typically so many more or less
> > broken changes that any extensive work on there is futile
> > because you never know whose bugs you're debugging
> > (and if the patch that is broken will even make it anywhere)
> >
> > In short mainline is frozen too long and -mm* is too unstable.
>
> Are you implying that if mainline wasn't frozen so much, it would
> still be
> more stable than -mm?
Yes. Historically 2.<odd> kernels were like this.
-Andi
Am Di 01.11.2005 01:34 schrieb Andrew Morton <[email protected]>:
> It happened somewhere between 2.5.71 and 2.6.8.
>
bloat-o-meter might tell who is to blame. Just run it
with the two vmlinuxes and it will display the differences in symbol
sizes
ftp://ftp.firstfloor.org/pub/ak/bloat-o-meter
-Andi
On Tue, 1 Nov 2005, Roland Dreier wrote:
>
> I think we're actually agreeing. My kmalloc/kzalloc patch is a cute
> hack but magic tricks like that aren't going to shrink the kernel by
> very much and it's probably not worth merging complications like that.
Right. I don't like having too many ways of doing the same thing - it just
confuses things and makes different people have different "coding styles"
and makes things less maintainable (I think the perl philosophy is
broken).
> The way to a smaller kernel is for a lot of people to do a lot of hard
> work and look at where we can trim fat.
Yes. On the other hand, I do believe that bloat is a fact of life.
Eventually somebody younger and leaver will come along and displace us.
It's how things should be. We can (and should try to, of course) only
delay the inevitable ;)
The fact is, we do do more, and we're more complex. Out VM is a _lot_ more
complex, and our VFS layer has grown a lot due to all the support it has
for situations that simply weren't an issue before. And even when not
used, that support is there.
> For your last suggestion, maybe someone can automate running Andi's
> bloat-o-meter? I think the hard part is maintaining comparable configs.
Yes. And we should probably make -Os the default. Apparently Fedora
already does that by just forcibly hacking the Kconfig files.
With modern CPU's, instructions are almost "free". The real cost is in
cache misses, and that tends to be doubly true of system software that
tends to have a lot more cache misses than "normal" programs (because
people try hard to batch up system calls like write etc, so by the time
the kernel is called, the L1 cache is mostly flushed already - possibly
the L2 too. And interrupts may be in the "fast path", but they'd sure as
hell better not happen so often that they stay cached very well etc etc).
So -Os probably performs better in real life, and likely only performs
worse on micro-benchmarks. Sadly, micro-benchmarks are often very
instructive in many other ways.
Linus
On Wed, Nov 02, 2005 at 07:54:04AM -0800, Linus Torvalds wrote:
> > For your last suggestion, maybe someone can automate running Andi's
> > bloat-o-meter? I think the hard part is maintaining comparable configs.
>
> Yes. And we should probably make -Os the default. Apparently Fedora
> already does that by just forcibly hacking the Kconfig files.
(excuse any typos, this wireless connection is god-awful)
We do. We rip out the dependancyon CONFIG_EMBEDDED, and build
with OPTIMISE_FOR_SIZE set. At least we usually do.
Once every so often, we hit something which throws a spanner
in the works, like the "x86-64 doesn't boot any more" problem
that was fixed by the patch that Alexandre posted earlier
this week.
Most of the time now, when we hit bugs with -Os, it seems to be due
to broken asm constraints in the kernel rather than actual
gcc bugs, but of course, they also occur from time to time,
whereas the same code works just fine with -O2.
I think part of th reason for this is exactly because it
doesn't get a great deal of testing.
Dave
On Wed, Nov 02, 2005 at 12:48:52PM -0500, Dave Jones wrote:
> On Wed, Nov 02, 2005 at 07:54:04AM -0800, Linus Torvalds wrote:
>
> > > For your last suggestion, maybe someone can automate running Andi's
> > > bloat-o-meter? I think the hard part is maintaining comparable configs.
> >
> > Yes. And we should probably make -Os the default. Apparently Fedora
> > already does that by just forcibly hacking the Kconfig files.
>
> (excuse any typos, this wireless connection is god-awful)
> We do. We rip out the dependancyon CONFIG_EMBEDDED, and build
> with OPTIMISE_FOR_SIZE set. At least we usually do.
> Once every so often, we hit something which throws a spanner
> in the works, like the "x86-64 doesn't boot any more" problem
> that was fixed by the patch that Alexandre posted earlier
> this week.
>
> Most of the time now, when we hit bugs with -Os, it seems to be due
> to broken asm constraints in the kernel rather than actual
> gcc bugs, but of course, they also occur from time to time,
> whereas the same code works just fine with -O2.
> I think part of th reason for this is exactly because it
> doesn't get a great deal of testing.
Is the usage of -Os in Fedora based on actual measurements?
> Dave
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
On Wed, 2 Nov 2005, Dave Jones wrote:
> On Wed, Nov 02, 2005 at 07:54:04AM -0800, Linus Torvalds wrote:
>
> > > For your last suggestion, maybe someone can automate running Andi's
> > > bloat-o-meter? I think the hard part is maintaining comparable configs.
> >
> > Yes. And we should probably make -Os the default. Apparently Fedora
> > already does that by just forcibly hacking the Kconfig files.
>
> (excuse any typos, this wireless connection is god-awful)
> We do. We rip out the dependancyon CONFIG_EMBEDDED, and build
> with OPTIMISE_FOR_SIZE set. At least we usually do.
> Once every so often, we hit something which throws a spanner
> in the works, like the "x86-64 doesn't boot any more" problem
> that was fixed by the patch that Alexandre posted earlier
> this week.
>
> Most of the time now, when we hit bugs with -Os, it seems to be due
> to broken asm constraints in the kernel rather than actual
> gcc bugs, but of course, they also occur from time to time,
> whereas the same code works just fine with -O2.
> I think part of th reason for this is exactly because it
> doesn't get a great deal of testing.
I used to compile with Os on all my kernels, but when the alignment
settings got added to the embedded section a few kernels ago and I saw
that they all default to 0 (no alignment) it scared me off, becouse I know
that unaligned accesses can be extremely expensive, and I wasn't
comfortable that I had the right info to set them for the different CPU's
that I have.
I would really like to see Os move out of the embedded section so that it
can be tested (and used) without having to worry about all those other
settings. The other option would be to have all those other settings get
good defaults (with an option to reset them to the defaults when you
change the CPU type).
David Lang
--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
> I used to compile with Os on all my kernels, but when the alignment
> settings got added to the embedded section a few kernels ago and I saw
> that they all default to 0 (no alignment) it scared me off
Alignment settings in the EMBEDDED menu are ignored if set to 0, in
other words setting alignment to 0 in Kconfig will fall back to
compilers default values.
Thats also documented in the help for the config options.
Sam
On Wednesday 02 November 2005 00:05, Linus Torvalds wrote:
> On Tue, 1 Nov 2005, Roland Dreier wrote:
> > Anyway, it would be great to find ways to make big improvements. But
> > I think the most realistic way to shrink the kernel is the same way it
> > grows in the first place -- one small piece at a time.
>
> No, I think that's a lost cause.
>
> It doesn't grow by 700 bytes once in a while. It grows by much more, and
> much more often. And we can't fight it that way, that's just not going to
> work. Maybe have something that tracks individual object file sizes and
> shames people into not growing them..
nm --size-sort vmlinux
Old busybox size optimization trick. Running it on 2.6.14 "allnoconfig"
attached. Lots and lots of small symbols (sizes are in hex), with no really
big low-hanging fruit jumping out.
Rob
On Wednesday 02 November 2005 23:31, Sam Ravnborg wrote:
> > I used to compile with Os on all my kernels, but when the alignment
> > settings got added to the embedded section a few kernels ago and I saw
> > that they all default to 0 (no alignment) it scared me off
>
> Alignment settings in the EMBEDDED menu are ignored if set to 0, in
> other words setting alignment to 0 in Kconfig will fall back to
> compilers default values.
>
> Thats also documented in the help for the config options.
In my opinion the alignment settings are completely useless
(far too low level for a user configuration) and should be just
removed. The per cpu or -Os defaults are totally adequate.
-Andi
Linus Torvalds wrote:
>
> On Tue, 1 Nov 2005, Roland Dreier wrote:
>
>>Anyway, it would be great to find ways to make big improvements. But
>>I think the most realistic way to shrink the kernel is the same way it
>>grows in the first place -- one small piece at a time.
>
>
> No, I think that's a lost cause.
>
> It doesn't grow by 700 bytes once in a while. It grows by much more, and
> much more often. And we can't fight it that way, that's just not going to
> work. Maybe have something that tracks individual object file sizes and
> shames people into not growing them..
As mentioned at the kernel summit, we're working on it
at the CE Linux Forum. These things take time to set up,
but we have code already for something that tests
the size impact of individual kernel configs, and we're
working on a test to track individual function sizes for
each new kernel (using Andi Kleen's bloat-o-meter).
We're still in process of setting up the test lab,
but we have a number of hardware boards already,
and we're hoping to be publishing size regression data
for the kernel on a regular basis, starting in about
April of next year.
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================
On Friday 04 November 2005 23:08, Tim Bird wrote:
> starting in about April of next year.
Any particular reason it's taking so long? Would it perhaps be possible
to do a subset (e.g. only a single architecture and less configs) sooner?
If the problem is writing the necessary scripts, I'm sure some people
(e.g. M. Bligh or Al Viro) have already written some that do
at least the automated building and would be happy to
share them.
-Andi
Andi Kleen wrote:
> On Friday 04 November 2005 23:08, Tim Bird wrote:
>
>>starting in about April of next year.
>
> Any particular reason it's taking so long? Would it perhaps be possible
> to do a subset (e.g. only a single architecture and less configs) sooner?
> If the problem is writing the necessary scripts, I'm sure some people
> (e.g. M. Bligh or Al Viro) have already written some that do
> at least the automated building and would be happy to
> share them.
The main reason it's taking so long is that it's
a background task for everyone working on it.
(Basically, me and one guy at NEC. ;-)
We've gotten sidetracked with lab setup issues
(signing a lease for office space, processing
paperwork for board donations, etc. etc.)
But I can run some tests now on a few different
platforms just on my desk.
I had hoped to have at least some preliminary
results by the end of November, with a bigger
call for help inside the forum to finish up
something big and professional-looking in April.
However, maybe I can cobble together some
things, and let people (both in and out of
the forum) look at the preliminary
results sooner.
I'll see what I can do by the end of November.
Thanks for the encouragement.
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Mon, 31 Oct 2005, Willy Tarreau wrote:
> On Sat, Oct 29, 2005 at 01:28:23PM -0700, Linus Torvalds wrote:
>
>> So I'm planning on continuing with it unchanged for now. Two-week merge
>> window until -rc1, and then another -rc kernel roughly every week until
>> release. With the goal being 6 weeks, and 8 weeks being ok.
>>
>> I don't think anybody has been really unhappy with this approach? Hmm?
>
> I believe it was a good experience. However, I still find it sad that
> there are changes between the latest -rc and the final version. This
> time, it seems it did not cause trouble, but in the past, it happened
> several times, because any valid fix can have side effects.
>
> I think that if you could announce what you intend to release with a
> message like "I will make this -final tomorrow", there would be some
> time to test builds in various configs, and check that no obvious bug
> has been introduced.
A -final should never be changed from the last -rc. That defeats the
purpose of having -rc releases (rc == 'release candidate' ;)). If you
do make changes to the last -rc, then you need to release another -rc
with those changes. If all's good, then you can release that /last/
- -rc with /no/ changes as the One and True Release.
- -sandalle
- --
Eric Sandall | Source Mage GNU/Linux Developer
[email protected] | http://www.sourcemage.org/
http://eric.sandall.us/ | SysAdmin @ Inst. Shock Physics @ WSU
http://counter.li.org/ #196285 | http://www.shock.wsu.edu/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
iD8DBQFDbt4KHXt9dKjv3WERAiIXAKClGeeapTkGJ7gFWymHISb99JGw7QCgrPag
lArvgGMUprksA28CX+9I2Bc=
=8xAI
-----END PGP SIGNATURE-----
Eric Sandall <[email protected]> writes:
> A -final should never be changed from the last -rc. That defeats the
> purpose of having -rc releases (rc == 'release candidate' ;)).
This logic is flawed. RCs are for performing tests. If you don't want
further tests (for example, tests on previous RC completed and you're
quite sure new changes introduce no new bugs) you don't need further
RCs.
--
Krzysztof Halasa
Krzysztof Halasa wrote:
> Eric Sandall <[email protected]> writes:
>>A -final should never be changed from the last -rc. That defeats the
>>purpose of having -rc releases (rc == 'release candidate' ;)).
>
>
> This logic is flawed. RCs are for performing tests. If you don't want
> further tests (for example, tests on previous RC completed and you're
> quite sure new changes introduce no new bugs) you don't need further
> RCs.
How do you ever know that new change introduced no new bugs? Maybe
there was a latent race condition that is activated by timing
differences caused by the new code. Maybe it shifts the spacing of the
code just enough to get hit by a pre-existing trampler. Unless you test
it, you *can't* know.
The safe bet is to simply rename the final -rc with no further changes.
Chris
On Mon, 7 Nov 2005, Christopher Friesen wrote:
>
> The safe bet is to simply rename the final -rc with no further changes.
That's not actually a very safe bet at all.
Why? Because only a small percentage of people actually test -rc kernels
in the first place.
So if you release the last -rc as the standard kernel, you (a) get no real
coverage advantage anyway and (b) you just wasted a week in order to try
to get some coverage that you aren't getting.
The current kernel development model is to merge stuff early, which
hopefully motivates the people who _do_ test -rc kernels to actually test
-rc1, since they know that that's the one that has _all_ the really
relevant goodies.
And most of those that do test -rc1 will never see any problems at all.
Those that do, are now more likely to test the rest of the -rcs, hopefully
until their problem is resolved. And those that don't test -rc releases
(because they simply don't upgrade very often) will _never_ test an -rc
release, whether it's the first one or the last one.
So what we have to fight this problem is the stable kernel series. We
release the final 2.6.x kernel with as much testing as we can, but it's
just an undeniable fact that a lot of people will try that kernel only
after the release, and often it might be weeks after the release. Doing
-rc kernels didn't do anything for those cases.
But when they find a problem (or somebody who _did_ test an -rc kernel,
but didn't notice a problem until much later), we try to have a process to
get those issues fixed.
So repeat after me: "Most people never test -rc kernels".
Linus
On Mon, 7 Nov 2005, Linus Torvalds wrote:
>
> So repeat after me: "Most people never test -rc kernels".
Btw, the ones that _do_ test -rc kernels usually don't test all of them.
The current model is set up in a way where there is _one_ special -rc
kernel that we should try to get people to test: the first one.
That hopefully encourages people to try an -rc kernel who might otherwise
decide that there's too many -rc kernels to bother with. If they know that
all of the real development happened before -rc1, they also are thus aware
that it doesn't really matter which -rc kernel they test, so just testing
_one_ is very good indeed.
The first -rc kernel is also special in another way: it's the one we
"wait" for. It's the one that happens after two weeks, and has a deadline.
The others happen more frequently, and are really objectively less
important than the first one.
(In contrast, some other projects try to make the _last_ -rc be the
important one. That's totally the wrong way around, because if there are
more people testing the last one, the testing happens at _exactly_ the
wrong point in time from a "let's fix the problems" standpoint)
So the call to people who can be bothered to test at all: if you
only test one -rc kernel, please test the first one. That way we get a
heads-up on problems earlier.
(And if you like testing -rc kernels, please test all of them. Or even the
nightly snapshots. Or track the git tree several times a day. The more,
the merrier, but if you only want to boot one kernel a month, make it be
the -rc1 kernel).
Linus
"Christopher Friesen" <[email protected]> writes:
> How do you ever know that new change introduced no new bugs?
Changing comments doesn't change generated code or your tools are
screwed. Trivial changes - you can be reasonably sure, too. Note
it's different from not having bugs at all.
> Maybe
> there was a latent race condition that is activated by timing
> differences caused by the new code.
Then the bug already existed, you aren't adding any.
> The safe bet is to simply rename the final -rc with no further changes.
Safe? You can't be really safe here. Bad luck and version string
change (compile time or -rc* removal) will trigger disaster.
--
Krzysztof Halasa
Hi Linus,
you made valid points, you know your public very well :-)
However, I have an objection on one point below.
On Mon, Nov 07, 2005 at 09:28:35AM -0800, Linus Torvalds wrote:
> On Mon, 7 Nov 2005, Linus Torvalds wrote:
> >
> > So repeat after me: "Most people never test -rc kernels".
>
> Btw, the ones that _do_ test -rc kernels usually don't test all of them.
That's true at least for me, except that I *try* to test *ALL* versions which
are announced as the most likely future -final. When Marcelo tells us that
-rc2 will be -final if nobody complains, I really test it. I build it on
several archs and try to catch stupid bugs because I hate stupid bugs in
final releases, they pollute bug reports (worst ones being build errors).
However, when he announces -preX (X>1) or when you annonce any -rc which
is not likely to become -final, I just check the changelog, and build it
if I both see something which applies to my setup, and I have nothing else
to do.
When I ask you to turn -rc into -final, it's not just to avoid adding
new bugs (eventhough the most quickly fixed bugs may be the most dangerous),
but it's mainly to add credibility to the call so that many people who would
be hesitant will try it.
> The current model is set up in a way where there is _one_ special -rc
> kernel that we should try to get people to test: the first one.
>
> That hopefully encourages people to try an -rc kernel who might otherwise
> decide that there's too many -rc kernels to bother with. If they know that
> all of the real development happened before -rc1, they also are thus aware
> that it doesn't really matter which -rc kernel they test, so just testing
> _one_ is very good indeed.
Agreed too. That's the same reason I asked Marcelo if he would agree to
merge all the uncertain recent mcast fixes early in 2.4.33-pre1.
> The first -rc kernel is also special in another way: it's the one we
> "wait" for. It's the one that happens after two weeks, and has a deadline.
> The others happen more frequently, and are really objectively less
> important than the first one.
>
> (In contrast, some other projects try to make the _last_ -rc be the
> important one. That's totally the wrong way around, because if there are
> more people testing the last one, the testing happens at _exactly_ the
> wrong point in time from a "let's fix the problems" standpoint)
Are there really projects managed like this ? I hope I don't use anything
from them !!!
> So the call to people who can be bothered to test at all: if you
> only test one -rc kernel, please test the first one. That way we get a
> heads-up on problems earlier.
Agreed here too from the developper standpoint. However, your final release
are references for all trees. It's better if the common ancestor between
many trees at least builds and does not propagate one horrible bug discovered
10 minutes after release. The least differences there are in code base between
tree, the most relevant the bug reports are. And yes, I know that there is
-stable for this, but most people don't know about it and will simply build
the 2.6.15 announced on slashdot then be the 100th to complain that 2.6.15
does not build because of this or that. (I hope it will not happen :-))
> (And if you like testing -rc kernels, please test all of them. Or even the
> nightly snapshots. Or track the git tree several times a day. The more,
> the merrier, but if you only want to boot one kernel a month, make it be
> the -rc1 kernel).
I consider two rcs as important :
- the first one, to test bugs in features
- the last one to increase the signal/noise ration in bug reports.
Then it makes sense to test them both. However, I agree that if only one can
be tested, the first one is most interesting to us.
> Linus
Regards,
Willy