Hi all,
Andrew was looking for someone to run a linux-next tree that just
contained the subsystem git and quilt trees for 2.6.x+1 and I (in a
moment of madness) volunteered. So, this is to announce the creating of
such a tree (it doesn't exist yet) which will require some (hopefully)
small amount of work on the part of subsystem maintainers.
Those interested in discussion about this are encouraged to join the
[email protected] mailing list.
The first things I need from the subsystem maintainers (you know who you
are) are a contact address (a list address is fine) and at least one git
branch or quilt series that contains all the things you want to see go
into 2.6.26. I am happy for there to be multiple branches/series (in
fact there probably will need to be in some cases where there are
dependencies on others work).
The tree will be based on the current version of Linus' tree but you may
specify an earlier branch point if you need to (hopefully not - but more
likely for quilters, I guess).
I hope to recreate this tree every day automatically. In order to do
this, any tree that has a conflict will be dropped from that days tree.
The maintainer will be notified. I hope to provide some clue as to what
the conflict is with, but probably not initially.
I will attempt to build the tree between each merge (and a failed build
will again cause the offending tree to be dropped). These builds will be
necessarily restricted to probably one architecture/config. I will build
the entire tree on as many architectures/configs as seem sensible and
the results of that will be available on a web page (to be announced).
This is just a start ...
Comments?
[I suspect that Andrew's dream includes actual time to dream :-)]
--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/
For reference - Andrew's original musings:
I perceive the following problems with kernel development:
- The 80-odd subsystem trees are developed in isolation, leading to
- large numbers of patch-time conflicts when I merge them all
- large numbers of compile-time problems
- occasional runtime problems as a result of the merging
This is largely because the subsystem tree owners aren't being
particularly disciplined in keeping their changes contained to their area of
responsibility.
- It's a large amount of work to merge all the trees and it is very dull.
This is the main reason why -mm releases are down to one every few weeks
rather than the old two or more per week.
- Those few -mm releases have a large number of regressions
- Developers aren't particularly responsive in handling those regression
reports.
- Mainline kernel releases also have a large number of regressions: ones
which are leaking through the -mm process.
- Very few kernel developers actually bother testing the -mm tree.
- Consequently most code hits mainline without its submitter having done
much testing of the resulting code combination.
- Few kernel developers are testing other kernel developers' development
code.
- Practically the only public testing which the various trees obtain is via
the -mm process, and that process is collapsing for the above reasons.
- People who develop against a particular tree are not spending sufficient
time reviewing and testing changes which other developers are committing to
other trees.
- People who raise patches are preparing them against Linus's tree. But by
the time we get into the late -rc timeframe, Linus's tree is very different
from the accumulated pending trees. So many patches fail to apply, or don't
work correctly and the originator runtime tested them against Linus's tree,
not against the tree into which they are to be merged.
- Developers are sneaking patches into the mainline merge window at the last
minute, before they have had a decent period of review and testing.
So what I propose is that we start a formal unified development tree (called
the "linux-next" tree). It will probably have the following characteristics:
- Once per day it merges all the subsystem trees: 75 git trees, 5-10 quilt
trees.
- It will need to run compilation tests after applying each tree
It's unreasonable to do randconfig on arch/kooky-arch/ during linux-next's
integration, but a basic x86_64 allmodconfig would be good: something which
is well-known by the subsystem maintainers so they can check that it will
work.
- It will send status messages to a new mailing list and also to individual
developers when something goes wrong.
- There are several trees which are overtly cross-subsystem and which might
not fit into this model: mainly the driver tree, the -mm tree and perhaps
the PCI tree.
otoh, a lot of the -mm queue is _not_ cross-subsystem: fbdev, uml, various
new drivers and features, etc. So I'll need to split -mm into multiple
queues and feed many of them into linux-next.
- Advanced topic: mechanisms will be put in place to detect a patch which
hasn't been in linux-next for "long enough". These patches will be flagged
for extra consideration when offered for a mainline merge.
- Note that James is already running a cut-down version of this. See
http://lkml.org/lkml/2008/1/17/481
Once the linux-next tree is up and running and is reliable and useful, I would
like to see the following policies implemented around it:
- If a tree doesn't automerge it is dropped and a notification is sent
- If a tree breaks the build it is dropped and a notification is sent
- If a tree's server cannot be contacted (a fairly regular event) it is
dropped and a notification is sent
- If a tree is causing runtime problems (ie: regressions) which impact
testing of the whole and those problems are not getting appropriately
resolved, the tree is dropped. (This will require manual intervention and
visible decision-making).
- If a tree was dropped for N% of the M days before 2.6.x is released, that
tree is ineligible for a 2.6.x+1 merge. This is for good software
engineering reasons, and to provide an incentive for developers to pay more
attention to integration and testing issues.
Other:
- The -mm tree will become a much smaller patchset, based on linux-next.
- People can set up linux-next cross-compilation farms on external servers.
Compilation errors can be fed back as regular bug reports.
Advantages:
- Will lead to kernel developers testing (and hopefully reviewing) each
others code more
- Will lead to kernel developers paying more attention to integration issues
- I expect that tree-owners will be a bit more careful about what they check
into their trees, knowing that it'll go straight into linux-next.
- Less bugs will hit Linus's tree:
- I expect we will get a lot of public testing of the linux-next tree
- Most kernel developers will also need to test linux-next, as that is
the tree they're integrating into. And kernel developers are the best
testers, bug-reporters and bug-fixers.
- It will give me far more time to do useful things like testing, bugfixing
and reviewing. Particularly because I don't intend to be the person who
operates linux-next.
- Patch submitters can now prepare and test against the linux-next tree
rather than against the relatively out-of-date mainline tree.
Disadvantages
- More work for subsystem maintainers.
But I don't think it's wasted time - it's largely work which would have
needed to be done later anyway.
If the subsystem maintainer is merging non-buggy patches and is staying
strictly within area-of-responsibility then there should be close to zero
extra work.
- Subsystem maintainers will need new strategies for those patches which
touch other subsystems.
- Ask submitters to split things up better (not a bad thing)
- Arrange for interface changes to be implemented in a
temporarily-back-compatible manner if poss (also not a bad thing).
- Send some patches over to other subsystem maintainers rather than
merging them locally (also not a bad thing).
- Subsystem maintainers will discover that getting other subsystem
maintainers to look at and merge their patches can be a right royal pita.
Trust me.
It is inefficient to have 80 subsystem maintainers re-re-re-sending
patches at each other.
- People can copy me on them and I can do the re-re-re-send for them,
while including the patches in the linux-next lineup.
- Or the subsystem maintainer can set up a standalone "patches from Tom
to Dick" tree, and include that in the linux-next lineup (staged after
Dick's tree) until Dick wakes up and merges it.
Open issues:
I don't want to over-design this at this stage. This hasn't been done
before and I'd expect the linux-next owner would be the person who
encounters and solves most of the problems which crop up. But I do
anticipate a few issues:
- Tree prioritisation. Often, two trees will collide. Say, git-net and
git-nfs. If this happens, which tree will be autodropped?
I'd suggest that we'll just need to make decisions about which tree is
most important, and stage that tree ahead of less important ones.
- I'd expect subsystem tree maintainers to decide to start running multiple
trees (like Jeff's "topic branches"). This way, if there are collisions or
other failures, then not all of their work is dropped.
- Also, a subsystem tree maintainer may have some changes which are
considered likely to collide with other people's work. These can be placed
in a best-effort standalone branch which is staged late in the linux-next
merging order and if it breaks, well, no great loss.
- A consequence of all the above is that it should be easy for subsystem
tree maintainers to add new trees to linux-next, and for them to decide
where in the merging order those trees are staged.
They won't want to have to send an email to another developer for this -
each tree-owner will need the ability to add trees with no turnaround delay.
- This could be implemented with a world-writeable control file on
kernel.org
- Or a world-writeable directory on kernel.org with control files which
start with a six-digit number
- Or some fancy web interface or command-line tool on kernel.org
- Or whatever.
- The need to perform compilation testing at merge-time might cause overload
at kernel.org, so an upgraded or alternative server might be needed.
And experience tells me that we _will_ need to perform compilation
testing.
I'm assuming that compilation testing will be a linear search: once per
tree. One could do bisection searching, but a) that gets tricky when more
than one tree has a failure and b) it probably means that the whole tree
needs to be re-merged once the offending tree has been weeded out.
- Ideally, all the subsystem trees are standalone and can be merged by Linus
in any order. There will be _some_ trees which are dependent upon other
subsystem trees, but these are special cases and can be handled by staging
them late in the linux-next merging order, and by not merging them in
mainline until the prerequisites are merged.
Obviously, the safest approach would be to merge trees into mainline in
the same order as they are merged into linux-next, but this is probably
impractical.
On Tue, 2008-02-12 at 12:02 +1100, Stephen Rothwell wrote:
> Hi all,
>
> Andrew was looking for someone to run a linux-next tree that just
> contained the subsystem git and quilt trees for 2.6.x+1 and I (in a
> moment of madness) volunteered. So, this is to announce the creating of
> such a tree (it doesn't exist yet) which will require some (hopefully)
> small amount of work on the part of subsystem maintainers.
Actually, it sort of does. If you look here:
http://www.kernel.org/pub/linux/kernel/people/jejb/merge-tree/
You'll find a merge candidate tree that builds nightly from everyone's
git and quilt trees. I'm using it to track merge conflicts (so I only
build the patch, I don't check it compiles).
You're welcome to the scripts that do this:
http://www.kernel.org/pub/linux/kernel/people/jejb/build.pl
And the config file that runs it:
http://www.kernel.org/pub/linux/kernel/people/jejb/merge-tree-build
I don't plan to do much more than keep it building to check conflicts,
so you're welcome to take it over.
> Those interested in discussion about this are encouraged to join the
> [email protected] mailing list.
>
> The first things I need from the subsystem maintainers (you know who you
> are) are a contact address (a list address is fine) and at least one git
> branch or quilt series that contains all the things you want to see go
> into 2.6.26. I am happy for there to be multiple branches/series (in
> fact there probably will need to be in some cases where there are
> dependencies on others work).
>
> The tree will be based on the current version of Linus' tree but you may
> specify an earlier branch point if you need to (hopefully not - but more
> likely for quilters, I guess).
>
> I hope to recreate this tree every day automatically. In order to do
> this, any tree that has a conflict will be dropped from that days tree.
> The maintainer will be notified. I hope to provide some clue as to what
> the conflict is with, but probably not initially.
Actually the experiment with the -mc tree shows that most of the
conflicts are trivial in nature (usually docbook stuff or
feature-removal.txt stuff), so you can do a trivial triage by hand. You
can't automatically drop them (well, not unless you want to end up
dropping half the trees).
The other problem is that we actually maintain deliberate conflicts with
a last person to merge fixes it type attitude. Again, it's usually in
minor areas, and the fixups are fairly trivial, but it illustrates why
conflicts can't be a reason to drop a tree, you have to maintain some
sort of automatic fixup (at least I had to with the -mc tree). The
reason we do this is that it would give the maintainers a nasty web of
included trees (which is almost impossible for the quilt trees anyway)
if we tried to resolve the conflicts and destroy our ability to rebase.
> I will attempt to build the tree between each merge (and a failed build
> will again cause the offending tree to be dropped). These builds will be
> necessarily restricted to probably one architecture/config. I will build
> the entire tree on as many architectures/configs as seem sensible and
> the results of that will be available on a web page (to be announced).
Yes, this is the bit I've never dared do ... principally because it's
such a time sink.
James
Hi James,
On Mon, 11 Feb 2008 19:36:49 -0600 James Bottomley <[email protected]> wrote:
>
> On Tue, 2008-02-12 at 12:02 +1100, Stephen Rothwell wrote:
> >
> > Andrew was looking for someone to run a linux-next tree that just
> > contained the subsystem git and quilt trees for 2.6.x+1 and I (in a
> > moment of madness) volunteered. So, this is to announce the creating of
> > such a tree (it doesn't exist yet) which will require some (hopefully)
> > small amount of work on the part of subsystem maintainers.
>
> Actually, it sort of does. If you look here:
Yes, Andrew pointed me there and I should have mentioned it, sorry.
> http://www.kernel.org/pub/linux/kernel/people/jejb/merge-tree/
>
> You'll find a merge candidate tree that builds nightly from everyone's
> git and quilt trees. I'm using it to track merge conflicts (so I only
> build the patch, I don't check it compiles).
>
> You're welcome to the scripts that do this:
>
> http://www.kernel.org/pub/linux/kernel/people/jejb/build.pl
>
> And the config file that runs it:
>
> http://www.kernel.org/pub/linux/kernel/people/jejb/merge-tree-build
>
> I don't plan to do much more than keep it building to check conflicts,
> so you're welcome to take it over.
Thanks, they are very useful.
> > I hope to recreate this tree every day automatically. In order to do
> > this, any tree that has a conflict will be dropped from that days tree.
> > The maintainer will be notified. I hope to provide some clue as to what
> > the conflict is with, but probably not initially.
>
> Actually the experiment with the -mc tree shows that most of the
> conflicts are trivial in nature (usually docbook stuff or
> feature-removal.txt stuff), so you can do a trivial triage by hand. You
> can't automatically drop them (well, not unless you want to end up
> dropping half the trees).
Well I did a trial run with the 40 git trees in the latest -mm and got
only 6 conflicts (which actually were not trivial, unfortunately). Of
course, a lot of them have already been pulled into Linus' tree by now.
> The other problem is that we actually maintain deliberate conflicts with
> a last person to merge fixes it type attitude. Again, it's usually in
I was hoping to be able to automatically find the other tree involved in
a conflict and point both maintainers at the problem. Also, there is
always the possibility of reordering the trees ;-)
> minor areas, and the fixups are fairly trivial, but it illustrates why
> conflicts can't be a reason to drop a tree, you have to maintain some
> sort of automatic fixup (at least I had to with the -mc tree). The
OK, maybe if the conflict is trivial, we can do fixups.
> reason we do this is that it would give the maintainers a nasty web of
> included trees (which is almost impossible for the quilt trees anyway)
> if we tried to resolve the conflicts and destroy our ability to rebase.
I am hoping (one of Andrew's bugbears) that over time we will end up with
several branches from git using trees (the vast majority) most of which
are completely contained within their own subsystem and don't depend on
anything but Linus' tree. The conflicting and dependent branches will be
merged later in the sequence. Thus we will end up with a large amount of
the tree becoming stable as the merge window approaches. (Yes, sometimes
I am an optimist :-))
> > I will attempt to build the tree between each merge (and a failed build
> > will again cause the offending tree to be dropped). These builds will be
> > necessarily restricted to probably one architecture/config. I will build
> > the entire tree on as many architectures/configs as seem sensible and
> > the results of that will be available on a web page (to be announced).
>
> Yes, this is the bit I've never dared do ... principally because it's
> such a time sink.
I have a couple of lackeys who already do this stuff :-)
Thanks for the comments - I will keep them in mind as I sink into the abyss :-)
--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/
On Tue, 12 Feb 2008 12:02:08 +1100 Stephen Rothwell <[email protected]> wrote:
>
> Andrew was looking for someone to run a linux-next tree that just
> contained the subsystem git and quilt trees for 2.6.x+1 and I (in a
> moment of madness) volunteered.
I neglected to mention the other brave souls who volunteered: Frank
Seidel, Ann Davis, and Harvey Harrison who I hope to be able to call on
in times of need.
--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/
On Tue, 2008-02-12 at 13:23 +1100, Stephen Rothwell wrote:
> Hi James,
>
> On Mon, 11 Feb 2008 19:36:49 -0600 James Bottomley <[email protected]> wrote:
> >
> > On Tue, 2008-02-12 at 12:02 +1100, Stephen Rothwell wrote:
> > >
> > > Andrew was looking for someone to run a linux-next tree that just
> > > contained the subsystem git and quilt trees for 2.6.x+1 and I (in a
> > > moment of madness) volunteered. So, this is to announce the creating of
> > > such a tree (it doesn't exist yet) which will require some (hopefully)
> > > small amount of work on the part of subsystem maintainers.
> >
> > Actually, it sort of does. If you look here:
>
> Yes, Andrew pointed me there and I should have mentioned it, sorry.
>
> > http://www.kernel.org/pub/linux/kernel/people/jejb/merge-tree/
> >
> > You'll find a merge candidate tree that builds nightly from everyone's
> > git and quilt trees. I'm using it to track merge conflicts (so I only
> > build the patch, I don't check it compiles).
> >
> > You're welcome to the scripts that do this:
> >
> > http://www.kernel.org/pub/linux/kernel/people/jejb/build.pl
> >
> > And the config file that runs it:
> >
> > http://www.kernel.org/pub/linux/kernel/people/jejb/merge-tree-build
> >
> > I don't plan to do much more than keep it building to check conflicts,
> > so you're welcome to take it over.
>
> Thanks, they are very useful.
>
> > > I hope to recreate this tree every day automatically. In order to do
> > > this, any tree that has a conflict will be dropped from that days tree.
> > > The maintainer will be notified. I hope to provide some clue as to what
> > > the conflict is with, but probably not initially.
> >
> > Actually the experiment with the -mc tree shows that most of the
> > conflicts are trivial in nature (usually docbook stuff or
> > feature-removal.txt stuff), so you can do a trivial triage by hand. You
> > can't automatically drop them (well, not unless you want to end up
> > dropping half the trees).
>
> Well I did a trial run with the 40 git trees in the latest -mm and got
> only 6 conflicts (which actually were not trivial, unfortunately). Of
> course, a lot of them have already been pulled into Linus' tree by now.
Hmm, OK of the 46 I pull in, it's only xfs causing difficulties at the
moment. Pre 2.6.24 being declared, we had about 10 fixups, all of which
were trivial except for one or two I got the subsystems to fix.
> > The other problem is that we actually maintain deliberate conflicts with
> > a last person to merge fixes it type attitude. Again, it's usually in
>
> I was hoping to be able to automatically find the other tree involved in
> a conflict and point both maintainers at the problem. Also, there is
> always the possibility of reordering the trees ;-)
It's frequently just not possible. There are a lot of global API
changes that sweep across subsystems. For instance, I had a trivial
fixup script for scsi and net because of a netlink API change.
> > minor areas, and the fixups are fairly trivial, but it illustrates why
> > conflicts can't be a reason to drop a tree, you have to maintain some
> > sort of automatic fixup (at least I had to with the -mc tree). The
>
> OK, maybe if the conflict is trivial, we can do fixups.
Right .. it's really not possible to work without an infrastructure that
does this.
> > reason we do this is that it would give the maintainers a nasty web of
> > included trees (which is almost impossible for the quilt trees anyway)
> > if we tried to resolve the conflicts and destroy our ability to rebase.
>
> I am hoping (one of Andrew's bugbears) that over time we will end up with
> several branches from git using trees (the vast majority) most of which
> are completely contained within their own subsystem and don't depend on
> anything but Linus' tree. The conflicting and dependent branches will be
> merged later in the sequence. Thus we will end up with a large amount of
> the tree becoming stable as the merge window approaches. (Yes, sometimes
> I am an optimist :-))
We can't really do this. We don't work in a utopian my area only
environment. Almost every release cycle involves dealing with patches
that cross subsystem boundaries. Often we try to minimise the problems
by taking patches from different subsystems via our trees to keep the
dependencies straight, but it does lead to conflicts. Most of them are
ones we're aware of, and easily resolvable as the trees go in in the
merge window. So any infrastructure that builds a unified tree has to
be aware of them too.
James
On Tue, Feb 12, 2008 at 12:02:08PM +1100, Stephen Rothwell wrote:
> Hi all,
>
> Andrew was looking for someone to run a linux-next tree that just
> contained the subsystem git and quilt trees for 2.6.x+1 and I (in a
> moment of madness) volunteered. So, this is to announce the creating of
> such a tree (it doesn't exist yet) which will require some (hopefully)
> small amount of work on the part of subsystem maintainers.
>
> Those interested in discussion about this are encouraged to join the
> [email protected] mailing list.
>
> The first things I need from the subsystem maintainers (you know who you
> are) are a contact address (a list address is fine) and at least one git
> branch or quilt series that contains all the things you want to see go
> into 2.6.26.
Note that a lot of these are already in the MAINTAINERS file.
But for the record, here's mine, in the order they need to be pulled
from.
Driver core:
kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/
PCI:
kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-02-pci/
USB:
kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-03-usb/
These are all quilt trees, with the series file in the directory for the
order of the patches, and a README saying what kernel version they have
been rebased against.
> I am happy for there to be multiple branches/series (in
> fact there probably will need to be in some cases where there are
> dependencies on others work).
>
> The tree will be based on the current version of Linus' tree but you may
> specify an earlier branch point if you need to (hopefully not - but more
> likely for quilters, I guess).
>
> I hope to recreate this tree every day automatically. In order to do
> this, any tree that has a conflict will be dropped from that days tree.
Oh oh oh, I get merged first! me me me!
> The maintainer will be notified. I hope to provide some clue as to what
> the conflict is with, but probably not initially.
>
> I will attempt to build the tree between each merge (and a failed build
> will again cause the offending tree to be dropped).
This is going to get really interesting, especially when (not if) we do
more global api changes. Look at the last round of kobject changes.
That touched a lot of different places, and other trees ended up not
building because of it, because I changed apis and they had added new
code based on the old apis.
I think the only way to fix this is not going to just "drop the tree"
like you are suggesting, but to let both people know (the person who
caused the change, and the person who's tree broke after the merge), and
then either add a "fixup patch" for the build like Andrew has been
doing, or disabling something from the build section.
As I know I'm going to be changing more driver core apis[1] this week,
I'm sure we will get a very good set of examples of this for you to see
in action :)
Good luck,
greg k-h
[1] Hopefully the "multiple drivers for a single device" feature people
have been asking for for years will be landing soon, of course the
number of odd places in the kernel that made the assumption that we
could only have one driver per device is causing lots of fun...
On Mon, 11 Feb 2008 20:21:33 -0800
Greg KH <[email protected]> wrote:
> > The maintainer will be notified. I hope to provide some clue as to
> > what the conflict is with, but probably not initially.
> >
> > I will attempt to build the tree between each merge (and a failed
> > build will again cause the offending tree to be dropped).
>
> This is going to get really interesting, especially when (not if) we
> do more global api changes. Look at the last round of kobject
> changes. That touched a lot of different places, and other trees
> ended up not building because of it, because I changed apis and they
> had added new code based on the old apis.
>
> I think the only way to fix this is not going to just "drop the tree"
> like you are suggesting, but to let both people know (the person who
> caused the change, and the person who's tree broke after the merge),
> and then either add a "fixup patch" for the build like Andrew has been
> doing, or disabling something from the build section.
>
in my experience, the only chance you have is doing API changes as first in the set of changes,
and then hoping (making) all other trees use the new APIs. Any other order just turns into
an impossible mismash.
I can see the point of doing an LKML annouce of the new API after the series is done, so that
all other maintainers have a chance to fix their trees (this of course is only for new occurances
of the old api showing up; the api change series should convert all existing users)
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Mon, Feb 11, 2008 at 08:31:46PM -0800, Arjan van de Ven wrote:
> On Mon, 11 Feb 2008 20:21:33 -0800
> Greg KH <[email protected]> wrote:
>
>
> > > The maintainer will be notified. I hope to provide some clue as to
> > > what the conflict is with, but probably not initially.
> > >
> > > I will attempt to build the tree between each merge (and a failed
> > > build will again cause the offending tree to be dropped).
> >
> > This is going to get really interesting, especially when (not if) we
> > do more global api changes. Look at the last round of kobject
> > changes. That touched a lot of different places, and other trees
> > ended up not building because of it, because I changed apis and they
> > had added new code based on the old apis.
> >
> > I think the only way to fix this is not going to just "drop the tree"
> > like you are suggesting, but to let both people know (the person who
> > caused the change, and the person who's tree broke after the merge),
> > and then either add a "fixup patch" for the build like Andrew has been
> > doing, or disabling something from the build section.
> >
>
> in my experience, the only chance you have is doing API changes as
> first in the set of changes, and then hoping (making) all other trees
> use the new APIs. Any other order just turns into an impossible
> mismash.
I agree, and that's what I do.
The problem is, the API change is still in my tree. So, if for example,
the IB tree goes and adds some new functionality before my API changes
have landed, they need to use the "old" API in order for them to be able
to test and build things on their own. Then, when the -next tree merges
everything together, the IB tree breaks the build, not my driver tree.
It's those "who goes first" type things that end up being the cause of a
lot of Andrew's headaches I think :)
thanks,
greg k-h
On Mon, 2008-02-11 at 20:31 -0800, Arjan van de Ven wrote:
> in my experience, the only chance you have is doing API changes as first in the set of changes,
> and then hoping (making) all other trees use the new APIs. Any other order just turns into
> an impossible mismash.
It would be very nice to have a separate tree with _only_ API changes
that could be frozen well before Linus' merge window opens. It should be
a requirement that maintainers use this tree as a basis for testing API
changes and even test that their own changesets were properly integrated
with the changed APIs.
As it is, we tend to get the order entirely wrong: API changes typically
get merged _after_ the maintainer changes have been merged, and so no
API changes are ever really integration tested.
Cheers
Trond
On Mon, 11 Feb 2008 20:21:33 -0800 Greg KH <[email protected]> wrote:
>
> Note that a lot of these are already in the MAINTAINERS file.
>
> But for the record, here's mine, in the order they need to be pulled
> from.
> Driver core:
> kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/
> PCI:
> kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-02-pci/
> USB:
> kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-03-usb/
>
> These are all quilt trees, with the series file in the directory for the
> order of the patches, and a README saying what kernel version they have
> been rebased against.
Thank you. Over time we might think about some sort of standard for
these.
> Oh oh oh, I get merged first! me me me!
What's it worth to you? :-)
> This is going to get really interesting, especially when (not if) we do
> more global api changes. Look at the last round of kobject changes.
> That touched a lot of different places, and other trees ended up not
> building because of it, because I changed apis and they had added new
> code based on the old apis.
This is one of the things that linux-next will hopefully let us discover
more easily/faster.
> I think the only way to fix this is not going to just "drop the tree"
> like you are suggesting, but to let both people know (the person who
> caused the change, and the person who's tree broke after the merge), and
> then either add a "fixup patch" for the build like Andrew has been
> doing, or disabling something from the build section.
Right. Except that "drop the tree" will probably only mean for a day or
so i.e. it will be taken out of the current round but will reappear
automatically when the conflict/dependency is sorted out.
> As I know I'm going to be changing more driver core apis[1] this week,
> I'm sure we will get a very good set of examples of this for you to see
> in action :)
Excellent!
However, I am hoping that these global api changes may be introduced in a
more orderly fashion (some of which is happening already) by creating new
api's and then switching to them (and them maybe changing the names back
if necessary). And, yes, I realise that this is sometimes not possible
(or at least not worth the extra effort).
> Good luck,
Thanks, I will probably need it :-(
--
Cheers,
Stephen Rothwell [email protected]
On Mon, 11 Feb 2008 20:43:14 -0800
Greg KH <[email protected]> wrote:
> On Mon, Feb 11, 2008 at 08:31:46PM -0800, Arjan van de Ven wrote:
> > On Mon, 11 Feb 2008 20:21:33 -0800
> > Greg KH <[email protected]> wrote:
> >
> >
> > > > The maintainer will be notified. I hope to provide some clue
> > > > as to what the conflict is with, but probably not initially.
> > > >
> > > > I will attempt to build the tree between each merge (and a
> > > > failed build will again cause the offending tree to be dropped).
> > >
> > > This is going to get really interesting, especially when (not if)
> > > we do more global api changes. Look at the last round of kobject
> > > changes. That touched a lot of different places, and other trees
> > > ended up not building because of it, because I changed apis and
> > > they had added new code based on the old apis.
> > >
> > > I think the only way to fix this is not going to just "drop the
> > > tree" like you are suggesting, but to let both people know (the
> > > person who caused the change, and the person who's tree broke
> > > after the merge), and then either add a "fixup patch" for the
> > > build like Andrew has been doing, or disabling something from the
> > > build section.
> > >
> >
> > in my experience, the only chance you have is doing API changes as
> > first in the set of changes, and then hoping (making) all other
> > trees use the new APIs. Any other order just turns into an
> > impossible mismash.
>
> I agree, and that's what I do.
>
> The problem is, the API change is still in my tree. So, if for
> example, the IB tree goes and adds some new functionality before my
> API changes have landed, they need to use the "old" API in order for
> them to be able to test and build things on their own. Then, when
> the -next tree merges everything together, the IB tree breaks the
> build, not my driver tree.
>
> It's those "who goes first" type things that end up being the cause
> of a lot of Andrew's headaches I think :)
>
this is why you need specific trees for just the API change, and these
need to EXPLICITLY go first before EVERYTHING ELSE. Yes this needs a bit of coordination,
but it's the only way.
On Mon, Feb 11, 2008 at 11:45:55PM -0500, Trond Myklebust wrote:
> It would be very nice to have a separate tree with _only_ API changes
> that could be frozen well before Linus' merge window opens. It should be
> a requirement that maintainers use this tree as a basis for testing API
> changes and even test that their own changesets were properly integrated
> with the changed APIs.
The other way that might work in some circumstances would be if we
tried a little harder to avoid API changes that don't involve an
interface naming change. That is, instead of adding a new parameter
to a function, and then having to sweep through all of the trees to
catch all of the users of siad function, we could instead add a new a
new interface, __deprecate the old one, and then give enough time for
trees to adapt, you can avoid needing to do flag day transitions. If
the old interface is __deprecated at the beginning of the merge
window, and then disappears at the very end of the merge window,
that's plenty of time for the subsystem maintainers to move to the new
interface.
This doesn't always work, of course (for example, if we make a
fundamental change in how some critical low-level data structure is
locked). But every little bit that we can do to avoid the tree
integration pain would be a win.
Regards,
- Ted
On Mon, Feb 11, 2008 at 09:17:51PM -0800, Arjan van de Ven wrote:
> On Mon, 11 Feb 2008 20:43:14 -0800
> Greg KH <[email protected]> wrote:
>
> > On Mon, Feb 11, 2008 at 08:31:46PM -0800, Arjan van de Ven wrote:
> > > On Mon, 11 Feb 2008 20:21:33 -0800
> > > Greg KH <[email protected]> wrote:
> > >
> > >
> > > > > The maintainer will be notified. I hope to provide some clue
> > > > > as to what the conflict is with, but probably not initially.
> > > > >
> > > > > I will attempt to build the tree between each merge (and a
> > > > > failed build will again cause the offending tree to be dropped).
> > > >
> > > > This is going to get really interesting, especially when (not if)
> > > > we do more global api changes. Look at the last round of kobject
> > > > changes. That touched a lot of different places, and other trees
> > > > ended up not building because of it, because I changed apis and
> > > > they had added new code based on the old apis.
> > > >
> > > > I think the only way to fix this is not going to just "drop the
> > > > tree" like you are suggesting, but to let both people know (the
> > > > person who caused the change, and the person who's tree broke
> > > > after the merge), and then either add a "fixup patch" for the
> > > > build like Andrew has been doing, or disabling something from the
> > > > build section.
> > > >
> > >
> > > in my experience, the only chance you have is doing API changes as
> > > first in the set of changes, and then hoping (making) all other
> > > trees use the new APIs. Any other order just turns into an
> > > impossible mismash.
> >
> > I agree, and that's what I do.
> >
> > The problem is, the API change is still in my tree. So, if for
> > example, the IB tree goes and adds some new functionality before my
> > API changes have landed, they need to use the "old" API in order for
> > them to be able to test and build things on their own. Then, when
> > the -next tree merges everything together, the IB tree breaks the
> > build, not my driver tree.
> >
> > It's those "who goes first" type things that end up being the cause
> > of a lot of Andrew's headaches I think :)
> >
>
> this is why you need specific trees for just the API change, and these
> need to EXPLICITLY go first before EVERYTHING ELSE. Yes this needs a
> bit of coordination, but it's the only way.
Even then, it will not work.
Again, Roland isn't going to want to always pull in my driver tree just
to build his tree. He wants to, and needs to do his own development
effort.
But when we merge them together, there would be problems.
So, you can't just "drop" the IB tree.
You can't just "drip" my tree.
Where do you "fix this up" at? I can send a patch for the IB tree, but
Roland can't put it in his tree, and I can't put it in my tree, it needs
to go _after_ both of our trees.
That's what -mm has been able to handle so far, and that needs to also
work with -next.
thanks,
greg k-h
On Tue, Feb 12, 2008 at 04:07:04PM +1100, Stephen Rothwell wrote:
> On Mon, 11 Feb 2008 20:21:33 -0800 Greg KH <[email protected]> wrote:
> > I think the only way to fix this is not going to just "drop the tree"
> > like you are suggesting, but to let both people know (the person who
> > caused the change, and the person who's tree broke after the merge), and
> > then either add a "fixup patch" for the build like Andrew has been
> > doing, or disabling something from the build section.
>
> Right. Except that "drop the tree" will probably only mean for a day or
> so i.e. it will be taken out of the current round but will reappear
> automatically when the conflict/dependency is sorted out.
See my response to Arjan for how this is going to be very difficult to
do, unless you are willing to have patches in -next that are not in any
other tree.
> > As I know I'm going to be changing more driver core apis[1] this week,
> > I'm sure we will get a very good set of examples of this for you to see
> > in action :)
>
> Excellent!
>
> However, I am hoping that these global api changes may be introduced in a
> more orderly fashion (some of which is happening already) by creating new
> api's and then switching to them (and them maybe changing the names back
> if necessary). And, yes, I realise that this is sometimes not possible
> (or at least not worth the extra effort).
It's possible to do in a series of patches, yes, but again, development
happens in parallel, with no one stopping for anyone else, and that's
fine, we work it out when we send stuff to Linus at merge time.
So with that parallel development effort, there are problems like this,
I just want you to be aware of it and plan properly for it, as it is
going to happen...
thanks,
greg k-h
From: Theodore Tso <[email protected]>
Date: Tue, 12 Feb 2008 00:11:36 -0500
> __deprecate the old one,
Deprecate is garbage, shit hangs around in the tree forever
and people just turn off the warnings.
Clean sweeps work much better, albeit with some merge pain,
we'll cope.
From: Greg KH <[email protected]>
Date: Mon, 11 Feb 2008 21:53:12 -0800
> Where do you "fix this up" at? I can send a patch for the IB tree, but
> Roland can't put it in his tree, and I can't put it in my tree, it needs
> to go _after_ both of our trees.
Totally agreed.
The fact is there are interdependencies, especially in driver
land and you have to either:
1) Make the driver folks work on top of Greg's tree.
2) Constantly rebase the -next tree to deal with the
conflicts.
There are some other issues related to this which haven't
be touched upon greatly yet.
I rebase my tree all the time, at least once or twice per
week. Why?
Firstly, to remove crap. When you have "great idea A" then "oh shit A
won't work, revert that" there is zero sense in keeping both
changesets around.
Secondly, I want to fix up the rejects caused by conflicts with
upstream bug fixes and the like (and there are tons when the tree gets
to 1500 or so patches like the networking did). I don't want git to
merge the thing by hand, I want to see what the conflict is and make
sure the "obvious" resolution is OK and the most efficient way I know
how to do that is to suck my tree apart as patches, then suck them
back into a fresh tree.
It therefore might make sense to the linux-next tree to do something
similar, constantly rebasing so that all the conflicts and reverted
shit changes can be sorted out without having an incredibly ugly
GIT history.
On Mon, 11 Feb 2008 21:56:48 -0800 Greg KH <[email protected]> wrote:
>
> It's possible to do in a series of patches, yes, but again, development
> happens in parallel, with no one stopping for anyone else, and that's
> fine, we work it out when we send stuff to Linus at merge time.
>
> So with that parallel development effort, there are problems like this,
> I just want you to be aware of it and plan properly for it, as it is
> going to happen...
Understood.
--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/
From: Arjan van de Ven <[email protected]>
Date: Mon, 11 Feb 2008 21:17:51 -0800
> this is why you need specific trees for just the API change
API trees don't work, just like other changes they will have
interdependencies on things like fixups, cleanups, etc.
This is why, with the networking, we've just tossed all of the network
driver stuff in there too. I can rebase freely, remove changesets,
rework them, etc. and this causes a very low amount of pain for Jeff
Garzik and John Linville.
This may or not be a legitimate thing to do with other heavily
dependant subsystems.
On Tue, 12 Feb 2008 00:11:36 -0500 Theodore Tso <[email protected]> wrote:
> On Mon, Feb 11, 2008 at 11:45:55PM -0500, Trond Myklebust wrote:
> > It would be very nice to have a separate tree with _only_ API changes
> > that could be frozen well before Linus' merge window opens. It should be
> > a requirement that maintainers use this tree as a basis for testing API
> > changes and even test that their own changesets were properly integrated
> > with the changed APIs.
>
> The other way that might work in some circumstances would be if we
> tried a little harder to avoid API changes that don't involve an
> interface naming change. That is, instead of adding a new parameter
> to a function, and then having to sweep through all of the trees to
> catch all of the users of siad function, we could instead add a new a
> new interface, __deprecate the old one, and then give enough time for
> trees to adapt, you can avoid needing to do flag day transitions.
yup, in many cases that's pretty easy to do. But people dive straight
for the end result, which of course provides the cleanest outcome but
isn't really very real-wordly.
Another thing we could do when these things happen is all gang up on Linus
and ask him to merge the API change into mainline. Because often the
change can be done in two stages: 1) change the interface then 2) add the code
in the callee which _uses_ that changed interface. Part 1 is an obviously-safe
do-nothing change and fixes the merge problems. Part 2 is at a single site
and can be merged in 2.6.x+1.
There are lots of well-known things we can do to simplify Stephen's life.
But first we have to have the motivation to do that thing. If linux-next
proves to be useful and developers are seeing good testing and
bug-reporting coming out of it then I expect people will be happy to make
such accommodations. Let's see if we can get to that stage first.
A fully running and successful linux-next won't change outcomes a lot, I
expect. It will help to pick up on obvious bugs and build errors and
bisection breakages prior to them hitting mainline. But we fix those
things by -rc1 or -rc2 anyway. So it's just a time-shift, causing no great
change in the final 2.6.x.
linux-next will do little to solve our (IMO) largest problem: unfixed
regressions and bugs. otoh it will free up a lot of my time and
hair-tearing, so I can devote more attention to the bugs. (where
"attention" usually= "sending irritiating emails")
On Mon, 2008-02-11 at 22:11 -0800, David Miller wrote:
> From: Arjan van de Ven <[email protected]>
> Date: Mon, 11 Feb 2008 21:17:51 -0800
>
> > this is why you need specific trees for just the API change
>
> API trees don't work, just like other changes they will have
> interdependencies on things like fixups, cleanups, etc.
>
> This is why, with the networking, we've just tossed all of the network
> driver stuff in there too. I can rebase freely, remove changesets,
> rework them, etc. and this causes a very low amount of pain for Jeff
> Garzik and John Linville.
>
> This may or not be a legitimate thing to do with other heavily
> dependant subsystems.
I was hoping that eventually certain pieces will stop moving and will
become a base that we can ask people to rebase on top of. Perhaps
even asking that stuff destined for 2.6.x+1 go into a -next branch
and the more devel quality stuff go into a -mm branch that is based
off of their -next branch. Gradually stuff would move from -mm
to -next, getting constantly integrated against other peoples' stuff.
More disruptive changes could go in and gradually as people rebase their
trees the conflicts would hopefully get less painful.
But, let's see how things go.
Harvey
On Mon, 11 Feb 2008 22:02:22 -0800 (PST)
David Miller <[email protected]> wrote:
> From: Theodore Tso <[email protected]>
> Date: Tue, 12 Feb 2008 00:11:36 -0500
>
> > __deprecate the old one,
>
> Deprecate is garbage, shit hangs around in the tree forever
> and people just turn off the warnings.
>
> Clean sweeps work much better, albeit with some merge pain,
> we'll cope.
I agree with that.
There is maybe a middle ground in this -next idea;
as very first part of the series, the new api gets added, current users converted
and api marked __deprecated.
Then there's a second part to the patch, which is a separate tree, which gets
added at the very end, which removed the old api.
Both will go in at the same merge window, and the next-meister needs to track that
no new users show up... but the final tree allows this to be done somewhat more gentle.
Doesn't work for API changes that just change the API rather than extending it, and
doesn't solve the dependency issues. So I still think a cleansweep works best in general,
but I suspect Andrew just disagrees with that.
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Mon, 11 Feb 2008 22:15:35 -0800 Andrew Morton <[email protected]> wrote:
>
> Another thing we could do when these things happen is all gang up on Linus
> and ask him to merge the API change into mainline. Because often the
> change can be done in two stages: 1) change the interface then 2) add the code
> in the callee which _uses_ that changed interface. Part 1 is an obviously-safe
> do-nothing change and fixes the merge problems. Part 2 is at a single site
> and can be merged in 2.6.x+1.
I have an alternative (with help from Paul Mackerras).
We have a branch in linux-next that is stable and only progresses forward
i.e. is never rebased. Global API changes/introductions that are of type
1 above go into this branch (after appropriate etc). Fixes to them go in
as well. People can base development on this branch.
But ... they can only really do that *if* we can guarantee that part of
linux-next will be in Linus' tree very early in the next merge window.
So here is the had part ...
We need to ask Linus to promise that he will pull the stable branch from
linux-next first in the merge window. For that to happen, I would expect
that Linus would also review and sign off (or ack) these commits to the
linux-next tree.
The rest of linux-next would always be based on the stable part plus
Linus' current tree.
Will this fly?
Comments?
> A fully running and successful linux-next won't change outcomes a lot, I
> expect. It will help to pick up on obvious bugs and build errors and
> bisection breakages prior to them hitting mainline. But we fix those
> things by -rc1 or -rc2 anyway. So it's just a time-shift, causing no great
> change in the final 2.6.x.
I am hoping that this time shift will allow more time for bug
fixing/polishing. But maybe I am being to optimistic again :-)
> linux-next will do little to solve our (IMO) largest problem: unfixed
> regressions and bugs. otoh it will free up a lot of my time and
> hair-tearing, so I can devote more attention to the bugs. (where
> "attention" usually= "sending irritiating emails")
And may be you can get more sleep and dream up some more schemes :-)
--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/
On Tue, Feb 12, 2008 at 10:57:16PM +1100, Stephen Rothwell wrote:
> On Mon, 11 Feb 2008 22:15:35 -0800 Andrew Morton <[email protected]> wrote:
> >
> > Another thing we could do when these things happen is all gang up on Linus
> > and ask him to merge the API change into mainline. Because often the
> > change can be done in two stages: 1) change the interface then 2) add the code
> > in the callee which _uses_ that changed interface. Part 1 is an obviously-safe
> > do-nothing change and fixes the merge problems. Part 2 is at a single site
> > and can be merged in 2.6.x+1.
>
> I have an alternative (with help from Paul Mackerras).
>
> We have a branch in linux-next that is stable and only progresses forward
> i.e. is never rebased. Global API changes/introductions that are of type
> 1 above go into this branch (after appropriate etc). Fixes to them go in
> as well. People can base development on this branch.
>
> But ... they can only really do that *if* we can guarantee that part of
> linux-next will be in Linus' tree very early in the next merge window.
> So here is the had part ...
>
> We need to ask Linus to promise that he will pull the stable branch from
> linux-next first in the merge window. For that to happen, I would expect
> that Linus would also review and sign off (or ack) these commits to the
> linux-next tree.
>
> The rest of linux-next would always be based on the stable part plus
> Linus' current tree.
>
> Will this fly?
I doubt it. See David's previous post about constantly rebasing his
tree over the lifetime of development during a -rc series. I do the
same thing, dropping things in the middle, re-arranging stuff, handling
merge issues that came into Linus's tree, etc.
I wouldn't want to change our current development flow to force such a
thing on Linus, or us right now.
My main point in all of this was just that you are going to have to help
with some of these merges by hand, adding patches to get things to
build, and not just blindly dropping trees when things go wrong, as you
originally stated. After that, we can continue doing what we have been
doing in the merges with Linus, as that is working very well (see our
rate of change metrics for proof of that.)
thanks,
greg k-h
On Mon, 2008-02-11 at 21:53 -0800, Greg KH wrote:
> > this is why you need specific trees for just the API change, and these
> > need to EXPLICITLY go first before EVERYTHING ELSE. Yes this needs a
> > bit of coordination, but it's the only way.
>
> Even then, it will not work.
>
> Again, Roland isn't going to want to always pull in my driver tree just
> to build his tree. He wants to, and needs to do his own development
> effort.
>
> But when we merge them together, there would be problems.
>
> So, you can't just "drop" the IB tree.
> You can't just "drip" my tree.
>
> Where do you "fix this up" at? I can send a patch for the IB tree, but
> Roland can't put it in his tree, and I can't put it in my tree, it needs
> to go _after_ both of our trees.
Actually, we had exactly this issue with the SCSI bidirectional patches:
They depended on the sg_table patches in block. The solution I adopted
was two merge trees: One to go in immediately with no dependencies
(scsi-misc-2.6) and the other based on the pieces of block (so it would
compile and apply) to go in mid way through the merge round after block
(scsi-bidi-2.6). What I did was keep rebasing the bidi tree until I
could see there was nothing other than a plane base before merging it.
Of course, this only worked because Jens has a git tree ... it would
have been a lot harder (but not impossible) if I'd had entangled patches
from a quilt tree.
So I've already proven that the split tree solution is viable, if not
pretty. The bidi tree had to be rebased an awful lot as the block trees
changed and rebased. Unfortunately, git isn't very good at this, I
eventually had to keep a base and a top reference and just try to cherry
pick this series into the new constructed block tree. But it can be
done...
> That's what -mm has been able to handle so far, and that needs to also
> work with -next.
Actually, we never successfully got block and bidi via -mm.
James
On Feb. 12, 2008, 17:07 +0200, James Bottomley <[email protected]> wrote:
> On Mon, 2008-02-11 at 21:53 -0800, Greg KH wrote:
>>> this is why you need specific trees for just the API change, and these
>>> need to EXPLICITLY go first before EVERYTHING ELSE. Yes this needs a
>>> bit of coordination, but it's the only way.
>> Even then, it will not work.
>>
>> Again, Roland isn't going to want to always pull in my driver tree just
>> to build his tree. He wants to, and needs to do his own development
>> effort.
>>
>> But when we merge them together, there would be problems.
>>
>> So, you can't just "drop" the IB tree.
>> You can't just "drip" my tree.
>>
>> Where do you "fix this up" at? I can send a patch for the IB tree, but
>> Roland can't put it in his tree, and I can't put it in my tree, it needs
>> to go _after_ both of our trees.
>
> Actually, we had exactly this issue with the SCSI bidirectional patches:
> They depended on the sg_table patches in block. The solution I adopted
> was two merge trees: One to go in immediately with no dependencies
> (scsi-misc-2.6) and the other based on the pieces of block (so it would
> compile and apply) to go in mid way through the merge round after block
> (scsi-bidi-2.6). What I did was keep rebasing the bidi tree until I
> could see there was nothing other than a plane base before merging it.
My take on this, in retrospect, is that the code should probably have been
developed in one branch off of one of the trees, or maybe even better in a
third tree.
The point is that the structure of git trees followed the organizational
structure rather than the problem at hand and if contributions coming
from different trees depend on each other, staging branches should be created
in the integration tree for working out the conflicts and when ready,
the integrated branches should be pushed towards the tree's trunk.
>
> Of course, this only worked because Jens has a git tree ... it would
> have been a lot harder (but not impossible) if I'd had entangled patches
> from a quilt tree.
>
> So I've already proven that the split tree solution is viable, if not
> pretty. The bidi tree had to be rebased an awful lot as the block trees
> changed and rebased. Unfortunately, git isn't very good at this, I
> eventually had to keep a base and a top reference and just try to cherry
> pick this series into the new constructed block tree. But it can be
> done...
I developed git-rebase-tree (http://git.bhalevy.com/git/gitweb.cgi?p=git-tools.git;a=blob_plain;f=git-rebase-tree;hb=HEAD)
for exactly this reason - frequently rebasing sub-branches in the linux-pnfs tree
and my experience so far is that it really speeds things up for me and Boaz.
Please feel invited to try it out, I think it's very to close to be mature enough
for general availability (I know, I know, it'd be perfect if this functionality
would be merged into git-rebase :)
>
>> That's what -mm has been able to handle so far, and that needs to also
>> work with -next.
>
> Actually, we never successfully got block and bidi via -mm.
>
> James
>
>
On Tue, 2008-02-12 at 17:32 +0200, Benny Halevy wrote:
> On Feb. 12, 2008, 17:07 +0200, James Bottomley <[email protected]> wrote:
> > On Mon, 2008-02-11 at 21:53 -0800, Greg KH wrote:
> >>> this is why you need specific trees for just the API change, and these
> >>> need to EXPLICITLY go first before EVERYTHING ELSE. Yes this needs a
> >>> bit of coordination, but it's the only way.
> >> Even then, it will not work.
> >>
> >> Again, Roland isn't going to want to always pull in my driver tree just
> >> to build his tree. He wants to, and needs to do his own development
> >> effort.
> >>
> >> But when we merge them together, there would be problems.
> >>
> >> So, you can't just "drop" the IB tree.
> >> You can't just "drip" my tree.
> >>
> >> Where do you "fix this up" at? I can send a patch for the IB tree, but
> >> Roland can't put it in his tree, and I can't put it in my tree, it needs
> >> to go _after_ both of our trees.
> >
> > Actually, we had exactly this issue with the SCSI bidirectional patches:
> > They depended on the sg_table patches in block. The solution I adopted
> > was two merge trees: One to go in immediately with no dependencies
> > (scsi-misc-2.6) and the other based on the pieces of block (so it would
> > compile and apply) to go in mid way through the merge round after block
> > (scsi-bidi-2.6). What I did was keep rebasing the bidi tree until I
> > could see there was nothing other than a plane base before merging it.
>
> My take on this, in retrospect, is that the code should probably have been
> developed in one branch off of one of the trees, or maybe even better in a
> third tree.
For your development, possibly, but not for my merge ... there were a
lot of other patches besides yours in the bidi tree (it was, in fact,
misnamed, but I thought at the time I created it it would only contain
the bidi patches).
> The point is that the structure of git trees followed the organizational
> structure rather than the problem at hand and if contributions coming
> from different trees depend on each other, staging branches should be created
> in the integration tree for working out the conflicts and when ready,
> the integrated branches should be pushed towards the tree's trunk.
Actually, I think you'll find the problem is that we can't build an
integration tree that's both stable and long lived, and hence you can't
base anything off it that requires a long lived base.
So, the way I worked for your patches was to use the -mc tree to
identify my conflicts, and only include the actual conflicting branches
as a basis for the scsi-bidi tree beofre fixing up the patches. I then
actually used the -mc tree as a canary to tell me when to rebase.
James
Greg KH wrote:
> [1] Hopefully the "multiple drivers for a single device" feature people
> have been asking for for years will be landing soon, of course the
> number of odd places in the kernel that made the assumption that we
> could only have one driver per device is causing lots of fun...
Color me a bit skeptical... we've always had this capability really:
for multi-function devices, just register as many subsystems as you
need. If your device is a network _and_ SCSI device, you can do that
with today's APIs.
I just went through a long thread (e1000/e1000e) explaining how much
pain multiple drivers for the same PCI ID cause on the distro side of
things... its a mess. In each case you must write special case code to
resolve the ambiguity. You never know which is the best driver to
choose, or proper module load order -- which may vary depending on
machine, not just static information captured from a kernel Makefile.
Jeff
David Miller wrote:
> I rebase my tree all the time, at least once or twice per
> week. Why?
>
> Firstly, to remove crap. When you have "great idea A" then "oh shit A
> won't work, revert that" there is zero sense in keeping both
> changesets around.
>
> Secondly, I want to fix up the rejects caused by conflicts with
> upstream bug fixes and the like (and there are tons when the tree gets
> to 1500 or so patches like the networking did). I don't want git to
> merge the thing by hand, I want to see what the conflict is and make
> sure the "obvious" resolution is OK and the most efficient way I know
> how to do that is to suck my tree apart as patches, then suck them
> back into a fresh tree.
FWIW, that is annoying and painful for us downstream jobbers, since it
isn't really how git was meant to be used. You use it more like a patch
queue, where commits are very fluid.
Unfortunately, if there is any synchronization lag between me and you --
not uncommon -- then I cannot commit changes on top of the changes just
sent, in my own local tree. Why? Because you rebase so often, I cannot
even locally commit dependent patches due to the end result merge
getting so nasty.
I understand the desire to want a nice and clean history, but the
frequency here really has a negative impact on your downstreams.
It also totally screws the commit statistics, wiping me and John and the
committers we have preserved out, replacing everybody's committer with
David Miller.
Jeff
David Miller wrote:
> This is why, with the networking, we've just tossed all of the network
> driver stuff in there too. I can rebase freely, remove changesets,
> rework them, etc. and this causes a very low amount of pain for Jeff
> Garzik and John Linville.
s/very low/not low/
Rebasing is always a pain, and John and I both agreed the other day that
you do it too often.
I've complained about this before, too... but figured this was just
another thing I was getting ignored on, and so life moved on. But don't
try to sell rebasing as "low pain".
Rebasing makes the history all nice and pretty, but by totalling
flattening the history, trashing all the commit ids (and rewriting
associated metadata), you create obvious downstream problems.
Rebasing is low impact only if you don't have git downstream people.
Otherwise, you're just treating it as a useful quilt clone, really.
Jeff
On Tue, Feb 12, 2008 at 11:24:09AM -0500, Jeff Garzik wrote:
> Greg KH wrote:
>> [1] Hopefully the "multiple drivers for a single device" feature people
>> have been asking for for years will be landing soon, of course the
>> number of odd places in the kernel that made the assumption that we
>> could only have one driver per device is causing lots of fun...
>
>
> Color me a bit skeptical... we've always had this capability really: for
> multi-function devices, just register as many subsystems as you need. If
> your device is a network _and_ SCSI device, you can do that with today's
> APIs.
Yes, but that's not the type of devices this is for.
> I just went through a long thread (e1000/e1000e) explaining how much pain
> multiple drivers for the same PCI ID cause on the distro side of things...
> its a mess. In each case you must write special case code to resolve the
> ambiguity. You never know which is the best driver to choose, or proper
> module load order -- which may vary depending on machine, not just static
> information captured from a kernel Makefile.
Oh, I agree, multiple drivers for the same functionality is not a nice
thing from a distro standpoint.
The work I'm doing here is for stupid PCI firmware engineers, who have
created devices that are different things, all bound up under the same
PCI device. I'm thinking of watchdog timers and random number
generator and i2c controller on the same PCI device, or even the more
basic, frame buffer and DRM access to the same PCI video device.
The OLPC is a good example of hardware that needs this kind of
functionality.
In order to take advantage of the "multiple drivers per device" all
drivers need to guarantee to play nice with others, as they need to be
aware that other drivers will be sharing the hardware.
I'll write up more details later this week, if I get this stuff to work
properly :)
thanks,
greg k-h
On Feb. 12, 2008, 18:36 +0200, Jeff Garzik <[email protected]> wrote:
> David Miller wrote:
>> This is why, with the networking, we've just tossed all of the network
>> driver stuff in there too. I can rebase freely, remove changesets,
>> rework them, etc. and this causes a very low amount of pain for Jeff
>> Garzik and John Linville.
>
>
> s/very low/not low/
>
> Rebasing is always a pain, and John and I both agreed the other day that
> you do it too often.
>
> I've complained about this before, too... but figured this was just
> another thing I was getting ignored on, and so life moved on. But don't
> try to sell rebasing as "low pain".
>
> Rebasing makes the history all nice and pretty, but by totalling
> flattening the history, trashing all the commit ids (and rewriting
> associated metadata), you create obvious downstream problems.
FWIW, when I rebase branches in my tree that others depend on
I keep tags on the old heads for reference. Once the work-in-progress
is done (e.g. tree pulled upstream) the reference tags can be cleaned
up and the tree can be pruned.
Benny
>
> Rebasing is low impact only if you don't have git downstream people.
> Otherwise, you're just treating it as a useful quilt clone, really.
>
> Jeff
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
On Tue, 2008-02-12 at 11:36 -0500, Jeff Garzik wrote:
> David Miller wrote:
> > This is why, with the networking, we've just tossed all of the network
> > driver stuff in there too. I can rebase freely, remove changesets,
> > rework them, etc. and this causes a very low amount of pain for Jeff
> > Garzik and John Linville.
>
>
> s/very low/not low/
>
> Rebasing is always a pain, and John and I both agreed the other day that
> you do it too often.
>
> I've complained about this before, too... but figured this was just
> another thing I was getting ignored on, and so life moved on. But don't
> try to sell rebasing as "low pain".
>
> Rebasing makes the history all nice and pretty, but by totalling
> flattening the history, trashing all the commit ids (and rewriting
> associated metadata), you create obvious downstream problems.
>
> Rebasing is low impact only if you don't have git downstream people.
> Otherwise, you're just treating it as a useful quilt clone, really.
I used to be in this camp too: seeing rebasing as the ultimate evil.
However, what I found was that as I tried to preserve my merge tree's
history, I was picking up merge points with non trivial fixups. You can
argue that this is how git is supposed to operate, but the problem is
that these merge points become immovable objects in the tree history ...
I can't rebase the tree at all any more and I can't simply remove a
commit (which gets messy for commits duplicated by other trees) They
also effectively separate the fix up from the patch causing it. Plus,
it became difficult for people to go to gitweb to see the tree history,
because everything before the merge point got buried.
I finally concluded that I needed to rebase my trees to resolve
conflicts rather than merging them. The advantages are I know the exact
patch with the problem and I can keep the history annotated with what
had to be done to get it to apply. Plus with git (and this is a huge
advantage over quilt) knowing the base of the patch, I can still use the
merge algorithms to help me do the fixup.
I'm not saying rebase constantly ... I'm still mindful of the disruption
it causes, so I only do it if I pick up an actual conflict. However, I
do think it does becomes necessary at times.
The problem rebasing causes is that git is pretty horrible at rebasing a
tree which is based on your now rebased tree (try saying that after a
few martinis). Git really needs to acquire better rebasing tools for
this one case, but if that were solved, I think a lot of the objections
to rebasing could be answered.
James
On Tue, 12 Feb 2008, Jeff Garzik wrote:
> David Miller wrote:
> > This is why, with the networking, we've just tossed all of the network
> > driver stuff in there too. I can rebase freely, remove changesets,
> > rework them, etc. and this causes a very low amount of pain for Jeff
> > Garzik and John Linville.
>
> Rebasing is always a pain, and John and I both agreed the other day that you
> do it too often.
I do think that some people rebase too often (and that David is in that
number), often with no real discernible reason. I think rebasing is great
when you are doing active development (ie you're really acting in "quilt
mode") and I actually think git could and should integrate more of the
queues modes, but I don't think it should be a default action for an
up-stream developer.
I also don't think rebasing helps the particular problem under discussion
(ie conflicts due to having to sort out dependencies between different
trees), and in some ways hurts it.
One thing that I personally react to is that
- I think sysfs and the device model layer has had too much churn, and
I'm unhappy that people seem to expect that to continue.
[ NOTE!! I'm picking on the device model/sysfs stuff here not because
it's the only one, but because it's the obvious and good example. I
do think we have other cases of the same thing. ]
Really. I do agree that we need to fix up bad designs, but I disagree
violently with the notion that this should be seen as some ongoing
thing. The API churn should absolutely *not* be seen as a constant
pain, and if it is (and it clearly is) then I think the people involved
should start off not by asking "how can we synchronize", but looking a
bit deeper and saying "what are we doing wrong?"
It may well be that part of the problem is that the people causing the
churn don't realize the downsides of the pain they are causing, because
THEY aren't the generally ones that see it!
For example, it's easy for Greg to change his driver core, and he can
obviously synchronize with himself in the other trees (because his left
hand is hopefully somewhat aware of what his right hand is doing), so I
suspect Greg simply doesn't see the pain that much. So Greg thinks that
the solution is to just have me merge his changes early, and the pain
is all gone as far as he is concerned.
- That said, I'm also a bit unhappy about the fact you think all merging
has to go through my tree and has to be visible during the two-week
merge period. Quite frankly, I think that you guys could - and should -
just try to sort API changes out more actively against each other, and
if you can't, then that's a problem too.
In other words, please do use the distributed nature of git to your
advantage, when there are things you guys know you need to sort out.
So there are two separate and totally independent issues here.
One is that I suspect some people are a bit too willing to do cleanup for
its own sake, and do not realize that backwards compatibility does
actually help too, and that "better solutions" are sometimes worse than
"keep things stable". We should always *allow* major breakage when
necessary, but I think the threshold for them should be higher than I
think it currently is.
The other is that once somebody says "ok, I *really* need to cause this
breakage, because there's a major bug or we need it for fundamental reason
XYZ", then that person should
(a) create a base tree with _just_ that fundamental infrastructure change,
and make sure that base branch is so obviously good that there is no
question about merging it.
(b) tell other people about the reason for the infrastructure change, and
simply allow others to merge it. You don't have to wait for *me* to
open the merge window, you need to make sure that the people that get
impacted most can continue development!
This is where "rebases really are bad" comes in. When the above sequence
happens, the fundamental infrastructure change obviously does need to be
solid and not shifting under from other people who end up merging it. I do
not want to see five different copies of the fundamental change either
because the original source fixed it up and rebased it, or because the
people who merged it rebased _their_ trees and rebased the fundamental
change in the process.
Can that (b) be my tree? Sure. That's been the common case, and I'll
happily continue it, of course, so I'm not arguing for that to go away.
Merging is my job, I'll do it. But when the merge window is a problem, my
merge window should *not* hold up people from using the distributed nature
of git for their advantage.
But yes, obviously when doing cross-merges, you'd better be really
*really* sure that the base is solid and will get merged. But let's face
it, all the really core maintainers should damn well know that by now:
you've all worked with me for years, so you should be able to trivially be
able to tell whether you *might* need to worry about something, and when
it's a slam dunk.
And it's the "it's a slam dunk" cases that I think are (a) the common ones
and (b) the ones where you can just do cross-merges to satisfy each others
needs.
Hmm? Does that sound palatable to people?
Linus
On Tue, Feb 12, 2008 at 11:31:48AM -0500, Jeff Garzik wrote:
> David Miller wrote:
>> I rebase my tree all the time, at least once or twice per
>> week. Why?
<snip>
> FWIW, that is annoying and painful for us downstream jobbers, since it
> isn't really how git was meant to be used. You use it more like a patch
> queue, where commits are very fluid.
<snip>
> I understand the desire to want a nice and clean history, but the frequency
> here really has a negative impact on your downstreams.
FWIW, I definitely have a (vocal minority) group of contributors
who resent all the rebasing. There may be a few cases of 'vanity'
represented here, but there are definitely reasonable complaints about
not being able to do simple pulls to stay up-to-date and/or having
to rebase before they can send patches to me. FWIW, I think it might
save a bit of my time as well, although I have become pretty good and
"riding the tide" of rebasing... :-(
Unfortunately, I don't have any good suggestions to remedy the issue.
One strategy might be a third layer of trees, e.g.:
net-2.6 fixes for the current release
net-2.6.26 updates certain to go to the next release
net-2.6.26-maybe updates that might not make it to the next release
Of course, managing what goes moves-up from -maybe is probably a big
headache, and just sucks-up more of Dave's time. And, of course,
virtually no one will run the -maybe tree...
Just my $0.02...
John
--
John W. Linville
[email protected]
> The other is that once somebody says "ok, I *really* need to cause this
> breakage, because there's a major bug or we need it for fundamental reason
> XYZ", then that person should
>
> (a) create a base tree with _just_ that fundamental infrastructure change,
> and make sure that base branch is so obviously good that there is no
> question about merging it.
I don't disagree with this, but I think I should point out that making
something "obviously good" may be pretty hard. It's clearly a common
case that the infrastructure change goes through several rounds of
change -- perhaps prompted by exposure in -mm that shows a subtle
issue. So then if all other maintainers based their trees on this
tree, we're left with two not-so-great alternatives:
1) merge the original, broken infrastructure change into your
(Linus's) tree, leaving a known problem for bisecters to trip
over.
2) rebase the world.
I don't know if there's really a perfect answer here. I hope that
tree-wide infrastructure breakage is uncommon enough that we can just
handle these issues "by hand" as they come up.
- R.
On Tue, 2008-02-12 at 09:09 -0800, Linus Torvalds wrote:
> (a) create a base tree with _just_ that fundamental infrastructure change,
> and make sure that base branch is so obviously good that there is no
> question about merging it.
The problem is how do we use a? Usually we need to track your -rc tree
as our fixes go in ... some of which affect our development trees.
If we stick with (a) as the base, we don't get to pull in the fixes in
your tree. If we use your tree we have to pull in (a) creating n
different merge points for the n different upstream trees..
> (b) tell other people about the reason for the infrastructure change, and
> simply allow others to merge it. You don't have to wait for *me* to
> open the merge window, you need to make sure that the people that get
> impacted most can continue development!
Yes, this is effectively what I did with the post merge SCSI tree.
However, if you do this rebasing becomes a fact of life because you need
to rebase out all the dependencies you have before you merge (in fact,
it's a good way of checking whether your dependencies have been merged
yet or not, seeing what survives a rebase).
James
> The work I'm doing here is for stupid PCI firmware engineers, who have
> created devices that are different things, all bound up under the same
> PCI device. I'm thinking of watchdog timers and random number
> generator and i2c controller on the same PCI device, or even the more
> basic, frame buffer and DRM access to the same PCI video device.
>
> The OLPC is a good example of hardware that needs this kind of
> functionality.
Sounds interesting. I've been meaning to work on this too for quite a
while, but I'm glad to see you beat me to it.
An example of an in-tree use case for this would be the mlx4 drivers--
you can look at drivers/net/mlx4/intf.c to see the simple stupid
solution I came up with to allow an IB and a NIC (not yet upstream)
driver to share the same PCI device. A good test for your stuff would
be if it simplifies the code from the ad hoc solution I came up with.
- R.
On Tue, Feb 12, 2008 at 09:09:34AM -0800, Linus Torvalds wrote:
> One thing that I personally react to is that
>
> - I think sysfs and the device model layer has had too much churn, and
> I'm unhappy that people seem to expect that to continue.
>
> [ NOTE!! I'm picking on the device model/sysfs stuff here not because
> it's the only one, but because it's the obvious and good example. I
> do think we have other cases of the same thing. ]
>
> Really. I do agree that we need to fix up bad designs, but I disagree
> violently with the notion that this should be seen as some ongoing
> thing. The API churn should absolutely *not* be seen as a constant
> pain, and if it is (and it clearly is) then I think the people involved
> should start off not by asking "how can we synchronize", but looking a
> bit deeper and saying "what are we doing wrong?"
I may be a bit defensive here, but I hope that all of the recent
kobject/kset/driver core changes have been done with the thought of
"what are we doing wrong".
For the kset rework, we went back and looked at how people were trying
to use this code, realized that it was way too complex, and reworked it
all, making both the implementation simpler, the kernel usage model
simpler, and documented the whole thing so that everyone knows exactly
what is now going on.
To quote the original developer of that code when hearing of the
rewrite, "thank you for doing it, I have no idea what I was thinking
when I wrote that code originally." The end result caused more code to
be removed from the kernel than was added, always a nice thing.
The rework went through many iteratations, reviews, rebases, and touched
many portions of the kernel. In the end, there was only 1 merge issue,
in a new IB driver, and Roland and I handled that after Andrew pointed
out. That kind of dependency was what I was trying to warn the -next
maintainers about.
> It may well be that part of the problem is that the people causing the
> churn don't realize the downsides of the pain they are causing, because
> THEY aren't the generally ones that see it!
Oh, I see it, and so does my inbox :)
> For example, it's easy for Greg to change his driver core, and he can
> obviously synchronize with himself in the other trees (because his left
> hand is hopefully somewhat aware of what his right hand is doing), so I
> suspect Greg simply doesn't see the pain that much. So Greg thinks that
> the solution is to just have me merge his changes early, and the pain
> is all gone as far as he is concerned.
No, I understand the issues here, and am working hard to resolve them.
Now that the kobject underlying layer looks very good, I'm moving a bit
higher up, into the driver core to be able to handle a long-standing
requirement by a lot of hardware vendor and driver authors. And yes,
that is going to cause a few problems in a few places in the
bus-specific logic (the PCI core looks to be the biggest issue, as it
does some nasty things with some internal device lists), but for the
rest of the kernel, and individual driver authors, it will not be an
issue at all.
But I understand your main point here, a lot of time I might come across
as wanting to constantly change this chunk of code, but I'm only doing
it because it's necessary. I'd much rather be off just writing new
drivers, and not having to touch this stuff at all.
If people see changes in which they think I and Kay are unnecessarily
causing churn, please call us out on it, I have no problem with that.
</defensive_posturing> :)
> - That said, I'm also a bit unhappy about the fact you think all merging
> has to go through my tree and has to be visible during the two-week
> merge period. Quite frankly, I think that you guys could - and should -
> just try to sort API changes out more actively against each other, and
> if you can't, then that's a problem too.
>
> In other words, please do use the distributed nature of git to your
> advantage, when there are things you guys know you need to sort out.
A lot of time that already happens today, between the different
subsystem maintainers. We routinely pass PCI and driver core changes
through the network and scsi and ata trees in order to handle merge
issues properly. This already happened in a few places in the
2.6.25-rc1 merge cycle.
But so far we have been doing this on a per-patch level, not really on a
git-tree level. Maybe we might want to re-think this if needed.
> So there are two separate and totally independent issues here.
>
> One is that I suspect some people are a bit too willing to do cleanup for
> its own sake, and do not realize that backwards compatibility does
> actually help too, and that "better solutions" are sometimes worse than
> "keep things stable". We should always *allow* major breakage when
> necessary, but I think the threshold for them should be higher than I
> think it currently is.
>
> The other is that once somebody says "ok, I *really* need to cause this
> breakage, because there's a major bug or we need it for fundamental reason
> XYZ", then that person should
>
> (a) create a base tree with _just_ that fundamental infrastructure change,
> and make sure that base branch is so obviously good that there is no
> question about merging it.
>
> (b) tell other people about the reason for the infrastructure change, and
> simply allow others to merge it. You don't have to wait for *me* to
> open the merge window, you need to make sure that the people that get
> impacted most can continue development!
I'll do this next time I do kobject/driver core changes and see how it
works out.
thanks,
greg k-h
On Tue, Feb 12, 2008 at 09:42:20AM -0800, Roland Dreier wrote:
> > The work I'm doing here is for stupid PCI firmware engineers, who have
> > created devices that are different things, all bound up under the same
> > PCI device. I'm thinking of watchdog timers and random number
> > generator and i2c controller on the same PCI device, or even the more
> > basic, frame buffer and DRM access to the same PCI video device.
> >
> > The OLPC is a good example of hardware that needs this kind of
> > functionality.
>
> Sounds interesting. I've been meaning to work on this too for quite a
> while, but I'm glad to see you beat me to it.
>
> An example of an in-tree use case for this would be the mlx4 drivers--
> you can look at drivers/net/mlx4/intf.c to see the simple stupid
> solution I came up with to allow an IB and a NIC (not yet upstream)
> driver to share the same PCI device. A good test for your stuff would
> be if it simplifies the code from the ad hoc solution I came up with.
Great, I'll look into that as I need a test case for this kind of stuff.
thanks,
greg k-h
On Feb. 12, 2008, 19:41 +0200, James Bottomley <[email protected]> wrote:
> On Tue, 2008-02-12 at 09:09 -0800, Linus Torvalds wrote:
>> (a) create a base tree with _just_ that fundamental infrastructure change,
>> and make sure that base branch is so obviously good that there is no
>> question about merging it.
>
> The problem is how do we use a? Usually we need to track your -rc tree
> as our fixes go in ... some of which affect our development trees.
>
> If we stick with (a) as the base, we don't get to pull in the fixes in
> your tree. If we use your tree we have to pull in (a) creating n
> different merge points for the n different upstream trees..
IMHO, this base tree should typically be based off of linus' tree
and kept rebased on top of it. This way you get the mainline fixes
through the integration base tree.
Benny
Stephen Rothwell wrote:
> Andrew was looking for someone to run a linux-next tree that just
> contained the subsystem git and quilt trees for 2.6.x+1 and I (in a
> moment of madness) volunteered. So, this is to announce the creating of
> such a tree (it doesn't exist yet) which will require some (hopefully)
> small amount of work on the part of subsystem maintainers.
As noted to Andrew in private email, I think this would be great.
akpm's -mm tree tends to have "x+1" changes in it, and also stuff that's
not fully baked yet. I would definitely like to see a tree composed of
all the sub-trees where maintainers said "yes, I think this is baked and
ready for x+1". Then -mm and not-yet-baked git changes can layer on top
of that.
As others are noting, API changes require some coordination amongst
ourselves (should be our job, not yours, really) and historically we
haven't been so good at that -- in part because IMO there hasn't been
any system thing better than a heads-up email in place.
Jeff
On Tue, 12 Feb 2008, James Bottomley wrote:
>
> On Tue, 2008-02-12 at 09:09 -0800, Linus Torvalds wrote:
> > (a) create a base tree with _just_ that fundamental infrastructure change,
> > and make sure that base branch is so obviously good that there is no
> > question about merging it.
>
> The problem is how do we use a? Usually we need to track your -rc tree
> as our fixes go in ... some of which affect our development trees.
So?
> If we stick with (a) as the base, we don't get to pull in the fixes in
> your tree. If we use your tree we have to pull in (a) creating n
> different merge points for the n different upstream trees..
I don't understand what you mean. This is true whether you pulled (a) or
not. If you have any changes what-so-ever in your tree, if you pull in
fixes from my tree, you'll get a merge.
But if you mean that you cannot rebase (a), then yes. That was what I
said. Rebases *do*not*work* (and fundamentally cannot work) in a
distributed environment.
But why would you merge with my tree in the first place? My tree won't
normally have any conflicts or anything like that anyway.
With a "Linux-next" tree, you'll see the conflicts if they occur (since
*that* tree would merge!), and in that case you would say "now I need to
merge Linus' tree just to resolve the conflicts!"
But before that, merging my tree (or rebasing on top of it) is simply
*wrong*. It has nothing to do with your SCSI development.
> Yes, this is effectively what I did with the post merge SCSI tree.
> However, if you do this rebasing becomes a fact of life because you need
> to rebase out all the dependencies you have before you merge (in fact,
> it's a good way of checking whether your dependencies have been merged
> yet or not, seeing what survives a rebase).
I don't see the logic. You shouldn't need to rebase at all. I don't see
why you claim that this makes rebasing more of a fact. It doesn't. It has
no impact at all, except making rebasing _less_ possible!
Linus
Roland Dreier wrote:
> > The work I'm doing here is for stupid PCI firmware engineers, who have
> > created devices that are different things, all bound up under the same
> > PCI device. I'm thinking of watchdog timers and random number
> > generator and i2c controller on the same PCI device, or even the more
> > basic, frame buffer and DRM access to the same PCI video device.
> >
> > The OLPC is a good example of hardware that needs this kind of
> > functionality.
>
> Sounds interesting. I've been meaning to work on this too for quite a
> while, but I'm glad to see you beat me to it.
>
> An example of an in-tree use case for this would be the mlx4 drivers--
> you can look at drivers/net/mlx4/intf.c to see the simple stupid
> solution I came up with to allow an IB and a NIC (not yet upstream)
> driver to share the same PCI device. A good test for your stuff would
> be if it simplifies the code from the ad hoc solution I came up with.
Our APIs are written such that the PCI device probe function is the
starting point for registering IB and NIC. No need for anything new.
Jeff
John W. Linville wrote:
> FWIW, I definitely have a (vocal minority) group of contributors
> who resent all the rebasing. There may be a few cases of 'vanity'
> represented here, but there are definitely reasonable complaints about
> not being able to do simple pulls to stay up-to-date and/or having
> to rebase before they can send patches to me. FWIW, I think it might
> save a bit of my time as well, although I have become pretty good and
> "riding the tide" of rebasing... :-(
Definitely... that is the sort of stuff that upstream folks never see
when they rebase -- it causes an explosion of downstream rebasing, where
each rebase must wait for the one-step-up upstream to rebase, before
patches can be sent. It's easy for the rebasing to get exponential
quickly, the more downstream levels you have.
Jeff
On Tue, Feb 12, 2008 at 12:56:35PM -0500, Jeff Garzik wrote:
> Greg KH wrote:
>> The work I'm doing here is for stupid PCI firmware engineers, who have
>> created devices that are different things, all bound up under the same
>> PCI device. I'm thinking of watchdog timers and random number
>> generator and i2c controller on the same PCI device, or even the more
>> basic, frame buffer and DRM access to the same PCI video device.
>
> Yes, that has a known solution: have your driver register i2c, rng,
> watchdog, etc. functions.
>
> Works just fine inside today's infrastructure, no changes needed.
Except that the individual drivers are a lot of the time written by
different people, live in different portions of the tree, and are
combined into different combinations depending on the chipset.
For i2c devices, I see the scx200_acb, i2c-elektor, i2c-sis5595, and
i2c-sis630 drivers needing this. The last one happens to share the pci
device with a video driver, that doesn't always need to be / want to be
loaded by users just so they can read the temperature of their
processors.
Oh, the EDAC code also needs this, and I know that no one wants to merge
that stuff into their individual drivers :)
Same goes for framebuffer vs. DRM. Yeah, I know DRM ended up "winning"
here, but for some hardware setups, both are still valid, and it really
would be good for both drivers to be notified when the system is about
to go into suspend/resume and other stuff like that.
If the driver core can provide this in a simple manner, I think it is
worth it, especially as there are lots of hardware that seems to need
it.
thanks,
greg k-h
On Tue, 2008-02-12 at 10:00 -0800, Linus Torvalds wrote:
>
> On Tue, 12 Feb 2008, James Bottomley wrote:
> >
> > On Tue, 2008-02-12 at 09:09 -0800, Linus Torvalds wrote:
> > > (a) create a base tree with _just_ that fundamental infrastructure change,
> > > and make sure that base branch is so obviously good that there is no
> > > question about merging it.
> >
> > The problem is how do we use a? Usually we need to track your -rc tree
> > as our fixes go in ... some of which affect our development trees.
>
> So?
>
> > If we stick with (a) as the base, we don't get to pull in the fixes in
> > your tree. If we use your tree we have to pull in (a) creating n
> > different merge points for the n different upstream trees..
>
> I don't understand what you mean. This is true whether you pulled (a) or
> not. If you have any changes what-so-ever in your tree, if you pull in
> fixes from my tree, you'll get a merge.
>
> But if you mean that you cannot rebase (a), then yes. That was what I
> said.
Yes, that's what I meant ... no rebasing of (a) once it becomes the
fixed point we all work from.
> Rebases *do*not*work* (and fundamentally cannot work) in a
> distributed environment.
Hm ... I think net is a counter example to this. Rebases certainly work
for them. The issue, I thought, was around the policy of rebasing and
how often.
I see the question as being one of who creates the history. When
something goes into your tree, that's an obvious history point and we
can't rewrite it. However, when something goes into my trees, I don't
think it's as obviously a history point that has to be preserved, so I
can rebase obviously, rebasing causes disruption, so I don't do it
unless it's necessitated by an actual conflict.
> But why would you merge with my tree in the first place? My tree won't
> normally have any conflicts or anything like that anyway.
This normally happens when we have bug fixes to a driver that is
simultaneously being worked on for the merge window ... and it actually
happens quite a lot. Sometimes the conflicts actually don't come via my
tree (usually a -mm docbook update, include file removal or something
else that looked trivial to the submitter).
The worst examples of this are the "run lindent on the driver" type of
patches, but a lot of other changes give trivial conflict too.
> With a "Linux-next" tree, you'll see the conflicts if they occur (since
> *that* tree would merge!), and in that case you would say "now I need to
> merge Linus' tree just to resolve the conflicts!"
Right ... that's exactly why I created the -mc tree ... it warns me that
there's a looming problem I need to fix (and not just in your tree).
For that reason, a -next tree that does exactly the same thing will be
great ... I won't have to maintain the -mc tree.
> But before that, merging my tree (or rebasing on top of it) is simply
> *wrong*. It has nothing to do with your SCSI development.
Yes ... I don't do that ... Like I said, I only rebase for an actual
conflict.
> > Yes, this is effectively what I did with the post merge SCSI tree.
> > However, if you do this rebasing becomes a fact of life because you need
> > to rebase out all the dependencies you have before you merge (in fact,
> > it's a good way of checking whether your dependencies have been merged
> > yet or not, seeing what survives a rebase).
>
> I don't see the logic. You shouldn't need to rebase at all. I don't see
> why you claim that this makes rebasing more of a fact. It doesn't. It has
> no impact at all, except making rebasing _less_ possible!
Well, it came at me because Jens was rebasing the block tree as he
worked through issues in the two branches I was based on. If the block
tree hadn't rebased, I wouldn't have rebased that often. However, we
did come across other things that had to move from my tree to Jens' as
we developed this, mainly bug fixes moving closer to the source to
preserve bisectability, and in that case rebases were really necessary.
James
On Tue, 12 Feb 2008, Greg KH wrote:
>
> I may be a bit defensive here, but I hope that all of the recent
> kobject/kset/driver core changes have been done with the thought of
> "what are we doing wrong".
.. but are we expecting it to be finished?
That's the point.
This whole "Linux-next" discussion so far has almost been predicated on
the whole assumption that this is an on-going concern. And it really
should NOT be.
If it's an on-going concern, we need to tackle *that* issue, not the issue
that cross-subsystem merges are hard. They simply seem to happen too much.
In other words, I'm not AT ALL interested in the merges we've already
done. That's over and done with, and we'll never ever do those merges
again. Who cares? I don't.
I'm purely and _only_ interested in the merges of the future. You don't
need to be defensive about the things that led up to this discussion, I'm
more hoping that we can aim at fixing the problem at the source, rather
than trying to work around it.
We simply shouldn't have all that many conflicts. We've had *way* too many
of them lately, and I think it's because people have felt it wasn't too
painful.
Put another way: back when we worked with just patches, we avoided renames
like hell, and we also tried to simply even re-architect the whole tree so
that you didn't have so many patch conflicts. One main reason as far as I
was concerned for things like per-directory Kconfig files and the whole
initcall() stuff was the fact that the old single Kconfig file and the old
crazy init/main.c file were total *nightmares* when it came to conflict
resolution.
So we split things up more, and we didn't do renames (or were very careful
about it). We avoided the things that caused pain.
I think we need to remember that: yes, we'll always have to have ways to
fix the pain that does happen, but even more importantly, we should strive
for models where it doesn't happen in the first place!
And simply avoiding cross-subsystem API changes unless there is a major
*MAJOR* reason for them is the obvious thing to do. Simply face the fact
that even in open source there are major reasons to stay with an old
interface even if it's not optimal.
We absolutely MUST NOT have the mindset that "cross-subsystem conflicts
happen all the time".
That was my point.
Linus
On Tue, 12 Feb 2008, Jeff Garzik wrote:
> Greg KH wrote:
> > The work I'm doing here is for stupid PCI firmware engineers, who have
> > created devices that are different things, all bound up under the same
> > PCI device. I'm thinking of watchdog timers and random number
> > generator and i2c controller on the same PCI device, or even the more
> > basic, frame buffer and DRM access to the same PCI video device.
>
> Yes, that has a known solution: have your driver register i2c, rng, watchdog,
> etc. functions.
>
> Works just fine inside today's infrastructure, no changes needed.
Indeed. If you have a multi-function device that shows up as a single PCI
function, just make it have its own "private bus", and make it show up as
a "devices within a device".
Create the fake PCI subdevices that have no "real" counterpart, except as
parts of the stupid device that couldn't be bothered to be seen as
multiple _real_ functions.
That not only solves the infrastructure issues, it's actually The Truth
with capital letters. It is, after all, how the device actually works
internally.
Linus
Greg KH wrote:
> Except that the individual drivers are a lot of the time written by
> different people, live in different portions of the tree, and are
> combined into different combinations depending on the chipset.
Yes -- the worst case is that people have to work together, and it
tweaks people who like to organize source files nicely into directories :)
> For i2c devices, I see the scx200_acb, i2c-elektor, i2c-sis5595, and
> i2c-sis630 drivers needing this. The last one happens to share the pci
> device with a video driver, that doesn't always need to be / want to be
> loaded by users just so they can read the temperature of their
> processors.
Sure. Reasonable request, and doable within today's APIs.
> Oh, the EDAC code also needs this, and I know that no one wants to merge
> that stuff into their individual drivers :)
So people have to work together... darn :) most drivers these days are
organized nicely into nicely modular units anyway, making it easy to
write a "shell" driver that simply registers each sub-driver, and helps
arbitrate/provide resources to sub-units.
Consider, for example, a PCI driver that loads, and then fills in
platform_data to provide a specific set of resources to a platform
driver (a common idiom). You would need to create parented struct
devices (parent: pci_dev's device), but everything else should work within
I could even forsee a future where most drivers are written in a generic
platform-driver style, and PCI|sbus|embedded-bus|blah are simply
bus-specific shells that fill in platform data.
All of this should be nicely possible within the existing usage of
struct device, platform drivers, generic DMA API, iomap, etc.
Jeff
Greg KH wrote:
> The work I'm doing here is for stupid PCI firmware engineers, who have
> created devices that are different things, all bound up under the same
> PCI device. I'm thinking of watchdog timers and random number
> generator and i2c controller on the same PCI device, or even the more
> basic, frame buffer and DRM access to the same PCI video device.
Yes, that has a known solution: have your driver register i2c, rng,
watchdog, etc. functions.
Works just fine inside today's infrastructure, no changes needed.
Jeff
On Tue, 12 Feb 2008, Benny Halevy wrote:
>
> IMHO, this base tree should typically be based off of linus' tree
> and kept rebased on top of it. This way you get the mainline fixes
> through the integration base tree.
Hell no!
No rebasing! If people rebase, then it's useless as a base.
That base tree needs to be something people can *depend* on. It contains
the API changes, and not anything else. Otherwise I will never ever pull
the resulting mess, and you all end up with tons of extra work.
Just say *no* to rebasing.
Rebasing is fine for maintaining *your* own patch-set, ie it is an
alternative to using quilt. But it is absolutely not acceptable for
*anythign* else.
In particular, people who rebase other peoples trees should just be shot
(*). It's simply not acceptable behaviour. It screws up the sign-off
procedure, it screws up the people whose code was merged, and it's just
WRONG.
Linus
(*) The exception being if there is something seriously wrong with the
tree. I think I've had trees which I just refused to pull, and while most
of the time I just say "I refuse to pull", early on in git development I
actually ended up fixing some of those trees up because my refusal was due
to people mis-using git in the first place. So I have actually effectively
rebased a maintainer tree at least once. But I still think it is seriously
screwed up.
Greg KH wrote:
> A lot of time that already happens today, between the different
> subsystem maintainers. We routinely pass PCI and driver core changes
> through the network and scsi and ata trees in order to handle merge
> issues properly. This already happened in a few places in the
> 2.6.25-rc1 merge cycle.
Yep. I do that with the powerpc tree sometimes, too: I'll ACK a
drivers/net change that is obviously PPC-specific, and that drivers/net
change will go in via paulus.
Like you said, it's done on a per-patch not git tree basis at the moment.
That sort of cooperation has become common: in areas of shared
responsibility, request an ack, and then the "most obvious" person
(generally the one with most merge dependencies) merges that
shared-responsibility change along with some other changes specific to
their own tree.
Jeff
On Tue, 12 Feb 2008, James Bottomley wrote:
>
> Hm ... I think net is a counter example to this. Rebases certainly work
> for them.
They consider themselves to be "one tree" and are thus largely a totally
different issue than the one discussed here.
Also, I actually flamed David a lot last round over his rebasing of
netfilter etc. It was no acceptably done. That network tree was all
screwed up, with the git committer information not matching the signed-off
path etc.
If you do cross-tree rebasing, you need to consider it 100% equivalent to
just passing patches around in emails. Because it really is.
> Yes ... I don't do that ... Like I said, I only rebase for an actual
> conflict.
And this is how things should work.
> Well, it came at me because Jens was rebasing the block tree as he
> worked through issues in the two branches I was based on.
Yes, and I am in no way saying that the core driver model has been the
only problem spot.
And also, I do not like "hard rules". Every rule always has an exception,
and sometimes a rebase-based strategy can be the right thing even across
trees.
But you're all ignoring my fundamental objection: you're talking as if
cross-tree fundamental API changes should be the "norm", and that we
should try to solve the workflow issues that stem from that. And I'm
saying that I think we should try to FIX the issue, and make sure that
it simply *isn't* the norm.
In other words, I'm perfectly happy to be an a*hole and tell people that I
simply won't merge things that cause undue API churn at all, and that were
not thought out sufficiently.
We've had too many issues like that (SG chaining, iommu, driver core, not
to mention the upheavals in x86) lately, but realistically, which
subsystem remains a problem for the future? And maybe the correct thing to
do is to just say "enough!".
I'm perfectly happy being hardnosed and saying "nope, that's crap, it
doesn't matter if the code is slightly better if you cause those kinds of
issues".
The thing is, sometimes the answer really *is* "Don't do that then!". If
our model becomes bogged up by cross-subsystem serialization issues, that
means that we (a) spend too much time handling the fall-out from that and
(b) too little time just refining the details.
And quite frankly, while "good core architecture" matters a lot, in the
end, "getting all the boring small things right" matters even more! Big
architectural churn that cleans up and fixes the "big issues" is totally
anti-productive if it means that we don't look at the boring small detail
stuff.
Linus
On Tue, 12 Feb 2008, Linus Torvalds wrote:
>
> In other words, I'm perfectly happy to be an a*hole and tell people that I
> simply won't merge things that cause undue API churn at all, and that were
> not thought out sufficiently.
.. btw: I'd need to know this in advance. I usually don't see the problem
until it's too late.
And this is very much an area where "Linux-next" can help: if some
subsystem causes problems in Linux-next for other maintainers, I really
think it shouldn't just be a matter of "drop the git tree that didn't
merge cleanly", but it should literally be a question of "maybe we should
drop the _earlier_ git tree that caused the later one not to merge
cleanly".
In other words, maybe things like core block layer changes or device model
changes should be *last* in the merge-list (or if first, also be first to
be dropped if they cause merge errors downstream!).
That way, infrastructure changes that screw up others can only happen if
the maintainer actively works with the others to make sure it works even
before it would ever merge into Linux-next successfully.
That may sound odd, but it actually matches what I personally believe in:
we have more driver code and other "outlying" things than we have core
things, and most of our problems come from that - so we should prioritize
*those* things, not the "fundmantal core changes".
So how about making that the default situation: drivers and other outliers
merge first. If fundamental API changes happen, they merge last, and if
their maintainers can't make it in time in the merge window, they just get
dropped.
That sure as hell would put the pain on API changes solidly where it
belongs.
Linus
On Tue, 2008-02-12 at 10:48 -0800, Linus Torvalds wrote:
> But you're all ignoring my fundamental objection: you're talking as if
> cross-tree fundamental API changes should be the "norm", and that we
> should try to solve the workflow issues that stem from that. And I'm
> saying that I think we should try to FIX the issue, and make sure that
> it simply *isn't* the norm.
OK, I'll address it then (or try to). Just don't eat me please ... I'm
only a small and timid storage maintainer ...
> In other words, I'm perfectly happy to be an a*hole and tell people that I
> simply won't merge things that cause undue API churn at all, and that were
> not thought out sufficiently.
>
> We've had too many issues like that (SG chaining, iommu, driver core, not
> to mention the upheavals in x86) lately, but realistically, which
> subsystem remains a problem for the future? And maybe the correct thing to
> do is to just say "enough!".
I can sort of agree, but I don't think you can say enough == no more API
changes. There's the request_irq argument removal patch floating around
somewhere for instance ...but as someone said "Every rule always has an
exception".
I think what's really needed is a balance where it's painful (on the
originator) to make an API change. Part of what -mc or -next trees can
do is to make that pain visible early (and often).
> I'm perfectly happy being hardnosed and saying "nope, that's crap, it
> doesn't matter if the code is slightly better if you cause those kinds of
> issues".
OK, so you can supply the pain as well.
> The thing is, sometimes the answer really *is* "Don't do that then!". If
> our model becomes bogged up by cross-subsystem serialization issues, that
> means that we (a) spend too much time handling the fall-out from that and
> (b) too little time just refining the details.
I think this is exaggerated. By the end of the -rc cycle for 2.6.24 I
had six merge fixup patches because of trivial cross subsystem API
changes and two dropped trees because of intractable problems (and they
were notified and later fixed themselves up). Given that I was
including about 50 git trees and 4 quilt ones, that's not too bad,
Especially as we had all the API churn you mentioned.
> And quite frankly, while "good core architecture" matters a lot, in the
> end, "getting all the boring small things right" matters even more! Big
> architectural churn that cleans up and fixes the "big issues" is totally
> anti-productive if it means that we don't look at the boring small detail
> stuff.
I agree with this too. When I introduce an API, I usually try to make
sure it will last the test of time ... the trouble is, I'm not always
omniscient.
So the point, I think we've reached is that:
1. we want to try to enforce good design for APIs to try to ensure
they're as future proof as foreseeable.
2. We have to recognise that 1. is impossible in practice and
provide a mechanism for correcting APIs
So, I think our current system is good in that it enforces pain on
people who change APIs. However, perhaps we are missing the bit where
we reflect longer before introducing APIs in the first place. (hey,
better reviewing ... I've heard that one before ...)
James
On Tue, Feb 12, 2008 at 10:26:53AM -0800, Linus Torvalds wrote:
> On Tue, 12 Feb 2008, Greg KH wrote:
> >
> > I may be a bit defensive here, but I hope that all of the recent
> > kobject/kset/driver core changes have been done with the thought of
> > "what are we doing wrong".
>
> .. but are we expecting it to be finished?
Nope, never.
> That's the point.
Not it isn't. To quote you a number of years ago:
"Linux is evolution, not intelligent design"
We need to constantly be able to make these kinds of changes in order to
continue to remain relevant.
> This whole "Linux-next" discussion so far has almost been predicated on
> the whole assumption that this is an on-going concern. And it really
> should NOT be.
>
> If it's an on-going concern, we need to tackle *that* issue, not the issue
> that cross-subsystem merges are hard. They simply seem to happen too much.
>
> In other words, I'm not AT ALL interested in the merges we've already
> done. That's over and done with, and we'll never ever do those merges
> again. Who cares? I don't.
>
> I'm purely and _only_ interested in the merges of the future. You don't
> need to be defensive about the things that led up to this discussion, I'm
> more hoping that we can aim at fixing the problem at the source, rather
> than trying to work around it.
>
> We simply shouldn't have all that many conflicts. We've had *way* too many
> of them lately, and I think it's because people have felt it wasn't too
> painful.
Oh, it's been painful at times, but they are, overall, very rare.
If you look at the rate of change we are currently running at, it's
amazing that we do not get _more_ of these kinds of problems.
Remember, we are currently clocking along at the steady rate of:
4000 lines added every day
1900 lines removed every day
1300 lines modified every day
And the rate of change in each major portion of the kernel (drivers,
arch, core, network, etc) is exactly proportional to the amount of the
kernel that that portion takes up (I have detailed numbers if people
really want to see them.)
Because of this rate of change, we are surviving, and evolving into what
is exactly needed at this point in time from a kernel. To try to quench
this change, and force us to stick with things that are no longer
optimal, is going to cause us to make the same mistakes that AIX and
Solaris and Vista have. To stop this, is to die :)
> Put another way: back when we worked with just patches, we avoided renames
> like hell, and we also tried to simply even re-architect the whole tree so
> that you didn't have so many patch conflicts. One main reason as far as I
> was concerned for things like per-directory Kconfig files and the whole
> initcall() stuff was the fact that the old single Kconfig file and the old
> crazy init/main.c file were total *nightmares* when it came to conflict
> resolution.
>
> So we split things up more, and we didn't do renames (or were very careful
> about it). We avoided the things that caused pain.
>
> I think we need to remember that: yes, we'll always have to have ways to
> fix the pain that does happen, but even more importantly, we should strive
> for models where it doesn't happen in the first place!
We have done very good at that. The problem comes in when one subsystem
depends on another one.
Like the kobject core, or driver core. Or network core. Changes there
trickle out into the other portions of the kernel that depend on them.
And we fix them up, and move on. And so far, we're doing this very
well, at a rate faster than any other software project ever has.
> And simply avoiding cross-subsystem API changes unless there is a major
> *MAJOR* reason for them is the obvious thing to do. Simply face the fact
> that even in open source there are major reasons to stay with an old
> interface even if it's not optimal.
I strongly disagree here. We lived with that kset/ktype crap for years,
and I finally broke down and cleaned it up, simplifying things, removing
code, making the kernel smaller, leaner, and easier for others to change
and use in the future. With your statement, such a change should have
never taken place as it what we had at the time was "not optimal", but
good enough to live with.
> We absolutely MUST NOT have the mindset that "cross-subsystem conflicts
> happen all the time".
They usually don't, by virtue of our current development model and how
we have the kernel structured.
But they do happen about once or twice a kernel release, just by virtue
of the way things need to happen. And since the last kernel had 10353
different commits, only 2 conflicts is a pretty good rate of merge
issues :)
And when they happen, so far _everyone_ involved instantly rushes to fix
them up. I say the fact that we are able to detect these problems, fix
them, and continue to move on and work, is a testament to the fact that
our current model is working _very_ well.
I think that you aren't seeing all of the "whacks" that Andrew is
constantly giving us that break his -mm tree with problems at times, so
when it gets to you, things are cleaned up.
The goal of -next is to take that "whacking" chore off of Andrew's
shoulders, and onto others, so Andrew can move off doing something like
development better.
My point is that the -next tree needs to work like -mm does today, a
policy of "merge broke, drop the tree" isn't going to help us subsystem
maintainers out in the end, we need a way to shim it up for a while as
merges and other communication happens, just like -mm provides, so that
when we get to the 2 week merge window, things are properly worked out
and what goes to you is all clean and purty.
thanks,
greg k-h
On Tue, Feb 12, 2008 at 10:59:00AM -0800, Linus Torvalds wrote:
>
>
> On Tue, 12 Feb 2008, Linus Torvalds wrote:
> >
> > In other words, I'm perfectly happy to be an a*hole and tell people that I
> > simply won't merge things that cause undue API churn at all, and that were
> > not thought out sufficiently.
>
> .. btw: I'd need to know this in advance. I usually don't see the problem
> until it's too late.
>
> And this is very much an area where "Linux-next" can help: if some
> subsystem causes problems in Linux-next for other maintainers, I really
> think it shouldn't just be a matter of "drop the git tree that didn't
> merge cleanly", but it should literally be a question of "maybe we should
> drop the _earlier_ git tree that caused the later one not to merge
> cleanly".
We usually get this warning today in -mm.
> In other words, maybe things like core block layer changes or device model
> changes should be *last* in the merge-list (or if first, also be first to
> be dropped if they cause merge errors downstream!).
>
> That way, infrastructure changes that screw up others can only happen if
> the maintainer actively works with the others to make sure it works even
> before it would ever merge into Linux-next successfully.
>
> That may sound odd, but it actually matches what I personally believe in:
> we have more driver code and other "outlying" things than we have core
> things, and most of our problems come from that - so we should prioritize
> *those* things, not the "fundmantal core changes".
>
> So how about making that the default situation: drivers and other outliers
> merge first. If fundamental API changes happen, they merge last, and if
> their maintainers can't make it in time in the merge window, they just get
> dropped.
>
> That sure as hell would put the pain on API changes solidly where it
> belongs.
Sure, I have no objection to that at all.
thanks,
greg k-h
On Feb. 12, 2008, 20:36 +0200, Linus Torvalds <[email protected]> wrote:
>
> On Tue, 12 Feb 2008, Benny Halevy wrote:
>> IMHO, this base tree should typically be based off of linus' tree
>> and kept rebased on top of it. This way you get the mainline fixes
>> through the integration base tree.
>
> Hell no!
>
> No rebasing! If people rebase, then it's useless as a base.
Linus, what you're saying is pretty extreme.
Our experience with pnfs development taught us that rebase was actually the only
method that worked on the long run because it allows us to maintain a *clean* series of
patchsets over an extended period of time (order of two years).
In the past, when we merged your tree into ours rather than using git-remote update and
rebase we ended up with a cluttered mess that buried all the resolved merge conflicts
in merge commits and maintaining it became an increasingly harder nightmare.
Nowadays, I rebase the linux-pnfs git tree almost on a daily basis, keeping tags
on historical refs (to be cleaned up when a branch stabilizes) and the other
contributors simply use git-remote update to refresh their tracking branches and
rebase their stuff onto the new heads. It took the team a couple weeks to get used
to this methodology but it really works great for us now.
It seems to me that merging works well one (your :) way when patches are merged
upstream and become immutable at this point, while on the other way, keeping
a patchset fresh and comprehensible is feasible only with rebase when there
are occasional conflicts with stuff coming down from upstream. The domino-
effect caused by rebase is manageable and not worse than merge if people
just keep track of the original bases of their development branch before
updating their tracking branches.
Maybe the missing link is a tool to help do rebasing more easily.
I wrote my own tool for that: git-rebase-tree (available on
git://git.bhalevy.com/git-tools.git) and using it makes the mechanics
of rebasing our patchsets almost trivial. I really encourage folks
to try it out and give me feedback about it.
Benny
>
> That base tree needs to be something people can *depend* on. It contains
> the API changes, and not anything else. Otherwise I will never ever pull
> the resulting mess, and you all end up with tons of extra work.
>
> Just say *no* to rebasing.
>
> Rebasing is fine for maintaining *your* own patch-set, ie it is an
> alternative to using quilt. But it is absolutely not acceptable for
> *anythign* else.
So what's the solution for maintaining a series of patchsets that depend
on one another?
>
> In particular, people who rebase other peoples trees should just be shot
> (*). It's simply not acceptable behaviour. It screws up the sign-off
> procedure, it screws up the people whose code was merged, and it's just
> WRONG.
>
> Linus
>
> (*) The exception being if there is something seriously wrong with the
> tree. I think I've had trees which I just refused to pull, and while most
> of the time I just say "I refuse to pull", early on in git development I
> actually ended up fixing some of those trees up because my refusal was due
> to people mis-using git in the first place. So I have actually effectively
> rebased a maintainer tree at least once. But I still think it is seriously
> screwed up.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
On Tue, Feb 12, 2008 at 11:31:48AM -0500, Jeff Garzik wrote:
> David Miller wrote:
>> I rebase my tree all the time, at least once or twice per
>> week. Why?
>>
>> Firstly, to remove crap. When you have "great idea A" then "oh shit A
>> won't work, revert that" there is zero sense in keeping both
>> changesets around.
>>
>> Secondly, I want to fix up the rejects caused by conflicts with
>> upstream bug fixes and the like (and there are tons when the tree gets
>> to 1500 or so patches like the networking did). I don't want git to
>> merge the thing by hand, I want to see what the conflict is and make
>> sure the "obvious" resolution is OK and the most efficient way I know
>> how to do that is to suck my tree apart as patches, then suck them
>> back into a fresh tree.
>
> FWIW, that is annoying and painful for us downstream jobbers, since it
> isn't really how git was meant to be used. You use it more like a patch
> queue, where commits are very fluid.
>
> Unfortunately, if there is any synchronization lag between me and you --
> not uncommon -- then I cannot commit changes on top of the changes just
> sent, in my own local tree. Why? Because you rebase so often, I cannot
> even locally commit dependent patches due to the end result merge
> getting so nasty.
>
> I understand the desire to want a nice and clean history, but the
> frequency here really has a negative impact on your downstreams.
>
> It also totally screws the commit statistics, wiping me and John and the
> committers we have preserved out, replacing everybody's committer with
> David Miller.
But the "author" is still preserved, right? Why do you need the
committer name to be preserved? (I'm not denying that there could be
reasons, I'm just curious what they are.)
--b.
On Tue, Feb 12, 2008 at 10:59:00AM -0800, Linus Torvalds wrote:
>
>
> On Tue, 12 Feb 2008, Linus Torvalds wrote:
> >
> > In other words, I'm perfectly happy to be an a*hole and tell people that I
> > simply won't merge things that cause undue API churn at all, and that were
> > not thought out sufficiently.
>
> .. btw: I'd need to know this in advance. I usually don't see the problem
> until it's too late.
We could simply decide that API changes affecting more than one subsystem
Must Be Serialized(tm). Explicitly. As in "any such change is posted
and discussed in advance, order of merges decided upon and we have merge
window for one decided set of API changes + fallout *ONLY*". With merge
window in question normally taking a few days.
On Tue, Feb 12, 2008 at 11:15:53AM -0800, Greg KH wrote:
> Remember, we are currently clocking along at the steady rate of:
> 4000 lines added every day
> 1900 lines removed every day
> 1300 lines modified every day
???? lines reviewed every day.
AFAICS, we are in situation when review bandwidth is where the bottleneck
is. Not the merge one...
On Tue, 12 Feb 2008, Greg KH wrote:
>
> > That's the point.
>
> Not it isn't. To quote you a number of years ago:
> "Linux is evolution, not intelligent design"
Umm. Have you read a lot of books on evolution?
It doesn't sound like you have.
The fact is, evolution often does odd (and "suboptimal") things exactly
because it does incremental changes that DO NOT BREAK at any point.
The examples are legion. The mammalian eye has the retina "backwards",
with the blind spot appearing because the fundmanetal infrastructure (the
optical nerves) actually being in *front* of the light sensor and needing
a hole in the retina to get the information (and blood flow) to go to the
brain!
In other words, exactly *because* evolution requires "bisectability" (any
non-viable point in between is a dead end by definition) and does things
incrementally, it doesn't do big flips. It fixes the problems on an
incremental scale both when it comes to the details and when it comes to
both "details" (actual protein-coding genes that code directly for some
expression) and "infrastructure" (homeobox and non-coding genes).
So quite frankly, you're the "intelligent designer" here. You're the one
who seems to claim that we need those leaps of faith and wild jumps.
> Oh, it's been painful at times, but they are, overall, very rare.
No, overall, they have *not* been rare lately. We've had them all over.
And not just the one introduced by you.
> If you look at the rate of change we are currently running at, it's
> amazing that we do not get _more_ of these kinds of problems.
I don't think that's a valid argument.
Sure, we have lots of changes, but 99.9% of them have no cross-subsystem
effect what-so-ever.
> > And simply avoiding cross-subsystem API changes unless there is a major
> > *MAJOR* reason for them is the obvious thing to do. Simply face the fact
> > that even in open source there are major reasons to stay with an old
> > interface even if it's not optimal.
>
> I strongly disagree here. We lived with that kset/ktype crap for years,
> and I finally broke down and cleaned it up, simplifying things, removing
> code, making the kernel smaller, leaner, and easier for others to change
> and use in the future. With your statement, such a change should have
> never taken place as it what we had at the time was "not optimal", but
> good enough to live with.
You didn't listen at all.
I said that the threshold should be high, not that it should be
impossible. I also said that we should strive for making it unnecessary to
have the painful total synchronization points.
The fact is, we *have* been able to do things like this gradually and
well, without introducing breakage. Take the VM changes, for example:
those were pretty damn fundamental, where we've changed the calling
convention totally for fault handling.
But that thing was done without at any point really seriously breaking
code. It involved adding the new interface, and letting the old one live
in parallel.
The last remnant of the old "nopage()" interface still exists, but I think
right now it's only used by DRM.
Did it require the drivers to be updated? Yes. But it did NOT require the
total synchronization, because it still worked with the old interface.
> But they do happen about once or twice a kernel release, just by virtue
> of the way things need to happen.
And I violently disagree.
It should not be "once of twice a kernel release".
It should be "once or twice a year" that you hit a flag-day issue. The
rest of the time you should be able to do it without breakage. It's
doable. You just HAVEN'T EVEN TRIED, and seem to be actively against even
doing so.
Linus
On Tue, 12 Feb 2008, J. Bruce Fields wrote:
>
> But the "author" is still preserved, right? Why do you need the
> committer name to be preserved? (I'm not denying that there could be
> reasons, I'm just curious what they are.)
It's not that the committer should be preserved, but:
- the chain from author -> committer should be visible in the
Signed-off-by: lines.
If you rebase somebody elses tree, you screw that up. You need to add
your sign-off, since now *you* are the new committer, and *you* took
somebody elses work!
- you should respect the down-stream developer, and if that downstream
developer continues to work with his branch or works with other people,
you shouldn't screw that up!
Both of those basically say that you should never rebase somebody elses
work. You can use rebase to rebase your *own* work on top of somebody
elses thing (since that doesn't change the sign-off chain, and you still
respect the downstream developers development model)!
But of course, if you rebase, you should respect the wishes of the
up-stream developer too. I don't do rebases. So if you asked me to pull,
the stuff I pulled can never be rebased, because it just *is* in my tree.
Put another way: think of the absolute *chaos* that would happen if I were
to rebase instead of just merging. Every time I pull from you I'd
invalidate your whole tree, and you'd have to re-generate. It gets
unmaintainable very quickly.
And that's actually ignoring a real issue: stability of commits. The nice
thing about stable commit naming is that all bug-reports from other people
that told where the bug happened are basically 100% trust-worthy and the
code is 100% reproducible not just for you, but for everybody else.
In other words, you really shouldn't rebase stuff that has been exposed
anywhere outside of your own private tree. But *within* your own private
tree, and within the commits that have never seen the light of day,
rebasing is fine.
(And yes, there are exceptions. If it's a clear "throw-away tree" all the
rules go out the window, of course, as long as everybody involved *knows*
it's a throw-away tree, and know that if they pull it they have to
synchronise 100% with you - so within a very tight-knit case or within a
very specific small detail that is actively being worked on, those rebases
with cleanups make tons of sense).
Linus
On Tue, Feb 12, 2008 at 09:09:34AM -0800, Linus Torvalds wrote:
> I also don't think rebasing helps the particular problem under discussion
> (ie conflicts due to having to sort out dependencies between different
> trees), and in some ways hurts it.
How do you suggest that we fix the problems we had with the initial
merge with the ARM tree.
We were in the situation where, had I asked you to pull my tree, you'd
have had a fair number of conflicts to resolve - I tested this by trying
to merge your tree into my 'devel' head.
The result wasn't pleasant - even I didn't know how to fix up a lot of
those conflicts, so the only way I could move forward was to reconstruct
the 'devel' head (which is just merges of other branches) omitting the
problem branches.
This then posed something of a problem - of which I saw three solutions:
1. ask the original authors of changes in those problem branches to come
up with patches or fixes in their git trees (in the rare case that
they have git trees) to bring the branch into line with the conflicting
changes in mainline. The resulting branch with that change applied
probably won't build, which means bisect pain for people who happen
to bisect to that point.
2. botch the merge, publish the tree, and then hit heads together to try
and get the problem resolved - again resulting in a commit point (the
merge) which is nonsense and unbuildable.
3. rebase the branch on top of the conflicting change, throw out the
patches which prove to be a problem and ask the original author of
those patches to fix them up for the conflicting change. The result
is a completely bisectable tree.
(3) is the solution which I chose, and it worked _extremely_ well.
(3) is effectively what akpm does with his tree - when a patch conflicts
with other changes, he throws the changes out and bangs peoples heads
together to get a new set of patches generated which work together. In
that respect, it's no different, and it's been proven there to work well.
So I see git as being no different.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of:
On Tue, 12 Feb 2008, James Bottomley wrote:
> > Rebases *do*not*work* (and fundamentally cannot work) in a
> > distributed environment.
> Hm ... I think net is a counter example to this. Rebases certainly work
> for them. The issue, I thought, was around the policy of rebasing and
> how often.
Hmm ... as far as I can see, Jeff and John (i.e. net tree downstreams) are
pretty loudly unhappy with Dave rebasing too often, right?
--
Jiri Kosina
On Tue, 12 Feb 2008 20:03:54 +0000
Russell King <[email protected]> wrote:
> (3) is effectively what akpm does with his tree - when a patch conflicts
> with other changes, he throws the changes out and bangs peoples heads
> together to get a new set of patches generated which work together.
Actually I'll usually (much >90%) just queue another patch which fixes the
fallout and will drop that once the upstreams sort things out.
I'll only drop a subsystem tree in extremis. One reason for this is
that I'll have a pile of other patches which are dependent upon that
subsystem tree and which would need reworking. And un-reworking once the
subsytem tree is undropped.
And the trees which are most likely to cause conflicts tend to also be
the trees upon which other patches are dependent. Mainly because those
trees are larger and more spread out.
On Tue, 12 Feb 2008, Russell King wrote:
>
> 3. rebase the branch on top of the conflicting change, throw out the
> patches which prove to be a problem and ask the original author of
> those patches to fix them up for the conflicting change. The result
> is a completely bisectable tree.
>
> (3) is the solution which I chose, and it worked _extremely_ well.
Don't get me wrong at all. Rebasing is fine for stuff you have committed
yourself (which I assume was the case here).
Rebasing is also a fine conflict resolution strategy when you try to
basically turn a "big and complex one-time merge conflict" into "multiple
much smaller ones by doing them one commit at a time".
But what rebasing is _not_ is a fine "default strategy", especially if
other people are depending on you.
> (3) is effectively what akpm does with his tree - when a patch conflicts
> with other changes, he throws the changes out and bangs peoples heads
> together to get a new set of patches generated which work together.
Right. And it's one of the fundamental differences between git and a patch
queue.
Patch queues are very flexible, but they simply don't scale. They don't
scale in history (ie you cannot sanely keep track of multiple queues as
they grow in the long run - you need a way to regularly "freeze" things
into a release tar-ball or something like that). But they also don't scale
with number of users - even just read-only ones.
The latter example is something we see in -mm right now. When people
report problems against -mm, it's usually an all-or-nothing thing
(so-and-so mm release doesn't work), but even when people sometimes bisect
to a specific point in mm, it's not "stable" in the sense that that point
may not make any sense in a subsequent -mm queue version.
And all of these issues are not about "-mm" per se, but are about patch
queues in general. And "rebase" turns a git repository effectively to a
patch queue too, with all the same downsides.
And I don't think patch queues are evil: I use git rebase all the time
myself (not for the kernel, but for git, where I'm not the top-level
thing). But I'd not rebase commits from other peoples trees: I rebase
commits that I applied as patches (ie the *author* may be somebody else,
but the committer is me!) or that I committed myself, and that I haven't
pushed out.
Note that difference between "committer" and "author". There's nothing
wrong with rebasing commits that are _authored_ by other people. But there
absolutely is something wrong rebasing commits that are _commmitted_ by
somebody else, unless you two are best buddies and know and trust each
other and are ok with each other messing up the commits (and use some
external method of serialization, so that you don't step on each others
toes).
Linus
On Tue, Feb 12, 2008 at 11:55:45AM -0800, Linus Torvalds wrote:
>
>
> On Tue, 12 Feb 2008, Greg KH wrote:
> >
> > > That's the point.
> >
> > Not it isn't. To quote you a number of years ago:
> > "Linux is evolution, not intelligent design"
>
> Umm. Have you read a lot of books on evolution?
>
> It doesn't sound like you have.
>
> The fact is, evolution often does odd (and "suboptimal") things exactly
> because it does incremental changes that DO NOT BREAK at any point.
Hm, I think we are in violent agreement here :)
> The examples are legion. The mammalian eye has the retina "backwards",
> with the blind spot appearing because the fundmanetal infrastructure (the
> optical nerves) actually being in *front* of the light sensor and needing
> a hole in the retina to get the information (and blood flow) to go to the
> brain!
>
> In other words, exactly *because* evolution requires "bisectability" (any
> non-viable point in between is a dead end by definition) and does things
> incrementally, it doesn't do big flips. It fixes the problems on an
> incremental scale both when it comes to the details and when it comes to
> both "details" (actual protein-coding genes that code directly for some
> expression) and "infrastructure" (homeobox and non-coding genes).
>
> So quite frankly, you're the "intelligent designer" here. You're the one
> who seems to claim that we need those leaps of faith and wild jumps.
No, I might have mistyped previously, but I really do not believe in
this at all.
> > > And simply avoiding cross-subsystem API changes unless there is a major
> > > *MAJOR* reason for them is the obvious thing to do. Simply face the fact
> > > that even in open source there are major reasons to stay with an old
> > > interface even if it's not optimal.
> >
> > I strongly disagree here. We lived with that kset/ktype crap for years,
> > and I finally broke down and cleaned it up, simplifying things, removing
> > code, making the kernel smaller, leaner, and easier for others to change
> > and use in the future. With your statement, such a change should have
> > never taken place as it what we had at the time was "not optimal", but
> > good enough to live with.
>
> You didn't listen at all.
>
> I said that the threshold should be high, not that it should be
> impossible. I also said that we should strive for making it unnecessary to
> have the painful total synchronization points.
I agree with this. But I was reacting to your "there are major reasons
to stay with an old interface..." portion above, while I feel we should
always work to fixe those interfaces as best as possible.
> > But they do happen about once or twice a kernel release, just by virtue
> > of the way things need to happen.
>
> And I violently disagree.
>
> It should not be "once of twice a kernel release".
>
> It should be "once or twice a year" that you hit a flag-day issue. The
> rest of the time you should be able to do it without breakage. It's
> doable. You just HAVEN'T EVEN TRIED, and seem to be actively against even
> doing so.
No, not at all.
I have tried, and successfully done this many times in the past. The
kobject change was one example: add a new function, migrate all users of
a direct pointer over to that function, after that work is all done and
in, change the structure and do the needed work afterward. All is
bisectable completly, with no big "flag day" needed.
Same goes for a lot of USB changes. Hell, we've been slowly moving
toward changing the USB urb callback for _years_ yet, not quite getting
there, but the groundwork is done, making it much simpler and easy for
it to happen. Eventually things will flip over and the eventual patch
for all callback functions will be trivial and easily reviewed and
merged.
In short, I agree that flag days are bad and should be avoided at all
costs, and that slow incremental, bisectable changes are how we evolve
properly. And that we need to always be able to do this kind of work,
and never be stuck with the "can't change an old, internal api" model.
To bring it around to the original topic, -next should help us in
finding these issues out, exactly like -mm is.
If we want to make a new rule of "only merging api or cross-subsystem
changes after everything else", that's wonderful with me, I have no
objection to it at all.
thanks,
greg k-h
On Tue, Feb 12, 2008 at 07:46:51PM +0000, Al Viro wrote:
> On Tue, Feb 12, 2008 at 11:15:53AM -0800, Greg KH wrote:
>
> > Remember, we are currently clocking along at the steady rate of:
> > 4000 lines added every day
> > 1900 lines removed every day
> > 1300 lines modified every day
>
> ???? lines reviewed every day.
I can run the numbers, but almost every one of those changes has at
least 2 signed-off-by: on them, so they should all be being reviewed
properly.
> AFAICS, we are in situation when review bandwidth is where the bottleneck
> is. Not the merge one...
Are there still large numbers of posted patches, not reviewed or picked
up by anyone laying around somewhere? I thought Andrew-the-patch-vacuum
had been doing a great job of keeping that from happening lately.
thanks,
greg k-h
On Tue, 2008-02-12 at 21:18 +0100, Jiri Kosina wrote:
> On Tue, 12 Feb 2008, James Bottomley wrote:
>
> > > Rebases *do*not*work* (and fundamentally cannot work) in a
> > > distributed environment.
> > Hm ... I think net is a counter example to this. Rebases certainly work
> > for them. The issue, I thought, was around the policy of rebasing and
> > how often.
>
> Hmm ... as far as I can see, Jeff and John (i.e. net tree downstreams) are
> pretty loudly unhappy with Dave rebasing too often, right?
That's true ... but irrelevant to the argument of whether rebasing does
or doesn't work. I don't think anyone's arguing that rebasing doesn't
cause real problems to downstream users ... when I was based on block
for the scsi post merge tree, I was just such a user ...
James
On Tue, Feb 12, 2008 at 12:50:51PM -0800, Greg KH wrote:
> I can run the numbers, but almost every one of those changes has at
> least 2 signed-off-by: on them, so they should all be being reviewed
> properly.
>
> > AFAICS, we are in situation when review bandwidth is where the bottleneck
> > is. Not the merge one...
>
> Are there still large numbers of posted patches, not reviewed or picked
> up by anyone laying around somewhere? I thought Andrew-the-patch-vacuum
> had been doing a great job of keeping that from happening lately.
Er... Surely you can think of examples of patches that went through
-mm without serious review? Not to mention that 2 S-o-B normally
means only one thing: "went to Linus through some tree". Which does
not guarantee any real review, as you damn well know.
On Tue, Feb 12, 2008 at 09:08:08PM +0000, Al Viro wrote:
> On Tue, Feb 12, 2008 at 12:50:51PM -0800, Greg KH wrote:
>
> > I can run the numbers, but almost every one of those changes has at
> > least 2 signed-off-by: on them, so they should all be being reviewed
> > properly.
> >
> > > AFAICS, we are in situation when review bandwidth is where the bottleneck
> > > is. Not the merge one...
> >
> > Are there still large numbers of posted patches, not reviewed or picked
> > up by anyone laying around somewhere? I thought Andrew-the-patch-vacuum
> > had been doing a great job of keeping that from happening lately.
>
> Er... Surely you can think of examples of patches that went through
> -mm without serious review? Not to mention that 2 S-o-B normally
> means only one thing: "went to Linus through some tree". Which does
> not guarantee any real review, as you damn well know.
Yes, I agree, there are lots of examples of this, but the overall
majority are reviewed by 2 people at least (or sure as hell should be,
maybe we need to bring into existance the "reviewed-by" marking to
ensure this.)
Do you know of a way to help this get better?
thanks,
greg k-h
On Tue, Feb 12, 2008 at 12:48:13PM -0800, Greg KH wrote:
> I have tried, and successfully done this many times in the past. The
> kobject change was one example: add a new function, migrate all users of
> a direct pointer over to that function, after that work is all done and
> in, change the structure and do the needed work afterward. All is
> bisectable completly, with no big "flag day" needed.
That takes care of bisectability, but doesn't take care of the problem
of new stuff being merged which used the old API and is now broken.
Maybe the last step (removal of old API) needs to be taken a lot later
than it currently is (maybe a second merge window between -rc1 and -rc2
for removal of interfaces that became unused during the first merge
window before -rc1?)
--
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."
On Tue, 12 Feb 2008, Greg KH wrote:
>
> Yes, I agree, there are lots of examples of this, but the overall
> majority are reviewed by 2 people at least (or sure as hell should be,
> maybe we need to bring into existance the "reviewed-by" marking to
> ensure this.)
Well, I don't really "review" any patches that come through Andrew. What I
do is:
- global search-and-replace Andrew's "acked-by:" with one that is both
him and me (that way I make sure that I _only_ sign off on patches that
he has signed off on!)
- look through all the commit *messages* (but not patches). This
sometimes involves also editing up grammar etc - some of those messages
just make me wince - but it also tends to include things like adding
commit one-liner information if only a git commit ID is mentioned etc.
- and only for areas that I feel competent in, I look at the patches too.
So, to take an example, when Andrew passes on uml patches that only touch
arch/um and include/asm-um, my sign-off does not mean *any* kind of review
at all. It's purely a sign that it's passed the sign-off requirements
properly.
When it comes to VM issues or other things, things are different, and I
actually review the patch (and occasionally send it back with "nope, I'm
not applying this"). But for stuff that comes through Andrew, that's
probably less than a quarter of the patches. And I don't mark the ones
I've reviewed specially in any way.
And I suspect I'm not at all alone in this. People simply have maintainers
they trust (and _need_ to trust in order to not become a bottleneck).
Linus
On Tue, Feb 12, 2008 at 10:30:04AM -0800, Linus Torvalds wrote:
>
>
> On Tue, 12 Feb 2008, Jeff Garzik wrote:
>
> > Greg KH wrote:
> > > The work I'm doing here is for stupid PCI firmware engineers, who have
> > > created devices that are different things, all bound up under the same
> > > PCI device. I'm thinking of watchdog timers and random number
> > > generator and i2c controller on the same PCI device, or even the more
> > > basic, frame buffer and DRM access to the same PCI video device.
> >
> > Yes, that has a known solution: have your driver register i2c, rng, watchdog,
> > etc. functions.
> >
> > Works just fine inside today's infrastructure, no changes needed.
>
> Indeed. If you have a multi-function device that shows up as a single PCI
> function, just make it have its own "private bus", and make it show up as
> a "devices within a device".
That's how I first tried to do this (old pci-piggy patches in -mm were a
result of this...).
But, we have the issue of the foolish-in-retrospect sysdev devices.
They are an example of "multiple drivers per device" type thing. To
clean them up properly, they need to move to the general 'struct device'
and if that is going to already support the one-to-many model, well,
then any PCI implementation of such an internal bus can also use the
same code.
This is a lot of handwaving right now, let me go work on some code and
see how it turns out.
Don't worry, I'll expect a lot of review cycles :)
thanks,
greg k-h
> We could simply decide that API changes affecting more than one subsystem
> Must Be Serialized(tm). Explicitly. As in "any such change is posted
Welcome to dreamland. The only way I can get serial changes done is to
wait months and eventually simply persuade Andrew to ignore the
"maintainers" who are basically not responding.
Maybe it works for vfs and mm internals but out here in driverspace
forget it - and if I simply break drivers people get cross instead of
fixing them nowdays.
On Tue, 12 Feb 2008 21:51:52 +0000
Alan Cox <[email protected]> wrote:
> > We could simply decide that API changes affecting more than one subsystem
> > Must Be Serialized(tm). Explicitly. As in "any such change is posted
>
> Welcome to dreamland. The only way I can get serial changes done is to
> wait months and eventually simply persuade Andrew to ignore the
> "maintainers" who are basically not responding.
"maintainers"?
I think the best way to get the serial drivers maintained would be to cat
them all onto the end of synclink.c and hope that Paul thinks he did it.
> I think the best way to get the serial drivers maintained would be to cat
> them all onto the end of synclink.c and hope that Paul thinks he did it.
Well I've already broken the buffering so he'd fix it ;)
We have a pile of old ISA drivers that are going to break soon with the
locking changes and a pile of USB drivers that when I looked at the
locking were so terminally broken I couldn't be bothered to fix them.
So it'll be fun, lots of && BROKENs to add
On Feb 11 2008 20:21, Greg KH wrote:
>>
>> I hope to recreate this tree every day automatically. In order to do
>> this, any tree that has a conflict will be dropped from that days tree.
>
>Oh oh oh, I get merged first! me me me!
No, you can't have a tree like that. [森林 Not yours. 森林]
Let's maximize the profit, by keeping the largest number of trees which
do not cause a conflict amongst each other.
On Tue, Feb 12, 2008 at 10:20:44PM +0000, Alan Cox wrote:
> > I think the best way to get the serial drivers maintained would be to cat
> > them all onto the end of synclink.c and hope that Paul thinks he did it.
>
> Well I've already broken the buffering so he'd fix it ;)
>
> We have a pile of old ISA drivers that are going to break soon with the
> locking changes and a pile of USB drivers that when I looked at the
> locking were so terminally broken I couldn't be bothered to fix them.
>
> So it'll be fun, lots of && BROKENs to add
Hrm... How badly is pl2303 broken? I actually use that sucker, so if
it needs help - count me in.
On Mon, Feb 11, 2008 at 11:06:17PM -0800, Arjan van de Ven wrote:
> There is maybe a middle ground in this -next idea; as very first
> part of the series, the new api gets added, current users converted
> and api marked __deprecated.
>
> Then there's a second part to the patch, which is a separate tree,
> which gets added at the very end, which removed the old api.
>
> Both will go in at the same merge window, and the next-meister needs
> to track that no new users show up... but the final tree allows this
> to be done somewhat more gentle.
>
> Doesn't work for API changes that just change the API rather than
> extending it, and doesn't solve the dependency issues. So I still
> think a cleansweep works best in general, but I suspect Andrew just
> disagrees with that.
Yes, that's exactly what I was suggesting. The __deprecate only lasts
for the merge window, and we remove the old API at the end of the
merge window. So it's only there for a very short time, and it's only
there to make the cleen sweep a little less painful --- not one where
"shit hangs around in the tree forever".
- Ted
On Tue, Feb 12, 2008 at 10:20:44PM +0000, Alan Cox wrote:
> > I think the best way to get the serial drivers maintained would be to cat
> > them all onto the end of synclink.c and hope that Paul thinks he did it.
>
> Well I've already broken the buffering so he'd fix it ;)
>
> We have a pile of old ISA drivers that are going to break soon with the
> locking changes and a pile of USB drivers that when I looked at the
> locking were so terminally broken I couldn't be bothered to fix them.
Let me know which USB ones are broken, I'll work to fix them.
thanks,
greg k-h
On Tue, 12 Feb 2008 14:55:31 -0800
Greg KH <[email protected]> wrote:
> On Tue, Feb 12, 2008 at 10:20:44PM +0000, Alan Cox wrote:
> > > I think the best way to get the serial drivers maintained would be to cat
> > > them all onto the end of synclink.c and hope that Paul thinks he did it.
> >
> > Well I've already broken the buffering so he'd fix it ;)
> >
> > We have a pile of old ISA drivers that are going to break soon with the
> > locking changes and a pile of USB drivers that when I looked at the
> > locking were so terminally broken I couldn't be bothered to fix them.
>
> Let me know which USB ones are broken, I'll work to fix them.
That I noticed doing an audit for unlocking the mctrl functions:
ir-usb: global variables without locking used in per port operations
iuu_phoenix: no locking on internal data structures
mos7840: ditto
option: ditto
kobil_sct: ditto
These drivers do interesting things (where interesting is probably not too
evil on a PC - except ir-usb) involving playing with data structures
without locks. It seems there was some kind of evolution along the way as
some drivers do have a carefully used port private data structure lock
(or two) but many do not.
> Hrm... How badly is pl2303 broken? I actually use that sucker, so if
> it needs help - count me in.
2303 is pretty good (in fact by usb serial standards outstanding). It has
all the internal locking needed for now and right down to killing
lock_kernel entirely outside of open/close (which is going to hit
everything).
Only fixme I have tagged for it is
- if you set an unsupported baud rate it reports it set rather than the
one you got
Which is a trivial mend for someone who has suitable docs (its marked
FIXME: at the right spot)
On Tue, Feb 12, 2008 at 10:59:00PM +0000, Alan Cox wrote:
> On Tue, 12 Feb 2008 14:55:31 -0800
> Greg KH <[email protected]> wrote:
>
> > On Tue, Feb 12, 2008 at 10:20:44PM +0000, Alan Cox wrote:
> > > > I think the best way to get the serial drivers maintained would be to cat
> > > > them all onto the end of synclink.c and hope that Paul thinks he did it.
> > >
> > > Well I've already broken the buffering so he'd fix it ;)
> > >
> > > We have a pile of old ISA drivers that are going to break soon with the
> > > locking changes and a pile of USB drivers that when I looked at the
> > > locking were so terminally broken I couldn't be bothered to fix them.
> >
> > Let me know which USB ones are broken, I'll work to fix them.
>
> That I noticed doing an audit for unlocking the mctrl functions:
>
> ir-usb: global variables without locking used in per port operations
> iuu_phoenix: no locking on internal data structures
> mos7840: ditto
> option: ditto
> kobil_sct: ditto
>
> These drivers do interesting things (where interesting is probably not too
> evil on a PC - except ir-usb) involving playing with data structures
> without locks. It seems there was some kind of evolution along the way as
> some drivers do have a carefully used port private data structure lock
> (or two) but many do not.
Ok, I'll look into them this week. I'm currently working on revamping
the option driver as it had the mistake of being based on the keyspan
drivers, which weren't the nicest to start with...
thanks,
greg k-h
On Tue, Feb 12, 2008 at 11:01:32PM +0000, Alan Cox wrote:
> > Hrm... How badly is pl2303 broken? I actually use that sucker, so if
> > it needs help - count me in.
>
> 2303 is pretty good (in fact by usb serial standards outstanding). It has
> all the internal locking needed for now and right down to killing
> lock_kernel entirely outside of open/close (which is going to hit
> everything).
>
> Only fixme I have tagged for it is
>
> - if you set an unsupported baud rate it reports it set rather than the
> one you got
>
> Which is a trivial mend for someone who has suitable docs (its marked
> FIXME: at the right spot)
I'll see what I can do about fixing this.
thanks,
greg k-h
On Tue, Feb 12, 2008 at 12:48:13PM -0800, Greg KH wrote:
> On Tue, Feb 12, 2008 at 11:55:45AM -0800, Linus Torvalds wrote:
> > >
> > > Not it isn't. To quote you a number of years ago:
> > > "Linux is evolution, not intelligent design"
I think this statement has been used unfortunately as a hard and fast
rule (which we all know how much Linus loves :-) to mean, in its most
extreme form, that we should *never* try to do some careful reflection
about careful API design, and that the extremes of "no interface
without an in-tree user" applies to (a) parameters in a function call
(heck, we can always sweep through all the in-tree users to add that
extra parameter later, and thats a *good* thing because it breaks
those evil out-of-tree drivers) and (b) to not even thinking if some
particular interface (that is not needed now but which reasonably will
be needed later) is even *possible* without doing a sweep of all of
the in-tree users of the interface.
Related to this syndrome is the assumption that measuring the rate of
changes in lines of code changed per second implies that any
development process which causes the number of lines of code changed
second, including frequent sweeps through the tree changing all
interfaces, is a *good* thing.
Yes, this is an extreme position, and I'm not accusing anyone of
holding the above in its entirety --- but I've seen aspects of all of
these from one developer or another.
We come to it from the attacking another strawman, which assumes that
*all* interfaces which don't have an in-tree are evil, and that
keeping old __deprecated interfaces for a long time is an evil which
causes intolerable pain, and that it's never worthwhile to try to
anticipate future expandibility into an interface because you will
inevitably get it wrong.
Clearly, we are right to mock Solaris for making claims that they will
never, ever, *ever* change an interface, not even one that goes back
sixteen years to Solaris 2.3. But it doesn't follow the opposite
point of view, that constant mutability of kernel interfaces to make
sure that things are always perfect and pure and clean is the right
one either.
> > The examples are legion. The mammalian eye has the retina "backwards",
> > with the blind spot appearing because the fundmanetal infrastructure (the
> > optical nerves) actually being in *front* of the light sensor and needing
> > a hole in the retina to get the information (and blood flow) to go to the
> > brain!
Also, evolution also means that things like vestigal organs (like our
appendix) are tolerated. So are things like clearly very badly
designed things, like human backs. To the extent that we don't like
vestigal old __deprecated interfaces, and want things to be perfect,
we are actually straying into the realms where we want the sort of
things that you would get if you *did* have an intelligent designer
designing the human body from scratch.
So the "Linux is evolution, not intelligent design" quote is
unfortunately getting used to imply that no amount of intelligent
foresight is worthwhile, and I think that's unfortunate. It implies
an extreme position which is not warranted.
> > > But they do happen about once or twice a kernel release, just by virtue
> > > of the way things need to happen.
> >
> > And I violently disagree.
> >
> > It should not be "once of twice a kernel release".
> >
> > It should be "once or twice a year" that you hit a flag-day issue. The
> > rest of the time you should be able to do it without breakage. It's
> > doable. You just HAVEN'T EVEN TRIED, and seem to be actively against even
> > doing so.
>
> No, not at all.
>
> I have tried, and successfully done this many times in the past. The
> kobject change was one example: add a new function, migrate all users of
> a direct pointer over to that function, after that work is all done and
> in, change the structure and do the needed work afterward. All is
> bisectable completly, with no big "flag day" needed.
Collectively, we need to try harder.
We can debate exactly where the right line is, in terms of whether
it's only "once or twice a kernel release", or "once or twice a year",
but clearly the current amount of interface changes and cross-tree
dependencies has been causing Andrew pain. And to me, that means we
need to turn the knob back a quarter turn towards tolerating
__deprecated old interfaces a little bit more, and trying to get
interfaces right just a little bit more and try building in just a
little bit more future expandability, and to try just *little* bit
harder to preserve a *little* bit more stable API.
In other words, maybe we need to write a counterpoint to the
stable_api_nonsense.txt and call it unstable_api_nonsense.txt --- and
in it, we note that if we start burning out Andrew and he starts
getting really, REALLY grumpy --- and if especially we start making
Stephen (normally a very mild-mannered and not terribly excitable guy)
grumpy, that it's time that we try just a little bit harder to make
our API's a little bit more stable. Suckers^H^H^H^H^H^H^H^H, err.,
dedicated release managers like like Andrew and Stephen are very
precious resources and we shouldn't be burning them out by assuming
that "stable_api_nonsense.txt" is a license to constantly churn our
internal API's, to the point where they start complaining.
- Ted
From: Jeff Garzik <[email protected]>
Date: Tue, 12 Feb 2008 11:31:48 -0500
> I understand the desire to want a nice and clean history, but the
> frequency here really has a negative impact on your downstreams.
Ok, fair enough. Any alternative suggestions on how to remove turds
without them being in the history?
> It also totally screws the commit statistics, wiping me and John and the
> committers we have preserved out, replacing everybody's committer with
> David Miller.
I am well aware of this downside, sorry.
On Tue, Feb 12, 2008 at 03:51:07PM -0800, David Miller wrote:
> From: Jeff Garzik <[email protected]>
> Date: Tue, 12 Feb 2008 11:31:48 -0500
>
> > I understand the desire to want a nice and clean history, but the
> > frequency here really has a negative impact on your downstreams.
>
> Ok, fair enough. Any alternative suggestions on how to remove turds
> without them being in the history?
>
> > It also totally screws the commit statistics, wiping me and John and the
> > committers we have preserved out, replacing everybody's committer with
> > David Miller.
>
> I am well aware of this downside, sorry.
Ahem... Use of git-cherry-pick preserves commit information just fine.
From: Jeff Garzik <[email protected]>
Date: Tue, 12 Feb 2008 11:36:24 -0500
> Rebasing is low impact only if you don't have git downstream people.
> Otherwise, you're just treating it as a useful quilt clone, really.
Understood.
One of the key operations that I'm interested in is removing things
from the history. If I could do that using GIT without any side
effects and in a way that really would remove it from the tree, I
would do that in a heartbeat.
At 1500 changesets, a merge conflict shows up about once
every day or two as 2.6.N nears it's release into final
as bug fixes trickle in.
I find using GIT to fixup merge errors on a tree of that
scale to be really painful. And it only fixes up the final
result in a merge changeset.
Just like we expect submitters to refresh their patches
as things change upstream, we as developers should be able
to refresh trees of changes we are maintaining for similar
effect.
This is all made difficult because GIT's commit IDs are tied
to where they are in the tree, and thus are dependant upon
the wholeness of parents and subsequently their parents.
That's why it's hard to "just remove" something completely
without rebasing the tree.
Let me give you a good example, just yesterday I had to rebase
my net-2.6 tree a few times. It's because I put a patch in there
that the person said was OK but it broke the build. There is
zero reason for me to push that tree to Linus with the bad
commit and the revert, it's just noise and makes the tree harder
for people to review.
Now, to preserver your tree's history I simply reapplied the
stuff I wanted then repulled your tree.
But right now I have to redo the include/linux/rcupdate.h that's in
there because it has a bug. I now want to rebase again to just kick
out the existing rcupdate.h changeset, because I know of no
alternative that really lets me get rid of the original bogus
changeset.
From: "John W. Linville" <[email protected]>
Date: Tue, 12 Feb 2008 12:04:22 -0500
> net-2.6.26 updates certain to go to the next release
> net-2.6.26-maybe updates that might not make it to the next release
If I knew something was "maybe" ahead of time I simply would not apply
it.
Everything I apply to my tree I feel is likely to get integrated. If
it isn't, I let the patch work itself out on the lists and amongst the
interested developers.
The real issue is deleting crap. Once something that seemed like a
good idea turns sour I want to remove it entirely. This isn't doable
without a rebase.
Also, and Andrew does this a lot, I want to clean up changes which
have problems I only notice later. In fact the rebase process turns
up all kinds of things such as whitespace errors that I get when
merging other people's trees. Having extra changesets with the
small fixups adds zero value and just more churn to go over when
reading changesets. A small annotation to the changelog will
do, that answers the "where did this come from?".
With patches and rebasing that operation is easy, and I'm willing to
deal with all of the rebasing of subsequent changesets that is usually
caused by such things. Yes, I'm even willing to do it to patch #1 in
a 1500 patch tree. :-)
On Tue, Feb 12, 2008 at 11:54:01PM +0000, Al Viro wrote:
> On Tue, Feb 12, 2008 at 03:51:07PM -0800, David Miller wrote:
> > From: Jeff Garzik <[email protected]>
> > Date: Tue, 12 Feb 2008 11:31:48 -0500
> >
> > > I understand the desire to want a nice and clean history, but the
> > > frequency here really has a negative impact on your downstreams.
> >
> > Ok, fair enough. Any alternative suggestions on how to remove turds
> > without them being in the history?
> >
> > > It also totally screws the commit statistics, wiping me and John and the
> > > committers we have preserved out, replacing everybody's committer with
> > > David Miller.
> >
> > I am well aware of this downside, sorry.
>
> Ahem... Use of git-cherry-pick preserves commit information just fine.
Not by default, at least (note they said "commiters", not "authors"):
bfields@pig:~/local/linux-2.6$ git cherry-pick v2.6.25-rc1
Finished one cherry-pick.
Created commit c445dc0: Linux 2.6.25-rc1
1 files changed, 2 insertions(+), 2 deletions(-)
bfields@pig:~/local/linux-2.6$ git show --pretty=raw HEAD
commit c445dc0a8b11877d6038dc528254733348c9f110
tree 4018d5d93f857d946dd89acbb4e45c9da04eadaf
parent b6ce068a1285a24185b01be8a49021827516b3e1
author Linus Torvalds <[email protected]> 1202681894 -0800
committer J. Bruce Fields <[email protected]> 1202861481 -0500
--b.
On Tue, Feb 12, 2008 at 03:58:53PM -0800, David Miller wrote:
> From: Jeff Garzik <[email protected]>
> Date: Tue, 12 Feb 2008 11:36:24 -0500
>
> > Rebasing is low impact only if you don't have git downstream people.
> > Otherwise, you're just treating it as a useful quilt clone, really.
>
> Understood.
>
> One of the key operations that I'm interested in is removing things
> from the history. If I could do that using GIT without any side
> effects and in a way that really would remove it from the tree, I
> would do that in a heartbeat.
>
> At 1500 changesets, a merge conflict shows up about once
> every day or two as 2.6.N nears it's release into final
> as bug fixes trickle in.
>
> I find using GIT to fixup merge errors on a tree of that
> scale to be really painful. And it only fixes up the final
> result in a merge changeset.
Perhaps you need to switch to using quilt. This is the main reason why
I use it.
thanks,
greg k-h
From: James Bottomley <[email protected]>
Date: Tue, 12 Feb 2008 12:24:42 -0600
> Hm ... I think net is a counter example to this. Rebases certainly work
> for them. The issue, I thought, was around the policy of rebasing and
> how often.
>
> I see the question as being one of who creates the history. When
> something goes into your tree, that's an obvious history point and we
> can't rewrite it. However, when something goes into my trees, I don't
> think it's as obviously a history point that has to be preserved, so I
> can rebase obviously, rebasing causes disruption, so I don't do it
> unless it's necessitated by an actual conflict.
And I realize that regrettably I end up rebasing a lot.
Let's say that today I merge a TCP bug fix into Linus's 2.6.24-rcX
tree. When I have the net-2.6.25 tree going I know that this is going
to create merge conflicts with the 80 or so odd TCP patches I have in
there.
Nobody can pull net-2.6.25 into Linus upstream without having to sift
through the merging of that stuff. It never merges cleanly using
the automated mechanisms, because since the changes are split up
nicely there are long changeset dependency chains.
So I rebase, and do the merging work by hand.
Next, let's say Jeff merges some net driver bug fixes into upstream,
resulting in potential conflicts with the several hundred or so driver
changes that are in the net-2.6.25 tree too.
In fact near the end of 2.6.24 development, there was a new merge
conflict created on a daily basis with the net-2.6.25 tree. You
simply cannot avoid this when you're managing 1500+ changes.
I even had to rebase the net-2.6.25 tree once or twice in Australia as
the merge window was openning up because I could push something
cleanly to Linus. There were conflicts created by stuff that got in
before the net-2.6.25 tree, mostly in files like the feature removal
schedule, Kconfig files, and whatnot.
At times I even felt the urge to avoid merging a bug fix upstream
because of all the merge conflicts it would create, but I of course
can't and won't do that. 8)
It actually turns out that things simplify once a tree gets into the
-stable folks hands. I pick out bug fixes as they go upstream, and
toss it to them once Linus sucks it in and I have an upstream
changeset ID to give them. I don't have to worry about -stable
changesets causing development merge grief later on.
And I've also yet to be shown how to completely remove a changeset
from a GIT tree without rebasing :-)
From: Linus Torvalds <[email protected]>
Date: Tue, 12 Feb 2008 10:26:53 -0800 (PST)
> We absolutely MUST NOT have the mindset that "cross-subsystem conflicts
> happen all the time".
Perhaps not, but self-conflicts are the bigger issue for the
networking.
If I (or Jeff or John) push a bug fix to you outside of the merge
window, that blows the entire development tree because at 1500+
changesets I can guarentee you there will be a merge conflict
somewhere.
Near the end of 2.6.24 every bug fix I merged to you created a
conflict with my net-2.6.25 tree. Every time I rebased there
were on the order of 10 to 20 patches with merge conflicts to
resolve.
From: Linus Torvalds <[email protected]>
Date: Tue, 12 Feb 2008 10:48:38 -0800 (PST)
> On Tue, 12 Feb 2008, James Bottomley wrote:
> > Yes ... I don't do that ... Like I said, I only rebase for an actual
> > conflict.
>
> And this is how things should work.
And if conflicts happen every day, what should someone do?
This isn't fantasy, that's exactly how things got with the
networking tree near the end of 2.6.24.
From: Linus Torvalds <[email protected]>
Date: Tue, 12 Feb 2008 10:59:00 -0800 (PST)
> That sure as hell would put the pain on API changes solidly where it
> belongs.
If a person does a driver API change and does all the work to sweep
the entire tree updating all the drivers, doesn't it penalize that
person a bit much to stick a new driver in front of that work?
People write code on top of infrastructure, both new and old, not the
other way around. At least to me, that seems how the merging ought to
work too.
On Tue, 12 Feb 2008 15:58:53 -0800 (PST)
David Miller <[email protected]> wrote:
> But right now I have to redo the include/linux/rcupdate.h that's in
> there because it has a bug.
Well there's a case in point. rcupdate.h is not a part of networking, and
it is random tree-wandering like this which causes me problems and which
will cause Stephen problems.
Now, I don't know which tree "owns" rcupdate.h but it ain't networking.
Probably git-sched.
Nothing in networking depends upon that change (which has a typo in the
comment, btw) hence it can and should have gone through
whichever-tree-owns-that-file.
For Stephen's sake: please.
From: Greg KH <[email protected]>
Date: Tue, 12 Feb 2008 11:15:53 -0800
> On Tue, Feb 12, 2008 at 10:26:53AM -0800, Linus Torvalds wrote:
> > We absolutely MUST NOT have the mindset that "cross-subsystem conflicts
> > happen all the time".
>
> They usually don't, by virtue of our current development model and how
> we have the kernel structured.
BTW, there are some things we can do to with the kernel structure
to help a lot of really idiotic cases. Here are some odd-the-cuff
suggestions:
1) Make feature-removal-schedule a directory with files in it.
Everyone touches that file, creating merge issues.
2) Let's move away from some/dir/{Kconfig,Makefile} schemes and
instead have each "thing" have it's own Kconfig.foo or
Makefile.foo that gets automatically sucked into the main
directory Makefile or Kconfig using file globs or similar.
Even better, encode the building of things into the *.[ch]
files themselves, and have the Kconfig/Makefile machinery
automatically extract this information when you build.
Little things like this would go a long way to eliminating merge
hassles.
For example, with #2, when a driver is added it would only
every add files never edit existing files. Merge conflicts
are impossible unless two new drivers try to use the same
file names. :-)
On Tue, 12 Feb 2008, David Miller wrote:
>
> At 1500 changesets, a merge conflict shows up about once
> every day or two as 2.6.N nears it's release into final
> as bug fixes trickle in.
>
> I find using GIT to fixup merge errors on a tree of that
> scale to be really painful. And it only fixes up the final
> result in a merge changeset.
Heh. I've had the reverse situation: "git rebase" often results in *more*
conflicts than "git merge" (ie "pull").
But one issue is also that when conflicts happen, different people are
used to different things. I'm particularly used to merge-type conflicts,
and in fact there's some fairly advanced support in git for helping
resolve them that people who *don't* do merge-level conflict resolution
may not even be aware of.
In particular, if you want to try it, do something that conflicts and then
do
gitk --merge
to see what the conflict is all about. That is just fancy shorthand for
gitk HEAD...MERGE_HEAD -- <list of conflicting files>
so what it does is to show only the relevant history (the three dots means
that it's a _symmetric_ set difference from HEAD to MERGE_HEAD) for the
merge, and only for the particular files that had conflicts!
This often means that even when you merge a thousand commits (or the thing
you merge *into* has thousands of commits since the merge base, which is
the common case for me), you actually only see a couple of commits - only
the ones that actually modified the conflicting files!
(If you have many files that conflict, you can further narrow it down to
just one at a time by explicitly listing the file/directory you want to
work on, ie do "gitk --merge <pathname-here>").
> Let me give you a good example, just yesterday I had to rebase
> my net-2.6 tree a few times. It's because I put a patch in there
> that the person said was OK but it broke the build. There is
> zero reason for me to push that tree to Linus with the bad
> commit and the revert, it's just noise and makes the tree harder
> for people to review.
Actually, better than rebase in that situation is to just remove the bad
commit. Yes, you'd use "rebase" for it, but you'd use it not to move the
whole series to a newer place, you'd use it just to rebase the commits
*after* the commit you remove.
This is something where I actually think git could and should do better:
git has the capability to act as more of a "quilt replacement", but
because it wasn't part of the original design, we never actualy exposed
the simple queue management commands to do this (stgit does things like
that, though).
So if you haven't pushed out, right now you'd have to do this stupid
thing:
[ start (and activate) a 'fixup' branch at X ]
git checkout -b fixup X
[ edit edit edit to fix it up ]
..
[ commit the fixed state ]
git commit --amend
[ go back to the old broken state ]
git checkout master
[ now, rebase 'master' on top of the fix ]
git rebase fixup
[ ok, done, forget the fixup branch ]
git branch -d fixup
and I don't discourage this kind of behaviour at all, but it is only good
iff:
- you have not pushed things out (obviously), so nobody ever even notices
that you've fixed up stuff
- you haven't pulled anything from outside (so you aren't trying to
rebase other peoples commits).
If you *really* want to try to do this even across merges you've done,
there is fancy a "--preserve-merges" thing that you can try to use
(needs "-i" to work, but "-i" is often cool for other reasons too!)
Basically, what I'm trying to say is that "git rebase" can be used in
fancy ways to do things that people outside your repository will never
even *know* were done. It's only when outsiders can see the effects of git
rebase that you're in trouble!
Linus
On Tue, 12 Feb 2008 16:41:49 -0800 (PST)
David Miller <[email protected]> wrote:
> Here are some odd-the-cuff
> suggestions:
>
> 1) Make feature-removal-schedule a directory with files in it.
> Everyone touches that file, creating merge issues.
>
> 2) Let's move away from some/dir/{Kconfig,Makefile} schemes and
> instead have each "thing" have it's own Kconfig.foo or
> Makefile.foo that gets automatically sucked into the main
> directory Makefile or Kconfig using file globs or similar.
>
> Even better, encode the building of things into the *.[ch]
> files themselves, and have the Kconfig/Makefile machinery
> automatically extract this information when you build.
3) teach people that you don't always have to add new includes right
at the end of the list
4) teach people that you don't have to add Makefile rules right at the
end of the list
5) ditto feature-removal
6) ditto lots of other things.
My usual way of fixing these things when they pop up is to just move the
offending addition to some random position other than end-of-list. I must
have done this hundreds of times and as yet I don't think anyone has noticed ;)
On Tue, Feb 12, 2008 at 07:16:50PM -0500, J. Bruce Fields wrote:
> > Ahem... Use of git-cherry-pick preserves commit information just fine.
>
> Not by default, at least (note they said "commiters", not "authors"):
That's why you give it -r.
From: Linus Torvalds <[email protected]>
Date: Tue, 12 Feb 2008 12:07:07 -0800 (PST)
>
>
> On Tue, 12 Feb 2008, J. Bruce Fields wrote:
> >
> > But the "author" is still preserved, right? Why do you need the
> > committer name to be preserved? (I'm not denying that there could be
> > reasons, I'm just curious what they are.)
>
> It's not that the committer should be preserved, but:
>
> - the chain from author -> committer should be visible in the
> Signed-off-by: lines.
>
> If you rebase somebody elses tree, you screw that up. You need to add
> your sign-off, since now *you* are the new committer, and *you* took
> somebody elses work!
I agree with this and that is exactly what I screwed up by mistake
this time around.
Normally when I rebase I walk through the patches that came from other
people's trees and add signoffs as needed. I understand that this
is frowned upon to some extent as well.
> Put another way: think of the absolute *chaos* that would happen if I were
> to rebase instead of just merging. Every time I pull from you I'd
> invalidate your whole tree, and you'd have to re-generate. It gets
> unmaintainable very quickly.
I actually wouldn't mind that, the first thing I do when sending a
pull request is I stop putting things into my tree and as soon as the
recipient pulls I wipe out my tree and clone a fresh copy of their's.
It's really not a big deal. The pusher can queue patches and other
stuff up in their mailbox or in a directory somewhere. This quiet
period also allows those patches to have some time to be reviewed on
the lists before they actually end up in anyone's tree.
I really like that mode of operation.
On Tue, 12 Feb 2008, Greg KH wrote:
>
> Perhaps you need to switch to using quilt. This is the main reason why
> I use it.
Btw, on that note: if some quilt user can send an "annotated history file"
of their quilt usage, it's something that git really can do, and I'll see
if I can merge (or rather, coax Junio to merge) the relevant part of stgit
to make it possible to just basically get "quilt behaviour" for the parts
of a git tree that you haven't pushed out yet.
A pure patch-stack will be faster at that thing than git would be (it's
simply easier to just track patches), but on the other hand, using git
would get some other advantages outside of the integration issue (eg the
cherry-pick thing really is a proper three-way merge, not just an "apply
patch", so it can do better).
It wasn't the original goal of git, but not only are really doing all the
series management anyway (that is largely what "rebase" is all about,
after all), but the git goals have obviously expanded over time too.
Linus
On Tue, 12 Feb 2008, David Miller wrote:
> From: Linus Torvalds <[email protected]>
> Date: Tue, 12 Feb 2008 10:59:00 -0800 (PST)
>
> > That sure as hell would put the pain on API changes solidly where it
> > belongs.
>
> If a person does a driver API change and does all the work to sweep
> the entire tree updating all the drivers, doesn't it penalize that
> person a bit much to stick a new driver in front of that work?
If that API change doesn't conflict with the work that hundreds of other
people do, it's obviously not a problem whichever way it goes.
And if the API change *does* cause conflicts, then yes, the onus of fixing
those conflicts (again) goes to the person who changed the API. Everybody
else did everything right.
> People write code on top of infrastructure, both new and old, not the
> other way around. At least to me, that seems how the merging ought to
> work too.
You think that infrastructure is more important than outlying code. But
you do that only because you write the infrastructure, not because you
have any logical reason to think so.
The fact is, that "outlying code" is where we have all the bulk of the
code, and it's also where we have all those developers who aren't on the
"inside track". So we should help the outliers, not the core code.
And very fundamentally, API changes are to be discouraged. If we make them
harder to do and make people think twice (and occasionally say "not worth
it"), that sounds like a damn good thing to me.
Linus
On Wed, Feb 13, 2008 at 12:48:41AM +0000, Al Viro wrote:
> On Tue, Feb 12, 2008 at 07:16:50PM -0500, J. Bruce Fields wrote:
> > > Ahem... Use of git-cherry-pick preserves commit information just fine.
> >
> > Not by default, at least (note they said "commiters", not "authors"):
>
> That's why you give it -r.
What version of git have you tried this with? At least on my version,
-r is a no-op, and the results are the same; the author is still kept
and the maintainer changed. I thought it had been that way for ages.
--b.
bfields@pig:~/local/linux-2.6$ git --version
git version 1.5.4.rc2.60.gb2e62
bfields@pig:~/local/linux-2.6$ git cherry-pick -r v2.6.25-rc1
Finished one cherry-pick.
Created commit 0277143: Linux 2.6.25-rc1
1 files changed, 2 insertions(+), 2 deletions(-)
bfields@pig:~/local/linux-2.6$ git cat-file -p HEAD
tree 4018d5d93f857d946dd89acbb4e45c9da04eadaf
parent b6ce068a1285a24185b01be8a49021827516b3e1
author Linus Torvalds <[email protected]> 1202681894 -0800
committer J. Bruce Fields <[email protected]> 1202863894 -0500
Linux 2.6.25-rc1
.. and I really need to call it something else. Maybe it is time to
bring back the weasel series, since weasels always make me feel good
about a kernel.
On Wed, 13 Feb 2008, Al Viro wrote:
> On Tue, Feb 12, 2008 at 07:16:50PM -0500, J. Bruce Fields wrote:
> > > Ahem... Use of git-cherry-pick preserves commit information just fine.
> >
> > Not by default, at least (note they said "commiters", not "authors"):
>
> That's why you give it -r.
Hmm. "-r" is a no-op to git-cherry-pick.
And even if you thought it should preserve committer information, it
really _really_ shouldn't. You're creating a new commit, you're the new
committer. The old committer is meaningless. It doesn't matter at all if
you try to keep the old committer information (which you can do by faking
GIT_COMMITER_NAME?EMAIL): you're simply just _lying_ at that point. The
original committer has a different commit in his tree, and if you try to
claim that your cherry-picked commit is his, you're only doing everybody a
disservice.
If you meant using "-x", then yes, that retains the actual pointer to the
original commit, but it's not the default, because it shouldn't be used
unless you plan to carry both around on purpose (ie it's mainly useful for
"maintain a stable branch that has commits cherry-picked from mainline"
kinds of things).
Linus
From: Andrew Morton <[email protected]>
Date: Tue, 12 Feb 2008 16:37:42 -0800
> Well there's a case in point. rcupdate.h is not a part of networking, and
> it is random tree-wandering like this which causes me problems and which
> will cause Stephen problems.
>
> Now, I don't know which tree "owns" rcupdate.h but it ain't networking.
> Probably git-sched.
>
> Nothing in networking depends upon that change (which has a typo in the
> comment, btw) hence it can and should have gone through
> whichever-tree-owns-that-file.
>
> For Stephen's sake: please.
At least thie time I did make sure that change got posted to
linux-kernel and got properly reviewed by the de-facto maintainer
(Paul McKenney). :-)
I'll toss it.
But how do I do that using GIT without rebasing and without
having this ugly changeset and revert in there?
That's the thing I want answered, and although Al claims it does,
git cherry-pick does not seem to do what I want either.
From: Linus Torvalds <[email protected]>
Date: Tue, 12 Feb 2008 16:44:47 -0800 (PST)
> gitk --merge
...
> This is something where I actually think git could and should do better:
> git has the capability to act as more of a "quilt replacement", but
> because it wasn't part of the original design, we never actualy exposed
> the simple queue management commands to do this (stgit does things like
> that, though).
>
> So if you haven't pushed out, right now you'd have to do this stupid
> thing:
Thanks for all the useful info.
But as soon as I've applied any patches to my tree I've "pushed out".
So this scheme doesn't work for me. The first thing I do when I have
changes to apply is clone a tree locally and on master.kernel.org,
then I apply that first patch locally and push it out to master.
What would be really cool is if you could do the rebase thing, push
that to a remote tree you were already pushing into and others could
pull from that and all the right things happen.
A rebase is just a series of events, and those could propagate to
people who are pulling from the tree. I'm pretty sure GIT could
support this.
On Tue, 12 Feb 2008 12:02:08 +1100 Stephen Rothwell wrote:
> Hi all,
>
> Andrew was looking for someone to run a linux-next tree that just
> contained the subsystem git and quilt trees for 2.6.x+1 and I (in a
> moment of madness) volunteered. So, this is to announce the creating of
> such a tree (it doesn't exist yet) which will require some (hopefully)
> small amount of work on the part of subsystem maintainers.
>
> Those interested in discussion about this are encouraged to join the
> [email protected] mailing list.
>
> The first things I need from the subsystem maintainers (you know who you
> are) are a contact address (a list address is fine) and at least one git
> branch or quilt series that contains all the things you want to see go
> into 2.6.26. I am happy for there to be multiple branches/series (in
> fact there probably will need to be in some cases where there are
> dependencies on others work).
Hi Stephen,
Please merge the quilt series/patchset from
http://oss.oracle.com/~rdunlap/kernel-doc-patches/current/
I expect this to be fairly small and localized (very few conflicts).
Thanks,
---
~Randy
From: Andrew Morton <[email protected]>
Date: Tue, 12 Feb 2008 16:47:26 -0800
> My usual way of fixing these things when they pop up is to just move
> the offending addition to some random position other than
> end-of-list. I must have done this hundreds of times and as yet I
> don't think anyone has noticed ;)
That sounds like a real useful usage of your time.
I think my proposal saves everyone, including you, this time and the
effort necessary to implement it would only need to be expended once
instead of this ever present time and learning sink you seem to be
advocating.
From: Linus Torvalds <[email protected]>
Date: Tue, 12 Feb 2008 16:49:46 -0800 (PST)
> Btw, on that note: if some quilt user can send an "annotated history file"
> of their quilt usage, it's something that git really can do, and I'll see
> if I can merge (or rather, coax Junio to merge) the relevant part of stgit
> to make it possible to just basically get "quilt behaviour" for the parts
> of a git tree that you haven't pushed out yet.
Sounds interesting and nice, but not relevant for my current
workflow.
I've always "already pushed out".
From: Linus Torvalds <[email protected]>
Date: Tue, 12 Feb 2008 16:53:50 -0800 (PST)
> The fact is, that "outlying code" is where we have all the bulk of the
> code, and it's also where we have all those developers who aren't on the
> "inside track". So we should help the outliers, not the core code.
Good point.
For the record I write drivers that use infrastructure too
and have to deal with the API changes just as equally :-)
On Tue, Feb 12, 2008 at 04:59:23PM -0800, Linus Torvalds wrote:
>
>
> On Wed, 13 Feb 2008, Al Viro wrote:
>
> > On Tue, Feb 12, 2008 at 07:16:50PM -0500, J. Bruce Fields wrote:
> > > > Ahem... Use of git-cherry-pick preserves commit information just fine.
> > >
> > > Not by default, at least (note they said "commiters", not "authors"):
> >
> > That's why you give it -r.
>
> Hmm. "-r" is a no-op to git-cherry-pick.
*duh*
OK, I plead the lack of caffeine when reading the original posting.
-r used to be "reproduce the changeset", but that _excludes_ committer.
Nevermind.
FWIW, I prefer to keep many branches and use suffix (.b<number>) to
distinguish between them. And cherry-pick/reorder/split/collapse
as needed on transition to the next one. At least that avoids
some problems for secondary trees - branches do not jump.
Since branches tend to be relatively small, they don't get conflicts
that open and I can postpone switch to new branch until it really
has to be done.
I don't know how to deal with tricky branch topology; every time when I
get to structure like <branch X is on top of Y+Z> it becomes very painful
to work on all these topics in parallel. For trees maintained by different
people... <shudder>
On Tue, 12 Feb 2008, David Miller wrote:
>
> > Put another way: think of the absolute *chaos* that would happen if I were
> > to rebase instead of just merging. Every time I pull from you I'd
> > invalidate your whole tree, and you'd have to re-generate. It gets
> > unmaintainable very quickly.
>
> I actually wouldn't mind that, the first thing I do when sending a
> pull request is I stop putting things into my tree and as soon as the
> recipient pulls I wipe out my tree and clone a fresh copy of their's.
You *really* don't see the problem here?
> I really like that mode of operation.
*YOU* like it, because it never generates any issues for *you*. You're the
top in your heap, and the people above you don't do that insane thing, so
you get all of the advantages, with none of the downsides. Of *course* you
like it.
But as people have pointed out, it generates issues for the people under
you! If I did it, the people who now complain about networking would
not just be a couple, it would be everybody. Nobody could depend on
anything out there, because everything would have to rebase.
You just don't see the problems, because the only person above you isn't
crazy enough to do what you propose. You also don't do ten merges a day of
subsystems you don't know.
The importance of merging (rather, not screwing up history in general)
becomes really obvious when things go tits-up. Then they go tits-up
*without* screwing up the history of the trees that were hopefully tested
individually.
If you re-base things that others developed, you lose that. Imagine if I
merged first Greg's tree (by rebasing), and then there was some
fundamental thing that didn't cause a conflict, but just made something
not work, when I rebased yours on top. Think about what happens.
Now I've merged (say) 1500 networking-related commits by rebasing, but
because I rebased on top of Greg's tree that I had also rebased,
absolutely *none* of that has been tested in any shape of form. I'd not
use most of the things I pulled, so I'd never see it, I'd just push out
something that was very different from *both* trees I pulled, with no way
to really blame the merge - because it doesn't even exist.
So as a result, some *random* commit that was actually fine on its own has
now become a bug, just because it was re-written.
You don't see the problem here?
Yes, this is the *crap* you do all the time. You don't see the problems as
much, because you merge probably only about a tenth of the volume I merge,
and you can keep track of the subsystem more. But even though you don't
have nearly the same kinds of problems, people have complained about your
process.
So there's a real reason why we strive to *not* rewrite history. Rewriting
history silently turns tested code into totally untested code, with
absolutely no indication left to say that it now is untested.
You can limit the damage by keeping it to a single subsystem and by
serializing that subsystem by - for example - talking it over amonst
yourself what order you do things in, and yes, most of the time rewriting
doesn't hurt anything at all, but I guarantee you that it's a big mistake
to do, and the mistake gets bigger the more _independent_ people you have
involved.
Linus
From: Linus Torvalds <[email protected]>
Date: Tue, 12 Feb 2008 17:31:10 -0800 (PST)
> You don't see the problems as much, because you merge probably only
> about a tenth of the volume I merge, and you can keep track of the
> subsystem more.
Good point.
Now how do I remove a bogus commit for a tree that I've already pushed
out and published for other people, without any record of it appearing
in the GIT tree any more?
How do I insert build fixes into existing changesets so that the tree
is more bisectable?
If Jeff merged in a tree that introduced a ton of whitespace errors
git is complaing about, is there a way I can fixup that changeset
in-place? (or should I tell Jeff to start adhering to GIT's whitespace
warning messages when he applies patches?)
Those are basically the major operations that would allow me to
seriously consider rebasing a ton less often.
On Tue, 12 Feb 2008, David Miller wrote:
>
> But as soon as I've applied any patches to my tree I've "pushed out".
> So this scheme doesn't work for me. The first thing I do when I have
> changes to apply is clone a tree locally and on master.kernel.org,
> then I apply that first patch locally and push it out to master.
I actually suggest you literally delay your push-out.
I don't generally delay things by a lot, but I tend to try to at least do
a compile in between pushing out - and even if I've pulled something in
between the thing that broke, I'll just "git reset --hard" to a working
state if something broke, and just re-pull instead of even trying to
rebase or anything like that.
(IOW, I often find it much easier to just start over and re-do than
actually doing a rebase).
I don't do it all the time, by any means, but there's really no huge
reason to push out all the time. And that's doubly true for subsystem
maintainers. Quite often, the right thing to do is to only push out when
you are ready to do the "please pull" message.
> What would be really cool is if you could do the rebase thing, push
> that to a remote tree you were already pushing into and others could
> pull from that and all the right things happen.
It would also be really cool if Claudia Schiffer had decided that hiding
under my desk is a good idea.
IOW, you really haven't though that through. That is how TLA and darcs
worked, and it's a total disaster.
Trust me, you don't know how good you have it.
Linus
From: Linus Torvalds <[email protected]>
Date: Tue, 12 Feb 2008 17:41:19 -0800 (PST)
> Trust me, you don't know how good you have it.
I know, preserving history is valuable.
I'll take up the various suggestions and try working
a little differently this time around. We'll see
how well it works.
On Tue, 12 Feb 2008 17:16:03 -0800 (PST) David Miller <[email protected]> wrote:
> From: Andrew Morton <[email protected]>
> Date: Tue, 12 Feb 2008 16:37:42 -0800
>
> > Well there's a case in point. rcupdate.h is not a part of networking, and
> > it is random tree-wandering like this which causes me problems and which
> > will cause Stephen problems.
> >
> > Now, I don't know which tree "owns" rcupdate.h but it ain't networking.
> > Probably git-sched.
> >
> > Nothing in networking depends upon that change (which has a typo in the
> > comment, btw) hence it can and should have gone through
> > whichever-tree-owns-that-file.
> >
> > For Stephen's sake: please.
>
> At least thie time I did make sure that change got posted to
> linux-kernel and got properly reviewed by the de-facto maintainer
> (Paul McKenney). :-)
Ah, thanks for that - I'm behind in my lkml reading. Again.
> I'll toss it.
While I was there I spotted a howling bug in rcu_assign_pointer(): a
double-touch of the second arg. Nobody has done
rcu_assign_pointer(p, something_with_side_effects);
before? That would be surpising...
Paul has been informed ;)
> But how do I do that using GIT without rebasing and without
> having this ugly changeset and revert in there?
Who, me? umm, get git changed? It seems pretty clear that it isn't
matching legitimate kernel development workflow. And it is a tool's job to
do that, rather than forcing humans to change there practices.
> That's the thing I want answered, and although Al claims it does,
> git cherry-pick does not seem to do what I want either.
On Tue, 12 Feb 2008, David Miller wrote:
>
> Now how do I remove a bogus commit for a tree that I've already pushed
> out and published for other people, without any record of it appearing
> in the GIT tree any more?
So, the answer is: if others have actually pulled, it's simply not
possible.
There simply is no way to undo history that isn't local any more. You just
*have* to revert, or you need to find every single person that pulled
(directly or indirectly) and ask them to undo all the work they did on top
of your bad commit.
This is not git-related, btw. That's just how the world works. It's a bit
like the internet - if you said something stupid on #IRC and it made it to
bash.org, there's not a whole lot you can do to "undo" your stupidity. You
can only hope to do better in the future and assume people forget.
> How do I insert build fixes into existing changesets so that the tree
> is more bisectable?
Just delay pushing out. There really is _zero_ downside to this. None at
all. There are only upsides.
> If Jeff merged in a tree that introduced a ton of whitespace errors
> git is complaing about, is there a way I can fixup that changeset
> in-place? (or should I tell Jeff to start adhering to GIT's whitespace
> warning messages when he applies patches?)
Umm. Git doesn't complain about whitespace _except_ when applying patches.
So if you don't rebase his work, you'll never see the whitespace warnings
either!
Of course, we'd probably wish that Jeff cared about the whitespace
warnings and pushed back on them, but the fact is, that warning isn't
meant for you - because by the time you pull Jeff's tree, it's simply not
your issue any more. Jeff already applied it. Either he's your trusted
lietenant or he's not.
Quite frankly, to me it sounds like you're not ready to "let go" and trust
the people under you. Trust me, it's worth it. It's why your life is easy:
I have let go and I trust you.
Also, I'd *much* rather have a few problems in the tree than have people
screw up history in order to hide them. Sure, we want to keep things
bisectable, but quite frankly, if you do a reasonable job and compile the
kernel before you push out, it will be "mostly bisectable".
And yes, mistakes happen. Mistakes will *always* happen. It's ok. Relax.
Let me put it another way: You're _both_ going to be *much* better off
pushing back on Jeff, telling him that "I can't pull from you because your
tree is ugly and doesn't compile", than taking his tree and rebasing it.
Remember? I used to do that all the time. I berated the ACPI people for
creating monster trees that were horrible and contained fifteen merges and
two real commits. I didn't try to clean it up for them, I just told them
what the problem was, and you know what? The ACPI tree is one of the
cleanest ones out there now!
So in short:
- clean trees and bisectability are all wonderful things. No doubt about
that at all.
- but if getting there means that you lose a lot of _other_ wonderful
things (like being able to trust history, and the people being under
your watchful eyes havign to deal with you re-writing their trees),
we'd be much better off taking the occasional commit that fixes things
up _after_ the fact rather than before!
- and you actually can help fix your issues by doing some simple things
*before* pushing out, rather than push out immediately. IOW, do your
whitespace sanity fixes, your compile checks etc early, and don't push
out until after you've done them.
Hmm?
Linus
On Tue, 12 Feb 2008 17:57:19 -0800 (PST) Linus Torvalds <[email protected]> wrote:
> - and you actually can help fix your issues by doing some simple things
> *before* pushing out, rather than push out immediately. IOW, do your
> whitespace sanity fixes, your compile checks etc early, and don't push
> out until after you've done them.
>
One of the things which linux-next could/should do is to help weed out the
silly build breaks, typos, missing documentation updates, missed checkpatch
opportunities, etc, etc. As well as real bugs.
So it would not be efficient for David to do all this queue-cleaning
*prior* to putting the tree into linux-next, because more stuff will pop up
anyway.
So perhaps a better workflow would be keep the linux-next trees all messy,
and then each developer can consolidate, rebase, join and drop things prior
to sending their individual trees to Linus.
If so, then git could perhaps help, by retaining sufficient metadata for
the maintainer to track the fact that this-patch-here fixes
that-patch-there.
On Tue, Feb 12, 2008 at 04:49:46PM -0800, Linus Torvalds wrote:
> On Tue, 12 Feb 2008, Greg KH wrote:
> >
> > Perhaps you need to switch to using quilt. This is the main reason why
> > I use it.
>
> Btw, on that note: if some quilt user can send an "annotated history file"
> of their quilt usage, it's something that git really can do, and I'll see
> if I can merge (or rather, coax Junio to merge) the relevant part of stgit
> to make it possible to just basically get "quilt behaviour" for the parts
> of a git tree that you haven't pushed out yet.
So this is what I do for ext4 development. We maintain a quilt series
in git, which is located here at: http://repo.or.cz/w/ext4-patch-queue.git
A number of ext4 developers have write access to commit into that
tree, and we coordinate amongst ourselves and on
[email protected]. I tend to suck it into git using the
"guilt" package, and do periodic merge testing with a number of git
queues to detect potential merge conflicts. Not as many as James
does, but I may start doing more of that once I steal his scripts. :-)
The patch queue also gets automatic testing on a number different
platforms; for that reason the series files comments which version of
the kernel it was last based off of, so the ABAT system can know what
version of the kernel to use as the base of the quilt series.
I do a fair amount of QA, including copy editing and in some cases
rewriting the patch descriptions (which are often pretty vile, due to
a number of the ext4 developers not being native English speakers; not
their fault, but more than once I've had no idea what the patch
description is trying to say until I read through the patch very
closely, which is also good for me to do from a code QA point of view :-).
Periodically, the patch queue gets pushed into the ext4.git tree and
as a patch series on ftp.kernel.org.
I've never been very happy with stgit because of past experiences
which has scarred me when it got get confused and lost my entire patch
series (this was before git reflogs, so recovery was.... interesting).
There's always been something deeply comforting about having the ASCII
patch series since it's easy to back it up and know you're in no
danger of losing everything in case of a bug. Also, having the patch
series stored in ASCII as a quilt stack means that we can store the
quilt stack itself in git, and with repo.or.cz it allows us to have
multiple write access to the shared quilt stack, while still giving us
the off-line access advantages of git. (Yes, I've spent plane rides
rewriting patch descriptions. :-)
The other advantage of storing the patch stack as a an ASCII quilt
series is we have a history of changes of the patches, which we don't
necessarily have if you just use stgit to rewrite the patch. So we
have the best of both worlds; what gets checked into Linus's tree is a
clean patch series, but we keep some history of different versions of
a patch over time in the ext4-patch-queue git repository. (I wish we
had better changelog comments there too, but I'll take what I can
get.)
- Ted
On Tue, 12 Feb 2008, Andrew Morton wrote:
>
> So it would not be efficient for David to do all this queue-cleaning
> *prior* to putting the tree into linux-next, because more stuff will pop up
> anyway.
Well, what others have done is to have special "temporary branches".
This is what git itself does, for example. The "pu" branch in git is used
for experimental stuff, and it's _declared_ to be rebased, redone, and
generally just unsafe at any moment.
So it is easy to have a special "testing" branch that is just declared to
be unsafe. Make Linux-next pull that testing branch - it will pollute the
Linux-next tree (and anybody else who just wants to see what the current
state is), but since those are re-generatd from scratch every day
_anyway_, so who cares?
But don't make it something people pull by mistake (ie never call it
"master", and when mentioning it in some email message, always mention the
fact that it's not a stable branch, and never ask anybody to pull it
without making it very clear that it's just for testing, not for real
merging).
So git does have support for those things. They are very much "secondary"
branches (any tree they are pulled into will itself become "poisoned" and
unstable), but it's easy enough to have something like that for testing
purposes. And if it all tests out fine, you can just move it as-is into
the "real" branch if you want to.
Linus
From: Andrew Morton <[email protected]>
Date: Tue, 12 Feb 2008 18:06:13 -0800
> So perhaps a better workflow would be keep the linux-next trees all
> messy, and then each developer can consolidate, rebase, join and
> drop things prior to sending their individual trees to Linus.
We could do that, but I believe Linus's main point is that if we go
and change the trees then in the end we're merging in something
different than what was actually tested, namely all the original trees
as merged into linux-next.
And I kind of agree with that.
On Tue, 2008-02-12 at 17:20 -0800, David Miller wrote:
> What would be really cool is if you could do the rebase thing, push
> that to a remote tree you were already pushing into and others could
> pull from that and all the right things happen.
>
> A rebase is just a series of events, and those could propagate to
> people who are pulling from the tree. I'm pretty sure GIT could
> support this.
Yes, this is exactly the feature I'm looking for. It would allow the
downstream users of a rebased tree to rebase themselves correctly.
All the information about the rebase is in the reflog ... it can't be
too difficult to pass it through on a pull and allow the downstream tree
to do the right thing.
James
On Tue, 12 Feb 2008, James Bottomley wrote:
>
> Yes, this is exactly the feature I'm looking for. It would allow the
> downstream users of a rebased tree to rebase themselves correctly.
>
> All the information about the rebase is in the reflog ... it can't be
> too difficult to pass it through on a pull and allow the downstream tree
> to do the right thing.
Guys, you simply have no idea what you're talking about.
Those downstream trees can have done things themselves. They *depended* on
the state you gave them.
You can't just say "oops, I lied, this is the state you should have used,
now it's _your_ mess to sort out".
OF COURSE it's what you'd like to use - it absolves you of any and all
actual responsibility. But dammit, that's not what anybody else wants than
the irresponsible person who cannot be bothered to stand up for his work!
If you're not confident enough about your work, don't push it out! It's
that simple. Pushing out to a public branch is a small "release".
Have the f*cking back-bone to be able to stand behind what you did!
Linus
On Tue, Feb 12, 2008 at 06:20:12PM -0800, David Miller wrote:
> From: Andrew Morton <[email protected]>
> Date: Tue, 12 Feb 2008 18:06:13 -0800
>
> > So perhaps a better workflow would be keep the linux-next trees all
> > messy, and then each developer can consolidate, rebase, join and
> > drop things prior to sending their individual trees to Linus.
>
> We could do that, but I believe Linus's main point is that if we go
> and change the trees then in the end we're merging in something
> different than what was actually tested, namely all the original trees
> as merged into linux-next.
>
> And I kind of agree with that.
Make the distinction earlier. With ocfs2 and configfs (we got
this scheme from Jeff), we keep the topic branches as "unsafe" - that
is, officially rebaseable . We merge them all into a big "ALL" branch,
which is also "unsafe". Andrew pulls this for -mm, and it gets tested
here. If there is a brown-paper-bag problem, we can tell the original
author to fix it. Then we re-pull the topic - effectively a rebase.
The ALL is also rebased. But that's Ok, it will never go towards Linus.
When a topic is considered worthy of going upstream, we pull it
to a branch called "upstream-linus". This branch is *NEVER* rebased.
Now that the topic is in upstream-linus, the original topic branch can't
be rebased either. So any fixes to that topic going forward will stay
in the history. Since that topic was pulled into ALL for testing, we
are using the identical commits that got tested.
Joel
--
"I have never let my schooling interfere with my education."
- Mark Twain
Joel Becker
Principal Software Developer
Oracle
E-mail: [email protected]
Phone: (650) 506-8127
On Tue, 2008-02-12 at 18:35 -0800, Linus Torvalds wrote:
>
> On Tue, 12 Feb 2008, James Bottomley wrote:
> >
> > Yes, this is exactly the feature I'm looking for. It would allow the
> > downstream users of a rebased tree to rebase themselves correctly.
> >
> > All the information about the rebase is in the reflog ... it can't be
> > too difficult to pass it through on a pull and allow the downstream tree
> > to do the right thing.
>
> Guys, you simply have no idea what you're talking about.
>
> Those downstream trees can have done things themselves. They *depended* on
> the state you gave them.
>
> You can't just say "oops, I lied, this is the state you should have used,
> now it's _your_ mess to sort out".
>
> OF COURSE it's what you'd like to use - it absolves you of any and all
> actual responsibility. But dammit, that's not what anybody else wants than
> the irresponsible person who cannot be bothered to stand up for his work!
>
> If you're not confident enough about your work, don't push it out! It's
> that simple. Pushing out to a public branch is a small "release".
>
> Have the f*cking back-bone to be able to stand behind what you did!
Erm, I would like this feature as a downstream user.
I'm not asking for this to be the default or even easily available.
However, when you know you've based a downstream tree on what you know
to be a volatile base, it would be very useful information to have.
Right at the moment, I maintain a <branch> and a <branch>-base and
simply cherry pick the commits between the two to do the right thing
when I know my volatile base has changed. It would be very helpful to
have a version of rebase that new my base had been rebased.
Basing on a tree I know to be volatile is sometimes a development
decision I make as a downstream user ... I'm just wishing the tools
could help me handle the problems better.
James
On Tue, 12 Feb 2008, James Bottomley wrote:
>
> Right at the moment, I maintain a <branch> and a <branch>-base and
> simply cherry pick the commits between the two to do the right thing
> when I know my volatile base has changed. It would be very helpful to
> have a version of rebase that new my base had been rebased.
Hey, I know, you could use.. drumroll..
"git rebase"
I know that's a big leap of faith, to use git rebase for rebasing, but
there you have it. Us git people are kind of odd that way.
IOW, if you know the old broken base, and the new base, just do
git rebase --onto newbase oldbase
and it should do exactly that (basically lots of automated cherry-picks).
[ But the fact is, if you did anything fancy (like pulled in other peoples
work), you cannot sanely rebase _those_ peoples work. They didn't screw
up to begin with! You can play with "git rebase -i --preserve-merges",
of course, but I really think you're doing something wrong if you start
pulling other peoples work into an unstable thing, so while it may work,
I'd strongly suggest against even trying, because the problem is your
workflow ]
So let's say that you have a remote branch that you track that goes
rebasing (let's call it "origin/pu" to match the real-life git behaviour),
then you should literally be able to do
old=$(git rev-parse origin/pu) &&
git fetch &&
new=$(git rev-parse origin/pu) &&
git rebase --onto $new $old
and no, I didn't actually test it, but hey, it really should be that
simple.
[ And no, you don't really need to do those "old=" and "new=" things, they
are there to make it explicit - you could easily just have done
git fetch
.. oh, noticed that origin/pu changed ..
git rebase --onto origin/pu origin/pu@{1}
where we just let git take care of the old/new itself using the reflog,
so that "origin/pu@{1}" assumes that you just know that the only thing
that has changed origin/pu was that previous "git fetch", and that
really *did* change it. ]
In other words, git does give you exactly what you want, but nothing
really changes the fact that you should only rebase like this only if:
- you haven't already exported the result (or only exported it as those
unstables branches that people know to avoid)
- your changes on top are just your own linear series of commits (where
"applying a patch from somebody else" is still _your_ commit, of
course, just with authorship attributed to somebody else)
so that part really is very fundamental.
Linus
On Tue, 12 Feb 2008, Linus Torvalds wrote:
>
> git rebase --onto $new $old
..and in case it wasn't clear - this is just a general way of saying "move
the commits on this branch since $old to be based on top of $new" instead.
You can pick out those old/new commit ID's using gitk or whatever if you
wish. Neither the $new or the $old needs to even be an existing branch -
just pick them with gitk.
So if you literally want to just move the top 5 commits (assuming those
top five cmmits are just a nice linear thing you did) from the current
branch to be on top on another branch instead, you can literally do this:
# save this state, maybe we want to keep it around. Call it "old"
git branch old-branch
# rebase the top five commits onto $target
git rebase --onto $target HEAD~5
ta-daa - all done. The branch you are on will now have been rewritten to
be the top five commits moved to be on top of the $target you chose, and
if you want to get back the old state, it's nicely squirrelled away in
"old-branch".
(That obviously assumes no merge conflicts - you'll have to resolve those
yourself ;)
Of course, if you didn't even want to save the old branch, just skip the
first step. If you have reflogs enabled (and git does that by default in
any half-way recent version), you can always find it again, even without
having to do "git fsck --lost-found", at least as long as you don't delete
that branch, and it hasn't gotten pruned away (kept around for the next 90
days by default, iirc)
Linus
On Tue, 2008-02-12 at 19:31 -0800, Linus Torvalds wrote:
>
> On Tue, 12 Feb 2008, James Bottomley wrote:
> >
> > Right at the moment, I maintain a <branch> and a <branch>-base and
> > simply cherry pick the commits between the two to do the right thing
> > when I know my volatile base has changed. It would be very helpful to
> > have a version of rebase that new my base had been rebased.
>
> Hey, I know, you could use.. drumroll..
>
> "git rebase"
>
> I know that's a big leap of faith, to use git rebase for rebasing, but
> there you have it. Us git people are kind of odd that way.
>
> IOW, if you know the old broken base, and the new base, just do
>
> git rebase --onto newbase oldbase
>
> and it should do exactly that (basically lots of automated cherry-picks).
OK, smarty-pants ... that works nicely, thanks!
I'm used to maintaining <branch> and <branch>-base, so this probably
suits my workflow better than getting the information from the reflog.
It wasn't clear from the git rebase man page that it would work like
that.
James
On Tue, Feb 12, 2008 at 09:00:16PM -0600, James Bottomley wrote:
> On Tue, 2008-02-12 at 18:35 -0800, Linus Torvalds wrote:
> >
> > On Tue, 12 Feb 2008, James Bottomley wrote:
> > >
> > > Yes, this is exactly the feature I'm looking for. It would allow the
> > > downstream users of a rebased tree to rebase themselves correctly.
> > >
> > > All the information about the rebase is in the reflog ... it can't be
> > > too difficult to pass it through on a pull and allow the downstream tree
> > > to do the right thing.
> >
> > Guys, you simply have no idea what you're talking about.
> >
> > Those downstream trees can have done things themselves. They *depended* on
> > the state you gave them.
> >
> > You can't just say "oops, I lied, this is the state you should have used,
> > now it's _your_ mess to sort out".
> >
> > OF COURSE it's what you'd like to use - it absolves you of any and all
> > actual responsibility. But dammit, that's not what anybody else wants than
> > the irresponsible person who cannot be bothered to stand up for his work!
> >
> > If you're not confident enough about your work, don't push it out! It's
> > that simple. Pushing out to a public branch is a small "release".
> >
> > Have the f*cking back-bone to be able to stand behind what you did!
>
> Erm, I would like this feature as a downstream user.
>
> I'm not asking for this to be the default or even easily available.
> However, when you know you've based a downstream tree on what you know
> to be a volatile base, it would be very useful information to have.
>
> Right at the moment, I maintain a <branch> and a <branch>-base and
> simply cherry pick the commits between the two to do the right thing
> when I know my volatile base has changed. It would be very helpful to
> have a version of rebase that new my base had been rebased.
>
> Basing on a tree I know to be volatile is sometimes a development
> decision I make as a downstream user ... I'm just wishing the tools
> could help me handle the problems better.
>
There's also a difference between a downstream user and a downstream
developer. While rebasing does cause trouble for folks doing downstream
development off of the tree in question, there's no reason why this
should be the case for users that are simply trying to follow the tree.
I push changes out to my tree so people have a chance to poke through it
and to see what's going on, though I do not generally encourage people to
fork off of it given that I end up rebasing periodically. At the moment
the options seem to be down to the following:
1 - Push changes out without rebasing
2 - Push changes out periodically rebased
3 - Hide the tree away in my home directory on hera
4 - Force people to get at the current tree through -mm
4) generally isn't very realistic, given that -mm releases have far too
many other changes and the releases are quite spread out. 3) is doable,
but I publish the tree as a convenience to the folks wanting to see
what's going on in the tree, and I would rather continue doing so.
This leaves 2) my current workflow, and 1) which ends up creating a
lot of extra metadata. The common cases thelre are patches + reversions
and merge points. Holding on to a patch for some period of time before
pushing it out to ensure that there won't be a reversion later rarely
tends to work in practice. Most of the time I end up having to revert
something it's because someone else found a problem with a given change
_after_ the it was pushed out, and which was not caught with my local
tree or testing.
On the other hand, perhaps it's also useful to see the reversions in the
history, particularly to see what the rationale for the change not
working out was (which could be helpful for people working on the same
stuff later on).
This then leaves merge points. During merge window time people are
pulling on a pretty frequent basis, which also breaks down to a few
fairly common cases:
1 - Bringing in new stuff to be supported (ie, system calls).
2 - Infrastructure support bits that have gone in through a
subsystem tree, when you have local patches blocked until the
subsystem tree has merge.
3 - Catching and resolving conflicts before bisect gets broken.
4 - Trying to make sure that it all merges cleanly.
I've generally worked around 2) by doing multiple merges during the merge
window, but it's also appealing to just rebase and throw in the rest of
the outstanding bits before sending out the initial merge request.
1) and 3) often tend to have dependencies on each other, and tend to be
the area where the most troubles arise. 3) and 4) are really the places
where rebasing appeals the most, and is also where we see the highest
concentration of merge points.
If there's a nice way to resolve this without a rebase that would be
nice. For users in general, I suspect most people are just interested in
a pull that can traverse a rebase without having them wipe the tree. Most
users that are tracking a tree of interest seem to understand that a
development tree is highly volatile during merge window time, and that
they track the tree knowing that they will occasionally have to toss it
and start over when it's rebased. Having a way to simplify this that
doesn't create an additional maintenance burden would be ideal. I suspect
that if people are never rebasing in publicly visible trees, we will
simply see a shift to unpublished trees instead, which doesn't seem to be
what anyone wants either.
On Tue, Feb 12, 2008 at 05:31:10PM -0800, Linus Torvalds wrote:
> The importance of merging (rather, not screwing up history in general)
> becomes really obvious when things go tits-up. Then they go tits-up
> *without* screwing up the history of the trees that were hopefully tested
> individually.
>
> If you re-base things that others developed, you lose that. Imagine if I
> merged first Greg's tree (by rebasing), and then there was some
> fundamental thing that didn't cause a conflict, but just made something
> not work, when I rebased yours on top. Think about what happens.
>
> Now I've merged (say) 1500 networking-related commits by rebasing, but
> because I rebased on top of Greg's tree that I had also rebased,
> absolutely *none* of that has been tested in any shape of form. I'd not
> use most of the things I pulled, so I'd never see it, I'd just push out
> something that was very different from *both* trees I pulled, with no way
> to really blame the merge - because it doesn't even exist.
>
> So as a result, some *random* commit that was actually fine on its own has
> now become a bug, just because it was re-written.
If there was a "fundamental thing that didn't cause a conflict", then
the two trees in question probably didn't touch the same code, so would
probably merge cleanly, for the same reason that one rebased onto the
other cleanly. But depending on what the "fundamental thing" was, the
merge might still introduce the same bug, right?
--b.
On Tue, Feb 12, 2008 at 05:20:51PM -0800, David Miller wrote:
> From: Linus Torvalds <[email protected]>
> Date: Tue, 12 Feb 2008 16:44:47 -0800 (PST)
>
> > gitk --merge
> ...
> > This is something where I actually think git could and should do better:
> > git has the capability to act as more of a "quilt replacement", but
> > because it wasn't part of the original design, we never actualy exposed
> > the simple queue management commands to do this (stgit does things like
> > that, though).
> >
> > So if you haven't pushed out, right now you'd have to do this stupid
> > thing:
>
> Thanks for all the useful info.
>
> But as soon as I've applied any patches to my tree I've "pushed out".
> So this scheme doesn't work for me. The first thing I do when I have
> changes to apply is clone a tree locally and on master.kernel.org,
> then I apply that first patch locally and push it out to master.
>
> What would be really cool is if you could do the rebase thing, push
> that to a remote tree you were already pushing into and others could
> pull from that and all the right things happen.
>
> A rebase is just a series of events, and those could propagate to
> people who are pulling from the tree. I'm pretty sure GIT could
> support this.
git checkout -b new-tree old-tree
# hack on new-tree: rebase, drop bad commits, whatever
git merge -s ours old-tree
git push your-public-repo new-tree:public-tree
Now public-tree has a merge commit on the top with two parents,
public-tree^1 and public-tree^2. public-tree^1 is the tip of the new
cleaned-up history, and public-tree^2 points to the old stuff.
I sometimes use that privately as a way to keep track of the history of
a patch-series, but I assume Linus would shoot anyone that tried to
submit such a monstrosity upstream....
--b.
After glancing at some of this thread its clear to me what Stephen's
real goal is:
1. collect kernel trees (or underpants)
2. ?
3. profit
http://en.wikipedia.org/wiki/Underpants_Gnomes
- k
On Tue, 12 Feb 2008, Linus Torvalds wrote:
> So let's say that you have a remote branch that you track that goes
> rebasing (let's call it "origin/pu" to match the real-life git behaviour),
> then you should literally be able to do
>
> old=$(git rev-parse origin/pu) &&
> git fetch &&
> new=$(git rev-parse origin/pu) &&
> git rebase --onto $new $old
>
> and no, I didn't actually test it, but hey, it really should be that
> simple.
Or, with Git 1.5.4, just:
git pull --rebase
And I didn't test it yet either. Same caveats do apply.
Nicolas
On Tue, 12 Feb 2008, Linus Torvalds wrote:
> Of course, if you didn't even want to save the old branch, just skip the
> first step. If you have reflogs enabled (and git does that by default in
> any half-way recent version), you can always find it again, even without
> having to do "git fsck --lost-found", at least as long as you don't delete
> that branch, and it hasn't gotten pruned away (kept around for the next 90
> days by default, iirc)
Even if you delete that branch, the "HEAD" reflog will still contain it,
since it is separate from any particular branch reflog.
Nicolas
On Tue, 12 Feb 2008, J. Bruce Fields wrote:
> > So as a result, some *random* commit that was actually fine on its own has
> > now become a bug, just because it was re-written.
>
> If there was a "fundamental thing that didn't cause a conflict", then
> the two trees in question probably didn't touch the same code, so would
> probably merge cleanly, for the same reason that one rebased onto the
> other cleanly. But depending on what the "fundamental thing" was, the
> merge might still introduce the same bug, right?
Absolutely. But if you do a true merge, the bug is clearly in the merge
(automatedly clean or not), and the blame is there too. IOW, you can blame
me for screwing up. Now, I will say "oh, me bad, I didn't realize how
subtle the interaction was", so it's not like I'll be all that contrite,
but at least it's obvious where the blame lies.
In contrast, when you rebase, the same problem happens, but now a totally
innocent commit is blamed just because it happened to no longer work in
the location it was not tested in. The person who wrote that commit, the
people who tested it and said it works, all that work is now basically
worthless: the testing was done with another version, the original patch
is bad, and the history and _reason_ for it being bad has been lost.
And there's literally nothing left to indicate the fact that the patch and
the testing _used_ to be perfectly valid.
That may not sound like such a big deal, but what does that make of code
review and tested-by, and the like? It just makes a mockery of trying to
do a good job testing any sub-trees, when you know that eventually it will
all quite possibly be pointless, and the fact that maybe the networking
tree was tested exhaustively is all totally moot, because in the end the
stuff that hit the main tree is something else altogether?
I don't know about you, but I'd personally be really disappointed if it
happened to me, and I felt that I did a really good job as a
submaintainer. I'd also feel that the source control management sucked.
Contrast that to the case where somebody simply does a merge error. The
original work doesn't lose it's validity - so the original maintainer
hasn't lost anything. And quite frankly, even the person who "screwed up"
with the merge hasn't really done anything bad: these things _do_ happen.
So bugs happen; not big deal. But the fact that the bugs are correctly
attributed - or rather, not mis-attributed to somebody blameless - that
_is_ a big deal.
It's not like I will guarantee that all my manual merges are always 100%
correct, much less try to guarantee that no subtle merge issue can make
things not work even if it all merged totally cleanly. That isn't my
point. And others will make merge mistakes too. But the people they merged
from will not be blamed.
So just the fact that the right commit gets blamed when somebody does a
"git bisect" is I think a big issue. It's just fundamentally more fair to
everybody. And it means that the people who push their work to me can
really choose to stand behind it, knowing that whatever happens, their
work won't get diluted by bad luck or others' incompetence.
And no, maybe most people don't feel things like that matters. But I do
think it's important.
Linus
On Tue, Feb 12, 2008 at 04:49:46PM -0800, Linus Torvalds wrote:
>
>
> On Tue, 12 Feb 2008, Greg KH wrote:
> >
> > Perhaps you need to switch to using quilt. This is the main reason why
> > I use it.
>
> Btw, on that note: if some quilt user can send an "annotated history file"
> of their quilt usage, it's something that git really can do, and I'll see
> if I can merge (or rather, coax Junio to merge) the relevant part of stgit
> to make it possible to just basically get "quilt behaviour" for the parts
> of a git tree that you haven't pushed out yet.
Ted's description matches mine (keep quilt tree in git, edit changelog
entries, rebase on newer kernel versions, etc.) I can go into details
if needed.
> A pure patch-stack will be faster at that thing than git would be (it's
> simply easier to just track patches), but on the other hand, using git
> would get some other advantages outside of the integration issue (eg the
> cherry-pick thing really is a proper three-way merge, not just an "apply
> patch", so it can do better).
I was amazed at how slow stgit was when I tried it out. I use
git-quiltimport a lot and I don't think it's any slower than just using
quilt on its own. So I think that the speed issue should be the same.
I had a number of issues last time I tried stgit out, but maybe they are
now resolved, I'll try it out tomorrow and report to the git list
anything I find that doesn't work for me.
thanks,
greg k-h
On Tue, 12 Feb 2008, Greg KH wrote:
> On Tue, Feb 12, 2008 at 04:49:46PM -0800, Linus Torvalds wrote:
> > On Tue, 12 Feb 2008, Greg KH wrote:
> > > Perhaps you need to switch to using quilt. This is the main reason why
> > > I use it.
> >
> > Btw, on that note: if some quilt user can send an "annotated history file"
> > of their quilt usage, it's something that git really can do, and I'll see
> > if I can merge (or rather, coax Junio to merge) the relevant part of stgit
> > to make it possible to just basically get "quilt behaviour" for the parts
> > of a git tree that you haven't pushed out yet.
>
> Ted's description matches mine (keep quilt tree in git, edit changelog
> entries, rebase on newer kernel versions, etc.) I can go into details
> if needed.
Ack. Same for PS3 and m68k (except I don't have the m68k patches in git (yet)).
Two issues with using quilt:
1. Sometimes a patch still applies after it was integrated upstream,
2. Sometimes it's a lot of work to fixup the patches after an upstream
change. I wrote a script to do a tree-way-merge in case of a
conflict, but I haven't really tested it yet (not suitable big
conflict has happened since ;-)
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
On Tue, 12 Feb 2008, Andrew Morton wrote:
> On Tue, 12 Feb 2008 16:41:49 -0800 (PST)
> David Miller <[email protected]> wrote:
>
> > Here are some odd-the-cuff
> > suggestions:
> >
> > 1) Make feature-removal-schedule a directory with files in it.
> > Everyone touches that file, creating merge issues.
> >
> > 2) Let's move away from some/dir/{Kconfig,Makefile} schemes and
> > instead have each "thing" have it's own Kconfig.foo or
> > Makefile.foo that gets automatically sucked into the main
> > directory Makefile or Kconfig using file globs or similar.
> >
> > Even better, encode the building of things into the *.[ch]
> > files themselves, and have the Kconfig/Makefile machinery
> > automatically extract this information when you build.
>
> 3) teach people that you don't always have to add new includes right
> at the end of the list
>
> 4) teach people that you don't have to add Makefile rules right at the
> end of the list
If the list is already sorted, I insert it at the right position.
If not, well...
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
>
> 2) Let's move away from some/dir/{Kconfig,Makefile} schemes and
> instead have each "thing" have it's own Kconfig.foo or
> Makefile.foo that gets automatically sucked into the main
> directory Makefile or Kconfig using file globs or similar.
So we could do:
config foo
tristate "do you want foo?"
depends on USB && BAR
module
obj-$(CONFIG_FOO) += foo.o
foo-y := file1.o file2.o
help
foo will allow you to explode your PC
The [module] part would go unedited into a kbuild file
located in same dir as the Kconfig file and included by kbuild.
That would solve the Makefile issue.
And we could introduce support for
source "drivers/net/Kconfig.*"
But then we would have to make the kconfig step mandatory
for each build as we would otherwise not know if there
were added any Kconfig files.
Does this fit what you had in mind?
Sam
From: Sam Ravnborg <[email protected]>
Date: Wed, 13 Feb 2008 09:45:41 +0100
> So we could do:
>
> config foo
> tristate "do you want foo?"
> depends on USB && BAR
> module
> obj-$(CONFIG_FOO) += foo.o
> foo-y := file1.o file2.o
> help
> foo will allow you to explode your PC
...
> Does this fit what you had in mind?
Yes it does.
Now I'll ask if you think embedding this information in one of the C
files for a module would be even nicer?
Also, we need to make sure we can properly handle top-level
container-like items. For example, where would menuconfigs like
NETDEV_10000 go if we adopt this kind of scheme?
On Wed, Feb 13, 2008 at 12:54:33AM -0800, David Miller wrote:
> From: Sam Ravnborg <[email protected]>
> Date: Wed, 13 Feb 2008 09:45:41 +0100
>
> > So we could do:
> >
> > config foo
> > tristate "do you want foo?"
> > depends on USB && BAR
> > module
> > obj-$(CONFIG_FOO) += foo.o
> > foo-y := file1.o file2.o
> > help
> > foo will allow you to explode your PC
> ...
> > Does this fit what you had in mind?
>
> Yes it does.
>
> Now I'll ask if you think embedding this information in one of the C
> files for a module would be even nicer?
I have no good idea for the syntax and I and not sure what is gained
by reducing a driver with one file.
Agreed - simple drivers would then be a single file - and thats a good argument.
>
> Also, we need to make sure we can properly handle top-level
> container-like items. For example, where would menuconfigs like
> NETDEV_10000 go if we adopt this kind of scheme?
If it makes sense to group stuff inside a menuconfig it would
also make sense to put the same modules in a subdirectory.
And then we would have the menuconfig in the Kconfig
file that would source the others.
So I do not see this as an issue for the 'embedded' syntax described above.
Sam
From: Sam Ravnborg <[email protected]>
Date: Wed, 13 Feb 2008 10:04:51 +0100
> On Wed, Feb 13, 2008 at 12:54:33AM -0800, David Miller wrote:
> > Also, we need to make sure we can properly handle top-level
> > container-like items. For example, where would menuconfigs like
> > NETDEV_10000 go if we adopt this kind of scheme?
>
> If it makes sense to group stuff inside a menuconfig it would
> also make sense to put the same modules in a subdirectory.
> And then we would have the menuconfig in the Kconfig
> file that would source the others.
Sounds good to me.
Sam Ravnborg wrote:
> On Wed, Feb 13, 2008 at 12:54:33AM -0800, David Miller wrote:
>> From: Sam Ravnborg <[email protected]>
>> Date: Wed, 13 Feb 2008 09:45:41 +0100
>>
>>> So we could do:
>>>
>>> config foo
>>> tristate "do you want foo?"
>>> depends on USB && BAR
>>> module
>>> obj-$(CONFIG_FOO) += foo.o
>>> foo-y := file1.o file2.o
>>> help
>>> foo will allow you to explode your PC
>> ...
>>> Does this fit what you had in mind?
>> Yes it does.
>>
>> Now I'll ask if you think embedding this information in one of the C
>> files for a module would be even nicer?
> I have no good idea for the syntax and I and not sure what is gained
> by reducing a driver with one file.
> Agreed - simple drivers would then be a single file - and thats a good argument.
I like the Sam proposal, but maybe we can simplify the rules on
"module" segment: some informations are often redundant, dependencies
are sometime calculated by config part and sometime by Makefile
(and sometime in the Makefile there are some wrong hacks).
I would really like a good section like:
module
foo : file1.o file2.o
and let the complex rules in the normal Makefile
(which is also good because the complex rules are often not
specific to a single driver).
But I don't like merging all info in a single file:
- not so clean in case of multiple source-file driver
- it would be more complex the "copy and paste"
from other drivers: most developers are not
comfortable with Kconfig and Makefile, so easy to grep
others Kconfig/Makefile could help developers not do
do strange/wrong hacks.
ciao
cate
On Tue, Feb 12, 2008 at 11:19:14AM -0800, Greg KH wrote:
> We usually get this warning today in -mm.
We don't always - and I'd say in terms of ARM it would be extremely rare.
The sysfs API changes at the start of the last merge window is one example
of this.
I had everything nicely prepared in the ARM tree for merging shortly
after the window opened, and then wham bam thankyou mam, the merge
window opened and a lot ended up breaking.
We know that the -mm tree is pretty much useless in terms of code
coverage for ARM, and it's getting increasingly unlikely that anything
short of a build of all ARM defconfigs will pick up on merge issues -
which is a lot of CPU cycles, and I'm not going to insist its something
that should be done.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of:
On Tue, Feb 12, 2008 at 09:51:52PM +0000, Alan Cox wrote:
> > We could simply decide that API changes affecting more than one subsystem
> > Must Be Serialized(tm). Explicitly. As in "any such change is posted
>
> Welcome to dreamland. The only way I can get serial changes done is to
> wait months and eventually simply persuade Andrew to ignore the
> "maintainers" who are basically not responding.
... because we didn't realise that the pre-requisits for those patches
had already gone in. If that had been the case, I'd have applied the
ARM changes for the new tty ioctls immediately.
It was a communication problem, not a process problem.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of:
On Tue, Feb 12, 2008 at 10:16:53PM -0800, Greg KH wrote:
> I was amazed at how slow stgit was when I tried it out. I use
> git-quiltimport a lot and I don't think it's any slower than just using
> quilt on its own. So I think that the speed issue should be the same.
I like using "guilt" because I can easily reapply the patchset using
"guilt push -a", which is just slightly fewer characters to type than
"git-quiltimport". This also means that I don't need to switch back
and forth between "git mode" and "quilt mode" when I'm editing the
patches (either directly by editing the patch files, in which case
afterwards I do a "guilt pop -a; guilt push -a", or by using "guilt
pop", "guilt push", and "guilt refresh").
"guilt push -a" is a little bit slower than "quilt push -a", but not
enough to be seriously annoying. And besides, "guilt pop -a" is
slightly faster than "quilt pop -a".
Using guilt is also nice because there is a bit of additional backup
for previous work via the git reflogs, although to be honest I've
rarely needed to use it.
- Ted
On Tue, Feb 12, 2008 at 12:50:51PM -0800, Greg KH wrote:
> I can run the numbers, but almost every one of those changes has at
> least 2 signed-off-by: on them, so they should all be being reviewed
> properly.
Good joke..
On Tue, 2008-02-12 at 21:16 -0500, Theodore Tso wrote:
> I've never been very happy with stgit because of past experiences
> which has scarred me when it got get confused and lost my entire patch
> series (this was before git reflogs, so recovery was.... interesting).
It got much better now :-). We are even working on transactions support
and, if something fails, it restores the state of the stack.
> There's always been something deeply comforting about having the ASCII
> patch series since it's easy to back it up and know you're in no
> danger of losing everything in case of a bug.
There is "stg export" which could be made to export a patch
automatically at every change.
> The other advantage of storing the patch stack as a an ASCII quilt
> series is we have a history of changes of the patches, which we don't
> necessarily have if you just use stgit to rewrite the patch.
As I said in a different e-mail, we got patch history tracking in StGIT.
You can even annotate specific changes.
For editing, there is "stg edit [--diff]" which allows both description
and patch changes.
However, the StGIT approach is not fully suited for multiple developers
sharing the same set of patches, especially if they are allowed to
modify the same patches. For this kind of workflow, you can export the
series and add the patches to a separate shared repository. StGIT has a
"sync" command to synchronise changes to patches in different series
(either stored as quilt series or in a different branch).
--
Catalin
On Tue, 2008-02-12 at 22:16 -0800, Greg KH wrote:
> Ted's description matches mine (keep quilt tree in git, edit changelog
> entries, rebase on newer kernel versions, etc.) I can go into details
> if needed.
I added some time ago patch history tracking in stgit and you can run
"stg log [--graphical] <patch>" to see all the changes for that patch
(as a list or via gitk). This is done by keeping a separate DAG of
commits linking small changes to a patch.
> I was amazed at how slow stgit was when I tried it out. I use
> git-quiltimport a lot and I don't think it's any slower than just using
> quilt on its own. So I think that the speed issue should be the same.
It shouldn't be slower than git-quiltimport (at least the recent stgit
versions) as they seem to have a similar approach (using "git apply").
There is probably an extra check stgit does for local changes before
starting the import. Otherwise, just use git-quiltimport and "stg
uncommit" to generate the patches.
StGIT approach for pushing patches is to use git-apply and, only if this
fails, switch to a three-way merge. These days it seems that the
three-way merge is pretty fast anyway, we might drop the former (after
some benchmarking).
> I had a number of issues last time I tried stgit out, but maybe they are
> now resolved, I'll try it out tomorrow and report to the git list
> anything I find that doesn't work for me.
Please try the last stable release, 0.14. The current HEAD has some
restructuring done (but gets nice features like transactions, undo).
Thanks.
--
Catalin
Russell King wrote:
> We know that the -mm tree is pretty much useless in terms of code
> coverage for ARM, and it's getting increasingly unlikely that anything
> short of a build of all ARM defconfigs will pick up on merge issues -
> which is a lot of CPU cycles, and I'm not going to insist its something
> that should be done.
If you need help in that area (build-testing ARM), surely we can find
some fast machines that can do builds for you. Any interest?
Jeff
On Wed, Feb 13, 2008 at 07:06:24AM -0500, Jeff Garzik wrote:
> Russell King wrote:
> >We know that the -mm tree is pretty much useless in terms of code
> >coverage for ARM, and it's getting increasingly unlikely that anything
> >short of a build of all ARM defconfigs will pick up on merge issues -
> >which is a lot of CPU cycles, and I'm not going to insist its something
> >that should be done.
>
>
> If you need help in that area (build-testing ARM), surely we can find
> some fast machines that can do builds for you. Any interest?
It would probably make some sense - it would help to get hold of the
infrastructure that armlinux.simtec.co.uk uses to run their test builds
of Linus' tree and apply it (on a separate machine) to -mm (or
indeed whatever other) kernels.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of:
Stephen Rothwell wrote:
> On Tue, 12 Feb 2008 12:02:08 +1100 Stephen Rothwell wrote:
>> Andrew was looking for someone to run a linux-next tree that just
>> contained the subsystem git and quilt trees for 2.6.x+1 and I (in a
>> moment of madness) volunteered.
>
> I neglected to mention the other brave souls who volunteered: Frank
> Seidel, Ann Davis, and Harvey Harrison who I hope to be able to call on
> in times of need.
Hey, I'm sure you just temporarily forgot the discussion about that point
we had the days before ;-) Never mind, just kidding! :-)
Lets get serious. I cannot speak for Ann and Harvey, but I'm quite sure they
also really hope - at least i very strongly do - you not only call on us when
things become a burden, but let us help and assist you right from the start.
Thanks,
Frank
On Tue, Feb 12, 2008 at 06:47:30PM -0800, Joel Becker wrote:
> Make the distinction earlier. With ocfs2 and configfs (we got
> this scheme from Jeff), we keep the topic branches as "unsafe" - that
> is, officially rebaseable . We merge them all into a big "ALL" branch,
> which is also "unsafe". Andrew pulls this for -mm, and it gets tested
> here. If there is a brown-paper-bag problem, we can tell the original
> author to fix it. Then we re-pull the topic - effectively a rebase.
> The ALL is also rebased. But that's Ok, it will never go towards Linus.
> When a topic is considered worthy of going upstream, we pull it
> to a branch called "upstream-linus". This branch is *NEVER* rebased.
> Now that the topic is in upstream-linus, the original topic branch can't
> be rebased either. So any fixes to that topic going forward will stay
> in the history. Since that topic was pulled into ALL for testing, we
> are using the identical commits that got tested.
This is essentially the same process I've been using in wireless-2.6
with the (regularly rebased) 'everything' branch. Still I find
that it causes lots of confusion and complaining. Perhaps I am not
communicating clearly enough... :-)
Do you find that people are happy with that process? Forgive me for
not knowing, but how many developers are actively (or occasionaly)
involved in ocfs2 and configfs? How many 'normal' users pull your
tree looking for 'latest and greatest' code?
John
--
John W. Linville
[email protected]
On Wed, Feb 13, 2008 at 10:06:16AM -0500, John W. Linville wrote:
> On Tue, Feb 12, 2008 at 06:47:30PM -0800, Joel Becker wrote:
> > Make the distinction earlier. With ocfs2 and configfs (we got
> > this scheme from Jeff), we keep the topic branches as "unsafe" - that
> > is, officially rebaseable . We merge them all into a big "ALL" branch,
> > which is also "unsafe". Andrew pulls this for -mm, and it gets tested
> > here. If there is a brown-paper-bag problem, we can tell the original
> > author to fix it. Then we re-pull the topic - effectively a rebase.
> > The ALL is also rebased. But that's Ok, it will never go towards Linus.
>
> This is essentially the same process I've been using in wireless-2.6
> with the (regularly rebased) 'everything' branch. Still I find
> that it causes lots of confusion and complaining. Perhaps I am not
> communicating clearly enough... :-)
>
> Do you find that people are happy with that process? Forgive me for
> not knowing, but how many developers are actively (or occasionaly)
> involved in ocfs2 and configfs? How many 'normal' users pull your
> tree looking for 'latest and greatest' code?
Oh, I have no pretense that ocfs2 is as busy as wireless or
net. Certainly Andrew has no problem with it. People I've interacted
with while working on features, etc, also get it, but I'm usually
pointing them to it up-front. I think you'd probably have lessons that
larger trees could learn from. For example, do you have a 'things that
will go to linus soon" branch that people can grab - something that's
now stable/non-rebasing? Do you advertise the rebaseableness of
"everything" in the web page where you say "grab 'everthing' for the
latest stuff"?
Joel
--
"The suffering man ought really to consume his own smoke; there is no
good in emitting smoke till you have made it into fire."
- thomas carlyle
Joel Becker
Principal Software Developer
Oracle
E-mail: [email protected]
Phone: (650) 506-8127
Linus Torvalds wrote:
>
> On Tue, 12 Feb 2008, Greg KH wrote:
>>> That's the point.
>> Not it isn't. To quote you a number of years ago:
>> "Linux is evolution, not intelligent design"
>
> Umm. Have you read a lot of books on evolution?
>
> It doesn't sound like you have.
>
> The fact is, evolution often does odd (and "suboptimal") things exactly
> because it does incremental changes that DO NOT BREAK at any point.
This is not entirely true if the pressure for changes are removed. For
instance in mammals the bones in the ear are what used to be gills in fish.
When fish became amphibians the gills weren't needed as much and evolution
took a side path.
> The examples are legion. The mammalian eye has the retina "backwards",
> with the blind spot appearing because the fundmanetal infrastructure (the
> optical nerves) actually being in *front* of the light sensor and needing
> a hole in the retina to get the information (and blood flow) to go to the
> brain!
>
> In other words, exactly *because* evolution requires "bisectability" (any
> non-viable point in between is a dead end by definition) and does things
> incrementally, it doesn't do big flips. It fixes the problems on an
> incremental scale both when it comes to the details and when it comes to
> both "details" (actual protein-coding genes that code directly for some
> expression) and "infrastructure" (homeobox and non-coding genes).
In nature there is a lot of duplication: several copies of genes can exist
and different copies may have a distinct evolution. There is also a lot of
'junk' DNA that doesn't code for anything (although it may have regulating
functions). In there some copies of genes may remain that are inactivated,
as well as parts of virusses, slowly obtaining random mutations because
there is no pressure on the evolution of them. Some may eventually become
active again and have different functions.
The duplication also often ensures there is fallback when random mutations
are acquired and a protein is knocked out. Besides the two chromosomes
several proteins also can have overlapping functions. The result is more
like a balance.
Evolution in nature and changes in code are different because in code junk
and bugs are constantly removed. In biology junk is allowed and may provide
a pool for future development. Linux development is intended and not
survival.
On Tue, Feb 12, 2008 at 12:50:51PM -0800, Greg KH wrote:
> On Tue, Feb 12, 2008 at 07:46:51PM +0000, Al Viro wrote:
>...
> > AFAICS, we are in situation when review bandwidth is where the bottleneck
> > is. Not the merge one...
>
> Are there still large numbers of posted patches, not reviewed or picked
> up by anyone laying around somewhere? I thought Andrew-the-patch-vacuum
> had been doing a great job of keeping that from happening lately.
I wrote in one of the many bug reports I sent for compile errors
introduced in Linus' tree during this merge window:
An architecture specific patch that breaks the one architecture it
touches at the first file being compiled is even for kernel standards
unusually bad...
> thanks,
>
> greg k-h
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
On Tue, Feb 12, 2008 at 09:43:10PM -0800, Linus Torvalds wrote:
> So just the fact that the right commit gets blamed when somebody does a
> "git bisect" is I think a big issue. It's just fundamentally more fair to
> everybody. And it means that the people who push their work to me can
> really choose to stand behind it, knowing that whatever happens, their
> work won't get diluted by bad luck or others' incompetence.
>
> And no, maybe most people don't feel things like that matters. But I do
> think it's important.
The obvious advantage to rebasing in this case is that the blame
(misplaced though it may be), at least lands on a commit that made a
single small change, likely making the problem easier to diagnose.
(As opposed to the case of a large merge, where all you may know is that
somewhere in the hundreds of commits done on one side of the merge there
was a conflict with the hundreds of commits on the other side.)
I think a lot of people would see rebasing as an acceptable tradeof that
gives up a small amount of accuracy in assigning blame to individuals in
return for a large increase in ability to debug problems.
I suppose one response to that would be that it's important that people
learn how to work in parallel, that failures to do so are particularly
important failures in the process, and that it's therefore worth it to
make sure that such failures are always identified specifically as merge
failures.
It would be nice if merges, like patches, were broken up into somewhat
smaller units. There's an understandable desire to wait to the last
minute to actually commit to one's commits, but a willingness to do so a
little earlier might avoid some of the problems that seem to come from
having a lot of large merges happen all at once.
--b.
On Tue, Feb 12, 2008 at 09:09:34AM -0800, Linus Torvalds wrote:
>...
> The other is that once somebody says "ok, I *really* need to cause this
> breakage, because there's a major bug or we need it for fundamental reason
> XYZ", then that person should
>
> (a) create a base tree with _just_ that fundamental infrastructure change,
> and make sure that base branch is so obviously good that there is no
> question about merging it.
>
> (b) tell other people about the reason for the infrastructure change, and
> simply allow others to merge it. You don't have to wait for *me* to
> open the merge window, you need to make sure that the people that get
> impacted most can continue development!
>
> This is where "rebases really are bad" comes in. When the above sequence
> happens, the fundamental infrastructure change obviously does need to be
> solid and not shifting under from other people who end up merging it. I do
> not want to see five different copies of the fundamental change either
> because the original source fixed it up and rebased it, or because the
> people who merged it rebased _their_ trees and rebased the fundamental
> change in the process.
>
> Can that (b) be my tree? Sure. That's been the common case, and I'll
> happily continue it, of course, so I'm not arguing for that to go away.
> Merging is my job, I'll do it. But when the merge window is a problem, my
> merge window should *not* hold up people from using the distributed nature
> of git for their advantage.
>
> But yes, obviously when doing cross-merges, you'd better be really
> *really* sure that the base is solid and will get merged. But let's face
> it, all the really core maintainers should damn well know that by now:
> you've all worked with me for years, so you should be able to trivially be
> able to tell whether you *might* need to worry about something, and when
> it's a slam dunk.
>
> And it's the "it's a slam dunk" cases that I think are (a) the common ones
> and (b) the ones where you can just do cross-merges to satisfy each others
> needs.
>
> Hmm? Does that sound palatable to people?
I'm sure I understand only less than half of what you said.
My current understanding is that all the aspects of your proposal that
are interesting for me can be summarized as follows:
- your tree stops compiling at the beginning of the merge window when
you merge the first subsystem tree
- your tree might compile again at the end of the merge window when you
merge the last subsystem tree
- git-bisect -> /dev/null
What do I miss?
> Linus
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
On Wed, Feb 13, 2008 at 05:36:41AM -0500, Theodore Tso wrote:
> On Tue, Feb 12, 2008 at 10:16:53PM -0800, Greg KH wrote:
> > I was amazed at how slow stgit was when I tried it out. I use
> > git-quiltimport a lot and I don't think it's any slower than just using
> > quilt on its own. So I think that the speed issue should be the same.
>
> I like using "guilt" because I can easily reapply the patchset using
> "guilt push -a", which is just slightly fewer characters to type than
> "git-quiltimport". This also means that I don't need to switch back
> and forth between "git mode" and "quilt mode" when I'm editing the
> patches (either directly by editing the patch files, in which case
> afterwards I do a "guilt pop -a; guilt push -a", or by using "guilt
> pop", "guilt push", and "guilt refresh").
I had problems getting guilt to preserve metadata properly last time I
tried it, and it forced me to work with the same location and format
that the original developers used, which wasn't as flexable as quilt
could handle from what I recall.
But I'll try it again, it might have gotten better...
thanks,
greg k-h
On Wed, 13 Feb 2008, Roel Kluin wrote:
>
> In nature there is a lot of duplication: several copies of genes can exist
> and different copies may have a distinct evolution.
This is true of very complex animals, but much less so when looking at
things like bacteria (and arguably, any current sw project is closer to
bacteria in complexity than anything mammalian).
In bacteria (and viruses), duplication of DNA/RNA is a big cost of living
in general, and as a result there is *much* less junk DNA. So in an
evolutionary sense, it's much closer to what the kernel should have (with
occasional duplication of code and interfaces to allow new functionality,
but rather aggressive pruning of the excess baggage).
In other words, all of these choices are a matter of "balance". In some
areas, excess code is not a sufficient downside, and we keep even broken
source code around with no actual function, "just because" (or rather,
because the cost of carrying it around is so small that nobody cares).
That's true in the kernel as in biology: check out not just deprecated
code, but the drivers and other odds-and-ends that are explicitly marked
as non-coding DNA (we just happen to call them BROKEN in our Kconfig).
Linus
> Evolution in nature and changes in code are different because in code junk
> and bugs are constantly removed. In biology junk is allowed and may provide
> a pool for future development. Linux development is intended and not
> survival.
I would be interested to see any evidence (rather than intuition) to
support that given that both appear to be the same kind of structure and
that structure is nowdays fairly well understood.
Alan
Frank Seidel wrote:
>
> Lets get serious. I cannot speak for Ann and Harvey, but I'm quite sure they
> also really hope - at least i very strongly do - you not only call on us when
> things become a burden, but let us help and assist you right from the start.
>
>
Agreed. I'm happy to do daily builds (I have x86 and ppc systems
available), write scripts, etc. In the meantime I'll just lurk and
learn. :-)
Thx,
Ann
On Wed, Feb 13, 2008 at 01:24:41PM -0700, Ann Davis wrote:
> Frank Seidel wrote:
>>
>> Lets get serious. I cannot speak for Ann and Harvey, but I'm quite sure they
>> also really hope - at least i very strongly do - you not only call on us when
>> things become a burden, but let us help and assist you right from the start.
>
> Agreed. I'm happy to do daily builds (I have x86 and ppc systems
> available),
>...
Jan already does builds of all -git (and previously -bk) and -mm kernels
for all architectures for some years, and it might be easier if he
simply adds -next to his builds.
> Thx,
>
> Ann
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Hi,
On Wednesday 13. February 2008, Sam Ravnborg wrote:
> config foo
> tristate "do you want foo?"
> depends on USB && BAR
> module
> obj-$(CONFIG_FOO) += foo.o
> foo-y := file1.o file2.o
> help
> foo will allow you to explode your PC
I'm more thinking about something like this:
module foo [FOO]
tristate "do you want foo?"
depends on USB && BAR
source file1.c
source file2.c if BAZ
Avoiding direct Makefile fragments would give us far more flexibility in the
final Makefile output.
> And we could introduce support for
>
> source "drivers/net/Kconfig.*"
>
> But then we would have to make the kconfig step mandatory
> for each build as we would otherwise not know if there
> were added any Kconfig files.
That's a real problem and it would be a step back of what we have right now,
so I'm not exactly comfortable with it.
bye, Roman
On Tue, Feb 12, 2008 at 10:57:16PM +1100, Stephen Rothwell wrote:
> We need to ask Linus to promise that he will pull the stable branch from
> linux-next first in the merge window. For that to happen, I would expect
> that Linus would also review and sign off (or ack) these commits to the
> linux-next tree.
Changing the commits in git in anyway changes their ID, which has the
same effects as a rebase.
With this idea, Linus only has two choices:
1. pull the entire set of linux-next changes whether he likes them or not,
because he's going to get them either from the linux-next tree or someone
elses tree which is based upon that.
2. don't pull the changes, nor anyone elses tree if he hates the changes
in linux-next.
So really, Linus needs to ack the changes _before_ they go into linux-next.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of:
On Thu, 14 Feb 2008, Roman Zippel wrote:
> On Wednesday 13. February 2008, Sam Ravnborg wrote:
> > config foo
> > tristate "do you want foo?"
> > depends on USB && BAR
> > module
> > obj-$(CONFIG_FOO) += foo.o
> > foo-y := file1.o file2.o
> > help
> > foo will allow you to explode your PC
>
> I'm more thinking about something like this:
>
> module foo [FOO]
> tristate "do you want foo?"
> depends on USB && BAR
> source file1.c
> source file2.c if BAZ
And we can finally distinguish between
config bar
bool "do you want bar?"
for boolean options and
module baz
bool "do you want baz?"
for modules that cannot be modular?
And we can make `depends on' do the right thing for dependencies on modules
that are modular (cfr. e.g. commit e11a6c236b3070ed05b079f91a9b3defa48b54d3,
[VIDEO]: XVR500 and XVR2500 require FB=y)?
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Hi Russell,
On Thu, 14 Feb 2008 08:14:05 +0000 Russell King <[email protected]> wrote:
>
> On Tue, Feb 12, 2008 at 10:57:16PM +1100, Stephen Rothwell wrote:
> > We need to ask Linus to promise that he will pull the stable branch from
> > linux-next first in the merge window. For that to happen, I would expect
> > that Linus would also review and sign off (or ack) these commits to the
> > linux-next tree.
>
> Changing the commits in git in anyway changes their ID, which has the
> same effects as a rebase.
Correct.
> With this idea, Linus only has two choices:
>
> 1. pull the entire set of linux-next changes whether he likes them or not,
> because he's going to get them either from the linux-next tree or someone
> elses tree which is based upon that.
>
> 2. don't pull the changes, nor anyone elses tree if he hates the changes
> in linux-next.
>
> So really, Linus needs to ack the changes _before_ they go into linux-next.
This is exactly what I suggested (or meant to) *except* that I want it to
only apply to the stable branch. I intend that the stable branch of
linux-next will never be rebased and so is suitable for others to base
their trees off. The master branch will be continually rebased as the
subsystem trees change over time.
Originally, I assumed the stable branch would be for our "usual" API
changes, but it appears we are not having any more of those. :-) However,
I see an argument for attempting to stabilise possible conflicting
changes get Linus' review/ack and add them to the stable branch.
Linus suggested that such changes should go into an independent tree that
everyone could pull into their trees with the full confidence that that
tree would be merged into Linus' tree when the merge window opens. I am
suggesting that that tree be the stable branch of linux-next.
I know I haven't thought through all the consequences of this, so
discussion is encouaged.
--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/
On Feb. 13, 2008, 19:52 +0200, "J. Bruce Fields" <[email protected]> wrote:
> On Tue, Feb 12, 2008 at 09:43:10PM -0800, Linus Torvalds wrote:
>> So just the fact that the right commit gets blamed when somebody does a
>> "git bisect" is I think a big issue. It's just fundamentally more fair to
>> everybody. And it means that the people who push their work to me can
>> really choose to stand behind it, knowing that whatever happens, their
>> work won't get diluted by bad luck or others' incompetence.
>>
>> And no, maybe most people don't feel things like that matters. But I do
>> think it's important.
>
> The obvious advantage to rebasing in this case is that the blame
> (misplaced though it may be), at least lands on a commit that made a
> single small change, likely making the problem easier to diagnose.
>
> (As opposed to the case of a large merge, where all you may know is that
> somewhere in the hundreds of commits done on one side of the merge there
> was a conflict with the hundreds of commits on the other side.)
>
> I think a lot of people would see rebasing as an acceptable tradeof that
> gives up a small amount of accuracy in assigning blame to individuals in
> return for a large increase in ability to debug problems.
>
> I suppose one response to that would be that it's important that people
> learn how to work in parallel, that failures to do so are particularly
> important failures in the process, and that it's therefore worth it to
> make sure that such failures are always identified specifically as merge
> failures.
>
> It would be nice if merges, like patches, were broken up into somewhat
> smaller units. There's an understandable desire to wait to the last
> minute to actually commit to one's commits, but a willingness to do so a
> little earlier might avoid some of the problems that seem to come from
> having a lot of large merges happen all at once.
>
> --b.
One idea that I thought about when debating rebase vs. merge (and it's
far far from being fully baked) is versioned commits. The gist of it
is that patches are assigned an hash identifier like today when they are
first committed into the tree, but, and this is the main change: if they mutate,
e.g. by a rebase, or even git commit --amend, their version is bumped up rather
than SHA changed.
This way all versions of the commit would be accessible and addressable using
their respective SHA *and* version. I think that this can help keep
the tree's history in a more intuitive way (since the patches' base identifier
won't change, just its version number), and you get a bonus of seeing each
commit's history, who changed it, and what was the change.
Benny
On Thu, 14 Feb 2008, Stephen Rothwell wrote:
>
> Originally, I assumed the stable branch would be for our "usual" API
> changes, but it appears we are not having any more of those. :-)
It's not that we should _never_ have them, it's that they shouldn't be
"business as usual".
I'm happy with them being a "a couple of times a year". I'm not happy with
them being "once or twice for every release cycle". That's the big deal
for me.
If we have a big flag-day that affects a lot of drivers (or architectures)
once or twice a year, I think everybody involved will be happy to stand up
and say "ok, that fixes problem X, and the new thing really is better, so
let's do it, it's worth it".
But if it's something that happens essentially every single release, that
is somethign else altogether. Then it's not a "ok, let's bite the bullet
and make the kernel better" thing any more, but instead it devolves into
"f*ck, the merge window is open again, now I have to fix up all the crap
people pushed on me".
See? That's a *huge* difference, even if it is "only" a mental one (and
clearly it isn't - there's the actual real work of the "I have to fix
things up" part too).
So to recap: I have absolutely nothing against fixing up bad internal
API's and breaking things. But 99% of the time that should be something we
can do incrementally (ie introduce the new API, and simply accept the
fact that removing the old API will take a few months). And the case when
that _really_ doesn't work should be rare enough that it doesn't wear
people down.
Because if you listen to the tone of people in this discussion, much of it
is about people being _tired_ of having to fix things up. It's not exactly
been a "wow, the end result sure was nice!" kind of discussion, is it?
And this is where "process" really matters. Making sure people don't get
too frustrated about the constant grind.
> However,
> I see an argument for attempting to stabilise possible conflicting
> changes get Linus' review/ack and add them to the stable branch.
>
> Linus suggested that such changes should go into an independent tree that
> everyone could pull into their trees with the full confidence that that
> tree would be merged into Linus' tree when the merge window opens. I am
> suggesting that that tree be the stable branch of linux-next.
I absolutely have no problem with having a "this is the infrastrcture
changes that will go into the next release". In fact, I can even
*maintain* such a branch.
I've not wanted to open up a second branch for "this is for next release",
because quite frankly, one of the other problems we have is that people
already spend way too much time on the next release compared to just
looking at regressions in the current one. But especially if we're talking
about _purely_ API changes etc infrastructure, I could certainly do a
"next" branch.
Linus
On Thursday 14 February 2008, Linus Torvalds wrote:
[...]
>And this is where "process" really matters. Making sure people don't get
>too frustrated about the constant grind.
One of the problems caused by this 'grind' is being locked out of using 3rd
party closed drivers until the vendor decides its stable enough to make the
effort to update their binary blobs to match the newer functions.
Nvidia vs 2.6.25-rc1 being a case in point, and they (nvidia) are appearing to
indicate its not a problem until some distro actually ships a kernel with the
changes that broke it. That could be months or even a year plus.
So you've lost one 'canary in the coal mine' tester at least until that
happens as I don't have a spare box I can setup to run the nv driver, which
itself seems to be suffering from bit rot recently and cannot run this screen
at its native 1680x1050 resolution, reverting to something that resembles
what I used to get from a timex 1000 in 1978 but with a few colors. I just
recently had to install the nvidia driver on my milling machines kubuntu 6.06
box cuz an xorg update put it back to 640x400 if the nv driver was used.
This is the real world, where politics aside, it just has to work... :-(
But I'll still be lurking and building to test anyway even if I don't boot it
for more than 10 minutes. :)
--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
I'm not a lawyer. I don't even play one on TV.
- Linus Torvalds on the gcc mailing list
On Thu, Feb 14, 2008 at 01:32:02PM -0500, Gene Heskett wrote:
> On Thursday 14 February 2008, Linus Torvalds wrote:
> [...]
> >And this is where "process" really matters. Making sure people don't get
> >too frustrated about the constant grind.
>
> One of the problems caused by this 'grind' is being locked out of using 3rd
> party closed drivers until the vendor decides its stable enough to make the
> effort to update their binary blobs to match the newer functions.
>
> Nvidia vs 2.6.25-rc1 being a case in point, and they (nvidia) are appearing to
> indicate its not a problem until some distro actually ships a kernel with the
> changes that broke it. That could be months or even a year plus.
How about "weeks". Both Fedora and openSUSE's next release is going to
be based on 2.6.25, and the first round of -rc1 kernels should be
showing up in their trees in a few days. So for this instance, I think
you will be fine :)
thanks,
greg k-h
On Thursday 14 February 2008, Greg KH wrote:
>On Thu, Feb 14, 2008 at 01:32:02PM -0500, Gene Heskett wrote:
>> On Thursday 14 February 2008, Linus Torvalds wrote:
>> [...]
>>
>> >And this is where "process" really matters. Making sure people don't get
>> >too frustrated about the constant grind.
>>
>> One of the problems caused by this 'grind' is being locked out of using
>> 3rd party closed drivers until the vendor decides its stable enough to
>> make the effort to update their binary blobs to match the newer functions.
>>
>> Nvidia vs 2.6.25-rc1 being a case in point, and they (nvidia) are
>> appearing to indicate its not a problem until some distro actually ships a
>> kernel with the changes that broke it. That could be months or even a
>> year plus.
>
>How about "weeks". Both Fedora and openSUSE's next release is going to
>be based on 2.6.25, and the first round of -rc1 kernels should be
>showing up in their trees in a few days. So for this instance, I think
>you will be fine :)
>
>thanks,
That is good news, thanks Greg.
>
>greg k-h
--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
There's small choice in rotten apples.
-- William Shakespeare, "The Taming of the Shrew"
On Thu, Feb 14, 2008 at 01:56:13AM +0100, Roman Zippel wrote:
> Hi,
>
> On Wednesday 13. February 2008, Sam Ravnborg wrote:
>
> > config foo
> > tristate "do you want foo?"
> > depends on USB && BAR
> > module
> > obj-$(CONFIG_FOO) += foo.o
> > foo-y := file1.o file2.o
> > help
> > foo will allow you to explode your PC
>
> I'm more thinking about something like this:
>
> module foo [FOO]
> tristate "do you want foo?"
> depends on USB && BAR
> source file1.c
> source file2.c if BAZ
>
> Avoiding direct Makefile fragments would give us far more flexibility in the
> final Makefile output.
Much better and now I see it I recall you posted something
along these lines before.
Is this something that you plan to look into implementing?
I can do the kbuild bits but I need you to do the kconfig
stuff (which is by far the biggest effort too).
It would be much appreciated to get this.
Sam
> The first things I need from the subsystem maintainers (you know who you
> are) are a contact address (a list address is fine) and at least one git
> branch or quilt series that contains all the things you want to see go
> into 2.6.26.
For InfiniBand/RDMA, the tree is:
master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-next
or via git protocol:
git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-next
contact addresses (me plus a mailing list):
[email protected]
[email protected]
thanks!
Hi Roland,
On Thu, 14 Feb 2008 15:22:46 -0800 Roland Dreier <[email protected]> wrote:
>
> For InfiniBand/RDMA, the tree is:
>
> master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-next
>
> or via git protocol:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-next
>
> contact addresses (me plus a mailing list):
>
> [email protected]
> [email protected]
Added, thanks.
--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/
* Linus Torvalds <[email protected]> wrote:
> On Thu, 14 Feb 2008, Stephen Rothwell wrote:
> >
> > Originally, I assumed the stable branch would be for our "usual" API
> > changes, but it appears we are not having any more of those. :-)
>
> It's not that we should _never_ have them, it's that they shouldn't be
> "business as usual".
>
> I'm happy with them being a "a couple of times a year". I'm not happy
> with them being "once or twice for every release cycle". That's the
> big deal for me.
very much agreed. I've yet to see a _single_ wide-scale API change that
broke stuff left and right where that breakage was technically
justified. I have not seen a single one.
All those cases were just plain old botched attempts. Either someone can
do a large-scale API change like the irq_regs() cleanups with near-zero
breakages, or someone cannot. In the latter case, gradual introduction
and trickling it through subsystem trees is a _must_.
and if it's _hard_ to do a particular large-scale change, then i think
the right answer is to _not do it_ in a large-scale way, but do it
gradually.
I claim that there's just not a single valid case of doing wide-scale
changes atomically and departing from the current to-be-stabilized
kernel tree materially. _Every_ large-scale API change can be done in a
staged way, with each subsystem adopting to the change at their own
pace, it just has to be planned well and tested well enough and has to
be executed persistently. And the moment we trickle things through
subsystem trees, there's no integration pain, as subsystem trees are
largely disjunct anyway.
i also fear that having an API-changes-only tree will dillute our
testing effort of the current to-be-stabilized upstream tree, as it
materially disrupts the flow of patches. Most maintainers should
concentrate on stabilizing current -git with only one serial queue of
fixes and enhancements ontop of that tree. I dont see how having a
second queue would help - it clearly splits attention.
widescale API changes should be discouraged, and forcing them through
the harder, "gradual, per subsystem" route is _exactly_ such a strong
force that discourages people from doing them.
Ingo
On Thu, 14 Feb 2008 13:32:02 EST, Gene Heskett said:
> Nvidia vs 2.6.25-rc1 being a case in point, and they (nvidia) are appearing to
> indicate its not a problem until some distro actually ships a kernel with the
> changes that broke it. That could be months or even a year plus.
Actually following the NVidia forums indicates otherwise:
http://www.nvnews.net/vbulletin/showthread.php?t=107144
I expect Zander will be posting a patch rather soonish, for some value of
soonish. And if you're running a -rc or -mm kernel, patching the 169.09
drivers should be well within your abilities....
On Thu, 14 Feb 2008 12:32:29 PST, Greg KH said:
> How about "weeks". Both Fedora and openSUSE's next release is going to
> be based on 2.6.25, and the first round of -rc1 kernels should be
> showing up in their trees in a few days. So for this instance, I think
> you will be fine :)
a few days == *NOW*
kernel-2.6.25-0.40.rc1.git2.fc9.x86_64 is in Fedora Rawhide already.
On Friday 15 February 2008, [email protected] wrote:
>On Thu, 14 Feb 2008 13:32:02 EST, Gene Heskett said:
>> Nvidia vs 2.6.25-rc1 being a case in point, and they (nvidia) are
>> appearing to indicate its not a problem until some distro actually ships a
>> kernel with the changes that broke it. That could be months or even a
>> year plus.
>
>Actually following the NVidia forums indicates otherwise:
>
>http://www.nvnews.net/vbulletin/showthread.php?t=107144
>
>I expect Zander will be posting a patch rather soonish, for some value of
>soonish. And if you're running a -rc or -mm kernel, patching the 169.09
>drivers should be well within your abilities....
Not so for the binaries, existing patches do make it compile but it still
upchucks someplace in the binary, or was this time yesterday.
--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
It is not well to be thought of as one who meekly submits to insolence and
intimidation.
On Fri, 15 Feb 2008 04:26:45 EST, Gene Heskett said:
> On Friday 15 February 2008, [email protected] wrote:
> >On Thu, 14 Feb 2008 13:32:02 EST, Gene Heskett said:
> >> Nvidia vs 2.6.25-rc1 being a case in point, and they (nvidia) are
> >> appearing to indicate its not a problem until some distro actually ships a
> >> kernel with the changes that broke it. That could be months or even a
> >> year plus.
> >
> >Actually following the NVidia forums indicates otherwise:
> >
> >http://www.nvnews.net/vbulletin/showthread.php?t=107144
> >
> >I expect Zander will be posting a patch rather soonish, for some value of
> >soonish. And if you're running a -rc or -mm kernel, patching the 169.09
> >drivers should be well within your abilities....
>
> Not so for the binaries, existing patches do make it compile but it still
> upchucks someplace in the binary, or was this time yesterday.
Umm.. if you actually *read* the mentioned thread, you'll see that "existing
patches" are known to be incomplete, complete with the "upchucks in the binary"
(mentioned at entry number 9 of the thread), and that Zander already knows
about it (entry #11), and has apparently one remaining issue left to resolve
(entries #32 and #36).
And entry #36 is what the NVidia engineer doing the work was thinking about
20 hours ago. Interpret it as you will...
On Thu, Feb 14, 2008 at 07:35:03PM +0200, Benny Halevy wrote:
> One idea that I thought about when debating rebase vs. merge (and it's
> far far from being fully baked) is versioned commits. The gist of it
> is that patches are assigned an hash identifier like today when they
> are first committed into the tree, but, and this is the main change:
> if they mutate, e.g. by a rebase, or even git commit --amend, their
> version is bumped up rather than SHA changed.
The SHA1 is uniquely determined by the contents of that commit--commit
and author names and times, changelog message, snapshot of the tree at
that point, and parents--hence, recursively, by the entire history
leading up to that commit.
Naming objects by their content in that way has a lot of advantages--for
example, you can sign an entire history by signing just the SHA1 of the
commit at its tip. So you can't break that link between the names of
commits and their contents without ending up with a fundamentally
different (and probably weaker, in some sense), system.
I suspect there's an unavoidable tradeoff--if you want to be able to
reliably and efficiently determine how two branches are related, then
you can't just throw away their (possibly messy) history. The best you
may be able to do, if you want the advantages of both rebasing and
merging, is to maintain on your own the messier meta-history as a
superset of the simpler history that you end up submitting.
--b.
> This way all versions of the commit would be accessible and addressable using
> their respective SHA *and* version. I think that this can help keep
> the tree's history in a more intuitive way (since the patches' base identifier
> won't change, just its version number), and you get a bonus of seeing each
> commit's history, who changed it, and what was the change.
Linus Torvalds wrote:
>
> On Wed, 13 Feb 2008, Roel Kluin wrote:
>> In nature there is a lot of duplication: several copies of genes can exist
>> and different copies may have a distinct evolution.
>
> This is true of very complex animals, but much less so when looking at
> things like bacteria (and arguably, any current sw project is closer to
> bacteria in complexity than anything mammalian).
>
> In bacteria (and viruses), duplication of DNA/RNA is a big cost of living
> in general, and as a result there is *much* less junk DNA. So in an
> evolutionary sense, it's much closer to what the kernel should have (with
> occasional duplication of code and interfaces to allow new functionality,
> but rather aggressive pruning of the excess baggage).
I like the comparison, and while I wrote my comment I have to admit I was
also thinking of bacteria and virusses as an exception: There the speed of
replication can be an important factor for survival. Less DNA means faster
replication and therefore the pressure is on removal of junk DNA. It can
be disputed however that removal of 'junk sourcecode' is a survival factor
for the linux kernel but the benefit may be disputable as well.
> In other words, all of these choices are a matter of "balance". In some
> areas, excess code is not a sufficient downside, and we keep even broken
> source code around with no actual function, "just because" (or rather,
> because the cost of carrying it around is so small that nobody cares).
>
> That's true in the kernel as in biology: check out not just deprecated
> code, but the drivers and other odds-and-ends that are explicitly marked
> as non-coding DNA (we just happen to call them BROKEN in our Kconfig).
>
> Linus
Maybe we can elaborate a bit on this comparison, just for fun:
I think not the linux kernel alone, but rather the entire Linux OS could
be compared with a cell. The kernel source encodes vital software parts
including the interactions with hardware - the environment for the
computer. Gcc can be compared with the (transcription and) translation
machinery. DNA can be seen as a language that encodes proteins, with
biological functions: Some are vital, including ones that allow
interactions with the environment: The cellular environment is beyond
the membrane. Interactions occur through membrane receptors, channels,
etc.
Interaction between proteins can be compared with functions selectively
calling other functions. Activation of certain proteins can cause a
cascade of protein interactions, comparable with function calls in a
loop: the activated protein activates particular protein(s) several
times. Some proteins influence intracellular messengers: cellular
global variables.
Transmembrane receptors responding to extracellular signals transmit
this through conformational changes across the membrane to the
intracellular region: These structural changes may allow interactions
with new proteins. Maybe comparable with a combination of hardware
interrupts, signals and the userspace?
The response to extracellular signals often depends on several
sequential interactions between proteins. This provides a protective
layer that can be compared with the kernelspace layer. This is where
the comparison probably becomes too biased to continue.
Alan Cox wrote:
>> Evolution in nature and changes in code are different because in code junk
>> and bugs are constantly removed. In biology junk is allowed and may provide
>> a pool for future development. Linux development is intended and not
>> survival.
>
> I would be interested to see any evidence (rather than intuition) to
> support that, given that both appear to be the same kind of structure and
> that structure is nowadays fairly well understood.
What do you mean with structure, the evolution? that both are a language?
On Tue, Feb 12, 2008 at 12:48:13PM -0800, Greg KH wrote:
> I have tried, and successfully done this many times in the past. The
> kobject change was one example: add a new function, migrate all users of
> a direct pointer over to that function, after that work is all done and
> in, change the structure and do the needed work afterward. All is
> bisectable completly, with no big "flag day" needed.
Incorrect - because this all happened far too quickly. This is one of
the reasons that I ended up having to redo various parts of the ARM tree
because stuff broke - set_kset_name() completely vanished introducing
compile errors, and iirc some merge issues as well.
I had patches introducing new system objects which use that, and
modifications extremely close to other uses in the PXA code.
The end result (through rebuilding the affected parts of my git tree, and
asking people for replacement patches) was something that is bisectable -
but had I tried to merge stuff as is, it would've been an utter mess, and
_was_ unbuildable.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of:
On Fri, 15 Feb 2008 23:23:08 +0000
Russell King <[email protected]> wrote:
> On Tue, Feb 12, 2008 at 12:48:13PM -0800, Greg KH wrote:
> > I have tried, and successfully done this many times in the past. The
> > kobject change was one example: add a new function, migrate all users of
> > a direct pointer over to that function, after that work is all done and
> > in, change the structure and do the needed work afterward. All is
> > bisectable completly, with no big "flag day" needed.
>
> Incorrect - because this all happened far too quickly. This is one of
> the reasons that I ended up having to redo various parts of the ARM tree
> because stuff broke - set_kset_name() completely vanished introducing
> compile errors, and iirc some merge issues as well.
>
> I had patches introducing new system objects which use that, and
> modifications extremely close to other uses in the PXA code.
>
> The end result (through rebuilding the affected parts of my git tree, and
> asking people for replacement patches) was something that is bisectable -
> but had I tried to merge stuff as is, it would've been an utter mess, and
> _was_ unbuildable.
>
I wonder why I didn't see any of this - I build arm allmodconfig at least
once a week, usually more frequently.
So either the offending patches weren't in my pile or arm allmodconfig is
worse than I thought :(
It really is in arch maintainers' best interest to keep their allmodconfig
in good shape, for this reason. arm's _isn't_ in good shape: the compile
fails for several long-standing reasons (eg: no hope of building DRM) and I
don't think the coverage is very broad either.
On Fri, 15 Feb 2008 15:37:32 -0800 Andrew Morton wrote:
> On Fri, 15 Feb 2008 23:23:08 +0000
> Russell King <[email protected]> wrote:
>
> > On Tue, Feb 12, 2008 at 12:48:13PM -0800, Greg KH wrote:
> > > I have tried, and successfully done this many times in the past. The
> > > kobject change was one example: add a new function, migrate all users of
> > > a direct pointer over to that function, after that work is all done and
> > > in, change the structure and do the needed work afterward. All is
> > > bisectable completly, with no big "flag day" needed.
> >
> > Incorrect - because this all happened far too quickly. This is one of
> > the reasons that I ended up having to redo various parts of the ARM tree
> > because stuff broke - set_kset_name() completely vanished introducing
> > compile errors, and iirc some merge issues as well.
> >
> > I had patches introducing new system objects which use that, and
> > modifications extremely close to other uses in the PXA code.
> >
> > The end result (through rebuilding the affected parts of my git tree, and
> > asking people for replacement patches) was something that is bisectable -
> > but had I tried to merge stuff as is, it would've been an utter mess, and
> > _was_ unbuildable.
> >
>
> I wonder why I didn't see any of this - I build arm allmodconfig at least
> once a week, usually more frequently.
>
> So either the offending patches weren't in my pile or arm allmodconfig is
> worse than I thought :(
>
> It really is in arch maintainers' best interest to keep their allmodconfig
> in good shape, for this reason. arm's _isn't_ in good shape: the compile
> fails for several long-standing reasons (eg: no hope of building DRM) and I
> don't think the coverage is very broad either.
I think that Russell has said that allmodconfig isn't very realistic
for ARM, with its 70+ config files. Nevertheless, having a usable
allmodconfig would be very helpful.
---
~Randy
On Tue, Feb 12, 2008 at 12:02:08PM +1100, Stephen Rothwell wrote:
> I will attempt to build the tree between each merge (and a failed build
> will again cause the offending tree to be dropped). These builds will be
> necessarily restricted to probably one architecture/config. I will build
> the entire tree on as many architectures/configs as seem sensible and
> the results of that will be available on a web page (to be announced).
This restriction means that the value for the ARM architecture is soo
limited it's probably not worth the hastle participating in this project.
We already know that -mm picks up on very few ARM conflicts because
Andrew doesn't run through the entire set of configurations; unfortunately
ARM is one of those architectures which is very diverse [*], and because
of that, ideas like "allyconfig" are just completely irrelevant to it.
As mentioned elsewhere, what we need for ARM is to extend the kautobuild
infrastructure (see armlinux.simtec.co.uk) so that we can have more trees
at least compile tested regularly - but that requires the folk there to
have additional compute power (which isn't going to happen unless folk
stamp up some machines _or_ funding).
More trees maintained by more people isn't going to help the situation -
if anything, it's going to make it worse - more trees needing more testing
by the already extremely limited resources we have available.
For reference, even _I_ don't build test the entire set of ARM defconfigs -
at about 7 minutes a build, 75 defconfigs, that's about 9 hours... I
just build those which are important to myself, hope that the others are
fine, and rely on kautobuild finding any breakage.
* - plus, its very difficult to get maintainers to see why having a
kernel able to support multiple platforms is a good thing. For example,
I would absolutely love to be able to combine more platforms into one
build (such as Lubbock, Mainstone and probably other PXA stuff), but
there's issues with drivers preventing it. (For those two I just
mentioned, it's the SMC91x net driver whose build needs to be configured
to the precise machine due to the multitude of different ways to connect
the hardware to the processor.)
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of:
On Fri, 15 Feb 2008 15:47:24 -0800
Randy Dunlap <[email protected]> wrote:
> On Fri, 15 Feb 2008 15:37:32 -0800 Andrew Morton wrote:
>
> > On Fri, 15 Feb 2008 23:23:08 +0000
> > Russell King <[email protected]> wrote:
> >
> > > On Tue, Feb 12, 2008 at 12:48:13PM -0800, Greg KH wrote:
> > > > I have tried, and successfully done this many times in the past. The
> > > > kobject change was one example: add a new function, migrate all users of
> > > > a direct pointer over to that function, after that work is all done and
> > > > in, change the structure and do the needed work afterward. All is
> > > > bisectable completly, with no big "flag day" needed.
> > >
> > > Incorrect - because this all happened far too quickly. This is one of
> > > the reasons that I ended up having to redo various parts of the ARM tree
> > > because stuff broke - set_kset_name() completely vanished introducing
> > > compile errors, and iirc some merge issues as well.
> > >
> > > I had patches introducing new system objects which use that, and
> > > modifications extremely close to other uses in the PXA code.
> > >
> > > The end result (through rebuilding the affected parts of my git tree, and
> > > asking people for replacement patches) was something that is bisectable -
> > > but had I tried to merge stuff as is, it would've been an utter mess, and
> > > _was_ unbuildable.
> > >
> >
> > I wonder why I didn't see any of this - I build arm allmodconfig at least
> > once a week, usually more frequently.
> >
> > So either the offending patches weren't in my pile or arm allmodconfig is
> > worse than I thought :(
> >
> > It really is in arch maintainers' best interest to keep their allmodconfig
> > in good shape, for this reason. arm's _isn't_ in good shape: the compile
> > fails for several long-standing reasons (eg: no hope of building DRM) and I
> > don't think the coverage is very broad either.
>
> I think that Russell has said that allmodconfig isn't very realistic
> for ARM, with its 70+ config files.
You'd need to pick one board support and enable everything else you
possibly can.
On Sat, 16 Feb 2008 00:05:59 +0100
Roel Kluin <[email protected]> wrote:
> Alan Cox wrote:
> >> Evolution in nature and changes in code are different because in code junk
> >> and bugs are constantly removed. In biology junk is allowed and may provide
> >> a pool for future development. Linux development is intended and not
> >> survival.
> >
> > I would be interested to see any evidence (rather than intuition) to
> > support that, given that both appear to be the same kind of structure and
> > that structure is nowadays fairly well understood.
>
> What do you mean with structure, the evolution? that both are a language?
No that they show the same mathematical structure and behaviour - both
are scale free networks.
Alan
On Sat, 16 Feb 2008 00:09:43 +0000
Russell King <[email protected]> wrote:
> For reference, even _I_ don't build test the entire set of ARM defconfigs -
> at about 7 minutes a build, 75 defconfigs, that's about 9 hours... I
> just build those which are important to myself, hope that the others are
> fine, and rely on kautobuild finding any breakage.
>
you need a better box ;)
cerfcube_defconfig: 35 seconds
carmeva_defconfig: 23 seconds
spitz_defconfig (one of the biggest): 45 seconds
so would a stupid `for i in arch/arm/configs/*' script be sufficient
coverage?
On Fri, Feb 15, 2008 at 03:47:24PM -0800, Randy Dunlap wrote:
> On Fri, 15 Feb 2008 15:37:32 -0800 Andrew Morton wrote:
> > I wonder why I didn't see any of this - I build arm allmodconfig at least
> > once a week, usually more frequently.
Basically, you don't build any of the PXA platforms, which is what was
affected.
> > So either the offending patches weren't in my pile or arm allmodconfig is
> > worse than I thought :(
> >
> > It really is in arch maintainers' best interest to keep their allmodconfig
> > in good shape, for this reason. arm's _isn't_ in good shape: the compile
> > fails for several long-standing reasons (eg: no hope of building DRM) and I
> > don't think the coverage is very broad either.
>
> I think that Russell has said that allmodconfig isn't very realistic
> for ARM, with its 70+ config files. Nevertheless, having a usable
> allmodconfig would be very helpful.
Which is quite impossible. I've said all along that all*config is bad
news for ARM and folk haven't listened.
allmodconfig can (and does) work on some platform configurations such as
the one Andrew builds - which will be based on the Versatile platform.
However, that doesn't get _any_ of the PXA SoC drivers scattered throughout
the tree, the PXA SoC support in arch/arm/mach-pxa, none of the PXA platform
support files.
If you built an allmodconfig PXA configuration, you'd get those, but you
wouldn't get the OMAP SoC drivers, nor the ARM Primecell drivers found on
ARMs Integrator, Versatile and Realview platforms and the Cirrus EP93xx
SoCs. Nor the Atmel AT91 drivers... and so the list goes on.
Each family of platforms are, unfortunately, quite distinct from each
other.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of:
Russell King wrote:
> On Fri, Feb 15, 2008 at 03:47:24PM -0800, Randy Dunlap wrote:
>> On Fri, 15 Feb 2008 15:37:32 -0800 Andrew Morton wrote:
>>> I wonder why I didn't see any of this - I build arm allmodconfig at least
>>> once a week, usually more frequently.
>
> Basically, you don't build any of the PXA platforms, which is what was
> affected.
>
>>> So either the offending patches weren't in my pile or arm allmodconfig is
>>> worse than I thought :(
>>>
>>> It really is in arch maintainers' best interest to keep their allmodconfig
>>> in good shape, for this reason. arm's _isn't_ in good shape: the compile
>>> fails for several long-standing reasons (eg: no hope of building DRM) and I
>>> don't think the coverage is very broad either.
>> I think that Russell has said that allmodconfig isn't very realistic
>> for ARM, with its 70+ config files. Nevertheless, having a usable
>> allmodconfig would be very helpful.
>
> Which is quite impossible. I've said all along that all*config is bad
> news for ARM and folk haven't listened.
>
> allmodconfig can (and does) work on some platform configurations such as
> the one Andrew builds - which will be based on the Versatile platform.
> However, that doesn't get _any_ of the PXA SoC drivers scattered throughout
> the tree, the PXA SoC support in arch/arm/mach-pxa, none of the PXA platform
> support files.
>
> If you built an allmodconfig PXA configuration, you'd get those, but you
> wouldn't get the OMAP SoC drivers, nor the ARM Primecell drivers found on
> ARMs Integrator, Versatile and Realview platforms and the Cirrus EP93xx
> SoCs. Nor the Atmel AT91 drivers... and so the list goes on.
>
> Each family of platforms are, unfortunately, quite distinct from each
> other.
>
Does that mean that an artificial allarmconfig can't be made to build?
We clearly don't care whether it can boot or work, but we would just as
clearly like to be able to build more ARM stuff without N (N > 10) configs.
--
~Randy
On Fri, Feb 15, 2008 at 04:21:21PM -0800, Andrew Morton wrote:
> On Sat, 16 Feb 2008 00:09:43 +0000
> Russell King <[email protected]> wrote:
>
> > For reference, even _I_ don't build test the entire set of ARM defconfigs -
> > at about 7 minutes a build, 75 defconfigs, that's about 9 hours... I
> > just build those which are important to myself, hope that the others are
> > fine, and rely on kautobuild finding any breakage.
> >
>
> you need a better box ;)
Maybe - it's a lowly 2.6GHz P4 with 1GB RAM, ICH5, and SATA disk.
> cerfcube_defconfig: 35 seconds
> carmeva_defconfig: 23 seconds
> spitz_defconfig (one of the biggest): 45 seconds
>
> so would a stupid `for i in arch/arm/configs/*' script be sufficient
> coverage?
It will certainly improve the situation significantly, and pick up
on some non-ARM problems like (badge4_defconfig, since 2.6.24-git2):
drivers/built-in.o: In function `v4l2_i2c_attach':
drivers/media/video/v4l2-common.c:1035: undefined reference to `i2c_attach_client'
Currently, the defconfigs known to fail (long term) in Linus' tree
are clps7500_defconfig and trizeps4_defconfig - the former I'm tempted
to remove, the latter seems to be because the PCMCIA changes were lost
on the linux-pcmcia list and the trizeps folk have probably given up.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of:
On Fri, Feb 15, 2008 at 04:21:21PM -0800, Andrew Morton wrote:
> On Sat, 16 Feb 2008 00:09:43 +0000
> Russell King <[email protected]> wrote:
>
> > For reference, even _I_ don't build test the entire set of ARM defconfigs -
> > at about 7 minutes a build, 75 defconfigs, that's about 9 hours... I
> > just build those which are important to myself, hope that the others are
> > fine, and rely on kautobuild finding any breakage.
> >
>
> you need a better box ;)
>
> cerfcube_defconfig: 35 seconds
> carmeva_defconfig: 23 seconds
> spitz_defconfig (one of the biggest): 45 seconds
>
> so would a stupid `for i in arch/arm/configs/*' script be sufficient
> coverage?
I do this wildcard together with
yes '' | make ARCH=arm ... oldconfig
make
Sans toolchain issues, it's pretty good -- Russell larts you when some
defconfig becomes broken anyway. :^)
On Sat, 16 Feb 2008 00:31:36 +0000
Russell King <[email protected]> wrote:
> > so would a stupid `for i in arch/arm/configs/*' script be sufficient
> > coverage?
>
> It will certainly improve the situation significantly, and pick up
> on some non-ARM problems like (badge4_defconfig, since 2.6.24-git2):
OK, I'll toss something together.
> the latter seems to be because the PCMCIA changes were lost
> on the linux-pcmcia list and the trizeps folk have probably given up.
I appear to be pcmcia maintainer lately so if someone wants to dust them
off and send them over we can see what we can do?
On Sat, Feb 16, 2008 at 03:42:49AM +0300, Alexey Dobriyan wrote:
> On Fri, Feb 15, 2008 at 04:21:21PM -0800, Andrew Morton wrote:
> > On Sat, 16 Feb 2008 00:09:43 +0000
> > Russell King <[email protected]> wrote:
> >
> > > For reference, even _I_ don't build test the entire set of ARM defconfigs -
> > > at about 7 minutes a build, 75 defconfigs, that's about 9 hours... I
> > > just build those which are important to myself, hope that the others are
> > > fine, and rely on kautobuild finding any breakage.
> > >
> >
> > you need a better box ;)
> >
> > cerfcube_defconfig: 35 seconds
> > carmeva_defconfig: 23 seconds
> > spitz_defconfig (one of the biggest): 45 seconds
> >
> > so would a stupid `for i in arch/arm/configs/*' script be sufficient
> > coverage?
>
> I do this wildcard together with
>
> yes '' | make ARCH=arm ... oldconfig
> make
>
> Sans toolchain issues, it's pretty good -- Russell larts you when some
> defconfig becomes broken anyway. :^)
Only when I check the kautobuild website - which I've not been doing
regularly since about end of November, and it only covers Linus'
kernels.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of:
On Wed, 13 Feb 2008, Geert Uytterhoeven wrote:
> On Tue, 12 Feb 2008, Greg KH wrote:
> > On Tue, Feb 12, 2008 at 04:49:46PM -0800, Linus Torvalds wrote:
> > > On Tue, 12 Feb 2008, Greg KH wrote:
> > > > Perhaps you need to switch to using quilt. This is the main reason why
> > > > I use it.
> > >
> > > Btw, on that note: if some quilt user can send an "annotated history file"
> > > of their quilt usage, it's something that git really can do, and I'll see
> > > if I can merge (or rather, coax Junio to merge) the relevant part of stgit
> > > to make it possible to just basically get "quilt behaviour" for the parts
> > > of a git tree that you haven't pushed out yet.
> >
> > Ted's description matches mine (keep quilt tree in git, edit changelog
> > entries, rebase on newer kernel versions, etc.) I can go into details
> > if needed.
>
> Ack. Same for PS3 and m68k (except I don't have the m68k patches in git (yet)).
>
> Two issues with using quilt:
> 1. Sometimes a patch still applies after it was integrated upstream,
Add QUILT_PATCH_OPTS="--fuzz=0" to your ~/.quiltrc file.
The default is fuzz=2 IIRC which tries to be too clever. I set fuzz
to 0 after I got burned by an unnoticed "quilt patched it somewhere
else" bug, which took me half a day to figure out.
Thanks,
tglx
On Fri, 2008-02-15 at 12:02 +1100, Stephen Rothwell wrote:
> Hi Roland,
>
> On Thu, 14 Feb 2008 15:22:46 -0800 Roland Dreier <[email protected]> wrote:
> >
> > For InfiniBand/RDMA, the tree is:
> >
> > master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-next
> >
> > or via git protocol:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-next
> >
> > contact addresses (me plus a mailing list):
> >
> > [email protected]
> > [email protected]
>
> Added, thanks.
Do you have the tree and build logs available anywhere? I'd like to
turn off the merge tree builds when this is able to replace it.
James
Hi James,
On Sat, 16 Feb 2008 09:14:32 -0600 James Bottomley <[email protected]> wrote:
>
> Do you have the tree and build logs available anywhere? I'd like to
> turn off the merge tree builds when this is able to replace it.
The tree is at
git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git - or did
you mean the logs of creating the tree. I currently ceate the tree
fairly manually (as I slowly script what can be) and so have no logs of
that process.
The build logs that I have some control over are at
http://kisskb.ellerman.id.au/kisskb/branch/9/. I am hoping to expand on
the arch/config combinations over time (I have to convince my tame cross
compiler builder :-)). I also hope that others will build this tree for
themselves and publish the results.
--
Cheers,
Stephen Rothwell [email protected]
On Tue, 2008-02-12 at 12:24 -0600, James Bottomley wrote:
> Hm ... I think net is a counter example to this. Rebases certainly work
> for them.
That's a matter of opinion.
I'm working on cleaning up the libertas driver as and when I have time,
and the constant rebasing of the git trees effectively means that I
can't just use git as it was intended; I feel that I'm being forced back
into the 1990s by having to deal with sequences of patches instead.
It's such a pain that I'm seriously tempted to push my changes directly
to Linus instead of through the subsystem maintainers.
--
dwmw2
On Sun, 2008-02-17 at 16:25 +1100, Stephen Rothwell wrote:
> Hi James,
>
> On Sat, 16 Feb 2008 09:14:32 -0600 James Bottomley <[email protected]> wrote:
> >
> > Do you have the tree and build logs available anywhere? I'd like to
> > turn off the merge tree builds when this is able to replace it.
>
> The tree is at
> git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git - or did
> you mean the logs of creating the tree.
Yes, the logs of creating the tree.
> I currently ceate the tree
> fairly manually (as I slowly script what can be) and so have no logs of
> that process.
Um, well, I've already pointed to tools that currently do this bit
automatically. Do you have a list of input trees anywhere?
> The build logs that I have some control over are at
> http://kisskb.ellerman.id.au/kisskb/branch/9/. I am hoping to expand on
> the arch/config combinations over time (I have to convince my tame cross
> compiler builder :-)). I also hope that others will build this tree for
> themselves and publish the results.
Oh, OK ... by build I mean combine the trees and quilts into the actual
tree. Compiling is nice, but it's the tree construction logs I want to
see to make sure there aren't any impending merge problems. Most people
verify on a fairly constant basis that their own trees actually build.
The only problems we have are edge configurations (like arm and sparc,
which have SCSI drivers that don't build except on those architectures).
James
Frank Seidel wrote:
> Lets get serious. I cannot speak for Ann and Harvey, but I'm quite sure they
> also really hope - at least i very strongly do - you not only call on us when
> things become a burden, but let us help and assist you right from the start.
I just started a little naive webpage where i collected some of the main
infos about linux-next on
http://linux.f-seidel.de/linux-next/
while the wiki there is probably the best filled part. Does this
make any sense to you to continue this? Or do you already have something
else in mind or even already prepared?
Thanks,
Frank
Hi Linus,
On Thu, 14 Feb 2008 10:01:14 -0800 (PST) Linus Torvalds <[email protected]> wrote:
>
> I absolutely have no problem with having a "this is the infrastrcture
> changes that will go into the next release". In fact, I can even
> *maintain* such a branch.
>
> I've not wanted to open up a second branch for "this is for next release",
> because quite frankly, one of the other problems we have is that people
> already spend way too much time on the next release compared to just
> looking at regressions in the current one. But especially if we're talking
> about _purely_ API changes etc infrastructure, I could certainly do a
> "next" branch.
So, will you open such a branch? If so, what would be the mechanics of
having patches applied to it? I assume people would have to suggest such
changes explicitly and have them reviewed (hopefully more thoroughly than
usual) in that light. I guess one place these "infrastructure" changes
may be noticed would be when subsystem maintainers stray outside their
subsystem in what they submit to the linux-next tree (or break it).
Then I assume most people would start working on a merge of this "next"
branch and your "master" branch, right? Consequently, each linux-next
would also be based on that merge.
I suppose I am stating the obvious (or asking the dumb questions), but I
always find it easier to have explicit answers to these sorts of things.
--
Cheers,
Stephen Rothwell [email protected]
Stephen Rothwell wrote:
> On Thu, 14 Feb 2008 10:01:14 -0800 (PST) Linus Torvalds <[email protected]> wrote:
>>
>> I absolutely have no problem with having a "this is the infrastrcture
>> changes that will go into the next release". In fact, I can even
>> *maintain* such a branch.
>>
>> I've not wanted to open up a second branch for "this is for next release",
>> because quite frankly, one of the other problems we have is that people
>> already spend way too much time on the next release compared to just
>> looking at regressions in the current one. But especially if we're talking
>> about _purely_ API changes etc infrastructure, I could certainly do a
>> "next" branch.
>
> So, will you open such a branch? If so, what would be the mechanics of
> having patches applied to it? I assume people would have to suggest such
> changes explicitly and have them reviewed (hopefully more thoroughly than
> usual) in that light. I guess one place these "infrastructure" changes
> may be noticed would be when subsystem maintainers stray outside their
> subsystem in what they submit to the linux-next tree (or break it).
Two things may largely eliminate the need for parallel branches.
1. Do infrastructure changes and whole tree wide refactoring etc. in a
compatible manner with a brief but nonzero transition period.
2. Insert a second merge window right after the usual merge window for
changes which cannot be well done with a transition period.
(I probably missed a number of points why these two things are not
always feasible, because I am just a downstream person.)
--
Stefan Richter
-=====-==--- --=- =-=--
http://arcgraph.de/sr/
On Wed, Feb 20, 2008 at 04:38:52PM +0100, Stefan Richter wrote:
> Two things may largely eliminate the need for parallel branches.
>
> 1. Do infrastructure changes and whole tree wide refactoring etc. in a
> compatible manner with a brief but nonzero transition period.
>
> 2. Insert a second merge window right after the usual merge window for
> changes which cannot be well done with a transition period.
A third option would be if people add new functions (with no users) in
-rc2 or -rc3 timeframes as long as it is part of a fully reviewed
patch with users that will use those new features in various kernel
development trees.
Since there wouldn't be any users in Linus's tree, there's no risk in
making those functions available in mainline ahead of time.
- Ted
On Wed, Feb 20, 2008 at 10:42:35AM -0500, Theodore Tso wrote:
> On Wed, Feb 20, 2008 at 04:38:52PM +0100, Stefan Richter wrote:
> > Two things may largely eliminate the need for parallel branches.
> >
> > 1. Do infrastructure changes and whole tree wide refactoring etc. in a
> > compatible manner with a brief but nonzero transition period.
> >
> > 2. Insert a second merge window right after the usual merge window for
> > changes which cannot be well done with a transition period.
>
> A third option would be if people add new functions (with no users) in
> -rc2 or -rc3 timeframes as long as it is part of a fully reviewed
> patch with users that will use those new features in various kernel
> development trees.
>...
I don't like suggestions based on unrealistic assumptions like
"a fully reviewed patch".
E.g. userspace ABI's are much more stable and everyone is aware that
they must be gotten right with the first try since they are then cast in
stone - but we all remember the recent timerfd fiasco.
> - Ted
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
On Wed, Feb 20, 2008 at 07:13:16PM +0200, Adrian Bunk wrote:
> > A third option would be if people add new functions (with no users) in
> > -rc2 or -rc3 timeframes as long as it is part of a fully reviewed
> > patch with users that will use those new features in various kernel
> > development trees.
> >...
>
> I don't like suggestions based on unrealistic assumptions like
> "a fully reviewed patch".
>
> E.g. userspace ABI's are much more stable and everyone is aware that
> they must be gotten right with the first try since they are then cast in
> stone - but we all remember the recent timerfd fiasco.
I'm talking about kernel interfaces, not userspace API's. And we can
change them if they are wrong, since they *are* kernel interfaces; but
if they correct, they ease the cross-tree merge pain.
- Ted
Hi!
> > Does this fit what you had in mind?
>
> Yes it does.
>
> Now I'll ask if you think embedding this information in one of the C
> files for a module would be even nicer?
I kind of like to be able to grep over just Kconfig files, to find out
what is going on...
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Hi Russell,
On Sat, 16 Feb 2008 00:09:43 +0000 Russell King <[email protected]> wrote:
>
> On Tue, Feb 12, 2008 at 12:02:08PM +1100, Stephen Rothwell wrote:
> > I will attempt to build the tree between each merge (and a failed build
> > will again cause the offending tree to be dropped). These builds will be
> > necessarily restricted to probably one architecture/config. I will build
> > the entire tree on as many architectures/configs as seem sensible and
> > the results of that will be available on a web page (to be announced).
>
> This restriction means that the value for the ARM architecture is soo
> limited it's probably not worth the hastle participating in this project.
>
> We already know that -mm picks up on very few ARM conflicts because
> Andrew doesn't run through the entire set of configurations; unfortunately
> ARM is one of those architectures which is very diverse [*], and because
> of that, ideas like "allyconfig" are just completely irrelevant to it.
>
> As mentioned elsewhere, what we need for ARM is to extend the kautobuild
> infrastructure (see armlinux.simtec.co.uk) so that we can have more trees
> at least compile tested regularly - but that requires the folk there to
> have additional compute power (which isn't going to happen unless folk
> stamp up some machines _or_ funding).
I now have an arm cross compiler (gcc-4.0.2-glibc-2.3.6
arm-unknown-linux-gnu). (See the results page at
http://kisskb.ellerman.id.au/kisskb/branch/9/ - I must get a better
name/place :-(.) Is this sufficient to help you out? What configs would
be useful to build (as Andrew said, they don't take very long each).
I really want as many subsystems as possible in the linux-next tree in an
attempt to avoid some of the merge/conflict problems we have had in the
past. What can we do to help?
--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/
Hi Russell,
On Tue, 26 Feb 2008 14:54:18 +1100 Stephen Rothwell <[email protected]> wrote:
>
> On Sat, 16 Feb 2008 00:09:43 +0000 Russell King <[email protected]> wrote:
> >
> > As mentioned elsewhere, what we need for ARM is to extend the kautobuild
> > infrastructure (see armlinux.simtec.co.uk) so that we can have more trees
> > at least compile tested regularly - but that requires the folk there to
> > have additional compute power (which isn't going to happen unless folk
> > stamp up some machines _or_ funding).
>
> I now have an arm cross compiler (gcc-4.0.2-glibc-2.3.6
> arm-unknown-linux-gnu). (See the results page at
> http://kisskb.ellerman.id.au/kisskb/branch/9/ - I must get a better
> name/place :-(.) Is this sufficient to help you out? What configs would
> be useful to build (as Andrew said, they don't take very long each).
>
> I really want as many subsystems as possible in the linux-next tree in an
> attempt to avoid some of the merge/conflict problems we have had in the
> past. What can we do to help?
OK, I tried this out: I built all 75 arm defconfigs with the cross
compiler above (the resulting log file is at
http://ozlabs.org/~sfr/armall.log.bz2). 17 failed (I don't know why - I
didn't even really look). If this is useful, I can get this added to our
infrastructure so that every linux-next kernel (and others as well, maybe)
will be built for all these.
Would that help, Russell?
--
Cheers,
Stephen Rothwell [email protected]
P.S. From the timestamps in the log file you can see that this would take
around an hour each time.
On Fri, Feb 29, 2008 at 11:45:51PM +1100, Stephen Rothwell wrote:
> Hi Russell,
>
> On Tue, 26 Feb 2008 14:54:18 +1100 Stephen Rothwell <[email protected]> wrote:
> >
> > On Sat, 16 Feb 2008 00:09:43 +0000 Russell King <[email protected]> wrote:
> > >
> > > As mentioned elsewhere, what we need for ARM is to extend the kautobuild
> > > infrastructure (see armlinux.simtec.co.uk) so that we can have more trees
> > > at least compile tested regularly - but that requires the folk there to
> > > have additional compute power (which isn't going to happen unless folk
> > > stamp up some machines _or_ funding).
> >
> > I now have an arm cross compiler (gcc-4.0.2-glibc-2.3.6
> > arm-unknown-linux-gnu). (See the results page at
> > http://kisskb.ellerman.id.au/kisskb/branch/9/ - I must get a better
> > name/place :-(.) Is this sufficient to help you out? What configs would
> > be useful to build (as Andrew said, they don't take very long each).
> >
> > I really want as many subsystems as possible in the linux-next tree in an
> > attempt to avoid some of the merge/conflict problems we have had in the
> > past. What can we do to help?
>
> OK, I tried this out: I built all 75 arm defconfigs with the cross
> compiler above (the resulting log file is at
> http://ozlabs.org/~sfr/armall.log.bz2). 17 failed (I don't know why - I
> didn't even really look).
2 errors are expected (long-standing breakages).
14 failures are due to your ancient gcc 4.0.2 not supporting -mabi=aapcs-linux:
/scratch/sfr/next/scripts/mod/empty.c:1: error: invalid ABI option: -mabi=aapcs-linux
Another one is an internal error in your ancient compiler:
/scratch/sfr/next/arch/arm/common/it8152.c:148: internal compiler error: in trunc_int_for_mode, at explow.c:53
Can you build a cross compiler using gcc 4.2.3 from ftp.gnu.org instead?
That one serves me perfectly for my cross-compile tests.
>...
> Cheers,
> Stephen Rothwell [email protected]
>...
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Hi Adrian,
On Fri, 29 Feb 2008 15:04:26 +0200 Adrian Bunk <[email protected]> wrote:
>
> 2 errors are expected (long-standing breakages).
>
> 14 failures are due to your ancient gcc 4.0.2 not supporting -mabi=aapcs-linux:
> /scratch/sfr/next/scripts/mod/empty.c:1: error: invalid ABI option: -mabi=aapcs-linux
>
> Another one is an internal error in your ancient compiler:
> /scratch/sfr/next/arch/arm/common/it8152.c:148: internal compiler error: in trunc_int_for_mode, at explow.c:53
>
> Can you build a cross compiler using gcc 4.2.3 from ftp.gnu.org instead?
> That one serves me perfectly for my cross-compile tests.
OK, thanks for the feedback it is very helpful. I will look at building
a newer compiler on Monday.
--
Cheers,
Stephen Rothwell [email protected]