Hi,
In working with some of the current out-of-tree drivers, some of them
are asking if they could keep their past development history when
merging the code into the main kernel tree.
Now normally we don't do this for new drivers, just dropping them in in
one big patch, or sometimes multiple patches to get it through email
filters.
The comedi group (data acquisition subsystem for Linux) have their whole
history going back to 2000 in a git tree (well, a cvs->git repo.)
I was wondering if it would be acceptable to graft their tree into the
linux-2.6 tree (after moving the files to the proper location) to keep
their whole old history alive.
Now what good that old history would do, I really don't know and can't
think of a solid reason to need it, other than to give proper authorship
credit for the various individual drivers and parts of the code (which
is good to have at tims.)
But the merge looks pretty cool, and it is impressive that git can allow
this to happen, so there are some extra "style" points a merge like this
would give :)
So, any thoughts? Should I graft the trees, or just stick to simple
"here's the whole driver" type patches like we have been doing?
thanks,
greg k-h
Greg KH wrote:
> In working with some of the current out-of-tree drivers, some of them
> are asking if they could keep their past development history when
> merging the code into the main kernel tree.
>
> Now normally we don't do this for new drivers, just dropping them in in
> one big patch, or sometimes multiple patches to get it through email
> filters.
>
> The comedi group (data acquisition subsystem for Linux) have their whole
> history going back to 2000 in a git tree (well, a cvs->git repo.)
>
> I was wondering if it would be acceptable to graft their tree into the
> linux-2.6 tree (after moving the files to the proper location) to keep
> their whole old history alive.
>
> Now what good that old history would do, I really don't know and can't
> think of a solid reason to need it, other than to give proper authorship
> credit for the various individual drivers and parts of the code (which
> is good to have at tims.)
[...]
There can be other reasons to keep the history in /some/ publicly
accessible repo (maybe not in the mainline though):
- Changelogs sometimes explain why something was written in a
specific way, which may not be obvious from the code or comments.
- Some modifications during mainline merge preparation, e.g.
simplifications, may turn out undesirable later and one wants to
revert some of such modifications.
--
Stefan Richter
-=====-==--- =-=- ====-
http://arcgraph.de/sr/
On Wed, 29 Oct 2008, Greg KH wrote:
>
> In working with some of the current out-of-tree drivers, some of them
> are asking if they could keep their past development history when
> merging the code into the main kernel tree.
>
> Now normally we don't do this for new drivers, just dropping them in in
> one big patch, or sometimes multiple patches to get it through email
> filters.
I'd suggest you talk to Chris Mason about his btrfs import.
I'd _like_ for old history to be merged, but quite frankly, bisectability
is a fairly big deal, and while we often have cases where a _few_ commits
don't build and make bisecting hard, if you import the past development
history badly, you can easily end up with _hundreds_ of commits that
simply don't build as a kernel at all.
And at some point the "nice to have" history is suddenly "more pain than
it is worth".
> The comedi group (data acquisition subsystem for Linux) have their whole
> history going back to 2000 in a git tree (well, a cvs->git repo.)
>
> I was wondering if it would be acceptable to graft their tree into the
> linux-2.6 tree (after moving the files to the proper location) to keep
> their whole old history alive.
If you mean "graft" in the git technical sense, where you actually use a
grafts file to fake ancestry, then the answer is "Hell no".
If you mean "graft" in the sense of merging a unrelated tree, the same way
git itself merged the gitk tree, then the answer is "yes, we can do that,
but bisectability is really important".
And quite frankly, if you don't spend time looking at it and doing it
well, it's probably not worth doing at all. Are you ready to really try to
do a good job?
That's why I'd suggest you talk to Chris - because he did an import from
an external mercurial repo that wasn't even a full kernel, and with some
help from me got a really good history in his git tree by using a number
of tricks, notably using "git filter-branch" to create a new tree with the
whole history as part of a whole tree, and nicely bisectable too.
(See the result at
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=summary
if you want to).
Anyway, I'll happily help with any cleanup and/or git questions, but Chris
can talk about the issues he had - he did all the actual work.
In contrast, if what you just want to do is to take some nasty straight
CVS import, and just do a git merge, and not try to make it bisect sanely,
at that point I'd say that the history is absolutely _not_ worth it.
Linus
On Thu, 2008-10-30 at 12:04 -0700, Linus Torvalds wrote:
>
> On Wed, 29 Oct 2008, Greg KH wrote:
> >
> > In working with some of the current out-of-tree drivers, some of them
> > are asking if they could keep their past development history when
> > merging the code into the main kernel tree.
> >
> > Now normally we don't do this for new drivers, just dropping them in in
> > one big patch, or sometimes multiple patches to get it through email
> > filters.
>
> I'd suggest you talk to Chris Mason about his btrfs import.
>
> I'd _like_ for old history to be merged, but quite frankly, bisectability
> is a fairly big deal, and while we often have cases where a _few_ commits
> don't build and make bisecting hard, if you import the past development
> history badly, you can easily end up with _hundreds_ of commits that
> simply don't build as a kernel at all.
Is bisection really important for a new driver? I wouldn't expect
anyone to have a config that builds a new driver when bisecting prior to
the point where the driver was merged. Just to be safe, it would be
easy enough to have the last commit be the one that adds the driver to
Kconfig and the higher-level Makefile.
Shaggy
--
David Kleikamp
IBM Linux Technology Center
On Thu, Oct 30, 2008 at 12:04:34PM -0700, Linus Torvalds wrote:
>
>
> On Wed, 29 Oct 2008, Greg KH wrote:
> >
> > In working with some of the current out-of-tree drivers, some of them
> > are asking if they could keep their past development history when
> > merging the code into the main kernel tree.
> >
> > Now normally we don't do this for new drivers, just dropping them in in
> > one big patch, or sometimes multiple patches to get it through email
> > filters.
>
> I'd suggest you talk to Chris Mason about his btrfs import.
>
> I'd _like_ for old history to be merged, but quite frankly, bisectability
> is a fairly big deal, and while we often have cases where a _few_ commits
> don't build and make bisecting hard, if you import the past development
> history badly, you can easily end up with _hundreds_ of commits that
> simply don't build as a kernel at all.
>
> And at some point the "nice to have" history is suddenly "more pain than
> it is worth".
>
> > The comedi group (data acquisition subsystem for Linux) have their whole
> > history going back to 2000 in a git tree (well, a cvs->git repo.)
> >
> > I was wondering if it would be acceptable to graft their tree into the
> > linux-2.6 tree (after moving the files to the proper location) to keep
> > their whole old history alive.
>
> If you mean "graft" in the git technical sense, where you actually use a
> grafts file to fake ancestry, then the answer is "Hell no".
>
> If you mean "graft" in the sense of merging a unrelated tree, the same way
> git itself merged the gitk tree, then the answer is "yes, we can do that,
> but bisectability is really important".
I ment the later, like what you did with the gitk tree and the git repo.
> And quite frankly, if you don't spend time looking at it and doing it
> well, it's probably not worth doing at all. Are you ready to really try to
> do a good job?
>
> That's why I'd suggest you talk to Chris - because he did an import from
> an external mercurial repo that wasn't even a full kernel, and with some
> help from me got a really good history in his git tree by using a number
> of tricks, notably using "git filter-branch" to create a new tree with the
> whole history as part of a whole tree, and nicely bisectable too.
>
> (See the result at
>
> http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=summary
>
> if you want to).
Thanks, I'll take a look at this.
But as for the 'bisectability' at one point in the merge, you will be
adding a stand-alone driver into the kernel itself. So for anyone
traversing down that path, all you would be building would be the driver
itself, the whole rest of the kernel is "gone". So, keeping the ability
for those points along the stand-alone driver might be nice, I can see
this confusing the heck out of people who would wonder where in the
world the kernel went away to.
> Anyway, I'll happily help with any cleanup and/or git questions, but Chris
> can talk about the issues he had - he did all the actual work.
>
> In contrast, if what you just want to do is to take some nasty straight
> CVS import, and just do a git merge, and not try to make it bisect sanely,
> at that point I'd say that the history is absolutely _not_ worth it.
Fair enough, I'll play around with the comedi git tree, look at what
Chris did with btrfs, and then see if it's even worth it.
thanks for your comments,
greg k-h
On Thu, 30 Oct 2008, Dave Kleikamp wrote:
>
> Is bisection really important for a new driver?
It's not the new driver that matters.
It's bisecting all the other _unrelated_ bugs, and then ending up in the
middle of the driver history.
Linus
On Thu, 30 Oct 2008, Greg KH wrote:
>
> But as for the 'bisectability' at one point in the merge, you will be
> adding a stand-alone driver into the kernel itself. So for anyone
> traversing down that path, all you would be building would be the driver
> itself, the whole rest of the kernel is "gone".
Right. This was exactly what happened initially in the btrfs thing. And it
was horrid.
It was horrid because it was totally unexpected for users, and causes huge
churn and confusion when trying to check out a totally different directory
layout (and git won't remove the old *.o files, so trust me, it _will_ be
confusing).
But it was horrid also because if you do it the straightforward but stupid
way (the way that git definitely does support) by using a "subdirectory
merge" to merge the individual driver into a new location, it also makes
the history almost totally useless. Suddenly trivial things like
git log drivers/xyz/file.c
don't work well at all any more, because in the old history it will all be
under the pathname "file.c" _without_ the drivers/xyz/ part. And the
merge diff will be just one huge mess..
So yes, git can do it that way, but the end result is so horrible as to be
worse than not importing the history at all!
What I got Chris Mason to do was to run
git filter-branch --index odd-script-goes-here
with that odd script looking something like:
> #!/bin/sh
> exists=$(git ls-files fs/btrfs/)
> [ -z "$exists" ] &&
> git ls-files --stage |
> awk -F '\t' '{ print "0 0000000000000000000000000000000000000000\t" $2 "\n" $1 "\tfs/btrfs/" $2 }' |
> git update-index --index-info
which basically does a directory rename in the branch (obviously, in this
case into fs/btrfs, which is not what _you_ want). That way, at least the
directory structure of the tree you merge has the same layout, and you
don't get _that_ particular directory jumping back and forth.
Chris also merged in the history at the 2.6.26 tree (I think), so that
while his original history had had just a stand-alone btrfs build, the end
result was actually *totally* bisectable. Again, you should ask him about
any other scripts he ran.
Linus
On Thu, Oct 30, 2008 at 1:49 AM, Greg KH <[email protected]> wrote:
> Hi,
>
> In working with some of the current out-of-tree drivers, some of them
> are asking if they could keep their past development history when
> merging the code into the main kernel tree.
>
> Now normally we don't do this for new drivers, just dropping them in in
> one big patch, or sometimes multiple patches to get it through email
> filters.
>
> The comedi group (data acquisition subsystem for Linux) have their whole
> history going back to 2000 in a git tree (well, a cvs->git repo.)
That seems to go back further than Linus's kernel tree goes. Would
this be a problem?
>
> I was wondering if it would be acceptable to graft their tree into the
> linux-2.6 tree (after moving the files to the proper location) to keep
> their whole old history alive.
>
> Now what good that old history would do, I really don't know and can't
> think of a solid reason to need it, other than to give proper authorship
> credit for the various individual drivers and parts of the code (which
> is good to have at tims.)
>
> But the merge looks pretty cool, and it is impressive that git can allow
> this to happen, so there are some extra "style" points a merge like this
> would give :)
>
> So, any thoughts? Should I graft the trees, or just stick to simple
> "here's the whole driver" type patches like we have been doing?
The Comedi project looks really big, so I can understand wanting to
get the history for it simply because it's a ton of code. But if this
is as hard to do properly as Linus says, I would be wary of the
precedent merging in their history might set. Suddenly we could have
tons of developers clamoring for the same treatment.
If files are being moved and the directory layout of the foreign
repository is nothing like the kernel's layout, could all the old
revisions be formated in one huge patch series against a single
subdirectory? Sure the dates relative to the old work would be totally
wrong, but we would not have to worry about breaking bisections, et
al, while still retaining the content's change history.
That course would depend on how Comedi's repo is laid out and how the
layout will be done to put them in the kernel tree, as well as how
important the difference between 'content creation time' and 'commit
time' are to those involved. If it's just a matter of having
accountability and the ability to look at old versions, perhaps that
difference is not very important. Any way you do this besides the One
Big Patch approach would seem to screw up the history for someone. I
think it would be safer to tinker with the outside project's view of
the past than the kernel's.
>
> thanks,
>
> greg k-h
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
On Thu, 30 Oct 2008, J.R. Mauro wrote:
>
> But if this is as hard to do properly as Linus says, I would be wary of
> the precedent merging in their history might set. Suddenly we could have
> tons of developers clamoring for the same treatment.
I don't think it's _hard_. But I think it does require some effort.
Quite frankly, if it comes from CVS, I expect the logs, for example, to be
totally inappropriate. I've never ever seen a CVS project that has sane
commit messages (ok, so they're not "commits" in CVS, but you know what I
mean).
That's just one example. I'm not willing to pull history if it's full of
crap. Pulling history for "history's sake" is worthless. The end result
has to actually be meaningful and help _future_.
So I strongly suspect that an "automated" history merge is simply not
appropriate, and almost certainly results in an end result that is not up
to snuff. And the question is whether anybody is willing to actually put
in the effort to make the history actually be useful.
Linus
On Thu, Oct 30, 2008 at 9:47 PM, Linus Torvalds
<[email protected]> wrote:
>
> I don't think it's _hard_. But I think it does require some effort.
Right, my point is if this turns into a trend, will that effort be
well-spent? Or will it become too much of a pain to do all the time
and start making the history overly complex? I don't think the hard
part is in the import itself, but keeping up with more and more
imports that have to be done on a case-by-case basis, whereas patch
imports are automated already.
>
> Quite frankly, if it comes from CVS, I expect the logs, for example, to be
> totally inappropriate. I've never ever seen a CVS project that has sane
> commit messages (ok, so they're not "commits" in CVS, but you know what I
> mean).
>
> That's just one example. I'm not willing to pull history if it's full of
> crap. Pulling history for "history's sake" is worthless. The end result
> has to actually be meaningful and help _future_.
Yes, CVS is just braindead. I'm pretty sure we're on the same page
there. That's why I suggested the patch series idea. If we're less
interested in the actual date and time of the 'commit' and really just
interested in tracking and retaining contributor and change
information, then patches might be vastly easier, and they'll show up
as real git commits. So the commit date is messed up with respect to
when the CVS commit actually happened. Git already separates the
notion of 'author' and 'committer'... can we sanely separate the
notion of creation time (outside of tree) versus commit time (into
tree)?
If that separation is not acceptable and for some good reasons the
original commit dates and times *must* be preserved, then I freely
admit my idea is hare-brained and should be laughed off as the
ramblings of a young crackpot.
But I think it's a good middle-ground between a big-bang patch and
full-blown absorption of another project. From the looks of the
repository, there are less than 150 CVS commits for each driver file.
Most have less than 10. So per driver it would be a pretty short patch
series for the most part. And if Greg is tracking the CVS repo with
git, it would be fairly easy to create the series.
>
> So I strongly suspect that an "automated" history merge is simply not
> appropriate, and almost certainly results in an end result that is not up
> to snuff. And the question is whether anybody is willing to actually put
> in the effort to make the history actually be useful.
I agree. The only viable automated approach would be something akin to
git-submodule but with the ability to integrate the submodule'd
project upon import in such a way that things like bisection would
follow the main project's ancestry and ignore the submodule's. But
that's not possible right now and probably won't be in the foreseeable
future.
And the willingness to put in the effort may be there now, but what
I'm worried about is 6 months from now, when everyone who has an
out-of-tree driver wants history imported because "well, you did it
for the Comedi guys". Greg is a pretty busy guy, and we don't want him
buried under something like that, especially if the payoff is
marginal.
But hey, maybe if this is the direction we should go in, the tools to
automate it sanely will get written.
~J.R.
On Thu, Oct 30, 2008 at 04:28:46PM -0700, Linus Torvalds wrote:
>
>
> On Thu, 30 Oct 2008, Greg KH wrote:
> >
> > But as for the 'bisectability' at one point in the merge, you will be
> > adding a stand-alone driver into the kernel itself. So for anyone
> > traversing down that path, all you would be building would be the driver
> > itself, the whole rest of the kernel is "gone".
>
> Right. This was exactly what happened initially in the btrfs thing. And it
> was horrid.
>
> It was horrid because it was totally unexpected for users, and causes huge
> churn and confusion when trying to check out a totally different directory
> layout (and git won't remove the old *.o files, so trust me, it _will_ be
> confusing).
I agree, I don't want this to happen.
> What I got Chris Mason to do was to run
>
> git filter-branch --index odd-script-goes-here
>
> with that odd script looking something like:
>
> > #!/bin/sh
> > exists=$(git ls-files fs/btrfs/)
> > [ -z "$exists" ] &&
> > git ls-files --stage |
> > awk -F '\t' '{ print "0 0000000000000000000000000000000000000000\t" $2 "\n" $1 "\tfs/btrfs/" $2 }' |
> > git update-index --index-info
>
> which basically does a directory rename in the branch (obviously, in this
> case into fs/btrfs, which is not what _you_ want). That way, at least the
> directory structure of the tree you merge has the same layout, and you
> don't get _that_ particular directory jumping back and forth.
>
> Chris also merged in the history at the 2.6.26 tree (I think), so that
> while his original history had had just a stand-alone btrfs build, the end
> result was actually *totally* bisectable. Again, you should ask him about
> any other scripts he ran.
Ok, this seems a lot more reasonable, and doable.
But then I took a long look at the cvs commit logs.
And you are right, they are crap, and pointless to have here.
So, thanks a lot for showing me how to do this properly, in the future I
might do this (actually, for one wireless driver, I might do this now,
as it was always developed in git), but for the comedi code, I'll just
use the "normal" method of adding drivers one at a time, with none of
the crazy past history sucked in as it doesn't add any value.
thanks,
greg k-h
On Thu, Oct 30, 2008 at 10:38:36PM -0400, J.R. Mauro wrote:
> And the willingness to put in the effort may be there now, but what
> I'm worried about is 6 months from now, when everyone who has an
> out-of-tree driver wants history imported because "well, you did it
> for the Comedi guys". Greg is a pretty busy guy, and we don't want him
> buried under something like that, especially if the payoff is
> marginal.
Greg's a big boy and he can decide what he wants to do with his own
time --- but what I'd recommend is that if "other projects" want to
merge their history, it should be up to the upstream projects to
massage their git history into something that his suitable for
merging. It doesn't scale to have core maintainers doing heavy-duty
git history rewrites for everybody else. Of course, if someone wants
to volunteer their time to help other projects do this, that's
completely up to them.
- Ted
On Thu, Oct 30, 2008 at 10:38 PM, J.R. Mauro <[email protected]> wrote:
> I agree. The only viable automated approach would be something akin to
> git-submodule but with the ability to integrate the submodule'd
> project upon import in such a way that things like bisection would
> follow the main project's ancestry and ignore the submodule's. But
> that's not possible right now and probably won't be in the foreseeable
> future.
Here's one relatively easy solution:
(1) Fetch the to-be-merged branch of the "child" project into a
repository with the parent project ($KIDBRANCH)
(2) Find a branch on which you are going to rebase the child
project's history ($BASEBRANCH)
(3) Determine a directory into which you are going to put the child
project's history ($SUBDIR)
(4) Identify the first revision of the child project by looking at
the last (first-in-time) entry in "git log" of that branch.
($KIDSTART)
(5) Graft the kid's history onto the parent's history at that point:
echo "$KIDSTART $(git rev-parse --verify $BASEBRANCH)" >.git/info/grafts
(6) Rewrite the kid's history as a subdir of that version of the parent:
git filter-branch --index-filter 'git read-tree master && git
read-tree --prefix=drivers/newkiddriver/ "$GIT_COMMIT"' --
"$BASEBRANCH..$KIDBRANCH"
Now the history of $KIDBRANCH will appear to be a series of commits
which add files in $SUBDIR of the parent project, without any changes
at all to the parent project's tree. You can then do a merge of what
appears to be a normal development branch into the main repository.
Since none of the files are even remotely referenced, there can be no
possible build failures (hence bisection is not broken). You may then
add commits which add Kconfig options and Makefile references and
happily develop away.
Cheers,
Kyle Moffett
On Fri, Oct 31, 2008 at 12:08 AM, Kyle Moffett <[email protected]> wrote:
> On Thu, Oct 30, 2008 at 10:38 PM, J.R. Mauro <[email protected]> wrote:
>> I agree. The only viable automated approach would be something akin to
>> git-submodule but with the ability to integrate the submodule'd
>> project upon import in such a way that things like bisection would
>> follow the main project's ancestry and ignore the submodule's. But
>> that's not possible right now and probably won't be in the foreseeable
>> future.
>
> Here's one relatively easy solution:
>
> (1) Fetch the to-be-merged branch of the "child" project into a
> repository with the parent project ($KIDBRANCH)
>
> (2) Find a branch on which you are going to rebase the child
> project's history ($BASEBRANCH)
>
> (3) Determine a directory into which you are going to put the child
> project's history ($SUBDIR)
>
> (4) Identify the first revision of the child project by looking at
> the last (first-in-time) entry in "git log" of that branch.
> ($KIDSTART)
>
> (5) Graft the kid's history onto the parent's history at that point:
> echo "$KIDSTART $(git rev-parse --verify $BASEBRANCH)" >.git/info/grafts
>
> (6) Rewrite the kid's history as a subdir of that version of the parent:
> git filter-branch --index-filter 'git read-tree master && git
> read-tree --prefix=drivers/newkiddriver/ "$GIT_COMMIT"' --
> "$BASEBRANCH..$KIDBRANCH"
>
> Now the history of $KIDBRANCH will appear to be a series of commits
> which add files in $SUBDIR of the parent project, without any changes
> at all to the parent project's tree. You can then do a merge of what
> appears to be a normal development branch into the main repository.
>
> Since none of the files are even remotely referenced, there can be no
> possible build failures (hence bisection is not broken). You may then
> add commits which add Kconfig options and Makefile references and
> happily develop away.
>
> Cheers,
> Kyle Moffett
>
So this would keep their history but make it look like their first
commit (years ago) happened today? Or do I misunderstand you?
If it is, this is basically what I was trying to propose, except I
didn't have nearly enough git-fu to put it like that.
But then if Greg thinks the history is rubbish, I doubt it will get
imported, although I think these methods are worth experimenting with
for the future.
~J.R.
On Fri, Oct 31, 2008 at 8:57 AM, J.R. Mauro <[email protected]> wrote:
> On Fri, Oct 31, 2008 at 12:08 AM, Kyle Moffett <[email protected]> wrote:
>> Now the history of $KIDBRANCH will appear to be a series of commits
>> which add files in $SUBDIR of the parent project, without any changes
>> at all to the parent project's tree. You can then do a merge of what
>> appears to be a normal development branch into the main repository.
>>
>> Since none of the files are even remotely referenced, there can be no
>> possible build failures (hence bisection is not broken). You may then
>> add commits which add Kconfig options and Makefile references and
>> happily develop away.
>
> So this would keep their history but make it look like their first
> commit (years ago) happened today? Or do I misunderstand you?
Not really, since GIT will preserve the original commit date
information when doing the filtering. Essentially it's equivalent to
taking some ancient long git branch of a project and doing a "git
rebase" onto a recent kernel. The difference is that there is no
common ancestor and the entire contents of one are mapped onto a
subdirectory of the other. I'm trying to figure out if this would be
a useful addition to the "git rebase" functionality (since that's
essentially what it is).
> But then if Greg thinks the history is rubbish, I doubt it will get
> imported, although I think these methods are worth experimenting with
> for the future.
Oh, I very much agree that importing a bunch of dodgy changelogs is
hardly worth it. The biggest difference between the Linux kernel and
most other software projects is the Linux kernel has some of the best
changelogs and one of the most bisectable histories.
Greg did say he had another project he wanted to import, though, so I
figured I'd post another suggestion about how to merge the projects
histories together.
Cheers,
Kyle Moffett