Doing the BK Thing, Penguin-Style
This set of notes is intended mainly for kernel developers, occasional
or full-time, but sysadmins and power users may find parts of it useful
as well. It assumes at least a basic familiarity with CVS, both at a
user level (use on the cmd line) and at a higher level (client-server model).
Due to the author's background, an operation may be described in terms
of CVS, or in terms of how that operation differs from CVS.
This is -not- intended to be BitKeeper documentation. Always run
"bk help <command>" or in X "bk helptool <command>" for reference
documentation.
BitKeeper Concepts
------------------
In the true nature of the Internet itself, BitKeeper is a distributed
system. When applied to revision control, this means doing away with
client-server, and changing to a parent-child model... essentially
peer-to-peer. On the developer's end, this also represents a
fundamental disruption in the standard workflow of changes, commits,
and merges. You will need to take a few minutes to think about
how to best work under BitKeeper, and re-optimize things a bit.
In some sense it is a bit radical, because it might described as
tossing changes out into a maelstrom and having them them magically
land at the right destination... but I'm getting ahead of myself.
Let's start with this progression:
Each BitKeeper source tree on disk is a repository unto itself.
Each repository has a parent.
Each repository contains a set of a changsets ("csets").
Each cset is one or more changed files, bundled together.
Each tree is a repository, so all changes are checked into the local
tree. When a change is checked in, all modified files are grouped
into a logical unit, the changeset. Internally, BK links these
changesets in a tree, representing various converging and diverging
lines of development. These changesets are the bread and butter of
the BK system.
After the concept of changesets, the next thing you need to get used
to having multiple copies of source trees lying around. This -really-
takes some getting used to, for some people. Separate source trees
are the means in BitKeeper by which you delineate parallel lines
of development, both minor and major. What would be branches in
CVS become separate source trees, or "clones" in BitKeeper [heh,
or Star Wars] terminology.
Clones and changesets are the tools from which most of the power of
BitKeeper is derived. As mentioned earlier, each clone has a parent,
the tree used as the source when the new clone was created. In a
CVS-like setup, the parent would be a remote server on the Internet,
and the child is your local clone of that tree.
Once you have established a common baseline between two source trees --
a common parent -- then you can merge changesets between those two
trees with ease. Merging changes into a tree is called a "pull", and
is analagous to 'cvs update'. A pull downloads all the changesets in
the remote tree you do not have, and merges them. Sending changes in
one tree to another tree is called a "push". Push sends all changes
in the local tree the remote does not yet have, and merges them.
>From these concepts come some initial command examples:
1) bk clone -q http://linux.bkbits.net/linux-2.5 linus-2.5
Download a 2.5 stock kernel tree, naming it "linus-2.5" in the local dir.
The "-q" disables listing every single file as it is downloaded.
2) bk clone -ql linus-2.5 alpha-2.5
Create a separate source tree for the Alpha AXP architecture.
The "-l" uses hard links instead of copying data, since both trees are
on the local disk. You can also replace the above with "bk lclone -q ..."
You only clone a tree -once-. After cloning the tree lives a long time
on disk, being updating by pushes and pulls.
3) cd alpha-2.5 ; bk pull http://gkernel.bkbits.net/alpha-2.5
Download changes in "alpha-2.5" repository which are not present
in the local repository, and merge them into the source tree.
4) bk -r co -q
Because every tree is a repository, files must be checked out before
they will be in their standard places in the source tree.
5) bk vi fs/inode.c # example change...
bk citool # checkin, using X tool
bk push bk://[email protected]/alpha-2.5 # upload change
Typical example of a BK sequence that would replace the analagous CVS
situation,
vi fs/inode.c
cvs commit
As this is just supposed to be a quick BK intro, for more in-depth
tutorials, live working demos, and docs, see http://www.bitkeeper.com/
BK and Kernel Development Workflow
----------------------------------
Currently the latest 2.5 tree is available via "bk clone $URL"
and "bk pull $URL" at http://linux.bkbits.net/linux-2.5
This should change in a few weeks to a kernel.org URL.
A big part of using BitKeeper is organizing the various trees you have
on your local disk, and organizing the flow of changes among those
trees, and remote trees. If one were to graph the relationships between
a desired BK setup, you are likely to see a few-many-few graph, like
this:
linux-2.5
|
merge-to-linus-2.5
/ | |
/ | |
vm-hacks bugfixes filesys personal-hacks
\ | | /
\ | | /
\ | | /
testing-and-validation
Since a "bk push" sends all changes not in the target tree, and
since a "bk pull" receives all changes not in the source tree, you want
to make sure you are only pushing specific changes to the desired tree,
not all changes from "peer parent" trees. For example, pushing a change
from the testing-and-validation tree would probably be a bad idea,
because it will push all changes from vm-hacks, bugfixes, filesys, and
personal-hacks trees into the target tree.
One would typically work on only one "theme" at a time, either
vm-hacks or bugfixes or filesys, keeping those changes isolated in
their own tree during development, and only merge the isolated with
other changes when going upstream (to Linus or other maintainers) or
downstream (to your "union" trees, like testing-and-validation above).
It should be noted that some of this separation is not just recommended
practice, it's actually [for now] -enforced- by BitKeeper. BitKeeper
requires that changesets maintain a certain order, which is the reason
that "bk push" sends all local changesets the remote doesn't have. This
separation may look like a lot of wasted disk space at first, but it
helps when two unrelated changes may "pollute" the same area of code, or
don't follow the same pace of development, or any other of the standard
reasons why one creates a development branch.
Small development branches (clones) will appear and disappear:
-------- A --------- B --------- C --------- D -------
\ /
-----short-term devel branch-----
While long-term branches will parallel a tree (or trees), with period
merge points. In this first example, we pull from a tree (pulls,
"\") periodically, such a what occurs when tracking changes in a
vendor tree, never pushing changes back up the line:
-------- A --------- B --------- C --------- D -------
\ \ \
----long-term devel branch-----------------
And then a more common case in Linux kernel development, a long term
branch with periodic merges back into the tree (pushes, "/"):
-------- A --------- B --------- C --------- D -------
\ \ / \
----long-term devel branch-----------------
Submitting Changes to Linus
---------------------------
There's a bit of an art, or style, of submitting changes to Linus.
Since Linus's tree is now (you might say) fully integrated into the
distributed BitKeeper system, there are several prerequisites to
properly submitting a BitKeeper change. All these prereq's are just
general cleanliness of BK usage, so as people become experts at BK, feel
free to optimize this process further (assuming Linus agrees, of
course).
0) Make sure your tree was originally cloned from the linux-2.5 tree
created by Linus. If your tree does not have this as its ancestor, it
is impossible to reliably exchanges changesets.
1) Pay attention to your commit text. The commit message that
accompanies each changeset you submit will live on forever in history,
and is used by Linus to accurately summarize the changes in each
pre-patch. Remember that there is no context, so
"fix for new scheduler changes"
would be too vague, but
"fix mips64 arch for new scheduler switch_to(), TIF_xxx semantics"
would be much better.
You can and should use the command "bk comment -C<rev>" to update the
commit text, and improve it after the fact. This is very useful for
development: poor, quick descriptions during development, which get
cleaned up using "bk comment" before issuing the "bk push" to submit the
changes.
2) Include an Internet-available URL for Linus to pull from, such as
Pull from: http://gkernel.bkbits.net/net-drivers-2.5
3) Include a summary and "diffstat -p1" of each changeset that will be
downloaded, when Linus issues a "bk pull". The author auto-generates
these summaries using "bk push -nl <parent> 2>&1", to obtain a listing
of all the pending-to-send changesets, and their commit messages.
It is important to show Linus what he will be downloading when he issues
a "bk pull", to reduce the time required to sift the changes once they
are downloaded to Linus's local machine.
IMPORTANT NOTE: One of the features of BK is that your repository does
not have to be up to date, in order for Linus to receive your changes.
It is considered a courtesy to keep your repository fairly recent, to
lessen any potential merge work Linus may need to do.
4) Split up your changes. Each maintainer<->Linus situation is likely
to be slightly different here, so take this just as general advice. The
author splits up changes according to "themes" when merging with Linus.
Simultaneous pushes from local development to goes special trees which
exist solely to house changes "queued" for Linus. Example of the trees:
net-drivers-2.5 -- on-going net driver maintenance
vm-2.5 -- VM-related changes
fs-2.5 -- filesystem-related changes
Linus then has much more freedom for pulling changes. He could (for
example) issue a "bk pull" on vm-2.5 and fs-2.5 trees, to merge their
changes, but hold off net-drivers-2.5 because of a change that needs
more discussion.
Other maintainers may find that a single linus-pull-from tree is
adequate for passing BK changesets to him.
Frequently Answered Questions
-----------------------------
1) How do I change the e-mail address shown in the changelog?
A. When you run "bk citool" or "bk commit", set environment
variables BK_USER and BK_HOST to the desired username
and host/domain name.
2) How do I use tags / get a diff between two kernel versions?
A. Pass the tags Linus uses to 'bk export'.
ChangeSets are in a forward-progressing order, so it's pretty easy
to get a snapshot starting and ending at any two points in time.
Linus puts tags on each release and pre-release, so you could use
these two examples:
bk export -tpatch -hdu -rv2.5.4,v2.5.5 | less
# creates patch-2.5.5 essentially
bk export -tpatch -du -rv2.5.5-pre1,v2.5.5 | less
# changes from pre1 to final
A tag is just an alias for a specific changeset... and since changesets
are ordered, a tag is thus a marker for a specific point in time (or
specific state of the tree).
Jeff Garzik wrote:
> 3) Include a summary and "diffstat -p1" of each changeset that will be
> downloaded, when Linus issues a "bk pull". The author auto-generates
> these summaries using "bk push -nl <parent> 2>&1", to obtain a listing
> of all the pending-to-send changesets, and their commit messages.
Two notes I wanted to get out before the rewrite:
1) The preferred format is likely to be a -single- diffstat followed by
a changeset, and
2) "bk changes -L <local-linus-repository>" is the preferred way to get
a listing of changesets not yet sent to Linus, not "bk push -nl"
A script should be appearing in the very near future, which generates
the desired for-Linus changeset format automatically. In the medium
term future and beyond, BK itself will hopefully generate the text.
Jeff
--
Jeff Garzik | XXX FREE! secure AFSPC AK-47 unclassified CDC
Building 1024 | NATO SAS CDMA fun with filters Bellcore kibo SSL
MandrakeSoft | high security goat clones infowar 2600 Magazine
On Thu, Feb 21, 2002 at 11:13:38AM -0500, Jeff Garzik wrote:
> Let's start with this progression:
> Each BitKeeper source tree on disk is a repository unto itself.
> Each repository has a parent.
except the official Linus one, http://linux.bkbits.net
> Submitting Changes to Linus
I would really like this section to be splitted in two:
- one for Linus' lieutenants:
your explanation, with tree to pull from is ok.
- one for occasionnal contributors:
either plain patch on l-k
or
plain patch + BK changeset on l-k (using the
Andreas Dilger script wrapper maybe, see below)
> You can and should use the command "bk comment -C<rev>" to update the
> commit text, and improve it after the fact.
Or use bk revtool in X.
Stelian.
#!/bin/sh
# A script to format BK changeset output in a manner that is easy to read.
# Andreas Dilger <[email protected]> 13/02/2002
PROG=bksend
usage() {
echo "usage: $PROG -r<rev>"
echo -e "\twhere <rev> is of the form '1.23', '1.23..', '1.23..1.27',"
echo -e "\tor '+' to indicate the most recent revision"
exit 1
}
case $1 in
-r) REV=$2; shift ;;
-r*) REV=`echo $1 | sed 's/^-r//'` ;;
*) echo "$PROG: no revision given, you probably don't want that";;
esac
[ -z "$REV" ] && usage
bk changes -r$REV
bk export -tpatch -du -h -r$REV
echo -e "\n================================================================\n\n"
bk send -wgzip_uu -r$REV -
--
Stelian Pop <[email protected]>
Alcove - http://www.alcove.com
> Comments welcome...
Jeff, this is a kind initiative from you. I've
downloaded bk but didn't have time nor
courage to install it and test it yet, but with
this documentation, I think I soon will.
Thanks
Willy
___________________________________________________________
Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en fran?ais !
Yahoo! Mail : http://fr.mail.yahoo.com
On Feb 21, 2002 17:40 +0100, Stelian Pop wrote:
> On Thu, Feb 21, 2002 at 11:13:38AM -0500, Jeff Garzik wrote:
> > Submitting Changes to Linus
>
> I would really like this section to be splitted in two:
>
> - one for Linus' lieutenants:
> your explanation, with tree to pull from is ok.
>
> - one for occasionnal contributors:
> either plain patch on l-k
> or
> plain patch + BK changeset on l-k (using the
> Andreas Dilger script wrapper maybe, see below)
I agree (not just because Stelian mentioned my script ;-). To burden
bkbits.net or any other site to keep a BK repository online just to
submit a trivial change from an occasional contributor is too much.
BK can easily handle CSETs sent inline with the mail, and if Linus
has the original email saying "pull from here" it could just as well
have a small CSET attached at the end. Proposed wording:
For occasional contributors who want to use BK, it is possible
to send a BK changeset directly in the mail. In order to make
it easier to view what is in the changeset, you need to include
a good description of the change (as always) and both a unified
diff and the changeset. Since the unified diff is the readable
part of the message, it makes sense to make the BK changeset part
as small as possible. This can be done by using one of the BK
wrappers when generating the changeset. For example:
To: <maintainer>
Subject: [PATCH] fixes to FAT filesystem detection
This change fixes the FAT filesystem code so that it does not
break when we try to mount something that isn't a FAT filesystem.
You can import this changeset into BK by piping this whole message to
'bk receive [path to repository]' or apply the patch as usual.
=================================================================
<changeset comment: 'bk changes -r<rev>'>
<patch summary: 'bk export -t patch -du -h -r<rev> | diffstat'>
<unified diff: 'bk export -t patch -du -h -r<rev>'>
=================================================================
<compressed changeset: 'bk send -wgzip_uu -r<rev> -'>
(this may become part of BK itself so at that time we can just say "use
'bk send -linus -r<rev>' to format the output".
New version of script which includes diffstat output below.
Cheers, Andreas
=============================================================================
#!/bin/sh
# A script to format BK changeset output in a manner that is easy to read.
# Andreas Dilger <[email protected]> 13/02/2002
#
# Add diffstat output after Changelog <[email protected]> 21/02/2002
PROG=bksend
usage() {
echo "usage: $PROG -r<rev>"
echo -e "\twhere <rev> is of the form '1.23', '1.23..', '1.23..1.27',"
echo -e "\tor '+' to indicate the most recent revision"
exit 1
}
case $1 in
-r) REV=$2; shift ;;
-r*) REV=`echo $1 | sed 's/^-r//'` ;;
*) echo "$PROG: no revision given, you probably don't want that";;
esac
[ -z "$REV" ] && usage
echo "You can import this changeset into BK by piping this whole message to"
echo "'| bk receive [path to repository]' or apply the patch as usual."
SEP="\n===================================================================\n\n"
echo -e $SEP
bk changes -r$REV
echo
bk export -tpatch -du -h -r$REV | diffstat
echo; echo
bk export -tpatch -du -h -r$REV
echo -e $SEP
bk send -wgzip_uu -r$REV -
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
On Thu, Feb 21, 2002 at 11:40:16AM -0700, Andreas Dilger wrote:
> Proposed wording:
[...]
> To: <maintainer>
> Subject: [PATCH] fixes to FAT filesystem detection
Maybe make this BKPATCH to make clear to the maintainer that
there is a BK changeset at the end...
Stelian.
--
Stelian Pop <[email protected]>
Alcove - http://www.alcove.com
On Thu, 21 Feb 2002, Jeff Garzik wrote:
> Comments welcome...
| Submitting Changes to Linus
| ---------------------------
[...]
| 2) Include an Internet-available URL for Linus to pull from, such as
|
| Pull from: http://gkernel.bkbits.net/net-drivers-2.5
|
|
|
| 3) Include a summary and "diffstat -p1" of each changeset that will be
| downloaded, when Linus issues a "bk pull". The author auto-generates
| these summaries using "bk push -nl <parent> 2>&1", to obtain a listing
| of all the pending-to-send changesets, and their commit messages.
|
| It is important to show Linus what he will be downloading when he issues
| a "bk pull", to reduce the time required to sift the changes once they
| are downloaded to Linus's local machine.
|
| IMPORTANT NOTE: One of the features of BK is that your repository does
| not have to be up to date, in order for Linus to receive your changes.
| It is considered a courtesy to keep your repository fairly recent, to
| lessen any potential merge work Linus may need to do.
|
|
| 4) Split up your changes. Each maintainer<->Linus situation is likely
| to be slightly different here, so take this just as general advice. The
| author splits up changes according to "themes" when merging with Linus.
| Simultaneous pushes from local development to goes special trees which
| exist solely to house changes "queued" for Linus. Example of the trees:
|
| net-drivers-2.5 -- on-going net driver maintenance
| vm-2.5 -- VM-related changes
| fs-2.5 -- filesystem-related changes
|
| Linus then has much more freedom for pulling changes. He could (for
| example) issue a "bk pull" on vm-2.5 and fs-2.5 trees, to merge their
| changes, but hold off net-drivers-2.5 because of a change that needs
| more discussion.
|
| Other maintainers may find that a single linus-pull-from tree is
| adequate for passing BK changesets to him.
So what if Linus isn't happy with the changes you made in the for-Him-to-pull
tree? How do I back off (part of the changes)?
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
On Feb 26, 2002 18:45 +0100, Geert Uytterhoeven wrote:
> So what if Linus isn't happy with the changes you made in the for-Him-to-pull
> tree? How do I back off (part of the changes)?
Well, assuming he tells you that there is a problem, run "bk undo -r <rev>.."
to remove the patchset that he doesn't like (in theory). If there have been
a large number of changes after <rev> then they are all removed (there is no
way to remove only a single CSET from within the middle of the tree. Then
you re-do the changes in a way that Linus likes, and submit a new CSET.
You could also add the fix to the same tree and hope he accepts both CSETs,
but I think Linus doesn't want to clutter up his tree with <patch>+<fix>
instead of a single <patch> that was correct in the first place.
In some cases, you are probably better off to export the changes in <rev>
into a diff, get a new Linus BK tree, and re-apply the patches + fixes
and generate a new CSET from that.
Not perfect, but that's life.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
On Tue, 26 Feb 2002, Andreas Dilger wrote:
> On Feb 26, 2002 18:45 +0100, Geert Uytterhoeven wrote:
> > So what if Linus isn't happy with the changes you made in the for-Him-to-pull
> > tree? How do I back off (part of the changes)?
>
> Well, assuming he tells you that there is a problem, run "bk undo -r <rev>.."
> to remove the patchset that he doesn't like (in theory). If there have been
> a large number of changes after <rev> then they are all removed (there is no
> way to remove only a single CSET from within the middle of the tree. Then
> you re-do the changes in a way that Linus likes, and submit a new CSET.
>
> You could also add the fix to the same tree and hope he accepts both CSETs,
> but I think Linus doesn't want to clutter up his tree with <patch>+<fix>
> instead of a single <patch> that was correct in the first place.
>
> In some cases, you are probably better off to export the changes in <rev>
> into a diff, get a new Linus BK tree, and re-apply the patches + fixes
> and generate a new CSET from that.
>
> Not perfect, but that's life.
So in case he wants a few csets only, I have to redo my for-Him-to-pull-tree.
In which case I don't see any advantages compared to emailing a patch with
changeset- and file-specific comments. Especially since setting up a
for-Him-to-pull-tree requires setting up a publically accessible BK server.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
> So in case he wants a few csets only, I have to redo my for-Him-to-pull-tree.
> In which case I don't see any advantages compared to emailing a patch with
> changeset- and file-specific comments. Especially since setting up a
> for-Him-to-pull-tree requires setting up a publically accessible BK server.
You can set up a publically accessible tree here, if you need one,
see the Hosting link on our website. You can make your tree publically
accessible in multiple ways, with varying levels of security, see
"bk help bkd".
The advantage of allowing him to pull is that you won't have the same data
in your BK tree twice, which you would have if you sent him diffs and then
pulled the diffs from him. This is ESPECIALLY IMPORTANT if you have renames
(and creates/deletes are a sort of rename) in your patch.
If the situation is that you've created a scratch tree, specifically for
sending stuff to Linus and you aren't going to use it for anything else
or build on it, then you can send him regular diffs, and toss the tree
once you know he accepted them.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Tue, 26 Feb 2002, Larry McVoy wrote:
> > So in case he wants a few csets only, I have to redo my for-Him-to-pull-tree.
> > In which case I don't see any advantages compared to emailing a patch with
> > changeset- and file-specific comments. Especially since setting up a
> > for-Him-to-pull-tree requires setting up a publically accessible BK server.
>
> You can set up a publically accessible tree here, if you need one,
> see the Hosting link on our website. You can make your tree publically
> accessible in multiple ways, with varying levels of security, see
> "bk help bkd".
OK.
> The advantage of allowing him to pull is that you won't have the same data
> in your BK tree twice, which you would have if you sent him diffs and then
> pulled the diffs from him. This is ESPECIALLY IMPORTANT if you have renames
> (and creates/deletes are a sort of rename) in your patch.
Yes, that's true.
> If the situation is that you've created a scratch tree, specifically for
> sending stuff to Linus and you aren't going to use it for anything else
> or build on it, then you can send him regular diffs, and toss the tree
> once you know he accepted them.
I guess in most cases we (the m68k boys, who got their CVS tree only a few
months ago ;-)) will end up using this `plan B' anyway...
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
On Feb 26, 2002 21:19 +0100, Geert Uytterhoeven wrote:
> So in case he wants a few csets only, I have to redo my for-Him-to-pull-tree.
> In which case I don't see any advantages compared to emailing a patch with
> changeset- and file-specific comments. Especially since setting up a
> for-Him-to-pull-tree requires setting up a publically accessible BK server.
If you want to just send occasional CSETs to Linus or Dave Jones (which
is the category that a large majority of people are in) then you can
always send a CSET in the mail instead of having a "pull" tree available.
Search the l-k archives for my "bksend" script which formats this nicely.
One of the benefits of using BK over patches, even in this situation, is
that if your tree is not exactly the same as Linus' the BK CSET will know
what version of the tree it was generated against and will make applying
and resolving conflicts a lot easier for Linus.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/