As many of you have figured out, I took kernel.bkbits.net (aka bk.kernel.org,
cvs.kernel.org, and svn.kernel.org) of the air yesterday due to the breakin
that attempted to add a trojan horse to the kernel source.
I took it down after talking with Linus and Dave about it, the point was to
shut down the disk drive so that we can go do forensics on it after the fact
and see what we can figure out. Maybe someone can track down who caused the
problem.
This means someone has to go down to the colo with a new disk and do
an install and we've been too busy to do this. Would anyone object if
this wasn't done until this weekend? We're pretty booked up here with
other work. Last I checked only about 6 IP addresses where using the CVS
server, I've never checked on the SVN server (Ben? You have any idea?).
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Larry McVoy wrote:
> This means someone has to go down to the colo with a new disk and do
> an install and we've been too busy to do this. Would anyone object if
> this wasn't done until this weekend? We're pretty booked up here with
> other work. Last I checked only about 6 IP addresses where using the CVS
> server, I've never checked on the SVN server (Ben? You have any idea?).
5 IP addresses, I switched to bk yesterday night ;)
- --
Clemens Schwaighofer - IT Engineer & System Administration
==========================================================
Tequila Japan, 6-17-2 Ginza Chuo-ku, Tokyo 104-8167, JAPAN
Tel: +81-(0)3-3545-7703 Fax: +81-(0)3-3545-7343
http://www.tequila.jp
==========================================================
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQE/q2uIjBz/yQjBxz8RAqozAJ4z8DQsfWuudcl2rHrxMi7nCO7kggCgoO43
dAMfTKS/ag3o81aqavLNzAk=
=71I2
-----END PGP SIGNATURE-----
Followup to: <[email protected]>
By author: Larry McVoy <[email protected]>
In newsgroup: linux.dev.kernel
>
> As many of you have figured out, I took kernel.bkbits.net (aka bk.kernel.org,
> cvs.kernel.org, and svn.kernel.org) of the air yesterday due to the breakin
> that attempted to add a trojan horse to the kernel source.
>
> I took it down after talking with Linus and Dave about it, the point was to
> shut down the disk drive so that we can go do forensics on it after the fact
> and see what we can figure out. Maybe someone can track down who caused the
> problem.
>
> This means someone has to go down to the colo with a new disk and do
> an install and we've been too busy to do this. Would anyone object if
> this wasn't done until this weekend? We're pretty booked up here with
> other work. Last I checked only about 6 IP addresses where using the CVS
> server, I've never checked on the SVN server (Ben? You have any idea?).
>
That doesn't include anyone who uses the mirrored repository on the
main kernel.org machines. Doubt it's a big deal with the timing,
though.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
If you send me mail in HTML format I will assume it's spam.
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
> That doesn't include anyone who uses the mirrored repository on the
> main kernel.org machines.
Last I checked, kernel.org isn't offering pserver access, just ftp. If you
want to take over the CVS access just say the word.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Sun, 9 Nov 2003, Larry McVoy wrote:
> On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
> > That doesn't include anyone who uses the mirrored repository on the
> > main kernel.org machines.
>
> Last I checked, kernel.org isn't offering pserver access, just ftp. If you
> want to take over the CVS access just say the word.
It is faster for me to use rsync on the CVS root locally, and then use the
local repository instead. Rsync is better than CVS when it comes to syncs.
Cvsps expecially, really wants a local repository when you start playing
heavily with -g.
- Davide
On Sun, Nov 09, 2003 at 07:42:09AM -0800, Davide Libenzi wrote:
> On Sun, 9 Nov 2003, Larry McVoy wrote:
>
> > On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
> > > That doesn't include anyone who uses the mirrored repository on the
> > > main kernel.org machines.
> >
> > Last I checked, kernel.org isn't offering pserver access, just ftp. If you
> > want to take over the CVS access just say the word.
>
> It is faster for me to use rsync on the CVS root locally, and then use the
> local repository instead. Rsync is better than CVS when it comes to syncs.
> Cvsps expecially, really wants a local repository when you start playing
> heavily with -g.
If that's the preferred interface then maybe we should shut down the pserver
completely and let people rsync it from kernel.org.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Larry McVoy wrote:
> On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
>
>>That doesn't include anyone who uses the mirrored repository on the
>>main kernel.org machines.
>
> Last I checked, kernel.org isn't offering pserver access, just ftp. If you
> want to take over the CVS access just say the word.
>
No, we don't do the pserver access, but some people get the whole
repository through the mirror. The current division seems to work well,
so I don't see any reason to change it.
-hpa
On Sun, 9 Nov 2003, Larry McVoy wrote:
> If that's the preferred interface then maybe we should shut down the pserver
> completely and let people rsync it from kernel.org.
Larry, that's just my way. Don't make me blame by others to have you shut
down the pserver :-)
- Davide
On Sun, Nov 09, 2003 at 11:28:54AM -0800, H. Peter Anvin wrote:
> Larry McVoy wrote:
> >On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
> >
> >>That doesn't include anyone who uses the mirrored repository on the
> >>main kernel.org machines.
> >
> >Last I checked, kernel.org isn't offering pserver access, just ftp. If you
> >want to take over the CVS access just say the word.
> >
>
> No, we don't do the pserver access, but some people get the whole
> repository through the mirror. The current division seems to work well,
> so I don't see any reason to change it.
Except for the higher likelihood of pserver being an exploit vector.
--
Matt Mackall : http://www.selenic.com : Linux development and consulting
Matt Mackall wrote:
>
> Except for the higher likelihood of pserver being an exploit vector.
>
True... for that it would be nice to have the pserver run in an isolated
environment (separate server, UML, ...) on a copy of the tree. That's
more work, though...
-hpa
On Sun, Nov 09, 2003 at 07:42:09AM -0800, Davide Libenzi wrote:
> On Sun, 9 Nov 2003, Larry McVoy wrote:
>
> > On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
> > > That doesn't include anyone who uses the mirrored repository on the
> > > main kernel.org machines.
> >
> > Last I checked, kernel.org isn't offering pserver access, just ftp. If you
> > want to take over the CVS access just say the word.
>
> It is faster for me to use rsync on the CVS root locally, and then use the
> local repository instead. Rsync is better than CVS when it comes to syncs.
> Cvsps expecially, really wants a local repository when you start playing
> heavily with -g.
same here, however the rsync export on kernel.org is lacking a two
sequence number locking that would allow us to checkout a coherent copy
of the cvs repository. Currently it works by luck.
if we add the two sequence numbers to the rsync on kernel.org, I believe
shutting down the pserver is ok.
On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> same here, however the rsync export on kernel.org is lacking a two
> sequence number locking that would allow us to checkout a coherent copy
> of the cvs repository. Currently it works by luck.
Peter, how the current rsync BKCVS root is updated at kernel.org? Is
coherency guaranteed in some way?
- Davide
On Mon, Nov 10, 2003 at 08:49:20AM -0800, Davide Libenzi wrote:
> On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
>
> > same here, however the rsync export on kernel.org is lacking a two
> > sequence number locking that would allow us to checkout a coherent copy
> > of the cvs repository. Currently it works by luck.
>
> Peter, how the current rsync BKCVS root is updated at kernel.org? Is
> coherency guaranteed in some way?
our rsync isn't going to be coherency guaranteed anyways, no matter how
the updates shows up on kernel.org, rsync can't even hang on per-file
locks, sure it can't be coherent as a whole.
The best way to fix this isn't to add locking to rsync, but to add two
files inside or outside the tree, each one is a sequence number, so you
fetch file1 first, then you rsync and you fetch file2, then you compare
them. If they're the same, your rsync copy is coherent. It's the same
locking we introduced with vgettimeofday.
Ideally rsync could learn to check the sequence numbers by itself but I
don't mind a bit of scripting outside of rsync.
On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> the updates shows up on kernel.org, rsync can't even hang on per-file
> locks, sure it can't be coherent as a whole.
>
> The best way to fix this isn't to add locking to rsync, but to add two
> files inside or outside the tree, each one is a sequence number, so you
> fetch file1 first, then you rsync and you fetch file2, then you compare
> them. If they're the same, your rsync copy is coherent. It's the same
> locking we introduced with vgettimeofday.
>
> Ideally rsync could learn to check the sequence numbers by itself but I
> don't mind a bit of scripting outside of rsync.
Wouldn't a simpler "stop-rsync -> update-root -> start-rsync" work? If
you'll hit an update you will get a error from your local rsync, that will
let you know to retry the operation.
- Davide
Davide Libenzi wrote:
> On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
>
>
>>the updates shows up on kernel.org, rsync can't even hang on per-file
>>locks, sure it can't be coherent as a whole.
>>
>>The best way to fix this isn't to add locking to rsync, but to add two
>>files inside or outside the tree, each one is a sequence number, so you
>>fetch file1 first, then you rsync and you fetch file2, then you compare
>>them. If they're the same, your rsync copy is coherent. It's the same
>>locking we introduced with vgettimeofday.
>>
>>Ideally rsync could learn to check the sequence numbers by itself but I
>>don't mind a bit of scripting outside of rsync.
>
> Wouldn't a simpler "stop-rsync -> update-root -> start-rsync" work? If
> you'll hit an update you will get a error from your local rsync, that will
> let you know to retry the operation.
>
Part of the problem is that there are multiple steps in the rsync chain,
some of which can't be stopped in this way.
The sequence number idea looks sensible to me. Larry, would it be too
much work to have the cvs repository generator generate these files?
-hpa
On Mon, 10 Nov 2003, H. Peter Anvin wrote:
> >>The best way to fix this isn't to add locking to rsync, but to add two
> >>files inside or outside the tree, each one is a sequence number, so you
> >>fetch file1 first, then you rsync and you fetch file2, then you compare
> >>them. If they're the same, your rsync copy is coherent. It's the same
> >>locking we introduced with vgettimeofday.
> >>
> >>Ideally rsync could learn to check the sequence numbers by itself but I
> >>don't mind a bit of scripting outside of rsync.
> >
> > Wouldn't a simpler "stop-rsync -> update-root -> start-rsync" work? If
> > you'll hit an update you will get a error from your local rsync, that will
> > let you know to retry the operation.
>
> Part of the problem is that there are multiple steps in the rsync chain,
> some of which can't be stopped in this way.
>
> The sequence number idea looks sensible to me. Larry, would it be too
> much work to have the cvs repository generator generate these files?
So the update of the rsync repo should do something like:
update file1
update repo
update file2
Isn't it? I do not understand how this guarantee coherency:
Kernel.org Me
get file1 (old value)
update file1 get repo-file1 (old value)
update repo-file1
...
update repo-fileJ
... get repo-fileJ (new value)
update repo-fileN get file2 (old value)
update file2
- Davide
On Mon, Nov 10, 2003 at 10:27:33AM -0800, Davide Libenzi wrote:
> So the update of the rsync repo should do something like:
>
> update file1
> update repo
> update file2
>
> Isn't it? I do not understand how this guarantee coherency:
>
> Kernel.org Me
> get file1 (old value)
> update file1 get repo-file1 (old value)
> update repo-file1
> ...
> update repo-fileJ
> ... get repo-fileJ (new value)
> update repo-fileN get file2 (old value)
> update file2
The kernel.org side goes
update file2
update repo
update file1
--
David Roundy
http://civet.berkeley.edu/droundy/
On Mon, Nov 10, 2003 at 10:27:33AM -0800, Davide Libenzi wrote:
> On Mon, 10 Nov 2003, H. Peter Anvin wrote:
>
> > >>The best way to fix this isn't to add locking to rsync, but to add two
> > >>files inside or outside the tree, each one is a sequence number, so you
> > >>fetch file1 first, then you rsync and you fetch file2, then you compare
> > >>them. If they're the same, your rsync copy is coherent. It's the same
> > >>locking we introduced with vgettimeofday.
> > >>
> > >>Ideally rsync could learn to check the sequence numbers by itself but I
> > >>don't mind a bit of scripting outside of rsync.
> > >
> > > Wouldn't a simpler "stop-rsync -> update-root -> start-rsync" work? If
> > > you'll hit an update you will get a error from your local rsync, that will
> > > let you know to retry the operation.
> >
> > Part of the problem is that there are multiple steps in the rsync chain,
> > some of which can't be stopped in this way.
> >
> > The sequence number idea looks sensible to me. Larry, would it be too
> > much work to have the cvs repository generator generate these files?
>
> So the update of the rsync repo should do something like:
>
> update file1
> update repo
> update file2
>
> Isn't it? I do not understand how this guarantee coherency:
>
> Kernel.org Me
> get file1 (old value)
> update file1 get repo-file1 (old value)
> update repo-file1
> ...
> update repo-fileJ
> ... get repo-fileJ (new value)
> update repo-fileN get file2 (old value)
> update file2
you must pick file2 before file1:
you:
do
get file2
get repo-file1-j
get file1
while file2 != file1 && sleep 10
On Nov 10, 2003 10:27 -0800, Davide Libenzi wrote:
> So the update of the rsync repo should do something like:
>
> update file1
> update repo
> update file2
>
> Isn't it? I do not understand how this guarantee coherency:
>
> Kernel.org Me
> get file1 (old value)
> update file1 get repo-file1 (old value)
> update repo-file1
> ...
> update repo-fileJ
> ... get repo-fileJ (new value)
> update repo-fileN get file2 (old value)
> update file2
The source (kernel.org or bkbits) would update file2 first.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> you must pick file2 before file1:
>
> you:
>
> do
> get file2
> get repo-file1-j
> get file1
> while file2 != file1 && sleep 10
Yes, this closes ;)
- Davide
Andrea Arcangeli wrote:
>
> you must pick file2 before file1:
>
> you:
>
> do
> get file2
> get repo-file1-j
> get file1
> while file2 != file1 && sleep 10
>
Okay... I'm starting to think the sequencing requirements on these files
may be hard to maintain across multiple levels of rsync... but perhaps
I'm wrong, in particular if 'file2' sorts hierachially-lexically last
and 'file1' first...
-hpa
On Mon, Nov 10, 2003 at 11:08:27AM -0800, H. Peter Anvin wrote:
> Andrea Arcangeli wrote:
> >
> > you must pick file2 before file1:
> >
> > you:
> >
> > do
> > get file2
> > get repo-file1-j
> > get file1
> > while file2 != file1 && sleep 10
> >
>
> Okay... I'm starting to think the sequencing requirements on these files
> may be hard to maintain across multiple levels of rsync... but perhaps
> I'm wrong, in particular if 'file2' sorts hierachially-lexically last
> and 'file1' first...
we've to start rsync three times to get them in order, 3 tcp
connections, there's no way to specify the order in the rsync command
line, infact those two sequence files can be as well outside the tree
and we can fetch temporarily in a /tmp/ directory or similar. However we
can probably hack the rsync client to be able to specify the two
sequence numbers on the command line.
It maybe also cleaner to use a slightly more complicated but more
compact algorithm, this would make a potential new rsync command line
option cleaner since only 1 sequence file would need to be specified:
do {
seq = fetch(sequence-file);
if (seq & 1)
break;
rsync
if (seq != fetch(sequence-file))
seq = 1;
} while (seq & 1 && sleep 10 /* ideally exponential backoff */)
this way only 1 sequence-file is needed for each repository that we want
to checkout. the server side only has to increase twice the same file
before and after each update of the repository, so the server side is
even simpler (with the only additional requirement that the sequence
number has to start "even"), only the client side is a bit more complicated.
On Mon, 10 Nov 2003, H. Peter Anvin wrote:
> Andrea Arcangeli wrote:
> >
> > you must pick file2 before file1:
> >
> > you:
> >
> > do
> > get file2
> > get repo-file1-j
> > get file1
> > while file2 != file1 && sleep 10
> >
>
> Okay... I'm starting to think the sequencing requirements on these files
> may be hard to maintain across multiple levels of rsync... but perhaps
> I'm wrong, in particular if 'file2' sorts hierachially-lexically last
> and 'file1' first...
Doing something like:
rsync file2
rsync repo
rsync file1
should work, doesn't it?
- Davide
Andrea Arcangeli wrote:
>
> we've to start rsync three times to get them in order, 3 tcp
> connections, there's no way to specify the order in the rsync command
> line, infact those two sequence files can be as well outside the tree
> and we can fetch temporarily in a /tmp/ directory or similar. However we
> can probably hack the rsync client to be able to specify the two
> sequence numbers on the command line.
>
> It maybe also cleaner to use a slightly more complicated but more
> compact algorithm, this would make a potential new rsync command line
> option cleaner since only 1 sequence file would need to be specified:
>
> do {
> seq = fetch(sequence-file);
> if (seq & 1)
> break;
> rsync
> if (seq != fetch(sequence-file))
> seq = 1;
> } while (seq & 1 && sleep 10 /* ideally exponential backoff */)
>
> this way only 1 sequence-file is needed for each repository that we want
> to checkout. the server side only has to increase twice the same file
> before and after each update of the repository, so the server side is
> even simpler (with the only additional requirement that the sequence
> number has to start "even"), only the client side is a bit more complicated.
>
Good grief. This is messy as hell, and really interferes rather badly
with the whole kernel.org mirror setup.
I guess the "best" solution is to use LVM atomic snapshots, and only
allow rsync off the atomic snapshot. That way any particular rsync
session would always be consistent. That's a *HUGE* amount of work,
though, and still doesn't solve the mirrors issue -- I don't control
what the mirrors run. On the other hand, I don't know how many mirror
sites actually mirror /pub/scm since it's not a requirement.
-hpa
On Mon, Nov 10, 2003 at 11:42:44AM -0800, H. Peter Anvin wrote:
> Good grief. This is messy as hell, and really interferes rather badly
> with the whole kernel.org mirror setup.
allowing coherent checkouts from the mirrors too is an interesting
matter, I believe I've a solution for it, but it obviously requires a
special script to spread the mirror of /pub/scm. However pserver is also
not being mirrored right now, and rsync is still more efficient (and it
carries all the info, not just a certain local copy), so even w/o
mirror-coherency it would be better.
> I guess the "best" solution is to use LVM atomic snapshots, and only
> allow rsync off the atomic snapshot. That way any particular rsync
> session would always be consistent. That's a *HUGE* amount of work,
> though, and still doesn't solve the mirrors issue -- I don't control
> what the mirrors run. On the other hand, I don't know how many mirror
> sites actually mirror /pub/scm since it's not a requirement.
I'm unsure how you can leave an rsync running on the old snapshot and
the new forked off ones running in the new snapshot. as for the mirror
issues it should be possible to make it work like this:
1) increase file1 on the mirror
2) read file2 on the master and store it on the userspace stack
3) copy the tree from master to mirror
4) read file1 on the master and compare it with file2 on the stack
5) if they're different goto 2 after a delay
6) increase file2 on the mirror
basically those sequence numbers should not be copied, they should be
completely separated, each server exporting the tree will have its own
sequence numbers. ideally the master could be at sequence number 1000
and the mirror could be at sequence number 100, depends on the frequency
of the mirror sync and the age of the mirror. the mirror could fetch
multiple updats at once and its sequence number would advance slower, or
it could sync more frequently than there are updates on the server and
its sequence number would advance more quickly than the master.
(for simplicity I'm using the two sequence number model, but clearly the
above can be easily converted to the single sequence number model, and
infact that's preferable since having a single sequence number is
cleaner from a filesystem maintainance point of view, both models are
functionally equivalent)
On Mon, 10 Nov 2003, H. Peter Anvin wrote:
> I guess the "best" solution is to use LVM atomic snapshots, and only
> allow rsync off the atomic snapshot. That way any particular rsync
> session would always be consistent. That's a *HUGE* amount of work,
> though, and still doesn't solve the mirrors issue -- I don't control
> what the mirrors run. On the other hand, I don't know how many mirror
> sites actually mirror /pub/scm since it's not a requirement.
BTW, is rsync.kernel.org::pub/scm/linux/kernel/bkcvs currently being fed
with new data? I don't get any updates.
- Davide
Davide Libenzi wrote:
> On Mon, 10 Nov 2003, H. Peter Anvin wrote:
>
>
>>I guess the "best" solution is to use LVM atomic snapshots, and only
>>allow rsync off the atomic snapshot. That way any particular rsync
>>session would always be consistent. That's a *HUGE* amount of work,
>>though, and still doesn't solve the mirrors issue -- I don't control
>>what the mirrors run. On the other hand, I don't know how many mirror
>>sites actually mirror /pub/scm since it's not a requirement.
>
>
> BTW, is rsync.kernel.org::pub/scm/linux/kernel/bkcvs currently being fed
> with new data? I don't get any updates.
>
It should be... I'll look at it in a bit.
-hpa
On Mon, Nov 10, 2003 at 11:42:44AM -0800, H. Peter Anvin wrote:
> Andrea Arcangeli wrote:
> >
> > we've to start rsync three times to get them in order, 3 tcp
> > connections, there's no way to specify the order in the rsync command
> > line, infact those two sequence files can be as well outside the tree
> > and we can fetch temporarily in a /tmp/ directory or similar. However we
> > can probably hack the rsync client to be able to specify the two
> > sequence numbers on the command line.
> >
> > It maybe also cleaner to use a slightly more complicated but more
> > compact algorithm, this would make a potential new rsync command line
> > option cleaner since only 1 sequence file would need to be specified:
> >
> > do {
> > seq = fetch(sequence-file);
> > if (seq & 1)
> > break;
> > rsync
> > if (seq != fetch(sequence-file))
> > seq = 1;
> > } while (seq & 1 && sleep 10 /* ideally exponential backoff */)
> >
> > this way only 1 sequence-file is needed for each repository that we want
> > to checkout. the server side only has to increase twice the same file
> > before and after each update of the repository, so the server side is
> > even simpler (with the only additional requirement that the sequence
> > number has to start "even"), only the client side is a bit more complicated.
> >
>
> Good grief. This is messy as hell, and really interferes rather badly
> with the whole kernel.org mirror setup.
>
> I guess the "best" solution is to use LVM atomic snapshots, and only
> allow rsync off the atomic snapshot. That way any particular rsync
> session would always be consistent. That's a *HUGE* amount of work,
> though, and still doesn't solve the mirrors issue -- I don't control
> what the mirrors run. On the other hand, I don't know how many mirror
> sites actually mirror /pub/scm since it's not a requirement.
The LVM snapshots would provide the guarantee of consistancy
which would be nice for those who aren't setting it up as
part of their infrastructure but it isn't all that messy.
The issue came up recently on the rsync list of syncing CVS
repositories. The discussion concluded with:
If the history file (or another single file) is enough:
starthist=`ls -l $CVSROOT/CVSROOT/history`
curhist=""
while [ "$Starthist" != "$curhist" ]
do
starthist=`ls -l $CVSROOT/CVSROOT/history`
rsync .....
curhist=`ls -l $CVSROOT/CVSROOT/history`
done
If not you can test the directory.
ls -l $CVSROOT/CVSROOT >$TMPFILE.start
touch $TMPFILE.test
until diff -q $TMPFILE.start $TMPFILE.test >/dev/null
do
ls -l $CVSROOT/CVSROOT >$TMPFILE.start
rsync .....
ls -l $CVSROOT/CVSROOT >$TMPFILE.test
done
rm -f $TMPFILE.start $TMPFILE.test
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: [email protected]
Remember Cernan and Schmitt
Andrea Arcangeli wrote:
>>On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> The best way to fix this isn't to add locking to rsync, but to add two
> files inside or outside the tree, each one is a sequence number, so you
> fetch file1 first, then you rsync and you fetch file2, then you compare
> them. If they're the same, your rsync copy is coherent. It's the same
> locking we introduced with vgettimeofday...
How is this different from writing one file named LOCK while updating
the tree?
I know this is a really basic question, but it also seems like a really
old problem which must have been solved multiple times by now. Am I
really wrong about this?
On Mon, 10 Nov 2003, jw schultz wrote:
> If the history file (or another single file) is enough:
>
> starthist=`ls -l $CVSROOT/CVSROOT/history`
> curhist=""
>
> while [ "$Starthist" != "$curhist" ]
> do
> starthist=`ls -l $CVSROOT/CVSROOT/history`
> rsync .....
> curhist=`ls -l $CVSROOT/CVSROOT/history`
> done
BK2CVS does not compile $CVSROOT/CVSROOT/history, but the second one
should work:
> If not you can test the directory.
>
> ls -l $CVSROOT/CVSROOT >$TMPFILE.start
> touch $TMPFILE.test
>
> until diff -q $TMPFILE.start $TMPFILE.test >/dev/null
> do
> ls -l $CVSROOT/CVSROOT >$TMPFILE.start
> rsync .....
> ls -l $CVSROOT/CVSROOT >$TMPFILE.test
> done
> rm -f $TMPFILE.start $TMPFILE.test
- Davide
On Mon, 10 Nov 2003, walt wrote:
> Andrea Arcangeli wrote:
>
> >>On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> > The best way to fix this isn't to add locking to rsync, but to add two
> > files inside or outside the tree, each one is a sequence number, so you
> > fetch file1 first, then you rsync and you fetch file2, then you compare
> > them. If they're the same, your rsync copy is coherent. It's the same
> > locking we introduced with vgettimeofday...
>
> How is this different from writing one file named LOCK while updating
> the tree?
This is even simpler I believe. If you happen to fetch it, you restart the
rsync. Peter ?
(maybe the name LOCK should be replaced by something more "uniq")
- Davide
Davide Libenzi wrote:
>On Mon, 10 Nov 2003, walt wrote:
>
>
>>Andrea Arcangeli wrote:
>>
>>
>>>>On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
>>>>
>>>The best way to fix this isn't to add locking to rsync, but to add two
>>>files inside or outside the tree, each one is a sequence number, so you
>>>fetch file1 first, then you rsync and you fetch file2, then you compare
>>>them. If they're the same, your rsync copy is coherent. It's the same
>>>locking we introduced with vgettimeofday...
>>>
>>How is this different from writing one file named LOCK while updating
>>the tree?
>>
>
>This is even simpler I believe. If you happen to fetch it, you restart the
>rsync. Peter ?
>(maybe the name LOCK should be replaced by something more "uniq")
>
>
What happens if the the tree is updated while the client is fetching it?
On Mon, 10 Nov 2003, jw schultz wrote:
> If not you can test the directory.
>
> ls -l $CVSROOT/CVSROOT >$TMPFILE.start
> touch $TMPFILE.test
>
> until diff -q $TMPFILE.start $TMPFILE.test >/dev/null
> do
> ls -l $CVSROOT/CVSROOT >$TMPFILE.start
> rsync .....
> ls -l $CVSROOT/CVSROOT >$TMPFILE.test
> done
> rm -f $TMPFILE.start $TMPFILE.test
It does not work either. If CVSROOT files are updated at the end, you can
still fetch some new files and be able to fetch the old CVSROOT before the
update process will be able to do it. I believe that the LOCK file idea
should work pretty nicely. The update process goes like:
create LOCK
update repo
remove LOCK
While the client can simply:
do
rsync
while exist LOCK
- Davide
On Tue, 11 Nov 2003, Nick Piggin wrote:
> What happens if the the tree is updated while the client is fetching it?
Surprise, it breaks :-) Yes, the double file approach is needed (excluding
changes to rsync).
- Davide
Davide Libenzi wrote:
> On Tue, 11 Nov 2003, Nick Piggin wrote:
>
>
>> What happens if the the tree is updated while the client is fetching
>> it [using a single LOCKfile method]?
> Surprise, it breaks :-) Yes, the double file approach is needed
> (excluding changes to rsync).
Sorry to be so dumb, but it seems to me that the two methods are exactly
equivalent in every way:
A test for file1 != file2 is exactly eqivalent to testing LOCK != NULL.
It's a simple binary TRUE/FALSE test.
What am I missing? (BTW I'm not arguing against the two-file method.
I just don't understand why it's different.)
Now, if multiple people are updating the tree at the same time then a
simple TRUE/FALSE test provides insufficient information: you would
then need enough 'bits' of information to count the number of people
updating at the same time.
But this problem is certainly not unique to this group. Anyone using
a source repository has the same problem to deal with. Or am I not
with the program here at all?
On Tuesday 11 Nov 2003 2:14 pm, walt wrote:
> Sorry to be so dumb, but it seems to me that the two methods are exactly
> equivalent in every way:
>
> A test for file1 != file2 is exactly eqivalent to testing LOCK != NULL.
> It's a simple binary TRUE/FALSE test.
>
> What am I missing? (BTW I'm not arguing against the two-file method.
> I just don't understand why it's different.)
>
So you check the lock, do rsync, and check the lock again. But the lock could
have flipped several times during the rsync and you wouldn't know about it.
My preferred solution is a single sequence file as described by Adreas:
Assuming sequence starts at 0,
To modify the repository, +1 to sequence file contents, modify repo, +1 to
sequence
To get a coherent copy,
do
seq1 = read(sequence file)
rsync repo
seq2 = read(sequence file)
until seq1==seq2 and !(seq1&1)
On Mon, Nov 10, 2003 at 11:34:18PM -0800, Davide Libenzi wrote:
> On Mon, 10 Nov 2003, jw schultz wrote:
>
> > If not you can test the directory.
> >
> > ls -l $CVSROOT/CVSROOT >$TMPFILE.start
> > touch $TMPFILE.test
> >
> > until diff -q $TMPFILE.start $TMPFILE.test >/dev/null
> > do
> > ls -l $CVSROOT/CVSROOT >$TMPFILE.start
> > rsync .....
> > ls -l $CVSROOT/CVSROOT >$TMPFILE.test
> > done
> > rm -f $TMPFILE.start $TMPFILE.test
>
> It does not work either. If CVSROOT files are updated at the end, you can
> still fetch some new files and be able to fetch the old CVSROOT before the
> update process will be able to do it. I believe that the LOCK file idea
> should work pretty nicely. The update process goes like:
>
> create LOCK
> update repo
> remove LOCK
>
> While the client can simply:
>
> do
> rsync
> while exist LOCK
that can't work either, the client can't take the lock, rsync is a
readonly thing. the create lock update repo, remove lock will all three
run during the client rsync. the "exist LOCK" will never notice.
unless you want to use snapshots etc.. the seq lock is the only viable
way to checkout coherent with rsync.
hope this isn't going too offtopic for l-k though.
On Mon, Nov 10, 2003 at 08:00:30PM -0800, walt wrote:
> Andrea Arcangeli wrote:
>
> >>On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> >The best way to fix this isn't to add locking to rsync, but to add two
> >files inside or outside the tree, each one is a sequence number, so you
> >fetch file1 first, then you rsync and you fetch file2, then you compare
> >them. If they're the same, your rsync copy is coherent. It's the same
> >locking we introduced with vgettimeofday...
>
> How is this different from writing one file named LOCK while updating
> the tree?
it's very different, your file named LOCK can be created and deleted
while rsync was running, and rsync will never notice. rsync is readonly,
it'll never care about locks.
On Tue, Nov 11, 2003 at 06:37:36PM +1100, Nick Piggin wrote:
>
>
> Davide Libenzi wrote:
>
> >On Mon, 10 Nov 2003, walt wrote:
> >
> >
> >>Andrea Arcangeli wrote:
> >>
> >>
> >>>>On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> >>>>
> >>>The best way to fix this isn't to add locking to rsync, but to add two
> >>>files inside or outside the tree, each one is a sequence number, so you
> >>>fetch file1 first, then you rsync and you fetch file2, then you compare
> >>>them. If they're the same, your rsync copy is coherent. It's the same
> >>>locking we introduced with vgettimeofday...
> >>>
> >>How is this different from writing one file named LOCK while updating
> >>the tree?
> >>
> >
> >This is even simpler I believe. If you happen to fetch it, you restart the
> >rsync. Peter ?
> >(maybe the name LOCK should be replaced by something more "uniq")
> >
> >
>
> What happens if the the tree is updated while the client is fetching it?
the usual problem, and the reason we need a sequence number (increased
before and after the repo update). A file lock not.
On Tue, Nov 11, 2003 at 02:38:47PM +0000, Andrew Walrond wrote:
> So you check the lock, do rsync, and check the lock again. But the lock could
> have flipped several times during the rsync and you wouldn't know about it.
>
> My preferred solution is a single sequence file as described by Adreas:
>
> Assuming sequence starts at 0,
>
> To modify the repository, +1 to sequence file contents, modify repo, +1 to
> sequence
>
> To get a coherent copy,
> do
> seq1 = read(sequence file)
> rsync repo
> seq2 = read(sequence file)
> until seq1==seq2 and !(seq1&1)
yep.
Please pardon my late intrusion into this discussion of what appears to
be CVS mirror coherency. The impression I get, which may be wrong, is
that one or both of these problems is happening:
- A user is pulling from one CVS mirror, but that mirror is in the
process of being updated from the BK source, so the user gets some new
files and some old ones.
- A user is pulling from more than one CVS mirror at the same time, and
not all mirrors are at the same revision level.
Either way, and I would not be surprised if someone else had suggested
this already, but being a graphics person, let me suggest a common
technique for dealing with this problem: double-buffering.
Let's assume a combination of the two above cases, because the general
solution applies to all cases.
To begin with, all mirrors serve from the "front buffer", all of which
are known to be at the same revision level for all files. While serving
the front buffers, the "back buffers" are being updated.
At a prescribed time, all servers go off-line (to deal with
asynchronicity between clocks), swap the front and back buffers, and
then go back online.
The synchronization problem can be dealt with either by having everyone
have their clocks set relatively close but go offline for a few seconds
just in case there is some drift, or there can be a master server which
signals the others when to swap buffers.
Followup to: <[email protected]>
By author: Andrew Walrond <[email protected]>
In newsgroup: linux.dev.kernel
>
> My preferred solution is a single sequence file as described by Adreas:
>
> Assuming sequence starts at 0,
>
> To modify the repository, +1 to sequence file contents, modify repo, +1 to
> sequence
>
> To get a coherent copy,
> do
> seq1 = read(sequence file)
> rsync repo
> seq2 = read(sequence file)
> until seq1==seq2 and !(seq1&1)
>
OK... this still doesn't deal with how to get mirrors to pick stuff up
with a minimum of fuss. The "minimum of fuss" bit is *extremely*
important... I still haven't managed to get all mirrors to use rsync.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
If you send me mail in HTML format I will assume it's spam.
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
On Tuesday 11 Nov 2003 7:43 pm, H. Peter Anvin wrote:
>
> OK... this still doesn't deal with how to get mirrors to pick stuff up
> with a minimum of fuss. The "minimum of fuss" bit is *extremely*
> important... I still haven't managed to get all mirrors to use rsync.
>
1. Don't bother with cvs.Just host a clone of the main bk repository
2. Persuade Larry to release a 'clone/pull-only' version of bk which *anyone*
can use to access open source software
3. Having lit the blue touch paper, run...
Seriously though; There isn't another way that I can see for mirroring cvs
repos coherently, unless cvsup does something clever? Anyone know?
Andrew Walrond
On Tue, 11 Nov 2003, Andrew Walrond wrote:
> 2. Persuade Larry to release a 'clone/pull-only' version of bk which *anyone*
> can use to access open source software
I tried to ask this to Larry but no one else seemed interested at the
time. Personally I would be happy even with only the removal of "you
cannot work/dev on any other SCM" kinda clause.
- Davide
On Tue, Nov 11, 2003 at 08:21:34PM +0000, Andrew Walrond wrote:
> On Tuesday 11 Nov 2003 7:43 pm, H. Peter Anvin wrote:
> >
> > OK... this still doesn't deal with how to get mirrors to pick stuff up
> > with a minimum of fuss. The "minimum of fuss" bit is *extremely*
> > important... I still haven't managed to get all mirrors to use rsync.
>
> 1. Don't bother with cvs.Just host a clone of the main bk repository
There's a good idea :)
> 2. Persuade Larry to release a 'clone/pull-only' version of bk which *anyone*
> can use to access open source software
As I've explained in the past, this doesn't make sense. I'd be far more
likely to build a sort of CVS like client that could do checkouts and
updates of read only files. That's a pretty straightforward thing to
do, in fact, nobody needs BK source to do that, it could all be done as
wrappers pretty trivially. If someone wanted to code that up and make the
code available under a BSD license we'd take a good look at adding that
into the BK server side. It doesn't need to be bundled in BK however.
Anyone could write a daemon that locally called BK to get the data and
a client that talked to the daemon.
The hard part is renames but even that can be handled reasonably easily.
> Seriously though; There isn't another way that I can see for mirroring cvs
> repos coherently, unless cvsup does something clever? Anyone know?
I could make some comment about this being a good example of one of
the zillion little problems we've had to solve but if I go there it's
going to start a flame war. So I won't. I will note that none of the
solutions proposed come close to being acceptable, they all fail on NFS
and on SMB shares. And they don't cascade properly as HPA has noted.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Tue, Nov 11, 2003 at 03:52:15PM -0800, Larry McVoy wrote:
> going to start a flame war. So I won't. I will note that none of the
> solutions proposed come close to being acceptable, they all fail on NFS
> and on SMB shares. And they don't cascade properly as HPA has noted.
FWIW arch would solve this completely too, but we need to checkout a
coherent cvs to import the data into arch.
On Tue, Nov 11, 2003 at 08:21:34PM +0000, Andrew Walrond wrote:
> Seriously though; There isn't another way that I can see for mirroring cvs
> repos coherently, unless cvsup does something clever? Anyone know?
cvsup does something clever.
For non-CVS repositories, it uses the "rsync" algorithm.
For CVS repositories, it will do per-change updates.
I'm not sure what it does if a change disappears from the source
(haven't tried that yet), but I do know that if you are careful enough
to avoid creating conflicting revision numbers in the mirror, you can
*commit* to it as well.
I'm using cvsup for mirroring a CVS repository at work, and it's worked
very well (and is remarkably low load).
I believe the "mirror consistency" problem isn't solved by this
unfortunately, but considering that there are mirrors not using rsync, I
have a feeling that it's not truly solvable, entirely.
I think I can provide an example set of configuration files for cvsup
now that I've configured it 3 times, if anyone is interested.
--
Ryan Anderson
sometimes Pug Majere
Andrea Arcangeli <[email protected]> writes:
> the usual problem, and the reason we need a sequence number (increased
> before and after the repo update). A file lock not.
Or a file that contains md5sums of the other files in the tree.
After the rsync, you recompute the md5sums file, and if it does not match,
rsync again. As a bonus feature, the md5sums file can be pgp-signed.
-- Benoit
> > 2. Persuade Larry to release a 'clone/pull-only' version of bk which
> > *anyone* can use to access open source software
>
> As I've explained in the past, this doesn't make sense. I'd be far more
> likely to build a sort of CVS like client that could do checkouts and
> updates of read only files. That's a pretty straightforward thing to
> do, in fact, nobody needs BK source to do that, it could all be done as
I'm a bit confused (not unusual). I think what I'm suggesting is exactly what
you've just described and doesn't involve releasing any bk source; Release a
binary only tool which will clone and pull only, (Ie can be used to access
open source software but not develop it) which is free of the license
restrictions of the full bk (ie can be used to access open source software by
anybody, regardless of what they might be working on)
Or am I missing something? How does this hurt the bk business model?
> I could make some comment about this being a good example of one of
> the zillion little problems we've had to solve but if I go there it's
> going to start a flame war. So I won't. I will note that none of the
> solutions proposed come close to being acceptable, they all fail on NFS
> and on SMB shares. And they don't cascade properly as HPA has noted.
Absolutely. Bk is, undeniably, brilliant, and would solve all these problems
at a stroke, except that the open source community cannot with good
conscience exclude *anyone* from being able to access the sources.
> On Tue, Nov 11, 2003 at 03:52:15PM -0800, Larry McVoy wrote:
> > going to start a flame war. So I won't. I will note that none of the
> > solutions proposed come close to being acceptable, they all fail on NFS
> > and on SMB shares. And they don't cascade properly as HPA has noted.
>
> FWIW arch would solve this completely too, but we need to checkout a
> coherent cvs to import the data into arch.
Okay, folks. Now i propose to go further than that.
I hope that Larry recognises that at some point the linux kernel community
_will_ switch to a free software alternative.
I think we should have a better choice than the lossy bk->cvs.
I propose we as a whole ask Larry to create a bk->arch gateway _not_ via cvs
but directly, and therefore using all the extended metadata possible.
- this way we`ll get whole-tree changesets, file renames etc.
- this way Larry will have a terrific argument to shut whiners.
The choice of Arch is not random, but motivated by my belief in that it is
the leading Free Software revision control effort.
If Larry wishes to give a birth to this idea, the list to contact is:
[email protected]
regards, Samium Gromoff
On Thu, Nov 13, 2003 at 03:59:42PM +0300, Samium Gromoff wrote:
> Okay, folks. Now i propose to go further than that.
>
> I hope that Larry recognises that at some point the linux kernel community
> _will_ switch to a free software alternative.
>
> I think we should have a better choice than the lossy bk->cvs.
>
> I propose we as a whole ask Larry to create a bk->arch gateway _not_ via cvs
> but directly, and therefore using all the extended metadata possible.
I appreciate Arch and am actually using it for some things, and can't
give an educated opinion on BK as I have never used it.
But this is just ridiculous.
First of all, there always serious pain in switching. Whatever free
alternative is provided better be _a lot_ better compared to what is
currently used. And although Arch might be more in-line with the
political goals of many developers, frankly I don't believe Arch is
technologically there yet.
But then asking Larry to implement a gateway seems to be the plan of the
century. Hey why doesn't Larry help out with the Arch or SVN effort
while he's at it? How would you react if ClearCase started demanding
that Arch/SVN/CVS developers should implement a one-directional
gateway to their software to make it easier for people to migrate.
Jan
On Wed, Nov 12, 2003 at 10:35:22PM +0100, Benoit Poulot-Cazajous wrote:
> Andrea Arcangeli <[email protected]> writes:
>
> > the usual problem, and the reason we need a sequence number (increased
> > before and after the repo update). A file lock not.
>
> Or a file that contains md5sums of the other files in the tree.
> After the rsync, you recompute the md5sums file, and if it does not match,
> rsync again. As a bonus feature, the md5sums file can be pgp-signed.
agreed, this would work too and it has the advantage of working with the
mirrors too as far as the per-file updates are atomic (should always be
the case). This has the only disavanage of forcing the client and the
server to read all file contents (I normally don't rsync with -c). But
if it simplifies life a lot, it's sure acceptable.
Andrea Arcangeli <[email protected]> writes:
> On Wed, Nov 12, 2003 at 10:35:22PM +0100, Benoit Poulot-Cazajous wrote:
>> Andrea Arcangeli <[email protected]> writes:
>>
>> > the usual problem, and the reason we need a sequence number (increased
>> > before and after the repo update). A file lock not.
>>
>> Or a file that contains md5sums of the other files in the tree.
>> After the rsync, you recompute the md5sums file, and if it does not match,
>> rsync again. As a bonus feature, the md5sums file can be pgp-signed.
>
> agreed, this would work too and it has the advantage of working with the
> mirrors too as far as the per-file updates are atomic (should always be
> the case). This has the only disavanage of forcing the client and the
> server to read all file contents (I normally don't rsync with -c).
This is not necessary, you only need to recompute the md5sums of changed
files.
Andreas.
--
Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 N?rnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
At Thu, 13 Nov 2003 09:44:54 -0500,
Jan Harkes wrote:
>
> On Thu, Nov 13, 2003 at 03:59:42PM +0300, Samium Gromoff wrote:
> > Okay, folks. Now i propose to go further than that.
> >
> > I hope that Larry recognises that at some point the linux kernel community
> > _will_ switch to a free software alternative.
> >
> > I think we should have a better choice than the lossy bk->cvs.
> >
> > I propose we as a whole ask Larry to create a bk->arch gateway _not_ via cvs
> > but directly, and therefore using all the extended metadata possible.
>
> I appreciate Arch and am actually using it for some things, and can't
> give an educated opinion on BK as I have never used it.
>
> But this is just ridiculous.
>
> First of all, there always serious pain in switching. Whatever free
> alternative is provided better be _a lot_ better compared to what is
> currently used. And although Arch might be more in-line with the
> political goals of many developers, frankly I don't believe Arch is
> technologically there yet.
You miss my point.
I don`t call for switching (although would like it to happen).
I call for a _better replacement for bk->cvs_.
bk->cvs loses data, which could be captured in a more sensible manner
than cvs is.
(I think that everyone agrees here that it`s not even remotely arguable)
bk->arch would regain that data back to our hands, thats what i`m proposing.
(here i presume that that data being constantly lost _is_ really useful)
> But then asking Larry to implement a gateway seems to be the plan of the
> century. Hey why doesn't Larry help out with the Arch or SVN effort
> while he's at it? How would you react if ClearCase started demanding
> that Arch/SVN/CVS developers should implement a one-directional
> gateway to their software to make it easier for people to migrate.
I decide to conveniently skip answering that part.
(That`s a suggestion to use your imagination here, really)
> Jan
regards, Samium Gromoff
At Thu, 13 Nov 2003 17:26:20 +0300,
Samium Gromoff wrote:
[snip]
> bk->arch would regain that data back to our hands, thats what i`m proposing.
> (here i presume that that data being constantly lost _is_ really useful)
^^^^ should be "the"
[snip]
> regards, Samium Gromoff
On Thu, Nov 13, 2003 at 04:23:24PM +0100, Andreas Schwab wrote:
> This is not necessary, you only need to recompute the md5sums of changed
> files.
agreed, theoretically I could intercept the changed files through rsync
output, it's more tricky but doable.
On Thu, Nov 13, 2003 at 04:23:24PM +0100, Andreas Schwab wrote:
> Andrea Arcangeli <[email protected]> writes:
>
> > On Wed, Nov 12, 2003 at 10:35:22PM +0100, Benoit Poulot-Cazajous wrote:
> >> Andrea Arcangeli <[email protected]> writes:
> >>
> >> > the usual problem, and the reason we need a sequence number (increased
> >> > before and after the repo update). A file lock not.
> >>
> >> Or a file that contains md5sums of the other files in the tree.
> >> After the rsync, you recompute the md5sums file, and if it does not match,
> >> rsync again. As a bonus feature, the md5sums file can be pgp-signed.
> >
> > agreed, this would work too and it has the advantage of working with the
> > mirrors too as far as the per-file updates are atomic (should always be
> > the case). This has the only disavanage of forcing the client and the
> > server to read all file contents (I normally don't rsync with -c).
>
> This is not necessary, you only need to recompute the md5sums of changed
> files.
If we had this approach we wouldn't have caught the torjan horse in the
CVS tree. We checksum all the data, changed or not. Your approach
pushes that duty onto the end users, and let's have a show of hands,
how many of you md5sum every bit of data that you download from the net?
(Please don't reply to that, it's a rhetorical question and I think the
vast majority of the people already know the answer.)
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Thu, Nov 13, 2003 at 10:10:26AM +0000, Andrew Walrond wrote:
> > > 2. Persuade Larry to release a 'clone/pull-only' version of bk which
> > > *anyone* can use to access open source software
> >
> > As I've explained in the past, this doesn't make sense. I'd be far more
> > likely to build a sort of CVS like client that could do checkouts and
> > updates of read only files. That's a pretty straightforward thing to
> > do, in fact, nobody needs BK source to do that, it could all be done as
>
> I'm a bit confused (not unusual). I think what I'm suggesting is exactly what
> you've just described and doesn't involve releasing any bk source; Release a
> binary only tool which will clone and pull only,
>
> Or am I missing something? How does this hurt the bk business model?
It doesn't hurt the BK business model. It also doesn't require any
access to BK source to do it. Anyone could wrap a network daemon
around BK and serve up repositories. So that means you could do it
if it is important to you. So could anyone else.
> > I could make some comment about this being a good example of one of
> > the zillion little problems we've had to solve but if I go there it's
> > going to start a flame war. So I won't. I will note that none of the
> > solutions proposed come close to being acceptable, they all fail on NFS
> > and on SMB shares. And they don't cascade properly as HPA has noted.
>
> Absolutely. Bk is, undeniably, brilliant, and would solve all these problems
> at a stroke, except that the open source community cannot with good
> conscience exclude *anyone* from being able to access the sources.
But noone is excluded from having access to the sources.
I suppose it sounds like we don't want to give out more free engineering
but let's put things into perspective. The CVS server has about 6 users.
It's cost us a pile of money to build and support that technology.
For 6 users. On the other hand, there are thousands if not tens of
thousands of BK users for the kernel alone. About a year ago we added
unique ID's to the repositories themselves (the clones) and a few months
later I counted over 10,000 clones having connected to to openlogging.org
for the kernel tree (people do use BK for other things).
So what do you want? Do you want us to spend what little resources we
have to give you on something that is used by less than 1/100th of one
percent of the people? Are you sure? What about us spending some time
and making performance and functionality of BK itself better? Wouldn't
that be a better use of our time and help dramatically more people?
Remember, there is a limited amount that we can do. Ask for the work
that helps the most people.
I'm starting to (finally) learn that there are a small number of people
who complain loudly (not you Andrew, I just happened to hit reply to your
mail) but they in no way represent the majority of the people doing work.
As far as I can tell, most people are using BK and if there wasn't any
whining they'd be content with that and what they'd really like is for
BK itself to improve (*). Linus doesn't get any benefit from one more
enhancement to the CVS gateway, he doesn't use it. Neither do thousands
of other people. All those people would be one heck of a lot happier
if the BK integrity checks were faster, if the GUI tools were faster,
if we added digital signatures to changesets, the list goes on and on.
As much as I'd love to have zero complaining about BK, I don't see that
ever happening. And it is a mistake for me to listen to that complaining
and help people who dislike BK, our license, and who knows what else.
There are a lot of silent people who are going about getting their job
done using BK and they deserve our help one heck of a lot more than a
bunch of whiners.
--lm
(*) Insert obligatory statement about everyone also wanting BK GPLed,
a chicken in every pot, a porsche in every garage, and a supermodel with
a PhD opening up your door to greet you with a smile at the end of your
hard day.
On Thu, Nov 13, 2003 at 08:29:45AM -0800, Larry McVoy wrote:
> If we had this approach we wouldn't have caught the torjan horse in the
> CVS tree. We checksum all the data, changed or not. Your approach
> pushes that duty onto the end users, and let's have a show of hands,
the suggestion of the md5sum wasn't related to keeping the data
secure/robust, we don't want to move the "robustness" duty onto the end
users of course, we only want to know when we can break the rsync loop.
The fact we'll do a further check possibly with a signature on the
md5sums, is a bonus, but it's not meant to replace in any way the
robusteness effort on the server side and we don't do it for robustness,
we do it only for providing a means of coherency to the changesets in
the tree.
On Thu, Nov 13, 2003 at 08:27:13AM -0800, Larry McVoy wrote:
> I'm starting to (finally) learn that there are a small number of people
> who complain loudly (not you Andrew, I just happened to hit reply to your
> mail) but they in no way represent the majority of the people doing work.
> As far as I can tell, most people are using BK and if there wasn't any
> whining they'd be content with that and what they'd really like is for
> BK itself to improve (*). Linus doesn't get any benefit from one more
> ...
Larry: You tried to get it working, and keep it working. To me, this is
more than enough effort. The whiners (yes, whiners - every participant has
the opportunity of *doing* something - choosing to complain and demand
that others *do* something is whining in my books) are an unfortunate fact
of life.
mark
--
[email protected]/[email protected]/[email protected] __________________________
. . _ ._ . . .__ . . ._. .__ . . . .__ | Neighbourhood Coder
|\/| |_| |_| |/ |_ |\/| | |_ | |/ |_ |
| | | | | \ | \ |__ . | | .|. |__ |__ | \ |__ | Ottawa, Ontario, Canada
One ring to rule them all, one ring to find them, one ring to bring them all
and in the darkness bind them...
http://mark.mielke.cc/
On Thu, 13 Nov 2003, Larry McVoy wrote:
> I suppose it sounds like we don't want to give out more free engineering
> but let's put things into perspective. The CVS server has about 6 users.
> It's cost us a pile of money to build and support that technology.
> For 6 users. On the other hand, there are thousands if not tens of
Larry, if there are really six users (i'm one of them, rsync) among
pserver and rsync access, I am the first to tell you shut it down. It is
not worth. On the other hand IIRC it was you that, when Pavel showed up
with the bitbucket hack to extract metadata from BK, volunteered to do it
internally inside BM. Do I remember correctly?
- Davide
On Thu, Nov 13, 2003 at 09:14:27AM -0800, Davide Libenzi wrote:
> On Thu, 13 Nov 2003, Larry McVoy wrote:
>
> > I suppose it sounds like we don't want to give out more free engineering
> > but let's put things into perspective. The CVS server has about 6 users.
> > It's cost us a pile of money to build and support that technology.
> > For 6 users. On the other hand, there are thousands if not tens of
>
> Larry, if there are really six users (i'm one of them, rsync) among
> pserver and rsync access, I am the first to tell you shut it down. It is
> not worth. On the other hand IIRC it was you that, when Pavel showed up
> with the bitbucket hack to extract metadata from BK, volunteered to do it
> internally inside BM. Do I remember correctly?
Not really, the revision history of the CVS gateway predates Pavel's so-called
hacks.
I don't mind us supporting this gateway for the small number of users, if it
makes them happy, that's fine. What I mind is people coming back and asking
for more stuff for a tiny number of people. That doesn't make sense. We
should put our efforts into helping the people using BK, not the people using
CVS. If you want to help yourselves, that's a fine idea, go for it.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Davide Libenzi wrote:
>Larry, if there are really six users (i'm one of them, rsync) among
>pserver and rsync access, I am the first to tell you shut it down. It is
>not worth. On the other hand IIRC it was you that, when Pavel showed up
>with the bitbucket hack to extract metadata from BK, volunteered to do it
>internally inside BM. Do I remember correctly?
>
>
>
>- Davide
>
>
As one of the six, I would happily 2nd the shutting down of the
pserver...rsync is fine with me. I would actually prefer no CVS archive
at all as long as the raw changesets were rsyncable...then the community
would be responsible for doing something useful with them instead of BM.
-Tupshin
This is degenerating, but I don't want anyone mistaking my position, so...
For the record and so nobody gets the the wrong end of the stick; I am a bk
advocate. I use it for my open source activities, and urge others to do so as
well.
But... There are tiny minority of people (who work on other SCM projects) who
cannot access my sources because they can't use BK.
I don't particularly care, and I'm still using bk, but that particular clause
of the bk license has caused Larry ridiculous amounts of grief for no
discernable return.
And I still maintain that a stripped out, redistributable, clone/pull only
binary tool without the restrictive license would be a real smart business
move.
But it ain't my business, so I'll leave it there :)
Andrew Walrond
On Thu, Nov 13, 2003 at 07:17:11PM +0000, Andrew Walrond wrote:
> And I still maintain that a stripped out, redistributable, clone/pull only
> binary tool without the restrictive license would be a real smart business
> move.
I'm trying to see why.
Our commercial customers don't care about this so there isn't any business
incentive there.
The actual BK users don't care about this, they use BK.
The BK detractors aren't going to touch anything that we produce, they are
convinced we're out to do them harm somehow.
So who's left?
And a better question is why doesn't some open source person do this? All
you need to do is create a network daemon which responds to clone requests
and pull requests. The clone request sends across a tarball and the client
side untars it. The pull request sends a changeset key (which was sent in
the last clone/pull) and gets a tarball of new/changed files, the new
top of trunk changeset key, and a list of renames. Actually the easier way
is to do it more like diff does, for each file that has been changed, send
across a "delete this file and any directories the deletion leaves empty"
list, and then you unpack the tarball on top of the tree.
This would be easy to build but I don't see it solving anything. All it
does is act as a replacement for rsync. It doesn't have revision history,
you can't do diffs, etc. As soon as this exists, people will ask you
for all the other features. Which is why I don't want to build it, I
don't want to get sucked into the "please rebuild BK as a GPLed product"
business.
What am I missing?
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Thursday 13 Nov 2003 7:28 pm, Larry McVoy wrote:
You're getting to be a cynical old git ;)
> What am I missing?
The point. You got one major o/s project hosted on bk when you ought to have
them all. Loads more developers using bk at home means loads more demanding
it at work.
And all it would take is a lobotomised, redistributable, license free client
so anyone can pull o/s software from bk repos.
And, for the record, you'd be a hungry fool if you GPLed bk.
On Thu, Nov 13, 2003 at 08:57:53PM +0000, Andrew Walrond wrote:
>
> And all it would take is a lobotomised, redistributable, license free client
> so anyone can pull o/s software from bk repos.
Larry, ain't that what you plan to do with bkbits?
To be able to pull regular patches.
No binaries, just a simple URL.
Sam
Larry McVoy wrote:
> On Thu, Nov 13, 2003 at 07:17:11PM +0000, Andrew Walrond wrote:
>
>>And I still maintain that a stripped out, redistributable, clone/pull only
>>binary tool without the restrictive license would be a real smart business
>>move.
>
>
> I'm trying to see why.
>
> Our commercial customers don't care about this so there isn't any business
> incentive there.
>
> The actual BK users don't care about this, they use BK.
>
> The BK detractors aren't going to touch anything that we produce, they are
> convinced we're out to do them harm somehow.
>
> So who's left?
Well, me for one. (I don't use BK, but I also don't complain about it.
;) ) The reason I don't use it, is I've been playing around in the SCM
space as an interesting diversion/exploration/mental-exercise, and I
don't want to violate your license. (Letter or spirit.)
But I've been happy using the offical release tarballs and -rmk patches,
so I don't count from a business view. :)
> And a better question is why doesn't some open source person do this? All
> you need to do is create a network daemon which responds to clone requests
> and pull requests. The clone request sends across a tarball and the client
> side untars it. The pull request sends a changeset key (which was sent in
> the last clone/pull) and gets a tarball of new/changed files, the new
> top of trunk changeset key, and a list of renames. Actually the easier way
> is to do it more like diff does, for each file that has been changed, send
> across a "delete this file and any directories the deletion leaves empty"
> list, and then you unpack the tarball on top of the tree.
>
> This would be easy to build but I don't see it solving anything. All it
> does is act as a replacement for rsync. It doesn't have revision history,
> you can't do diffs, etc. As soon as this exists, people will ask you
> for all the other features. Which is why I don't want to build it, I
> don't want to get sucked into the "please rebuild BK as a GPLed product"
> business.
>
> What am I missing?
If (as I understand the above to imply[1]) the daemon responding to
requests runs BK locally, then those who can't run BK can't run the
daemon, and so the above doesn't actually solve their problem. (They
would have to find someone willing to accept the license to run the
daemon and test it during development, and there are no guarrantees there.)
Anyway, I'm just adding noise... I'm happy using the tarballs, etc., and
I'm glad you've improved Linus's development life.
C-ya,
Eli
[1] If you are saying that daemon talks the BK protocol, then my point
no longer stands.
--------------------. "If it ain't broke now,
Eli Carter \ it will be soon." -- crypto-gram
eli.carter(a)inet.com `-------------------------------------------------
On Thu, 13 Nov 2003, Larry McVoy wrote:
> On Thu, Nov 13, 2003 at 09:14:27AM -0800, Davide Libenzi wrote:
> > On Thu, 13 Nov 2003, Larry McVoy wrote:
> >
> > > I suppose it sounds like we don't want to give out more free engineering
> > > but let's put things into perspective. The CVS server has about 6 users.
> > > It's cost us a pile of money to build and support that technology.
> > > For 6 users. On the other hand, there are thousands if not tens of
> >
> > Larry, if there are really six users (i'm one of them, rsync) among
> > pserver and rsync access, I am the first to tell you shut it down. It is
> > not worth. On the other hand IIRC it was you that, when Pavel showed up
> > with the bitbucket hack to extract metadata from BK, volunteered to do it
> > internally inside BM. Do I remember correctly?
>
> Not really, the revision history of the CVS gateway predates Pavel's so-called
> hacks.
Looking here:
http://lkml.org/lkml/2003/2/15/96
it didn't seem so, but it's not that makes a huge difference.
> I don't mind us supporting this gateway for the small number of users, if it
> makes them happy, that's fine. What I mind is people coming back and asking
> for more stuff for a tiny number of people. That doesn't make sense. We
> should put our efforts into helping the people using BK, not the people using
> CVS. If you want to help yourselves, that's a fine idea, go for it.
I really would like to know from Peter/DaveM how many hits the rsync of
the CVS repo at kernel.org has. If really a few cats are looking at it and
if for BM is an hassle to support it, we have to ask ourselves if it is
worth. Both for kernel.org maintainers and for BM.
(rsync on CVS repo on kernel.org does not give me any new data by days)
- Davide
On Mon, Nov 10, 2003 at 08:31:01PM +0100, Andrea Arcangeli wrote:
> It maybe also cleaner to use a slightly more complicated but more
> compact algorithm, this would make a potential new rsync command line
> option cleaner since only 1 sequence file would need to be specified:
>
> do {
> seq = fetch(sequence-file);
> if (seq & 1)
> break;
> rsync
> if (seq != fetch(sequence-file))
> seq = 1;
> } while (seq & 1 && sleep 10 /* ideally exponential backoff */)
>
> this way only 1 sequence-file is needed for each repository that we want
> to checkout. the server side only has to increase twice the same file
> before and after each update of the repository, so the server side is
> even simpler (with the only additional requirement that the sequence
> number has to start "even"), only the client side is a bit more complicated.
For transparency, I would change the file contents to "updating"
during an update, instead of the even-odd thing. I think this will
make it more obvious to people how to use it properly.
Andrew
On Thu, 2003-11-13 10:21:10 -0800, Tupshin Harper <[email protected]>
wrote in message <[email protected]>:
> Davide Libenzi wrote:
> >Larry, if there are really six users (i'm one of them, rsync) among
> >pserver and rsync access, I am the first to tell you shut it down. It is
> >not worth. On the other hand IIRC it was you that, when Pavel showed up
> >with the bitbucket hack to extract metadata from BK, volunteered to do it
> >internally inside BM. Do I remember correctly?
> As one of the six, I would happily 2nd the shutting down of the
> pserver...rsync is fine with me. I would actually prefer no CVS archive
> at all as long as the raw changesets were rsyncable...then the community
> would be responsible for doing something useful with them instead of BM.
That would be fine with me, too, but there's one little drawback: The
changeset format. You can't simply use a patch(1) file because there is
a (really little) number of non-text files in the kernel source tree
that won't diff...
MfG, JBG
--
Jan-Benedict Glaw [email protected] . +49-172-7608481
"Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg
fuer einen Freien Staat voll Freier B?rger" | im Internet! | im Irak!
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
Andreas Schwab <[email protected]> writes:
> Andrea Arcangeli <[email protected]> writes:
>
> > On Wed, Nov 12, 2003 at 10:35:22PM +0100, Benoit Poulot-Cazajous wrote:
> >> Andrea Arcangeli <[email protected]> writes:
> >>
> >> > the usual problem, and the reason we need a sequence number (increased
> >> > before and after the repo update). A file lock not.
> >>
> >> Or a file that contains md5sums of the other files in the tree.
> >> After the rsync, you recompute the md5sums file, and if it does not match,
> >> rsync again. As a bonus feature, the md5sums file can be pgp-signed.
> >
> > agreed, this would work too and it has the advantage of working with the
> > mirrors too as far as the per-file updates are atomic (should always be
> > the case). This has the only disavanage of forcing the client and the
> > server to read all file contents (I normally don't rsync with -c).
>
> This is not necessary, you only need to recompute the md5sums of changed
> files.
Well, if you really want to optimize (md5sum is fast, faster than a typical hd,
and much faster than gcc), check the most recent file. If its md5sum does not
match, rsync again.
There are many ways to optimize the md5sum of the whole tree, when a previous
tree with a trusted md5sum file is available. "find -ls" can help there. But
make sure not to over-optimize, as servers with wrong md5sum files
will be DDOSed by their clients ;-)
-- Benoit
Jan-Benedict Glaw wrote:
>On Thu, 2003-11-13 10:21:10 -0800, Tupshin Harper <[email protected]>
>wrote in message <[email protected]>:
>
>
>>Davide Libenzi wrote:
>>
>>
>>>Larry, if there are really six users (i'm one of them, rsync) among
>>>pserver and rsync access, I am the first to tell you shut it down. It is
>>>not worth. On the other hand IIRC it was you that, when Pavel showed up
>>>with the bitbucket hack to extract metadata from BK, volunteered to do it
>>>internally inside BM. Do I remember correctly?
>>>
>>>
>>As one of the six, I would happily 2nd the shutting down of the
>>pserver...rsync is fine with me. I would actually prefer no CVS archive
>>at all as long as the raw changesets were rsyncable...then the community
>>would be responsible for doing something useful with them instead of BM.
>>
>>
>
>That would be fine with me, too, but there's one little drawback: The
>changeset format. You can't simply use a patch(1) file because there is
>a (really little) number of non-text files in the kernel source tree
>that won't diff...
>
>MfG, JBG
>
>
>
An acceptable hurdle to overcome.
-Tupshin
Am Don, den 13.11.2003 schrieb Larry McVoy um 17:27:
> I suppose it sounds like we don't want to give out more free engineering
> but let's put things into perspective. The CVS server has about 6 users.
I really do believe you about the amount of users. That's probably
because CVS sucks rocks and everyone complaining about it doesn't make
it better. Shut it down and we'll all have a merrier live.
What really makes me mad though is that there's no permanent way to
retrieve and update current kernels at the moment for those who don't
want or cannot use BK. I'm waiting for quite some period to get SVN back
online so I can continue the merging work I've been trying to do. And
yes, I'm one of those fools who are stuck behind a huge 64k
pay-big-bucks-for-the-minute and really have troubles changing back and
forth between several mechanisms to get current versions
(ftp/rsync/CVS/SVN), especially since Linus obviously doesn't release
patches as frequently anymore since he uses BK. :(
--
Servus,
Daniel
On Thu, 13 Nov 2003, Tupshin Harper wrote:
> Davide Libenzi wrote:
> >Larry, if there are really six users (i'm one of them, rsync) among
> >pserver and rsync access, I am the first to tell you shut it down. It is
> >not worth. On the other hand IIRC it was you that, when Pavel showed up
> >with the bitbucket hack to extract metadata from BK, volunteered to do it
> >internally inside BM. Do I remember correctly?
> >
> As one of the six, I would happily 2nd the shutting down of the
> pserver...rsync is fine with me. I would actually prefer no CVS archive
> at all as long as the raw changesets were rsyncable...then the community
> would be responsible for doing something useful with them instead of BM.
Just wondering: the emails sent to the bk-commits mailing lists are just all
changesets in a `neutral' format that contains all meta information, right?
So if all individual mails were archived somewhere with correct sequence
numbers, they could be used to recreate the whole repository in whatever format
you want. I guess it's just a matter of importing them like patches into arch.
Gr{oetje,eeting}s,
Geert
P.S. I did use the CVS gateway before to have a base tree to verify that the
patches I send to Linus and Marcelo still apply cleanly. But these days
full releases are so frequent that I can use these as base trees. And I
monitor the bk-commits lists so I know what's happening in between.
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Daniel Egger wrote:
>Am Don, den 13.11.2003 schrieb Larry McVoy um 17:27:
>
>
>>I suppose it sounds like we don't want to give out more free engineering
>>but let's put things into perspective. The CVS server has about 6 users.
>>
>
>I really do believe you about the amount of users. That's probably
>because CVS sucks rocks and everyone complaining about it doesn't make
>it better. Shut it down and we'll all have a merrier live.
>
>What really makes me mad though is that there's no permanent way to
>retrieve and update current kernels at the moment for those who don't
>want or cannot use BK.
>
Actually, at http://www.kernel.org/ there is a link to daily snapshots.
There are also changesets generated every couple of hours at the "C" link
at the right of the page.
Even if Linus doesn't release as often (doesn't he? I don't know), this
is surely much better than pre BK. Maybe I didn't understand you right?
Best regards,
Nick
Am Fre, den 14.11.2003 schrieb Nick Piggin um 10:56:
> Actually, at http://www.kernel.org/ there is a link to daily snapshots.
> There are also changesets generated every couple of hours at the "C" link
> at the right of the page.
> Even if Linus doesn't release as often (doesn't he? I don't know), this
> is surely much better than pre BK. Maybe I didn't understand you right?
Seems so. I assume you missed the "bandwidth constraint" part. Fetching
a whole snapshot every day is not even close to workable. The snapshots
in patch form are nice however patching forth and back is not really an
option. If svn doesn't get back up I'd be tempted to use rsync and use
vendor branches in my own SVN repository but this also seems far from
optimal to me. rsync alone doesn't cut it because there's no version
management and I've lost quite a few patches due to an not thoroughly
considered rsync use.
--
Servus,
Daniel
On Fri, Nov 14, 2003 at 12:13:00AM -0500, Andrew Pimlott wrote:
> For transparency, I would change the file contents to "updating"
> during an update, instead of the even-odd thing. I think this will
> make it more obvious to people how to use it properly.
we've more solutions now. The file contents to "updating" would work too
but I believe it would be by far the most complicated, the md5sum has
the advantage of valiating the file contents too and it only requires 1
file update to be atomic, no matter how the upload of the data paylod
happens, so I tend to like it most even if it only works
probabilitsically (but it's sure safe enough).
However I understand Larry has no real interest in helping us to
rsync a known to be coherent copy of the repository (very understandable
from his own business standpoint), so I guess this is all wasted time,
and we've to live with an heuristic like:
1:rsync
sleep 60
rsync -> if something changed goto 1
I doubt Peter can provide the coherency guarantee with the md5sum on his
side, unless it's Peter fetching the update of the scm data, and not the
other way around.
On Fri, Nov 14, 2003 at 03:01:24PM +0100, Andrea Arcangeli wrote:
> However I understand Larry has no real interest in helping us to
> rsync a known to be coherent copy of the repository (very understandable
> from his own business standpoint), so I guess this is all wasted time,
Rsync coherence is your problem, there is nothing I can do about that,
I don't admin kernel.org. Peter has a login on kernel.bkbits.net and
we can coordinate so he gets a coherent snapshot. It's trivial, he
could sync the data to kernel.org at 2PM and we update the data at 2AM,
I tend to think that there won't be any coherency problems.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Thu, Nov 13, 2003 at 10:17:52PM +0100, Sam Ravnborg wrote:
> On Thu, Nov 13, 2003 at 08:57:53PM +0000, Andrew Walrond wrote:
> >
> > And all it would take is a lobotomised, redistributable, license free client
> > so anyone can pull o/s software from bk repos.
> Larry, ain't that what you plan to do with bkbits?
> To be able to pull regular patches.
Yes.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Thu, Nov 13, 2003 at 08:57:53PM +0000, Andrew Walrond wrote:
> The point. You got one major o/s project hosted on bk when you ought to have
> them all. Loads more developers using bk at home means loads more demanding
> it at work.
>
> And all it would take is a lobotomised, redistributable, license free client
> so anyone can pull o/s software from bk repos.
One of us is not getting it, maybe it's me. To build something like
you describe is pretty easy IF AND ONLY IF all you are asking for is an
update mechanism. As soon as you want revision history, diffs, rollbacks,
modifiable files, etc., you have to go to real BK. Is that OK? All you
want is a "keep me up to date" mechanism? No diffs, no history, it's a
replacement for tarballs and patches?
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Fri, Nov 14, 2003 at 06:53:33AM -0800, Larry McVoy wrote:
> Rsync coherence is your problem, there is nothing I can do about that,
> I don't admin kernel.org. Peter has a login on kernel.bkbits.net and
> we can coordinate so he gets a coherent snapshot. It's trivial, he
> could sync the data to kernel.org at 2PM and we update the data at 2AM,
> I tend to think that there won't be any coherency problems.
if we know rsync.kernel.org is getting the update at 2PM we can safely
fetch it from kernel.org at 2AM. It's not very robust and it would be
hard to enforce it with the mirrors, but it should work and it's much
better than nothing ;).
On Friday 14 Nov 2003 3:00 pm, Larry McVoy wrote:
> On Thu, Nov 13, 2003 at 08:57:53PM +0000, Andrew Walrond wrote:
> > The point. You got one major o/s project hosted on bk when you ought to
> > have them all. Loads more developers using bk at home means loads more
> > demanding it at work.
> >
> > And all it would take is a lobotomised, redistributable, license free
> > client so anyone can pull o/s software from bk repos.
>
> One of us is not getting it, maybe it's me. To build something like
> you describe is pretty easy IF AND ONLY IF all you are asking for is an
> update mechanism. As soon as you want revision history, diffs, rollbacks,
> modifiable files, etc., you have to go to real BK. Is that OK? All you
> want is a "keep me up to date" mechanism? No diffs, no history, it's a
> replacement for tarballs and patches?
Yes exactly. Fundamentally I want *anybody*, without restriction, to ge able
to pull and update sources from any open-source project hosted with bk.
The requirements are the equivalent functionality to:
lobobk clone ...
lobobk -r co
lobobk pull
lobobk export -r tag dest
Ie, Get a local clone of the repo, Be able to keep it updated, checkout the
head version and export any particular tagged version.
and absolutely nothing else.
Not necessary, but if the cloned repo was complete enough to be recloned, It
would also make the perfect tool for anyone wanting to host a coherent mirror
of the repo. Which takes us full circle I believe :)
Andrew Walrond
On Fri, 14 Nov 2003, Larry McVoy wrote:
> One of us is not getting it, maybe it's me. To build something like
> you describe is pretty easy IF AND ONLY IF all you are asking for is an
> update mechanism. As soon as you want revision history, diffs, rollbacks,
> modifiable files, etc., you have to go to real BK. Is that OK? All you
Spec for bk-lite:
1) Binary with "no worky on other SCM" kinda license
2) update+history+diff (no rollbacks, no modifiable files, no etc...)
In that way all current users of bk2cvs, bk2svn, bk2xxx can simply do a
pull from a bk repo and have they own scripts on their local machine to do
their bk2xxx. It will be a lower headache for you and for kernel.org
maintainers. Is it feasible ?
- Davide
On Fri, Nov 14, 2003 at 04:24:31PM +0000, Andrew Walrond wrote:
> On Friday 14 Nov 2003 3:00 pm, Larry McVoy wrote:
> > On Thu, Nov 13, 2003 at 08:57:53PM +0000, Andrew Walrond wrote:
> > > The point. You got one major o/s project hosted on bk when you ought to
> > > have them all. Loads more developers using bk at home means loads more
> > > demanding it at work.
> > >
> > > And all it would take is a lobotomised, redistributable, license free
> > > client so anyone can pull o/s software from bk repos.
> >
> > One of us is not getting it, maybe it's me. To build something like
> > you describe is pretty easy IF AND ONLY IF all you are asking for is an
> > update mechanism. As soon as you want revision history, diffs, rollbacks,
> > modifiable files, etc., you have to go to real BK. Is that OK? All you
> > want is a "keep me up to date" mechanism? No diffs, no history, it's a
> > replacement for tarballs and patches?
>
> Yes exactly. Fundamentally I want *anybody*, without restriction, to ge able
> to pull and update sources from any open-source project hosted with bk.
>
> The requirements are the equivalent functionality to:
>
> lobobk clone ...
Sure.
> lobobk -r co
There are no local revision history files in lobobk, it's just a file transport.
> lobobk pull
Sure.
> lobobk export -r tag dest
That's
lobobk clone -r tag FROM DEST
And you of course realize that you as a BK user could code up this system with
zero changes needed from us, right?
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Fri, Nov 14, 2003 at 08:35:33AM -0800, Davide Libenzi wrote:
> On Fri, 14 Nov 2003, Larry McVoy wrote:
>
> > One of us is not getting it, maybe it's me. To build something like
> > you describe is pretty easy IF AND ONLY IF all you are asking for is an
> > update mechanism. As soon as you want revision history, diffs, rollbacks,
> > modifiable files, etc., you have to go to real BK. Is that OK? All you
>
> Spec for bk-lite:
>
> 1) Binary with "no worky on other SCM" kinda license
> 2) update+history+diff (no rollbacks, no modifiable files, no etc...)
>
> In that way all current users of bk2cvs, bk2svn, bk2xxx can simply do a
> pull from a bk repo and have they own scripts on their local machine to do
> their bk2xxx. It will be a lower headache for you and for kernel.org
> maintainers. Is it feasible ?
update, yes. history, no, use bk/web. diffs, no, use bk/web or bk. building
it is certainly possible and you could do it yourself. We already built the
cvs exporter which is a lot nicer than what you are asking for and building
another thing for another 6 users seems pointless. If you want to pay for the
engineering then contact me offline.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Fri, 14 Nov 2003, Larry McVoy wrote:
> update, yes. history, no, use bk/web. diffs, no, use bk/web or bk. building
> it is certainly possible and you could do it yourself. We already built the
> cvs exporter which is a lot nicer than what you are asking for and building
> another thing for another 6 users seems pointless. If you want to pay for the
> engineering then contact me offline.
I want nothing new. A BK with only the CVS exporter that you already have
will work plain fine. We do not need even 25 exports formats. Once we
have your existing binary that does:
bk2cvs remote-bkrepo local-cvs-repo
would be great. All other export can be based from that with local
scripts. With a little bit less restrictive license, Is it something
painful for BM to do?
- Davide
On Fri, Nov 14, 2003 at 09:03:05AM -0800, Davide Libenzi wrote:
> On Fri, 14 Nov 2003, Larry McVoy wrote:
>
> > update, yes. history, no, use bk/web. diffs, no, use bk/web or bk. building
> > it is certainly possible and you could do it yourself. We already built the
> > cvs exporter which is a lot nicer than what you are asking for and building
> > another thing for another 6 users seems pointless. If you want to pay for the
> > engineering then contact me offline.
>
> I want nothing new. A BK with only the CVS exporter that you already have
> will work plain fine. We do not need even 25 exports formats. Once we
> have your existing binary that does:
>
> bk2cvs remote-bkrepo local-cvs-repo
>
> would be great. All other export can be based from that with local
> scripts. With a little bit less restrictive license, Is it something
> painful for BM to do?
Yes. bk2cvs is not a supported product, and it is based on the commercial
only version of BK. We're not giving that out for free.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Fri, Nov 14, 2003 at 09:08:03AM -0800, Davide Libenzi wrote:
> On Fri, 14 Nov 2003, Larry McVoy wrote:
>
> > Yes. bk2cvs is not a supported product, and it is based on the commercial
> > only version of BK. We're not giving that out for free.
>
> Fine then. I guess we will live with the existing CVS repo rsync from
> kernel.org, that is plain good for me. I was trying to find a solution to
> reduce headaches for BM and kernel.org maintainers ...
It's not a headache for us to do the conversion, that's fine. I'd like to
get rid of the pserver on kernel.bkbits.net because it and the SVN server
beat up the machine quite a bit. So an rsync based solution sounds good
to me if HPA can handle it.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Fri, 14 Nov 2003, Larry McVoy wrote:
> Yes. bk2cvs is not a supported product, and it is based on the commercial
> only version of BK. We're not giving that out for free.
Fine then. I guess we will live with the existing CVS repo rsync from
kernel.org, that is plain good for me. I was trying to find a solution to
reduce headaches for BM and kernel.org maintainers ...
- Davide
On Friday 14 Nov 2003 4:46 pm, Larry McVoy wrote:
> And you of course realize that you as a BK user could code up this system
> with zero changes needed from us, right?
A client only solution that would talk to bkd? In a flash mate. But I don't
think you'd like that ;)
You I'm sure mean a server side wrapper which calls on bk to extract sources
and diffs and handles transport to the client. Not exactly hastle free and
encouraging people to host their o/s projects with bk, is it? Probably of
dubious legality as well (for scm developers).
Andrew Walrond
On Fri, Nov 14, 2003 at 05:34:57PM +0000, Andrew Walrond wrote:
> On Friday 14 Nov 2003 4:46 pm, Larry McVoy wrote:
>
> > And you of course realize that you as a BK user could code up this system
> > with zero changes needed from us, right?
>
> A client only solution that would talk to bkd? In a flash mate. But I don't
> think you'd like that ;)
I think that might be harder than you think but you're right, we wouldn't be
too thrilled with that. If for no other reason than the amount of cursing
you'd do as you figured out that our protocol is pretty ad-hoc and should be
rewritten.
> You I'm sure mean a server side wrapper which calls on bk to extract sources
> and diffs and handles transport to the client. Not exactly hastle free and
> encouraging people to host their o/s projects with bk, is it? Probably of
> dubious legality as well (for scm developers).
You're already a BK user and I'd happily make it clear that it is OK for you
to do this work, we can have that discussion offline. As for hassle free,
I don't get that comment at all. What you describe is exactly what we would
do anyway. The only difference is that we could extend the bkd itself to
answer these requests and you couldn't without a lot of headaches. But if
you built this and wanted us to included it in the bkd, we'd look at doing
that.
The points are
a) I'm not at all convinced this is going to make anyone other than you
happy. They all want a BK replacement, not a tarball+patch replacement.
b) If I'm wrong, there isn't anything preventing you from building this.
Convince me that (a) is all wrong and I'll see about building it.
So far, all the other people have made it clear they want all the
functionality of BK, the transport, the diffs, the history, etc.
Giving them a tarball/patch transport isn't like to make them happy and
I'll be in another one of these conversations after having wasted a pile
of engineering (which is not free, nor cheap).
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
> The points are
> a) I'm not at all convinced this is going to make anyone other than you
> happy. They all want a BK replacement, not a tarball+patch
> replacement
No. The fundamental point is:
There are a small group of people who cannot legally extract sources from o/s
projects hosted with bk. However small the group, this is unacceptible for an
o/s project.
So Bk is unsuitable for general hosting of o/s, 'free and available to all'
projects. They will use cvs or arch.
Thats why you should provide this facility. Not because I want it, but because
you should want as many o/s projects as possible to use bk, for sound
business reasons.
I think I can see granny looking hungrily at a basket of eggs, so I'm
finished.
Andrew Walrond
Andrea Arcangeli wrote:
>
> I doubt Peter can provide the coherency guarantee with the md5sum on his
> side, unless it's Peter fetching the update of the scm data, and not the
> other way around.
>
It is, actually... Larry runs the bkcvs site, but I have a login on that
machine and do everything from there.
-hpa
On Fri, 2003-11-14 10:56:56 +0100, Geert Uytterhoeven <[email protected]>
wrote in message <[email protected]>:
> On Thu, 13 Nov 2003, Tupshin Harper wrote:
> > Davide Libenzi wrote:
> > As one of the six, I would happily 2nd the shutting down of the
> > pserver...rsync is fine with me. I would actually prefer no CVS archive
> > at all as long as the raw changesets were rsyncable...then the community
> > would be responsible for doing something useful with them instead of BM.
>
> Just wondering: the emails sent to the bk-commits mailing lists are just all
> changesets in a `neutral' format that contains all meta information, right?
These changesets represent what someone "submitted". You can't expect
these to apply cleanly. I've tried once, it will not work.
> So if all individual mails were archived somewhere with correct sequence
> numbers, they could be used to recreate the whole repository in whatever format
> you want. I guess it's just a matter of importing them like patches into arch.
Nope. You'll have to heal with conflicts first.
MfG, JBG
--
Jan-Benedict Glaw [email protected] . +49-172-7608481
"Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg
fuer einen Freien Staat voll Freier B?rger" | im Internet! | im Irak!
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
Hi,
On Fri, 14 Nov 2003, Geert Uytterhoeven wrote:
> So if all individual mails were archived somewhere with correct sequence
> numbers, they could be used to recreate the whole repository in whatever format
> you want. I guess it's just a matter of importing them like patches into arch.
The problematic part are "correct sequence numbers" here. These numbers in
the commit list mails are pretty much useless. At least the numbers in
this line:
# ChangeSet 1.1437+1.1350.5.10 -> 1.1438
have to be replaced with a unique identifier. This identifier does exist,
e.g. it's visible in the exclude mails.
As soon as someone finds out how to export changesets in this format and
stores them somewhere, other people can make use of this data.
Unfortunately it's rather unlikely that anyone who can do the first, knows
also how to do the latter (but maybe Larry can tell us the trick).
Anyway, if someone has this data I'd be interested in a copy.
bye, Roman
Carl-Daniel Hailfinger wrote:
> Larry,
>
> regarding rsync from bkbits.net to kernel.org, would it be possible to do
> that with a post-incoming trigger in kernel.bkbits.net which starts rsync
> to kernel.org? That should solve all atomicity requirements, at least on
> the way from bkbits.net to kernel.org.
> Same way for the CVS tree. Since you are starting the conversion (I assume
> it's at least half automated), you could also add a call to rsync at the
> end of that script.
> Using rsync over ssh with pubkey authentication should be pretty
> straightforward and also mostly secure, since no incoming connections to
> bkbits.net are needed. The only thing listening to network connections
> would be bkd.
>
> Comments on the (in)feasibility of my suggestion are welcome.
>
>
> Carl-Daniel
> (happy bk user)
One way to do think would be to have bkcvs flock() a particular key
file; we can then have the upload script flock() the same file, which
will wait if it's already busy.
It does *not* solve all atomicity problems, however.
-hpa
Larry,
regarding rsync from bkbits.net to kernel.org, would it be possible to do
that with a post-incoming trigger in kernel.bkbits.net which starts rsync
to kernel.org? That should solve all atomicity requirements, at least on
the way from bkbits.net to kernel.org.
Same way for the CVS tree. Since you are starting the conversion (I assume
it's at least half automated), you could also add a call to rsync at the
end of that script.
Using rsync over ssh with pubkey authentication should be pretty
straightforward and also mostly secure, since no incoming connections to
bkbits.net are needed. The only thing listening to network connections
would be bkd.
Comments on the (in)feasibility of my suggestion are welcome.
Carl-Daniel
(happy bk user)
Daniel Egger wrote:
>Am Fre, den 14.11.2003 schrieb Nick Piggin um 10:56:
>
>
>>Actually, at http://www.kernel.org/ there is a link to daily snapshots.
>>There are also changesets generated every couple of hours at the "C" link
>>at the right of the page.
>>
>
>>Even if Linus doesn't release as often (doesn't he? I don't know), this
>>is surely much better than pre BK. Maybe I didn't understand you right?
>>
>
>Seems so. I assume you missed the "bandwidth constraint" part. Fetching
>a whole snapshot every day is not even close to workable. The snapshots
>in patch form are nice however patching forth and back is not really an
>option. If svn doesn't get back up I'd be tempted to use rsync and use
>vendor branches in my own SVN repository but this also seems far from
>optimal to me. rsync alone doesn't cut it because there's no version
>management and I've lost quite a few patches due to an not thoroughly
>considered rsync use.
>
There are compressed incremental patches of the snapshots available:
http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/incr/
They average maybe 150KB per day.
Nick
On Sat, 15 Nov 2003, Nick Piggin wrote:
>
> There are compressed incremental patches of the snapshots available:
> http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/incr/
> They average maybe 150KB per day.
I'd hope they average a _lot_ less lately. I've been trying to be an
absolute _bastard_ when it comes to patches.
Yeah, I just looked. Lately they've been averaging about 3-4kB per day.
And the sick thing is, I'm still not satisfied. I want it to become an
absolute _trickle_ of one-liners that fix real bugs.
So if you want incrementals and have a 1200 baud modem - be happy. The
last week or so has been a pretty good week.
Linus
On Fri, 14 Nov 2003, Linus Torvalds wrote:
> I'd hope they average a _lot_ less lately. I've been trying to be an
> absolute _bastard_ when it comes to patches.
So it didn't change much. You've *always* been a bastard when it comes to
patches :-)
- Davide
On Fri, 14 Nov 2003, Davide Libenzi wrote:
>
> So it didn't change much. You've *always* been a bastard when it comes to
> patches :-)
*Snif*. I don't know what to say. Thank you. Your kind words make it all
worth it.
Linus "kernel bastard, and proud of it" Torvalds
On Fri, 14 Nov 2003, Linus Torvalds wrote:
> *Snif*. I don't know what to say. Thank you. Your kind words make it all
> worth it.
>
> Linus "kernel bastard, and proud of it" Torvalds
Your welcome ;)
Davide "gotta wait big time before next patch will be merged" Libenzi
I find it incredible that this thread has been allowed to continue for so
long when in recent memory a certain someone was kicked off of lkml for
posting off-topic garbage.
Yes, this thread is closer to home, but I don't see what it has to do
with Linux-Kernel. Especially considering that it does seem appropriate
for bk-users (if such a list exists).
/fc
Am Sam, den 15.11.2003 schrieb Nick Piggin um 04:02:
> There are compressed incremental patches of the snapshots available:
> http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/incr/
> They average maybe 150KB per day.
Me likee. Thanks a lot, this is a missing piece in the puzzle. With this
I can easily build my own repository.
--
Servus,
Daniel
Followup to: <[email protected]>
By author: Larry McVoy <[email protected]>
In newsgroup: linux.dev.kernel
>
> It's not a headache for us to do the conversion, that's fine. I'd like to
> get rid of the pserver on kernel.bkbits.net because it and the SVN server
> beat up the machine quite a bit. So an rsync based solution sounds good
> to me if HPA can handle it.
>
Machine-resource-wise or bandwidth-wise it's not a problem. The
coherency mechanism -- in particular, deploying it -- is the only
issue.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
If you send me mail in HTML format I will assume it's spam.
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
On Sat, 2003-11-15 at 04:43, Larry McVoy wrote:
> The points are
> a) I'm not at all convinced this is going to make anyone other than you
> happy. They all want a BK replacement, not a tarball+patch replacement.
As a quite irrelevant (from a kernel development point - as i don't do
anything other than test / use and bug report) data point, this is
somehting I would like to see, and preferably with a license that would
make it possible to incluse in the mainline debian distribution. that
way it would be possible for more people to test bk repository versions
of software (not jsut the kernel) without having to install the full
version of BK.
so all I'd like is to be able to do a get/update to the head revision of
the repository, and if possible get/convert to a tagged version.
having the second would have made chasing down what version of the
kernel broke my pcmcia support easier :)
cheers
Sven
Sven Dowideit wrote:
> On Sat, 2003-11-15 at 04:43, Larry McVoy wrote:
>
>>The points are
>> a) I'm not at all convinced this is going to make anyone other than you
>> happy. They all want a BK replacement, not a tarball+patch replacement.
> ...so all I'd like is to be able to do a get/update to the head revision of
> the repository, and if possible get/convert to a tagged version.
>
> having the second would have made chasing down what version of the
> kernel broke my pcmcia support easier :)
Just a MeeToo for the record. Of course, I'm perfectly happy using the free
full-version of bk for pulling from Linus's tree, but I don't use all the other
functions.
Hi!
> > > I could make some comment about this being a good example of one of
> > > the zillion little problems we've had to solve but if I go there it's
> > > going to start a flame war. So I won't. I will note that none of the
> > > solutions proposed come close to being acceptable, they all fail on NFS
> > > and on SMB shares. And they don't cascade properly as HPA has noted.
> >
> > Absolutely. Bk is, undeniably, brilliant, and would solve all these problems
> > at a stroke, except that the open source community cannot with good
> > conscience exclude *anyone* from being able to access the sources.
>
> But noone is excluded from having access to the sources.
>
> I suppose it sounds like we don't want to give out more free engineering
> but let's put things into perspective. The CVS server has about 6
> users.
I do not know where you got that number, but its wrong. You cited 6
unique IP addresses... I certainly did updates from more than
_that_. But I use time rsync -zav --delete
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.5 ., so I'm
probably not counted in your statistics.
Pavel
--
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]
On Tue, Nov 18, 2003 at 10:59:13AM +0100, Pavel Machek wrote:
> > I suppose it sounds like we don't want to give out more free engineering
> > but let's put things into perspective. The CVS server has about 6
> > users.
>
> I do not know where you got that number, but its wrong. You cited 6
> unique IP addresses... I certainly did updates from more than
> _that_. But I use time rsync -zav --delete
> rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.5 ., so I'm
> probably not counted in your statistics.
"CVS server", Pavel. That means people talking to the pserver process, not
rsync.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Sun, Nov 16, 2003 at 04:12:45PM +1100, Sven Dowideit wrote:
> On Sat, 2003-11-15 at 04:43, Larry McVoy wrote:
> > The points are
> > a) I'm not at all convinced this is going to make anyone other than you
> > happy. They all want a BK replacement, not a tarball+patch replacement.
> As a quite irrelevant (from a kernel development point - as i don't do
> anything other than test / use and bug report) data point, this is
> somehting I would like to see, and preferably with a license that would
> make it possible to incluse in the mainline debian distribution. that
> way it would be possible for more people to test bk repository versions
> of software (not jsut the kernel) without having to install the full
> version of BK.
>
> so all I'd like is to be able to do a get/update to the head revision of
> the repository, and if possible get/convert to a tagged version.
>
> having the second would have made chasing down what version of the
> kernel broke my pcmcia support easier :)
We'd be happy to build this if there was some indication that it would
actually make a lot more people happy. Examples of that would be
agreement from the debian crowd to include it, find some BK flamers who
say this would resolve their issues, etc.
We've been burned once by doing a bunch of work and getting beat up for
it, i.e., doing the BK2CVS converter and people still aren't happy.
We spend a lot of unseen time maintaining bkbits.net and the hosting
machines. I'd *love* to have a better relationship with the open source
world but if this is a solution for a handful of people but it does
nothing for the other people then we aren't moving forward.
Just to be clear, what we are talking about is a free client which talks to
a modified BK server. The client has the ability to
- get a workspace (bk clone equiv)
- get a version of the workspace as of any tag (bk clone -r equiv)
- update a workspace (bk pull equiv)
Anything more than that could be done but would be left to you to extend
the client (you'd get source to the client).
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
> Just to be clear, what we are talking about is a free client which talks to
> a modified BK server. The client has the ability to
I just wanted to be clear on what you meant by "free". Is that free as
in binary with less restrictive license than the current bk client, or
"free" as in includes the source under the GPL? If the latter, you can
bet your ass I'd use it. If the former, it would allow me to use it, but
I can't guarantee I would.
--
Debian - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
WatchGuard - http://www.watchguard.com/
On Tue, Nov 18, 2003 at 01:30:58PM -0500, Ben Collins wrote:
> > Just to be clear, what we are talking about is a free client which talks to
> > a modified BK server. The client has the ability to
>
> I just wanted to be clear on what you meant by "free". Is that free as
> in binary with less restrictive license than the current bk client, or
> "free" as in includes the source under the GPL? If the latter, you can
> bet your ass I'd use it. If the former, it would allow me to use it, but
> I can't guarantee I would.
It would be free as in w/ source, probably BSD rather than GPL but some
license you'd like. About the only thing we'd need to worry about is
our commercial customers taking this and using it as a way to not pay
for some seats so there is some chance that we'd want to put in some
hook there, I'd have to think about that before promising anything.
I'm curious as to why you would think this is better than the CVS gateway.
The CVS gateway is actually a really nice thing. The whiners think we
have somehow hamstrung the data in the gateway but that's only because
they haven't looked at the data, if they had done a careful comparison
then they'd know it's all in there.
So what's the attraction? Having a client that will work with any BK
server? Do you realize that the client is just a way to get at the head?
And tagged releases? It doesn't have 1/10th the functionality of BK itself.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Ben Collins wrote:
>>Just to be clear, what we are talking about is a free client which talks to
>>a modified BK server. The client has the ability to
>
>
> I just wanted to be clear on what you meant by "free". Is that free as
> in binary with less restrictive license than the current bk client, or
> "free" as in includes the source under the GPL? If the latter, you can
> bet your ass I'd use it. If the former, it would allow me to use it, but
> I can't guarantee I would.
>
I think he said something about providing source code. GPL would be
wise to use because, for instance, BSD license would give away too much
to his competitors.
Maybe this will tone down some of the whining.
On Tue, 18 Nov 2003 19:50:18 +0100, you wrote in linux.kernel:
> I'm curious as to why you would think this is better than the CVS gateway.
Both things are tackling different issues.
The CVS gateway means the data (code + checkin comments) is available
in a free format. This means you as a company can do to BK's internal
format whatever you want if you decide the old format is no longer
up to it. The community still has all it wants in CVS format, as long
as you continue the gateway service.
A check-out and keep-up-to-date BK variant is a tool to access the
latest version of a BK-using project (or any tagged version). This
doesn't have to be the kernel obviously. It is still useful for
kernel development - before sending in a patch to Linus' or Marcelo,
people want to check out the latest BK tree to see if their patch
applies cleanly. The CVS gateway, at least the data on kernel.org,
lags behind the BK trees - for 2.4, the CVS repository at kernel.org
still has "-pre9" in the Makefile although rc1 has been released
already and the ChangeLog Marcelo posted indicated that he changed
the Makefile in the BK repo.
The lobotomized BK client would be useful for people who just want
to get the latest code from some project, and nothing else. It means
the project's maintainers wouldn't need to waste effort on creating
.tgz snapshots (or similar); people could instead point to the BK
repository and tell people who don't use closed-source tools on
principle that they can use the loboBK ;) client to get the sources.
As an aside, I personally rsync the CVS repository to be able to use
"cvs diff" quickly (no fun having to contact a remote server on ISDN).
I could use BK but it's not worth the effort for me to learn a new tool.
So a checkout-only BK would not be useful to me, but I can see reasons
why people would want one, outlined above.
--
Ciao,
Pascal
Larry,
On Wed, 2003-11-19 at 05:42, Larry McVoy wrote:
> It would be free as in w/ source, probably BSD rather than GPL but some
> license you'd like. About the only thing we'd need to worry about is
> our commercial customers taking this and using it as a way to not pay
> for some seats so there is some chance that we'd want to put in some
> hook there, I'd have to think about that before promising anything.
to my mind this would be brilliant, but i don't remember every whinging
about bk ;).
The only requirements that would need to be fulfilled for it to go into
debian, would be source, a licence that does not restrict the usage of
the code/ package, and a distinct lack of invariant sections in the
license. (a real debian developer can probably elaborate better than i)
>
> I'm curious as to why you would think this is better than the CVS gateway.
> The CVS gateway is actually a really nice thing. The whiners think we
> have somehow hamstrung the data in the gateway but that's only because
> they haven't looked at the data, if they had done a careful comparison
> then they'd know it's all in there.
last time i tried the cvs gateway, i think i had trouble getting a full
shadow.. I should try again really, but i would hope that the
openBKClient could be faster, better and more modern (oh, and sexy too)
(and would mean that you don't need the cvs gateway anymore!)
>
> So what's the attraction? Having a client that will work with any BK
> server? Do you realize that the client is just a way to get at the head?
> And tagged releases? It doesn't have 1/10th the functionality of BK itself.
yep. and for me, until i do some actual kernel development (unlikely as
i just can't find the time), this is all i think i want - i should be
able to do a bit more testing this way.
cheers
Sven
I'm likely to get a slap for this one, but....
On Tuesday 18 Nov 2003 6:42 pm, Larry McVoy wrote:
>
> I'm curious as to why you would think this is better than the CVS gateway.
> The CVS gateway is actually a really nice thing. The whiners think we
> have somehow hamstrung the data in the gateway but that's only because
> they haven't looked at the data, if they had done a careful comparison
> then they'd know it's all in there.
Since you've already done all the work for bk2cvs, why not add the
functionality to bkd which would allow a simple client to do
lobobk cvsclone
and
lobobk cvspull
Ie be able to clone and update a local cvs repo from bkd? This would have the
added advantage that users (kernel testers) could get any tagged versions
from their local repo using cvs, rather than requiring diff trasmission from
bkd to convert to another version. The client would be very simple too.
Obviously the local cvs repo would be read-only, and only useful for people to
fetch sources. Bitkeeper would be required for development work. It would
make the perfect mirroring tool aswell.
And when you've finished that, I'd like a moon rocket please. GPL, of
course ;)
Andrew Walrond
On Wed, Nov 19, 2003 at 12:24:51AM +0000, Andrew Walrond wrote:
> I'm likely to get a slap for this one, but....
>
> On Tuesday 18 Nov 2003 6:42 pm, Larry McVoy wrote:
> >
> > I'm curious as to why you would think this is better than the CVS gateway.
> > The CVS gateway is actually a really nice thing. The whiners think we
> > have somehow hamstrung the data in the gateway but that's only because
> > they haven't looked at the data, if they had done a careful comparison
> > then they'd know it's all in there.
>
> Since you've already done all the work for bk2cvs, why not add the
> functionality to bkd which would allow a simple client to do
>
> lobobk cvsclone
> and
> lobobk cvspull
>
> Ie be able to clone and update a local cvs repo from bkd? This would have the
> added advantage that users (kernel testers) could get any tagged versions
> from their local repo using cvs, rather than requiring diff trasmission from
> bkd to convert to another version. The client would be very simple too.
Presumably because he's said several times that the bk2cvs gateway
software is based on (and requires) the commercial version of bk. Not
to mention that the way it generates repositories isn't really
compatible with this model.
--
Daniel Jacobowitz
MontaVista Software Debian GNU/Linux Developer
On Wed, Nov 19, 2003 at 12:24:51AM +0000, Andrew Walrond wrote:
> On Tuesday 18 Nov 2003 6:42 pm, Larry McVoy wrote:
> > I'm curious as to why you would think this is better than the CVS gateway.
> > The CVS gateway is actually a really nice thing. The whiners think we
> > have somehow hamstrung the data in the gateway but that's only because
> > they haven't looked at the data, if they had done a careful comparison
> > then they'd know it's all in there.
>
> Since you've already done all the work for bk2cvs, why not add the
> functionality to bkd which would allow a simple client to do
>
> lobobk cvsclone
> and
> lobobk cvspull
Because the translation process is hugely CPU intensive. It takes
something like 6 hours to do the 2.5 tree on 2.2Ghz Athlon with a gig
of ram. And it uses every bit of that ram, that's my desktop machine
and _everything_ is paged out when I come in in the morning.
I agreed to the BK2CVS stuff as a way of ensuring that people had the
data in a format that they could use without BK and was useful. I'm
willing to do that for each main tree of each major project (i.e., if
XFree86 or something like that moved to BK we'd agree to do the CVS
conversion so that there was no lockin).
But doing it for every branch of every tree is nuts unless you are
donating a 16 way with a zillion gigs of ram. And a zillion disk
arms, this thrashes the heck out of the disk.
> And when you've finished that, I'd like a moon rocket please. GPL, of
> course ;)
Yeah, and I'd like all the open source developers of the world to acknowledge
that I'm a good guy and I'm trying to help. In writing, of course ;)
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Wednesday 19 Nov 2003 12:38 am, Daniel Jacobowitz wrote:
>
> Presumably because he's said several times that the bk2cvs gateway
> software is based on (and requires) the commercial version of bk. Not
> to mention that the way it generates repositories isn't really
> compatible with this model.
I was (obviously?) assuming that the bk(d) was the commercial version, but I
don't see how that would affect the client from being o/s?
And why is this different from bk2cvs? Because then I have to rsync the cvs
repo with all the problems (discussed at length in this thread) of getting a
coherent local copy of the repo.
I guess it all comes back to my wanting to host my (commercial and open-
source) public code repositories with bk, but have to guarantee 100% access
to my 'users'. 99% Isn't good enough.
I'm not, Daniel, a FSF/GPL whiner, and do not appreciate being labelled as
such.
Andrew Walrond
Hi Larry
On Wednesday 19 Nov 2003 12:49 am, Larry McVoy wrote:
>
> Because the translation process is hugely CPU intensive. It takes
> something like 6 hours to do the 2.5 tree on 2.2Ghz Athlon with a gig
> of ram. And it uses every bit of that ram, that's my desktop machine
> and _everything_ is paged out when I come in in the morning.
But once you've done it, incoming changesets are then added incrementally,
right? You don't just convert the tree once a day?
So you'd just need to maintain two copies of the cvs tree, one which currently
available for coherent transmission to lobobk clients, and the other which
can be updated, with a means of swapping them
> I agreed to the BK2CVS stuff as a way of ensuring that people had the
> data in a format that they could use without BK and was useful. I'm
> willing to do that for each main tree of each major project (i.e., if
> XFree86 or something like that moved to BK we'd agree to do the CVS
> conversion so that there was no lockin).
But, as I described in my previous post, it also has at least one useful
commercial application.
> But doing it for every branch of every tree is nuts unless you are
> donating a 16 way with a zillion gigs of ram. And a zillion disk
> arms, this thrashes the heck out of the disk.
The 2.5 tree is <300Mb, so just stick 4Gb in your machine, the bk and cvs tree
in tmpfs and, well, throw the HD away ;)
> > And when you've finished that, I'd like a moon rocket please. GPL, of
> > course ;)
This was in anticipation of somebody calling me a whinging pom. And sure
enough... (And yes, my antipodean mates, we are going to _stuff_ you on
saturday!)
> Yeah, and I'd like all the open source developers of the world to
> acknowledge that I'm a good guy and I'm trying to help. In writing, of
> course ;)
I think everyone who works for a living knows that already :)
On Tue, Nov 18, 2003 at 08:38:51PM +0100, Pascal Schmidt wrote:
> As an aside, I personally rsync the CVS repository to be able to use
> "cvs diff" quickly (no fun having to contact a remote server on ISDN).
> I could use BK but it's not worth the effort for me to learn a new tool.
> So a checkout-only BK would not be useful to me, but I can see reasons
> why people would want one, outlined above.
"I could use BK but it's not worth the effort for me to learn a new tool."
followed by
"a new tool would be useful to me"
doesn't parse. As one person said in private mail, the translation
seems to be "Please write us a new tool because I'm too lazy to learn
the one you last wrote."
Comments like this don't inspire us to start coding.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm