Jeff Garzik wrote:
> If you can't fit a whole tree including metadata into RAM, though,
> BK crawls... Going from "bk citool" at the command line to actually
> seeing the citool window approaches five minutes of runtime, on this
> 200MB laptop... [my dual athlon with 512MB RAM corroborates your
> numbers, though] "bk -r co -Sq" takes a similar amount of time...
>
> I also find that BK brings out the worst in the 2.4 kernel
> elevator/VM... mouse clicks in Mozilla take upwards of 10 seconds to
> respond, when "bk -r co -Sq" is running on this laptop [any other
> read-from-disk process behaves similarly]. And running any two BK
> jobs at the same time is a huge mistake. Two "bk -r co -Sq" runs
> easily take four or more times longer than a single run. Ditto for
> consistency checks, or any other disk-intensive activity BK indulges
> in.
Hello,
What kind of CPU and hard drive do your two machines above have? I'm
a developer for arch[1], and I'm wondering how fast things can get.
Note: If you answer, you'll certainly be aiding arch development. It
might be interpreted as "develop[ing] ... a product which
contains substantially similar capabilities of the BitKeeper
Software, or, in the reasonable opinion of BitMover, competes
with the BitKeeper Software". So you might lose the ability to
use the free license. But I'll let you decide if you want to
help us.
Thank you,
Walter Landry
[email protected]
[1] http://www.fifthvision.net/Arch
Walter Landry wrote:
> Jeff Garzik wrote:
>
>>If you can't fit a whole tree including metadata into RAM, though,
>>BK crawls... Going from "bk citool" at the command line to actually
>>seeing the citool window approaches five minutes of runtime, on this
>>200MB laptop... [my dual athlon with 512MB RAM corroborates your
>>numbers, though] "bk -r co -Sq" takes a similar amount of time...
>>
>>I also find that BK brings out the worst in the 2.4 kernel
>>elevator/VM... mouse clicks in Mozilla take upwards of 10 seconds to
>>respond, when "bk -r co -Sq" is running on this laptop [any other
>>read-from-disk process behaves similarly]. And running any two BK
>>jobs at the same time is a huge mistake. Two "bk -r co -Sq" runs
>>easily take four or more times longer than a single run. Ditto for
>>consistency checks, or any other disk-intensive activity BK indulges
>>in.
>
>
> Hello,
>
> What kind of CPU and hard drive do your two machines above have? I'm
> a developer for arch[1], and I'm wondering how fast things can get.
The laptop has 200MB RAM, and mozilla and a ton of xterms loaded. IDE
drives w/ Intel PIIX4 controller. The Dual Athlon has 512MB RAM, and I
forget what kind of IDE controller -- I think AMD. IDE drives as well.
BitKeeper must scan the entire tree when doing a checkin or checkout, so
that is impossible to optimize at the SCM level without compromising
features... if your source tree takes up ~190MB on disk, you have 200MB
of RAM total, and you need to sequentially scan the entire thing, there
is nothing that can be done at either the OS or app level... You're just
screwed. Things are extremely fast on the Dual Athlon because the
entire tree is in RAM.
> Note: If you answer, you'll certainly be aiding arch development. It
> might be interpreted as "develop[ing] ... a product which
> contains substantially similar capabilities of the BitKeeper
> Software, or, in the reasonable opinion of BitMover, competes
> with the BitKeeper Software". So you might lose the ability to
> use the free license. But I'll let you decide if you want to
> help us.
Don't be silly: I am more than willing to help, via answering
questions. If I had time I would help code. I'm not religious about
"only BitKeeper" to the exclusion of helping open source projects. I
simply use what is most expedient and productive for my kernel
development workflow.
Jeff
> The laptop has 200MB RAM, and mozilla and a ton of xterms loaded. IDE
> drives w/ Intel PIIX4 controller. The Dual Athlon has 512MB RAM, and I
> forget what kind of IDE controller -- I think AMD. IDE drives as well.
>
> BitKeeper must scan the entire tree when doing a checkin or checkout, so
> that is impossible to optimize at the SCM level without compromising
> features... if your source tree takes up ~190MB on disk, you have 200MB
> of RAM total, and you need to sequentially scan the entire thing, there
> is nothing that can be done at either the OS or app level... You're just
> screwed. Things are extremely fast on the Dual Athlon because the
> entire tree is in RAM.
In low memory situations you really want to run the tree compressed.
ON a fast machine do a "bk -r admin -Z" and then clone that onto your
laptop. I think that will drop the tree to about 145MB which will
help, maybe. I suspect that you use enough of the rest of your 200MB
that it still won't fit.
For the checkouts, always do a "bk -r get -S" the -S doesn't check out the
file again if it is already there. We could make that the default but
it is an interface change. A fairly minor one though.
We've got some other fixes in the pipeline for the checkin and integrity
check pass.
There is only so much we can do when you are trying to cram 10 pounds of
crap in a 5 pound bag :(
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Larry McVoy wrote:
>>The laptop has 200MB RAM, and mozilla and a ton of xterms loaded. IDE
>>drives w/ Intel PIIX4 controller. The Dual Athlon has 512MB RAM, and I
>>forget what kind of IDE controller -- I think AMD. IDE drives as well.
>>
>>BitKeeper must scan the entire tree when doing a checkin or checkout, so
>>that is impossible to optimize at the SCM level without compromising
>>features... if your source tree takes up ~190MB on disk, you have 200MB
>>of RAM total, and you need to sequentially scan the entire thing, there
>>is nothing that can be done at either the OS or app level... You're just
>>screwed. Things are extremely fast on the Dual Athlon because the
>>entire tree is in RAM.
>
>
> In low memory situations you really want to run the tree compressed.
> ON a fast machine do a "bk -r admin -Z" and then clone that onto your
> laptop. I think that will drop the tree to about 145MB which will
> help, maybe. I suspect that you use enough of the rest of your 200MB
> that it still won't fit.
Yeah, I don't think that will help at all, given that X and KDE and all
its acoutrements are loaded... I would rather run uncompressed anyway :)
> For the checkouts, always do a "bk -r get -S" the -S doesn't check out the
> file again if it is already there. We could make that the default but
> it is an interface change. A fairly minor one though.
I do "bk -r co -Sq", is the above faster than that?
> We've got some other fixes in the pipeline for the checkin and integrity
> check pass.
>
> There is only so much we can do when you are trying to cram 10 pounds of
> crap in a 5 pound bag :(
indeed :) That's why I keep repeating that it's not BK's fault, and
keep pointing out that my Dual Athlon with plenty of RAM does multiple
simultaneous checks/checkins quite rapidly.
Jeff
On Thu, Oct 10, 2002 at 07:28:18AM -0700, Larry McVoy wrote:
>
> In low memory situations you really want to run the tree compressed.
> ON a fast machine do a "bk -r admin -Z" and then clone that onto your
> laptop. I think that will drop the tree to about 145MB which will
> help, maybe. I suspect that you use enough of the rest of your 200MB
> that it still won't fit.
I believe it's the case that all of the trees on bkbits.net are
compressed, so if you originally cloned your kernel tree off of
bkbits.net, it should already be compressed. Larry, would this be a
correct statement?
- Ted
Jeff Garzik <[email protected]> wrote:
> Walter Landry wrote:
> > Jeff Garzik wrote:
> >
> >>If you can't fit a whole tree including metadata into RAM, though,
> >>BK crawls... Going from "bk citool" at the command line to actually
> >>seeing the citool window approaches five minutes of runtime, on this
> >>200MB laptop... [my dual athlon with 512MB RAM corroborates your
> >>numbers, though] "bk -r co -Sq" takes a similar amount of time...
> >>
> >>I also find that BK brings out the worst in the 2.4 kernel
> >>elevator/VM... mouse clicks in Mozilla take upwards of 10 seconds to
> >>respond, when "bk -r co -Sq" is running on this laptop [any other
> >>read-from-disk process behaves similarly]. And running any two BK
> >>jobs at the same time is a huge mistake. Two "bk -r co -Sq" runs
> >>easily take four or more times longer than a single run. Ditto for
> >>consistency checks, or any other disk-intensive activity BK indulges
> >>in.
> >
> >
> > Hello,
> >
> > What kind of CPU and hard drive do your two machines above have? I'm
> > a developer for arch[1], and I'm wondering how fast things can get.
>
> The laptop has 200MB RAM, and mozilla and a ton of xterms loaded. IDE
> drives w/ Intel PIIX4 controller. The Dual Athlon has 512MB RAM, and I
> forget what kind of IDE controller -- I think AMD. IDE drives as well.
Sorry, I wasn't specific enough. What is the clock speed (MHz) of the
CPU and the RPM and/or results of hparm -t of the hard drive?
> BitKeeper must scan the entire tree when doing a checkin or checkout, so
> that is impossible to optimize at the SCM level without compromising
> features... if your source tree takes up ~190MB on disk, you have 200MB
> of RAM total, and you need to sequentially scan the entire thing, there
> is nothing that can be done at either the OS or app level... You're just
> screwed. Things are extremely fast on the Dual Athlon because the
> entire tree is in RAM.
How fast? If you have, for example, the 2.4.18 tree and apply the
2.4.19 patches, approximately how long does it take for citool to show
up, showing all modified files (less than a second? 10 seconds? a
minute?)
> > Note: If you answer, you'll certainly be aiding arch development. It
> > might be interpreted as "develop[ing] ... a product which
> > contains substantially similar capabilities of the BitKeeper
> > Software, or, in the reasonable opinion of BitMover, competes
> > with the BitKeeper Software". So you might lose the ability to
> > use the free license. But I'll let you decide if you want to
> > help us.
>
>
> Don't be silly: I am more than willing to help, via answering
> questions.
Thanks,
Walter Landry
[email protected]
On Thu, Oct 10, 2002 at 07:28:18AM -0700, Larry McVoy wrote:
> > The laptop has 200MB RAM, and mozilla and a ton of xterms loaded. IDE
> > drives w/ Intel PIIX4 controller. The Dual Athlon has 512MB RAM, and I
> > forget what kind of IDE controller -- I think AMD. IDE drives as well.
> >
> > BitKeeper must scan the entire tree when doing a checkin or checkout, so
[...]
> In low memory situations you really want to run the tree compressed.
> ON a fast machine do a "bk -r admin -Z" and then clone that onto your
> laptop. I think that will drop the tree to about 145MB which will
> help, maybe. I suspect that you use enough of the rest of your 200MB
> that it still won't fit.
[...]
> There is only so much we can do when you are trying to cram 10 pounds of
> crap in a 5 pound bag :(
The reason that one or two years ago my "diff+multiple trees" beat
bitkeeper on the performance front was that diff would only touch
inode-metadata, and not the files themselves. You can cache the
file-metadata (inodes) of a 200M tree in a couple of megabytes of RAM.
Roger.
--
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* The Worlds Ecosystem is a stable system. Stable systems may experience *
* excursions from the stable situation. We are currenly in such an *
* excursion: The stable situation does not include humans. ***************
On Fri, Oct 11, 2002 at 03:35:38PM +0200, Rogier Wolff wrote:
> On Thu, Oct 10, 2002 at 07:28:18AM -0700, Larry McVoy wrote:
> > > The laptop has 200MB RAM, and mozilla and a ton of xterms loaded. IDE
> > > drives w/ Intel PIIX4 controller. The Dual Athlon has 512MB RAM, and I
> > > forget what kind of IDE controller -- I think AMD. IDE drives as well.
> > >
> > > BitKeeper must scan the entire tree when doing a checkin or checkout, so
>
> [...]
>
> > In low memory situations you really want to run the tree compressed.
> > ON a fast machine do a "bk -r admin -Z" and then clone that onto your
> > laptop. I think that will drop the tree to about 145MB which will
> > help, maybe. I suspect that you use enough of the rest of your 200MB
> > that it still won't fit.
>
> [...]
> > There is only so much we can do when you are trying to cram 10 pounds of
> > crap in a 5 pound bag :(
>
> The reason that one or two years ago my "diff+multiple trees" beat
> bitkeeper on the performance front was that diff would only touch
> inode-metadata, and not the files themselves. You can cache the
> file-metadata (inodes) of a 200M tree in a couple of megabytes of RAM.
Jeff:
The trick is that I would have hardlinked trees. Thus
linux-2.2.18.clean/drivers/char/tty_ioctl.c
and
linux-2.2.18.rio/drivers/char/tty_ioctl.c
would have the same inode number, and diff wouldn't even bother to
open the file....
Roger.
--
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* The Worlds Ecosystem is a stable system. Stable systems may experience *
* excursions from the stable situation. We are currenly in such an *
* excursion: The stable situation does not include humans. ***************
On Fri, Oct 11, 2002 at 04:08:28PM +0200, Rogier Wolff wrote:
> Jeff:
>
> The trick is that I would have hardlinked trees. Thus
>
> linux-2.2.18.clean/drivers/char/tty_ioctl.c
> and
> linux-2.2.18.rio/drivers/char/tty_ioctl.c
>
> would have the same inode number, and diff wouldn't even bother to
> open the file....
FYI:
Just tested the how long it takes to make a diff between my
2.2.18-clean tree and the 2.2.18.rio tree:
cold cache over NFS: 25 seconds
warm cache over NFS: 13 seconds
warm cache LOCAL disk: 0.34 seconds.
Roger.
--
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* The Worlds Ecosystem is a stable system. Stable systems may experience *
* excursions from the stable situation. We are currenly in such an *
* excursion: The stable situation does not include humans. ***************