LinuxLists.cc - log-buf-len dynamic

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Hi,

(it was from me)

On Mon, Sep 22, 2003 at 09:48:33PM +0200, Andrea Arcangeli wrote:
> Hi,
>
> I'm rejecting on the log-buf-len feature in 2.4.23pre5, the code in
> mainline is worthless for any distributor, shipping another rpm package
> just for the bufsize would be way overkill.
>
> Please backout the below (extracted from bkcvs) and apply this one
> instead:
>
> http://www.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.22aa1/00_log-buf-len-1

Well, Andrea, I've looked at your patch. I really like the dynamic size
reconfiguration, but:
- now it becomes mandatory to add a command line option, just for this. It's
annoying when you want to build install images for many systems. I've
always considered that command line parameters should be limited to the
strict minimum to have a system boot reliably (root=, console=, very few
IDE/SCSI/ACPI tuning when absolutely needed) and that's all.
- what does the initial __log_buf[] become after log_buf_len_setup() ? can
these 64 kB be freed or are they definitely lost ?

I think that perhaps we should merge the two things, but reconfigure them
differently :

- be able to specify de DEFAULT buffer size at compile time.
- have it reconfigurable at run time with a sysctl (this could be something
next to 'prink', or even a write to kmsg). This way, if you detect that
your system is still loosing messages under load, you have a chance to
catch them all.
- initialize the buffer with allocated memory from the beginning so that we
can free it when changing the buffer size.

I can spend a few hours working with you on this if you're interested. But be
assured that I know enough people who would complain about being forced to
add a new boot option to their lilo.conf.

Cheers,
Willy

2003-09-23 12:46:34

by Daniel Jacobowitz

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Mon, Sep 22, 2003 at 09:48:33PM +0200, Andrea Arcangeli wrote:
> +#if !defined(CONFIG_LOG_BUF_SHIFT) || (CONFIG_LOG_BUF_SHIFT - 0 == 0)

It shouldn't have an empty string value anyway, so this is just:
#if !CONFIG_LOG_BUF_SHIFT

That handles both !defined and !0.

--
Daniel Jacobowitz
MontaVista Software Debian GNU/Linux Developer

2003-09-23 12:49:31

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 06:28:55AM +0200, Willy Tarreau wrote:
> Hi,
>
> (it was from me)
>
> On Mon, Sep 22, 2003 at 09:48:33PM +0200, Andrea Arcangeli wrote:
> > Hi,
> >
> > I'm rejecting on the log-buf-len feature in 2.4.23pre5, the code in
> > mainline is worthless for any distributor, shipping another rpm package
> > just for the bufsize would be way overkill.
> >
> > Please backout the below (extracted from bkcvs) and apply this one
> > instead:
> >
> > http://www.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.22aa1/00_log-buf-len-1
>
> Well, Andrea, I've looked at your patch. I really like the dynamic size
> reconfiguration, but:
> - now it becomes mandatory to add a command line option, just for this. It's
> annoying when you want to build install images for many systems. I've
> always considered that command line parameters should be limited to the
> strict minimum to have a system boot reliably (root=, console=, very few
> IDE/SCSI/ACPI tuning when absolutely needed) and that's all.
> - what does the initial __log_buf[] become after log_buf_len_setup() ? can
> these 64 kB be freed or are they definitely lost ?
>
> I think that perhaps we should merge the two things, but reconfigure them
> differently :
>
> - be able to specify de DEFAULT buffer size at compile time.
> - have it reconfigurable at run time with a sysctl (this could be something
> next to 'prink', or even a write to kmsg). This way, if you detect that
> your system is still loosing messages under load, you have a chance to
> catch them all.
> - initialize the buffer with allocated memory from the beginning so that we
> can free it when changing the buffer size.
>
> I can spend a few hours working with you on this if you're interested. But be
> assured that I know enough people who would complain about being forced to
> add a new boot option to their lilo.conf.

The point here is that the default must work for 99% of the userbase.
Either that or the default is totally broken.

So the rest of 1% should be ok to add the command line, and they should
be very happy that when they overflow again even with 128k because their
cpu is too slow to keep up in klogd or whatever, they can change it to
256k without replacing kernels, some embedded platforms especially
should like it. And only this 1% would be the one recompiling the kernel
by themself anyways, and if they can recompile the kernel they can as
well edit the defaults in kernel/printk.c without pain.

I don't buy much the lazyness argument in changing lilo.conf, the people
who need this feature is a marginal part so they must be ok with the
parmeter. And don't tell me that you don't pass root= to the kernel at
boot. Do you want to fix that too? I do pass plenty of argumetns all the
time, starting from profile=0 (and often acpi=off).

however I won't complain if you put the compile time configurator on top
of my patch (that's easy) but personaly I think it's not needed, and
having it dyanmic is an order of magnitude more important than having it
static (again: if you can afford to recompile the kernel than you could
edit kernel/printk.c in the first place without much slowdown).

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-23 14:08:06

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Hi Andrea,

> The point here is that the default must work for 99% of the userbase.
> Either that or the default is totally broken.

You're only thinking about what distro vendors should do. If all kernels should
suit their needs, then there would not be any config anymore, and everyone
would have the same kernel with the same wagon of command line arguments.

Lots of people actually do recompile their kernel to change a few options from
the vendor config, and that's the expected behaviour. And be it 1%, I consider
that 1% of Linux users are *lots* of people.

> And only this 1% would be the one recompiling the kernel
> by themself anyways, and if they can recompile the kernel they can as
> well edit the defaults in kernel/printk.c without pain.

I don't agree. That's what I did for a long time, and finally got bored of it.
Then I wrote a patch and proposed it. Randy Dunlap had one included in 2.6, and
Marcelo thought this one would be useful to lots of people, *exactly* because
it avoids what you describe here, ie. changins constants by hand.

> I don't buy much the lazyness argument in changing lilo.conf, the people
> who need this feature is a marginal part so they must be ok with the
> parmeter. And don't tell me that you don't pass root= to the kernel at
> boot. Do you want to fix that too? I do pass plenty of argumetns all the
> time, starting from profile=0 (and often acpi=off).

sorry, I don't agree with you on this. My lilo.conf contains "root=/dev/root"
and "boot=/dev/mbr" on nearly all my systems, just because it's easier ton
maintain, especially those which are patched remotely. You are speaking about
a developer's system, and I too have lots of crap on my desktop machines, and I
too am regularly annoyed because lilo tells me that 19 boot images is the max.
But on production systems, you need to avoid tricks, and even more when other
people manage the system after you install it.

> however I won't complain if you put the compile time configurator on top
> of my patch (that's easy) but personaly I think it's not needed, and
> having it dyanmic is an order of magnitude more important than having it
> static

I said I agree on the dynamic advantage. BTW, you didn't reply me about what
the statically allocated 64 kB became with your patch. Will this memory be
wasted forever or is there a way to free it ? If I want to pre-allocate it
dynamically with alloc_space() as you did, what can I use to free it ? is
kfree() the right thing ?

Cheers,
Willy

2003-09-23 14:44:11

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 04:06:47PM +0200, Willy Tarreau wrote:
> Hi Andrea,
>
> > The point here is that the default must work for 99% of the userbase.
> > Either that or the default is totally broken.
>
> You're only thinking about what distro vendors should do. If all kernels should
> suit their needs, then there would not be any config anymore, and everyone
> would have the same kernel with the same wagon of command line arguments.

this is actually not true. Many options are way too costly to recompile
every time, or to keep enabled at runtime, it can't be the case for this
one. My own self compiled kernel has a very stripped down .config, it's
not like a distro kernel with everything included.

> I said I agree on the dynamic advantage. BTW, you didn't reply me about what
> the statically allocated 64 kB became with your patch. Will this memory be

the 64k are wasted only when you use the option, if you need more than
64k it's unlikely you care about 64k lost. But I'm ok to fix it, it just
looks a low priority to fix. Also since you will never use this option
anyways since you don't like adding parameters to the kernel, then you
don't care about fixing it either, you simply want the additional config
option on top of this.

My whole argument is that being able to do it dynamic is an order of
magnitude more important than being able to do it statically, no matter
the command line argument and no matter ther 64k lost. You must agree
with that. the amount of userbase is just not comparable.

> wasted forever or is there a way to free it ? If I want to pre-allocate it
> dynamically with alloc_space() as you did, what can I use to free it ? is
> kfree() the right thing ?

we should simply allocate it from the bootmem pool, it's trivial to
change it so the 64k aren't lost.

Not sure what I should do, personally I consider the additional .config
option as configuration pollution, but since you care that much I can
also change my patch not to reject yours, and to allow the two things to
coexist, but I don't think it's really needed. I believe people should
be forced to use the kernel params, and it's much more manageable to
know all the kernels behave the same if no param is passed (otherwise
you have to check the System.map to see how big the buffer is, to know
if this was the right or wrong kernel during your remote maintainance,
either that or you should check /proc/config.gz if you have it). It's
just a mess if each kernel with the same revision number behaves
differently, everything becomes less and less predictable. The number of
System.map checks increases and increases before I can identify what
kernel is that. Is there anybody else that has opinions on the matter?

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-23 15:02:33

by Jan Evert van Grootheest

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Andrea,

since you ask...
I consider myself a not very typical Linux user in that I am a
programmer too and I (sometimes) write programs for Linux as well. And
yes, I do compile my own kernels.

I think it is pretty senseless to have a configuation option for the
default size of that buffer. Especially if I can, in one of the first rc
scripts, do something like 'echo 128 > /proc/sys/kernel/printkbuffer'.
The only hard requirement for that default size is that all output up
till that rc script fits into a buffer of that size.

And yes, I do care about that static 64K buffer. I still have an old
pentium that only has 16M. Every byte counts!

On the other hand there are the really large systems with many CPUs and
such. Those might have problems with the small default. But, well, those
really could #ifdef a different default based on some configuration
options...

-- Jan Evert

Andrea Arcangeli wrote:

> On Tue, Sep 23, 2003 at 04:06:47PM +0200, Willy Tarreau wrote:
>
>>Hi Andrea,
>>
>>
>>>The point here is that the default must work for 99% of the userbase.
>>>Either that or the default is totally broken.
>>
>>You're only thinking about what distro vendors should do. If all kernels should
>>suit their needs, then there would not be any config anymore, and everyone
>>would have the same kernel with the same wagon of command line arguments.
>
>
> this is actually not true. Many options are way too costly to recompile
> every time, or to keep enabled at runtime, it can't be the case for this
> one. My own self compiled kernel has a very stripped down .config, it's
> not like a distro kernel with everything included.
>
>
>>I said I agree on the dynamic advantage. BTW, you didn't reply me about what
>>the statically allocated 64 kB became with your patch. Will this memory be
>
>
> the 64k are wasted only when you use the option, if you need more than
> 64k it's unlikely you care about 64k lost. But I'm ok to fix it, it just
> looks a low priority to fix. Also since you will never use this option
> anyways since you don't like adding parameters to the kernel, then you
> don't care about fixing it either, you simply want the additional config
> option on top of this.
>
> My whole argument is that being able to do it dynamic is an order of
> magnitude more important than being able to do it statically, no matter
> the command line argument and no matter ther 64k lost. You must agree
> with that. the amount of userbase is just not comparable.
>
>
>>wasted forever or is there a way to free it ? If I want to pre-allocate it
>>dynamically with alloc_space() as you did, what can I use to free it ? is
>>kfree() the right thing ?
>
>
> we should simply allocate it from the bootmem pool, it's trivial to
> change it so the 64k aren't lost.
>
> Not sure what I should do, personally I consider the additional .config
> option as configuration pollution, but since you care that much I can
> also change my patch not to reject yours, and to allow the two things to
> coexist, but I don't think it's really needed. I believe people should
> be forced to use the kernel params, and it's much more manageable to
> know all the kernels behave the same if no param is passed (otherwise
> you have to check the System.map to see how big the buffer is, to know
> if this was the right or wrong kernel during your remote maintainance,
> either that or you should check /proc/config.gz if you have it). It's
> just a mess if each kernel with the same revision number behaves
> differently, everything becomes less and less predictable. The number of
> System.map checks increases and increases before I can identify what
> kernel is that. Is there anybody else that has opinions on the matter?
>
> Andrea - If you prefer relying on open source software, check these links:
> rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
> http://www.cobite.com/cvsps/
> svn://svn.kernel.org/linux-2.[46]/trunk
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2003-09-23 15:41:43

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 05:01:26PM +0200, Jan Evert van Grootheest wrote:
> Andrea,
>
> since you ask...
> I consider myself a not very typical Linux user in that I am a
> programmer too and I (sometimes) write programs for Linux as well. And
> yes, I do compile my own kernels.
>
> I think it is pretty senseless to have a configuation option for the
> default size of that buffer. Especially if I can, in one of the first rc
> scripts, do something like 'echo 128 > /proc/sys/kernel/printkbuffer'.

having a sysctl can be an additional option (though it's tricky to
implement due the needed callbacks), but the problem I guess is that
most people needs a larger buf for the boot logs, so having only the
sysctl would be too late...

> The only hard requirement for that default size is that all output up
> till that rc script fits into a buffer of that size.
>
> And yes, I do care about that static 64K buffer. I still have an old
> pentium that only has 16M. Every byte counts!

Note however that on the old 16M pentium it's unlikely that your boot
logs will overflow the current 64k, so for your pentium I doubt there
would be any need to use the feature. At least from my point of view the
feature is meant mainly for the high end, like 16/32-way HT with 64G of
ram etc.. where 64k hardly matters.

> On the other hand there are the really large systems with many CPUs and
> such. Those might have problems with the small default. But, well, those
> really could #ifdef a different default based on some configuration
> options...

note the 64k are only wasted when you use the feature, there's nothing
wasted if you don't use it.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-23 16:07:48

[permalink] [raw]

Subject: Re: log-buf-len dynamic

[replying to both Jan and Andrea at once]

-- Jan:
> I think it is pretty senseless to have a configuation option for the
> default size of that buffer. Especially if I can, in one of the first rc
> scripts, do something like 'echo 128 > /proc/sys/kernel/printkbuffer'.
> The only hard requirement for that default size is that all output up
> till that rc script fits into a buffer of that size.

I totally agree with this possibility, and it's one I proposed too. I only
proposed to keep the default size in order to be able to catch all boot
messages, but I think that if we make this buffer hot-resizable (and a sysctl
is good for this), then we could systematically pre-allocate 64k at boot which
should be enough for boot messages (16k is too short), and could be reduced
if needed by a write to the sysctl. This way, the config option can go away.

> And yes, I do care about that static 64K buffer. I still have an old
> pentium that only has 16M. Every byte counts!

My previous firewall add 8M, and I second your statement : every byte counts !

> Andrea Arcangeli wrote:
> >>You're only thinking about what distro vendors should do. If all kernels
> >>should
> >>suit their needs, then there would not be any config anymore, and everyone
> >>would have the same kernel with the same wagon of command line arguments.
> >
> >
> >this is actually not true. Many options are way too costly to recompile
> >every time, or to keep enabled at runtime, it can't be the case for this
> >one. My own self compiled kernel has a very stripped down .config, it's
> >not like a distro kernel with everything included.

I think you're mistaken about what I meant. I didn't mean that I want to
recompile each kernel with a different option, I meant that for a large set
of identical systems, it's more convenient to have a single, stripped down
.config with the most default values, and propagate this same kernel everywhere,
than having to duplicate special kernel parameters in every lilo.conf.

> >the 64k are wasted only when you use the option, if you need more than
> >64k it's unlikely you care about 64k lost.

Absolutely, my concern was most about the case where I need less than 64k and
I have too few memory to agree to waste them.

> > But I'm ok to fix it, it just
> >looks a low priority to fix. Also since you will never use this option
> >anyways since you don't like adding parameters to the kernel, then you
> >don't care about fixing it either, you simply want the additional config
> >option on top of this.

:-)
Andrea, I'm not *that* selfish ! If I did this only for me, I wouldn't even
post it here and risk complaints from people who ask me to add other archs
too. I can live with this as a patch in my kernel, next to your VM and others.
It's just that I felt a need for it on production machines and thought there
were potentially other people in my case. In the same way, I could say that
you're wrong in believing that everyone wants to add a boot option, but I may
be wrong WRT this. That's not the problem. If I asked how to fix it, it's
because I'm interested in doing it myself, but I'm clueless about how to do it
(don't know how to allocate boot mem, don't know how to free it, don't know
if I can reallocate new mem further and free it again by the same means).

> >My whole argument is that being able to do it dynamic is an order of
> >magnitude more important than being able to do it statically, no matter
> >the command line argument and no matter ther 64k lost. You must agree
> >with that. the amount of userbase is just not comparable.

I said I did agree with that, but it's not the same feature. And concerning
the amount of userbase, I agree it's not comparable... I don't imagine desktop
users adding "logbuflen=1048576" to their boot command line, nor can I imagine
them recompiling their kernel changing this option.

> >Not sure what I should do, personally I consider the additional .config
> >option as configuration pollution, but since you care that much I can
> >also change my patch not to reject yours, and to allow the two things to
> >coexist, but I don't think it's really needed. I believe people should
> >be forced to use the kernel params,
^^^^^^^^^^^^^^^^^^^
why do you want to force people to do something the way you think is best ?
I personnaly think that both solutions are a convenient way of achieving the
same goal for different people. Please let them choose.

> >just a mess if each kernel with the same revision number behaves
> >differently, everything becomes less and less predictable.

nothing new here. There are already many options which will make your kernel
behave differently : CPU/ACPI/APM/SCSI/IDE DMA by default/Shutdown Watchdog on
close, to name a few... That's why many of us add a version suffix to
production kernels identifying a known .config.

> >The number of System.map checks increases and increases before I can
> >identify what kernel is that.

Then don't rely on the default value for your kernels, but only on the boot
option, and the problem is gone. Let us all have the choice. I insist that
your patch is appealing, but it is not the only solution and does not cover
all needs, nor does mine.

Cheers,
Willy

2003-09-23 16:11:26

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 05:41:37PM +0200, Andrea Arcangeli wrote:

> note the 64k are only wasted when you use the feature, there's nothing
> wasted if you don't use it.

There's 48k wasted compared to the default 16k, because Jan will not have the
choice of not allocating them with your current patch. And I'm not sure he'll
be happy to follow your advice, to tap into the source file to get them back :-)

Willy

2003-09-23 16:23:33

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 06:06:00PM +0200, Willy Tarreau wrote:
> why do you want to force people to do something the way you think is best ?

because it gives you no real disavantage to pass the parameter in grub.
You've to set root=/dev/root anyways, this way you would simply need to
add the parameter for log buf as well. I've always to add lots of other
stuff anyways, including vga= profile= etc..

> I personnaly think that both solutions are a convenient way of achieving the
> same goal for different people. Please let them choose.

they can already choose with the parameter at boot, it's not that I
don't let them choose.

You simply want *two* different APIs, where one is worhtless and
obsoleted by the more powerful one.

That's why I think it's useless to have *two* APIs where one overlaps
on the other *completely*.

My API is good for everyone, yours is not, I don't see why we've to keep
both when nobody depends on yours yet.

> your patch is appealing, but it is not the only solution and does not cover
> all needs, nor does mine.

I can easily fix the 64k ram waste, after that I can't see why my API
can't cover you needs. The init code even gets released because it's
__init.

I just don't buy the fact that you don't like to pass params to the
kernel because you already do, you have to or it won't boot on a system
different than the one that you compiled the kernel.

If you don't like to pass buf_log_len= then how can you be ok to pass
root=/dev/root on all your thousand of lilo.conf in production? How can
root=/dev/root be remotely acceptable if buf_log_len= can't be as well
added to your thousand of distributed lilo.confs for your all images out
there? This makes no sense at all to me.

Yeah, I'm forcing you to waste a dozen bytes of disk space in
/etc/lilo.conf, you can definitely claim that, but I don't that as an
argument either (delete the spaces/tabs and you may save more). And
maybe thanks to this you won't run into the number of kernel images
limitation anymore.

I'm deinitely against worthless APIs in the kernel, when more powerful
one exists, and they give you zero disavantage (I will fix the 64k waste
don't worry, I definitely agree that such bit must be fixed to make
everyone happy).

As for decreasing the buf size, my option will allow you to decrease it
too after I fix it! While the shift only let you increase the buf size,
not decrease. So you should prefer the dynamic API too.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-23 16:26:52

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 06:09:41PM +0200, Willy Tarreau wrote:
> On Tue, Sep 23, 2003 at 05:41:37PM +0200, Andrea Arcangeli wrote:
>
> > note the 64k are only wasted when you use the feature, there's nothing
> > wasted if you don't use it.
>
> There's 48k wasted compared to the default 16k, because Jan will not have the

48k wasted where? What arch, I thought you meant the original log_buf
memory is not released, I will fix that, then the waste will be zero and
there will be no advantage in having it at compile time anymore, only
disavantages.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-23 16:56:10

by Ruth Ivimey-Cook

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, 23 Sep 2003, Andrea Arcangeli wrote:

>On Tue, Sep 23, 2003 at 05:01:26PM +0200, Jan Evert van Grootheest wrote:
>> I think it is pretty senseless to have a configuation option for the
>> default size of that buffer. Especially if I can, in one of the first rc
>> scripts, do something like 'echo 128 > /proc/sys/kernel/printkbuffer'.
>
>having a sysctl can be an additional option (though it's tricky to
>implement due the needed callbacks), but the problem I guess is that
>most people needs a larger buf for the boot logs, so having only the
>sysctl would be too late...

IMO there are two issues:

1. On some systems it is possible to overflow the log buffer during boot and
before init runs

2. On some systems, there is enough going on that klogd cannot read the log
quick enough and so the log is missing lines.

IME (1) is the irritating one. I can (and have) edited the source, but it is
irritating. Setting up the config option is ok, but for default kernels (e.g.
a distro one) where you can't easily recompile, it's insufficient.

Proposal:

1. The default buffer size is huge: 128K or more, which should enable the
initial mesages to be kept.

2. The default buffer size is modifiable on the command line, but does not
have to be set there (mainly for those with v. limited RAM).

3. The log buffer can be resized smaller and bigger: using sysctl seems nice,
but might a dmesg option be ok?

4. klogd and dmesg, at all times, can tell how big the buffer is and where the
start and end are. That is, there is no need for the user to tell dmesg how
big the buffer is.

5. Lets just do something and move on: it's not important enough to waste
weeks talking about :-)

HTH,

Ruth

--
Ruth Ivimey-Cook
Software engineer and technical writer.

2003-09-23 17:40:32

by Tom Zanussi

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Ruth Ivimey-Cook wrote:
> On Tue, 23 Sep 2003, Andrea Arcangeli wrote:
>
>
>>On Tue, Sep 23, 2003 at 05:01:26PM +0200, Jan Evert van Grootheest wrote:
>>
>>>I think it is pretty senseless to have a configuation option for the
>>>default size of that buffer. Especially if I can, in one of the first rc
>>>scripts, do something like 'echo 128 > /proc/sys/kernel/printkbuffer'.
>>
>>having a sysctl can be an additional option (though it's tricky to
>>implement due the needed callbacks), but the problem I guess is that
>>most people needs a larger buf for the boot logs, so having only the
>>sysctl would be too late...
>
>
> IMO there are two issues:
>
> 1. On some systems it is possible to overflow the log buffer during boot and
> before init runs
>
> 2. On some systems, there is enough going on that klogd cannot read the log
> quick enough and so the log is missing lines.
>
> IME (1) is the irritating one. I can (and have) edited the source, but it is
> irritating. Setting up the config option is ok, but for default kernels (e.g.
> a distro one) where you can't easily recompile, it's insufficient.
>
>
> Proposal:
>
> 1. The default buffer size is huge: 128K or more, which should enable the
> initial mesages to be kept.
>
> 2. The default buffer size is modifiable on the command line, but does not
> have to be set there (mainly for those with v. limited RAM).
>
> 3. The log buffer can be resized smaller and bigger: using sysctl seems nice,
> but might a dmesg option be ok?
>
> 4. klogd and dmesg, at all times, can tell how big the buffer is and where the
> start and end are. That is, there is no need for the user to tell dmesg how
> big the buffer is.
>
> 5. Lets just do something and move on: it's not important enough to waste
> weeks talking about :-)
>
> HTH,
>
> Ruth
>
>

Hi,

It may be a little over the top for 2.4, but I posted a patch for a
dynamically resizable printk a while back:

http://marc.theaimsgroup.com/?l=linux-kernel&m=105828314919697&w=2

I depends on relayfs, which I also posted at the same time.

Basically, the idea is that the static printk buffer is used only at
init time for boot messages and is then discarded (so can be made very
large), and after that point printk uses the relayfs channel, which is
automatically grown and shrunk as necessary. Config options control the
buffer sizes (LOG_BUF_SHIFT for initial static, MIN_LOG_BUF_SHIFT for
the minimum dynamic) and the relay_open() call controls the maximum
dynamic size and other channel characteristics.

Changes I'm currently making to relayfs should simplify the patch
significantly and fix some other problems, but I thought I'd point it
out as an alternative approach.

Tom

2003-09-23 17:53:28

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 12:40:06PM -0500, Tom Zanussi wrote:
> It may be a little over the top for 2.4, but I posted a patch for a
> dynamically resizable printk a while back:
>
> http://marc.theaimsgroup.com/?l=linux-kernel&m=105828314919697&w=2
>
> I depends on relayfs, which I also posted at the same time.

I don't think we can merge this in 2.4 but it looks good for 2.6.
However note you can't close the overflow issue, you must still have an
high limit, but you can make lots more reliable that way, it's similar
to netlink anyways, that uses dynamic skb to avoid wasting ram. But it's too
invasive for 2.4, just to make a comparison look what my patch looks
like (it will get a bit more complicated because of the request not to
waste the 64k, you know we throw some hundred megs in the per-cpu
freelists so nobody could notice the 64k lost in the 16-way..):

diff -urNp --exclude CVS --exclude BitKeeper x-ref/kernel/printk.c x/kernel/printk.c
--- x-ref/kernel/printk.c 2003-08-26 00:13:17.000000000 +0200
+++ x/kernel/printk.c 2003-09-01 21:38:16.000000000 +0200
@@ -30,13 +30,13 @@
#include <asm/uaccess.h>

#if defined(CONFIG_MULTIQUAD) || defined(CONFIG_IA64)
-#define LOG_BUF_LEN (65536)
+#define __LOG_BUF_LEN (65536)
#elif defined(CONFIG_ARCH_S390)
-#define LOG_BUF_LEN (131072)
+#define __LOG_BUF_LEN (131072)
#elif defined(CONFIG_SMP)
-#define LOG_BUF_LEN (32768)
+#define __LOG_BUF_LEN (32768)
#else
-#define LOG_BUF_LEN (16384) /* This must be a power of two */
+#define __LOG_BUF_LEN (16384) /* This must be a power of two */
#endif

#define LOG_BUF_MASK (LOG_BUF_LEN-1)
@@ -78,7 +78,9 @@ struct console *console_drivers;
*/
static spinlock_t logbuf_lock = SPIN_LOCK_UNLOCKED;

-static char log_buf[LOG_BUF_LEN];
+static char __log_buf[__LOG_BUF_LEN];
+static char * log_buf = __log_buf;
+static int LOG_BUF_LEN = __LOG_BUF_LEN;
#define LOG_BUF(idx) (log_buf[(idx) & LOG_BUF_MASK])

/*
@@ -151,6 +153,39 @@ static int __init console_setup(char *st

__setup("console=", console_setup);

+static int __init log_buf_len_setup(char *str)
+{
+ unsigned long size = simple_strtoul(str, &str, 0);
+
+ if (size & (size-1))
+ printk("log_buf_len: invalid size - needs a power of two\n");
+
+ spin_lock_irq(&logbuf_lock);
+ if (size > LOG_BUF_LEN) {
+ unsigned long start, end, dest_idx, offset;
+
+ LOG_BUF_LEN = size;
+ log_buf = alloc_exact(LOG_BUF_LEN);
+
+ offset = start = min(con_start, log_start);
+ end = log_end;
+ dest_idx = 0;
+ while (start != end) {
+ log_buf[dest_idx] = __log_buf[start & (__LOG_BUF_LEN - 1)];
+ start++;
+ dest_idx++;
+ }
+ log_start -= offset;
+ con_start -= offset;
+ log_end -= offset;
+ }
+ spin_unlock_irq(&logbuf_lock);
+
+ return 1;
+}
+
+__setup("log_buf_len=", log_buf_len_setup);
+
/*
* Commands to do_syslog:
*

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-23 19:03:40

[permalink] [raw]

Subject: Re: log-buf-len dynamic

[Cc: list stripped down]

On Tue, Sep 23, 2003 at 06:23:19PM +0200, Andrea Arcangeli wrote:

> You simply want *two* different APIs, where one is worhtless and
> obsoleted by the more powerful one.

I never said I wanted *two* of them. I proposed first a fairly non-intrusive
one which doesn't add any line of code, just defines, and managed to reunify
architectures which used different values by default.

*You* proposed a second one. I have no objections against it, as I said, since
I certainly understand that some people would prefer to use it. But *you*
decided that mine was totally unusable and obsoleted by yours, reason for it to
be removed. And this is *that* I objected to. Since the can both coexist and
fit different people's needs without adding a byte of code (at least for mine),
I don't see the goal in eliminating it, deciding unilaterally that nobody will
use it.

I repeat, at the end, I don't mind. I have the patch and can live with one
more patch, as I did before. It's just the fact that you decide for everyone
that always adding a command line option is more convenient than a once-for-all
fixed config option.

> My API is good for everyone, yours is not

I'm impressed that you know so many people. I know that mine at least
satisfies a few collegues, customers, and I. So I deduced that it might be
useful to others too. Even Marcelo thought the same a time ago.

> I don't see why we've to keep both when nobody depends on yours yet.

Nobody depends on mine, since it doesn't change defaults nor semantics. Perhaps
some people depend on yours *not* being in. You changed LOG_BUF_LEN from a
define to a static, and THAT could break other external patches. BTW, I've
just checked 2.4.22aa1, I noticed that you removed kmsgdump, is it because it
doesn't apply, compile or work anymore since your API change ?

Honnestly, I simply took a conservative approach. My proposal not being what
will save the world, I'm 100% with you on that. Mine being useless to everyone,
I'm only 20% with you on that. Yours being "good for everyone", I'm only 20%
with you on that too, and perhaps less since it might break some compatibility.
Yours *is* interesting to some aspects, but is not an equivalent.

> I just don't buy the fact that you don't like to pass params to the
> kernel because you already do, you have to or it won't boot on a system
> different than the one that you compiled the kernel.

Believe it or not, there are people who don't like to pass params to the
kernel. I just told a friend while I was typing this mail, and he replied me
"he must be joking !". So at least, we're two on earth.

> Yeah, I'm forcing you to waste a dozen bytes of disk space in
> /etc/lilo.conf, you can definitely claim that, but I don't that as an
> argument either (delete the spaces/tabs and you may save more). And
> maybe thanks to this you won't run into the number of kernel images
> limitation anymore.

OK Andrea, you're in a bad mood today. You're trying to prove me crazy. I
won't fight. You're the kernel hacker. Not me. I have never been proud of
putting a patch in the kernel and never will be. I'm more ashamed when I send
an obvious "fix" which breaks someone's setup. Your attitude today seems to be
the opposite. Your patch is useful, I won't contest that. It won't annoy anyone,
I'll let others report it to you themselves, I've done my part of it. But they
will waste their time since you don't want to listen and will try to sell your
patch anyway because "it's good for everyone" as you stated it.

> I'm deinitely against worthless APIs in the kernel, when more powerful
> one exists, and they give you zero disavantage

There are people out there who are against worthless lines in config files,
and against APIs which break other patches.

> As for decreasing the buf size, my option will allow you to decrease it
> too after I fix it! While the shift only let you increase the buf size,
> not decrease. So you should prefer the dynamic API too.

I don't see why you say that ! Set LOG_BUF_SHIFT to 10 and you'll have 1kB !

Have a good night anyway,
Willy

2003-09-23 21:08:06

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 02:51:21AM +0100, Matthew Wilcox wrote:
> On Mon, Sep 22, 2003 at 09:48:33PM +0200, Andrea Arcangeli wrote:
> > Date: 2003/08/26 19:28:13
> > Author: willy
>
> Must be a different willy.
>
> > I added the config line to all archs. I used copy paste, so either I broke them
> > all, either none. I'll start a compilation on my alpha while I'm on the way
> > home, to check. I you have questions, please do ask.
>
> ... because I don't have an Alpha ;-)

;)

if Marcelo would be using open source code to exports the patchsets in
his tree, we could fix it to add the email address along the name in the
checkin logs metadata, to avoid this sort of mistakes.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-23 21:53:37

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, 23 Sep 2003, Andrea Arcangeli wrote:
>
> if Marcelo would be using open source code to exports the patchsets in
> his tree, we could fix it to add the email address along the name in the
> checkin logs metadata, to avoid this sort of mistakes.

Andrea - please just shut up.

Until you can point to anything even _remotely_ as good as BitKeeper,
there's no point in just continually trying to start a flame-war.

I agree that Larry ends up being a jerk about it too, but I can well see
that he reacts negatively to your totally unproductive comments.

BK has the email in the meta-data already, and a lot of tools to extract
all the necessary data from mailboxes etc.

In other words, your argument is nonexistant, and your whole posting is
pointless - except as flame-bait. And yes, Larry likely _will_ eat your
bait, something I've berated him for in private emails several times.

So shut up or put up. Go off and write your own tool. In the meantime,
stop complaining about people who wrote _their_ tools and selected a
license that you wouldn't choose.

When you write some code of your own, you get to choose the license. And I
haven't seen you make CVS usable - I've only seen you bitch, moan, and
complain..

Linus

2003-09-23 21:57:46

by Andrew Morton

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Andrea Arcangeli <[email protected]> wrote:
>
> I don't think we can merge this in 2.4 but it looks good for 2.6.

I think so: doing it via a __setup parameter is much smarter than requiring
a rebuild.

> +static int __init log_buf_len_setup(char *str)
> +{
> + unsigned long size = simple_strtoul(str, &str, 0);
> +
> + if (size & (size-1))
> + printk("log_buf_len: invalid size - needs a power of two\n");

You need to either panic or return here.

2003-09-23 22:15:29

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 02:53:32PM -0700, Linus Torvalds wrote:
> When you write some code of your own, you get to choose the license. And I
> haven't seen you make CVS usable - I've only seen you bitch, moan, and
> complain..

we have cvsps and subversion these days, it's not about cvs only anymore.

if I would know that you will eventually get interested into anything
not bitkeeper I would be glad to hack in this area (I can use some spare
time too) to provide a service to the community. The main reason I will
never use b*tkeeper is because I want to retain the freedom to do that,
something that you and many others won't be able to do anymore.

But note that cvs+cvsps is already perfectly usable for me, most people
is readonly anyways, and if you would checkin into cvs instead of
b*tkeeper I couldn't notice any difference (except that I could send you
a patch to add the email at the top of the cvs log). So from my part,
using cvsps is just good enough and I've no idea why you call it
"unusable" (just as example see the two patches I extracted in a second,
in the old days I certainly couldn't do that).

There's a chance cvsps is even more friendly than b*tkeeper for my
common usages.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-23 22:22:28

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 02:37:54PM -0700, Andrew Morton wrote:
> Andrea Arcangeli <[email protected]> wrote:
> >
> > I don't think we can merge this in 2.4 but it looks good for 2.6.
>
> I think so: doing it via a __setup parameter is much smarter than requiring
> a rebuild.

agreed. That's why I also preferred my simple hack. After I fix the 64k
memory loss, I will see no disavantages in the kernel boot time
parameter compared to the compile time option.

>
> > +static int __init log_buf_len_setup(char *str)
> > +{
> > + unsigned long size = simple_strtoul(str, &str, 0);
> > +
> > + if (size & (size-1))
> > + printk("log_buf_len: invalid size - needs a power of two\n");
>
> You need to either panic or return here.

woops sorry, that was the old version ;) it had other bugs too, the new
one is this one (Marcelo please don't merge the old version, I thought
it was just uptodate in 22aa1 but it was not, the new version wasn't in
the ftp site yet infact)

Also I have to fix the request for the 64k memory loss reported by Willy
first.

thanks for the review!

diff -urNp --exclude CVS --exclude BitKeeper ul-ref/kernel/printk.c ul/kernel/printk.c
--- ul-ref/kernel/printk.c 2003-09-06 01:45:25.000000000 +0200
+++ ul/kernel/printk.c 2003-09-06 01:46:09.000000000 +0200
@@ -27,15 +27,16 @@
#include <linux/interrupt.h> /* For in_interrupt() */
#include <linux/config.h>
#include <linux/delay.h>
+#include <linux/bootmem.h>

#include <asm/uaccess.h>

#if defined(CONFIG_MULTIQUAD) || defined(CONFIG_IA64)
-#define LOG_BUF_LEN (65536)
+#define __LOG_BUF_LEN (65536)
#elif defined(CONFIG_ARCH_S390) || defined(CONFIG_PPC64) || defined(CONFIG_PPC)
-#define LOG_BUF_LEN (131072)
+#define __LOG_BUF_LEN (131072)
#else
-#define LOG_BUF_LEN (32768) /* This must be a power of two */
+#define __LOG_BUF_LEN (32768) /* This must be a power of two */
#endif

#define LOG_BUF_MASK (LOG_BUF_LEN-1)
@@ -78,7 +79,9 @@ struct console *console_drivers;
*/
static spinlock_t logbuf_lock = SPIN_LOCK_UNLOCKED;

-static char log_buf[LOG_BUF_LEN];
+static char __log_buf[__LOG_BUF_LEN];
+static char * log_buf = __log_buf;
+static int LOG_BUF_LEN = __LOG_BUF_LEN;
#define LOG_BUF(idx) (log_buf[(idx) & LOG_BUF_MASK])

/*
@@ -165,6 +168,45 @@ static int __init console_setup(char *st

__setup("console=", console_setup);

+static int __init log_buf_len_setup(char *str)
+{
+ unsigned long size = memparse(str, &str);
+
+ if (size > LOG_BUF_LEN) {
+ unsigned long start, dest_idx, offset;
+ char * new_log_buf;
+
+ new_log_buf = alloc_bootmem(size);
+ if (!new_log_buf) {
+ printk("log_buf_len: allocation failed\n");
+ goto out;
+ }
+
+ spin_lock_irq(&logbuf_lock);
+ LOG_BUF_LEN = size;
+ log_buf = new_log_buf;
+
+ offset = start = min(con_start, log_start);
+ dest_idx = 0;
+ while (start != log_end) {
+ log_buf[dest_idx] = __log_buf[start & (__LOG_BUF_LEN - 1)];
+ start++;
+ dest_idx++;
+ }
+ log_start -= offset;
+ con_start -= offset;
+ log_end -= offset;
+ spin_unlock_irq(&logbuf_lock);
+
+ printk("log_buf_len: %d\n", LOG_BUF_LEN);
+ }
+ out:
+
+ return 1;
+}
+
+__setup("log_buf_len=", log_buf_len_setup);
+
/*
* Commands to do_syslog:
*

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-23 22:34:55

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 09:02:19PM +0200, Willy Tarreau wrote:
> > My API is good for everyone, yours is not
>
> I'm impressed that you know so many people. I know that mine at least
> satisfies a few collegues, customers, and I. So I deduced that it might be
> useful to others too. Even Marcelo thought the same a time ago.

I know your patch was just very good for many people, but not for
everyone. I really didn't want to say your patch didn't make any good,
I acknowledge it was just very good.

Here I need an argument from you that explains me why should I work
further on the current code to make the config option still usable. I
see no reason to spend time in that effort because the config option
will provide no benefit compared to the kernel parameter.

If you can explain me _why_ you don't want to pass the kernel parameter
than you will convince me, if it's just that "you don't want it" then I
can't buy that. Give me a valid argument and I will have a reason to
retain the config option.

I definitely agree my patch (btw, I posted the last one that had a few
bugs too) needs fixing to release the 64k of ram, and to allow a smaller
bufsize too (the latter will happen automatically while addressing the
former)

The fact you don't want to touch the lilo.conf doesn't sound to me.
Especially with lilo (not grub) you've to run lilo anyways every time
you replace the kernel, so a simple script adding the parameter in every
lilo.conf sounds very easy to provide (you can add it in all kernels,
the old ones will ignore it).

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-23 22:54:29

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, 24 Sep 2003, Andrea Arcangeli wrote:
>
> we have cvsps and subversion these days, it's not about cvs only anymore.

Neither of those are anywhere close to bk.

In particular, they don't support any kind of distributed development.
They aren't even trying to, I'm sad to say. And to me, distributed
development is the only thing that matters.

And I realise it isn't to you. You don't much care about merging, you only
have your tree you need to worry about.

And you know what? You shouldn't have to care. I'm not berating you for
using CVS/SVN/whatever. I'm berating you for complaining when _others_
have come to the conclusion that CVS/SVN/whatever really doesn't cut it
for them.

Use CVS and be happy. But don't complain to others that have needs that
CVS simply can't fill.

[ Bad analogy time ]

You're acting like a husband that has a wife that refuses to use make-up,
and thinks that everybody else should have ugly wives too, and calls them
whores for being prettier.

Actually, in the CVS analogy, I don't think it's that your wife refuses to
use make-up, but that make-up doesn't actually help.

[ Ok, let's see just _how_ badly I get flamed for that analogy. I
admit, it really sucks, and is tasteless to boot. My bad. ]

> But note that cvs+cvsps is already perfectly usable for me, most people
> is readonly anyways

Indeed. That's pretty much all non-distributed stuff is useful for, from
where I'm stading. Small projects with a few developers and a lot of
read-only stuff. And even there the developers will bitch about the
limitations.

Sure, SVN makes branches cheaper, but you still have to work with them
like under CVS, ie merging is a total disaster. And you still can't make
it your private repository.

But that wasn't really my point. My point is that you're only starting
flame wars, with no actual point to your emails. Please don't.

"Those that can, do. Those that can't, complain".

You're complaining, right now. Everybody who has ever used BK admits to
its techical superiority. That's just a fact.

I don't care about source control software, so I'm not likely to start
coding one any time soon (like "ever") - but if I did, I'd be totally
_ashamed_ to push lower-quality stuff on users. I'd make excuses for it,
and I'd 'fess up when they didn't work. And I'd try my best to make it
better. Even if it took me a decade.

In contrast, what you're doing is saying "ignore the good stuff: use this
crap, because I'm buddies with the people developing it. We aren't even
trying to compete on technical terms, but we'll push our version on you
because we've got religion, and this doesn't contain any cow-meat". That's
bad - especially if others DON'T share the religion.

I'm ok with other people using NT. When it's better for them, that's their
choice. I work hard to make sure that the Linux kernel is technically
superior, and if I feel it isn't I want to fix it. Because I do _not_ want
people using Linux for religious reasons. I want people to use Linux
because it is _better_ for them, of because they truly believe that they
can make it so (or at least have fun trying).

Take pride in what you do. But don't make that pride blind you to what is
good technology, and what isn't. Don't get religion. It's a science.

Oh well. I told you not to start a flamewar, and I told Larry to not raise
to the bait. Now I'll just take my own advice and stop responding. You too
please stop baiting,

Linus

2003-09-23 23:31:21

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, Sep 24, 2003 at 12:34:57AM +0200, Andrea Arcangeli wrote:

> I know your patch was just very good for many people, but not for
> everyone. I really didn't want to say your patch didn't make any good,
> I acknowledge it was just very good.

It's not about it to be good or not, but be useful. There's a difference.
When two things are good and do nearly the same, you get the best of them and
that's all. When two things are useful and nearly identical, there's a huge
potential that one is useful to some people, and the other one to other people.
I'm certain I would use your patch on my desktop PC (if it doesn't break
kmsgdump as it seems, though), because it will allow me to easily extend my
log buffer when I get repetitive oopses. But most probably not on remote
systems.

> I definitely agree my patch (btw, I posted the last one that had a few
> bugs too) needs fixing to release the 64k of ram, and to allow a smaller
> bufsize too (the latter will happen automatically while addressing the
> former)

That doesn't matter, I'm fairly certain you will fix these issues, they are
details since the patch is at its early stage.

> The fact you don't want to touch the lilo.conf doesn't sound to me.
> Especially with lilo (not grub) you've to run lilo anyways every time
> you replace the kernel, so a simple script adding the parameter in every
> lilo.conf sounds very easy to provide (you can add it in all kernels,
> the old ones will ignore it).

Yes it's the real reason, and I think that others will second me on this one.
You're seeing the benefit of *changing* the value on a kernel compile once.
I see the benefit of *deploying* a same kernel to a lot of remote hosts while
touching the fewest possible files. Sending a pre-configured kernel and running
lilo is already a lot.

I've had private discussions off-list with a few other people who don't agree
on the *obligation* to enter a new option on the command line because they will
not be able to dynamically generate their boot files anymore and will have to
add a config option to them just for this. Believe me, when you remotely and
blindly upgrade systems with commands such as :

for host in to_update/*; do update $host && mv $host up_to_date/ ; done

*not* having to touch a boot file is really recomforting. It's not funny
to remotely run 'ed' scripts on lilo.conf when you know that there's
nobody to replace the box if you fail because one of the file had once been
hand-edited and is slightly different. May be you'll say "well, simply prepare
all your lilo.conf, check them all then send them". If you think this, it's
because you obviously never felt your heart accelerate and beat very loud
during these operations and cannot understand why some people prefer to keep
these delicate upgrades as trivial as possible. When I said to you that I use
root=/dev/root and boot=/dev/mbr on all these hosts, it's exactly not to have
to touch these sensible files.

One of us told me that he will have to rebuild his BASE package just to
integrate this change, and deploy it to 1200 hosts around the world. When I see
how I worry for a few tens, I think he'd better revert your patch and stay on
the previous one. Think that some of these people are still running 2.2 because
they don't trust 2.4 enough to risk a remote upgrade.

Andrea, I won't blame you for not understanding that some people are afraid of
doing certain operations remotely. Perhaps you'll think that I'm totally
stupid. That's not a problem for me. I just wanted to let you know that this
sort of changes will be a regression for a few of us (mainly the few who've
been using the previous patch).

As I told you the first time, if I had the choice for a dynamic buffer
resizement, I'd be all for a sysctl so that I wouldn't need to touch any boot
file, and that would allow to resize a live system without needing a reboot.
Then, simply pre-initialize a 64kB buffer so that no boot message gets lost,
and the init scripts adjust the size to what is needed.

I repeat my proposal. I agree to spend a few of my time to work with you in
this direction, because I know it is a useful feature. But please don't make
it more difficult for people who already have a hard time.

Thanks,
Willy

2003-09-23 23:48:46

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, Sep 24, 2003 at 01:29:59AM +0200, Willy Tarreau wrote:
> because you obviously never felt your heart accelerate and beat very loud
> during these operations and cannot understand why some people prefer to keep

well, I kind of remeber that ;)

So ok, that sounds a good enough argument to keep the old patch too,
despite you may be very few people using this.

But let's do it only in 2.4, 2.6 should have it only dynamic.

clearly if you can try to go 2.6 (possibly using a wrong .config) then
you can risk editing the lilo/grub file as well. And with 2.6, with the
printk rewrite that shrinks and grows the buffer, you should have no
need of the parameter any more.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-23 23:52:17

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, Sep 24, 2003 at 01:48:49AM +0200, Andrea Arcangeli wrote:
> On Wed, Sep 24, 2003 at 01:29:59AM +0200, Willy Tarreau wrote:
> > because you obviously never felt your heart accelerate and beat very loud
> > during these operations and cannot understand why some people prefer to keep
>
> well, I kind of remeber that ;)
>
> So ok, that sounds a good enough argument to keep the old patch too,
> despite you may be very few people using this.

Thanks for your understanding ;-)

> But let's do it only in 2.4, 2.6 should have it only dynamic.
>
> clearly if you can try to go 2.6 (possibly using a wrong .config) then
> you can risk editing the lilo/grub file as well. And with 2.6, with the
> printk rewrite that shrinks and grows the buffer, you should have no
> need of the parameter any more.

I have not tried it remotely yet, but I may adapt to a new behaviour or write
a patch :-)

Cheers,
Willy

2003-09-24 00:13:43

by Andrew Morton

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Andrea Arcangeli <[email protected]> wrote:
>
> +__setup("log_buf_len=", log_buf_len_setup);

Specifying this in units of bytes is a pain.

How about we make it "log_buf_kbytes", and multiply it by 1024?

2003-09-24 00:36:47

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 03:54:22PM -0700, Linus Torvalds wrote:
> [ Bad analogy time ]
>
> You're acting like a husband that has a wife that refuses to use make-up,
> and thinks that everybody else should have ugly wives too, and calls them
> whores for being prettier.
>
> Actually, in the CVS analogy, I don't think it's that your wife refuses to
> use make-up, but that make-up doesn't actually help.
>
> [ Ok, let's see just _how_ badly I get flamed for that analogy. I
> admit, it really sucks, and is tasteless to boot. My bad. ]

amusing ;)

well, following your bad analogy time, with software we can do the
equivalent of changing the dna, that can fix the problem unlike than
markup. So even starting from a very bad dna may not be a disaster as
far as there's a critical mass behind it, able to eventually rewrite it
from scratch. sure if we could choose the dna of a beautiful woman, it
would be easier, but we don't have much choice (cvs/svn). either that
or we start from scratch.

> because we've got religion, and this doesn't contain any cow-meat".

this is not religion this is primarly a law matter from my point of view.

AFIK it's illegal for me to use bitkeeper, because I'm not giving up the
freedom to fix bugs in cvs or hack subversion, like the one that I'm
going to fix to avoid losing the domain, period.

It's a pleasure for me to be able to very seldom fix an annoying bug
once in a while in some random app, that nobody else cared about or
noticed. It doesn't happen frequently, but it's one of the things I also
like of using open source.

And secondly I prefer "open" to "immediatly better" or even to
"immediatly so much better". Again, not because of religion (at least I
don't feel it that way), but because of a better long term business
investment. What you call religion I call long term investment. yeah,
it maybe too long term, but I won't take the risk of trading it with the
freedom of innovating in this area (even if I may never will).

You're right I should provide new code, and avoid comments on the a bit
inferior info in bkcvs (that Larry nicely offered to even improve after
cvs gets properly fixed), but I had no real interest in this area todate
and my job keeps most of my time full already and that's higher
priority. So I'm trying to at least raise the issue to see if somebody
else shares my view, and maybe in more people it'll be easier to write
code. I'm not complaining anything to Larry, quite the opposite, he was
very annoyed by my signature, but I respect Larry and the signature has
really nothing directly against him or bitkeeper. I'm advertizing the
open links, that were not posted into any webpage or anywhere else, and
I tought they deserved to be better known.

As for the merging with mainline, these patches were merged from 2.4.22
to 2.4.23pre5 (not all of them have been developed from me):

Only in 2.4.22aa1: 00_config-smp-1
Only in 2.4.22aa1: 00_copy-namespace-1
Only in 2.4.22aa1: 00_panic-console-switch-1
Only in 2.4.22aa1: 00_pgt-cache-leak-2
Only in 2.4.22aa1: 00_read_full_page-get_block-err-2
Only in 2.4.22aa1: 00_sk98lin_2.4.22-20030902-1.gz
Only in 2.4.22aa1: 05_vm_03_vm_tunables-4
Only in 2.4.22aa1: 05_vm_05_zone_accounting-2
Only in 2.4.22aa1: 05_vm_06_swap_out-3
Only in 2.4.22aa1: 05_vm_07_local_pages-4
Only in 2.4.22aa1: 05_vm_09_misc_junk-3
Only in 2.4.22aa1: 05_vm_16_active_free_zone_bhs-1
Only in 2.4.22aa1: 05_vm_17_rest-10
Only in 2.4.22aa1: 70_xfs-sysctl-3
Only in 2.4.22aa1: 9999900_request-firmware-1

I know this time more patches have been merged than in the previous
kernels (in this case some of the vm enhacements), but almost the same
number gets regularly merged into mainline in the other releases too.

You're wrong saying that all I care is my tree, I'm deeply careful in
keeping those patches in a way that can be merged with the minor
possible pain from the kernel maintainer, infact I hope Marcelo also was
confortable with that. My object is to get everything merged into
mainline. Personally I wish my tree would be empty.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-24 00:38:40

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 05:15:05PM -0700, Andrew Morton wrote:
> Andrea Arcangeli <[email protected]> wrote:
> >
> > +__setup("log_buf_len=", log_buf_len_setup);
>
> Specifying this in units of bytes is a pain.
>
> How about we make it "log_buf_kbytes", and multiply it by 1024?

note: that's a memparse, that's not unit of bytes, you can wite 256k or
1M or 2m just now and it'll just work

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-24 01:20:06

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, Sep 24, 2003 at 02:36:52AM +0200, Andrea Arcangeli wrote:
> You're right I should provide new code, and avoid comments on the a bit
> inferior info in bkcvs (that Larry nicely offered to even improve after
> cvs gets properly fixed), but I had no real interest in this area todate
> and my job keeps most of my time full already and that's higher
> priority.

The problem I have is as follows:

a) I understand your point of view and from the very first version of BK
we released we addressed it. 100% of the data and the metadata is
available from the command line with BK. Always has been and always
will be, if that's not true that is is a bug and we'll fix it. People
use BK because they like it, not because we locked them in.

b) I understand your need to not be dependent on BitMover or BitKeeper.
That's why we built the CVS gateway, so you wouldn't need to depend
on us, the data you care about is available in a form that doesn't
require any license agreements.

What the above two points demonstrate, dramatically so, is that we
understand your concerns and agree with them. We have spent a lot of
time and money to make sure that you are happy. Not whining, not
flaming, we were writing code to make you happy. We were writing that
code long before you ever heard of BitKeeper and we have the revision
history to prove it.

What we expected in return was the same understanding.

What we got was you complaining about us, in public, over and over.

Andrea, you need to grow up and learn that biting the hand that is held
out to you and is helping you, that's not smart. What you have done
is to make all the people who tried so hard to help you dislike you.
That is your problem. You need to grow up. The world is based on
relationships, people helping each other. That means you help those
who help you, or at least don't piss all over those who help you.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

2003-09-24 02:04:08

by andrea

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 06:19:51PM -0700, Larry McVoy wrote:
> On Wed, Sep 24, 2003 at 02:36:52AM +0200, Andrea Arcangeli wrote:
> > You're right I should provide new code, and avoid comments on the a bit
> > inferior info in bkcvs (that Larry nicely offered to even improve after
> > cvs gets properly fixed), but I had no real interest in this area todate
> > and my job keeps most of my time full already and that's higher
> > priority.
>
> The problem I have is as follows:
>
> a) I understand your point of view and from the very first version of BK
> we released we addressed it. 100% of the data and the metadata is
> available from the command line with BK. Always has been and always
> will be, if that's not true that is is a bug and we'll fix it. People
> use BK because they like it, not because we locked them in.
>
> b) I understand your need to not be dependent on BitMover or BitKeeper.
> That's why we built the CVS gateway, so you wouldn't need to depend
> on us, the data you care about is available in a form that doesn't
> require any license agreements.
>
> What the above two points demonstrate, dramatically so, is that we
> understand your concerns and agree with them. We have spent a lot of
> time and money to make sure that you are happy. Not whining, not
> flaming, we were writing code to make you happy. We were writing that
> code long before you ever heard of BitKeeper and we have the revision
> history to prove it.

I defininitely agree with that and I appreciate that you acknowledge my
requests when the bkcvs didn't exist.

(I was also using cvs for kernel development way before I ever heard of
bitkeeper too, then I had to giveup because it was too slow to handle
branches)

> What we expected in return was the same understanding.

that is not accurate, you also asked us to giveup the freedom of
development in your area. And I wouldn't be too interested in a closed
software anyways but at least I could consider using it in the meantime
without feeling tainted.

NOTE: I'm not asking you to remove that clause, nor I'm complaining
about it, I would feel bad for you if you had problems because you
removed that clause. I perfectly understand why you put that clause in
and it makes sense to me.

> Andrea, you need to grow up and learn that biting the hand that is held

It's because I grow up that I can actually better understand the deals
it's in my own (again speaking only for myself and not for anybody else)
interest to avoid.

(changing email address as well to make it clear I'm speaking only for
myself here)

Last but not the least, if I was required to use bitkeeper as part of my
job, then I would use it and I'd giveup that bit of freedom, but as far
as I'm free to choose, I will avoid it. But that's my own choice, it has
nothing to do with anybody else.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-24 02:37:36

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, 24 Sep 2003 [email protected] wrote:
>
> It's because I grow up that I can actually better understand the deals
> it's in my own (again speaking only for myself and not for anybody else)
> interest to avoid.

You've claimed this now twice.

However, that only explains why you don't use BitKeeper. And everybody
accepts that. When I started to use BK, I made it _very_ clear that
service for non-BK users will be _at_least_ as good as it ever was before
I started using BK.

And I think everybody agrees that is true. ChangeLogs, CVS exports, daily
snapshots. And that's just the advantages to _others_.

But your lack of interest in BK does _not_ explain why you whine about it,
and try to goad Larry, and just generally are nasty about it.

You remind me of how some of the BSD people complaining about me using the
GPL. They whined and whined about how the GPL is not as free as the BSD
license.

In other words:

- you don't have to agree with another persons choice of license, and
you don't have to to use the software using it. That is _your_ choice.

- But you also don't have the moral right to whine about another persons
choice of license (or choice of using software under that license).
That was _their_ choice.

See? You're not just being impolite; your complaints are actually morally
offensive. The same way I found it morally offensive when people
complained about my choice of GPL. They didn't have the right. And _you_
don't have the right.

It's the old

"I disapprove of what you say, but I will defend to the death your
right to say it"

approach: even if you disapprove of Larry's license, you should defend his
_right_ to that license. Instead of whining about it.

Linus

2003-09-24 02:39:21

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 07:29:48PM -0700, Larry McVoy wrote:
> On Wed, Sep 24, 2003 at 04:04:09AM +0200, [email protected] wrote:
> > > What we expected in return was the same understanding.
> >
> > that is not accurate, you also asked us to giveup the freedom of
> > development in your area.
>
> If you were actually doing some significant development then maybe I'd
> respect you. But you aren't. So I don't. You don't have the slightest
> understanding in this area, you've proved that beyond all reasonable
> doubt. So you are just complaining about something you don't understand.
>
> I truly hope you follow in the footsteps of others who got pissed at the
> BK licensing and try to implement a replacement. BK makes VM systems
> seem like child's play. Centralized systems like CVS are child's play.

If you weren't using the linux VM I'd respect you, but you are, and
still I respected you so far, probably I was wrong.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-24 02:31:12

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, Sep 24, 2003 at 04:04:09AM +0200, [email protected] wrote:
> > What we expected in return was the same understanding.
>
> that is not accurate, you also asked us to giveup the freedom of
> development in your area.

If you were actually doing some significant development then maybe I'd
respect you. But you aren't. So I don't. You don't have the slightest
understanding in this area, you've proved that beyond all reasonable
doubt. So you are just complaining about something you don't understand.

I truly hope you follow in the footsteps of others who got pissed at the
BK licensing and try to implement a replacement. BK makes VM systems
seem like child's play. Centralized systems like CVS are child's play.
Distributed systems like BK actually have to address all the problems
that CVS ignores. Those problems are really hard. Not because they are
so hard (even though they are), but because there are so many of them.

It may be an eye opener to you if you realized that most of the problems
that the researchers are discussing about distributed file systems, those
are problems that we have solved. Computer science research is behind us.
No kidding. Go get your best and bring 'em on, we'll take the challenge.
There are at least a dozen excellent PhD theses in BK. And those are the
ones I can think of at the end of a very long tiring day.

If you understood that you'd understand why I am so unhappy with you.
We've bent over backwards to give you what you need and you still don't
understand what we have given you. You think it is something that you
can fix in a few days of hacking.

I'm reminded of something Peter Gutmann said recently:

Whenever someone thinks that they can replace SSL/SSH with something
much better that they designed this morning over coffee, their
computer speakers should generate some sort of penis-shaped sound
wave and plunge it repeatedly into their skulls until they achieve
enlightenment.

Change "SSL/SSH" to "useful source mangement" and it makes the point.
I hope your skull hurts. It should. Enlightenment is then forthcoming.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

2003-09-24 02:48:28

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 07:36:39PM -0700, Linus Torvalds wrote:
>
> On Wed, 24 Sep 2003 [email protected] wrote:
> >
> > It's because I grow up that I can actually better understand the deals
> > it's in my own (again speaking only for myself and not for anybody else)
> > interest to avoid.
>
> You've claimed this now twice.
>
> However, that only explains why you don't use BitKeeper. And everybody
> accepts that. When I started to use BK, I made it _very_ clear that
> service for non-BK users will be _at_least_ as good as it ever was before
> I started using BK.
>
> And I think everybody agrees that is true. ChangeLogs, CVS exports, daily
> snapshots. And that's just the advantages to _others_.
>
> But your lack of interest in BK does _not_ explain why you whine about it,
> and try to goad Larry, and just generally are nasty about it.

I wasn't whining about BK at all, nor about the licence, I just
advertized the open links in my signature, the word "refuse" may be not
a nice one, I didn't feel to be rude when I wrote it, now I changed it
and I hope it's a more friendly wording, though the meaning it's the
same.

> approach: even if you disapprove of Larry's license, you should defend his
> _right_ to that license. Instead of whining about it.

I defend his right to the licence, it makes perfect business sense, if
he wanted to be safer he had to take a couple of patents too I guess
(but I'm not a lawyer and I certainly prefer the licence to the
patents). I don't think I ever complained to the licence. I only said
I'm not applying to the licence and _you_ have to respect my _right_ not
to use the software and to advertize the _open_ links, without seeing it
as an offensive thing.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-24 03:07:56

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, 24 Sep 2003, Andrea Arcangeli wrote:
>
> I wasn't whining about BK at all, nor about the licence, I just
> advertized the open links in my signature, the word "refuse" may be not
> a nice one, I didn't feel to be rude when I wrote it, now I changed it
> and I hope it's a more friendly wording, though the meaning it's the
> same.

Andrea, be honest.

This latest thing didn't start with just the signature. In fact, the
signature flame-war I berated Larry for - ask Larry. I told him he
shouldn't let himself be goaded.

This one started with you very much making the whole point of your whole
post be a complaint about the license:

"if Marcelo would be using open source code to exports the patchsets in
his tree, we could fix it to add the email address along the name in the
checkin logs metadata, to avoid this sort of mistakes."

That quote is not just some small snippet: it is the _entirety_ of new
material in the whole post.

You are whining. You did nothing _but_ whine in the post. About other
peoples choice of license usage.

> I only said I'm not applying to the licence and _you_ have to respect
> my _right_ not to use the software and to advertize the _open_ links,
> without seeing it as an offensive thing.

(a) It wasn't all you said
(b) I can, and do, get offended by postings to public non-religious lists
that have no redeeming value except to push your religious views.

I despise people who come up to me in the street (or even worse - ring the
doorbell on my house) to tell me about Jesus, and how they found happiness
in him.

Good for you and your buddy Jesus/buddha/Moses/AlienFromAnotherPlanet. But
get the _hell_ out of my face. If I come to your church to listen to your
sermons, that's when I'll respect your right to tell me. But if you stand
up with a megaphone in the park when I'm having a picknick, I'd much
rather kick your sorry ass the h*ll out of there.

So go to gnu.misc.discuss if you want to discuss the immorality of
licenses. There you'll find people who care. linux-kernel is about
technical discussions about the kernel, and yes IT IS OFFENSIVE to push
religion here.

It's also offensive to have comments about abortion or guns in your
signature. For exactly the same reason (and btw, I complained to esr when
he had his gun-control signature, so I'm not just making these up).

Do you see?

Now can we get back to complaining about the scheduler behaviour and xmms?

Please?

Linus

2003-09-24 03:17:39

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, Sep 24, 2003 at 04:39:28AM +0200, Andrea Arcangeli wrote:
> On Tue, Sep 23, 2003 at 07:29:48PM -0700, Larry McVoy wrote:
> > On Wed, Sep 24, 2003 at 04:04:09AM +0200, [email protected] wrote:
> > > > What we expected in return was the same understanding.
> > >
> > > that is not accurate, you also asked us to giveup the freedom of
> > > development in your area.
> >
> > If you were actually doing some significant development then maybe I'd
> > respect you. But you aren't. So I don't. You don't have the slightest
> > understanding in this area, you've proved that beyond all reasonable
> > doubt. So you are just complaining about something you don't understand.
> >
> > I truly hope you follow in the footsteps of others who got pissed at the
> > BK licensing and try to implement a replacement. BK makes VM systems
> > seem like child's play. Centralized systems like CVS are child's play.
>
> If you weren't using the linux VM I'd respect you, but you are, and
> still I respected you so far, probably I was wrong.

Yeah. I'm using the Linux VM. And it _still_ isn't up to what I managed
to accomplish in SunOS. Wake up, dude. You aren't the first one to
work in this area, you are just one of the first to work in this area
without learning from the people who came before you. Don't believe me?
Go to OLS and read the papers. Then read the OS research that has
happened in the last 20 years or so. So far you are still catching up.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

2003-09-24 03:12:31

by David Miller

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, 24 Sep 2003 04:48:31 +0200
Andrea Arcangeli <[email protected]> wrote:

> I wasn't whining about BK at all, nor about the licence,

Let me remind you, what you said which prompted Linus to reply in the
first place:

"if Marcelo would be using open source code to exports the
patchsets in"

This is flame bait. The only thing a sentence like that will do is
spark a flame and incite emotional responses from others, not
constructive conversation.

That is what you need to stop doing.

2003-09-24 03:38:43

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, 24 Sep 2003, Andrea Arcangeli wrote:
>
> Really I don't find anything immoral, nor religious here. I only care
> about law. Law says if I use bitkeeper I can't develop in the same area,
> and so I refuse to use it. it's as simple as that.

Go away. I will tell you one more time: it is _fine_ that you don't use
BK. Nobody has ever asked you to, in fact.

So don't use it. Please. You don't like the license, and what it means.
Agreed. That license is a legal document. Agreed. BUT THAT ISN'T THE
ISSUE.

And that was NEVER the issue, even though you keep bringing it up. Again
and again.

The issue is that you should not complain about other peoples choices.
They are not _your_ choices. Never were, and never will be.

Linus

2003-09-24 03:33:56

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, 23 Sep 2003, Larry McVoy wrote:

> Yeah. I'm using the Linux VM. And it _still_ isn't up to what I managed
> to accomplish in SunOS. Wake up, dude. You aren't the first one to
> work in this area, you are just one of the first to work in this area
> without learning from the people who came before you. Don't believe me?
> Go to OLS and read the papers. Then read the OS research that has
> happened in the last 20 years or so. So far you are still catching up.

Bad example ;)

While it is true that the Linux VM doesn't do a whole
lot of the things a VM should do, there is also the
fact that the VM problem space has gotten a lot harder
over the last decade due to the fact that the size of
disk and memory has grown way faster than the speed of
disk and memory.

Simply put, nowadays it takes more than 10 times as
long to swap out a process then it took 10 years ago.
This puts interesting constraints on the design of a
VM...

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

2003-09-24 03:28:32

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 08:06:51PM -0700, Linus Torvalds wrote:
>
> On Wed, 24 Sep 2003, Andrea Arcangeli wrote:
> >
> > I wasn't whining about BK at all, nor about the licence, I just
> > advertized the open links in my signature, the word "refuse" may be not
> > a nice one, I didn't feel to be rude when I wrote it, now I changed it
> > and I hope it's a more friendly wording, though the meaning it's the
> > same.
>
> Andrea, be honest.
>
> This latest thing didn't start with just the signature. In fact, the

the latest not, but that was the latest, and I still recommend Marcelo
to stop using bitkeeper and to do the checkins directly in cvs. That
would be a start and it would motivate me more too into adding more
effort there.

I need a bit of feedback from a kernel maintainer that there will be a
slight chance to use something non bitkeeper in the future before I
invest a bit of (probably my spare) time into this.

> "if Marcelo would be using open source code to exports the patchsets in
> his tree, we could fix it to add the email address along the name in the
> checkin logs metadata, to avoid this sort of mistakes."
>
> That quote is not just some small snippet: it is the _entirety_ of new
> material in the whole post.

yes that was the whole point of the email. That was a suggestion for
Marcelo.

> > I only said I'm not applying to the licence and _you_ have to respect
> > my _right_ not to use the software and to advertize the _open_ links,
> > without seeing it as an offensive thing.
>
> (a) It wasn't all you said
> (b) I can, and do, get offended by postings to public non-religious lists
> that have no redeeming value except to push your religious views.
>
> I despise people who come up to me in the street (or even worse - ring the
> doorbell on my house) to tell me about Jesus, and how they found happiness
> in him.
>
> Good for you and your buddy Jesus/buddha/Moses/AlienFromAnotherPlanet. But
> get the _hell_ out of my face. If I come to your church to listen to your
> sermons, that's when I'll respect your right to tell me. But if you stand
> up with a megaphone in the park when I'm having a picknick, I'd much
> rather kick your sorry ass the h*ll out of there.
>
> So go to gnu.misc.discuss if you want to discuss the immorality of

Really I don't find anything immoral, nor religious here. I only care
about law. Law says if I use bitkeeper I can't develop in the same area,
and so I refuse to use it. it's as simple as that.

To me it looks like you're confusing law and free market, with religion.

To make an example, I used netscape for years, netscape was the only
reasonable software at the time, and I would have never expected that
they would release the sources anytime soon, nor I was extremely
interested into it, since netscape just worked fine (on alpha I used the
tru64 version for a long time too). I preferred an open alternative but
there wasn't so I happily used netscape. I still used netscape for a
long time even when mozilla was usable because some website didn't work
with mozilla and I had no time to fix it myself. The netscape licence
was acceptable to me to use for something not critical for me like
browsing the web (I mean who cares if netscape crashes). I can make lots
more examples like that. I don't feel religious about stuff, and you
certainly can't claim I'm religious because I refuse to giveup my rights
to fix or improve cvs or subversion or to write a filesystem with
versioning, that may effectively collide with future jobs as well.
that's about freedom, not religion.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-24 03:43:02

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, 24 Sep 2003, Andrea Arcangeli wrote:

> the latest not, but that was the latest, and I still recommend Marcelo
> to stop using bitkeeper and to do the checkins directly in cvs. That
> would be a start

That would break so many 3rd party trees. It would mean
Marcelo would no longer be able to just pull in eg. the
PPC tree, it would mean merging pain for branched trees,
etc...

> and it would motivate me more too into adding more effort there.

Effort where ?

If you're talking about adding more effort into merging
things into 2.4, then DUH. You would NEED more effort to
get stuff merged, if marcelo used something as bad as CVS.

If you're talking about improving CVS, don't bother. The
model is broken.

> I need a bit of feedback from a kernel maintainer that there will be a
> slight chance to use something non bitkeeper in the future before I
> invest a bit of (probably my spare) time into this.

There's more than a slight chance. Just come up with
something that's as good as, or better than, bitkeeeper
and people will switch in a heartbeat.

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

2003-09-24 03:46:05

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 08:16:02PM -0700, Larry McVoy wrote:
> Yeah. I'm using the Linux VM. And it _still_ isn't up to what I managed
> to accomplish in SunOS. Wake up, dude. You aren't the first one to

if this would be really true, then why do you use linux then? I don't
understand. I've to exclude the reason is that it's free and open.

I heard arguments like yours for years and years from tons of different
people, 7 years ago I heard the very same again and again about linux
at large compared to other operative systems (way before I was using
linux myself).

Personally I don't believe you. I'm not saying the vm is too complex,
but it's heavily dominatd by details, it's simply not possible to
unique describe the vm in terms of algorithms like most parers do, and I
believe what we have in linux (mainline 2.4.23pre5 and 2.6 [modulo
rmap]) is very optimal.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-24 03:47:03

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 11:31:05PM -0400, Rik van Riel wrote:
> On Tue, 23 Sep 2003, Larry McVoy wrote:
>
> > Yeah. I'm using the Linux VM. And it _still_ isn't up to what I managed
> > to accomplish in SunOS. Wake up, dude. You aren't the first one to
> > work in this area, you are just one of the first to work in this area
> > without learning from the people who came before you. Don't believe me?
> > Go to OLS and read the papers. Then read the OS research that has
> > happened in the last 20 years or so. So far you are still catching up.
>
> Bad example ;)
>
> While it is true that the Linux VM doesn't do a whole
> lot of the things a VM should do, there is also the
> fact that the VM problem space has gotten a lot harder
> over the last decade due to the fact that the size of
> disk and memory has grown way faster than the speed of
> disk and memory.

Oh, I agree. The problem space changes and you have to move with it.
I was just pointing out that by not learning from others you put yourself
at a disadvantage. There are a lot of bright minds in the Linux space,
there is no question of that. But they would do so much better if they
understood the past. The past holds a lot of information, ignoring it
is a disadvantage.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

2003-09-24 04:03:11

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, Sep 24, 2003 at 05:46:12AM +0200, Andrea Arcangeli wrote:
> On Tue, Sep 23, 2003 at 08:16:02PM -0700, Larry McVoy wrote:
> > Yeah. I'm using the Linux VM. And it _still_ isn't up to what I managed
> > to accomplish in SunOS. Wake up, dude. You aren't the first one to
>
> if this would be really true, then why do you use linux then? I don't
> understand. I've to exclude the reason is that it's free and open.

Because I have huge respect for Linus' leadership. I've met a lot
of talented people in my life, people who were good engineers, good
architects, and good managers. Linus is _all_ of those and that, in my
opinion, is unique. BitKeeper exists because I believe that Linus is
unique and BK is my way of helping him be productive and continue to be
the leader of the Linux effort.

I don't know if you get it, you could go a lifetime and not meet
someone like Linus. Don't get me wrong, we've had screaming matches,
I remember one where I was yelling "I just want you to work for me so I
can fire your sorry ass", it's not like I worship the ground he walks
on or anything. But I do respect, deeply respect, what he has done.
I've demonstrated here that I don't have what it takes to do his job.
You have too. So have others. If we really care about the good of the
world then we help in the way that helps the most. That's what I'm doing.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

2003-09-24 03:56:23

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, Sep 23, 2003 at 08:38:35PM -0700, Linus Torvalds wrote:
> And that was NEVER the issue, even though you keep bringing it up. Again

that is _the_ issue from my part, the thing that isn't remotely going to
make me think to use bitkeeper. If there wasn't that clause in the
licence I would probably find acceptable to use it in the meantime.

But I guess I'd better go away as you ask since I feel like nobody
understands why I'm not using it despite I'm trying to explain it a few
times already.

> The issue is that you should not complain about other peoples choices.

I'm not complaining about that, I was making a suggestion to Marcelo
outlighting what was possible to achieve if it was open (yeah, it's a
minor thing). Larry even said what I suggested was not an optimal fix
and I agree with that so I will try to fix it right. I think it was a
productive suggestion after all.

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-24 03:55:51

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Larry, I think you remember the good old days of SunOS, when 16MB of RAM
was a lot, and people expected less of their hardware. In particular,
interactive programs used to have a _tiny_ footprint. Often even under X.

Then we put Solaris, Motif and CDE on those suckers, and it was horrible.

Yeah, SunOS was nice. But I really think it's the access patterns that
changed.

Linus

2003-09-24 04:08:04

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, 24 Sep 2003, Andrea Arcangeli wrote:
> On Tue, Sep 23, 2003 at 08:16:02PM -0700, Larry McVoy wrote:
> > Yeah. I'm using the Linux VM. And it _still_ isn't up to what I managed
> > to accomplish in SunOS. Wake up, dude. You aren't the first one to
>
> I heard arguments like yours for years and years from tons of different
> people, 7 years ago I heard the very same again and again about linux
> at large compared to other operative systems (way before I was using
> linux myself).

And you know what? Larry is right that the Linux VM still
lacks a number of things, like being robust under heavy
overload situations, etc...

Yes, the Linux VM is fast, but that isn't much good if you
can drive it into a death spiral with a single load spike.

> Personally I don't believe you. I'm not saying the vm is too complex,
> but it's heavily dominatd by details, it's simply not possible to
> unique describe the vm in terms of algorithms like most parers do, and I
> believe what we have in linux (mainline 2.4.23pre5 and 2.6 [modulo
> rmap]) is very optimal.

You'll want to catch up to the 1990's some day. Maybe even
to the current decade. For example, it's pretty much proven
that lru-like algorithms (like in 2.4-rmap) are suboptimal,
while the use-once stuff in 2.4 mainline and 2.6 is only
metastable, meaning that it doesn't recover from load spikes
except through some combination of clever tricks and sheer
luck, that works only under some loads.

Likewise, the way we write out dirty pages on page replacement
just can't scale to huge amounts of memory, because it'd take
minutes to write out all of the pages on the inactive list. The
non-blocking VM stuff akpm implemented for 2.6 should alleviate
that on medium sized systems, but on truly huge systems we'll
need a page laundering strategy that also improves CPU efficiency
when faced with dozens of gigabytes of inactive pages.

I'll happily concede that you are the absolute champion when it
comes to tuning the VM to get the best possible benchmark numbers,
but that is just not the same as trying to deal with all the corner
cases people run into in real life.

Larry really does have a point.

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

2003-09-24 04:14:42

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, 23 Sep 2003, Linus Torvalds wrote:

> Yeah, SunOS was nice. But I really think it's the access patterns that
> changed.

Absolutely. Garbage collection and object orientation have
obsoleted LRU and related algorithms.

Huge memory sizes also have done their share to obsolete
recency based replacement algorithms.

I suspect we'll want something closer to a frequency based
replacement scheme in the future. Possibly something like
LIRS or ARC, but adapted to work in a general purpose OS in
a very light weight way.

The argument that because we have more memory replacement
is no longer as important doesn't quite hold because the
cost of page replacement, measured in cpu cycles, has gone
completely through the roof in the last decades.

Then there's the whole different bag of cookies that is the
caching of filesystem metadata, like our inode and dentry
caches. I'm not sure what would be a good replacement
strategy for those, but I do know that we'll want a good
algorithm here because hard disks (and the number of files
on them) are growing so much faster than anything else in
our computers...

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

2003-09-24 04:26:55

by Al Viro

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, Sep 24, 2003 at 05:56:29AM +0200, Andrea Arcangeli wrote:

> But I guess I'd better go away as you ask since I feel like nobody
> understands why I'm not using it despite I'm trying to explain it a few
> times already.

Three letters: TMI. Look, if you really wanted you could post
the description of your favorite positions and comparative study of
their effects on your productivity. Once. It would certainly raise
a lot of eyebrows, but that would be it. But if you kept explaining your
preferences again and again and again, people would get annoyed. And
if you would get into a habit of making recommendations to other folks,
people would get *really* annoyed. Fast. And that's pretty much what's
happening.

2003-09-24 07:56:56

by Pau Aliagas

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Tue, 23 Sep 2003, Linus Torvalds wrote:

> Use CVS and be happy. But don't complain to others that have needs that
> CVS simply can't fill.
> ....
> Indeed. That's pretty much all non-distributed stuff is useful for, from
> where I'm stading. Small projects with a few developers and a lot of
> read-only stuff. And even there the developers will bitch about the
> limitations.
>
> Sure, SVN makes branches cheaper, but you still have to work with them
> like under CVS, ie merging is a total disaster. And you still can't make
> it your private repository.

No flame wars intended, but arch does this and more. See:
http://savannah.gnu.org/projects/gnu-arch

Distributed, complex merging, easy branches... and even a linux repository
from 0.1, not yet in the detail of the current cvs or bk, but easyly
achievable if anyone is interested in spending the time to import it.

Pau

2003-09-24 13:13:57

by Alan

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Mer, 2003-09-24 at 04:45, Larry McVoy wrote:
> Oh, I agree. The problem space changes and you have to move with it.
> I was just pointing out that by not learning from others you put yourself
> at a disadvantage. There are a lot of bright minds in the Linux space,
> there is no question of that. But they would do so much better if they
> understood the past. The past holds a lot of information, ignoring it
> is a disadvantage.

It isnt just sizes. As the size has risen rapidly the disk data rates
have increased pretty well (factor of about 100 over an MFM disk 386)
but the seek time has shifted by a factor of 10. This has a huge impact
on the whole basic theory of things like paging in applications versus
doing a streaming preload of the code/data.

It also has a big impact on how swap is managed - it is pretty much as
cheap now to swap out 2Mb as 4K

2003-09-24 14:45:28

by Roman Zippel

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Hi,

On Tue, 23 Sep 2003, Linus Torvalds wrote:

> See? You're not just being impolite; your complaints are actually morally
> offensive.

You think this is impolite? I've been quiet lately in these flame wars,
because I couldn't have staid that polite to an arrogant asshole, but I
won't stay quiet when Larry can now insult other developers and gets away
with it. If someone is whining around here, then it's only Larry. A lot of
people are contributing to the kernel, but he is the only one constantly
whining around that he should be treated special and makes silly threats.
Can we please get cause and effect right here? Of course Larry is
completely free to choose whatever license he wants, but advertising a
stupid license in a free software environment will of course cause
disagreement and complaints. There is _nothing_ anyone but Larry can do
anything about it, but amazingly enough Larry manages to take every single
complaint as personal insult and makes all of the bk flame wars worse
than they already are.
Andreas sig is about as offensive as a "Don't drink and drive" spot. The
bk license is problematic for a number of developers and it's very
important that new developers are made aware of the problems and not just
think it's cool to use bk because the mighty Linus is using it.

bye, Roman

2003-09-24 17:39:31

by Ken Ryan

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Andrea Arcangeli <[email protected]> wrote in message news:<[email protected]>...
> On Tue, Sep 23, 2003 at 06:06:00PM +0200, Willy Tarreau wrote:
> > why do you want to force people to do something the way you think is best ?
>
> because it gives you no real disavantage to pass the parameter in grub.
> You've to set root=/dev/root anyways, this way you would simply need to
> add the parameter for log buf as well. I've always to add lots of other
> stuff anyways, including vga= profile= etc..
>
> > I personnaly think that both solutions are a convenient way of achieving the
> > same goal for different people. Please let them choose.
>
> they can already choose with the parameter at boot, it's not that I
> don't let them choose.

I would like to add my two pennies...

At home I have a box with 4 IDE controllers (8 channels). Up to and
including 2.4.20 I needed a kernel parameter list specifying
addresses, ata66, etc. for all but ide0 and ide1, as well as another
couple tweaks. I literally was *at* the line length limit. While my
specific situation was eased by .21 and .22 (all I need now is
'ide=reverse'), IMVHO as long as there is a hard limit on the
parameter line length it would be very nice to have the option of
minimizing what needs to be specified.

(FWIW I use lilo; I've never tried grub. I haven't actually looked
whether the line length limit is in lilo or the kernel.)

Ken Ryan

2003-09-24 18:57:23

by Jörn Engel

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Wed, 24 September 2003 14:09:38 +0100, Alan Cox wrote:
>
> It isnt just sizes. As the size has risen rapidly the disk data rates
> have increased pretty well (factor of about 100 over an MFM disk 386)
> but the seek time has shifted by a factor of 10. This has a huge impact
> on the whole basic theory of things like paging in applications versus
> doing a streaming preload of the code/data.
>
> It also has a big impact on how swap is managed - it is pretty much as
> cheap now to swap out 2Mb as 4K

How do you get to 2MB? My rule of thumb number is still 100kB.

J?rn

--
Invincibility is in oneself, vulnerability is in the opponent.
-- Sun Tzu

2003-09-24 21:19:19

by Victor Yodaiken

[permalink] [raw]

Subject: Re: log-buf-len dynamic

If people would just wake up to the lossy memory compression algorithms that
Steve Tweedie invented, the whole problem could be solved in a moment.

On Wed, Sep 24, 2003 at 12:12:51AM -0400, Rik van Riel wrote:
> On Tue, 23 Sep 2003, Linus Torvalds wrote:
>
> > Yeah, SunOS was nice. But I really think it's the access patterns that
> > changed.
>
> Absolutely. Garbage collection and object orientation have
> obsoleted LRU and related algorithms.
>
> Huge memory sizes also have done their share to obsolete
> recency based replacement algorithms.
>
> I suspect we'll want something closer to a frequency based
> replacement scheme in the future. Possibly something like
> LIRS or ARC, but adapted to work in a general purpose OS in
> a very light weight way.
>
> The argument that because we have more memory replacement
> is no longer as important doesn't quite hold because the
> cost of page replacement, measured in cpu cycles, has gone
> completely through the roof in the last decades.
>
> Then there's the whole different bag of cookies that is the
> caching of filesystem metadata, like our inode and dentry
> caches. I'm not sure what would be a good replacement
> strategy for those, but I do know that we'll want a good
> algorithm here because hard disks (and the number of files
> on them) are growing so much faster than anything else in
> our computers...
>
> --
> "Debugging is twice as hard as writing the code in the first place.
> Therefore, if you write the code as cleverly as possible, you are,
> by definition, not smart enough to debug it." - Brian W. Kernighan
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/

--
---------------------------------------------------------
Victor Yodaiken
Finite State Machine Labs: The RTLinux Company.
http://www.fsmlabs.com http://www.rtlinux.com
1+ 505 838 9109

2003-09-25 04:10:09

by Miles Bader

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Roman Zippel <[email protected]> writes:
> amazingly enough Larry manages to take every single complaint as
> personal insult and makes all of the bk flame wars worse than they
> already are. ... Andreas sig is about as offensive as a "Don't drink
> and drive" spot.

Yeah, what he said.

My ruleset looks kinda like this:

Larry contributing BK
==> good, flames inappropriate

Larry being an overbearing, thin-skinned, asshole
==> bad, flames appropriate

Developers using BK
==> their choice, tread lightly

Developers promoting non-BK systems in their sig
==> their choice, tread lightly

Andrea's sig in particular
==> pretty damn innocuous; why on earth is Larry whining about it?

-Miles
--
Would you like fries with that?

2003-09-25 04:21:35

by Nick Piggin

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Miles Bader wrote:

>Roman Zippel <[email protected]> writes:
>
>>amazingly enough Larry manages to take every single complaint as
>>personal insult and makes all of the bk flame wars worse than they
>>already are. ... Andreas sig is about as offensive as a "Don't drink
>>and drive" spot.
>>
>
>Yeah, what he said.
>
>My ruleset looks kinda like this:
>

Can we please drop it? Nobody is perfect, just live and let live. I don't
mean to sound rude so please don't take it that way. And this is not
directed at you in particular Miles.

Thanks,
Nick

2003-09-25 13:40:20

by Marcelo Tosatti

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Mon, 22 Sep 2003, Andrea Arcangeli wrote:

> Hi,
>
> I'm rejecting on the log-buf-len feature in 2.4.23pre5, the code in
> mainline is worthless for any distributor, shipping another rpm package
> just for the bufsize would be way overkill.

Andrea,

As Willy stated previously this is useful for people who want to change
the log buf size without having to change the code manually, and I think
this is a useful and non intrusive change.

Do you see any problem with this?

2003-09-25 17:22:06

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Linus Torvalds <[email protected]> writes:

> On Wed, 24 Sep 2003 [email protected] wrote:
> >
> > It's because I grow up that I can actually better understand the deals
> > it's in my own (again speaking only for myself and not for anybody else)
> > interest to avoid.
>
> You've claimed this now twice.
>
> However, that only explains why you don't use BitKeeper. And everybody
> accepts that. When I started to use BK, I made it _very_ clear that
> service for non-BK users will be _at_least_ as good as it ever was before
> I started using BK.

And for the core kernel development this is true. There are subprojects
that are currently using BK that you can't even get the code without
BK. And the only reason they are using BK is they are attempting to
following how Linux is managed. So having the Linux kernel
development use BK does have some down sides.

In addition there are some major gains to be had in standardizing on a
distributed version control system that everyone can use, and
unfortunately BK does not fill that position. So I think it is good
that there is enough general discontent it the air that people
continue to look for alternatives.

The current situation with version control is painful. CVS branches
poorly and is not distributed. SVN is not distributed. ARCH is
barely distributed and architecturally it makes distributed merging
hard. BK requires open logging which makes it unsuitable for working
on prerelease hardware. BK does not scale to the low end because
to use it successfully you need to make non-BK releases which is
an extra burden. Unless I missed something big all of the BK->foo
gateways are specific to a few source trees. And of course BK won't
let you hack on a replacement.

I don't think the flame wars should stop. The current situation is
not half as good as it could be. And discussion is needed to get us
there. Even pure flames which accomplish nothing technical accomplish
something socially by reminding people that there is the potential to
do much better, and that the current situation is painful.

It is clearly not a solution to simply drop BK, that is even more
painful. To reduce the pain will take a combination of frustration,
time, talent, and a bit of luck that has not happened yet.

Eric

2003-09-25 17:39:57

by Dave Jones

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, Sep 25, 2003 at 11:15:33AM -0600, Eric W. Biederman wrote:

> And for the core kernel development this is true. There are subprojects
> that are currently using BK that you can't even get the code without
> BK. And the only reason they are using BK is they are attempting to
> following how Linux is managed. So having the Linux kernel
> development use BK does have some down sides.

I was expecting this to come up when Linus first made sparse publically
available only by bitkeeper, so I started the nightly snapshots.
If there's any other Linux related project that's bitkeeper only
let me know, and I'll be happy to host nightly snapshots of that too.

Dave

--
Dave Jones http://www.codemonkey.org.uk

2003-09-25 17:37:42

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On 25 Sep 2003, Eric W. Biederman wrote:
> >
> > However, that only explains why you don't use BitKeeper. And everybody
> > accepts that. When I started to use BK, I made it _very_ clear that
> > service for non-BK users will be _at_least_ as good as it ever was before
> > I started using BK.
>
> And for the core kernel development this is true. There are subprojects
> that are currently using BK that you can't even get the code without
> BK. And the only reason they are using BK is they are attempting to
> following how Linux is managed. So having the Linux kernel
> development use BK does have some down sides.

That's actually a pretty good point. I end up releasing "sparse" only as a
BK archive, simply because I'm too lazy to care and there aren't enough
people involved (and those that _are_ involved do actually end up
re-exporting it as non-BK, but that doesn't invalidate your point).

I don't know what the solution to it might be - but I don't think the
reason they are using BK is that they are trying to emulate "the great
kernel project". I know it wasn't for me - it's just that once you get
used to BK, there's no way you'll ever go back to CVS willingly.

> In addition there are some major gains to be had in standardizing on a
> distributed version control system that everyone can use, and
> unfortunately BK does not fill that position.

I don't disagree, but I don't see a real way to solve it. As Larry will
tell you, the technical problems are bigger than you imagine. So a BK
killer won't be coming any time soon, methinks.

Linus

2003-09-25 17:37:42

by Christoph Hellwig

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, Sep 25, 2003 at 11:15:33AM -0600, Eric W. Biederman wrote:
> And for the core kernel development this is true. There are subprojects
> that are currently using BK that you can't even get the code without
> BK. And the only reason they are using BK is they are attempting to
> following how Linux is managed. So having the Linux kernel
> development use BK does have some down sides.

Stupid argument. E.g. the ppc folks used BK much longer than Linus.
And there are kernel projects using svn (ieee1394) or cvs that you
can't access without installing svn or cvs.

> In addition there are some major gains to be had in standardizing on a
> distributed version control system that everyone can use, and
> unfortunately BK does not fill that position. So I think it is good
> that there is enough general discontent it the air that people
> continue to look for alternatives.

Why should we standardize on one SCM? That's like we standadize
on Windows for all computers..

2003-09-25 17:58:25

by Jeff Garzik

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, Sep 25, 2003 at 10:30:58AM -0700, Linus Torvalds wrote:
> I don't know what the solution to it might be - but I don't think the
> reason they are using BK is that they are trying to emulate "the great
> kernel project". I know it wasn't for me - it's just that once you get
> used to BK, there's no way you'll ever go back to CVS willingly.

Nod. All my new projects start out in BitKeeper these days, because
it's so much better than cvs.

Jeff

2003-09-25 18:36:24

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, Sep 25, 2003 at 06:36:48PM +0100, Dave Jones wrote:
> On Thu, Sep 25, 2003 at 11:15:33AM -0600, Eric W. Biederman wrote:
>
> > And for the core kernel development this is true. There are subprojects
> > that are currently using BK that you can't even get the code without
> > BK. And the only reason they are using BK is they are attempting to
> > following how Linux is managed. So having the Linux kernel
> > development use BK does have some down sides.
>
> I was expecting this to come up when Linus first made sparse publically
> available only by bitkeeper, so I started the nightly snapshots.

Qwest is taking forever with our T1 lines but once that problem is solved
we'll put up a version of the BK server on bkbits that will give you
either GNU style patches or tarballs of an exported tree. Then *all*
projects mirrored on bkbits are automagically exported to good old plain
text format.

And, because people might want to build their own gateways (we can only
hope) the diffs can be extracted with checkin comments as shown below.
That ought to be sufficient so that you can do whatever you want. I'm
pretty sure that patch(1) will just ignore all the leading comments.

How's that?

===== panic.c 1.12 vs + =====
2003/08/07 10:29:13 [email protected] 1.13 +6 -1
Don't trigger NMI watchdog for panic delay

--- 1.12/kernel/panic.c Mon May 12 11:11:38 2003
+++ +/kernel/panic.c Thu Aug 7 10:29:13 2003
@@ -16,6 +16,7 @@
#include <linux/init.h>
#include <linux/sysrq.h>
#include <linux/interrupt.h>
+#include <linux/nmi.h>

asmlinkage void sys_sync(void); /* it's really int */

@@ -71,12 +72,16 @@ NORET_TYPE void panic(const char * fmt,

if (panic_timeout > 0)
{
+ int i;
/*
* Delay timeout seconds before rebooting the machine.
* We can't use the "normal" timers since we just panicked..
*/
printk(KERN_EMERG "Rebooting in %d seconds..",panic_timeout);
- mdelay(panic_timeout*1000);
+ for (i = 0; i < panic_timeout; i++) {
+ touch_nmi_watchdog();
+ mdelay(1000);
+ }
/*
* Should we run the reboot notifier. For the moment Im
* choosing not too. It might crash, be corrupt or do

2003-09-25 18:37:24

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Dave Jones <[email protected]> writes:

> On Thu, Sep 25, 2003 at 11:15:33AM -0600, Eric W. Biederman wrote:
>
> > And for the core kernel development this is true. There are subprojects
> > that are currently using BK that you can't even get the code without
> > BK. And the only reason they are using BK is they are attempting to
> > following how Linux is managed. So having the Linux kernel
> > development use BK does have some down sides.
>
> I was expecting this to come up when Linus first made sparse publically
> available only by bitkeeper, so I started the nightly snapshots.
> If there's any other Linux related project that's bitkeeper only
> let me know, and I'll be happy to host nightly snapshots of that too.

The linux infiniband project, is actually what I was thinking of.
http://infiniband.sourceforge.net/

Eric

2003-09-25 18:27:21

by Charles Cazabon

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Linus Torvalds <[email protected]> wrote:
> On 25 Sep 2003, Eric W. Biederman wrote:
>
> > And for the core kernel development this is true. There are subprojects
> > that are currently using BK that you can't even get the code without BK.

> That's actually a pretty good point.
[...]
> I don't know what the solution to it might be

Perhaps BitMover could release a client that can't do anything but keep a
local (unmodified) tree in sync with a public repository tree, so that the
"politically objectionable" (to some) parts of the BK license don't matter.

In an idea world, this read-only client could be released in source form, but
I'm under no illusions there :).

Charles
--
-----------------------------------------------------------------------
Charles Cazabon <[email protected]>
GPL'ed software available at: http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

2003-09-25 18:41:59

by Randy.Dunlap

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, 25 Sep 2003 20:22:33 +0200 J?rn Engel <[email protected]> wrote:

| On Thu, 25 September 2003 10:30:58 -0700, Linus Torvalds wrote:
| >
| > That's actually a pretty good point. I end up releasing "sparse" only as a
| > BK archive, simply because I'm too lazy to care and there aren't enough
| > people involved (and those that _are_ involved do actually end up
| > re-exporting it as non-BK, but that doesn't invalidate your point).
|
| Actually, sparse taught me how life was before the internet. I had to
| ask someone else to get it for me, don't want to bother him on a
| regular basis and am stuck with an old version. Not that I've done
| anything with it worth giving back, but I do feel the pain.
|
| On the other hand, if it matters enough, people will find solutions.
| Would be nice, if Larry coded up something central that all projects
| could benefit from, but that is not necessary. All projects that grow
| important enough will have a solution someday.
|
| BTW: Is there a non-bk way to get recent sparse code yet?

Snapshots are available at
http://www.codemonkey.org.uk/projects/sparse/

--
~Randy

2003-09-25 18:38:09

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, Sep 25, 2003 at 08:22:33PM +0200, J?rn Engel wrote:
> Would be nice, if Larry coded up something central that all projects
> could benefit from, but that is not necessary.

Is the "bkbits.net can export any patch and/or version of a tree as plain
text" match what you are asking for or not?

So if there was a URL that you could type in that fed you the patch as
plain text, with any associated checkin comments, is that what you meant?
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

2003-09-25 18:30:50

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, Sep 25, 2003 at 12:28:38PM -0600, Charles Cazabon wrote:
> Perhaps BitMover could release a client that can't do anything but keep a
> local (unmodified) tree in sync with a public repository tree, so that the
> "politically objectionable" (to some) parts of the BK license don't matter.
>
> In an idea world, this read-only client could be released in source form, but
> I'm under no illusions there :).

People ask us for this all the time and it just highlights the point that
people don't understand how BK works. It isn't client server, it's peer
to peer, every so-called client has to have all the smarts built in that
the so-called server has.

There isn't any way to release a stripped down version that makes sense.
If there was, we would.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

2003-09-25 18:27:21

by Jörn Engel

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, 25 September 2003 10:30:58 -0700, Linus Torvalds wrote:
>
> That's actually a pretty good point. I end up releasing "sparse" only as a
> BK archive, simply because I'm too lazy to care and there aren't enough
> people involved (and those that _are_ involved do actually end up
> re-exporting it as non-BK, but that doesn't invalidate your point).

Actually, sparse taught me how life was before the internet. I had to
ask someone else to get it for me, don't want to bother him on a
regular basis and am stuck with an old version. Not that I've done
anything with it worth giving back, but I do feel the pain.

On the other hand, if it matters enough, people will find solutions.
Would be nice, if Larry coded up something central that all projects
could benefit from, but that is not necessary. All projects that grow
important enough will have a solution someday.

BTW: Is there a non-bk way to get recent sparse code yet?

J?rn

--
He who knows others is wise.
He who knows himself is enlightened.
-- Lao Tsu

2003-09-25 19:03:32

by Jörn Engel

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, 25 September 2003 11:36:23 -0700, Larry McVoy wrote:
> On Thu, Sep 25, 2003 at 08:22:33PM +0200, J?rn Engel wrote:
> > Would be nice, if Larry coded up something central that all projects
> > could benefit from, but that is not necessary.
>
> Is the "bkbits.net can export any patch and/or version of a tree as plain
> text" match what you are asking for or not?

Something like that, yes.

> So if there was a URL that you could type in that fed you the patch as
> plain text, with any associated checkin comments, is that what you meant?

That would be perfect, I would even go for a lot less. As long as I
can recreate last weeks code of my pet project of the day, life is
good. But feel free to make it even better. :)

Actually, you already do that, just with some html around it. That
could be stripped off on your side to save bandwidth or I could do it
myself with a little perl.

Either way, nice service. Should have looked there earlier.

J?rn

--
He who knows that enough is enough will always have enough.
-- Lao Tsu

2003-09-25 18:50:39

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, Sep 25, 2003 at 11:15:33AM -0600, Eric W. Biederman wrote:
> In addition there are some major gains to be had in standardizing on a
> distributed version control system that everyone can use, and
> unfortunately BK does not fill that position. So I think it is good
> that there is enough general discontent it the air that people
> continue to look for alternatives.

Let's just postulate that my claim that this is harder than it looks is
true. You don't have to agree with it, just pretend you do. Given that,
it's going to be a while before any alternative shows up. People mumble
about arch until they go use it for a while and realize it is about 3-5
years behind BK. Linus isn't about to step backwards that far.

> The current situation with version control is painful.

No kidding. Do you have any suggestions, _realistic_ suggestions, that
we could do to help? Beyond making plain text patches/tarballs available
from every repo hosted on bkbits?
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

2003-09-25 19:29:48

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Linus Torvalds <[email protected]> writes:

> On 25 Sep 2003, Eric W. Biederman wrote:
> > >
> > > However, that only explains why you don't use BitKeeper. And everybody
> > > accepts that. When I started to use BK, I made it _very_ clear that
> > > service for non-BK users will be _at_least_ as good as it ever was before
> > > I started using BK.
> >
> > And for the core kernel development this is true. There are subprojects
> > that are currently using BK that you can't even get the code without
> > BK. And the only reason they are using BK is they are attempting to
> > following how Linux is managed. So having the Linux kernel
> > development use BK does have some down sides.
>
> That's actually a pretty good point. I end up releasing "sparse" only as a
> BK archive, simply because I'm too lazy to care and there aren't enough
> people involved (and those that _are_ involved do actually end up
> re-exporting it as non-BK, but that doesn't invalidate your point).
>
> I don't know what the solution to it might be - but I don't think the
> reason they are using BK is that they are trying to emulate "the great
> kernel project". I know it wasn't for me - it's just that once you get
> used to BK, there's no way you'll ever go back to CVS willingly.

I expressed it badly. But the kernel using BK both introduces people
to BK, and it validates BK as a useful tool. Especially as the kernel
tends to embody many of the best practices in the community.

The only sure piece of the solution I have it to keep saying there is
a problem, and to raise the level of the conversation to it's
technical merits.

> > In addition there are some major gains to be had in standardizing on a
> > distributed version control system that everyone can use, and
> > unfortunately BK does not fill that position.
>
> I don't disagree, but I don't see a real way to solve it. As Larry will
> tell you, the technical problems are bigger than you imagine. So a BK
> killer won't be coming any time soon, methinks.

It does not have to start as a BK killer. Just something that is
architecturally sound, once all of the basics work adding the polish
can be done by the community.

I am pretty certain I could write up something that got the core of
the repository right in a month or two. The hard part is avoiding
mistakes in repository architecture. CVS has never recovered, despite
a lot of good work put into it. For the few blunders Larry has made
he has worked very hard to correct things.

But of course if I do something I won't be targeting the kernel
developers directly. I will do it to solve my problems on the
projects I lead. Which may not map well to kernel development. But
doing anything else would be an academic exercise because I could
not maintain it.

Eric

2003-09-25 19:28:28

by Erik Andersen

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu Sep 25, 2003 at 06:31:49PM +0100, Christoph Hellwig wrote:
> Stupid argument. E.g. the ppc folks used BK much longer than Linus.
> And there are kernel projects using svn (ieee1394) or cvs that you
> can't access without installing svn or cvs.

I use the 'cvsweb' interface to track the ieee1394 codebase. I
can download specific patches, and anytime I feel like it, it
will automagically generate a current tarball for me. Therefore,
I can easily track it without needing to use svn (which I do
not have installed).

-Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

2003-09-25 19:40:08

[permalink] [raw]

Subject: OT go to gnu-arch-users for these matters (Re: log-buf-len dynamic)

On Thu, Sep 25, 2003 at 09:02:22PM +0200, J?rn Engel wrote:
> Actually, you already do that, just with some html around it. That
> could be stripped off on your side to save bandwidth or I could do it
> myself with a little perl.

I already did that in python, no need to reinvet the wheel :)

It works fine for any project hosted on bkbits but only as far as you've
a reference point to start with (i.e. you've an old tarball) and as far
as there are no merge from cloned trees in the main branch.

http://www.us.kernel.org/pub/linux/kernel/people/andrea/openbkweb/openbkweb-0.0.tar.gz

I'll let it die quickly first because it was using too much bandwidth
according to Larry, and secondly because it couldn't handle merges
across different trees. that's why Larry provided us with the bkcvs
then. But it should still work in theory, though I doubt it can be very
useful if they do merges.

BTW, for obvious reasons I'm posting emails on these matter _only_ on
gnu-arch-users, I already suggested a number of features I need and that
I'd like to add to arch to be usable for my 2.4-aa (for an usage like
b*tkeeper it seems just usable right now but I need more dedicated
features to be able to maintain my tree with a revision control system,
like being able to hook [tag in arch terms] to a unpacked tree with
hardlinks and be able to rollback rejecting patchsets and overwriting
them in new revisions, and automatic conversion from patchsets to an
ordered serie of patches with a meaningful name). All the details here:

http://mail.gnu.org/mailman/listinfo/gnu-arch-users

thanks,

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-25 19:43:34

by Mudama, Eric

[permalink] [raw]

Subject: RE: log-buf-len dynamic

> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> Subject: Re: log-buf-len dynamic
>
> I expressed it badly. But the kernel using BK both introduces people
> to BK, and it validates BK as a useful tool. Especially as the kernel
> tends to embody many of the best practices in the community.

Personally I fail to see the problem with this. BK must be (I assume) a
useful tool to someone, or it wouldn't exist.

> I am pretty certain I could write up something that got the core of
> the repository right in a month or two.

I've only been following LKML for about 4 months, and this must be the 5th
or
6th time I've seen someone post an email with this jist to it. If it only
took a month or two, I am sure someone would have posted an announcement
about their new tool in the time I've been on the list.

--eric

2003-09-25 20:04:18

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Larry McVoy <[email protected]> writes:

> On Thu, Sep 25, 2003 at 11:15:33AM -0600, Eric W. Biederman wrote:
> > In addition there are some major gains to be had in standardizing on a
> > distributed version control system that everyone can use, and
> > unfortunately BK does not fill that position. So I think it is good
> > that there is enough general discontent it the air that people
> > continue to look for alternatives.
>
> Let's just postulate that my claim that this is harder than it looks is
> true. You don't have to agree with it, just pretend you do. Given that,
> it's going to be a while before any alternative shows up. People mumble
> about arch until they go use it for a while and realize it is about 3-5
> years behind BK. Linus isn't about to step backwards that far.

Nope. I won't. I would much rather postulate that it is easier than
it looks so someone will get started :)

> > The current situation with version control is painful.
>
> No kidding. Do you have any suggestions, _realistic_ suggestions, that
> we could do to help? Beyond making plain text patches/tarballs available
> from every repo hosted on bkbits?

patches/tarballs and a clear way to find them would certainly help.
And would pretty much put a project on bkbits on par with the non bk
alternatives, for usability.

Eric

2003-09-25 20:19:40

by David Lang

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Larry, have you or could you publish the BK->?? export code?, This way
even if you do not host the exported stuff it would be trivial for a
person who is willing to use bk to setup a server to mirror any 'bk-only'
project.

David Lang

On Thu, 25 Sep 2003, Larry McVoy wrote:

> Date: Thu, 25 Sep 2003 11:29:22 -0700
> From: Larry McVoy <[email protected]>
> To: Kernel Mailing List <[email protected]>,
> Larry McVoy <[email protected]>
> Subject: Re: log-buf-len dynamic
>
> On Thu, Sep 25, 2003 at 12:28:38PM -0600, Charles Cazabon wrote:
> > Perhaps BitMover could release a client that can't do anything but keep a
> > local (unmodified) tree in sync with a public repository tree, so that the
> > "politically objectionable" (to some) parts of the BK license don't matter.
> >
> > In an idea world, this read-only client could be released in source form, but
> > I'm under no illusions there :).
>
> People ask us for this all the time and it just highlights the point that
> people don't understand how BK works. It isn't client server, it's peer
> to peer, every so-called client has to have all the smarts built in that
> the so-called server has.
>
> There isn't any way to release a stripped down version that makes sense.
> If there was, we would.
> --
> ---
> Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

2003-09-25 20:27:36

[permalink] [raw]

Subject: Re: log-buf-len dynamic

I can't, it's mostly C code in BK itself and it is in a higher
performance commercial version of BK that we haven't released yet.

On Thu, Sep 25, 2003 at 01:15:38PM -0700, David Lang wrote:
> Larry, have you or could you publish the BK->?? export code?, This way
> even if you do not host the exported stuff it would be trivial for a
> person who is willing to use bk to setup a server to mirror any 'bk-only'
> project.

2003-09-25 23:36:26

by Pau Aliagas

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, 25 Sep 2003, Larry McVoy wrote:

> On Thu, Sep 25, 2003 at 11:15:33AM -0600, Eric W. Biederman wrote:
> > In addition there are some major gains to be had in standardizing on a
> > distributed version control system that everyone can use, and
> > unfortunately BK does not fill that position. So I think it is good
> > that there is enough general discontent it the air that people
> > continue to look for alternatives.
>
> Let's just postulate that my claim that this is harder than it looks is
> true. You don't have to agree with it, just pretend you do. Given that,
> it's going to be a while before any alternative shows up. People mumble
> about arch until they go use it for a while and realize it is about 3-5
> years behind BK. Linus isn't about to step backwards that far.

Have you tried it lately? I'm not sure who's behind because I've never
used bk, but only having a look at arch and using it for a couple of
weeks would convince anybody of the opposite. Just wait that there is a
linux arch repo with the bk detail and facts will speak.

That's why there's the need of having all the bk linux related repos
exported to CVS, not to limit linux development. If only one of these
repos is not exported, many people have difficulties in keeping off bk. I
appreciate you comment that all will be exported RSN.

The same that you demand respect for bk, please respect the others too and
don't spread false comments. Moreover, you are on an advantadgeous
position as you can test arch, read its code and learn -it's GPLed-, while
the rest of people can't even benchmark arch with bk as it's forbidden by
your license. Don't misunderstand me, I'm not flaming anyone, I'm just
exposing facts.

> > The current situation with version control is painful.
>
> No kidding. Do you have any suggestions, _realistic_ suggestions, that
> we could do to help? Beyond making plain text patches/tarballs available
> from every repo hosted on bkbits?

It would be great to keep an arch gateway with the same detail than bk
changesets. It looks very easy to do.

Pau

2003-09-26 02:26:47

by Miles Bader

[permalink] [raw]

Subject: Re: log-buf-len dynamic

[email protected] (Eric W. Biederman) writes:
> ARCH is barely distributed and architecturally it makes distributed
> merging hard.

Are you are kidding? Arch is _insanely_ good at handling both
distributed repositories and merging -- those are if anything its
greatest strengths. Everyday development of tla (the latest/greatest
arch implementation) involves many people with their own repositories,
merging back and forth.

Really, if you have explicit complaints/observations about arch's
handling of these things, please share them, because on the surface
that statement just seems kind of bizarre.

-Miles
--
`The suburb is an obsolete and contradictory form of human settlement'

2003-09-26 04:43:41

by Davide Libenzi

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, 26 Sep 2003, Miles Bader wrote:

> [email protected] (Eric W. Biederman) writes:
> > ARCH is barely distributed and architecturally it makes distributed
> > merging hard.
>
> Are you are kidding? Arch is _insanely_ good at handling both
> distributed repositories and merging -- those are if anything its
> greatest strengths. Everyday development of tla (the latest/greatest
> arch implementation) involves many people with their own repositories,
> merging back and forth.

I definitely agree. It's about a couple of months that I'm playing with it
and I have to say that it works great with distributed development. It
basically born with that as the very first design rule. It also look very
stable AFAICS. And, the old collection of shell scripts (that I didn't
like in the beginning) is shaping out toward C code. In three words, I
like it. I cannot compare it to bk because I never used it (not because
of the license, and not even for political/personal reasons, but because I
haven't had any time to do it in the past), and I am sure it still lacks
of some useful features when matched with bk, but the fundamentals are
definitely there.

- Davide

2003-09-26 14:24:39

by Samium Gromoff

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Larry wrote:
> it's going to be a while before any alternative shows up. People mumble
> about arch until they go use it for a while and realize it is about 3-5
> years behind BK. Linus isn't about to step backwards that far.

Thanks for a dose of desinf^D^D^D^D^D^Denlightenment, Larry.

Seriously, someone whom Linus have entrusted his revision control
needs, should be a bit more objective, to say the least.

In this way Linus is going to keep telling things like:
> Andrea - please just shut up.
>
> Until you can point to anything even _remotely_ as good as BitKeeper,
> there's no point in just continually trying to start a flame-war.
_quite_ for some time...

BTW i really doubt you are so badly informed on Arch progress,
since it is you most close competitor.

And, as the last note, if Linus was at least somewhat interested
in replacing bk with something less community-irritating, he could
do google queries like that more often:

google://"revision control" "distributed development"

Guess what? Arch is on the _first_ hit.

cheers, Samium Gromoff

2003-09-26 14:49:33

by Al Viro

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Fri, Sep 26, 2003 at 05:24:31PM +0400, Samium Gromoff wrote:

> And, as the last note, if Linus was at least somewhat interested
> in replacing bk with something less community-irritating, he could
> do google queries like that more often:
>
> google://"revision control" "distributed development"
>
>
> Guess what? Arch is on the _first_ hit.

You know, if you look there for "object-oriented language" the first hit
you'll get will claim that C++ and Java are such...

2003-09-26 17:10:03

by John Goerzen

[permalink] [raw]

Subject: Re: log-buf-len dynamic

Before replying, I should let you know that I have made an Arch
repository containing every version of Linux since 0.01 clear through
2.6.0-test5 available at http://arch.debian.org/. Please see the docs
there for more info. While I have learned a lot about Arch since then
(particularly relating to proper naming of categories), you should
still be able to use it to experiment with the strengths and
weaknesses of arch, particularly relating to branching and marging.

[email protected] (Eric W. Biederman) writes:

> The current situation with version control is painful. CVS branches
> poorly and is not distributed. SVN is not distributed. ARCH is
> barely distributed and architecturally it makes distributed merging

That is completely wrong wrt arch; in fact, I would say it is the most
intrinsically distributed of any VC system I've seen. Please see:

Elementary Branches
http://regexps.srparish.net/tutorial-tla/elementary-branches.html

Development Branches with star-merge
http://regexps.srparish.net/tutorial-tla/development-branches.html

Introducing Changesets
http://regexps.srparish.net/tutorial-tla/introducing-changesets.html

-- John

2003-09-26 20:25:25

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thu, Sep 25, 2003 at 02:40:15PM +0100, [email protected] wrote:
> On Mon, 22 Sep 2003, Andrea Arcangeli wrote:
>
> > Hi,
> >
> > I'm rejecting on the log-buf-len feature in 2.4.23pre5, the code in
> > mainline is worthless for any distributor, shipping another rpm package
> > just for the bufsize would be way overkill.
>
> Andrea,
>
> As Willy stated previously this is useful for people who want to change
> the log buf size without having to change the code manually, and I think
> this is a useful and non intrusive change.
>
> Do you see any problem with this?

as wrote in a later email, technically the dynamic solution obsoletes
the static one, but only as far as the kernel is concerned.

It happens that to enable the dynamic solution you've to change 1 line
to lilo.conf/grub.lst. since 2.4 is already being used in production
distributed scenarios, there is at least a psycology value in providing
a static solution that doesn't involve lilo.conf changes, so people can
deploy the feature without having to edit conf files at all (with grub
they don't need to run lilo either).

So I will rewrite my patch in a way that doesn't reject the static
setting for next -aa (together with some other tons of pending things,
but the vm merge is already completed on my tree, so I'm in sync again
with pre5 on that side and that was top priority to fixup to be sure
nothing was missing, thanks for merging the watermarks already btw ;).

Andrea - If you prefer relying on open source software, check these links:
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
http://www.cobite.com/cvsps/
svn://svn.kernel.org/linux-2.[46]/trunk

2003-09-29 08:59:35

by Rob Landley

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Thursday 25 September 2003 13:29, Larry McVoy wrote:
> On Thu, Sep 25, 2003 at 12:28:38PM -0600, Charles Cazabon wrote:
> > Perhaps BitMover could release a client that can't do anything but keep a
> > local (unmodified) tree in sync with a public repository tree, so that
> > the "politically objectionable" (to some) parts of the BK license don't
> > matter.
> >
> > In an idea world, this read-only client could be released in source form,
> > but I'm under no illusions there :).
>
> People ask us for this all the time and it just highlights the point that
> people don't understand how BK works. It isn't client server, it's peer
> to peer, every so-called client has to have all the smarts built in that
> the so-called server has.
>
> There isn't any way to release a stripped down version that makes sense.
> If there was, we would.

I'm under the impression that the real smarts of bitkeeper is keeping a huge
database of independent changes and only making a tree out of them when you
ask to see the thing. I.E. bitkeeper is a huge pile of merge logic that can
take into account not just what the tree looks like now, but everything the
tree has _ever_ looked like.

So a read-only client still needs all this merge logic. Export is the easy
part. VIEW is the hard part. The protocols to marshall the patches from
point A to point B are almost irrelevant, it's sorting and integrating them
into something useful that requires work.

And the reason Andrea hasn't found bitkeeper to be nicer than CVS is he isn't
trying to integrate the work of 300 other developers into his personal tree
simultaneously. BK is really just a merging tool that fixes rejects
automatically, everything else is details...

Am I wrong?

Rob

2003-09-29 11:24:47

by John Bradford

[permalink] [raw]

Subject: Re: log-buf-len dynamic

> BK is really just a merging tool that fixes rejects
> automatically, everything else is details...

IFF that is true, then taking this to it's logical extreme, what is
the point in having an SCM system for kernel development at all?

It could be argued that what we really need is enhanced versions of
diff and patch that actually understand C constructs and are able to
make intellegent decisions about merging two pieces of code, even
without knowledge of other merges.

'Enhanced' is, of course, a complete understatement. What I am
suggesting is basicaly adding A.I. functionality to diff and patch, to
the point where they can merge three pieces of C code as efficiently
as a good developer can.

This would probably involve analysing code and identifying discrete
sections, (analogous to the way a human developer would draw a flow
chart), within which the purpose of algorithms and variable names
could be deduced. This knowledge could then be used to adapt code
that was submitted as a diff against a compltely different piece of
code.

Ultimately, it should be possible for two people to independently
write code to do a certain task, for one of them to add an extra
facility to their codebase, and for the enhanced diff and patch tools
to then add this facilty to the other, completely separate,
implementation.

John

2003-09-29 12:34:08

by Rob Landley

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Monday 29 September 2003 06:24, John Bradford wrote:
> > BK is really just a merging tool that fixes rejects
> > automatically, everything else is details...
>
> IFF that is true, then taking this to it's logical extreme, what is
> the point in having an SCM system for kernel development at all?
>
> It could be argued that what we really need is enhanced versions of
> diff and patch that actually understand C constructs and are able to
> make intellegent decisions about merging two pieces of code, even
> without knowledge of other merges.

But you can't always make intelligent merging decisions without knowledge
about other merges.

Suppose that tree 1 has a line "x=function(3*yy)+2", and tree 2 has the
corresponding line "x=fred+1". Your only merge choice is to pick one or the
other. You as a fully intelligent human being have no other choice unless
you're writing fresh code.

Now let's look at the patch history back to where the two trees diverged from:

Patch series:

original line:
x=yy+1;

Patch 1:
- x=yy+1;
+ x=function(3*yy)+2;

Patch 2:
- x=yy+1;
+ x=fred+1;

Looking at this, you can see that the result you probably want is
"x=function(3*fred)+2;", but you couldn't figure that out until you'd see the
original line they both diverged from. One tree probably had s/yy/fred/
applied to it, and the other tweaked the algorithm, thus the merges conflict
going either way.

But if the original line had instead been "x=fred+2;", the logical merge would
instead be "x=function(3*yy)+1;".

Original line:
x=fred+2;

Patch 1:
- x=fred+2;
+ x=function(3*yy)+2;

Patch 2:
- x=fred+2;
+ x=fred+1;

Logical merge: "x=function(3*yy)+1;"

You CANNOT KNOW THIS until you see the common ancestor they diverged from.
There just isn't enough information to work it out, unless you do so from
first principles by actually comprehending the algorithm being implemented
and what the changes are trying to accomplish.

That's why bitkeeper has to remember all the past changes to do its job of
merging new ones.

> 'Enhanced' is, of course, a complete understatement. What I am
> suggesting is basicaly adding A.I. functionality to diff and patch, to
> the point where they can merge three pieces of C code as efficiently
> as a good developer can.

If you've gotten AI working, why not just have it write the code and save us
the trouble?

> This would probably involve analysing code and identifying discrete
> sections, (analogous to the way a human developer would draw a flow
> chart), within which the purpose of algorithms and variable names
> could be deduced.

You're suggesting making a computer figure out the purpose of something.
Uh-huh. Has this ever happened, even once, in the entire history of
computing?

> This knowledge could then be used to adapt code
> that was submitted as a diff against a compltely different piece of
> code.

Written in a diferent language, performing a completely different task even...

> Ultimately, it should be possible for two people to independently
> write code to do a certain task, for one of them to add an extra
> facility to their codebase, and for the enhanced diff and patch tools
> to then add this facilty to the other, completely separate,
> implementation.

It'd be nice, sure. I do not know how to implement that. I don't know
anybody who does. If you do, by all means code it up. (My guess is you
don't understand the scope of the problem, but I could always be wrong....)

> John

Rob

2003-09-29 13:23:52

by Valdis Klētnieks

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Mon, 29 Sep 2003 12:24:47 BST, John Bradford <[email protected]> said:

> It could be argued that what we really need is enhanced versions of
> diff and patch that actually understand C constructs and are able to
> make intellegent decisions about merging two pieces of code, even
> without knowledge of other merges.

So what you're saying is that diff and patch should drag along all the BK
functionality, databases, and so forth.....

% ls -l /usr/bin/diff /usr/bin/patch
-rwxr-xr-x 1 root root 64540 Jun 5 14:57 /usr/bin/diff
-rwxr-xr-x 1 root root 83828 Jun 5 22:41 /usr/bin/patch
% man diff | wc -l
310
% man patch | wc -l
675

Larry, how big is the BK binary for an x86? And how big are the docs?

Attachments:

(No filename) (226.00 B)

2003-09-29 13:20:24

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Mon, 29 Sep 2003, John Bradford wrote:

> 'Enhanced' is, of course, a complete understatement. What I am
> suggesting is basicaly adding A.I. functionality to diff and patch, to
> the point where they can merge three pieces of C code as efficiently
> as a good developer can.

In short, you want to implement the "graduate student
algorithm" in software ?

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

2003-09-29 15:07:15

[permalink] [raw]

Subject: Re: log-buf-len dynamic

On Mon, Sep 29, 2003 at 03:56:27AM -0500, Rob Landley wrote:
> And the reason Andrea hasn't found bitkeeper to be nicer than CVS is he isn't
> trying to integrate the work of 300 other developers into his personal tree
> simultaneously. BK is really just a merging tool that fixes rejects
> automatically, everything else is details...
>
> Am I wrong?

I think so. That's sort of like saying "the kernel is really just a program
that multiplexes processes". Both are true statements if you look at the
world that way but both dramatically underestimate the real picture.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

2003-09-29 15:04:03