On Wed, 1 Sep 2004, David Masover wrote:
>
> I think transactions should be done in kernel space. It's the only way
> to "enforce" them sanely.
Well, they clearly can't, since it depends on the filesystem. Also,
transactions have a tendency to be expensive, and people have a tendency
to ask for more than you give them. Doing transactions on one file is
only the beginning - you'll find people who want transactions across file
boundaries etc.
That's a basic fact when it comes to pretty much anything: you can always
find people who want something better. You can't be all things to all
people, so what you actually _want_ to do is to expose the _minimal_ set
of capabilities that people can build on.
So I definitely don't want this discussion to degenerate (any more than it
lng since has ;) into what people _wish_ for. No, it should be a "what is
the _least_ we can absolutely do" that solves real problems.
Linus
Linus Torvalds wrote:
>
>. Doing transactions on one file is
>only the beginning - you'll find people who want transactions across file
>boundaries etc.
>
>
Yup, that's what reiser4 is aiming at and where things get exciting for
version control systems and the like. We are just missing the api code
for it at the moment, all the infrastructure is there, and the api is in
progress in sys_reiser4.
On Thu, 2 Sep 2004, Hans Reiser wrote:
> Linus Torvalds wrote:
> >
> >. Doing transactions on one file is
> >only the beginning - you'll find people who want transactions across file
> >boundaries etc.
>
> Yup, that's what reiser4 is aiming at and where things get exciting for
> version control systems and the like. We are just missing the api code
> for it at the moment, all the infrastructure is there, and the api is in
> progress in sys_reiser4.
But _my_ point is, no user program is going to take _advantage_ of
anything that only one filesystem on one system offers.
So there's no point.
It's much saner (from _any_ app standpoint) to roll their own database in
user space - that way it just works.
In other words, nobody is really ever going to take advantage of this.
This is _not_ how technical advanncement happens. The way you get people
to take advantage of something is to have a nice gradual ramp-up, not a
sudden new feature that they can't realistically use.
So before you do transactions in the filesystem, you have to be able to do
them in user space - and then make the filesystem version be _compatible_
with that user space library. That way you can get people to use the
library ("hey, it works anywhere") and then you can get them to use your
filesystem ("hey, if you use our super-duper filesystem, you can do what
you are already doing five times faster").
See? You're starting at the wrong end. Give me a portable interface to use
_first_. Then do transactions in the filesystem.
(Now, as an academic exercise and as a learnign experience and prototyping
work it migt make sense to do the filesystem first. But then you have to
really _consider_ it a prototype, and realize that to make it useful you'd
still have to do the user thing before you can "sell" people on the idea).
Linus
On Thursday 02 September 2004 09.15, Hans Reiser wrote:
> Linus Torvalds wrote:
> >. Doing transactions on one file is
> >only the beginning - you'll find people who want transactions across file
> >boundaries etc.
>
> Yup, that's what reiser4 is aiming at and where things get exciting for
> version control systems and the like. We are just missing the api code
> for it at the moment, all the infrastructure is there, and the api is in
> progress in sys_reiser4.
Hope that is the same as or implied XA-support. Having to syncrhonized file
updates with database transactions is not an uncommon problem.
-- robin
Linus Torvalds wrote:
> But _my_ point is, no user program is going to take _advantage_ of
>
>anything that only one filesystem on one system offers.
>
>
Apple does not have this problem....
and yes, the apps will take advantage of it, which is different from
depending on it. If you use the wrong fs you will lose some of the
features of the app.
For 30 years nothing much has happened in Unix filesystem semantics
because of sheer cowardice (excepting Clearcase, which priced itself
into a niche market). It is 25 years past time for someone to change
things. That someone will have first mover advantage, and the more
little semantic features possessed the more lure there will be to use it
which will increase market share which will lure more apps into
depending on it and in a few years the other filesystems will
(deservedly) have only a small market share because the apps won't all
work on them.
Besides, there are enhancements which are simply compelling. You can
write a dramatically better performance version control system with a
much simpler design if the FS is atomic. Our transaction manager
first draft was written by a version control guy, and he would probably
be happy to tell you how lack of atomicity other than rename makes
version control software design hideous.
We have the performance lead. By next year we will be stable enough for
mission critical servers, and then we start the serious semantic
enhancements.
If you don't embrace progress, then you doom Linux to following behind,
because the guys at Apple are pretty aggressive now that Jobs is back,
and they WILL change the semantics, and they will do so in compelling
ways, and Linux will be reduced to aping them when it should be leading
them.
Hans
Linus Torvalds wrote:
>
>But _my_ point is, no user program is going to take _advantage_ of
>anything that only one filesystem on one system offers.
>
>So there's no point.
>
>I
>
Filesystem diversity will only be crippling for Linux if you choose for
it to be so.
How would we go about finding out how many data forks were in a file?
Because in order to be able to retrieve data from a fork we would need
to know that the fork were there. Currently this would imply that we
go looking through mtab or some such to find out what fs we're running
on, which seems ugly.
Alternatively we go through the _exciting_ task of making every other
fs (with the exceptions of ntfs, and whatever it is that macs use,
which would need there own custom code) and add code that effectively
goes
getNumForks(fileref){ return 1;}
or add a new open call that can take a fork number...
either way we have to add a new syscall, that doesn't conform to any
real standard(though i suppose it would be possible to use
macOS/windows style fork iopening interface)
I personally like the concept of having multiple forks in a file, but
in this case I'm inclined towards usermode first, then if it takes off
add kernel level support.
--Oliver Hunt
On Thu, 02 Sep 2004 01:58:30 -0700, Hans Reiser <[email protected]> wrote:
> Linus Torvalds wrote:
>
> > But _my_ point is, no user program is going to take _advantage_ of
> >
> >anything that only one filesystem on one system offers.
> >
> >
> Apple does not have this problem....
>
> and yes, the apps will take advantage of it, which is different from
> depending on it. If you use the wrong fs you will lose some of the
> features of the app.
>
> For 30 years nothing much has happened in Unix filesystem semantics
> because of sheer cowardice (excepting Clearcase, which priced itself
> into a niche market). It is 25 years past time for someone to change
> things. That someone will have first mover advantage, and the more
> little semantic features possessed the more lure there will be to use it
> which will increase market share which will lure more apps into
> depending on it and in a few years the other filesystems will
> (deservedly) have only a small market share because the apps won't all
> work on them.
>
> Besides, there are enhancements which are simply compelling. You can
> write a dramatically better performance version control system with a
> much simpler design if the FS is atomic. Our transaction manager
> first draft was written by a version control guy, and he would probably
> be happy to tell you how lack of atomicity other than rename makes
> version control software design hideous.
>
> We have the performance lead. By next year we will be stable enough for
> mission critical servers, and then we start the serious semantic
> enhancements.
>
> If you don't embrace progress, then you doom Linux to following behind,
> because the guys at Apple are pretty aggressive now that Jobs is back,
> and they WILL change the semantics, and they will do so in compelling
> ways, and Linux will be reduced to aping them when it should be leading
> them.
>
> Hans
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Oliver Hunt wrote:
>How would we go about finding out how many data forks were in a file?
>Because in order to be able to retrieve data from a fork we would need
>to know that the fork were there. Currently this would imply that we
>go looking through mtab or some such to find out what fs we're running
>on, which seems ugly.
>
>
>
Depends on how the forks eventually get implemented.
With the file-as-directory concept, all you need is to
look at the file's directory part to see what is there. (The forks,
implemented as files in a subdirectory.) It is done the same way
as for an ordinary directory, so nothing "new".
>Alternatively we go through the _exciting_ task of making every other
>fs (with the exceptions of ntfs, and whatever it is that macs use,
>which would need there own custom code) and add code that effectively
>goes
>
>getNumForks(fileref){ return 1;}
>
>
>
Necessary if some other mechanism is used, sure.
Helge Hafting
> Depends on how the forks eventually get implemented.
> With the file-as-directory concept, all you need is to
> look at the file's directory part to see what is there. (The forks,
> implemented as files in a subdirectory.) It is done the same way
> as for an ordinary directory, so nothing "new".
>
This still doesn't solve the problem of how we distinguish between a
multi-fork file
and a directory, for those old programs(ie. almost all that currently
exist), to be able to access meaningful data we would need to either
return just the primary fork, a serialized version of all forks in the
file, or make the file look more or less identical to a directory.
The first option could cause problems when transfering files between
different filesystems,
the second results in programs getting metadata they can't handle, and
the third option results in few of the advantages over, well,
directories... And even those applications that could handle the fork
information nicely would need to fs type to find out whether they were
handling a directory or a multi-forked file...
As I say I like the idea, but I can't see anyway of implementing it in
a way that is useful without first putting considerable effort into at
least the VFS if not all the actual fs drivers.
--Oliver Hunt
> Linus Torvalds wrote:
>> But _my_ point is, no user program is going to take _advantage_ of
>>
>>anything that only one filesystem on one system offers.
>>
>>
> Apple does not have this problem....
> and yes, the apps will take advantage of it, which is different from
> depending on it. If you use the wrong fs you will lose some of the
> features of the app.
> For 30 years nothing much has happened in Unix filesystem semantics
> because of sheer cowardice (excepting Clearcase, which priced itself
> into a niche market). It is 25 years past time for someone to change
> things. That someone will have first mover advantage, and the more
> little semantic features possessed the more lure there will be to use it
> which will increase market share which will lure more apps into
> depending on it and in a few years the other filesystems will
> (deservedly) have only a small market share because the apps won't all
> work on them.
> Besides, there are enhancements which are simply compelling. You can
> write a dramatically better performance version control system with a
> much simpler design if the FS is atomic. Our transaction manager
> first draft was written by a version control guy, and he would probably
> be happy to tell you how lack of atomicity other than rename makes
> version control software design hideous.
Btw, version control for ordinary files would be a great feature. I
think something like it is available through Windows 2000/3 server.
Isn't it called "Shadow Copies". It works over network shares. :)
It allows you to restore previous versions of the file even if you
delete or overwrite it.
Features like this do make a good point and helps protecting data -
something that is important IMHO.
http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp
> We have the performance lead. By next year we will be stable enough for
> mission critical servers, and then we start the serious semantic
> enhancements.
> If you don't embrace progress, then you doom Linux to following behind,
> because the guys at Apple are pretty aggressive now that Jobs is back,
> and they WILL change the semantics, and they will do so in compelling
> ways, and Linux will be reduced to aping them when it should be leading
> them.
> Hans
>> Depends on how the forks eventually get implemented.
>> With the file-as-directory concept, all you need is to
>> look at the file's directory part to see what is there. (The forks,
>> implemented as files in a subdirectory.) It is done the same way
>> as for an ordinary directory, so nothing "new".
>>
> This still doesn't solve the problem of how we distinguish between a
> multi-fork file
> and a directory, for those old programs(ie. almost all that currently
> exist), to be able to access meaningful data we would need to either
> return just the primary fork, a serialized version of all forks in the
> file, or make the file look more or less identical to a directory.
Well. wasn't the idea that unless programs specifically tried to
open the file-as-dir as a directory it would look like a file?
ls -F would show it as file. Or have I understood wrong?
> The first option could cause problems when transfering files between
> different filesystems,
The meta-data should be deleted if it the file is copied or moved to
a medium that doesn't support it. However a _warning_ may be shown
to the user if there is risk to loose data.
I do not think we should not implement something like this because
most other filesystems in Linux doesn't support it. Think of other
limitations there is in various ones. Large file support. Limited
amount of files, length of file names, etc.
> the second results in programs getting metadata they can't handle, and
> the third option results in few of the advantages over, well,
> directories... And even those applications that could handle the fork
> information nicely would need to fs type to find out whether they were
> handling a directory or a multi-forked file...
No, if an application supported the file-as-dir then it could simply
try to open the file as a directory. It would fail on systems that
didn't support it.
The meta-data and file streams would be seen as ordinary files. If
applications support the contents or not isn't really relevant, is
it?.
> As I say I like the idea, but I can't see anyway of implementing it in
> a way that is useful without first putting considerable effort into at
> least the VFS if not all the actual fs drivers.
Indeed. It is important that something like this gets implemented as
a transparent way as possible. If it could be done in a general way
so other filesystems like ext3/4 can eventually support it then that
would be wonderful. I do not, however, think that we should block it
in reiser4 because no other filesystems support it.
~S
> --Oliver Hunt
On Thursday 02 September 2004 12:52, Spam wrote:
> > Linus Torvalds wrote:
> > Besides, there are enhancements which are simply compelling. You can
> > write a dramatically better performance version control system with a
> > much simpler design if the FS is atomic. Our transaction manager
> > first draft was written by a version control guy, and he would probably
> > be happy to tell you how lack of atomicity other than rename makes
> > version control software design hideous.
>
> Btw, version control for ordinary files would be a great feature. I
> think something like it is available through Windows 2000/3 server.
> Isn't it called "Shadow Copies". It works over network shares. :)
>
> It allows you to restore previous versions of the file even if you
> delete or overwrite it.
>
> Features like this do make a good point and helps protecting data -
> something that is important IMHO.
I believe you mean something simillar to:
file1.txt;1
file1.txt;2
file1.txt;3 (yeap, it's VMS)
where you'll have to cleanup old versions when you don't need them any more...
AFAIK that this is older than HDDs
<<V13>>
On Iau, 2004-09-02 at 10:52, Spam wrote:
> Btw, version control for ordinary files would be a great feature. I
> think something like it is available through Windows 2000/3 server.
> Isn't it called "Shadow Copies". It works over network shares. :)
Netapps have done this for years with NFS, Subversion does it with DAV,
clearcase does it with NFS.
On Thu, 2 Sep 2004, Spam wrote:
> Well. wasn't the idea that unless programs specifically tried to
> open the file-as-dir as a directory it would look like a file?
>
> ls -F would show it as file. Or have I understood wrong?
Yes, otherwise apps that do
if (S_ISDIR()) {
..
} else if (S_ISREG()) {
..
}
would behave differently from apps that check the file in
different order, and they would probably break because in
a regular fs that order is not important. The goal is to
keep the semantic changes as hidden as possible for apps
that don't know about the new features.
--
Giuliano.
Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
> ? Btw, version control for ordinary files would be a great feature. I
> ? think something like it is available through Windows 2000/3 server.
> ? Isn't it called "Shadow Copies". It works over network shares. :)
>
> ? It allows you to restore previous versions of the file even if you
> ? delete or overwrite it.
There's no need to do that in kernel, unless you want to be able
to force it unto users.
Regards
Oliver
> Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
>> ? Btw, version control for ordinary files would be a great feature. I
>> ? think something like it is available through Windows 2000/3 server.
>> ? Isn't it called "Shadow Copies". It works over network shares. :)
>>
>> ? It allows you to restore previous versions of the file even if you
>> ? delete or overwrite it.
> There's no need to do that in kernel, unless you want to be able
> to force it unto users.
Exactly ;)
Difference with having it in just certain applications like Gnome-VFS
etc is that it would work with all applications, over SMB shares
etc. This is a great advantage and benefit for many desktop users
but also in a corporate environment. Even though there are backups
of users documents it is a pain to restore them every time they
loose the original file. Besides, the backup is never as fresh
either.
> Regards
> Oliver
Linus Torvalds wrote:
>...
>But _my_ point is, no user program is going to take _advantage_ of
>anything that only one filesystem on one system offers.
>
>So there's no point.
>
>It's much saner (from _any_ app standpoint) to roll their own database in
>user space - that way it just works.
>
>In other words, nobody is really ever going to take advantage of this.
>This is _not_ how technical advanncement happens. The way you get people
>to take advantage of something is to have a nice gradual ramp-up, not a
>sudden new feature that they can't realistically use.
>...
>
Sure, but there are plenty of existing interfaces that you could
emulate. One could make a small library to use a transactional
filesystem to implement the Berkely DB interface (libdb) for example.
So on filesystems without such support your app could use the regular
userspace database, but on a transactional filesystem it'd just use
regular files, which would simplify database management and likely
increase performance over the userspace-only version (libdb is pretty
slow). In terms of functionality it'd just be a drop-in replacement,
just like math libraries that use MMX/SSE when available.
- Jim Bruce
Hans> For 30 years nothing much has happened in Unix filesystem
Hans> semantics because of sheer cowardice (excepting Clearcase, which
Hans> priced itself into a niche market).
Keeping a consistent namespace is cowardice? And ClearCase is an
example of an application which sets up it's own namespace and grafts
it onto the standard Unix one. Most standard Unix tools work just
fine inside clearcase Views, but to manage the metadata (xattrs, or
whatever you want to call them), you need to use the 'cleartool'
command. Hmmm... sounds like 'runat' to me.
Hans> It is 25 years past time for someone to change things. That
Hans> someone will have first mover advantage, and the more little
Hans> semantic features possessed the more lure there will be to use
Hans> it which will increase market share which will lure more apps
Hans> into depending on it and in a few years the other filesystems
Hans> will (deservedly) have only a small market share because the
Hans> apps won't all work on them.
This is all pure marketing speak and economic theory. Show us the
*technical* advantages, not just wishful thinking.
Hans> Besides, there are enhancements which are simply compelling.
Hans> You can write a dramatically better performance version control
Hans> system with a much simpler design if the FS is atomic.
Define atomic please, with state diagrams and clear examples.
Hans> We have the performance lead. By next year we will be stable
Hans> enough for mission critical servers, and then we start the
Hans> serious semantic enhancements.
I think you've got it backwards. Make your serious semantic
enhancements first, then make it stable, then make it fast. Because
when you change the semantics, you break all kinds of things and then
it doesn't matter how fast you are.
Hans> If you don't embrace progress, then you doom Linux to following
Hans> behind, because the guys at Apple are pretty aggressive now that
Hans> Jobs is back, and they WILL change the semantics, and they will
Hans> do so in compelling ways, and Linux will be reduced to aping
Hans> them when it should be leading them.
Monkey see, Monkey do then.
I'd like to point out another successful company which has extended
the standard Unix namespace and that's 'Network Appliance' with it's
.snapshot directory structure. It's a great idea and allows my users
to restore files from snapshots without me having to think about it.
But it still causes problems since when I use rsync to move data, I
need to put in stuff like:
rsync -az --exclude ".snapshot" --delete --delete-excluded <src> <dest>
to make sure it doesn't descend into that directory of previous
versions and try to copy them over as well. And of course now now
other apps can use the .snapshot name. But what if other vendors aped
(ook ook) this decided to use their own names? Who decides which gets
priority?
I think you need to go back and re-read Pike's paper on namespaces
that you pointed to before and mull it over. And look at how
simplicity is inherently powerful. If the design is too complicated,
you're probably doing it wrong.
John
V13> I believe you mean something simillar to:
V13> file1.txt;1
V13> file1.txt;2
V13> file1.txt;3 (yeap, it's VMS)
Or TOPS-20, a precursor to VMS in some ways. It was a nice feature.
V13> where you'll have to cleanup old versions when you don't need
V13> them any more... AFAIK that this is older than HDDs
It was usually an automatic cleanup past a certain point or if you
went over disk quota.
In any case, while I do like this feature, I'm not sure how we would
cleanly implement this inside the unix namespace, or if inside a new
namespace, how that new namespace would be joined with a standard Unix
one.
John
On Thursday 02 September 2004 12:02, Spam wrote:
> > As I say I like the idea, but I can't see anyway of implementing it in
> > a way that is useful without first putting considerable effort into at
> > least the VFS if not all the actual fs drivers.
>
> Indeed. It is important that something like this gets implemented as
> a transparent way as possible. If it could be done in a general way
> so other filesystems like ext3/4 can eventually support it then that
> would be wonderful. I do not, however, think that we should block it
> in reiser4 because no other filesystems support it.
What about extending the namespace with leading ".." and "...".
In Unix names starting with a "." already have the meaning of being a
hidden/config entry.
A name starting ".." means streams/metainformation, etc. something belonging
to the file/directory that should get backed up, copied, etc.
There could be a "..streams", "..metas" or a "..acl" entry.
If it starts with "..." it means some system specific information, like the
name of the hash algorithm used for the current directory. It's basically
information that is not portable and not required by applications, like
"...fsplugins".
There is still the big problem of how to copy files with their associated
streams or meta information onto a standard unix filesystem as the
file/directory duality cannot be expressed. (It's forbidden to have
a directory + file with the same name in the same directory)
Maybe copy could create something like a ".#filename" directory for this kind
of information if the advanced features are not supported on the target
filesystem. This is neither nice nor clean, but at least you don't loose
information. I do not suggest that the kernel should simulate the advanced
features with ".#filename" directories, it's more a backup/restore thing
that could work if it's written down properly.
--
lg, Chris
On Iau, 2004-09-02 at 15:24, John Stoffel wrote:
> This is all pure marketing speak and economic theory.
I wouldn't grace it with the label economic theory. Lets be honest about
this.. its pure -research-. Research is good but don't screw around with
Linux semantics until you've done the research, figured out how to also
make it useful to real users and proved its a good idea despite the past
evidence from things like MacOS
Alan
On Thu, 2 Sep 2004, Hans Reiser wrote:
>
> Apple does not have this problem....
Deep, Hans. Deep.
You're ignoring the issue. You're flaming about not "embracing progress",
when I was talking about how progress happens. You don't seem to
understand _how_ people embrace progress, Hans. Railing at windmills isn't
real smart.
Linus
On Thu, 2 Sep 2004, James Bruce wrote:
> Linus Torvalds wrote:
>
> >In other words, nobody is really ever going to take advantage of this.
> >This is _not_ how technical advanncement happens. The way you get people
> >to take advantage of something is to have a nice gradual ramp-up, not a
> >sudden new feature that they can't realistically use.
>
> Sure, but there are plenty of existing interfaces that you could
> emulate.
Absolutely. We should look at them, and whether they solve any issues.
I'm not saying that Hans would have to make up a new interface. Quite the
reverse. One interface I suggested to use for attributes (and everybody
hates ;) was the existing one from Solaris. Similarly, there are probably
perfectly fine interfaces for the issue of transactions..
Linus
On Thu, 2004-09-02 at 05:52, Spam wrote:
> Btw, version control for ordinary files would be a great feature. I
> think something like it is available through Windows 2000/3 server.
> Isn't it called "Shadow Copies". It works over network shares. :)
>
> It allows you to restore previous versions of the file even if you
> delete or overwrite it.
>
> Features like this do make a good point and helps protecting data -
> something that is important IMHO.
>
> http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp
>
Netapp's WAFL has had this for ages, they are called snapshots, and they
are one of the most useful file system features ever. But Microsoft
certainly did not invent them.
Lee
> For 30 years nothing much has happened in Unix filesystem semantics
> because of sheer cowardice
Or because it works fine, and isn't broken.
M.
"Martin J. Bligh" <[email protected]> writes:
> > For 30 years nothing much has happened in Unix filesystem semantics
> > because of sheer cowardice
>
> Or because it works fine, and isn't broken.
I've heard the same argument a lot of times. People complaining that
Unix is so seventies because it sticks to the old boring philosophy of
everything is a file and that a file is a stream of bytes, nothing
more. Modern operating systems such as VMS with basic database
handling in the OS itself, or MacOS or NT with named streams is so
much more modern. Why don't we get with the times?
It may be because just because of the simplicity it's fairly easy to
use, harder to break och does one thing well. If you want structured
storage, use a database, on top of the low level primitives, or use
multiple files in a directory. Why complicate things?
/Christer
--
"Just how much can I get away with and still go to heaven?"
Freelance consultant specializing in device driver programming for Linux
Christer Weinigel <[email protected]> http://www.weinigel.se
On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote:
> > For 30 years nothing much has happened in Unix filesystem semantics
> > because of sheer cowardice
>
> Or because it works fine, and isn't broken.
OK. I'm not a kernel hacker. I'm not a crack C programmer. Nor am I a
designer of filesystems. I'm just a guy that puts together and supports
Linux systems for my customers.
In following this thread, I may be missing huge chunks of concept.
However, a few things are becoming clear to me:
1. The file as directory thing adds complexity that the administrator
has to deal with. Symlinks are useful, but it's still aggravating to
tar off a directory structure, take it somewhere, and then realize that
all you have is links to something not in the archive because you didn't
get your tar switches just right. Now we're talking about adding
another set of "files which are not really files" to the semantics.
More complexity. I'll take simplicity over some ivory tower ideal of
"unified name space" any day.
2. The use of multiple streams within files by Linux apps would make
Linux as cross-platform unfriendly as MS is trying to be. Say these
features start getting used and you copy an OO.org document from a Linux
box to a BSD box. It's broken. Of course, OO.org wouldn't use the
streams in the first place because it would destroy their cross platform
portability. So what's the point? No one who cares about cross
platform portability can use it. Everyone who doesn't care about cross
platform portability please raise your hand.
3. MS does require attributes and multiple streams, which makes these
features important (even essential) to Samba, and Samba alone. Samba is
important to Linux, so this can't be ignored. (Here I am implicitly
assuming that Samba will need kernel support for this to do it right.)
So it seems to me that the only real consideration is giving Samba what
it needs without making the semantics one bit more complex than
absolutely necessary. It might even be wise to discourage use of these
ambiguous new objects by the casual application programmer.
Then again, maybe I just have tunnel vision...
-Steve Bergman
On Thu, 02 Sep 2004 12:02:59 +0200, Spam said:
> The meta-data should be deleted if it the file is copied or moved to
> a medium that doesn't support it. However a _warning_ may be shown
> to the user if there is risk to loose data.
OK... I'll bite. How do you report the warning to the user if you're using
an unenhanced utility to copy a file to a file system that may be lossy?
> On Thu, 02 Sep 2004 12:02:59 +0200, Spam said:
>> The meta-data should be deleted if it the file is copied or moved to
>> a medium that doesn't support it. However a _warning_ may be shown
>> to the user if there is risk to loose data.
> OK... I'll bite. How do you report the warning to the user if you're using
> an unenhanced utility to copy a file to a file system that may be lossy?
That I do not really know. But I think this is somewhat the
discussion here - how to make the support for metas and streams
as good as possible.
I suppose it would be difficult to actually query the user about
something like this unless the applications themselves do. Perhaps
warnings in the syslog is good for now?
Already now things like advanced attributes get lost when copying
data from a supported fs to a non-supported one.
Perhaps documentation around the ways to use streams and meta data
should mention that only the supported fs will keep the extra info.
As long as the user is aware that whatever information he puts in
the meta or streams will be lost then it is ok, isn't it?
~S
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> Martin J. Bligh wrote:
|>>For 30 years nothing much has happened in Unix filesystem semantics
|>>because of sheer cowardice
> |
> |
> | Or because it works fine, and isn't broken.
> Ok, maybe it wasn't cowardice. Maybe it was laziness.
> Because cowardice tells me that data corruption is bad. Data corruption
> kills kittens. End data corruption now!
> Unfortunately, the only way to allow files to survive a crash is to
> implement transactions. The simplest way is to do it in the filesystem
> and to export an interface to userland. This is what Hans is doing,
> though AFAIK it's not done yet.
I thought reiser4 had its journaling and atomic commits. Am I
mistaken? I run reiser4 as primary fs on my test systems and it seem
to work as expected.
> Laziness tells me that I should just use Windows anyway, because "it
> works". Switching to Linux requires training, porting apps requires
> work, and using Linux instead of Windows is a hell of a lot more work
> than just patching programs to do something like
> + start_transaction()
> ~ write()
> ~ blah()
> ~ write()
> + end_transaction()
> How'd you like to be able to tell users "If a file gets corrupted, it's
> either a bug in OpenOffice or your hardware eats data."
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.4 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
> iQIVAwUBQTe9pngHNmZLgCUhAQLTZQ/5AZq31sPKOWB/eI4WtVhM76V2+pmQdfNk
> aZkPtwhZlVrGDJscp45OVNthwMiSskD07om4obUMac79BCmA5805clZ7i0X1uzQB
> SecyX/7XITWz3iwF4VIaxJgDoo7YPFfbpWS0NTmhDJPt6h5Z+goEccLdTABj5kVk
> gbex4xNgXzPrnopvjnWx+K2K3ydvXiX+8bWV200F9S+j0uCObcfSZ357uQPqlReP
> gKQhEs0pCcoQQua81rXJfUhP8hCbegCpUmhJNI0MwoTUvHTnBU/+99+cCsTLR4gN
> 1XlbS9SIaaBXG7/AV9L7iQO5GF1t8mTMSe163EOzZ7ypLDb2km5e7zZ7t2DrA6SE
> Jgd0xT3cHjDgp1b3qhdeWYvt0QVfz5CGaiHukkT4pEOpsrM09BOhDhY72B74swuV
> byDIJ+y4X3J08i/a0zt15hBlVz1FUq7ac7doZZDt3orj93t3B/y4Xpb7tgzVGoWP
> T5AkR+uS5dGRUoMSI/btnPIK4ERlhbwFzCB5lgkas5kjp4dnawIKAk1wxPdhvEle
> g3SsNqSfBsA9hs6tzVw341GaWj6AWV7oOfg3j0c9I+7nt8SX5UeuTCwBWtCuwd/H
> +nnBulusiMnAm3tzQxq8aMpSVAqsZDywjgXSler1mNik4Ow5I5JdXl+Bf2ZWHF+G
> VHk4JVxMt/c=
> =3z3Z
> -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Martin J. Bligh wrote:
|>For 30 years nothing much has happened in Unix filesystem semantics
|>because of sheer cowardice
|
|
| Or because it works fine, and isn't broken.
Ok, maybe it wasn't cowardice. Maybe it was laziness.
Because cowardice tells me that data corruption is bad. Data corruption
kills kittens. End data corruption now!
Unfortunately, the only way to allow files to survive a crash is to
implement transactions. The simplest way is to do it in the filesystem
and to export an interface to userland. This is what Hans is doing,
though AFAIK it's not done yet.
Laziness tells me that I should just use Windows anyway, because "it
works". Switching to Linux requires training, porting apps requires
work, and using Linux instead of Windows is a hell of a lot more work
than just patching programs to do something like
+ start_transaction()
~ write()
~ blah()
~ write()
+ end_transaction()
How'd you like to be able to tell users "If a file gets corrupted, it's
either a bug in OpenOffice or your hardware eats data."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTe9pngHNmZLgCUhAQLTZQ/5AZq31sPKOWB/eI4WtVhM76V2+pmQdfNk
aZkPtwhZlVrGDJscp45OVNthwMiSskD07om4obUMac79BCmA5805clZ7i0X1uzQB
SecyX/7XITWz3iwF4VIaxJgDoo7YPFfbpWS0NTmhDJPt6h5Z+goEccLdTABj5kVk
gbex4xNgXzPrnopvjnWx+K2K3ydvXiX+8bWV200F9S+j0uCObcfSZ357uQPqlReP
gKQhEs0pCcoQQua81rXJfUhP8hCbegCpUmhJNI0MwoTUvHTnBU/+99+cCsTLR4gN
1XlbS9SIaaBXG7/AV9L7iQO5GF1t8mTMSe163EOzZ7ypLDb2km5e7zZ7t2DrA6SE
Jgd0xT3cHjDgp1b3qhdeWYvt0QVfz5CGaiHukkT4pEOpsrM09BOhDhY72B74swuV
byDIJ+y4X3J08i/a0zt15hBlVz1FUq7ac7doZZDt3orj93t3B/y4Xpb7tgzVGoWP
T5AkR+uS5dGRUoMSI/btnPIK4ERlhbwFzCB5lgkas5kjp4dnawIKAk1wxPdhvEle
g3SsNqSfBsA9hs6tzVw341GaWj6AWV7oOfg3j0c9I+7nt8SX5UeuTCwBWtCuwd/H
+nnBulusiMnAm3tzQxq8aMpSVAqsZDywjgXSler1mNik4Ow5I5JdXl+Bf2ZWHF+G
VHk4JVxMt/c=
=3z3Z
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Steve Bergman wrote:
[...]
| In following this thread, I may be missing huge chunks of concept.
You are, but don't worry. You aren't the only one. (Before I get
assaulted for this, let me say that there are people on this list
smarter than me and I miss concept chunks quite frequently.)
| 1. The file as directory thing adds complexity that the administrator
| has to deal with. Symlinks are useful, but it's still aggravating to
| tar off a directory structure, take it somewhere, and then realize that
| all you have is links to something not in the archive because you didn't
| get your tar switches just right. Now we're talking about adding
| another set of "files which are not really files" to the semantics.
| More complexity. I'll take simplicity over some ivory tower ideal of
| "unified name space" any day.
You want simplicity? How does this strike you:
scp file/serialize daemon@bsd:~/file.img
You don't have to worry about "file is not a file". You don't have to
worry about fifos or device nodes. You don't have to worry about files
in files. In fact, you don't have to worry about any new kind of file
added, ever.
| 2. The use of multiple streams within files by Linux apps would make
| Linux as cross-platform unfriendly as MS is trying to be. Say these
The use of ext3 as a filesystem isn't cross-platform. Every disk-write
is platform-specific! We should all be using captive-ntfs instead!
Seriously, this isn't hard -- it's called a "library". And anything as
bloated as OO.org has got to already be modular enough to add this sort
of thing.
| features start getting used and you copy an OO.org document from a Linux
| box to a BSD box. It's broken. Of course, OO.org wouldn't use the
| streams in the first place because it would destroy their cross platform
If that was so, OO.org wouldn't use a GUI, because that isn't
cross-platform. They'd use a purely commandline interface, because that
is universal.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTe8PHgHNmZLgCUhAQKiNQ//Zo6BLeGK6/GBbYzcgyp51Ra4fRf2xoVo
0aOve2vinVl+RRzFGWSa66p+/rHd339BhxTPYhlLkm9SWbU9Et80J0R/bySYtc17
RmUHhmukSEXhJx+py7Vc99jXr03zr2fNaA3UBIfpMiAlj1UB2yOls5iwJgqfPVnX
+YwYcVRfZXc3KWqH9SGcY9q+NX38wXoQh3rbeJ0OEzaIxnvPWlMvUFEN4OF92cub
on+n0h+JKptwtXOhLh9HaKAzG9fDhy4tJwmN3GIemSzKpEZ0ZGS4+Pz4dcnRXP+S
C6cMxpsmCnm5WVzmeoyBYBY28Fmnji00ShvxDGRFYOp48RPFhNaoMbVhLkQFoYfX
a5k3iedhbjaWOLmpfNGaDNdpKFtqD3LJO1xOSkXz3sndAiLW9ArKhuu2+63ASAQy
wlFknP09k13UHzD2iL5kPUPzy0A/BLqZPSpX4tXnF0xcSp4X8RoLnyu8BmrqaTjG
fi7FkEsoVEB7BWslNNtKrH596sDhWCP+BQdkAYqOBh0fzSIJ+0uyo2nJRw3d8FEi
/ynNVUqZEN8Xj08WHI+ucwUmRpTnh6spwXZC42K2tokAQa+95nSTegx3uNkypcGU
l12s+BtIHFB25BKDaBReyGoBqUj0sdFvoHVkRNc/3DNY6MfRHV3rdgjz5NieT4gH
uwMwLNstv8c=
=O3r8
-----END PGP SIGNATURE-----
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> Oliver Neukum wrote:
> | Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
> |
|>> Btw, version control for ordinary files would be a great feature. I
|>> think something like it is available through Windows 2000/3 server.
|>> Isn't it called "Shadow Copies". It works over network shares. :)
|>>
|>> It allows you to restore previous versions of the file even if you
|>> delete or overwrite it.
> |
> |
> | There's no need to do that in kernel, unless you want to be able
> | to force it unto users.
> And on apps. Should I teach OpenOffice.org to do version control?
> Seems a lot easier to just do it in the kernel, and teach everything to
> do version control in one fell swoop.
Do you mean in the kernel or as a filesystem/VFS plugin that would
extend the functionality to include version control?
~S
On Thu, 02 Sep 2004 19:43:34 CDT, David Masover said:
> And on apps. Should I teach OpenOffice.org to do version control?
> Seems a lot easier to just do it in the kernel, and teach everything to
> do version control in one fell swoop.
Including files you didn't really want to keep version control of?
How many temp files does gcc create and unlink in the course of a kernel build?
(And remember, you can't say "don't enable that on /tmp" - gcc respects the
setting of $TMPDIR - so an 'export TMPDIR=~/tmp' confuses things quite
nicely...)
And it's hard for the kernel to know that an unlink() done by gcc should be
treated differently than the "recover the last version" you *want* it do be able
to do after you work on a source file for a long while, save it, and then
fumble-finger a 'rm * .o' - you can't even use a heuristic like "don't version
control it unless it's N seconds or more old"
(Note that the "obvious" solution of creating a chattr flag has its own
complexity issues - should versioning be turned on by default for some types
and not others, etc...)
There be dragons here - it's not as simple as "drop in a plugin and be happy".
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Spam wrote:
[...]
|> I thought reiser4 had its journaling and atomic commits. Am I
|> mistaken? I run reiser4 as primary fs on my test systems and it seem
|> to work as expected.
consider this:
save_file () {
write() /* what if the write flushes halfway through
* then crashes?
*/
blah() /* what if "blah" crashes? */
write()
}
Some apps need consistency across multiple files, but we don't even have
it on a single file. You need a new interface to do that. As you can
see, reiser4 has absolutely no way of knowing, anywhere in the above
code, when you're done writing -- and when the file is consistent.
AFAIK, all that has to be done now for this to work is for them to
finish the userland interface to the journalling and atomic commits that
already exist for kernel space. But so far, all that is truly atomic is
metadata operations -- chmod, mv, mkdir, touch, and rm/rmdir are all
atomic, so long as you only use them on a single file/dir. But this has
been true in reiserfs3, xfs, ext3, and others.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTfKF3gHNmZLgCUhAQKp3Q//WcijAx8VVdKjJFHENZvrA2P3/wjdCLIn
OeIGi9VxbcQhS1cBM+BU6nBN6wz8S59o9hEM/F8L5oDxto8YW6sfyceoanbtVkLo
r7k4i39A4Ao1UQFzYEzxWzgCA607WvGBboNN1J/5pYe+bWATWBw+oQgc6XikQofQ
QwFX+0KxEDUva5Xkei/wBNhvpSveFBjAm/t79nVkANOVLji2KYALtewps+7QDiGc
PNwfjSUcAbNgXrvuhwpocFgAGEhgS/Y7ANR4DqFAiUseDwY6WgkZqhTaxC8t9ZzF
/Hd82sEb/J+hGpMfrTIe+D5VYhnjX+bPPs/U0OcfoH4CfkgOFgGOeuzlWg3xeTyX
vgtgYGwuGQuWux7WNte0GOCNxfaFDE8lLGexdQYLkdVCSsLmzACA17H1bQdeswgH
UnWL3oBoBx5BcdpkW1y1ZeoJv4xInazkzeU3GtEOd50tlJmOXmdjRNfj5LTrTh+1
CUpEKVvSjQ8MY70uQW6XKEUAkCYyBmSVeMay5SSQS/1Q0ISLEG3gMqPdsL2BOBO/
LkkQKjLe3LQS913wUtibL1CCfJCS/BrDswNCMsboivcg1SnwY+xJb9ffFMW0blBL
kQ0B1AGZCjWHDfxXKIlC31PJYKzgE5vaKZ6+INj8mG8t5PzqgikBDQStvsXqczJq
VgvvOVCjvho=
=I+Xw
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
[email protected] wrote:
| On Thu, 02 Sep 2004 19:43:34 CDT, David Masover said:
|
|
|>And on apps. Should I teach OpenOffice.org to do version control?
|>Seems a lot easier to just do it in the kernel, and teach everything to
|>do version control in one fell swoop.
|
|
| Including files you didn't really want to keep version control of?
No one said you have to apply it to everything. Only that you can.
Right now vim cannot do version control, AFAIK. But you could do
something like:
echo 1 > foo/version_control
vim foo
Default would be off, of course.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTfLR3gHNmZLgCUhAQKybRAAkF4sqLUCJEfpDNvKEyv8MJGcy0w+qV12
3Rf50OQeeurZMZrnli0vNIuSIWwIsGI+j7qbj1mlfi+Ps3dAjOlKXav3tGoGshLl
HH/Wl9XvGPOAWlIQOqqBGAyno8sDIvHAjh5lH3C7m+9/2Ao7s1G/bQ+hm/deYPJK
WPZxExd/1BdchrbjNjbfWHKrD6LdF/GBa/Vbj1F8m95WEvxsfbDnSTwb5HhvcUxD
TtqwkINmx7Tle2p9q7PcgR7y76dmcyoWw82ST3BfF/AbGauMQAr0h6jNjvzclfgJ
yrIOwGXdR3pn8ZK8dHVct79N0f+PYFdeAzLYxMZR2vefB2enQolbbs6Lf7X60g2g
5hrH/ezenFJIZMPhQtlJqBaFuA5GHw9B2An6VbOmDm7sSOFyHSlKoCUFlJ/I5nZd
Bcn46lhNxvTJhu3tvnvDqTB8f0/yVYt5QfoBFA1mzHKig2iNN6vr2xLGdWSLU6XD
vRjy+KyVC+PvjL13E0JrjXy9UuqY7CH2xAunSgLJ2cvfRmNRoVoV1hkp3McQcUqF
GiD3s0REcztlvsrP+DxKLRtPpnOJi5By5NmZpJkiPdVB9zpIsL/Ia7KL/A7IGGh3
IF3t/xSUmjgrek01SeadHqsIngeWQ+F8fiars7NtzNhrvRgPVfDVheey3vhQjFxM
3kDKH+I9rGk=
=3+nq
-----END PGP SIGNATURE-----
On Thu, 02 Sep 2004 20:11:13 CDT, David Masover said:
> It'd be like writing OpenGL entirely in software, before hardware
> accelerators work, and at the last minute have to change the library to
> use triangles instead of splines.
I expect that SGI did a software-only version of IrisGL first, so they could
figure out what the hardware accelerators needed to support. And even then,
the API for IrisGL got modified when it became OpenGL.....
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Spam wrote:
[...]
| Do you mean in the kernel or as a filesystem/VFS plugin that would
| extend the functionality to include version control?
The filesystem plugin. But this needs some kernel support, which scares
people. Read the archives if you don't believe me.
Besides, we'd want per-block copy-on-write support anyway to make it
sane. That has to go in the filesystem.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTfKhXgHNmZLgCUhAQIR5w//Z6hg58uVvIJuwNCb764lFj22iHwLvg0T
JvjQeOSCz7p2/uv+dHN2MRNDeezmvGfH49Y7RE78Q9e1xrfZFGZKyybHVisfea1t
f9Bk8M6ZtykKnUdPhq5MsDH9Oo98dXMNEWhDw5WUYMzvglitMUcd4np3gGEtr539
yKHuxZhqjc/lQ9rd6PFKJYab1m9RAabQnGMHRIvx0T98iW3EI9pyOd0u1oUVCiRi
RxllyP0KCg7wxFv93b4zYHsv5BO8E7MWVI4aKuBUCqrc8ObJnDLDbO/8FfMvDt/1
Do9nlikFh4IaRvwZtmQA0nVv80uxtjnSOCs7F2r+qqNyprcHIdj2m+3x5zE8Tbfz
YCkI5BopHub94yBvhlcFGiKBtigLDsH3zHc66dVDl4SreJPKo+yirjjZ+RrU3HN1
iK+aFp5GxapcYDkMNOS0ab9TPejqLVyJ+Ci5mSNckcMARC9v+2mT7JTCBupyXZar
LSLFYef+W7ipKq5ZMurDB17I5y7VXfwOTd+oSwQdMwlsQlJs/AFOOGdJywtLGLy1
6jhejQlziPLWB9hz9wFAjWyHPrS0XsCNbtGbBESIVmqvLsZmpcAULpMn7TGc8aJg
0r79ODKlGK8lYJRIquRhZzd51oqiBfFJcX969XGPMPUBX4x3GZjaOlj79OQ7Ih3s
Qd8PSfUVzGU=
=Cw9B
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Oliver Neukum wrote:
| Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
|
|> Btw, version control for ordinary files would be a great feature. I
|> think something like it is available through Windows 2000/3 server.
|> Isn't it called "Shadow Copies". It works over network shares. :)
|>
|> It allows you to restore previous versions of the file even if you
|> delete or overwrite it.
|
|
| There's no need to do that in kernel, unless you want to be able
| to force it unto users.
And on apps. Should I teach OpenOffice.org to do version control?
Seems a lot easier to just do it in the kernel, and teach everything to
do version control in one fell swoop.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTe+NXgHNmZLgCUhAQLW+RAAnoQIQLXSd2vQudOKAaLgXXCxoa+gEWIq
zcNd+bIWPuQxwcZYlqbs/pDFiDky3XqLRaZv8PZx++rs9mMnv36eAtMX7uUp3egz
MxymeDDI796wtsDDBGcZtO+WwmqJ0fPgQMN/Q237s2yccPGJoDIVle9mzZAfNIdO
RDdLq142D+0t1iqfiVqaA+NgkArrGhcr0Hs/3TdviJX30gPL/Jm9eOvc3H3NgIQu
XToDQB1UnLoyiIwc16IY4ZxYoxf1YnFtrivZb+YgC61mIQLeqJMjixCAtBVzAfKN
2GmlsnfWcMCgXSEX8imkoNSLcmkv886+esGXZVzBbY1/Qg/f5MCXZilbb8e6+4I9
U4ReEUg5fGJW8JyKAXqKGyvbC/lim55Vgcjp/J8mrhUa3Q3cwPKoi2GKNRcbdpT/
pPB2SADA1eNrmFyGB7kLcqt0F9i4fIyTrItpHq9+m72OdFEze2OnWn+eIY8dRut7
SaGTq6ZEYtkH8oWT9xSXp1d2TUxeSVTdx8EffIIdwXAVOO0Z9xd71MJPLMp3VOMH
Qld4gjYjKFzwj3QlcWcsiVhk6Zq2fmqoOCQNfxhBzH1Le3v0F0ugammkQUXtRAub
dKEWxGfkDGOwviJXKeckDNn98x88vh7C/7gxpAOIcVSRDhEV+MDWIfMMvotE/lxe
393hENsXOgQ=
=g+3J
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Linus Torvalds wrote:
[...]
| It's much saner (from _any_ app standpoint) to roll their own database in
| user space - that way it just works.
Then why do we have apps which use Windows' Internet Options where it
can, and otherwise ask the user for proxy settings?
|
| In other words, nobody is really ever going to take advantage of this.
| This is _not_ how technical advanncement happens. The way you get people
| to take advantage of something is to have a nice gradual ramp-up, not a
| sudden new feature that they can't realistically use.
|
| So before you do transactions in the filesystem, you have to be able
to do
| them in user space - and then make the filesystem version be _compatible_
| with that user space library. That way you can get people to use the
| library ("hey, it works anywhere") and then you can get them to use your
| filesystem ("hey, if you use our super-duper filesystem, you can do what
| you are already doing five times faster").
Why not:
do transactions in the filesystem
write a library to talk to that filesystem
have the library do emulation on other systems
in that order.
| See? You're starting at the wrong end. Give me a portable interface to
use
| _first_. Then do transactions in the filesystem.
If you're going to have to do both the portable interface and the fs
support, does it really matter which order you do them in?
Again, this is curiosity, not sarcasm. Here my reasons for wanting the
fs support first:
You don't always know exactly how the filesystem transactions will work.
~ You don't know until it's done whether you'll deviate from your design,
cut certain features, replace them with others...
And if you write the portable library first, and it ends up supporting
features a, b, and c, while the kernel supports features b, c, and d,
which obsoletes a, you have to add some features and remove some
features from the portable library to make it sane.
It'd be like writing OpenGL entirely in software, before hardware
accelerators work, and at the last minute have to change the library to
use triangles instead of splines. What's more, every single app now has
to be rewritten or use the library's emulation -- that's assuming many
apps would consider using a 3D library that doesn't have hardware accel
support yet.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTfEsHgHNmZLgCUhAQK1ORAAiSQ6UHSYc/PPMMOv2VjTkSuK2MgPylKP
d2jEitIuvWx01xAa++CLcUS6ncaiLuFYDA7TTHjhKYKAj+zuNN9OTI9EMGc6D+Ua
lFcbcRFapqbT9HPrUnGPM8VQIR8UrBt54hRAGm0/jOCc8DKLa3d6xnRfqmKQhnkv
w3ut4FiNo0GCd19hF82NxEA1YtwbOIgOIUxVlbADGI9QT/d+I3ZMA8QUNCrgrudn
e9g7o0yTa2c16t/rrqkiYCqSHIhCgY8ZH8VQzG4uYSjU+qYS+QbpJakD82EyuF/k
4T4jyJOqqSv6tLwE1j2y0gteXDYOdmfda/TpDeBSaRvGf+SmNxfSDFL448B2QJgA
e3V0HZHVVums1Vl6kX4pBaWM01V3n9BGeLaF2S9XSP7WCSWgW7t4ONRSLE3Bf2mm
1e5cMJ63mrtImb8Jyd8GhyCQ8/tkGnEYOIAHL3laGOjuc2ARBeyqmWrGKJsQ8iwu
i9NsyukQC3NP2FY1nBSmKs9J2bWtixqk5OXpzupkqBkbpeKL1H/b9Mxpnn5+Aomi
PjI/5i4QcaHMNBggxCG3B5zGJIWmvMkmZmHrHxvbLeENGvS5fiZ0Kscq7PL0FnE0
/2etDgjg65I66SqguXMzvtZdQ0hYgdwvoR83nZymHMSLYrFJ8FOxuB7kYQBeKZnu
QpUz0UDTUZA=
=jVvF
-----END PGP SIGNATURE-----
>> 1. The file as directory thing adds complexity that the administrator
>> has to deal with. Symlinks are useful, but it's still aggravating to
>> tar off a directory structure, take it somewhere, and then realize that
>> all you have is links to something not in the archive because you didn't
>> get your tar switches just right. Now we're talking about adding
>> another set of "files which are not really files" to the semantics.
>> More complexity. I'll take simplicity over some ivory tower ideal of
>> "unified name space" any day.
>
> Are you afraid to learn something new? ;) Just joking. But really,
> it doesn't have to be very difficult. The extra streams etc would
> just be saved as files. If tar is patched then it would be no
> problem and no extra stuff but perhaps a switch --save-metas.
If they're saved as files, and the app has to be changed to use them
anyway, then what's the point? Just change the app to use new files
instead.
> I would agree with tunnel vision. The kernel should provide the
> tools and options. Users and developers can then invent new things
> to use them. :)
Ugh. Change for changes sake is not a good thing. There's enough real
problems in the world without inventing random features. More complexity
without gain is a Bad Thing (tm).
I'm not saying streams is bad ... just that there don't seem to have
been very many convincing (to me) arguments raised for it yet. The
versioning stuff would be nice, IMHO, because the stuff mainly using it
wouldn't need to be modified in many instances ... "vi /etc/configfile"
would keep old copies for you (only the recovery tool would need to
understand it). Saving icons seems easy enough to do with "foo.icon"
as another file.
M.
> On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote:
>> > For 30 years nothing much has happened in Unix filesystem semantics
>> > because of sheer cowardice
>>
>> Or because it works fine, and isn't broken.
> OK. I'm not a kernel hacker. I'm not a crack C programmer. Nor am I a
> designer of filesystems. I'm just a guy that puts together and supports
> Linux systems for my customers.
> In following this thread, I may be missing huge chunks of concept.
> However, a few things are becoming clear to me:
> 1. The file as directory thing adds complexity that the administrator
> has to deal with. Symlinks are useful, but it's still aggravating to
> tar off a directory structure, take it somewhere, and then realize that
> all you have is links to something not in the archive because you didn't
> get your tar switches just right. Now we're talking about adding
> another set of "files which are not really files" to the semantics.
> More complexity. I'll take simplicity over some ivory tower ideal of
> "unified name space" any day.
Are you afraid to learn something new? ;) Just joking. But really,
it doesn't have to be very difficult. The extra streams etc would
just be saved as files. If tar is patched then it would be no
problem and no extra stuff but perhaps a switch --save-metas.
> 2. The use of multiple streams within files by Linux apps would make
> Linux as cross-platform unfriendly as MS is trying to be. Say these
> features start getting used and you copy an OO.org document from a Linux
> box to a BSD box. It's broken. Of course, OO.org wouldn't use the
> streams in the first place because it would destroy their cross platform
> portability. So what's the point? No one who cares about cross
> platform portability can use it. Everyone who doesn't care about cross
> platform portability please raise your hand.
Actually MacOS and Windows support multiple streams, only Linux
doesn't. But of course there are BSD's etc too. I'd say to leave
them behind.
The file streams would make my day a lot easier. The idea to
split up contents of OO.org files into streams is bad. But that
doesn't make the file streams bad. I see many uses that would make
my every day life easier.
It isn't about cross plat form compatibility, but to add features
that are useful and meta-data and file streams are.
Also. No one forces you to use either meta-data or streams, just as
no one forces you to use ACLs or other things.
> 3. MS does require attributes and multiple streams, which makes these
> features important (even essential) to Samba, and Samba alone. Samba is
> important to Linux, so this can't be ignored. (Here I am implicitly
> assuming that Samba will need kernel support for this to do it right.)
I do not think the Samba would really require the streams support
but it would certainly make life easier for Samba. Not to mention
that these files would also be natively viewed on the Linux host.
> So it seems to me that the only real consideration is giving Samba what
> it needs without making the semantics one bit more complex than
> absolutely necessary. It might even be wise to discourage use of these
> ambiguous new objects by the casual application programmer.
> Then again, maybe I just have tunnel vision...
I would agree with tunnel vision. The kernel should provide the
tools and options. Users and developers can then invent new things
to use them. :)
~S
> -Steve Bergman
>If tar is patched then it would be no
>problem and no extra stuff but perhaps a switch --save-metas.
Aaargh. How about a tar that recognizes when the filesystem has metas and
saves/restores them by default, with a warning when it's losings data because
it's restoring to a simpler filesystem?
And maybe a switch --forget-metas.
Thousands of sys-admins will thank you to not make it easier to lose data with
the scripts they have already written.
--
You are in a maze of twisty passages, all alike. Again.
http://www.hacksaw.org -- http://www.privatecircus.com -- KB1FVD
The reason for supporting transactions in the kernel is that there is a
whole slew of security bugs that have their origins in fs semantics
being non-transactional. rename does not cut it as a transactions API.
rename results in horribly performing apps that engage in contortions
that cost a lot of programmer time to code them to stuff a transaction
into a file that can be renamed. Most of the time programmers who
should use a transaction/rename are just too busy to do it, and they
take the risk that the user will lose data and be less secure rather
than write the code to do it right. When they do do it, because it is
complex code, they often screw it up.
That said, there is a whole slew of reasons for not supporting
transactions in the kernel:
* the history of the literature stars transactional filesystems that
paid too high a cost in performance
* isolation is very tricky to do without killing performance
* transactions which are kept open too long are a problem
* avoiding problems with locking shared data structures long enough to
cause painful exclusion of things that should be able to run in parallel
just fine
Reiser4, by being very modest in its ambitions, manages to not pay a
performance cost. We say, isolation and rollback are for user space to
manage until we can find a clever implementation, if such an
implementation can even be found. Even if such an implementation can be
found, there are plenty of cases where special case knowledge of what
the app is doing can improve performance a lot, and thus for that app it
will belong in user space.
All reiser4 does, is allow you to guarantee (using sys_reiser4) that a
limited size set of fs operations will be fused into one atom together,
and therefor will either all survive a crash or none will survive. This
does however meet a lot of apps needs.
Hans
David Masover wrote:
> Linus Torvalds wrote:
> [...]
> | It's much saner (from _any_ app standpoint) to roll their own
> database in
> | user space - that way it just works.
>
> Then why do we have apps which use Windows' Internet Options where it
> can, and otherwise ask the user for proxy settings?
>
> |
> | In other words, nobody is really ever going to take advantage of this.
> | This is _not_ how technical advanncement happens. The way you get people
> | to take advantage of something is to have a nice gradual ramp-up, not a
> | sudden new feature that they can't realistically use.
> |
> | So before you do transactions in the filesystem, you have to be able
> to do
> | them in user space - and then make the filesystem version be
> _compatible_
> | with that user space library. That way you can get people to use the
> | library ("hey, it works anywhere") and then you can get them to use your
> | filesystem ("hey, if you use our super-duper filesystem, you can do what
> | you are already doing five times faster").
>
> Why not:
> do transactions in the filesystem
> write a library to talk to that filesystem
> have the library do emulation on other systems
>
> in that order.
>
> | See? You're starting at the wrong end. Give me a portable interface to
> use
> | _first_. Then do transactions in the filesystem.
>
> If you're going to have to do both the portable interface and the fs
> support, does it really matter which order you do them in?
>
> Again, this is curiosity, not sarcasm. Here my reasons for wanting the
> fs support first:
>
> You don't always know exactly how the filesystem transactions will work.
> ~ You don't know until it's done whether you'll deviate from your design,
> cut certain features, replace them with others...
>
> And if you write the portable library first, and it ends up supporting
> features a, b, and c, while the kernel supports features b, c, and d,
> which obsoletes a, you have to add some features and remove some
> features from the portable library to make it sane.
>
> It'd be like writing OpenGL entirely in software, before hardware
> accelerators work, and at the last minute have to change the library to
> use triangles instead of splines. What's more, every single app now has
> to be rewritten or use the library's emulation -- that's assuming many
> apps would consider using a 3D library that doesn't have hardware accel
> support yet.
David Masover wrote:
> Spam wrote:
> [...]
> |> I thought reiser4 had its journaling and atomic commits. Am I
> |> mistaken? I run reiser4 as primary fs on my test systems and it seem
> |> to work as expected.
>
> consider this:
>
> save_file () {
> write() /* what if the write flushes halfway through
> * then crashes?
reiser4 does protect from this. reiserfs v3 does not.
> */
>
> blah() /* what if "blah" crashes? */
> write()
> }
>
> Some apps need consistency across multiple files, but we don't even have
> it on a single file. You need a new interface to do that. As you can
> see, reiser4 has absolutely no way of knowing, anywhere in the above
> code, when you're done writing -- and when the file is consistent.
>
> AFAIK, all that has to be done now for this to work is for them to
> finish the userland interface to the journalling and atomic commits that
> already exist for kernel space. But so far, all that is truly atomic is
> metadata operations -- chmod, mv, mkdir, touch, and rm/rmdir are all
> atomic, so long as you only use them on a single file/dir. But this has
> been true in reiserfs3, xfs, ext3, and others.
David Masover wrote:
>
> The use of ext3 as a filesystem isn't cross-platform. Every disk-write
> is platform-specific! We should all be using captive-ntfs instead!
;-)
All this stuff about how no filesystem should be allowed to have
semantic features others don't, it seems very Bolshevist to me.
Let Linux have an ecosystem with a diverse ecology of filesystems, and
the features that work will reproduce to other filesystems. I thought
that was the Linus way?
If not, why did I spend 10 years laying the storage layer groundwork for
semantic enhancements when I could have taken that job at Sun as
filesystems architect and made a lot more money?
I want to tinker. Let me play in my sandbox, and if you don't like what
I do, don't imitate it..... I think there are plenty of users who like
reiser4 though....
Linus, trying to outguess someone who has spent 2 decades studying
namespace design as to what will be useful to users is risky. Look at
reiser4's performance, see if it obsoletes V3, and if it does then let
me play a bit.
Objecting on the grounds that it causes VFS bugs is reasonable, but I
answered those questions and you did not respond (I can resend if
asked). If you really really don't like what we do to VFS, well, we can
confine ourselves to sys_reiser4(), but that is only a last resort from
my view.
Hans
On Thu, Sep 02, 2004 at 07:43:34PM -0500, David Masover wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Oliver Neukum wrote:
> | Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
> |
> |> Btw, version control for ordinary files would be a great feature. I
> |> think something like it is available through Windows 2000/3 server.
> |> Isn't it called "Shadow Copies". It works over network shares. :)
> |>
> |> It allows you to restore previous versions of the file even if you
> |> delete or overwrite it.
> |
> |
> | There's no need to do that in kernel, unless you want to be able
> | to force it unto users.
>
> And on apps. Should I teach OpenOffice.org to do version control?
> Seems a lot easier to just do it in the kernel, and teach everything to
> do version control in one fell swoop.
Just add a post-save trigger that can check it into any SCM you want.
Spam wrote:
>
> Btw, version control for ordinary files would be a great feature. I
> think something like it is available through Windows 2000/3 server.
> Isn't it called "Shadow Copies". It works over network shares. :)
>
> It allows you to restore previous versions of the file even if you
> delete or overwrite it.
>
>
Feel free to use cvs on your home directory. :-)
Helge Hafting
[email protected] wrote:
>On Thu, 02 Sep 2004 12:02:59 +0200, Spam said:
>
>
>
>> The meta-data should be deleted if it the file is copied or moved to
>> a medium that doesn't support it. However a _warning_ may be shown
>> to the user if there is risk to loose data.
>>
>>
>
>OK... I'll bite. How do you report the warning to the user if you're using
>an unenhanced utility to copy a file to a file system that may be lossy?
>
>
Well, _if_ you're using my sort of file-as-dir then you'd expect a
plain "cp" to only copy the file. You'd use cp -a to copy the
file's subdirectory too. If you're copying to anther filesystem
that doesn't support file-as-directory then "cp -a" won't be able
to create a directory with the same name as the file, and
will do whatever cp does in such a situation. I.e. something like
"NAME exists but is not a directory".
Helge Hafting
Oliver Hunt wrote:
>>Depends on how the forks eventually get implemented.
>>With the file-as-directory concept, all you need is to
>>look at the file's directory part to see what is there. (The forks,
>>implemented as files in a subdirectory.) It is done the same way
>>as for an ordinary directory, so nothing "new".
>>
>>
>>
>This still doesn't solve the problem of how we distinguish between a
>multi-fork file
>and a directory, for those old programs(ie. almost all that currently
>exist), to be able to access meaningful data we would need to either
>return just the primary fork, a serialized version of all forks in the
>file, or make the file look more or less identical to a directory.
>
>
If they open "filename" then they get what you calls the primary fork.
If they open "filename/someforkname" then they get some other fork, using
exactly the same mechanism as when they opens "directoryname/somefilename"
And if you want to use the old tools to look at some forks, just do a
$ cd filename
It will work if "filename" happens to be a file-as-dir thing, then do
$ ls
And you'll see the forks by name. Because the "forks" are merely
plain files in a subdirectory.
The only new thing needed is the ability for something to be both
file and directory at the same time. Some tools will need
a update - usually only because they blindly assume that a directory
isn't a file too, or that a file can't be a directory too. Remove the
mistaken
assumption and things will work because the underlying system
calls (chdir or open) _will_ work.
>The first option could cause problems when transfering files between
>different filesystems,
>the second results in programs getting metadata they can't handle, and
>the third option results in few of the advantages over, well,
>directories...
>
Few advantages over directories perhaps, but other implementatins
don't have more advantages over directories than this one.
Requiring another syscall to open forks other than the primary
breaks _all_ existing software because it obviously don't use that
syscall yet. And then it doesn't provide any advantages over the
file-as-plain-directory way. . .
>And even those applications that could handle the fork
>information nicely would need to fs type to find out whether they were
>handling a directory or a multi-forked file...
>
>
>
The distinction might not be important. A directory can be seen
as a multi-stream file without the main stream. And there's always
"stat" which should be able to tell what you're looking at.
>As I say I like the idea, but I can't see anyway of implementing it in
>a way that is useful without first putting considerable effort into at
>least the VFS if not all the actual fs drivers.
>
>
Sure, implementing it well will take effort, wich is why it is important
to find a good way of doing it so the effort won't be wasted.
Helge Hafting
Am Freitag, 3. September 2004 10:13 schrieb Christoph Hellwig:
> On Thu, Sep 02, 2004 at 07:43:34PM -0500, David Masover wrote:
> > | There's no need to do that in kernel, unless you want to be able
> > | to force it unto users.
> >
> > And on apps. Should I teach OpenOffice.org to do version control?
> > Seems a lot easier to just do it in the kernel, and teach everything to
> > do version control in one fell swoop.
>
> Just add a post-save trigger that can check it into any SCM you want.
That depends on whom you refer to. If you want to impose the RCS
on the users as an administrative measure and force compliance, then
you'll need to do it in kernel.
I see some nasty issues with disk quotas there.
Regards
Oliver
On Thu, 2004-09-02 at 22:06 +0200, Spam wrote:
[...]
> The file streams would make my day a lot easier. The idea to
> split up contents of OO.org files into streams is bad. But that
> doesn't make the file streams bad. I see many uses that would make
> my every day life easier.
When can we expect patches?
> It isn't about cross plat form compatibility, but to add features
> that are useful and meta-data and file streams are.
Putting everything (including web server and browser) in the kernel is
not a good (or even desireable) target - perhaps that's the reason for
Win* to be where there are.
And it *is* done in user-space (ok, there more than one apprach and
implementation but this may change. And you want to invent the next
version.).
> Also. No one forces you to use either meta-data or streams, just as
> no one forces you to use ACLs or other things.
Which is probably an argument against such features. How long do ACLs
exist - as well as standard and in implementation?
How many people and/or apps are using it?
No more questions.
> > 3. MS does require attributes and multiple streams, which makes these
> > features important (even essential) to Samba, and Samba alone. Samba is
> > important to Linux, so this can't be ignored. (Here I am implicitly
> > assuming that Samba will need kernel support for this to do it right.)
>
> I do not think the Samba would really require the streams support
> but it would certainly make life easier for Samba. Not to mention
> that these files would also be natively viewed on the Linux host.
It is not in the interest of MSFT that their aplication's files are
viewable on other OSes. Expect incompatible changes to formats, streams
and metadata contents (and surely patents on them - even if the idea is
not new) if it starts to be really useful. It happended already in the
past with the SMB protocol IIRC ....
Bernd
--
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and Services
> On Thu, 2004-09-02 at 22:06 +0200, Spam wrote:
> [...]
>> The file streams would make my day a lot easier. The idea to
>> split up contents of OO.org files into streams is bad. But that
>> doesn't make the file streams bad. I see many uses that would make
>> my every day life easier.
> When can we expect patches?
>> It isn't about cross plat form compatibility, but to add features
>> that are useful and meta-data and file streams are.
> Putting everything (including web server and browser) in the kernel is
> not a good (or even desireable) target - perhaps that's the reason for
> Win* to be where there are.
> And it *is* done in user-space (ok, there more than one apprach and
> implementation but this may change. And you want to invent the next
> version.).
As I said before. It doesn't matter if it is in the kernel or user
space. Not to me at least - as long as I could use extend the usage
of my filesystem with meta-data, my own descriptions, perhaps
streams, etc in a transparent way. I do not want to be confined to
only use future Gnome apps.
>> Also. No one forces you to use either meta-data or streams, just as
>> no one forces you to use ACLs or other things.
> Which is probably an argument against such features. How long do ACLs
> exist - as well as standard and in implementation?
> How many people and/or apps are using it?
> No more questions.
>> > 3. MS does require attributes and multiple streams, which makes these
>> > features important (even essential) to Samba, and Samba alone. Samba is
>> > important to Linux, so this can't be ignored. (Here I am implicitly
>> > assuming that Samba will need kernel support for this to do it right.)
>>
>> I do not think the Samba would really require the streams support
>> but it would certainly make life easier for Samba. Not to mention
>> that these files would also be natively viewed on the Linux host.
> It is not in the interest of MSFT that their aplication's files are
> viewable on other OSes. Expect incompatible changes to formats, streams
> and metadata contents (and surely patents on them - even if the idea is
> not new) if it starts to be really useful. It happended already in the
> past with the SMB protocol IIRC ....
No perhaps. But it doesn't make a difference to the ideas of having
plugins for reiser4 that can give such features.
~S
> Bernd
On Thu, 2004-09-02 at 14:38, Christer Weinigel wrote:
> "Martin J. Bligh" <[email protected]> writes:
>
> > > For 30 years nothing much has happened in Unix filesystem semantics
> > > because of sheer cowardice
> >
> > Or because it works fine, and isn't broken.
>
> I've heard the same argument a lot of times. People complaining that
> Unix is so seventies because it sticks to the old boring philosophy of
> everything is a file and that a file is a stream of bytes, nothing
> more. Modern operating systems such as VMS with basic database
> handling in the OS itself, or MacOS or NT with named streams is so
> much more modern. Why don't we get with the times?
Actually the handleing of special filesystem features in the OS is an
OLD idea. VMS is actually not more modern than Unix, it is older than
Unix owing much to earlier OS's such as RSX-11. It was a great insight
on the part of Ritche and Thompson that the kernel should present a file
as a byte stream and that the interpretation of the contents of a file
is the province of the application, not the operating system.
Now it maybe, that there are features that can only be provided by
adding support in the kernel, but we should be very careful not to be
distracted by what we see as new wizbang features. The filesystem as a
database has been tried before, ISAM is a very old concept.
>
> It may be because just because of the simplicity it's fairly easy to
> use, harder to break och does one thing well. If you want structured
> storage, use a database, on top of the low level primitives, or use
> multiple files in a directory. Why complicate things?
>
> /Christer
--
Brian Beattie LFS12947 | "Honor isn't about making the right choices.
[email protected] | It's about dealing with the consequences."
http://www.beattie-home.net | -- Midori Koto
On Fri, 2004-09-03 at 02:35, Hans Reiser wrote:
> All this stuff about how no filesystem should be allowed to have
> semantic features others don't, it seems very Bolshevist to me.
>
> Let Linux have an ecosystem with a diverse ecology of filesystems, and
> the features that work will reproduce to other filesystems. I thought
> that was the Linus way?
>
There's no reason you can't have semantics the other filesystems don't.
There's also no reason you shouldn't continue researching in any
direction you choose.
When you submit to the kernel (even just -mm), call the FS stable and
suggest people start broadly using it, you're telling them the parts you
have submitted are in fact fixed and reliable.
This implies you are done changing the semantics you have submitted,
which also implies you would like applications to start seriously using
it. Those application writers will want other filesystems to be able to
support these new semantics, which means they should have a race free
framework vetted by the kernel community and available for reuse by the
other filesystems. In other words, the semantics should be in the VFS,
without any requirement other FS's implement them, but the possibility
for every FS to implement them.
If you don't want the framework to be agreed on by the community and
available for other implementations, you either haven't stabilized the
semantics, or you don't want anyone else using them. Either way, it
precludes inclusion in the kernel. There are simply higher expectations
for a filesystem in these cases, and a higher burden for stable
semantics and interfaces.
This doesn't mean new semantics are impossible, it just means that
constantly changing semantics of the filesystem core are bad for users
and applications. They belong in patches that live outside the kernel
until they are sufficiently researched to justify their inclusion.
The biggest victim of this discussion is the non-semantic part of
reiser4. You've done a huge amount of research that is being held back
because you haven't separated the semantic research from the storage
layer areas. If the storage layers are ready now, please please just
submit them.
-chris
Salut,
On Thu, Sep 02, 2004 at 09:12:56PM +1200, Oliver Hunt wrote:
> getNumForks(fileref){ return 1;}
Actually it might be even cooler to have a generic function which you
can apply to any file on a file system (or whatever) and which returns
you the file system's capabilities (can write, has xattrs, has acl,
has multiple streams, *not* has btree lookups etc, as that is *not*
interesting to userland anyway.)
Tonnerre
Salut,
On Fri, Sep 03, 2004 at 10:22:55AM +0200, Helge Hafting wrote:
> Requiring another syscall to open forks other than the primary
> breaks _all_ existing software because it obviously don't use that
> syscall yet. And then it doesn't provide any advantages over the
> file-as-plain-directory way. . .
Actually...
We might tune the sys_open() call to take an additional argument (the
stream ID), and introduce a compatibility interface into *libc which
chooses strid=0 by default if the plain old open call is being used.
Maybe this can be handled transparently by cpp..
Tonnerre
> Salut,
> On Fri, Sep 03, 2004 at 10:22:55AM +0200, Helge Hafting wrote:
>> Requiring another syscall to open forks other than the primary
>> breaks _all_ existing software because it obviously don't use that
>> syscall yet. And then it doesn't provide any advantages over the
>> file-as-plain-directory way. . .
> Actually...
> We might tune the sys_open() call to take an additional argument (the
> stream ID), and introduce a compatibility interface into *libc which
> chooses strid=0 by default if the plain old open call is being used.
> Maybe this can be handled transparently by cpp..
How are things like copy handled today? Is it just some code linked
in during compilation (ie, the application handles the copy itself)
or is it a library function in libc? If it is in libc then it ought
to be enough if libc is patched to support the new semantics with
file-as-dir etc.
~S
> Tonnerre
Salut,
On Thu, Sep 02, 2004 at 08:50:10AM -0400, James Bruce wrote:
> filesystem to implement the Berkely DB interface (libdb) for example.
Problem is that BerkeleyDB changes its API all the time.
Tonnerre
Tonnerre wrote:
>Salut,
>
>On Fri, Sep 03, 2004 at 10:22:55AM +0200, Helge Hafting wrote:
>
>
>>Requiring another syscall to open forks other than the primary
>>breaks _all_ existing software because it obviously don't use that
>>syscall yet. And then it doesn't provide any advantages over the
>>file-as-plain-directory way. . .
>>
>>
>
>Actually...
>
>We might tune the sys_open() call to take an additional argument (the
>stream ID), and introduce a compatibility interface into *libc which
>chooses strid=0 by default if the plain old open call is being used.
>
>
But this isn't necessary, as I pointed out above.
There is no _need_ for a new kind of open(), so
why make one in libc?
If you can open a fork/substream/whatever by issuing
open("filename/forkname", ...
then the old-fashioned open() works with multi-fork files too.
An tools based on "open() something, then work with
the resulting file descriptor" will work _unchanged_
with such a multi-fork fs.
Tools that rely on "stat" or directory traversals might need
some updating to work perfectly with such a fs, but not something
that just opens by name.
Helge Hafting
Salut,
On Wed, Sep 08, 2004 at 11:24:40AM +0200, Helge Hafting wrote:
> If you can open a fork/substream/whatever by issuing
> open("filename/forkname", ...
> then the old-fashioned open() works with multi-fork files too.
> An tools based on "open() something, then work with
> the resulting file descriptor" will work _unchanged_
> with such a multi-fork fs.
In my version they'd run unchanged as well.
And BTW, I wasn't talking about introducing a new open at libc
level. I was talking about modifying the open system call in the
kernel and having libc provide compatibility for the old call.
Since I'm not sure how much breakage it takes to make a file a
directory.
Tonnerre
Tonnerre wrote:
>Salut,
>
>On Wed, Sep 08, 2004 at 11:24:40AM +0200, Helge Hafting wrote:
>
>
>>If you can open a fork/substream/whatever by issuing
>>open("filename/forkname", ...
>>then the old-fashioned open() works with multi-fork files too.
>>An tools based on "open() something, then work with
>>the resulting file descriptor" will work _unchanged_
>>with such a multi-fork fs.
>>
>>
>
>In my version they'd run unchanged as well.
>
>And BTW, I wasn't talking about introducing a new open at libc
>level. I was talking about modifying the open system call in the
>kernel and having libc provide compatibility for the old call.
>
>Since I'm not sure how much breakage it takes to make a file a
>directory.
>
I believe most of the changes necessary for supporting file-as-dir
will be in the internal workings of filesystems, not so much in
the interfaces. I don't think it is necessary to change the
open system call, because opening a file in a subdirectory
or a named substream is the same thing.
The _uses_ people have suggested for file-as-dir (complex file attributes
and different views of the data) tends to be different from what
a directory usually does, but the implementation doesn't have to be.
There's no need to add extra interfaces or restrictions. File-as-dir that
really is a directory, so that it may contain a whole tree is more flexible.
And the underlying fs can use existing directory code instead of
something extra.
So the only change needed for the open system call is that it will
suddenly succeed in the open("filename/forkname") case that
used to complain about an invalid path because "filename" isn't a
directory.
"stat" may have to change though, so it can identify a file-as-dir.
Helge Hafting
Hey, you know how device nodes have a bit set, indicating that they're
device nodes and not regular files? Can a set of such properties be
defined for reiser4 metadata properties?
Like a "metadata" bit so you can distinguish (if you wish) between
regular files and metadata objects, and in addition an "archivable
metadata" bit which indicates that the given piece of metadata is not
automatically generated and should be archived during backup (some
manually-generated metadata which does not need to be backed up will not
have this bit set -- perhaps add another flag indicating that it's not
automatic but unnecessary to archive).
Timothy Miller wrote:
>
> Hey, you know how device nodes have a bit set, indicating that they're
> device nodes and not regular files? Can a set of such properties be
> defined for reiser4 metadata properties?
>
> Like a "metadata" bit so you can distinguish (if you wish) between
> regular files and metadata objects, and in addition an "archivable
> metadata" bit which indicates that the given piece of metadata is not
> automatically generated and should be archived during backup (some
> manually-generated metadata which does not need to be backed up will
> not have this bit set -- perhaps add another flag indicating that it's
> not automatic but unnecessary to archive).
Interesting idea, particularly for autogenerated metadata. Some
metadata, like
thumbnails and icons might as well be ordinary files though. (Like some
icon and want to use it as background bitmap? Just copy it, because
it is a plain file. Or want to use it as clipart - fine, the word processor
supports bitmap files. . .)
This is a case where something is "metadata" only because it is used as
such,
not because it has to be. File-as-dir is an interesting concept for
such cases,
because it is convenient to group the "metadata" with the file, have it
move with the file and so on. But still have this metadata accessible as
a file because it isn't all that special.
Helge Hafting