Linus, you are looking at this like a lieutenant instead of an HQ
staffer, which is unusual of you.
You are saying, 1-2% simpler and better, no biggie, why work so hard to
get it?
And we are saying, 1-2% simpler and better, times thousands of
applications, wow! That's a lot!
Yes, changing cat to use openat() is no big deal. 1-2% additional coding
cost for cat, who cares? But if you add 1-2% coding cost to every
application which might access an attribute/stream/whatever, well, that
totals more than the effort of authoring emacs. Can you see that?
Namespace simplifications and empowerments are force multipliers. They
don't add to Linux like writing a new app adds to Linux, they add to
Linux like adding percentage improvements to every app in Linux adds to
Linux. HQ staffer types know that if you collect enough itty bitty
little force multipliers, you win the war. Whether the troops have to
spend five minutes a day polishing their shoes because their type of
shoe needs polishing, that matters more than losing a tank battalion,
when you are a major power. Linux is a major power.
This is a software engineering issue. We are discussing improvements
that because they are diffused throughout the OS in their impact, seem
like no big deal. But they are a big deal. One of the major determinants
of an organization's efficiency is whether the management can recognize
widely diffused inefficiencies as well as it can recognize focused
inefficiencies of the same magnitude.
David is so very right about the usability issues. Usability is all
about trivia. Usability battles feature no dragons, they feature armies
of spiders. If you add one step to what a user needs to do to get to the
data with an app, that matters a lot. That's doubling the spider army.
Making a 2 step process to access data into a one step process to access
the data halves the time cost of accessing the data. Lifestyle
efficiency is mostly about reducing the cost of trivia, because trivia
is most of what we do. Most people have as much interest in reading the
man page for tar as you have in learning how to turn the hand crank to
start your car. Just the look of "tar -xf" turns them away. Crypto-Geek
gobbledy-gook is what it is. Let's value their time, there are a lot of
them.
Hans
David Masover wrote:
>
>
> It's not about the kernel, it's about the interface. And see my other
> mail:
> cat foo.zip/README
> less foo.zip/contents/bar.c
> is a lot easier than
> lynx http://google.com/search?q=zip
> emerge zip
> man zip
> unzip foo.zip
> cat bar.c
> which already assumes quite a lot of expertise.
>
>
On Tue, 31 Aug 2004, Hans Reiser wrote:
>
> You are saying, 1-2% simpler and better, no biggie, why work so hard to
> get it?
>
> And we are saying, 1-2% simpler and better, times thousands of
> applications, wow! That's a lot!
But would thousands care? Seriously?
For example, you could make just _one_ program support "openat()", and
you'd get most of the advantages, with no possibility of _breaking_ any of
thousands of applications..
I know, you've ignored the "runat" program (which is just a wrapper around
the openat() system call), but it _has_ been mentioned several times in
this thread. Yes, you'd type a bit more to do
runat file ls -l
instead of
ls -l file/
but at least the openat/runat approach also works for directories, which
does actually make it a lot more _generic_ than the "show in the regular
filespace" approach. No special cases.
So your comparison isn't valid, because you're ignoring the people who
shout "runat" at you. You've also not ever actually answered about the
problems about directories with attributes.
Linus
Linus Torvalds wrote:
>On Tue, 31 Aug 2004, Hans Reiser wrote:
>
>
>>You are saying, 1-2% simpler and better, no biggie, why work so hard to
>>get it?
>>
>>And we are saying, 1-2% simpler and better, times thousands of
>>applications, wow! That's a lot!
>>
>>
>
>But would thousands care? Seriously?
>
>
>For example, you could make just _one_ program support "openat()", and
>you'd get most of the advantages, with no possibility of _breaking_ any of
>thousands of applications..
>
>I know, you've ignored the "runat" program (which is just a wrapper around
>the openat() system call), but it _has_ been mentioned several times in
>this thread. Yes, you'd type a bit more to do
>
> runat file ls -l
>
>instead of
>
> ls -l file/
>
>but at least the openat/runat approach also works for directories, which
>does actually make it a lot more _generic_ than the "show in the regular
>filespace" approach. No special cases.
>
>
So you are saying I can do runat emacs my-compound-document/stream-name?
and in place of CTRL-X CTRL-F to open a file in emacs I type what to
open an attribute?
I think it does not work Linus....
>So your comparison isn't valid, because you're ignoring the people who
>shout "runat" at you. You've also not ever actually answered about the
>problems about directories with attributes.
>
>
Which problem specifically? I will list them and hope I don't miss what
you want addressed. I won't claim to be sure I have the right answers
for all of these (because VFS has not been my interest in the past), but
these look feasible to me.;-)
Cycle detection:
We should either 1) make hard links only link to the file aspect of the
file-directory duality, and persons who want to link to the directory
aspect must use symlinks (best short term answer), or 2) ask Alexander
Smith to help us with applying his cycle detection algorithm and gain
the benefit of being able to hard link to directories (if it works well,
best long term answer).
Ambiguity:
Linus said:
- how do you handle the ambiguity of
//attr/usr/bin/emacs/icon
(is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the
"emacs/icon" attribute on "/usr/bin").
If you want to avoid crowding the namespace with some piece of meta
information that you want to be applicable to both directories and the
files in them, and also widely used enough that you care about crowding
the namespace, and unambiguous as to what level in the hierarchy it
metas, then you put it into the metas subdirectory. There is no
principled difference between a file that is a child of a directory and
an attribute. The ambiguity is inherent in hiearchical semantics I
think, and whether or not you have attributes you have this type of
ambiguity.
Thus, you have either:
/usr/bin/metas/emacs/icon
or /usr/bin/emacs/metas/icon
and these are easily distinguished and unambiguous. This makes metas a
reserved keyword, and there was a long discussion about whether to use
"metas" or "..metas" as the reserved subdirectory name which I can
summarize for you on request. There is also "pseudos", which contains
files implementing special methods of special plugins. Feel free to
invent more reserved subdirectories as needed, or to argue for fewer.
Seeing two trees, one with attrs and one without:
Linus said:
If you open a filename in some "secondary" tree (be it /proc or //attr or
whatever) based on the filename in the primary one, you have two issues
that you need to work out:
- how do you handle a name change in the primary tree at the same time as
lookup
- how do you handle the ambiguity of
//attr/usr/bin/emacs/icon
(is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the
"emacs/icon" attribute on "/usr/bin").
The ambiguity can be handled by saying that attributes only have one
component (ie only the _last_ component of a lookup is the attribute
name). But the race between primary tree and secondary tree cannot be
handled in a normal name-space.
Are you sure that you have to have two trees in order to see two trees?
That is, can you not refuse to follow or see a link depending on whether
some flag is set and what the type of the link is?
Locking:
Linus said:
let's say that you have filename "a" hard-linked to filename "b",
and you have a directory structure of streams under there. So you have
a/file1
a/dir1/file2
a/dir2/file3
and (through the hard-link with "b") you have aliases of all these same
names available as "b/file1", "b/dir1/file2" etc).
Now, imagine that you have two processes doing
mv a/dir1 a/dir2/newdir
and
mv b/dir2 b/dir1/newdir
at the same time. Both of them MUST NOT SUCCEED, for pretty obvious
reasons (you'd have moved two directories within each other, and now
neither would be accessible any more).
How do you handle locking for this situation?
(Here is where my knowledge of VFS is shakiest.) Making "hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks" seems to solve this. If we want to use the Alexander Smith solution to cycle detection, and allow hard links to directories, then we must make sure that for the hard linked entity, there is a single lock for it somewhere (e.g. the directory inode) that can be taken. If you create two dentries for the same object, then flag the dentries as needing to lock a lock in their shared inode and not in the dentry. If you don't want to read in the inode from disk to do that, then create a pointer to a lock shared by them both.
Linus, is this a start on solving some of the issues?
Hans Reiser wrote on Wed, 01 Sep 2004 00:22:49 -0700:
> Making "hard links only link to the file aspect of the file-directory
> duality, and persons who want to link to the directory aspect must use
> symlinks" seems to solve this. If we want to use the Alexander Smith
> solution to cycle detection, and allow hard links to directories, then
> we must make sure that for the hard linked entity, there is a single
> lock for it somewhere (e.g. the directory inode) that can be taken.
Oddly enough that's what I did in my RAM file system. No matter how
the file/directory node is found (the path can lead through different
parents), it has a unique ID and a corresponding single lock for it.
Rename/delete goes and locks all the nodes that might change parents
and/or children, then it does its operations (though it has to watch
out for deadlocks, which are unavoidable when two rename/deletes both
sneak up on the same file by different paths).
- Alex
On Mer, 2004-09-01 at 06:50, Hans Reiser wrote:
> Yes, changing cat to use openat() is no big deal. 1-2% additional coding
> cost for cat, who cares?
Large cost on its own. The FSF would almost certainly reject such a
change until someone had written portable emulation of the feature for
every other platform which would mean vendor patches which would
essentially mean it wouldn't happen.
I know I rarely agree with Hans but I think he's right on this one, it
has to work with the existing open() API.
Alan
On Tue, 31 Aug 2004, Hans Reiser wrote:
> learning how to turn the hand crank to start your car. Just the look of "tar
> -xf" turns them away. Crypto-Geek gobbledy-gook is what it is. Let's value
> their time, there are a lot of them.
These kinds of people would use something like file-roller in GNOME,
and they'd care not a jot whether file-rollers used built-in
tar support, and/or openat() or via file-as-dir fs support.
All they care about is that they get a window with icons for their
files that they can click on. How it's done does not matter - it only
matters to those who must write the code.
> Hans
regards,
--
Paul Jakma [email protected] [email protected] Key ID: 64A2FF6A
Fortune:
I haven't lost my mind; I know exactly where I left it.
Linus, did you get the email in which I (I think....) answered all the
problems with files as directories that you and Viro had raised?
You and Viro did not respond to it....
Hans
On Thu, Sep 02, 2004 at 11:06:39PM -0700, Hans Reiser wrote:
> Linus, did you get the email in which I (I think....) answered all the
> problems with files as directories that you and Viro had raised?
>
> You and Viro did not respond to it....
Message-ID, please. There was a lot of mails from you in this thread and
I probably have missed one that did address *any* problems. If such mail had
ever existed, that is.
[email protected] writes:
> On Thu, Sep 02, 2004 at 11:06:39PM -0700, Hans Reiser wrote:
>> Linus, did you get the email in which I (I think....) answered all
>> the problems with files as directories that you and Viro had
>> raised?
>>
>> You and Viro did not respond to it....
>
> Message-ID, please. There was a lot of mails from you in this
> thread and I probably have missed one that did address *any*
> problems. If such mail had ever existed, that is.
Perhaps this is one? Message-ID: <[email protected]>
On Wed, Sep 08, 2004 at 01:21:45PM +0530, Sriram Karra wrote:
> Perhaps this is one? Message-ID: <[email protected]>
OK...
One note before replying: current code deadlocks even if you make ->link()
*ALWAYS* return an error. It doesn't get to calling the method. No amount
of "disallow hard links to <something>" is going to help here, obviously.
<quote>
Cycle detection:
We should either 1) make hard links only link to the file aspect of the
file-directory duality, and persons who want to link to the directory
aspect must use symlinks (best short term answer), or 2) ask Alexander
Smith to help us with applying his cycle detection algorithm and gain
the benefit of being able to hard link to directories (if it works well,
best long term answer).
</quote>
... which doesn't address the problem at all. The question is what to do
with seeing directory "aspect..." in more than one place when we have many
links to file in question. So much for (1). And (2) is not feasible with
on-disk fs both due to memory, CPU and IO costs _and_ due to exclusion from
hell you'll need to make it safe.
Re: ambiguity - lots and lots of handwaving on both sides. FWIW, I agree
with Hans in one (and only one) respect here - openat() as a primary API
(and not a convenient libc function) is an atrocity. Simply because it
doesn't address operations beyond open (unlinkat(2), anyone?).
However, I still haven't seen any strong arguments for need of this "metas"
stuff _or_ the need to export mode/ownership as files, both for regular
files and for directories. Aside of "we can do that" [if we solve the
locking issues] and "xattrs are atrocious" [yes, they are; it doesn't make
alternative mechanism any better] there was nothing that even pretended to
be a technical reason.
Note that we also have fun issues with device nodes (Linus' "show partitions"
vs. "show metadata from hosting filesystem"), which makes it even more dubious.
We also have symlinks to deal with (do they have attributes? where should
that be exported?).
Reserved names have one more problem: to be useful, they'd have to be
hardcoded into applications. And that will create hell with use of
such applications on existing filesystems. Again, no feasible scheme
to deal with that in userland code had been proposed so far, AFAICS.
Locking: see above - links to regular files would create directories seen
in many places. With all related issues...
I'm probably not the first to suggest this idea, and it's probably not a
very good idea, but here's my idea anyhow:
You have a file "/usr/bin/emacs"
with a metadata property in the overlaid namespace
"/usr/bin/emacs/[[..]metas/]icon"
According to some, this could cause some confusion. Howabout instead:
You have a file "/usr/bin/emacs"
with a metadata property in a slightly separated namespace
"/metas/usr/bin/emacs/icon"
This has the advantage of still having the metadata in the filesystem
namespace but without the confusion of having files-as-directories,
ambiguity of filename, backup issues, etc. This is the reverse of
having the namespaces overlaid with a "/nometas" view which is separate.
Furthermore, you can split things further like this:
You have a file "/usr/bin/emacs"
with an automatically-generated metadata property that you don't want to
back up in "/autometas/usr/bin/emacs/modification_date"
and a manually generated metadata property that you MAY want to backup
in "/staticmetas/usr/bin/emacs/icon".
This is inelegant, I know. But if we do this, we can add the extra
features of reiser4 without confusing existing apps or having to modify
them to support the new functionality.
Furthermore, you can easily hide the extra features by not mounting the
meta top-level directories (assuming they're mounted like separate
filesystems, rather than just magically appearing there, which is okay too).
Timothy Miller wrote:
> I'm probably not the first to suggest this idea, and it's probably not
> a very good idea, but here's my idea anyhow:
>
> You have a file "/usr/bin/emacs"
> with a metadata property in the overlaid namespace
> "/usr/bin/emacs/[[..]metas/]icon"
>
> According to some, this could cause some confusion. Howabout instead:
>
> You have a file "/usr/bin/emacs"
> with a metadata property in a slightly separated namespace
> "/metas/usr/bin/emacs/icon"
>
And the problem with such a "solution" is
mv /usr/bin/emacs /usr/bin/old-emacs
Do this with an ordinary fs, and the /metas/usr/bin/emacs/icon
won't move with it. Now metas might might not be an ordinary fs,
so perhaps the move happens automatically there, but if so
it will be unexpected.
> This has the advantage of still having the metadata in the filesystem
> namespace but without the confusion of having files-as-directories,
> ambiguity of filename, backup issues, etc. This is the reverse of
> having the namespaces overlaid with a "/nometas" view which is separate.
>
> Furthermore, you can split things further like this:
>
> You have a file "/usr/bin/emacs"
> with an automatically-generated metadata property that you don't want
> to back up in "/autometas/usr/bin/emacs/modification_date"
> and a manually generated metadata property that you MAY want to backup
> in "/staticmetas/usr/bin/emacs/icon".
>
> This is inelegant, I know. But if we do this, we can add the extra
> features of reiser4 without confusing existing apps or having to
> modify them to support the new functionality.
>
> Furthermore, you can easily hide the extra features by not mounting
> the meta top-level directories (assuming they're mounted like separate
> filesystems, rather than just magically appearing there, which is okay
> too).
Having to go up to root and then down a similar but different path to
reach a
file's metadata seems very counterintuitive to me. And you have to
update all
tools to do this automatically, or it'll be hopeless to actually use.
Helge Hafting