Hi.
I just wondered about the current situation (respectively plans)
regarding XATTRs in NFS (I personally don't care much about the
version,... of course v4 would be nice, but v3 should do either).
I'm aware about the patches for v3 that James Morris has submitted some
time ago, but AFAICS these were never accepted/merged.
Asking him (off-list) he mentioned that any such work would need to take
place in NFS4.
I've seen some comments from the past, saying that it should never
happen since it would cause performance issues - but I guess one could
make it optional.
I have some use case where I need a network filesystem with XATTR
support providing the features of most standard filesystems (locking,
etc.).
And I'm sure there are plenty more use-cases.
So are there any plans to get this into NFS?
Cheers,
Chris.
btw: please keep me CCed.
On Oct 28, 2013, at 9:39 PM, Christoph Anton Mitterer <[email protected]> wrote:
> Trond,... since you ask for motivations pro NFS-XATTRs, I wondered what
> are the strong reasons against?
As I've already told you: you're asking for the addition of a feature which is inadequately defined, and is not documented in any standard.
> Of course someone has to do the work,... but that's not really and
> argument pro or con NFS-XATTRs, is it?
>
> The security problems with namespaces like trusted. or so can probably
> be solved quite easy, e.g. simply by not supporting or
> ignoring/rejecting them.
>
> You've mentioned caching issues,... could you elaborate a bit on that?
> How is that much different from caching any other file metadata NFS
> transfers?
The standard metadata such as ctime, mtime, size, .... are all part of an existing NFS caching model called close-to-open. We know how they change when the filesystem data changes.
How do I know when it is safe to cache your checksum xattr? I don't even know what it is, let alone what its relation is to the other filesystem objects.
Let's say client B modifies the file and updates the checksum. When client A opens the file, it will see that the data has changed, but how does it know that it also needs to update the xattr information?
Alternatively, let's say that client A reads the file and checksum data after client B has updated the file, but before it updates the checksum. What will cause client A to stop caching the stale checksum once client B has updated it?
Trond
On 10/25/2013 10:08 AM, J. Bruce Fields wrote:
> On Thu, Oct 24, 2013 at 07:22:30PM +0200, Christoph Anton Mitterer wrote:
>> On Thu, 2013-10-24 at 16:30 +0000, Myklebust, Trond wrote:
>>> Those programs need to recompute the checksum data anyway in order to
>>> verify and/or update it. Checksums that are computed by some third
>>> party application have exactly zero value for integrity checking.
>> No that's exactly the point,... the applications should _NOT_ set those
>> checksums, especially not automagically (since then you'd never notice
>> when just some application is buggy or writes/modifies when you don't
>> expect it to do so).
>> The idea is that there is on application (in my case it's just a
>> script), which sets the integrity data and verifies it.
>>
>> This works very well for e.g. large data archives, where you most of the
>> time (but not always) only read files, write new files or move around
>> existing ones - but only rarely modify existing file's contents.
>>
>>
>> I do this already like that on local filesystems, which works very
>> nicely with XATTRs... but now I want to move this on a central data
>> cluster (where clients connect to via NFS)... and here the problems
>> start... when I add new data to the archive (from the clients) I cannot
>> have XATTRs attached, nor can I verify them form the clients.
> Can you give any more details about your use case? (E.g. is there an
> article describing this system somewhere?)
>
> Just curious.--b.
>
I think that having xattrs in NFS would be very useful over time. A lot of user
space tools have been made aware of them, we are certainly lagging all (almost
all?) local file system here and it can cause a data loss when you copy a file
from a local file system to an NFS server.
It certainly violates the principle of least surprise that the xattrs are lost
when move through NFS!
Typical use cases I have seen include storing things like metadata that tracks
what application created the file, tags to let you know when the last time the
file was verified by a data integrity scrubber, tags that hold a checksum/crypto
has of the file contents along with the date of that signature.
Doing a file system service does not mean that we have to be personally
interested in all of the odd things that our users might use features for, but
at this point, xattrs are water under the bridge and NFS not supporting them
makes us look bad :)
Ric
On Oct 27, 2013, at 6:20 PM, Christoph Anton Mitterer <[email protected]> wrote:
> On Sun, 2013-10-27 at 18:41 +0000, Myklebust, Trond wrote:
>> I'm not going to start anything...
> So that's basically a: "XATTRs in NFS - never"?! :(
Your case for adding the feature seems to be "I have a couple of programs that use xattrs, so NFS should provide it.".
Well, that means at a minimum going to the IETF (possibly also POSIX) to convince them that the feature is worth adding, helping design the protocol to do so, and achieving consensus in a NFSv4.x spec, then doing the work designing and coding up a Linux implementation. Who is volunteering to do all that?
My statement above was that I'm not volunteering to do all this. You haven't convinced me that your programs cannot be modified to use other more portable solutions. You're not even generating the checksums when creating the data, so there is already an existing window during which corruption can occur.
Trond
On Sun, 2013-10-27 at 18:07 +0000, Myklebust, Trond wrote:
> On Oct 27, 2013, at 12:56 PM, Christoph Hellwig <[email protected]> wrote:
>
> > On Sun, Oct 27, 2013 at 12:31:46PM +0000, Myklebust, Trond wrote:
> >> The NFSv4 working group has no authority to do so. POSIX would be the right address.
> >
> > The complete lack of clue, authority or even defined semantics didn't
> > stop them from defining their ACL model. User extended attributes are
> > harmless compared to that, even if everyone has subtly different
> > flavours.
> >
>
> The user xattrs are in principle harmless. What about "trusted.*"?
>
> BTW: caching any xattrs is a problem. There is no close-to-open model that you can use and neither is there locking. This is already causing a problem for labeled nfs...
You could start without caching them.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
On Oct 24, 2013, at 5:01 PM, Christoph Anton Mitterer <[email protected]> wrote:
> On Thu, 2013-10-24 at 14:32 +0000, Myklebust, Trond wrote:
>> Linux xattrs are a rabid mess.
> Well... might be from a technical POV, but for users they're quite
> useful in some scenarios.
>
>
>> The whole "system" namespace is something that cannot and should not ever be exposed on a network.
>> The "trusted" and "user" namespaces just offer specialised storage. Why are they needed?
> Well what I do is attaching integrity information to files.
>
> You may say now that this is similar to what btrfs will provide
> anyway... but the problem with that is, that checksums are always
> updated when something in the system does valid changes to the file.
>
> What I however want is that I really manually have to set this, so that
> I notice "accidental" changes, e.g. by myself or by buggy software...
>
>
>> If the data needs to follow the file, then store it in the file. Why do you need the filesystem to manage that for you?
> ... and since this applies to arbitrary files, from text-files over
> pictures, videos to binaries,... it's neither possible to store this in
> the file, nor can I really track this with an database,... since
> literally any program that uses such files, from the picture editor to
> the file-manager would need to use such DB.
Those programs need to recompute the checksum data anyway in order to verify and/or update it. Checksums that are computed by some third party application have exactly zero value for integrity checking.
Trond
On Sun, 2013-10-27 at 18:41 +0000, Myklebust, Trond wrote:
> I'm not going to start anything...
So that's basically a: "XATTRs in NFS - never"?! :(
Cheers,
Chris.
On Oct 27, 2013, at 12:56 PM, Christoph Hellwig <[email protected]> wrote:
> On Sun, Oct 27, 2013 at 12:31:46PM +0000, Myklebust, Trond wrote:
>> The NFSv4 working group has no authority to do so. POSIX would be the right address.
>
> The complete lack of clue, authority or even defined semantics didn't
> stop them from defining their ACL model. User extended attributes are
> harmless compared to that, even if everyone has subtly different
> flavours.
>
The user xattrs are in principle harmless. What about "trusted.*"?
BTW: caching any xattrs is a problem. There is no close-to-open model that you can use and neither is there locking. This is already causing a problem for labeled nfs...
Trond
On Oct 24, 2013, at 4:16 PM, Simo Sorce <[email protected]>
wrote:
> On Thu, 2013-10-24 at 15:11 +0000, Myklebust, Trond wrote:
>> On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:
>>
>>> Because the filesystem can do that when multiple applications are
>>> involved without having to change them all to talk to each other and
>>> invent custom protocol all the time just to keep some additional
>>> metadata associated to a file..
>>>
>> It's still a custom protocol. The applications need to agree on a data
>> format and store it somewhere. The portable way to do this is to write
>> an application library that they can link to.
>
> Perhaps I was unclear, you are never going to see that custom library
> linked into the 'mv' command.
>
Why should my mv need to link into such a library?
> So your approach makes little sense if the object is to maintain data
> coherent when people need to handle files from random applications and
> scripts and general system maintenance.
>
See the earlier admonition: store data that needs to be kept together either in the same file, or in the same directory. Use a library when different applications need to access the same data.
> The data may be relevant only to a specific application.
>
> I am not saying you *have* to implement xattrs support, just saying that
> it is not a mere 'applications should synchronize data themselves'
> problem.
_portable_ applications do not use xattrs. They are a Linuxism that is not described by either POSIX or any other similar standard.
Trond
On Oct 27, 2013, at 8:14 PM, Christoph Anton Mitterer <[email protected]> wrote:
> On Sun, 2013-10-27 at 12:48 +0000, Myklebust, Trond wrote:
>> Let's get the basic discussion of motivation right first.
> Actually I might know another use case.
>
> dCache[0] which implements a (p)NFS 4.1 server and is heavily used
> within the high energy physics community, could benefit from XATTRs as
> well.
>
> Ever since, dCache has had a concept called PNFS tags (where the PNFS
> had nothing to do with parallel NFS, but is called so for other
> historical reasons), which are basically key/value pairs belonging to
> some file (well at least directory, I don't remember where PNFS tags
> exist for regular files).
>
> These tags are used to control several things, indirectly, e.g. to which
> data pool node a file may go, and via that whether it may be moved to
> tape, whether it is duplicated, etc. pp.
>
> Since there were no XATTRs (at least I guess that's the historical
> reason), they used special files for this having the name:
> ".(tag)(key)" and the content of the file is the value.
>
> So for a path,
> /data/foo/bar/
> one can set the key "sGroup" (aka storage group) to a value say
> "ATLAS" (which (very simplified) will tell dCache in the end that this
> is storage belonging to the ATLAS experiment) by creating a file
> echo "ATLAS" > '/data/foo/bar/.(tag)(sGroup)'
>
>
> Obviously having such special files isn't that nice, and having real
> XATTRs would be the better solution.
Ordinary groups can do the same.
Trond
On Fri, 2013-10-25 at 11:32 -0400, Chuck Lever wrote:
> Local file systems can support whatever they like, since they are
> Linux-only.
> NFS has to interoperate with other operating systems that may or may
> not support xattrs, or may have something that looks like an xattr,
> but really is subtly incompatible.
Well but that alone can apply to most other things that NFS does support
as well.
Take NFS4 ACLs,... not (fully) supported by I'd guess every Linux
filesystem (or was there some proposal for btrfs to get NFS4 ACLs in
addition?)
Or take very old filesystem who have no concept of different owners.
etc. pp.
In any case (as with Linux filesystems as well), I think XATTRs
shouldn't be enabled by default, if NFS was to support them... and if
someone activates it, he probably knows whether the backend fs does
support it or not.
> So we would like to see some very clear evidence that it's worth the
> effort. Until then, Linux distributors are free to implement NFS
> xattr support outside the standard NFS protocol (like the Solaris
> NFSv3 ACL protocol) and support this themselves, perhaps as proof that
> "if you build it, they will come."
This always bears the danger that it will become a de facto standard
that one cannot get rid of anymore.
Cheers,
Chris.
On Sun, 2013-10-27 at 09:56 -0700, Christoph Hellwig wrote:
> On Sun, Oct 27, 2013 at 12:31:46PM +0000, Myklebust, Trond wrote:
> > The NFSv4 working group has no authority to do so. POSIX would be the right address.
>
> The complete lack of clue, authority or even defined semantics didn't
> stop them from defining their ACL model. User extended attributes are
> harmless compared to that, even if everyone has subtly different
> flavours.
+1
--
Simo Sorce * Red Hat, Inc * New York
On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
> Then you end up with large directories and an extra name per inode that needs to
> be stored and extra lookups for each file when you do a whole file system crawl.
>
> Certainly not as easy as adding and xattrs with that information :)
And I think there's another reason why it wouldn't work...
Imagine I change my system to encode what should be XATTRs in hardlink
pseudo files...
If I have such pair locally e.g. on my ext4:
/foo/bar/actual/file
/meta/<SHA512 identifier>.2342348324
And now move/copy the file via the network to the archive, I'd have to
copy both files (which is really annoying), and I'd guess the inode
coupling would get los (and at least the name wouldn't fit anymore).
So the whole thing is IMHO not even a workaround.
Cheers,
Chris.
Hi Spencer,
Is it still possible to get on the agenda fir IETF 88, I just got approval
to travel. We can use about 15 minutes, or what ever is available to
discussion the future of named attributes in NFS. The two main questions
that we need answer are:
1. Do we need them? what applications use them and how.
2. Can we have a more simple model that handles just user attributes.
Input is welcome to help me make the case at the meeting.
Thanks, Marc.
From: Ric Wheeler <[email protected]>
To: Dr Fields James Bruce <[email protected]>,
Cc: "Myklebust, Trond" <[email protected]>, Christoph Anton
Mitterer <[email protected]>, Mailing List Linux NFS
<[email protected]>, Steve Dickson <[email protected]>
Date: 10/28/2013 11:32 AM
Subject: Re: XATTRs in NFS?
Sent by: [email protected]
On 10/28/2013 02:08 PM, Dr Fields James Bruce wrote:
> On Mon, Oct 28, 2013 at 02:00:58PM -0400, Ric Wheeler wrote:
>> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
>>> On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer
<[email protected]> wrote:
>>>
>>>> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>>>>> Then you end up with large directories and an extra name per inode
that needs to
>>>>> be stored and extra lookups for each file when you do a whole file
system crawl.
>>>>>
>>>>> Certainly not as easy as adding and xattrs with that information :)
>>>> And I think there's another reason why it wouldn't work...
>>>>
>>>> Imagine I change my system to encode what should be XATTRs in
hardlink
>>>> pseudo files...
>>>>
>>>> If I have such pair locally e.g. on my ext4:
>>>> /foo/bar/actual/file
>>>> /meta/<SHA512 identifier>.2342348324
>>>>
>>>> And now move/copy the file via the network to the archive, I'd have
to
>>>> copy both files (which is really annoying), and I'd guess the inode
>>>> coupling would get los (and at least the name wouldn't fit anymore).
>>>>
>>>> So the whole thing is IMHO not even a workaround.
>>> OK. So you're going to do XATTRs for us?
>>>
>>> Trond
>> Now that pNFS is perfect and labeled NFS has made it upstream, I
>> think that Steve D must be looking for something to keep him busy :)
> I agree with Trond that we first really need good evidence about exactly
> who wants this and why.
>
> --b.
For the user space xattrs, many applications store various types of
metadata.
Gluster for example heavily uses xattrs, other programs do things like
data
scrubbing (look for a long unchanged file, compute a has and store it as
an
xattr) or simply use it to annotate the file with the name of the program
that
created it. Think of it as file decorations or annotations.
Today, if we store files in NFS that have xattrs, we do in fact cause data
loss.
I can understand an answer of "this would be hard to do for NFS and need
to go
through IETF" but think that xattrs are well enough established in Linux
and
supported in the tool chain that it is way too late to question whether or
not
supporting them is a worth our time.
Ric
On Oct 27, 2013, at 8:27 PM, Christoph Anton Mitterer <[email protected]> wrote:
> On Mon, 2013-10-28 at 00:17 +0000, Myklebust, Trond wrote:
>> ...and if the checksums are any good, then all you need to do to
>> substitute a database is to realise that a good data checksum is
>> invariant under renames.
>
> Don't quite see what you mean...
>
> Sure the checksums stay the same, but consider you have many millions of
> files, and you moved them around and thus the paths in the DB are
> incorrect... verifying the files will become very much a pain in the
> a**, especially when multiple files don't verify anymore.
>
> Or what if you have many small similar files, where errors could lead to
> a checksum that was a correct one for another file,... when the paths
> are no longer valid you cannot know if this was a correct file or not.
If you have lots of small files, and you really do need to associate them uniquely with the checksum, then try something like:
ln <filename> /path/to/database/<SHA512 identifier>.<inode number>
Hard links and inode numbers are also generally invariant under 'mv'.
Trond
On Thu, 2013-10-24 at 16:30 +0000, Myklebust, Trond wrote:
> Those programs need to recompute the checksum data anyway in order to
> verify and/or update it. Checksums that are computed by some third
> party application have exactly zero value for integrity checking.
No that's exactly the point,... the applications should _NOT_ set those
checksums, especially not automagically (since then you'd never notice
when just some application is buggy or writes/modifies when you don't
expect it to do so).
The idea is that there is on application (in my case it's just a
script), which sets the integrity data and verifies it.
This works very well for e.g. large data archives, where you most of the
time (but not always) only read files, write new files or move around
existing ones - but only rarely modify existing file's contents.
I do this already like that on local filesystems, which works very
nicely with XATTRs... but now I want to move this on a central data
cluster (where clients connect to via NFS)... and here the problems
start... when I add new data to the archive (from the clients) I cannot
have XATTRs attached, nor can I verify them form the clients.
Cheers,
Chris.
Please don't trim the cc list.
On Sat, Oct 26, 2013 at 07:12:27PM +0200, Christoph Anton Mitterer wrote:
> On Fri, 2013-10-25 at 10:08 -0400, J. Bruce Fields wrote:
> > Can you give any more details about your use case? (E.g. is there an
> > article describing this system somewhere?)
> Okay let me try to explain the motivation more elaborately.
>
> The general idea is getting data integrity, i.e. being able to see
> whether some files are in a valid state.
>
> This is similar to what e.g. btrfs/zfs provide at (IIRC block/extent
> level) with checksuming.
> But a) btrfs is not yet fully production ready (IMHO) and zfs in Linux
> has it's (license) issues and more important b) the checksuming there
> happens always and automatically as soon as some valid filesystem
> operations occur on the files.
>
> So what you basically notice are errors on the physical medium (or at
> least on block layers below the filesystem).
>
>
> You won't notice many other cases of file "corruptions":
>
> - when you have programs which do subtly manipulate the files and you
> don't notice,...e.g. some image organisation programs store their
> meta-data crap in the Exif/XMP fields, even when you don't actively tell
> them to do so (and sometimes even when the files are read-only)
>
> - when there are e.g. kernel bugs (in some other places of the kernel)
> and you copy the files around. Some years ago I found a bug (not the
> solution) where single bit errors happened more or less randomly every
> 40-100 GB of writing data. In the end it was found to be a IOMMU related
> bug and a single one liner of flushing some buffers cleared the problem.
> Since such errors would happen already at earlier stages, when writing
> such files btrfs/zfs would simply write the checksums of the corrupted
> data.
Was this problem actually caught using checksums stored in xattrs, or
did the problem predate your use of xattrs?
> So the idea of my integrity data is, that I really manually say "now the
> data is in the state where I consider it to be consistent and I want to
> have checksums stored and attached to the files, for exactly that
> state", e.g. after I have read out some images from the SD card (perhaps
> even twice with the cache cleared and the results diffed) and placed in
> my archive.
> Afterwards I can regularly verify the whole archive and if at some stage
> corruptions as the above would have happened, I can simply take the
> respective files from backups.
How long have you been using this for? How many problems has it caught?
How often do you checksum or verify files, and how expensive is that?
--b.
>
>
> Obviously that cannot be stored *in* the files,... and a database
> solution fails since, it wouldn't track the changes when I move files
> around within the archive.
>
>
> So IMHO the best solution were XATTRs, which do work fine for that
> purpose... just the problem that I can't have it via NFS ;)
>
>
> Cheers,
> Chris.
On Thu, 24 Oct 2013 11:16:10 -0400
Simo Sorce <[email protected]> wrote:
> On Thu, 2013-10-24 at 15:11 +0000, Myklebust, Trond wrote:
> > On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:
> >
> > > Because the filesystem can do that when multiple applications are
> > > involved without having to change them all to talk to each other and
> > > invent custom protocol all the time just to keep some additional
> > > metadata associated to a file..
> > >
> > It's still a custom protocol. The applications need to agree on a data
> > format and store it somewhere. The portable way to do this is to write
> > an application library that they can link to.
>
> Perhaps I was unclear, you are never going to see that custom library
> linked into the 'mv' command.
>
> So your approach makes little sense if the object is to maintain data
> coherent when people need to handle files from random applications and
> scripts and general system maintenance.
>
> The data may be relevant only to a specific application.
>
> I am not saying you *have* to implement xattrs support, just saying that
> it is not a mere 'applications should synchronize data themselves'
> problem.
>
I think the real solution if people need this is to lead an effort to
put xattrs into the spec. I think there is still time to get new
features into v4.3 if someone wants to champion it...
--
Jeff Layton <[email protected]>
On Thu, 2013-10-24 at 11:23 -0400, Jeff Layton wrote:
> I think the real solution if people need this is to lead an effort to
I'm often reading this argument,... "if people need this", respectively
the argument "XATTRs/ACLs" aren't widely used so we're not going to
support them.
IMHO that's no really an argument.... the simple reason that this isn't
widely used, is likely that there are still many places that don't
support it.
All the major Linux (disk) filesystems support it, right? ext*/btrfs/XFS
But many (even standard clients) do either not (e.g. normal GNU tar
format) or behave in a way that somehow break or lose XATTRs/ACLs (e.g.
vim),... NFS is another example - a very important IMHO since it is THE
network filesystem.
So sure,... not many people are using XATTRs, ACLs, but the reason is
simply, that there are still many obstacles in the way - in many cases
upstream refusing to solve these, for the reason that allegedly noone
uses XATTRs/ACLs.
(Which is btw. not fully true,... many modernish stuff like uses e.g.
ACLs like in /dev/).
I do agree that especially the ACLs from POSIX are not the best thing
(NFS4 ACLs are much better),... but I don't see the big problem with
key/value pairs from XATTRs.
Cheers,
Chris.
On Thu, Oct 24, 2013 at 07:22:30PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2013-10-24 at 16:30 +0000, Myklebust, Trond wrote:
> > Those programs need to recompute the checksum data anyway in order to
> > verify and/or update it. Checksums that are computed by some third
> > party application have exactly zero value for integrity checking.
> No that's exactly the point,... the applications should _NOT_ set those
> checksums, especially not automagically (since then you'd never notice
> when just some application is buggy or writes/modifies when you don't
> expect it to do so).
> The idea is that there is on application (in my case it's just a
> script), which sets the integrity data and verifies it.
>
> This works very well for e.g. large data archives, where you most of the
> time (but not always) only read files, write new files or move around
> existing ones - but only rarely modify existing file's contents.
>
>
> I do this already like that on local filesystems, which works very
> nicely with XATTRs... but now I want to move this on a central data
> cluster (where clients connect to via NFS)... and here the problems
> start... when I add new data to the archive (from the clients) I cannot
> have XATTRs attached, nor can I verify them form the clients.
Can you give any more details about your use case? (E.g. is there an
article describing this system somewhere?)
Just curious.--b.
On Thu, 2013-10-24 at 15:11 +0000, Myklebust, Trond wrote:
> On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:
>
> > Because the filesystem can do that when multiple applications are
> > involved without having to change them all to talk to each other and
> > invent custom protocol all the time just to keep some additional
> > metadata associated to a file..
> >
> It's still a custom protocol. The applications need to agree on a data
> format and store it somewhere. The portable way to do this is to write
> an application library that they can link to.
Perhaps I was unclear, you are never going to see that custom library
linked into the 'mv' command.
So your approach makes little sense if the object is to maintain data
coherent when people need to handle files from random applications and
scripts and general system maintenance.
The data may be relevant only to a specific application.
I am not saying you *have* to implement xattrs support, just saying that
it is not a mere 'applications should synchronize data themselves'
problem.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
Hi,
I was going for the point you make later in this thread: "user xattrs are in
principle harmless."
The exposure policy for non-harmless xattrs issue is real enough, but my point was,
it seems to be a Linux policy issue, at the NFSv4 level just as it was at the syscall level.
The caching and atomicity issues seem like they actually are issues for NFSv4 discussion.
If the named attribute interface actually fails to cover xattrs, that's a big gaffe.
Matt
----- "Trond Myklebust" <[email protected]> wrote:
> On Oct 26, 2013, at 10:01 AM, Matt W. Benjamin <[email protected]>
> wrote:
>
> > Hi,
> >
> > Surely you don't need NFSv4 to standardize the the (empty?) set of
> system or magic attributes Linux
> > should export?
>
> The NFSv4 working group has no authority to do so. POSIX would be the
> right address.
>
> > Besides, as you're well aware, most people who ask for xattrs are
> looking for an ability to associate
> > arbitrary specific data, not a back door ioctl interface. That's
> clearly what the NFSv4 named attributes as standardized were intended
> for.
>
> No. I'm not aware of that.
>
> > I'm well aware of other uses and plans that someone would want to
> standardize, but it seems
> > irrelevant to the discussion.
>
> It's very relevant to the discussion as it defines what namespace
> applications can expect to work.
>
> Trond
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:
+AD4- Because the filesystem can do that when multiple applications are
+AD4- involved without having to change them all to talk to each other and
+AD4- invent custom protocol all the time just to keep some additional
+AD4- metadata associated to a file..
+AD4-
It's still a custom protocol. The applications need to agree on a data
format and store it somewhere. The portable way to do this is to write
an application library that they can link to.
--
Trond Myklebust
Linux NFS client maintainer
NetApp
Trond.Myklebust+AEA-netapp.com
http://www.netapp.com
(Sorry for top post)
Ah, note that by "glusterfs.<something>" i was meaning "<ns>.glusterfs.<something>" where <ns> is like "user". It just needs to be a mechanism to store key values on the inode.
Avati
Sent from my Android phone using TouchDown (http://www.nitrodesk.com)
-----Original Message-----
From: Myklebust, Trond [[email protected]]
Received: Monday, 28 Oct 2013, 18:52
To: Anand Avati [[email protected]]
CC: Wheeler Ric [[email protected]]; Dr Fields James Bruce [[email protected]]; Christoph Anton Mitterer [[email protected]]; Mailing List Linux NFS [[email protected]]; Dickson Steve [[email protected]]
Subject: Re: XATTRs in NFS?
On Oct 28, 2013, at 9:24 PM, Anand Avati <[email protected]> wrote:
> On 10/28/2013 06:26 PM, Myklebust, Trond wrote:
>
>>
>> That battle may have been fought and won within the glusterfs community, but why should we wave the white flag without a discussion? I don't see how what he described above has anything to do with user defined attributes. He's describing how he wants to export quota information and xtime through a private xattr interface that is currently unique to glusterfs. How is that not a private syscall interface?
>>
>
> Exposing quota informtion is use "from the top". Note the other point I mention about using NFS volumes as "gluster bricks" where we store xattrs as dumb and persistent key/values associated with file/dir inodes (fresh/stale info for replication, hash ranges for dirs, quota acconting info per-dir, xtime per dir).
>> Which of the mainstream filesystems have their own private xattr namespaces like the above?
>>
>
> Why should NFS need to worry? As long as it acts like a pass-through (like every other call it supports).
We need to worry because we don't know what side-effects your private interface will have. How does it affect our caching model? How do we debug any problems that arise? Are there any security implications that we need to know about?
Trond
Hi,
It appears that there could be spec implications, but I have not so far been convinced that
nfsv4.x named attributes are not an appropriate vehicle for (some set of) extended attributes.
The fact that nfsv4.x named attributes are suitable to represent large objects, does not make
them unsuitable to represent key value properties.
Matt
----- "Jeff Layton" <[email protected]> wrote:
> On Thu, 24 Oct 2013 11:16:10 -0400
> Simo Sorce <[email protected]> wrote:
>
> > On Thu, 2013-10-24 at 15:11 +0000, Myklebust, Trond wrote:
> > > On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:
> > >
> > > > Because the filesystem can do that when multiple applications
> are
> > > > involved without having to change them all to talk to each other
> and
> > > > invent custom protocol all the time just to keep some
> additional
> > > > metadata associated to a file..
> > > >
> > > It's still a custom protocol. The applications need to agree on a
> data
> > > format and store it somewhere. The portable way to do this is to
> write
> > > an application library that they can link to.
> >
> > Perhaps I was unclear, you are never going to see that custom
> library
> > linked into the 'mv' command.
> >
> > So your approach makes little sense if the object is to maintain
> data
> > coherent when people need to handle files from random applications
> and
> > scripts and general system maintenance.
> >
> > The data may be relevant only to a specific application.
> >
> > I am not saying you *have* to implement xattrs support, just saying
> that
> > it is not a mere 'applications should synchronize data themselves'
> > problem.
> >
>
> I think the real solution if people need this is to lead an effort to
> put xattrs into the spec. I think there is still time to get new
> features into v4.3 if someone wants to champion it...
>
> --
> Jeff Layton <[email protected]>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
On Oct 27, 2013, at 5:57 PM, Christoph Anton Mitterer <[email protected]> wrote:
> On Sun, 2013-10-27 at 15:15 -0400, J. Bruce Fields wrote:
>> Was this problem actually caught using checksums stored in xattrs, or
>> did the problem predate your use of xattrs?
> Phew... don't remember actually... but I think I haven't used that
> already back then and noticed it by chance when I did some diffs.
>
>
>>> So the idea of my integrity data is, that I really manually say "now the
>>> data is in the state where I consider it to be consistent and I want to
>>> have checksums stored and attached to the files, for exactly that
>>> state", e.g. after I have read out some images from the SD card (perhaps
>>> even twice with the cache cleared and the results diffed) and placed in
>>> my archive.
>>> Afterwards I can regularly verify the whole archive and if at some stage
>>> corruptions as the above would have happened, I can simply take the
>>> respective files from backups.
>> How long have you been using this for?
> Uhm... about 3-4 years now.
>
>> How many problems has it caught?
> I do not keep exact statistics... but I remember a few cases where I
> found damaged backups (optimal media) which I replaced as a consequence.
>
>> How often do you checksum or verify files, and how expensive is that?
> Not that often,... on my actual data disks,... about every 4 months...
> on my backup media (I use to keep older generations of backups as well)
> about once a year.
>
> I've never really looked at how expensive it is,... all you need to do
> is simply reading all data + have their hashes calculated.
...and if the checksums are any good, then all you need to do to substitute a database is to realise that a good data checksum is invariant under renames.
On Thu, 2013-10-24 at 14:32 +0000, Myklebust, Trond wrote:
> Linux xattrs are a rabid mess.
Well... might be from a technical POV, but for users they're quite
useful in some scenarios.
> The whole "system" namespace is something that cannot and should not ever be exposed on a network.
> The "trusted" and "user" namespaces just offer specialised storage. Why are they needed?
Well what I do is attaching integrity information to files.
You may say now that this is similar to what btrfs will provide
anyway... but the problem with that is, that checksums are always
updated when something in the system does valid changes to the file.
What I however want is that I really manually have to set this, so that
I notice "accidental" changes, e.g. by myself or by buggy software...
> If the data needs to follow the file, then store it in the file. Why do you need the filesystem to manage that for you?
... and since this applies to arbitrary files, from text-files over
pictures, videos to binaries,... it's neither possible to store this in
the file, nor can I really track this with an database,... since
literally any program that uses such files, from the picture editor to
the file-manager would need to use such DB.
Cheers,
Chris.
Hi,
We are prepared to contribute a draft implementation of -something-, and have planned
to do so.
We're currently working on Ganesha support. I have assumed that what we
would implement is mapping of from user xattr to current nfsv4 named attributes.
Matt
----- "Ric Wheeler" <[email protected]> wrote:
> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
> > On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer
> <[email protected]> wrote:
> >
> >> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
> >>> Then you end up with large directories and an extra name per inode
> that needs to
> >>> be stored and extra lookups for each file when you do a whole file
> system crawl.
> >>>
> >>> Certainly not as easy as adding and xattrs with that information
> :)
> >> And I think there's another reason why it wouldn't work...
> >>
> >> Imagine I change my system to encode what should be XATTRs in
> hardlink
> >> pseudo files...
> >>
> >> If I have such pair locally e.g. on my ext4:
> >> /foo/bar/actual/file
> >> /meta/<SHA512 identifier>.2342348324
> >>
> >> And now move/copy the file via the network to the archive, I'd have
> to
> >> copy both files (which is really annoying), and I'd guess the
> inode
> >> coupling would get los (and at least the name wouldn't fit
> anymore).
> >>
> >> So the whole thing is IMHO not even a workaround.
> > OK. So you're going to do XATTRs for us?
> >
> > Trond
>
> Now that pNFS is perfect and labeled NFS has made it upstream, I think
> that
> Steve D must be looking for something to keep him busy :)
>
> ric
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
On Sun, Oct 27, 2013 at 12:31:46PM +0000, Myklebust, Trond wrote:
> The NFSv4 working group has no authority to do so. POSIX would be the right address.
The complete lack of clue, authority or even defined semantics didn't
stop them from defining their ACL model. User extended attributes are
harmless compared to that, even if everyone has subtly different
flavours.
On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
> On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer <[email protected]> wrote:
>
>> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>>> Then you end up with large directories and an extra name per inode that needs to
>>> be stored and extra lookups for each file when you do a whole file system crawl.
>>>
>>> Certainly not as easy as adding and xattrs with that information :)
>> And I think there's another reason why it wouldn't work...
>>
>> Imagine I change my system to encode what should be XATTRs in hardlink
>> pseudo files...
>>
>> If I have such pair locally e.g. on my ext4:
>> /foo/bar/actual/file
>> /meta/<SHA512 identifier>.2342348324
>>
>> And now move/copy the file via the network to the archive, I'd have to
>> copy both files (which is really annoying), and I'd guess the inode
>> coupling would get los (and at least the name wouldn't fit anymore).
>>
>> So the whole thing is IMHO not even a workaround.
> OK. So you're going to do XATTRs for us?
>
> Trond
Now that pNFS is perfect and labeled NFS has made it upstream, I think that
Steve D must be looking for something to keep him busy :)
ric
On Oct 26, 2013, at 10:01 AM, Matt W. Benjamin <[email protected]> wrote:
> Hi,
>
> Surely you don't need NFSv4 to standardize the the (empty?) set of system or magic attributes Linux
> should export?
The NFSv4 working group has no authority to do so. POSIX would be the right address.
> Besides, as you're well aware, most people who ask for xattrs are looking for an ability to associate
> arbitrary specific data, not a back door ioctl interface. That's clearly what the NFSv4 named attributes as standardized were intended for.
No. I'm not aware of that.
> I'm well aware of other uses and plans that someone would want to standardize, but it seems
> irrelevant to the discussion.
It's very relevant to the discussion as it defines what namespace applications can expect to work.
Trond
On 10/27/2013 08:44 PM, Myklebust, Trond wrote:
> On Oct 27, 2013, at 8:27 PM, Christoph Anton Mitterer <[email protected]> wrote:
>
>> On Mon, 2013-10-28 at 00:17 +0000, Myklebust, Trond wrote:
>>> ...and if the checksums are any good, then all you need to do to
>>> substitute a database is to realise that a good data checksum is
>>> invariant under renames.
>> Don't quite see what you mean...
>>
>> Sure the checksums stay the same, but consider you have many millions of
>> files, and you moved them around and thus the paths in the DB are
>> incorrect... verifying the files will become very much a pain in the
>> a**, especially when multiple files don't verify anymore.
>>
>> Or what if you have many small similar files, where errors could lead to
>> a checksum that was a correct one for another file,... when the paths
>> are no longer valid you cannot know if this was a correct file or not.
> If you have lots of small files, and you really do need to associate them uniquely with the checksum, then try something like:
>
> ln <filename> /path/to/database/<SHA512 identifier>.<inode number>
>
> Hard links and inode numbers are also generally invariant under 'mv'.
>
> Trond--
>
Then you end up with large directories and an extra name per inode that needs to
be stored and extra lookups for each file when you do a whole file system crawl.
Certainly not as easy as adding and xattrs with that information :)
Ric
On 10/28/2013 02:08 PM, Dr Fields James Bruce wrote:
> On Mon, Oct 28, 2013 at 02:00:58PM -0400, Ric Wheeler wrote:
>> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
>>> On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer <[email protected]> wrote:
>>>
>>>> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>>>>> Then you end up with large directories and an extra name per inode that needs to
>>>>> be stored and extra lookups for each file when you do a whole file system crawl.
>>>>>
>>>>> Certainly not as easy as adding and xattrs with that information :)
>>>> And I think there's another reason why it wouldn't work...
>>>>
>>>> Imagine I change my system to encode what should be XATTRs in hardlink
>>>> pseudo files...
>>>>
>>>> If I have such pair locally e.g. on my ext4:
>>>> /foo/bar/actual/file
>>>> /meta/<SHA512 identifier>.2342348324
>>>>
>>>> And now move/copy the file via the network to the archive, I'd have to
>>>> copy both files (which is really annoying), and I'd guess the inode
>>>> coupling would get los (and at least the name wouldn't fit anymore).
>>>>
>>>> So the whole thing is IMHO not even a workaround.
>>> OK. So you're going to do XATTRs for us?
>>>
>>> Trond
>> Now that pNFS is perfect and labeled NFS has made it upstream, I
>> think that Steve D must be looking for something to keep him busy :)
> I agree with Trond that we first really need good evidence about exactly
> who wants this and why.
>
> --b.
For the user space xattrs, many applications store various types of metadata.
Gluster for example heavily uses xattrs, other programs do things like data
scrubbing (look for a long unchanged file, compute a has and store it as an
xattr) or simply use it to annotate the file with the name of the program that
created it. Think of it as file decorations or annotations.
Today, if we store files in NFS that have xattrs, we do in fact cause data loss.
I can understand an answer of "this would be hard to do for NFS and need to go
through IETF" but think that xattrs are well enough established in Linux and
supported in the tool chain that it is way too late to question whether or not
supporting them is a worth our time.
Ric
On Oct 27, 2013, at 2:30 PM, Simo Sorce <[email protected]> wrote:
> On Sun, 2013-10-27 at 18:07 +0000, Myklebust, Trond wrote:
>> On Oct 27, 2013, at 12:56 PM, Christoph Hellwig <[email protected]> wrote:
>>
>>> On Sun, Oct 27, 2013 at 12:31:46PM +0000, Myklebust, Trond wrote:
>>>> The NFSv4 working group has no authority to do so. POSIX would be the right address.
>>>
>>> The complete lack of clue, authority or even defined semantics didn't
>>> stop them from defining their ACL model. User extended attributes are
>>> harmless compared to that, even if everyone has subtly different
>>> flavours.
>>>
>>
>> The user xattrs are in principle harmless. What about "trusted.*"?
>>
>> BTW: caching any xattrs is a problem. There is no close-to-open model that you can use and neither is there locking. This is already causing a problem for labeled nfs...
>
> You could start without caching them.
>
I'm not going to start anything...
Trond
On Sun, 2013-10-27 at 12:48 +0000, Myklebust, Trond wrote:
> Let's get the basic discussion of motivation right first.
Actually I might know another use case.
dCache[0] which implements a (p)NFS 4.1 server and is heavily used
within the high energy physics community, could benefit from XATTRs as
well.
Ever since, dCache has had a concept called PNFS tags (where the PNFS
had nothing to do with parallel NFS, but is called so for other
historical reasons), which are basically key/value pairs belonging to
some file (well at least directory, I don't remember where PNFS tags
exist for regular files).
These tags are used to control several things, indirectly, e.g. to which
data pool node a file may go, and via that whether it may be moved to
tape, whether it is duplicated, etc. pp.
Since there were no XATTRs (at least I guess that's the historical
reason), they used special files for this having the name:
".(tag)(key)" and the content of the file is the value.
So for a path,
/data/foo/bar/
one can set the key "sGroup" (aka storage group) to a value say
"ATLAS" (which (very simplified) will tell dCache in the end that this
is storage belonging to the ATLAS experiment) by creating a file
echo "ATLAS" > '/data/foo/bar/.(tag)(sGroup)'
Obviously having such special files isn't that nice, and having real
XATTRs would be the better solution.
Cheers,
Chris.
btw: Note that while I'm good friends with the dCache developers, I
don't speak for them, nor do I know whether they'd really like this or
not.
[0] http://www.dcache.org/
On Oct 24, 2013, at 3:13 PM, Christoph Anton Mitterer <[email protected]> wrote:
> On Thu, 2013-10-24 at 08:45 +0000, Myklebust, Trond wrote:
>> labeled NFS (i.e. security labels for NFS) is already supported in Linux 3.10 and newer.
> Sure, but that doesn't really help me.
>
>
>> There are no plans to merge general purpose xattrs.
> Why not? Is it a big deal?
>
Linux xattrs are a rabid mess.
The whole "system" namespace is something that cannot and should not ever be exposed on a network.
The "trusted" and "user" namespaces just offer specialised storage. Why are they needed?
>
>> Please just use an application-specific database.
> Well that won't work,... since that wouldn't be updated if e.g.
> pathnames are changed by any program (cp, mv)
If the data needs to follow the file, then store it in the file. Why do you need the filesystem to manage that for you?
Cheers
Trond
On Oct 23, 2013, at 9:37 PM, Christoph Anton Mitterer <[email protected]> wrote:
> Hi.
>
> I just wondered about the current situation (respectively plans)
> regarding XATTRs in NFS (I personally don't care much about the
> version,... of course v4 would be nice, but v3 should do either).
>
>
> I'm aware about the patches for v3 that James Morris has submitted some
> time ago, but AFAICS these were never accepted/merged.
>
> Asking him (off-list) he mentioned that any such work would need to take
> place in NFS4.
>
>
> I've seen some comments from the past, saying that it should never
> happen since it would cause performance issues - but I guess one could
> make it optional.
>
>
> I have some use case where I need a network filesystem with XATTR
> support providing the features of most standard filesystems (locking,
> etc.).
> And I'm sure there are plenty more use-cases.
>
>
> So are there any plans to get this into NFS?
>
labeled NFS (i.e. security labels for NFS) is already supported in Linux 3.10 and newer.
There are no plans to merge general purpose xattrs. Please just use an application-specific database.
Cheers
Trond
On Mon, 2013-10-28 at 17:49 +0000, Myklebust, Trond wrote:
> OK. So you're going to do XATTRs for us?
That's not what I've said ;-) ... neither have I said that you must or
should do it.
I asked for the status and mentioned the need of it. You asked me why -
so I tried to explain.
Cheers,
Chris.
On Oct 25, 2013, at 11:26 AM, Ric Wheeler <[email protected]> wrote:
> On 10/25/2013 10:08 AM, J. Bruce Fields wrote:
>> On Thu, Oct 24, 2013 at 07:22:30PM +0200, Christoph Anton Mitterer wrote:
>>> On Thu, 2013-10-24 at 16:30 +0000, Myklebust, Trond wrote:
>>>> Those programs need to recompute the checksum data anyway in order to
>>>> verify and/or update it. Checksums that are computed by some third
>>>> party application have exactly zero value for integrity checking.
>>> No that's exactly the point,... the applications should _NOT_ set those
>>> checksums, especially not automagically (since then you'd never notice
>>> when just some application is buggy or writes/modifies when you don't
>>> expect it to do so).
>>> The idea is that there is on application (in my case it's just a
>>> script), which sets the integrity data and verifies it.
>>>
>>> This works very well for e.g. large data archives, where you most of the
>>> time (but not always) only read files, write new files or move around
>>> existing ones - but only rarely modify existing file's contents.
>>>
>>>
>>> I do this already like that on local filesystems, which works very
>>> nicely with XATTRs... but now I want to move this on a central data
>>> cluster (where clients connect to via NFS)... and here the problems
>>> start... when I add new data to the archive (from the clients) I cannot
>>> have XATTRs attached, nor can I verify them form the clients.
>> Can you give any more details about your use case? (E.g. is there an
>> article describing this system somewhere?)
>>
>> Just curious.--b.
>>
>
> I think that having xattrs in NFS would be very useful over time. A lot of user space tools have been made aware of them, we are certainly lagging all (almost all?) local file system here and it can cause a data loss when you copy a file from a local file system to an NFS server.
>
> It certainly violates the principle of least surprise that the xattrs are lost when move through NFS!
>
> Typical use cases I have seen include storing things like metadata that tracks what application created the file, tags to let you know when the last time the file was verified by a data integrity scrubber, tags that hold a checksum/crypto has of the file contents along with the date of that signature.
>
> Doing a file system service does not mean that we have to be personally interested in all of the odd things that our users might use features for, but at this point, xattrs are water under the bridge and NFS not supporting them makes us look bad :)
Trond's point is that it still would be a very heavy lift for us.
Local file systems can support whatever they like, since they are Linux-only.
NFS has to interoperate with other operating systems that may or may not support xattrs, or may have something that looks like an xattr, but really is subtly incompatible.
Therefore we must have a standard for storing and accessing xattrs locally (POSIX) and a standard for expressing their name and content on the wire (IETF NFS) before we can consider an implementation.
So we would like to see some very clear evidence that it's worth the effort. Until then, Linux distributors are free to implement NFS xattr support outside the standard NFS protocol (like the Solaris NFSv3 ACL protocol) and support this themselves, perhaps as proof that "if you build it, they will come."
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Thu, 2013-10-24 at 11:05 -0400, Matt W. Benjamin wrote:
+AD4- Hi,
+AD4-
+AD4- Presumably Linux client and server implementations would restrict use of +ACI-system.+ACI-
+AD4- Perhaps would be more to talk about, if xattrs were implemented experimentally.
+AD4-
+AD4- I'm interested in knowing about patches that might be floating around for the Linux client, for NFSv4.x. (I've seen NFSv3 work.)
+AD4-
+AD4- Thanks,
+AD4-
+AD4- Matt
We've already talked about this, Matt. The answer is still NO.
Trond
+AD4- ----- +ACI-Trond Myklebust+ACI- +ADw-Trond.Myklebust+AEA-netapp.com+AD4- wrote:
+AD4-
+AD4- +AD4- On Oct 24, 2013, at 3:13 PM, Christoph Anton Mitterer
+AD4- +AD4- +ADw-calestyo+AEA-scientia.net+AD4- wrote:
+AD4- +AD4-
+AD4- +AD4- +AD4- On Thu, 2013-10-24 at 08:45 +-0000, Myklebust, Trond wrote:
+AD4- +AD4- +AD4APg- labeled NFS (i.e. security labels for NFS) is already supported in
+AD4- +AD4- Linux 3.10 and newer.
+AD4- +AD4- +AD4- Sure, but that doesn't really help me.
+AD4- +AD4- +AD4-
+AD4- +AD4- +AD4-
+AD4- +AD4- +AD4APg- There are no plans to merge general purpose xattrs.
+AD4- +AD4- +AD4- Why not? Is it a big deal?
+AD4- +AD4- +AD4-
+AD4- +AD4-
+AD4- +AD4- Linux xattrs are a rabid mess.
+AD4- +AD4-
+AD4- +AD4- The whole +ACI-system+ACI- namespace is something that cannot and should not
+AD4- +AD4- ever be exposed on a network.
+AD4- +AD4- The +ACI-trusted+ACI- and +ACI-user+ACI- namespaces just offer specialised storage.
+AD4- +AD4- Why are they needed?
+AD4- +AD4-
+AD4- +AD4- +AD4-
+AD4- +AD4- +AD4APg- Please just use an application-specific database.
+AD4- +AD4- +AD4- Well that won't work,... since that wouldn't be updated if e.g.
+AD4- +AD4- +AD4- pathnames are changed by any program (cp, mv)
+AD4- +AD4-
+AD4- +AD4- If the data needs to follow the file, then store it in the file. Why
+AD4- +AD4- do you need the filesystem to manage that for you?
+AD4- +AD4-
+AD4- +AD4- Cheers
+AD4- +AD4- Trond--
+AD4- +AD4- To unsubscribe from this list: send the line +ACI-unsubscribe linux-nfs+ACI-
+AD4- +AD4- in
+AD4- +AD4- the body of a message to majordomo+AEA-vger.kernel.org
+AD4- +AD4- More majordomo info at http://vger.kernel.org/majordomo-info.html
+AD4-
--
Trond Myklebust
Linux NFS client maintainer
NetApp
Trond.Myklebust+AEA-netapp.com
http://www.netapp.com
On Mon, Oct 28, 2013 at 08:55:06PM +0000, Haynes, Tom wrote:
>
> On Oct 28, 2013, at 1:44 PM, Marc Eshel <[email protected]> wrote:
>
> > Hi Spencer,
> > Is it still possible to get on the agenda fir IETF 88, I just got approval
> > to travel. We can use about 15 minutes, or what ever is available to
> > discussion the future of named attributes in NFS. The two main questions
> > that we need answer are:
> > 1. Do we need them? what applications use them and how.
> > 2. Can we have a more simple model that handles just user attributes.
> >
> > Input is welcome to help me make the case at the meeting.
> >
> > Thanks, Marc.
> >
>
>
> Be sure to call out why they currently do or do not work.
And we also need to keep clear the distinctions between:
- NFSv4 named attributes/streams/forks and "extended attributes"
as implemented in Linux and elsewhere.
- user.* xattrs and "back-door ioctls".
(Your description above seems to confuse named attributes and Linux
extended attributes.)
--b.
Trond,... since you ask for motivations pro NFS-XATTRs, I wondered what
are the strong reasons against?
Of course someone has to do the work,... but that's not really and
argument pro or con NFS-XATTRs, is it?
The security problems with namespaces like trusted. or so can probably
be solved quite easy, e.g. simply by not supporting or
ignoring/rejecting them.
You've mentioned caching issues,... could you elaborate a bit on that?
How is that much different from caching any other file metadata NFS
transfers?
Are there other (technical) reasons?
Cheers,
Chris.
On Thu, 2013-10-24 at 14:32 +0000, Myklebust, Trond wrote:
> On Oct 24, 2013, at 3:13 PM, Christoph Anton Mitterer <[email protected]> wrote:
>
> > On Thu, 2013-10-24 at 08:45 +0000, Myklebust, Trond wrote:
> >> labeled NFS (i.e. security labels for NFS) is already supported in Linux 3.10 and newer.
> > Sure, but that doesn't really help me.
> >
> >
> >> There are no plans to merge general purpose xattrs.
> > Why not? Is it a big deal?
> >
>
> Linux xattrs are a rabid mess.
First time I hear this :)
> The whole "system" namespace is something that cannot and should not ever be exposed on a network.
> The "trusted" and "user" namespaces just offer specialised storage. Why are they needed?
Samba for example stores various metadata bits of information there (DOS
bits, original ACLs before posix translation, etc..).
Loosing that data on an mv via NFS breaks stuff.
That said samba and NFS have other synchronization issues, so it is not
the best example, but you can't just discount xattrs. They exist, they
are used, and if you don't have support for them in NFS you are not
transparent to applications.
> >> Please just use an application-specific database.
> > Well that won't work,... since that wouldn't be updated if e.g.
> > pathnames are changed by any program (cp, mv)
>
> If the data needs to follow the file, then store it in the file. Why do you need the filesystem to manage that for you?
Because the filesystem can do that when multiple applications are
involved without having to change them all to talk to each other and
invent custom protocol all the time just to keep some additional
metadata associated to a file..
Simo.
--
Simo Sorce * Red Hat, Inc * New York
On Oct 25, 2013, at 5:26 PM, Ric Wheeler <[email protected]> wrote:
> On 10/25/2013 10:08 AM, J. Bruce Fields wrote:
>> On Thu, Oct 24, 2013 at 07:22:30PM +0200, Christoph Anton Mitterer wrote:
>>> On Thu, 2013-10-24 at 16:30 +0000, Myklebust, Trond wrote:
>>>> Those programs need to recompute the checksum data anyway in order to
>>>> verify and/or update it. Checksums that are computed by some third
>>>> party application have exactly zero value for integrity checking.
>>> No that's exactly the point,... the applications should _NOT_ set those
>>> checksums, especially not automagically (since then you'd never notice
>>> when just some application is buggy or writes/modifies when you don't
>>> expect it to do so).
>>> The idea is that there is on application (in my case it's just a
>>> script), which sets the integrity data and verifies it.
>>>
>>> This works very well for e.g. large data archives, where you most of the
>>> time (but not always) only read files, write new files or move around
>>> existing ones - but only rarely modify existing file's contents.
>>>
>>>
>>> I do this already like that on local filesystems, which works very
>>> nicely with XATTRs... but now I want to move this on a central data
>>> cluster (where clients connect to via NFS)... and here the problems
>>> start... when I add new data to the archive (from the clients) I cannot
>>> have XATTRs attached, nor can I verify them form the clients.
>> Can you give any more details about your use case? (E.g. is there an
>> article describing this system somewhere?)
>>
>> Just curious.--b.
>>
>
> I think that having xattrs in NFS would be very useful over time. A lot of user space tools have been made aware of them, we are certainly lagging all (almost all?) local file system here and it can cause a data loss when you copy a file from a local file system to an NFS server.
>
> It certainly violates the principle of least surprise that the xattrs are lost when move through NFS!
The _problem_ is that xattrs are a system call interface. Setting the xattrs in the "system", "security", or even the "trusted" namespace have undocumented side effects. There are very good security reasons why NFS doesn't have an IOCTL rpc call for executing arbitrary ioctls on the NFS server. Those same reasons apply to xattr.
> Typical use cases I have seen include storing things like metadata that tracks what application created the file, tags to let you know when the last time the file was verified by a data integrity scrubber, tags that hold a checksum/crypto has of the file contents along with the date of that signature.
Standardise those xattrs, and we can export them specifically as part of the NFSv4 spec.
> Doing a file system service does not mean that we have to be personally interested in all of the odd things that our users might use features for, but at this point, xattrs are water under the bridge and NFS not supporting them makes us look bad :)
I'd rather look bad for not supporting a broken filesystem feature than have to deal with the abiove mentioned side-effects. Until the xattr interface is properly documented and standardised so that it can be exported safely, it should be treated as a unexportable, in the same way we treat procfs and sysfs.
Cheers
Trond
On Oct 28, 2013, at 1:44 PM, Marc Eshel <[email protected]> wrote:
> Hi Spencer,
> Is it still possible to get on the agenda fir IETF 88, I just got approval
> to travel. We can use about 15 minutes, or what ever is available to
> discussion the future of named attributes in NFS. The two main questions
> that we need answer are:
> 1. Do we need them? what applications use them and how.
> 2. Can we have a more simple model that handles just user attributes.
>
> Input is welcome to help me make the case at the meeting.
>
> Thanks, Marc.
>
Be sure to call out why they currently do or do not work.
On Mon, Oct 28, 2013 at 02:00:58PM -0400, Ric Wheeler wrote:
> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
> >On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer <[email protected]> wrote:
> >
> >>On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
> >>>Then you end up with large directories and an extra name per inode that needs to
> >>>be stored and extra lookups for each file when you do a whole file system crawl.
> >>>
> >>>Certainly not as easy as adding and xattrs with that information :)
> >>And I think there's another reason why it wouldn't work...
> >>
> >>Imagine I change my system to encode what should be XATTRs in hardlink
> >>pseudo files...
> >>
> >>If I have such pair locally e.g. on my ext4:
> >>/foo/bar/actual/file
> >>/meta/<SHA512 identifier>.2342348324
> >>
> >>And now move/copy the file via the network to the archive, I'd have to
> >>copy both files (which is really annoying), and I'd guess the inode
> >>coupling would get los (and at least the name wouldn't fit anymore).
> >>
> >>So the whole thing is IMHO not even a workaround.
> >OK. So you're going to do XATTRs for us?
> >
> >Trond
>
> Now that pNFS is perfect and labeled NFS has made it upstream, I
> think that Steve D must be looking for something to keep him busy :)
I agree with Trond that we first really need good evidence about exactly
who wants this and why.
--b.
Hi Trond,
I am just trying to see if there any path here to get xattr over NFS.
Assuming that there is agreement that xattr are used and needed, and we
need to build a better case here but just give alternative solutions
doesn't mean that is not needed, almost every thing can be done with an
additional DB that associates attributes with files but it is not the best
or right way.
So if we had an NFS protocol extension (might not be named attributes)
that mapped to and was semantically compatibly to Linux xattr, as Nico
wrote, would that be enough to consider adding it to the Linux NFS?
I see the need to show requirement and better interface over the NFS
protocol (with no security holes) but not sure if we have another problem
with standardizing xattr in Linux?
Thanks, Marc.
From: "Myklebust, Trond" <[email protected]>
To: Christoph Anton Mitterer <[email protected]>,
Cc: Wheeler Ric <[email protected]>, Anand Avati
<[email protected]>, "Dr Fields James Bruce" <[email protected]>, Mailing
List Linux NFS <[email protected]>, Dickson Steve
<[email protected]>
Date: 10/28/2013 07:28 PM
Subject: Re: XATTRs in NFS?
Sent by: [email protected]
On Oct 28, 2013, at 9:39 PM, Christoph Anton Mitterer
<[email protected]> wrote:
> Trond,... since you ask for motivations pro NFS-XATTRs, I wondered what
> are the strong reasons against?
As I've already told you: you're asking for the addition of a feature
which is inadequately defined, and is not documented in any standard.
> Of course someone has to do the work,... but that's not really and
> argument pro or con NFS-XATTRs, is it?
>
> The security problems with namespaces like trusted. or so can probably
> be solved quite easy, e.g. simply by not supporting or
> ignoring/rejecting them.
>
> You've mentioned caching issues,... could you elaborate a bit on that?
> How is that much different from caching any other file metadata NFS
> transfers?
The standard metadata such as ctime, mtime, size, .... are all part of an
existing NFS caching model called close-to-open. We know how they change
when the filesystem data changes.
How do I know when it is safe to cache your checksum xattr? I don't even
know what it is, let alone what its relation is to the other filesystem
objects.
Let's say client B modifies the file and updates the checksum. When client
A opens the file, it will see that the data has changed, but how does it
know that it also needs to update the xattr information?
Alternatively, let's say that client A reads the file and checksum data
after client B has updated the file, but before it updates the checksum.
What will cause client A to stop caching the stale checksum once client B
has updated it?
Trond-
On Mon, 2013-10-28 at 00:44 +0000, Myklebust, Trond wrote:
> If you have lots of small files, and you really do need to associate them uniquely with the checksum, then try something like:
>
> ln <filename> /path/to/database/<SHA512 identifier>.<inode number>
>
> Hard links and inode numbers are also generally invariant under 'mv'.
Well that seems like a workaround, IMHO not like a real solution,...
since when you remove <filename> the file will stay due
to /path/to/database/<SHA512 identifier>.<inode number> .
Of course you can run regular scrubbing jobs, which remove
and /path/to/database/<SHA512 identifier>.<inode number> for which the
link count is one.
Anyway as said, IMHO it's a (nice) workaround which I haven't thought
of, admittedly, but not a clean solution... and AFAICS it fixes only my
personal use case.
Cheers,
Chris.
On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer <[email protected]> wrote:
> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>> Then you end up with large directories and an extra name per inode that needs to
>> be stored and extra lookups for each file when you do a whole file system crawl.
>>
>> Certainly not as easy as adding and xattrs with that information :)
>
> And I think there's another reason why it wouldn't work...
>
> Imagine I change my system to encode what should be XATTRs in hardlink
> pseudo files...
>
> If I have such pair locally e.g. on my ext4:
> /foo/bar/actual/file
> /meta/<SHA512 identifier>.2342348324
>
> And now move/copy the file via the network to the archive, I'd have to
> copy both files (which is really annoying), and I'd guess the inode
> coupling would get los (and at least the name wouldn't fit anymore).
>
> So the whole thing is IMHO not even a workaround.
OK. So you're going to do XATTRs for us?
Trond
On Oct 28, 2013, at 5:02 PM, "J. Bruce Fields" <[email protected]> wrote:
> On Mon, Oct 28, 2013 at 08:55:06PM +0000, Haynes, Tom wrote:
>>
>> On Oct 28, 2013, at 1:44 PM, Marc Eshel <[email protected]> wrote:
>>
>>> Hi Spencer,
>>> Is it still possible to get on the agenda fir IETF 88, I just got approval
>>> to travel. We can use about 15 minutes, or what ever is available to
>>> discussion the future of named attributes in NFS. The two main questions
>>> that we need answer are:
>>> 1. Do we need them? what applications use them and how.
>>> 2. Can we have a more simple model that handles just user attributes.
>>>
>>> Input is welcome to help me make the case at the meeting.
>>>
>>> Thanks, Marc.
>>>
>>
>>
>> Be sure to call out why they currently do or do not work.
>
> And we also need to keep clear the distinctions between:
>
> - NFSv4 named attributes/streams/forks and "extended attributes"
> as implemented in Linux and elsewhere.
> - user.* xattrs and "back-door ioctls".
>
> (Your description above seems to confuse named attributes and Linux
> extended attributes.)
I wonder if we will have a balanced discussion next week. The people who seem to know the most about these issues on Linux will not be attending. Perhaps it would be better to start this discussion on the mailing list instead.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On 10/28/2013 06:26 PM, Myklebust, Trond wrote:
>
> That battle may have been fought and won within the glusterfs community, but why should we wave the white flag without a discussion? I don't see how what he described above has anything to do with user defined attributes. He's describing how he wants to export quota information and xtime through a private xattr interface that is currently unique to glusterfs. How is that not a private syscall interface?
>
Exposing quota informtion is use "from the top". Note the other point I
mention about using NFS volumes as "gluster bricks" where we store
xattrs as dumb and persistent key/values associated with file/dir inodes
(fresh/stale info for replication, hash ranges for dirs, quota acconting
info per-dir, xtime per dir).
> Which of the mainstream filesystems have their own private xattr namespaces like the above?
>
Why should NFS need to worry? As long as it acts like a pass-through
(like every other call it supports).
Avati
On Fri, 2013-10-25 at 10:08 -0400, J. Bruce Fields wrote:
> Can you give any more details about your use case? (E.g. is there an
> article describing this system somewhere?)
Okay let me try to explain the motivation more elaborately.
The general idea is getting data integrity, i.e. being able to see
whether some files are in a valid state.
This is similar to what e.g. btrfs/zfs provide at (IIRC block/extent
level) with checksuming.
But a) btrfs is not yet fully production ready (IMHO) and zfs in Linux
has it's (license) issues and more important b) the checksuming there
happens always and automatically as soon as some valid filesystem
operations occur on the files.
So what you basically notice are errors on the physical medium (or at
least on block layers below the filesystem).
You won't notice many other cases of file "corruptions":
- when you have programs which do subtly manipulate the files and you
don't notice,...e.g. some image organisation programs store their
meta-data crap in the Exif/XMP fields, even when you don't actively tell
them to do so (and sometimes even when the files are read-only)
- when there are e.g. kernel bugs (in some other places of the kernel)
and you copy the files around. Some years ago I found a bug (not the
solution) where single bit errors happened more or less randomly every
40-100 GB of writing data. In the end it was found to be a IOMMU related
bug and a single one liner of flushing some buffers cleared the problem.
Since such errors would happen already at earlier stages, when writing
such files btrfs/zfs would simply write the checksums of the corrupted
data.
So the idea of my integrity data is, that I really manually say "now the
data is in the state where I consider it to be consistent and I want to
have checksums stored and attached to the files, for exactly that
state", e.g. after I have read out some images from the SD card (perhaps
even twice with the cache cleared and the results diffed) and placed in
my archive.
Afterwards I can regularly verify the whole archive and if at some stage
corruptions as the above would have happened, I can simply take the
respective files from backups.
Obviously that cannot be stored *in* the files,... and a database
solution fails since, it wouldn't track the changes when I move files
around within the archive.
So IMHO the best solution were XATTRs, which do work fine for that
purpose... just the problem that I can't have it via NFS ;)
Cheers,
Chris.
Hi,
Surely you don't need NFSv4 to standardize the the (empty?) set of system or magic attributes Linux
should export?
Besides, as you're well aware, most people who ask for xattrs are looking for an ability to associate
arbitrary specific data, not a back door ioctl interface. That's clearly what the NFSv4 named attributes as standardized were intended for.
I'm well aware of other uses and plans that someone would want to standardize, but it seems
irrelevant to the discussion.
Matt
----- "Trond Myklebust" <[email protected]> wrote:
>
> Standardise those xattrs, and we can export them specifically as part
> of the NFSv4 spec.
>
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
Hi, Marc.
At this point the agenda is full .. based on maximum time. But I am open to shifting things around a little if the other presenters are willing to adjust their time allotment.
I suggest that you and I work this together off the cc'd aliases and you can send a summary when we work out the details.
Spencer
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf
> Of Marc Eshel
> Sent: Monday, October 28, 2013 1:44 PM
> To: [email protected]
> Cc: Mailing List Linux NFS; Christoph Anton Mitterer; Myklebust, Trond; Steve
> Dickson; [email protected]; Dr Fields James Bruce; Ric Wheeler; linux-nfs-
> [email protected]
> Subject: Re: [nfsv4] XATTRs in NFS?
>
> Hi Spencer,
> Is it still possible to get on the agenda fir IETF 88, I just got approval to travel.
> We can use about 15 minutes, or what ever is available to discussion the
> future of named attributes in NFS. The two main questions that we need
> answer are:
> 1. Do we need them? what applications use them and how.
> 2. Can we have a more simple model that handles just user attributes.
>
> Input is welcome to help me make the case at the meeting.
>
> Thanks, Marc.
>
>
>
>
> From: Ric Wheeler <[email protected]>
> To: Dr Fields James Bruce <[email protected]>,
> Cc: "Myklebust, Trond" <[email protected]>, Christoph
> Anton
> Mitterer <[email protected]>, Mailing List Linux NFS <linux-
> [email protected]>, Steve Dickson <[email protected]>
> Date: 10/28/2013 11:32 AM
> Subject: Re: XATTRs in NFS?
> Sent by: [email protected]
>
>
>
> On 10/28/2013 02:08 PM, Dr Fields James Bruce wrote:
> > On Mon, Oct 28, 2013 at 02:00:58PM -0400, Ric Wheeler wrote:
> >> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
> >>> On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer
> <[email protected]> wrote:
> >>>
> >>>> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
> >>>>> Then you end up with large directories and an extra name per inode
> that needs to
> >>>>> be stored and extra lookups for each file when you do a whole file
> system crawl.
> >>>>>
> >>>>> Certainly not as easy as adding and xattrs with that information :)
> >>>> And I think there's another reason why it wouldn't work...
> >>>>
> >>>> Imagine I change my system to encode what should be XATTRs in
> hardlink
> >>>> pseudo files...
> >>>>
> >>>> If I have such pair locally e.g. on my ext4:
> >>>> /foo/bar/actual/file
> >>>> /meta/<SHA512 identifier>.2342348324
> >>>>
> >>>> And now move/copy the file via the network to the archive, I'd have
> to
> >>>> copy both files (which is really annoying), and I'd guess the inode
> >>>> coupling would get los (and at least the name wouldn't fit anymore).
> >>>>
> >>>> So the whole thing is IMHO not even a workaround.
> >>> OK. So you're going to do XATTRs for us?
> >>>
> >>> Trond
> >> Now that pNFS is perfect and labeled NFS has made it upstream, I
> >> think that Steve D must be looking for something to keep him busy :)
> > I agree with Trond that we first really need good evidence about exactly
> > who wants this and why.
> >
> > --b.
>
> For the user space xattrs, many applications store various types of
> metadata.
> Gluster for example heavily uses xattrs, other programs do things like
> data
> scrubbing (look for a long unchanged file, compute a has and store it as
> an
> xattr) or simply use it to annotate the file with the name of the program
> that
> created it. Think of it as file decorations or annotations.
>
> Today, if we store files in NFS that have xattrs, we do in fact cause data
> loss.
>
> I can understand an answer of "this would be hard to do for NFS and need
> to go
> through IETF" but think that xattrs are well enough established in Linux
> and
> supported in the tool chain that it is way too late to question whether or
> not
> supporting them is a worth our time.
>
> Ric
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> _______________________________________________
> nfsv4 mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/nfsv4
On Sun, 2013-10-27 at 15:15 -0400, J. Bruce Fields wrote:
> Was this problem actually caught using checksums stored in xattrs, or
> did the problem predate your use of xattrs?
Phew... don't remember actually... but I think I haven't used that
already back then and noticed it by chance when I did some diffs.
> > So the idea of my integrity data is, that I really manually say "now the
> > data is in the state where I consider it to be consistent and I want to
> > have checksums stored and attached to the files, for exactly that
> > state", e.g. after I have read out some images from the SD card (perhaps
> > even twice with the cache cleared and the results diffed) and placed in
> > my archive.
> > Afterwards I can regularly verify the whole archive and if at some stage
> > corruptions as the above would have happened, I can simply take the
> > respective files from backups.
> How long have you been using this for?
Uhm... about 3-4 years now.
> How many problems has it caught?
I do not keep exact statistics... but I remember a few cases where I
found damaged backups (optimal media) which I replaced as a consequence.
> How often do you checksum or verify files, and how expensive is that?
Not that often,... on my actual data disks,... about every 4 months...
on my backup media (I use to keep older generations of backups as well)
about once a year.
I've never really looked at how expensive it is,... all you need to do
is simply reading all data + have their hashes calculated.
Cheers,
Chris.
On Mon, 2013-10-28 at 00:19 +0000, Myklebust, Trond wrote:
> Ordinary groups can do the same.
For sure not,.. a) groups are already used for something else (namely
the normal semantics of groups, and the sGroup does not necessarily need
to correspond to the UNIX owning group (actually it usually does not).
b) sGroup is only one of several tags,... there are tags which control
space tokens, there are tags which contain dCache's own file checksums,
etc. pp.
Cheers,
Chris.
On Mon, 2013-10-28 at 00:17 +0000, Myklebust, Trond wrote:
> ...and if the checksums are any good, then all you need to do to
> substitute a database is to realise that a good data checksum is
> invariant under renames.
Don't quite see what you mean...
Sure the checksums stay the same, but consider you have many millions of
files, and you moved them around and thus the paths in the DB are
incorrect... verifying the files will become very much a pain in the
a**, especially when multiple files don't verify anymore.
Or what if you have many small similar files, where errors could lead to
a checksum that was a correct one for another file,... when the paths
are no longer valid you cannot know if this was a correct file or not.
Cheers,
Chris.
On Thu, 2013-10-24 at 08:45 +0000, Myklebust, Trond wrote:
> labeled NFS (i.e. security labels for NFS) is already supported in Linux 3.10 and newer.
Sure, but that doesn't really help me.
> There are no plans to merge general purpose xattrs.
Why not? Is it a big deal?
> Please just use an application-specific database.
Well that won't work,... since that wouldn't be updated if e.g.
pathnames are changed by any program (cp, mv)
Cheers,
Chris.
Hi,
Sorry I was not clear. I was not re-stating support for this banned feature.
I'm interested in implementation code I can use privately under GPLv2.
Thanks!
Matt
----- "Trond Myklebust" <[email protected]> wrote:
> On Thu, 2013-10-24 at 11:05 -0400, Matt W. Benjamin wrote:
> > Hi,
> >
> > Presumably Linux client and server implementations would restrict
> use of "system."
> > Perhaps would be more to talk about, if xattrs were implemented
> experimentally.
> >
> > I'm interested in knowing about patches that might be floating
> around for the Linux client, for NFSv4.x. (I've seen NFSv3 work.)
> >
> > Thanks,
> >
> > Matt
>
> We've already talked about this, Matt. The answer is still NO.
>
> Trond
>
> > ----- "Trond Myklebust" <[email protected]> wrote:
> >
> > > On Oct 24, 2013, at 3:13 PM, Christoph Anton Mitterer
> > > <[email protected]> wrote:
> > >
> > > > On Thu, 2013-10-24 at 08:45 +0000, Myklebust, Trond wrote:
> > > >> labeled NFS (i.e. security labels for NFS) is already supported
> in
> > > Linux 3.10 and newer.
> > > > Sure, but that doesn't really help me.
> > > >
> > > >
> > > >> There are no plans to merge general purpose xattrs.
> > > > Why not? Is it a big deal?
> > > >
> > >
> > > Linux xattrs are a rabid mess.
> > >
> > > The whole "system" namespace is something that cannot and should
> not
> > > ever be exposed on a network.
> > > The "trusted" and "user" namespaces just offer specialised
> storage.
> > > Why are they needed?
> > >
> > > >
> > > >> Please just use an application-specific database.
> > > > Well that won't work,... since that wouldn't be updated if e.g.
> > > > pathnames are changed by any program (cp, mv)
> > >
> > > If the data needs to follow the file, then store it in the file.
> Why
> > > do you need the filesystem to manage that for you?
> > >
> > > Cheers
> > > Trond--
> > > To unsubscribe from this list: send the line "unsubscribe
> linux-nfs"
> > > in
> > > the body of a message to [email protected]
> > > More majordomo info at
> http://vger.kernel.org/majordomo-info.html
> >
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> [email protected]
> http://www.netapp.com
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
On Sun, Oct 27, 2013 at 06:07:46PM +0000, Myklebust, Trond wrote:
> The user xattrs are in principle harmless. What about "trusted.*"?
trusted ones are still fairly harmless and exist in most implementations
under a slightly different name. They aren't quite as essential,
though.
The important point is to not carry over the system and security namespacs.
Those are a bad and Linux specific mistake to abuse the xattr interface for
all kinds of other interfaces. This isn't something that should be carried
over to other standards.
Hi,
Presumably Linux client and server implementations would restrict use of "system."
Perhaps would be more to talk about, if xattrs were implemented experimentally.
I'm interested in knowing about patches that might be floating around for the Linux client, for NFSv4.x. (I've seen NFSv3 work.)
Thanks,
Matt
----- "Trond Myklebust" <[email protected]> wrote:
> On Oct 24, 2013, at 3:13 PM, Christoph Anton Mitterer
> <[email protected]> wrote:
>
> > On Thu, 2013-10-24 at 08:45 +0000, Myklebust, Trond wrote:
> >> labeled NFS (i.e. security labels for NFS) is already supported in
> Linux 3.10 and newer.
> > Sure, but that doesn't really help me.
> >
> >
> >> There are no plans to merge general purpose xattrs.
> > Why not? Is it a big deal?
> >
>
> Linux xattrs are a rabid mess.
>
> The whole "system" namespace is something that cannot and should not
> ever be exposed on a network.
> The "trusted" and "user" namespaces just offer specialised storage.
> Why are they needed?
>
> >
> >> Please just use an application-specific database.
> > Well that won't work,... since that wouldn't be updated if e.g.
> > pathnames are changed by any program (cp, mv)
>
> If the data needs to follow the file, then store it in the file. Why
> do you need the filesystem to manage that for you?
>
> Cheers
> Trond--
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
On Thu, 2013-10-24 at 11:23 -0400, Jeff Layton wrote:
+AD4- On Thu, 24 Oct 2013 11:16:10 -0400
+AD4- Simo Sorce +ADw-simo+AEA-redhat.com+AD4- wrote:
+AD4-
+AD4- +AD4- On Thu, 2013-10-24 at 15:11 +-0000, Myklebust, Trond wrote:
+AD4- +AD4- +AD4- On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:
+AD4- +AD4- +AD4-
+AD4- +AD4- +AD4- +AD4- Because the filesystem can do that when multiple applications are
+AD4- +AD4- +AD4- +AD4- involved without having to change them all to talk to each other and
+AD4- +AD4- +AD4- +AD4- invent custom protocol all the time just to keep some additional
+AD4- +AD4- +AD4- +AD4- metadata associated to a file..
+AD4- +AD4- +AD4- +AD4-
+AD4- +AD4- +AD4- It's still a custom protocol. The applications need to agree on a data
+AD4- +AD4- +AD4- format and store it somewhere. The portable way to do this is to write
+AD4- +AD4- +AD4- an application library that they can link to.
+AD4- +AD4-
+AD4- +AD4- Perhaps I was unclear, you are never going to see that custom library
+AD4- +AD4- linked into the 'mv' command.
+AD4- +AD4-
+AD4- +AD4- So your approach makes little sense if the object is to maintain data
+AD4- +AD4- coherent when people need to handle files from random applications and
+AD4- +AD4- scripts and general system maintenance.
+AD4- +AD4-
+AD4- +AD4- The data may be relevant only to a specific application.
+AD4- +AD4-
+AD4- +AD4- I am not saying you +ACo-have+ACo- to implement xattrs support, just saying that
+AD4- +AD4- it is not a mere 'applications should synchronize data themselves'
+AD4- +AD4- problem.
+AD4- +AD4-
+AD4-
+AD4- I think the real solution if people need this is to lead an effort to
+AD4- put xattrs into the spec. I think there is still time to get new
+AD4- features into v4.3 if someone wants to champion it...
+AD4-
How would that help? Witness Oracle's success with named attributes,
which are +AF8-also+AF8- a non-standard filesystem feature that was hastily
pushed into the NFSv4 spec.
If you really need this for use by applications (as opposed to by
sysadmins - see labeled NFS), then get the functionality into POSIX
first, then add it to the NFS spec.
--
Trond Myklebust
Linux NFS client maintainer
NetApp
Trond.Myklebust+AEA-netapp.com
http://www.netapp.com
On Oct 28, 2013, at 9:24 PM, Anand Avati <[email protected]> wrote:
> On 10/28/2013 06:26 PM, Myklebust, Trond wrote:
>
>>
>> That battle may have been fought and won within the glusterfs community, but why should we wave the white flag without a discussion? I don't see how what he described above has anything to do with user defined attributes. He's describing how he wants to export quota information and xtime through a private xattr interface that is currently unique to glusterfs. How is that not a private syscall interface?
>>
>
> Exposing quota informtion is use "from the top". Note the other point I mention about using NFS volumes as "gluster bricks" where we store xattrs as dumb and persistent key/values associated with file/dir inodes (fresh/stale info for replication, hash ranges for dirs, quota acconting info per-dir, xtime per dir).
>> Which of the mainstream filesystems have their own private xattr namespaces like the above?
>>
>
> Why should NFS need to worry? As long as it acts like a pass-through (like every other call it supports).
We need to worry because we don't know what side-effects your private interface will have. How does it affect our caching model? How do we debug any problems that arise? Are there any security implications that we need to know about?
Trond
"J. Bruce Fields" <[email protected]> wrote on 10/28/2013 02:02:47 PM:
> From: "J. Bruce Fields" <[email protected]>
> To: "Haynes, Tom" <[email protected]>,
> Cc: Marc Eshel/Almaden/IBM@IBMUS, Spencer Shepler
> <[email protected]>, Mailing List Linux NFS <linux-
> [email protected]>, Christoph Anton Mitterer
> <[email protected]>, "Myklebust, Trond"
> <[email protected]>, Steve Dickson <[email protected]>,
> "[email protected] [email protected]" <[email protected]>, Ric Wheeler
> <[email protected]>, "[email protected]" <linux-
> [email protected]>
> Date: 10/28/2013 02:02 PM
> Subject: Re: [nfsv4] XATTRs in NFS?
>
> On Mon, Oct 28, 2013 at 08:55:06PM +0000, Haynes, Tom wrote:
> >
> > On Oct 28, 2013, at 1:44 PM, Marc Eshel <[email protected]> wrote:
> >
> > > Hi Spencer,
> > > Is it still possible to get on the agenda fir IETF 88, I just
> got approval
> > > to travel. We can use about 15 minutes, or what ever is available to
> > > discussion the future of named attributes in NFS. The two main
questions
> > > that we need answer are:
> > > 1. Do we need them? what applications use them and how.
> > > 2. Can we have a more simple model that handles just user
attributes.
> > >
> > > Input is welcome to help me make the case at the meeting.
> > >
> > > Thanks, Marc.
> > >
> >
> >
> > Be sure to call out why they currently do or do not work.
>
> And we also need to keep clear the distinctions between:
>
> - NFSv4 named attributes/streams/forks and "extended attributes"
> as implemented in Linux and elsewhere.
> - user.* xattrs and "back-door ioctls".
>
> (Your description above seems to confuse named attributes and Linux
> extended attributes.)
Yes, I used user which is the Linux type of xattr but what I meant is
somthing like the Linux user attributes with no special meaning to NFS, so
NFS can be used to just transfer opaque attributes.
Marc.
>
> --b.
>
[email protected] wrote on 10/28/2013 02:04:57 PM:
> From: Chuck Lever <[email protected]>
> To: "J. Bruce Fields" <[email protected]>,
> Cc: "Haynes, Tom" <[email protected]>, Marc Eshel/Almaden/
> IBM@IBMUS, Spencer Shepler <[email protected]>, Mailing List
> Linux NFS <[email protected]>, Christoph Anton Mitterer
> <[email protected]>, "Myklebust, Trond"
> <[email protected]>, Steve Dickson <[email protected]>,
> "[email protected] [email protected]" <[email protected]>, Ric Wheeler
> <[email protected]>, "[email protected]" <linux-
> [email protected]>
> Date: 10/28/2013 02:07 PM
> Subject: Re: [nfsv4] XATTRs in NFS?
> Sent by: [email protected]
>
>
> On Oct 28, 2013, at 5:02 PM, "J. Bruce Fields" <[email protected]>
wrote:
>
> > On Mon, Oct 28, 2013 at 08:55:06PM +0000, Haynes, Tom wrote:
> >>
> >> On Oct 28, 2013, at 1:44 PM, Marc Eshel <[email protected]> wrote:
> >>
> >>> Hi Spencer,
> >>> Is it still possible to get on the agenda fir IETF 88, I just
> got approval
> >>> to travel. We can use about 15 minutes, or what ever is available to
> >>> discussion the future of named attributes in NFS. The two main
questions
> >>> that we need answer are:
> >>> 1. Do we need them? what applications use them and how.
> >>> 2. Can we have a more simple model that handles just user
attributes.
> >>>
> >>> Input is welcome to help me make the case at the meeting.
> >>>
> >>> Thanks, Marc.
> >>>
> >>
> >>
> >> Be sure to call out why they currently do or do not work.
> >
> > And we also need to keep clear the distinctions between:
> >
> > - NFSv4 named attributes/streams/forks and "extended attributes"
> > as implemented in Linux and elsewhere.
> > - user.* xattrs and "back-door ioctls".
> >
> > (Your description above seems to confuse named attributes and Linux
> > extended attributes.)
>
> I wonder if we will have a balanced discussion next week. The
> people who seem to know the most about these issues on Linux will
> not be attending. Perhaps it would be better to start this
> discussion on the mailing list instead.
Like I said input is welcome to the mailing list and I can try to collect
all the main points into a presentation at the IETF. The main objective
for now is to see if this is an item for the working group to start
dealing with for future NFS protocol enhancements.
Marc.
>
> --
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>