Hi,
I have a somewhat odd use case that I think has some room for improvement.
I have a single fs exported on the server (/export/data). On the client I have mount /export/data/a and /export/data/b. Doing a rename("/export/data/a/foo", "/export/data/b/foo") results in -EXDEV.
Now this is a bit annoying from a program point of view as it is very difficult to detect this beforehand. Doing a stat() on source and destination indicates that they are the same fs. So it would seem POSIX would guarantee I would not get a -EXDEV in that case. I assume this is simply a local, client-side check that determines that two different mount entries are involved.
So, would it be possible to actually make the kernel realise that the source and target are in face the same fs on the server (so a NFS rename will succeed)? Or at least make the kernel present different device st_dev for the two trees.
Rgds
--
-- Pierre Ossman
Linux kernel, MMC maintainer http://www.kernel.org
PulseAudio, core developer http://pulseaudio.org
rdesktop, core developer http://www.rdesktop.org
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Mon, Feb 11, 2008 at 11:21:36AM -0500, J. Bruce Fields wrote:
> > http://verein.lst.de/~hch/xfs/xfs-multiple-containers.txt
>
> Excellent, thanks, yes that's just what we'd want....
The proposal still lacks the idea of encoding the projid in the file
handle. But I think we could do that quite easily, but I'd do it after
my planned next round of the export interface cleanups, so that a
filehandle can see at mount time how much of the filehandle space it
has left and refuse the export if it doesn't fit (e.g. exporting with
various 64bit inode fields and NFSv2, or projids on NFSv2)
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sat, Feb 09, 2008 at 08:05:03PM +0100, Pierre Ossman wrote:
> Hi,
>
> I have a somewhat odd use case that I think has some room for
> improvement.
>
> I have a single fs exported on the server (/export/data). On the
> client I have mount /export/data/a and /export/data/b. Doing a
> rename("/export/data/a/foo", "/export/data/b/foo") results in -EXDEV.
I'm afraid the answer is likely "don't do that!"
>
> Now this is a bit annoying from a program point of view as it is very
> difficult to detect this beforehand. Doing a stat() on source and
> destination indicates that they are the same fs. So it would seem
> POSIX would guarantee I would not get a -EXDEV in that case. I assume
> this is simply a local, client-side check that determines that two
> different mount entries are involved.
>
> So, would it be possible to actually make the kernel realise that the
> source and target are in face the same fs on the server (so a NFS
> rename will succeed)? Or at least make the kernel present different
> device st_dev for the two trees.
I believe the same thing happens even with --bind mounts of the same
local filesystem, and that it's by design. See this discussion:
http://marc.info/?t=119801816800006&r=1&w=2
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sat, 2008-02-09 at 20:05 +0100, Pierre Ossman wrote:
> Hi,
>
> I have a somewhat odd use case that I think has some room for improvement.
>
> I have a single fs exported on the server (/export/data). On the client I have mount /export/data/a and /export/data/b. Doing a rename("/export/data/a/foo", "/export/data/b/foo") results in -EXDEV.
>
> Now this is a bit annoying from a program point of view as it is very difficult to detect this beforehand. Doing a stat() on source and destination indicates that they are the same fs. So it would seem POSIX would guarantee I would not get a -EXDEV in that case. I assume this is simply a local, client-side check that determines that two different mount entries are involved.
>
> So, would it be possible to actually make the kernel realise that the source and target are in face the same fs on the server (so a NFS rename will succeed)? Or at least make the kernel present different device st_dev for the two trees.
>
> Rgds
Those semantics are _exactly_ the same as if you were to mount --bind
the same filesystem to two different mountpoints. This is not an NFS
bug, but rather a deliberate design within the VFS in order to avoid
mutex locking deadlocks during rename.
Cheers
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sat, 9 Feb 2008 14:10:21 -0500
"J. Bruce Fields" <[email protected]> wrote:
> On Sat, Feb 09, 2008 at 08:05:03PM +0100, Pierre Ossman wrote:
> > Hi,
> >
> > I have a somewhat odd use case that I think has some room for
> > improvement.
> >
> > I have a single fs exported on the server (/export/data). On the
> > client I have mount /export/data/a and /export/data/b. Doing a
> > rename("/export/data/a/foo", "/export/data/b/foo") results in -EXDEV.
>
> I'm afraid the answer is likely "don't do that!"
>
I don't think I find that an acceptable answer. Giving all systems complete access to your entire server side storage isn't really acceptable in many situations.
> >
> > So, would it be possible to actually make the kernel realise that the
> > source and target are in face the same fs on the server (so a NFS
> > rename will succeed)? Or at least make the kernel present different
> > device st_dev for the two trees.
>
> I believe the same thing happens even with --bind mounts of the same
> local filesystem, and that it's by design. See this discussion:
>
> http://marc.info/?t=119801816800006&r=1&w=2
>
They make some sound arguments for it, so I guess the optimisation is out of the question. I also couldn't find anything in POSIX linking rename() and st_dev, so it should be fine from a strict interpretation of the spec. Still, it is a bit unexpected to get -EXDEV when st_dev is the same for both points.
I though the NFS client allocates "fake" st_dev numbers to guarantee the st_dev/st_ino uniqueness, so how come each mount point doesn't get a new one?
Rgds
--
-- Pierre Ossman
Linux kernel, MMC maintainer http://www.kernel.org
PulseAudio, core developer http://pulseaudio.org
rdesktop, core developer http://www.rdesktop.org
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sat, 2008-02-09 at 21:35 +0100, Pierre Ossman wrote:
> > >
> > > So, would it be possible to actually make the kernel realise that the
> > > source and target are in face the same fs on the server (so a NFS
> > > rename will succeed)? Or at least make the kernel present different
> > > device st_dev for the two trees.
> >
> > I believe the same thing happens even with --bind mounts of the same
> > local filesystem, and that it's by design. See this discussion:
> >
> > http://marc.info/?t=119801816800006&r=1&w=2
> >
>
> They make some sound arguments for it, so I guess the optimisation is out of the question. I also couldn't find anything in POSIX linking rename() and st_dev, so it should be fine from a strict interpretation of the spec. Still, it is a bit unexpected to get -EXDEV when st_dev is the same for both points.
>
> I though the NFS client allocates "fake" st_dev numbers to guarantee the st_dev/st_ino uniqueness, so how come each mount point doesn't get a new one?
It has the same syntax and properties as a bind mount. If you don't like
it, then fix the VFS.
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sat, 09 Feb 2008 15:40:11 -0500
Trond Myklebust <[email protected]> wrote:
>
> It has the same syntax and properties as a bind mount. If you don't like
> it, then fix the VFS.
>
So it detects that it is the same server and server side fs and reuses the super block?
> Those semantics are _exactly_ the same as if you were to mount --bind
> the same filesystem to two different mountpoints. This is not an NFS
> bug, but rather a deliberate design within the VFS in order to avoid
> mutex locking deadlocks during rename.
Do you have a reference for that? The previous thread linked in suggested it was a security feature.
Rgds
--
-- Pierre Ossman
Linux kernel, MMC maintainer http://www.kernel.org
PulseAudio, core developer http://pulseaudio.org
rdesktop, core developer http://www.rdesktop.org
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sat, Feb 09, 2008 at 09:35:11PM +0100, Pierre Ossman wrote:
> On Sat, 9 Feb 2008 14:10:21 -0500
> "J. Bruce Fields" <[email protected]> wrote:
>
> > On Sat, Feb 09, 2008 at 08:05:03PM +0100, Pierre Ossman wrote:
> > > Hi,
> > >
> > > I have a somewhat odd use case that I think has some room for
> > > improvement.
> > >
> > > I have a single fs exported on the server (/export/data). On the
> > > client I have mount /export/data/a and /export/data/b. Doing a
> > > rename("/export/data/a/foo", "/export/data/b/foo") results in -EXDEV.
> >
> > I'm afraid the answer is likely "don't do that!"
> >
>
> I don't think I find that an acceptable answer. Giving all systems complete access to your entire server side storage isn't really acceptable in many situations.
I'd be curious to know what you're actually trying to accomplish. If
you're mounting subdirectories of the same server filesystem, then you
need to be aware that, for example, anyone with access to the network
can probably access that whole filesystem by guessing filehandles (which
is very easy).
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sat, 9 Feb 2008 18:14:35 -0500
"J. Bruce Fields" <[email protected]> wrote:
> >
> > I don't think I find that an acceptable answer. Giving all systems complete access to your entire server side storage isn't really acceptable in many situations.
>
> I'd be curious to know what you're actually trying to accomplish. If
> you're mounting subdirectories of the same server filesystem, then you
> need to be aware that, for example, anyone with access to the network
> can probably access that whole filesystem by guessing filehandles (which
> is very easy).
>
I thought the subtree check prevented that?
The point of the setup is of course to avoid exposing more information than is necessary, while still having the benefit of having a single pool of disk space for all exported resources.
Rgds
--
-- Pierre Ossman
Linux kernel, MMC maintainer http://www.kernel.org
PulseAudio, core developer http://pulseaudio.org
rdesktop, core developer http://www.rdesktop.org
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sun, Feb 10, 2008 at 01:12:24PM +0100, Pierre Ossman wrote:
> On Sat, 9 Feb 2008 18:14:35 -0500
> "J. Bruce Fields" <[email protected]> wrote:
>
> > >
> > > I don't think I find that an acceptable answer. Giving all systems complete access to your entire server side storage isn't really acceptable in many situations.
> >
> > I'd be curious to know what you're actually trying to accomplish. If
> > you're mounting subdirectories of the same server filesystem, then you
> > need to be aware that, for example, anyone with access to the network
> > can probably access that whole filesystem by guessing filehandles (which
> > is very easy).
> >
>
> I thought the subtree check prevented that?
Yes, at the expense of odd behavior in a couple corner cases: the parent
directory is encoded in the filehandle, so, for example, a client
holding a file open will get ESTALE if it moves to a different
directory.
> The point of the setup is of course to avoid exposing more information
> than is necessary, while still having the benefit of having a single
> pool of disk space for all exported resources.
I notice that btrfs claims support for "subvolumes"--multiple filesystem
namespaces sharing the same storage. Seems like a useful feature for
this situation. I don't know whether it'd be possible to add to other
filesystems.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sun, Feb 10, 2008 at 10:26:18AM -0500, J. Bruce Fields wrote:
> > The point of the setup is of course to avoid exposing more information
> > than is necessary, while still having the benefit of having a single
> > pool of disk space for all exported resources.
>
> I notice that btrfs claims support for "subvolumes"--multiple filesystem
> namespaces sharing the same storage. Seems like a useful feature for
> this situation. I don't know whether it'd be possible to add to other
> filesystems.
That stuff is done using a code sniplet from me, which does nothing but
allowing to mount into a subdirectory of a filesystem. It will look
exactly the same as bind mounts. I have hacked up implementations for
ext2 and xfs aswell and in the form btfs does it currently it would
be doable for any filesystem. But to actually make it useful you'd
need quotas for each subvolume which is something currently only
xfs could provide (if I could be arsed into finally implementing
my multiple subvolumes proposal fully).
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sun, Feb 10, 2008 at 10:33:40AM -0500, Christoph Hellwig wrote:
> On Sun, Feb 10, 2008 at 10:26:18AM -0500, J. Bruce Fields wrote:
> > > The point of the setup is of course to avoid exposing more information
> > > than is necessary, while still having the benefit of having a single
> > > pool of disk space for all exported resources.
> >
> > I notice that btrfs claims support for "subvolumes"--multiple filesystem
> > namespaces sharing the same storage. Seems like a useful feature for
> > this situation. I don't know whether it'd be possible to add to other
> > filesystems.
>
> That stuff is done using a code sniplet from me, which does nothing but
> allowing to mount into a subdirectory of a filesystem. It will look
> exactly the same as bind mounts.
To get the equivalent of subtree checking, we also need a reference from
each inode back to the subvolume it belongs to, so we can determine the
subvolume from a filehandle. I assume that's a more intrusive change.
--b.
> I have hacked up implementations for
> ext2 and xfs aswell and in the form btfs does it currently it would
> be doable for any filesystem. But to actually make it useful you'd
> need quotas for each subvolume which is something currently only
> xfs could provide (if I could be arsed into finally implementing
> my multiple subvolumes proposal fully).
>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sun, Feb 10, 2008 at 10:57:33AM -0500, J. Bruce Fields wrote:
> To get the equivalent of subtree checking, we also need a reference from
> each inode back to the subvolume it belongs to, so we can determine the
> subvolume from a filehandle. I assume that's a more intrusive change.
XFS actually has the infrastructure for that by having a project id
that's inheritable to child directories/files. But to actuall make
that useable we'd have to encode it in the fsid. I must admit
I haven't thought that far yet.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sun, Feb 10, 2008 at 10:59:23AM -0500, Christoph Hellwig wrote:
> On Sun, Feb 10, 2008 at 10:57:33AM -0500, J. Bruce Fields wrote:
> > To get the equivalent of subtree checking, we also need a reference from
> > each inode back to the subvolume it belongs to, so we can determine the
> > subvolume from a filehandle. I assume that's a more intrusive change.
>
> XFS actually has the infrastructure for that by having a project id
> that's inheritable to child directories/files. But to actuall make
> that useable we'd have to encode it in the fsid. I must admit
> I haven't thought that far yet.
It'd be useful for nfsd. So if you have any design notes or anything
I'd be interested.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sun, Feb 10, 2008 at 01:10:53PM -0500, J. Bruce Fields wrote:
> > XFS actually has the infrastructure for that by having a project id
> > that's inheritable to child directories/files. But to actuall make
> > that useable we'd have to encode it in the fsid. I must admit
> > I haven't thought that far yet.
>
> It'd be useful for nfsd. So if you have any design notes or anything
> I'd be interested.
http://verein.lst.de/~hch/xfs/xfs-multiple-containers.txt
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Sun, Feb 10, 2008 at 01:21:21PM -0500, Christoph Hellwig wrote:
> On Sun, Feb 10, 2008 at 01:10:53PM -0500, J. Bruce Fields wrote:
> > > XFS actually has the infrastructure for that by having a project id
> > > that's inheritable to child directories/files. But to actuall make
> > > that useable we'd have to encode it in the fsid. I must admit
> > > I haven't thought that far yet.
> >
> > It'd be useful for nfsd. So if you have any design notes or anything
> > I'd be interested.
>
> http://verein.lst.de/~hch/xfs/xfs-multiple-containers.txt
Excellent, thanks, yes that's just what we'd want....
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs