I was testing using the rootdir setting in nfs.config to allow using export
paths that would normally conflict with local systems directories with NFS v3.
The idea was to support re-exporting a source NFS server such as NetApp that
might exports arbitrary paths such as /home without overwriting the /home
directory on the NFS server, and even support exporting a root directory similar
to NFS v4.
While testing I ran into an issue where the NFS server would fail to start.
During start up, the NFS server would log the error:
$ systemctl status nfs-server.service
systemd[1]: Starting NFS server and services...
exportfs[2307]: exportfs: Failed to stat /srv/nfs/usr/bin: No such
file or directory
systemd[1]: nfs-server.service: Control process exited, code=exited,
status=1/FAILURE
$ cat nfs.config
[exports]
rootdir=/srv/nfs
$ cat /etc/exports
/ 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,fsid=0,nohide)
/assets 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,fsid=10,nohide)
/bin 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,fsid=30,nohide)
/software 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,fsid=40,nohide)
If I create the directory /srv/nfs/usr/bin then the NFS server will start.
Listing the actual exports shows that a different path was exported compared to
the path from /etc/exports.
$ exportfs -s
/ 10.0.0.0/8(sync,wdelay,nohide,no_subtree_check,fsid=0,sec=sys,rw,secure,no_root_squash,no_all_squ>
/assets 10.0.0.0/8(sync,wdelay,nohide,no_subtree_check,fsid=10,sec=sys,rw,secure,no_root_squash,no_>
/usr/bin 10.0.0.0/8(sync,wdelay,nohide,no_subtree_check,fsid=30,sec=sys,rw,secure,no_root_squash,no>
/software 10.0.0.0/8(sync,wdelay,nohide,no_subtree_check,fsid=40,sec=sys,rw,secure,no_root_squash,n>
To test this further I create a symlink from /software to /usr/lib. Once again
the server failed to start because it could not find /srv/nfs/usr/lib.
Reading through the source, I think the issue is in the getexportent function in
support/nfs/exports.c.
/* resolve symlinks */
if (realpath(ee.e_path, rpath) != NULL) {
rpath[sizeof (rpath) - 1] = '\0';
strncpy(ee.e_path, rpath, sizeof (ee.e_path) - 1);
ee.e_path[sizeof (ee.e_path) - 1] = '\0';
}
It appears this function does not take into account the rootdir property when
resolving e_path.
This was tested on Ubuntu 20.04 with the 5.11.8-051108-generic kernel. nfs-utils
version is 2.5.3.
On Fri, Jan 14, 2022 at 12:36:23PM +0000, Chris Chilvers wrote:
> I was testing using the rootdir setting in nfs.config to allow using export
> paths that would normally conflict with local systems directories with NFS v3.
>
> The idea was to support re-exporting a source NFS server such as NetApp that
> might exports arbitrary paths such as /home without overwriting the /home
> directory on the NFS server, and even support exporting a root directory similar
> to NFS v4.
>
> While testing I ran into an issue where the NFS server would fail to start.
> During start up, the NFS server would log the error:
>
> $ systemctl status nfs-server.service
> systemd[1]: Starting NFS server and services...
> exportfs[2307]: exportfs: Failed to stat /srv/nfs/usr/bin: No such
> file or directory
> systemd[1]: nfs-server.service: Control process exited, code=exited,
> status=1/FAILURE
>
> $ cat nfs.config
> [exports]
> rootdir=/srv/nfs
>
> $ cat /etc/exports
> / 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,fsid=0,nohide)
> /assets 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,fsid=10,nohide)
> /bin 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,fsid=30,nohide)
> /software 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,fsid=40,nohide)
>
> If I create the directory /srv/nfs/usr/bin then the NFS server will start.
> Listing the actual exports shows that a different path was exported compared to
> the path from /etc/exports.
>
> $ exportfs -s
> / 10.0.0.0/8(sync,wdelay,nohide,no_subtree_check,fsid=0,sec=sys,rw,secure,no_root_squash,no_all_squ>
> /assets 10.0.0.0/8(sync,wdelay,nohide,no_subtree_check,fsid=10,sec=sys,rw,secure,no_root_squash,no_>
> /usr/bin 10.0.0.0/8(sync,wdelay,nohide,no_subtree_check,fsid=30,sec=sys,rw,secure,no_root_squash,no>
> /software 10.0.0.0/8(sync,wdelay,nohide,no_subtree_check,fsid=40,sec=sys,rw,secure,no_root_squash,n>
>
> To test this further I create a symlink from /software to /usr/lib. Once again
> the server failed to start because it could not find /srv/nfs/usr/lib.
I'm probably just reading too quickly, but I'm not seeing how this
explains the problems with your original configuration.
Is it that /sr/nfs/bin on you system is a symlink? (And what exactly is
the content of that symlink?)
(Also, for what it's worth, we don't recommend using fsid=0 any more.
It should be unnecessary.)
--b.
>
> Reading through the source, I think the issue is in the getexportent function in
> support/nfs/exports.c.
>
> /* resolve symlinks */
> if (realpath(ee.e_path, rpath) != NULL) {
> rpath[sizeof (rpath) - 1] = '\0';
> strncpy(ee.e_path, rpath, sizeof (ee.e_path) - 1);
> ee.e_path[sizeof (ee.e_path) - 1] = '\0';
> }
>
> It appears this function does not take into account the rootdir property when
> resolving e_path.
>
> This was tested on Ubuntu 20.04 with the 5.11.8-051108-generic kernel. nfs-utils
> version is 2.5.3.
> I'm probably just reading too quickly, but I'm not seeing how this
> explains the problems with your original configuration.
>
> Is it that /sr/nfs/bin on you system is a symlink? (And what exactly is
> the content of that symlink?)
No, all the directories under /srv/nfs are ordinary directories. The symlinks
are in the root of the real file system.
So I have:
/bin -> /usr/bin
/software -> /usr/lib (created to test the hypothesis)
/srv/nfs/
/srv/nfs/assets
/srv/nfs/bin
/srv/nfs/software
Because exportfs is not taking into account the rootdir setting, when it tries
to resolve the path from /etc/exports, it sees the path /bin, and matches that
with the real /bin, and resolves it to /usr/bin.
Later exportfs errors when trying to resolve the real path (e_realpath) in the
exportent_mkrealpath function, as this function does prepend the rootdir. This
causes exportfs to look for /srv/nfs/usr/bin instead of the expected
/srv/nfs/bin.
At best this causes an error, but if that does happen to match an existing path
under /srv/nfs then it would export the wrong directory.
This will happen for any export that happens to match an existing symlink.
There are two issues though that I'm not sure how best to handle:
* Should the symlink be considered relative to the local system or relative to
the rootdir?
* What happens if the symlink points to a directory outside the rootdir?
In my case I think the symlink would need to be resolved relative to the
rootdir. This is because we're re-exporting an existing NFS server, so if the
existing NFS server is exporting some kind of symlink tree, the symlinks would
be relative to the existing mounts.
One option that would be valid for my case would be an setting to disable
symlink resolution and just consider it an error if an export is a symlink.
Since I'm re-exporting an existing NFS server, they can't be symlinks anyway.
-- Chris
On Fri, Jan 14, 2022 at 03:57:03PM +0000, Chris Chilvers wrote:
> > I'm probably just reading too quickly, but I'm not seeing how this
> > explains the problems with your original configuration.
> >
> > Is it that /sr/nfs/bin on you system is a symlink? (And what exactly is
> > the content of that symlink?)
>
> No, all the directories under /srv/nfs are ordinary directories. The symlinks
> are in the root of the real file system.
Oh, got it, thanks for the explanation.
> So I have:
> /bin -> /usr/bin
> /software -> /usr/lib (created to test the hypothesis)
> /srv/nfs/
> /srv/nfs/assets
> /srv/nfs/bin
> /srv/nfs/software
>
> Because exportfs is not taking into account the rootdir setting, when it tries
> to resolve the path from /etc/exports, it sees the path /bin, and matches that
> with the real /bin, and resolves it to /usr/bin.
>
> Later exportfs errors when trying to resolve the real path (e_realpath) in the
> exportent_mkrealpath function, as this function does prepend the rootdir. This
> causes exportfs to look for /srv/nfs/usr/bin instead of the expected
> /srv/nfs/bin.
>
> At best this causes an error, but if that does happen to match an existing path
> under /srv/nfs then it would export the wrong directory.
>
> This will happen for any export that happens to match an existing symlink.
>
> There are two issues though that I'm not sure how best to handle:
> * Should the symlink be considered relative to the local system or relative to
> the rootdir?
> * What happens if the symlink points to a directory outside the rootdir?
Yes, I'm not sure off hand what the right behavior is, but the existing
behavior certainly seems to violate the principal of least surprise.
--b.
>
> In my case I think the symlink would need to be resolved relative to the
> rootdir. This is because we're re-exporting an existing NFS server, so if the
> existing NFS server is exporting some kind of symlink tree, the symlinks would
> be relative to the existing mounts.
>
> One option that would be valid for my case would be an setting to disable
> symlink resolution and just consider it an error if an export is a symlink.
> Since I'm re-exporting an existing NFS server, they can't be symlinks anyway.
>
> -- Chris