2011-03-21 02:58:30

by Dennis Jacobfeuerborn

[permalink] [raw]
Subject: bind mounts, crossmnt and multi client nfsv3

Hi,
I have a storage system that is exporting many directories to multiple
clients resulting in a lot of mountpoints (156) on each client. What I'm
trying to do is to create a single directory on the server and then use
mount --bind to mount all the different directories (which are stored on
different LVM volumes) into this single export directory and finally export
this directory to the clients.
Now the exports man-page mentions the crossmnt option but it also mentions
that it cannot be used if I want to export the directory to multiple
clients. Is there another way to accomplish something like this?

Regards,
Dennis


2011-03-21 17:25:34

by J. Bruce Fields

[permalink] [raw]
Subject: Re: bind mounts, crossmnt and multi client nfsv3

On Mon, Mar 21, 2011 at 01:27:05PM +0100, Dennis Jacobfeuerborn wrote:
> On 03/21/2011 04:09 AM, NeilBrown wrote:
> >On Mon, 21 Mar 2011 03:24:50 +0100 Dennis Jacobfeuerborn
> ><[email protected]> wrote:
> >
> >>Hi,
> >>I have a storage system that is exporting many directories to multiple
> >>clients resulting in a lot of mountpoints (156) on each client. What I'm
> >>trying to do is to create a single directory on the server and then use
> >>mount --bind to mount all the different directories (which are stored on
> >>different LVM volumes) into this single export directory and finally export
> >>this directory to the clients.
> >>Now the exports man-page mentions the crossmnt option but it also mentions
> >>that it cannot be used if I want to export the directory to multiple
> >>clients. Is there another way to accomplish something like this?
> >>
> >
> >Just export the top directory with 'crossmnt' - it should work fine.
> >
> >The 'multiple clients' thing only affects 'nohide' and I think it only
> >affected it back in 2.4 days.
> >Lots changed with 2.6, but maybe not enough of the man page changed :-(
> >
> >So try with 'crossmnt' and if it doesn't work, then come back with details.
>
> I've tried this now but it doesn't seem to work. This is the setup so far:
>
> ===========
>
> On the server I've created the bind mounts like this:
>
> /mnt/vg0/vol01/country/de/a on /exports/country/de/a type none (rw,bind)
> /mnt/vg0/vol03/country/de/b on /exports/country/de/b type none (rw,bind)
> ...
>
> Then I put the following line in /etc/exports:
> /exports 192.168.0.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,no_root_squash,wdelay,sync,crossmnt,no_subtree_check)
>
> additionally these exports exist for each volume:
> /mnt/vg0/vol01/country 213.131.252.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,no_root_squash,wdelay,sync)

I'm a bit confused by the paths and ip blocks:

- You want to export the path that the client sees
(/exports/country/whatever), not the original path
(/mnt/vg0/...).
- If you expect clients to be able to traverse from /exports/ to
filesystems underneath, then you'd want to make sure /exports/
is exported to anything that the filesystems underneath are.

But actually you should only need the first /exports entry; filesystems
underneath should inherit export options from the parent.

--b.

>
> ===========
>
> On the client I've mounted this export like this:
> mount -o
> rw,noatime,tcp,rsize=8192,hard,intr,nfsvers=3,addr=192.168.0.8
> 192.168.0.8:/exports/country/de /data/country/de
>
> This succeeds and I can see the letters a-z in /data/country/de
> however these directories are all empty so crossing the filesystem
> border doesn't seem to work.
>
> ===========
>
> The server is an openfiler appliance:
>
> [root@exonas900 ~]# conary q|grep nfs
> libnfsidmap=0.20-1.1-1
> nfs-client=1.0.10-4.6-1
> nfs-server=1.0.10-4.6-1
> nfs-utils=1.0.10-4.6-1
> [root@exonas900 ~]# uname -a
> Linux exonas900 2.6.26.8-1.0.11.smp.gcc3.4.x86.i686 #1 SMP Sat Jan
> 10 22:09:09 GMT 2009 i686 i686 i386 GNU/Linux
>
> ===========
>
> The client I'm testing with is a Centos 5 system:
>
> [root@wmembf10 ~]# rpm -qa|grep nfs
> nfs-utils-lib-1.0.8-7.6.el5
> nfs-utils-1.0.9-47.el5_5
> [root@wmembf10 ~]# uname -a
> Linux wmembf10.tmm.cvsn.de 2.6.18-194.11.4.el5xen #1 SMP Tue Sep 21
> 05:40:24 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
>
> Regards,
> Dennis
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2011-03-22 17:43:32

by Dennis Jacobfeuerborn

[permalink] [raw]
Subject: Re: bind mounts, crossmnt and multi client nfsv3

On 03/21/2011 08:04 PM, Dennis Jacobfeuerborn wrote:

[SNIP]

> I'd be interested to move on to v4 but from what I've heard the
> user-mapping no longer is done using the uid. Since the users don't have
> entries in /etc/passwd and exist purely as id's in the filesystem the
> username based mapping in nfsv4 doesn't allow me to use it in this case.
> I guess the only way to accomplish that would be to write a custom
> rpcidmapd daemon?
>
> I'm currently downloading the openfiler installer so I can replicate the
> setup here for better testing. Maybe updating the nfs-utils package to a
> more current version will have an impact.

Apparently I need to export both the /exports directory and all the letter
directories beneath it for the crossmnt option to work, e.g.:

/exports 192.168...
/exports/country/de/a 192.168...
/exports/country/de/b 192.168...
/exports/country/de/c 192.168...
etc.

Regards,
Dennis

2011-03-21 19:04:23

by Dennis Jacobfeuerborn

[permalink] [raw]
Subject: Re: bind mounts, crossmnt and multi client nfsv3

On 03/21/2011 07:19 PM, J. Bruce Fields wrote:
> On Mon, Mar 21, 2011 at 06:53:18PM +0100, Dennis Jacobfeuerborn wrote:
>> On 03/21/2011 06:25 PM, J. Bruce Fields wrote:
>>> On Mon, Mar 21, 2011 at 01:27:05PM +0100, Dennis Jacobfeuerborn wrote:
>>>> On 03/21/2011 04:09 AM, NeilBrown wrote:
>>>>> On Mon, 21 Mar 2011 03:24:50 +0100 Dennis Jacobfeuerborn
>>>>> <[email protected]> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I have a storage system that is exporting many directories to multiple
>>>>>> clients resulting in a lot of mountpoints (156) on each client. What I'm
>>>>>> trying to do is to create a single directory on the server and then use
>>>>>> mount --bind to mount all the different directories (which are stored on
>>>>>> different LVM volumes) into this single export directory and finally export
>>>>>> this directory to the clients.
>>>>>> Now the exports man-page mentions the crossmnt option but it also mentions
>>>>>> that it cannot be used if I want to export the directory to multiple
>>>>>> clients. Is there another way to accomplish something like this?
>>>>>>
>>>>>
>>>>> Just export the top directory with 'crossmnt' - it should work fine.
>>>>>
>>>>> The 'multiple clients' thing only affects 'nohide' and I think it only
>>>>> affected it back in 2.4 days.
>>>>> Lots changed with 2.6, but maybe not enough of the man page changed :-(
>>>>>
>>>>> So try with 'crossmnt' and if it doesn't work, then come back with details.
>>>>
>>>> I've tried this now but it doesn't seem to work. This is the setup so far:
>>>>
>>>> ===========
>>>>
>>>> On the server I've created the bind mounts like this:
>>>>
>>>> /mnt/vg0/vol01/country/de/a on /exports/country/de/a type none (rw,bind)
>>>> /mnt/vg0/vol03/country/de/b on /exports/country/de/b type none (rw,bind)
>>>> ...
>>>>
>>>> Then I put the following line in /etc/exports:
>>>> /exports 192.168.0.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,no_root_squash,wdelay,sync,crossmnt,no_subtree_check)
>>>>
>>>> additionally these exports exist for each volume:
>>>> /mnt/vg0/vol01/country 213.131.252.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,no_root_squash,wdelay,sync)
>>>
>>> I'm a bit confused by the paths and ip blocks:
>>>
>>> - You want to export the path that the client sees
>>> (/exports/country/whatever), not the original path
>>> (/mnt/vg0/...).
>>> - If you expect clients to be able to traverse from /exports/ to
>>> filesystems underneath, then you'd want to make sure /exports/
>>> is exported to anything that the filesystems underneath are.
>>
>> The /mnt/vg0/volXX exports are currently used to export each
>> filesystem containing letters individually. On the client I then
>> mount each letter individually which means access to each mountpoint
>> doesn't have to traverse filesystem boundaries on the server but it
>> also means I end up with 156 individual mountpoints which is a pain
>> to deal with. I had to write a script that runs a loop mounting 5
>> letters then waiting 30 seconds then mount the next 5 letters etc.
>> If I try to mount these in one go about 10 mounts work fine but then
>> I only get errors. Apparently the server can only handle so many
>> mounts in a certain timeframe (socket timeouts?)
>
> Might be interesting to pin that problem down. I wonder if "insecure"
> would help?

I'm going to give it a try.

>> That's why I'm aiming for consolidating the letters on the server
>> using bind mounts and then just mount the country directories
>> leaving me with 6 mountpoints per client which would be *much*
>> easier to handle.
>
> Oh, so you're actually exporting each filesystem *twice*, in two
> different places?
>
> I think the server can deal with that. But probably only if the two
> filesystems have the same export options. (You've got crossmnt and
> no_subtree_check set on one but not the other.)

I've modified the entries so that the options are now exactly identical in
/var/lib/nfs/etab (all have set nohide,crossmnt,no_subtree_check now). This
doesn't seem to help so far.

> Wait, and looks like you're also bind mounting *subdirectories* of
> filesystems instead of whole filesystems. I'm not sure the server will
> handle that the way you'd expect.
>
> Is there any reason you can't just bind-mount all of /mnt/vg0/vol01/, or
> whatever, to the same path?

Each of the 10 volumes contains (not quite) random letter directories for
various countries and when the are mounted on the clients they end up in
the proper hierarchy e.g.:

vol01/de/a => country/de/a
vol03/de/b => country/de/b
vol07/de/c => country/de/c
...
vol03/fr/a => country/fr/a
vol05/fr/b => country/fr/b
...

so I cannot really mount entire volumes in one go on the client. I guess
another way would be to mount the volumes on the clients and then use bind
mounts there to create the appropriate hierarchy. Since I'd have to
duplicate that on each client though I'd really like to avoid this and
accomplish this on the server side.

>> The 192 IP range was a mistake and that should be the same 213 range
>> as the /exports export. If I can get this working then I plan to
>> eliminate the individual /mnt/vg0/... exports if they are not
>> needed.
>>
>>> But actually you should only need the first /exports entry; filesystems
>>> underneath should inherit export options from the parent.
>>
>> This is what /var/lib/nfs/etab says for /exports after an exportfs -r:
>> /exports 213.131.252.0/255.255.255.0(rw,sync,wdelay,hide,crossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=96,anongid=96)
>>
>> When I mount this I can see the proper subdirectories until I get to
>> the bind mounted letter directories which have their expected
>> contents on the server but appear empty on the client.
>
> Note also NFSv2/v3 clients aren't necessarily equipped to deal with
> filesystem boundaries, and some applications may be confused by seeing
> what look like files with the same inode number on the same filesystem
> that are actually different files.

I'd be interested to move on to v4 but from what I've heard the
user-mapping no longer is done using the uid. Since the users don't have
entries in /etc/passwd and exist purely as id's in the filesystem the
username based mapping in nfsv4 doesn't allow me to use it in this case.
I guess the only way to accomplish that would be to write a custom
rpcidmapd daemon?

I'm currently downloading the openfiler installer so I can replicate the
setup here for better testing. Maybe updating the nfs-utils package to a
more current version will have an impact.

Regards,
Dennis

2011-03-22 17:54:10

by J. Bruce Fields

[permalink] [raw]
Subject: Re: bind mounts, crossmnt and multi client nfsv3

On Tue, Mar 22, 2011 at 06:43:29PM +0100, Dennis Jacobfeuerborn wrote:
> On 03/21/2011 08:04 PM, Dennis Jacobfeuerborn wrote:
>
> [SNIP]
>
> >I'd be interested to move on to v4 but from what I've heard the
> >user-mapping no longer is done using the uid. Since the users don't have
> >entries in /etc/passwd and exist purely as id's in the filesystem the
> >username based mapping in nfsv4 doesn't allow me to use it in this case.
> >I guess the only way to accomplish that would be to write a custom
> >rpcidmapd daemon?
> >
> >I'm currently downloading the openfiler installer so I can replicate the
> >setup here for better testing. Maybe updating the nfs-utils package to a
> >more current version will have an impact.
>
> Apparently I need to export both the /exports directory and all the
> letter directories beneath it for the crossmnt option to work, e.g.:
>
> /exports 192.168...
> /exports/country/de/a 192.168...
> /exports/country/de/b 192.168...
> /exports/country/de/c 192.168...

Huh. I did just a quick test, mount --bind'ing a couple directories
under a crossmnt export, and verified that I could see the result from
an nfsv3 client.

2.6.38-ish kernel, fc13 userspace (nfs-utils-1.2.2-6).

--b.

2011-03-21 18:19:16

by J. Bruce Fields

[permalink] [raw]
Subject: Re: bind mounts, crossmnt and multi client nfsv3

On Mon, Mar 21, 2011 at 06:53:18PM +0100, Dennis Jacobfeuerborn wrote:
> On 03/21/2011 06:25 PM, J. Bruce Fields wrote:
> >On Mon, Mar 21, 2011 at 01:27:05PM +0100, Dennis Jacobfeuerborn wrote:
> >>On 03/21/2011 04:09 AM, NeilBrown wrote:
> >>>On Mon, 21 Mar 2011 03:24:50 +0100 Dennis Jacobfeuerborn
> >>><[email protected]> wrote:
> >>>
> >>>>Hi,
> >>>>I have a storage system that is exporting many directories to multiple
> >>>>clients resulting in a lot of mountpoints (156) on each client. What I'm
> >>>>trying to do is to create a single directory on the server and then use
> >>>>mount --bind to mount all the different directories (which are stored on
> >>>>different LVM volumes) into this single export directory and finally export
> >>>>this directory to the clients.
> >>>>Now the exports man-page mentions the crossmnt option but it also mentions
> >>>>that it cannot be used if I want to export the directory to multiple
> >>>>clients. Is there another way to accomplish something like this?
> >>>>
> >>>
> >>>Just export the top directory with 'crossmnt' - it should work fine.
> >>>
> >>>The 'multiple clients' thing only affects 'nohide' and I think it only
> >>>affected it back in 2.4 days.
> >>>Lots changed with 2.6, but maybe not enough of the man page changed :-(
> >>>
> >>>So try with 'crossmnt' and if it doesn't work, then come back with details.
> >>
> >>I've tried this now but it doesn't seem to work. This is the setup so far:
> >>
> >>===========
> >>
> >>On the server I've created the bind mounts like this:
> >>
> >>/mnt/vg0/vol01/country/de/a on /exports/country/de/a type none (rw,bind)
> >>/mnt/vg0/vol03/country/de/b on /exports/country/de/b type none (rw,bind)
> >>...
> >>
> >>Then I put the following line in /etc/exports:
> >>/exports 192.168.0.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,no_root_squash,wdelay,sync,crossmnt,no_subtree_check)
> >>
> >>additionally these exports exist for each volume:
> >>/mnt/vg0/vol01/country 213.131.252.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,no_root_squash,wdelay,sync)
> >
> >I'm a bit confused by the paths and ip blocks:
> >
> > - You want to export the path that the client sees
> > (/exports/country/whatever), not the original path
> > (/mnt/vg0/...).
> > - If you expect clients to be able to traverse from /exports/ to
> > filesystems underneath, then you'd want to make sure /exports/
> > is exported to anything that the filesystems underneath are.
>
> The /mnt/vg0/volXX exports are currently used to export each
> filesystem containing letters individually. On the client I then
> mount each letter individually which means access to each mountpoint
> doesn't have to traverse filesystem boundaries on the server but it
> also means I end up with 156 individual mountpoints which is a pain
> to deal with. I had to write a script that runs a loop mounting 5
> letters then waiting 30 seconds then mount the next 5 letters etc.
> If I try to mount these in one go about 10 mounts work fine but then
> I only get errors. Apparently the server can only handle so many
> mounts in a certain timeframe (socket timeouts?)

Might be interesting to pin that problem down. I wonder if "insecure"
would help?

> That's why I'm aiming for consolidating the letters on the server
> using bind mounts and then just mount the country directories
> leaving me with 6 mountpoints per client which would be *much*
> easier to handle.

Oh, so you're actually exporting each filesystem *twice*, in two
different places?

I think the server can deal with that. But probably only if the two
filesystems have the same export options. (You've got crossmnt and
no_subtree_check set on one but not the other.)

Wait, and looks like you're also bind mounting *subdirectories* of
filesystems instead of whole filesystems. I'm not sure the server will
handle that the way you'd expect.

Is there any reason you can't just bind-mount all of /mnt/vg0/vol01/, or
whatever, to the same path?

> The 192 IP range was a mistake and that should be the same 213 range
> as the /exports export. If I can get this working then I plan to
> eliminate the individual /mnt/vg0/... exports if they are not
> needed.
>
> >But actually you should only need the first /exports entry; filesystems
> >underneath should inherit export options from the parent.
>
> This is what /var/lib/nfs/etab says for /exports after an exportfs -r:
> /exports 213.131.252.0/255.255.255.0(rw,sync,wdelay,hide,crossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=96,anongid=96)
>
> When I mount this I can see the proper subdirectories until I get to
> the bind mounted letter directories which have their expected
> contents on the server but appear empty on the client.

Note also NFSv2/v3 clients aren't necessarily equipped to deal with
filesystem boundaries, and some applications may be confused by seeing
what look like files with the same inode number on the same filesystem
that are actually different files.

--b.

2011-03-21 17:53:22

by Dennis Jacobfeuerborn

[permalink] [raw]
Subject: Re: bind mounts, crossmnt and multi client nfsv3

On 03/21/2011 06:25 PM, J. Bruce Fields wrote:
> On Mon, Mar 21, 2011 at 01:27:05PM +0100, Dennis Jacobfeuerborn wrote:
>> On 03/21/2011 04:09 AM, NeilBrown wrote:
>>> On Mon, 21 Mar 2011 03:24:50 +0100 Dennis Jacobfeuerborn
>>> <[email protected]> wrote:
>>>
>>>> Hi,
>>>> I have a storage system that is exporting many directories to multiple
>>>> clients resulting in a lot of mountpoints (156) on each client. What I'm
>>>> trying to do is to create a single directory on the server and then use
>>>> mount --bind to mount all the different directories (which are stored on
>>>> different LVM volumes) into this single export directory and finally export
>>>> this directory to the clients.
>>>> Now the exports man-page mentions the crossmnt option but it also mentions
>>>> that it cannot be used if I want to export the directory to multiple
>>>> clients. Is there another way to accomplish something like this?
>>>>
>>>
>>> Just export the top directory with 'crossmnt' - it should work fine.
>>>
>>> The 'multiple clients' thing only affects 'nohide' and I think it only
>>> affected it back in 2.4 days.
>>> Lots changed with 2.6, but maybe not enough of the man page changed :-(
>>>
>>> So try with 'crossmnt' and if it doesn't work, then come back with details.
>>
>> I've tried this now but it doesn't seem to work. This is the setup so far:
>>
>> ===========
>>
>> On the server I've created the bind mounts like this:
>>
>> /mnt/vg0/vol01/country/de/a on /exports/country/de/a type none (rw,bind)
>> /mnt/vg0/vol03/country/de/b on /exports/country/de/b type none (rw,bind)
>> ...
>>
>> Then I put the following line in /etc/exports:
>> /exports 192.168.0.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,no_root_squash,wdelay,sync,crossmnt,no_subtree_check)
>>
>> additionally these exports exist for each volume:
>> /mnt/vg0/vol01/country 213.131.252.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,no_root_squash,wdelay,sync)
>
> I'm a bit confused by the paths and ip blocks:
>
> - You want to export the path that the client sees
> (/exports/country/whatever), not the original path
> (/mnt/vg0/...).
> - If you expect clients to be able to traverse from /exports/ to
> filesystems underneath, then you'd want to make sure /exports/
> is exported to anything that the filesystems underneath are.

The /mnt/vg0/volXX exports are currently used to export each filesystem
containing letters individually. On the client I then mount each letter
individually which means access to each mountpoint doesn't have to traverse
filesystem boundaries on the server but it also means I end up with 156
individual mountpoints which is a pain to deal with. I had to write a
script that runs a loop mounting 5 letters then waiting 30 seconds then
mount the next 5 letters etc. If I try to mount these in one go about 10
mounts work fine but then I only get errors. Apparently the server can only
handle so many mounts in a certain timeframe (socket timeouts?)

That's why I'm aiming for consolidating the letters on the server using
bind mounts and then just mount the country directories leaving me with 6
mountpoints per client which would be *much* easier to handle.

The 192 IP range was a mistake and that should be the same 213 range as the
/exports export. If I can get this working then I plan to eliminate the
individual /mnt/vg0/... exports if they are not needed.

> But actually you should only need the first /exports entry; filesystems
> underneath should inherit export options from the parent.

This is what /var/lib/nfs/etab says for /exports after an exportfs -r:
/exports
213.131.252.0/255.255.255.0(rw,sync,wdelay,hide,crossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=96,anongid=96)

When I mount this I can see the proper subdirectories until I get to the
bind mounted letter directories which have their expected contents on the
server but appear empty on the client.

Regards,
Dennis

2011-03-21 03:09:31

by NeilBrown

[permalink] [raw]
Subject: Re: bind mounts, crossmnt and multi client nfsv3

On Mon, 21 Mar 2011 03:24:50 +0100 Dennis Jacobfeuerborn
<[email protected]> wrote:

> Hi,
> I have a storage system that is exporting many directories to multiple
> clients resulting in a lot of mountpoints (156) on each client. What I'm
> trying to do is to create a single directory on the server and then use
> mount --bind to mount all the different directories (which are stored on
> different LVM volumes) into this single export directory and finally export
> this directory to the clients.
> Now the exports man-page mentions the crossmnt option but it also mentions
> that it cannot be used if I want to export the directory to multiple
> clients. Is there another way to accomplish something like this?
>

Just export the top directory with 'crossmnt' - it should work fine.

The 'multiple clients' thing only affects 'nohide' and I think it only
affected it back in 2.4 days.
Lots changed with 2.6, but maybe not enough of the man page changed :-(

So try with 'crossmnt' and if it doesn't work, then come back with details.

NeilBrown


2011-03-21 12:58:30

by Dennis Jacobfeuerborn

[permalink] [raw]
Subject: Re: bind mounts, crossmnt and multi client nfsv3

On 03/21/2011 04:09 AM, NeilBrown wrote:
> On Mon, 21 Mar 2011 03:24:50 +0100 Dennis Jacobfeuerborn
> <[email protected]> wrote:
>
>> Hi,
>> I have a storage system that is exporting many directories to multiple
>> clients resulting in a lot of mountpoints (156) on each client. What I'm
>> trying to do is to create a single directory on the server and then use
>> mount --bind to mount all the different directories (which are stored on
>> different LVM volumes) into this single export directory and finally export
>> this directory to the clients.
>> Now the exports man-page mentions the crossmnt option but it also mentions
>> that it cannot be used if I want to export the directory to multiple
>> clients. Is there another way to accomplish something like this?
>>
>
> Just export the top directory with 'crossmnt' - it should work fine.
>
> The 'multiple clients' thing only affects 'nohide' and I think it only
> affected it back in 2.4 days.
> Lots changed with 2.6, but maybe not enough of the man page changed :-(
>
> So try with 'crossmnt' and if it doesn't work, then come back with details.

I've tried this now but it doesn't seem to work. This is the setup so far:

===========

On the server I've created the bind mounts like this:

/mnt/vg0/vol01/country/de/a on /exports/country/de/a type none (rw,bind)
/mnt/vg0/vol03/country/de/b on /exports/country/de/b type none (rw,bind)
...

Then I put the following line in /etc/exports:
/exports
192.168.0.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,no_root_squash,wdelay,sync,crossmnt,no_subtree_check)

additionally these exports exist for each volume:
/mnt/vg0/vol01/country
213.131.252.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,no_root_squash,wdelay,sync)

===========

On the client I've mounted this export like this:
mount -o rw,noatime,tcp,rsize=8192,hard,intr,nfsvers=3,addr=192.168.0.8
192.168.0.8:/exports/country/de /data/country/de

This succeeds and I can see the letters a-z in /data/country/de however
these directories are all empty so crossing the filesystem border doesn't
seem to work.

===========

The server is an openfiler appliance:

[root@exonas900 ~]# conary q|grep nfs
libnfsidmap=0.20-1.1-1
nfs-client=1.0.10-4.6-1
nfs-server=1.0.10-4.6-1
nfs-utils=1.0.10-4.6-1
[root@exonas900 ~]# uname -a
Linux exonas900 2.6.26.8-1.0.11.smp.gcc3.4.x86.i686 #1 SMP Sat Jan 10
22:09:09 GMT 2009 i686 i686 i386 GNU/Linux

===========

The client I'm testing with is a Centos 5 system:

[root@wmembf10 ~]# rpm -qa|grep nfs
nfs-utils-lib-1.0.8-7.6.el5
nfs-utils-1.0.9-47.el5_5
[root@wmembf10 ~]# uname -a
Linux wmembf10.tmm.cvsn.de 2.6.18-194.11.4.el5xen #1 SMP Tue Sep 21
05:40:24 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

Regards,
Dennis