Hey Trond,
Here here is the first stab at having NFS use the CacheFS that is currently
in the -mm tree. As you know CacheFS is a caching filesystem that allows
network filesystems, such as AFS and now NFS, to cache data pages on
the local disk. This is a big win (especially with read-only filesystems)
since it drastically cuts down on the amount of data read across the
network. Plus the cached data survives umounts and reboots (since CacheFS
journals the data). So when a client comes back up or the filesystem is
remounted,
all that is needed is a few small getattrs to insure the data is still
valid. If it is,
NFS reads are turned into local disk reads.... A Huge gain in
performance!!!!
I have structured that patch so CacheFS is a mount option. I figured
this was the safest way to introduce CacheFS and not completely
break NFS.... And if for some reason, NFS is not able to get an
initial Cachefs cookie (i.e. it can't use CacheFS) , NFS will dynamically
turn off the mount option, disabling the use of the cache..
But as usual there are some issues....
1) NFS aliasing. The fact that:
mount hades:/hades /a
mount hades:/hades/xxx /b
creates separate super blocks causes problems for
CacheFS. With the current -mm code, these type of mount
actual causes CacheFS to crash (although I do have a proposed
patch that David is looking at).
2) NFS4 is not supported. I simply have not had time to get this working.
But I will!!
3) There is no user level support. I realize this is extremely cheesy
but I noticed that the NFS posix mount option (in the 2.6 kernel)
was no longer being used, so I high jacked it. Which means
to make NFS to used CacheFS you need to use the posix option:
mount -o posix server:/export/home /mnt/server/home
Comments?
SteveD.
PS. The new CacheFS mailing that have been set up is at
http://www.redhat.com/mailman/listinfo/linux-cachefs
I encourage you and anybody else interested in this
type of technology to subscribe....
P? m? , 04/10/2004 klokka 22:45, skreiv Steve Dickson:
> 3) There is no user level support. I realize this is extremely cheesy
> but I noticed that the NFS posix mount option (in the 2.6 kernel)
> was no longer being used, so I high jacked it. Which means
> to make NFS to used CacheFS you need to use the posix option:
>
> mount -o posix server:/export/home /mnt/server/home
This is my one and only real gripe about it. The posix mount option is
clearly documented, so we really cannot play around with it. Why can't
you just add a separate cachefs flag?
Otherwise, I'm a bit dubious about the wisdom of putting
nfs_invalidatepage() and nfs_releasepage() into fs/nfs/file.c. These are
not file operations, but rather pure page cache operations. I would have
thought that either read.c or possibly nfs-cachefs.c would be more
appropriate.
Please note too that Chuck has made generic functions for copying and
comparing NFS filehandles. They should be used in nfs_cache_fh_match() &
co.
I'm a bit worried about the use of the raw IP address in
nfs_cache_server_match(). It seems to me that when we add the NFSv4.1
support for trunking over several different transport mechanisms (RDMA,
IPv4/v6 etc) on the same mountpoint, then we may end up with a problem.
We can probably leave it in for now, but later we may want to consider
switching to using server->hostname or something equivalent.
Otherwise, it looks good. Looking forward to try it out...
Cheers,
Trond
> This is my one and only real gripe about it. The posix mount option is
> clearly documented, so we really cannot play around with it. Why can't
> you just add a separate cachefs flag?
That's just for the prototype. It will be changed later. It's also a stand-in
for the superblock un-aliasing patch.
> I'm a bit worried about the use of the raw IP address in
> nfs_cache_server_match(). It seems to me that when we add the NFSv4.1
> support for trunking over several different transport mechanisms (RDMA,
> IPv4/v6 etc) on the same mountpoint, then we may end up with a problem. We
> can probably leave it in for now, but later we may want to consider
> switching to using server->hostname or something equivalent.
There needs to be some way of matching NFS file handle sets in the cache if
you want persistant caches. The only way I could see was to use IP addr and
port number for NFS.
However, changing this later is trivial. If you bump the version number in the
netfs definition, then cachefs will nuke existing data in the cache that isn't
of the expected format.
David
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 10/05/2004 05:45 AM, Steve Dickson wrote:
| 3) There is no user level support. I realize this is extremely cheesy
| but I noticed that the NFS posix mount option (in the 2.6 kernel)
| was no longer being used, so I high jacked it. Which means
| to make NFS to used CacheFS you need to use the posix option:
|
| mount -o posix server:/export/home /mnt/server/home
brr :) why is it posix, this is so out of the context for me (as a
user). Is it possible to have a cachefs flag. Would make it more logical.
- --
[ Clemens Schwaighofer -----=====:::::~ ]
[ TBWA\ && TEQUILA\ Japan IT Group ]
[ 6-17-2 Ginza Chuo-ku, Tokyo 104-0061, JAPAN ]
[ Tel: +81-(0)3-3545-7703 Fax: +81-(0)3-3545-7343 ]
[ http://www.tequila.co.jp http://www.tbwajapan.co.jp ]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFBZhlPjBz/yQjBxz8RAtKbAKCLWJBqWQUcroSaLzlrH1r2nuqI8gCfUzyL
kpANJQ4pYt1026VBRm8UlpU=
=NGFC
-----END PGP SIGNATURE-----
> brr :) why is it posix, this is so out of the context for me (as a
> user). Is it possible to have a cachefs flag. Would make it more logical.
It's a prototype, so Steve's made use of an existing flag for convenience
sake.
David
Clemens Schwaighofer wrote:
> brr :) why is it posix, this is so out of the context for me (as a
> user). Is it possible to have a cachefs flag. Would make it more logical.
Because it was the easiest way to get things started (i.e. no userlevel
changes needed at all).... The 'fscache' flag will be coming along with
the nfs4 support, since nfs4 mounting code does not have an open
(unused) mounting flag....
SteveD.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 10/08/2004 08:22 PM, Steve Dickson wrote:
> Clemens Schwaighofer wrote:
>
>> brr :) why is it posix, this is so out of the context for me (as a
>> user). Is it possible to have a cachefs flag. Would make it more logical.
>
>
> Because it was the easiest way to get things started (i.e. no userlevel
> changes needed at all).... The 'fscache' flag will be coming along with
> the nfs4 support, since nfs4 mounting code does not have an open
> (unused) mounting flag....
okay, thank you for the information. Anyway inbetween I will try that
out with NFS3 ...
lg, clemens
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFBZrTAjBz/yQjBxz8RAoddAJwIVyxuMFy0tD9H8z8D/UKjXmvAtgCg4mcY
2NZyYRCsdR6CzbTKKcJiybw=
=sKPN
-----END PGP SIGNATURE-----
hi :)
On Fri, Oct 08, 2004 at 07:22:13AM -0400, Steve Dickson wrote:
> The 'fscache' flag will be coming along with the nfs4 support, since
> nfs4 mounting code does not have an open (unused) mounting flag....
is such a flag even neccessary?
The way I see fscache is that its operations will be no-ops anyway if you
haven't mounted any backing cache.
--
Martin Waitz
P? m? , 11/10/2004 klokka 16:23, skreiv Martin Waitz:
> hi :)
>
> On Fri, Oct 08, 2004 at 07:22:13AM -0400, Steve Dickson wrote:
> > The 'fscache' flag will be coming along with the nfs4 support, since
> > nfs4 mounting code does not have an open (unused) mounting flag....
>
> is such a flag even neccessary?
> The way I see fscache is that its operations will be no-ops anyway if you
> haven't mounted any backing cache.
You may not want to run cachefs on *all* your NFS partitions. It will
slow you down on those partitions that have lots of cache contention.
That said, David & co.: why did you choose not to use something similar
to the Solaris syntax for cachefs? The "layered filesystem" syntax has
the advantage that it would avoid entirely the need to change the NFS
mount syntax, and would make it easier to port cachefs to cifs etc.
Cheers,
Trond
> > The 'fscache' flag will be coming along with the nfs4 support, since
> > nfs4 mounting code does not have an open (unused) mounting flag....
>
> is such a flag even neccessary?
> The way I see fscache is that its operations will be no-ops anyway if you
> haven't mounted any backing cache.
There are two reasons:
(1) The fscache middle bit builds up a cookie tree even if there are no
backing caches. This means if the cache is added later (NFS root, for
example), the netfs then starts caching immediately without requiring a
remount.
(2) NFS is not currently safe with respect to fscache in the following
situation:
mount warthog:/warthog/x /x
mount warthog:/warthog /y
Because NFS ends up with two superblocks for what is one set of
filehandles. This means it tries to get the same cookies out of fscache
twice, which fscache is ill-equipped to handle.
This also introduces unnecessary (IMNSHO) aliasing in each client of
inodes and pages both.
I have a patch to fix NFS to cope with this by modifying the get_sb()
method to allow the filesystem to override the automatic selection of
super->s_root as the root dentry. However, Al Viro is leery of the patch
for some reason.
David
Trond Myklebust <[email protected]> wrote:
> That said, David & co.: why did you choose not to use something similar
> to the Solaris syntax for cachefs?
Well...
(1) when I wrote cachefs I didn't know that Solaris had such a thing
(2) I don't know what Solaris syntax is - I've seen one example, and that
didn't make a lot of sense
(3) my way doesn't involve any changes to userspace programs such as mount
> The "layered filesystem" syntax has the advantage that it would avoid
> entirely the need to change the NFS mount syntax,
And the disadvantage that we'd have to change the VFS to support it, I
think. I don't know how this "layered filesystem" thing operates or is used,
so I'm guessing.
> and would make it easier to port cachefs to cifs etc.
I doubt it. The netfs still has to interact with fscache internally to decide
how match netfs files to cache files, which is what most of the interface is
about; and to push/pull pages to/from the cache (this bit could possibly be
made transparent, but I'm not sure how you'd do it on Linux with the present
VM & VFS interfaces).
David