Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 8BIT
Subject: RE: Pull request for FS-Cache, including NFS patches
Date: Thu, 18 Dec 2008 19:45:57 -0800
Message-ID: <7A24DF798E223B4C9864E8F92E8C93EC01AAF3F8@SACMVEXC1-PRD.hq.netapp.com>
In-Reply-To: <7633.1229653644@redhat.com>
Thread-Topic: Pull request for FS-Cache, including NFS patches
Thread-Index: AclhgWf78h0pPgdFSainorUZMMENHQABqWtg
From: "Muntz, Daniel" <Dan.Muntz@netapp.com>
To: "David Howells" <dhowells@redhat.com>,
       "Andrew Morton" <akpm@linux-foundation.org>
Cc: <sfr@canb.auug.org.au>, <linux-kernel@vger.kernel.org>,
       <nfsv4@linux-nfs.org>, <steved@redhat.com>,
       <linux-fsdevel@vger.kernel.org>, <rwheeler@redhat.com>
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 6519
Lines: 160

Local disk cache was great for AFS back around 1992.  Typical networks
were 10 or 100Mbps (slower than disk access at the time), and memories
were small (typical 16MB).  FS-Cache appears to help only with read
traffic--one reason why the web loves caching--and only for reads that
would miss the buffer/page cache (memories are now "large").  Solaris
has had CacheFS since ~1995, HPUX had a port of it since ~1997.  I'd be
interested in evidence of even a small fraction of Solaris and/or HPUX
shops using CacheFS.  I am aware of customers who thought it sounded
like a good idea, but ended up ditching it for various reasons (e.g.,
CacheFS just adds overhead if you almost always hit your local mem
cache).

One argument in favor that I don't see here is that local disk cache is
persistent (I'm assuming it is in your implementation).

Addressing 1 and 2 in your list, I'd be curious how often a miss in core
is a hit on disk.
Number 3 scares me.  How does this play with the expected semantics of
NFS?
Number 5 is hard, if not provably requiring human intervention to do
syncs when writes are involved (see Disconnected AFS work by
UM/CITI/Huston, and work at CMU).
Add persistence as number 6.  This may be the best reason to have it,
imho.

  -Dan

-----Original Message-----
From: David Howells [mailto:dhowells@redhat.com] 
Sent: Thursday, December 18, 2008 6:27 PM
To: Andrew Morton
Cc: sfr@canb.auug.org.au; linux-kernel@vger.kernel.org;
nfsv4@linux-nfs.org; steved@redhat.com; dhowells@redhat.com;
linux-fsdevel@vger.kernel.org; rwheeler@redhat.com
Subject: Re: Pull request for FS-Cache, including NFS patches

Andrew Morton <akpm@linux-foundation.org> wrote:

> I don't believe that it has yet been convincingly demonstrated that we

> want to merge it at all.
> 
> It's a huuuuuuuuge lump of new code, so it really needs to provide 
> decent value.  Can we revisit this?  Yet again?  What do we get from 
> all this?

I should tell you to go and reread LKML at this point.

But...  What can FS-Cache do for you?  Well, in your specific case,
probably nothing.  If all your computers are local to your normal
desktop box and are connected by sufficiently fast network and you have
sufficiently few of them, or you don't use any of NFS, AFS, CIFS,
Lustre, CRFS, CD-ROMs then it is likely that won't gain you anything.

Even if you do use some of those "netfs's", it won't get you anything
yet because I haven't included patches to support anything other than
NFS and the in-kernel AFS client yet.

However, if you do use NFS (or my AFS client), and you are accessing
computers via slow networks, or you have lots of machines spamming your
NFS server, then it might avail you.

It's a compromise: a trade-off between the loading and latencies of your
network vs the loading and latencies of your disk; you sacrifice disk
space to make up for the deficiencies of your network.  The worst bit is
that the latencies are additive under some circumstances (when doing a
page read you may have to check disk and then go to the network).


So, FS-Cache can do the following for you:

 (1) Allow you to reduce network loading by diverting repeat reads to
local
     storage.

 (2) Allow you to reduce the latency of slow network links by diverting
repeat
     reads to local storage.

 (3) Allow you to reduce the effect of uncontactable servers by serving
data
     out of local storage.

 (4) Allows you to reduce the latency of slow rotating media (such as
CDROM
     and CD-changers).

 (5) Allow you to implement disconnected operation, partly by (3), but
also by
     caching changes for later syncing.

Now, (1) and (2) are readily demonstrable.  I have posted benchmarks to
do this.  (3) to (5) are not yet implemented; these have to be mostly
implemented in the filesystems that use FS-Cache rather than FS-Cache
itself.  FS-Cache currently has sufficient functionality to do (3) and
(4), but needs some extra bits to do (5).

I've tried to implement just the minimal useful functionality for
persistent caching.  There is more to be added, but it's not immediately
necessary.


FS-Cache tries to make its facilities as simple and as general as
possible so that _any_ filesystem or blockdev can use it.  I have
patches here to make NFS and AFS use it.  I know someone is working on
getting Lustre to use it, and other filesystem maintainers have
expressed interest, subject to the code getting upstream.

Furthermore, FS-Cache hides the implementation of the cache from the
netfs.
Not only that, it hides I/O errors in the cache from the netfs.  Why
should the netfs have to deal with such things?


Another way to look at things is to look at other cases of cached
netfs's.
OpenAFS for example.  It has a local cache of its own.  Solaris has
local NFS caching.  Windows has local caching for NFS and CIFS, I think.
Even web browsers have local caching.

>  303 files changed, 21049 insertions(+), 3726 deletions(-)

A big chunk of that, particularly the deletions, is the creds patches.
Excluding the stuff pulled from the security and NFS trees, the combined
FS-Cache, CacheFiles and AFS+ and NFS+FS-Cache patches are, in fact:

	86 files changed, 15385 insertions(+), 413 deletions(-)

and over 19% of the insertions is documentation.  Most of the deletions
(373) are in AFS.

> Are any distros pushing for this?  Or shipping it?  If so, are they 
> able to weigh in and help us with this quite difficult decision?

We (Red Hat) have shipped it in RHEL-5 and some Fedora releases.  Doing
so is quite an effort, though, precisely because the code is not yet
upstream.  We have customers using it and are gaining more customers who
want it.  There even appear to be CentOS users using it (or at least
complaining when it breaks).


I don't know what will convince you.  I've given you theoretical reasons
why caching ought to be useful; I've backed up the ones I've implemented
with benchmarks; I've given you examples of what our customers are doing
with it or want to do with it.  Please help me understand what else you
want.

Do you perhaps want the netfs maintainers (such as Trond) to say that
it's necessary?

David
_______________________________________________
NFSv4 mailing list
NFSv4@linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/