Hi all,
I was playing with nfs4 delegation, finding out that write delegations have
never been granted. I am using Benny Halevy's git tree with latest
pnfs-block-all branch. With rpcdebug, I found out that the write-delegation
grant was rejected due to setlease failure with error code (-11, which is
-EAGAIN). I finally figured out that this failure was caused by the
following lines in generic_setlease function,
if ((arg == F_WRLCK)
&& ((atomic_read(&dentry->d_count) > 1)
|| (atomic_read(&inode->i_count) > 1)))
goto out;
In my nfsd, the i_count, and d_count turn out to be 2 (no conflicts actaully
exist in my test case), so write-delegations always failed. I did find some
similar posts on nfs4 related mail lists about this problem. One possible
hack by Ajay could be found at
http://osdir.com/ml/linux.nfsv4/2006-08/msg00063.html
I am curious whether there is any progress in awarding write delegation in
current linux nfs4 implementation (for example, any git branch with write
delegation enabled I can pull?) Or, is write-delegation feature available in
other systems like OpenSolaris?
Any suggestion if I would like to enable and test write delegation? I think
some changes are needed to the setlease interface, right?
Thanks,
David
On Jul 23, 2009, at 11:10 AM, Rick Macklem wrote:
>>> I really don't want to enable write delegations until we figure
>>> out how
>>> to enforce them correctly against local (non-nfs) users of the
>>> exported
>>> filesystem as well. In addition to breaking delegations on read
>>> opens,
>>> that means breaking delegations or doing a cb_getattr on
>>> operations like
>>> stat.
>>
>> do you know whether there are local FS where the maintainers at
>> least plan
>> to incorporate delegations?
>
> I'm not a Linux guy, so I'm not familiar with the internal
> structure, but...
> in general, I don't think the problem is with local file systems.
> Usually
> the problem is with local system call access. For example, if a
> process running locally on the server opens a file, the delegation
> should
> be recalled, so that changes done locally on the client get flushed
> back
> to the server. Also, a write delegation allows a client to do byte
> range
> locking locally in the client, so the write delegation needs to be
> recalled before anything gets a byte range lock locally in the server.
The delegation implementation on the Linux server uses the vfs lease
subsystem, and so is integrated with local access - conflicting opens
done locally do recall delegations. But the last time I looked, the
lease subsystem is not complete as it doesn't recall leases (nor
delegations) on remove, rename, etc. Another problem is that while
write delegations improve performance for certain workloads, they kill
performance for others.
-->Andy
>
>
> A Samba server running in the nfs server would be doing "local" ops
> for the purpose of this discussion. (I'm not sure if Samba goes as far
> as doing Open/Share locks for clients?)
>
> rick
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
A simple one could be found in the following paper,
http://www.usenix.org/events/fast07/tech/full_papers/gulati/gulati_html/nache.html
-David
On Fri, Jul 24, 2009 at 2:44 PM, J. Bruce Fields <[email protected]>wrote:
> On Thu, Jul 23, 2009 at 02:11:46PM -0400, Andy Adamson wrote:
> >
> > On Jul 23, 2009, at 11:10 AM, Rick Macklem wrote:
> >
> >>>> I really don't want to enable write delegations until we figure
> >>>> out how
> >>>> to enforce them correctly against local (non-nfs) users of the
> >>>> exported
> >>>> filesystem as well. In addition to breaking delegations on read
> >>>> opens,
> >>>> that means breaking delegations or doing a cb_getattr on
> >>>> operations like
> >>>> stat.
> >>>
> >>> do you know whether there are local FS where the maintainers at
> >>> least plan
> >>> to incorporate delegations?
> >>
> >> I'm not a Linux guy, so I'm not familiar with the internal structure,
> >> but...
> >> in general, I don't think the problem is with local file systems.
> >> Usually
> >> the problem is with local system call access. For example, if a
> >> process running locally on the server opens a file, the delegation
> >> should
> >> be recalled, so that changes done locally on the client get flushed
> >> back
> >> to the server. Also, a write delegation allows a client to do byte
> >> range
> >> locking locally in the client, so the write delegation needs to be
> >> recalled before anything gets a byte range lock locally in the server.
> >
> > The delegation implementation on the Linux server uses the vfs lease
> > subsystem, and so is integrated with local access - conflicting opens
> > done locally do recall delegations. But the last time I looked, the
> > lease subsystem is not complete as it doesn't recall leases (nor
> > delegations) on remove, rename, etc. Another problem is that while write
> > delegations improve performance for certain workloads, they kill
> > performance for others.
>
> Are there any published results yet with real workloads?
>
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
On Tue, Jul 21, 2009 at 05:47:17PM -0400, David V. Cloud wrote:
> Hi all,
> I was playing with nfs4 delegation, finding out that write delegations have
> never been granted. I am using Benny Halevy's git tree with latest
> pnfs-block-all branch. With rpcdebug, I found out that the write-delegation
> grant was rejected due to setlease failure with error code (-11, which is
> -EAGAIN). I finally figured out that this failure was caused by the
> following lines in generic_setlease function,
>
> if ((arg == F_WRLCK)
> && ((atomic_read(&dentry->d_count) > 1)
> || (atomic_read(&inode->i_count) > 1)))
> goto out;
>
> In my nfsd, the i_count, and d_count turn out to be 2 (no conflicts actaully
> exist in my test case), so write-delegations always failed. I did find some
> similar posts on nfs4 related mail lists about this problem. One possible
> hack by Ajay could be found at
> http://osdir.com/ml/linux.nfsv4/2006-08/msg00063.html
>
> I am curious whether there is any progress in awarding write delegation in
> current linux nfs4 implementation (for example, any git branch with write
> delegation enabled I can pull?) Or, is write-delegation feature available in
> other systems like OpenSolaris?
>
> Any suggestion if I would like to enable and test write delegation? I think
> some changes are needed to the setlease interface, right?
I really don't want to enable write delegations until we figure out how
to enforce them correctly against local (non-nfs) users of the exported
filesystem as well. In addition to breaking delegations on read opens,
that means breaking delegations or doing a cb_getattr on operations like
stat.
--b.
Hello,
> I really don't want to enable write delegations until we figure out how
> to enforce them correctly against local (non-nfs) users of the exported
> filesystem as well. In addition to breaking delegations on read opens,
> that means breaking delegations or doing a cb_getattr on operations like
> stat.
do you know whether there are local FS where the maintainers at least plan
to incorporate delegations?
-- Eric
On Thu, Jul 23, 2009 at 10:12:05AM +0200, Eric Veith wrote:
> Hello,
>
> > I really don't want to enable write delegations until we figure out how
> > to enforce them correctly against local (non-nfs) users of the exported
> > filesystem as well. In addition to breaking delegations on read opens,
> > that means breaking delegations or doing a cb_getattr on operations like
> > stat.
>
> do you know whether there are local FS where the maintainers at least plan
> to incorporate delegations?
Local filesystems don't need support from the filesystem itself; the
only modifications necessary should be to the vfs.
We have patches that do that for read delegations:
http://git.linux-nfs.org/?p=bfields/linux-topics.git;a=shortlog;h=refs/heads/leases
but they still have a few problems. I hope to get them out for review
soon.
Write delegations are more difficult. Distributed filesystems will also
be harder.
--b.
> > I really don't want to enable write delegations until we figure out how
> > to enforce them correctly against local (non-nfs) users of the exported
> > filesystem as well. In addition to breaking delegations on read opens,
> > that means breaking delegations or doing a cb_getattr on operations like
> > stat.
>
> do you know whether there are local FS where the maintainers at least plan
> to incorporate delegations?
I'm not a Linux guy, so I'm not familiar with the internal structure, but...
in general, I don't think the problem is with local file systems. Usually
the problem is with local system call access. For example, if a
process running locally on the server opens a file, the delegation should
be recalled, so that changes done locally on the client get flushed back
to the server. Also, a write delegation allows a client to do byte range
locking locally in the client, so the write delegation needs to be
recalled before anything gets a byte range lock locally in the server.
A Samba server running in the nfs server would be doing "local" ops
for the purpose of this discussion. (I'm not sure if Samba goes as far
as doing Open/Share locks for clients?)
rick
On Thu, Jul 23, 2009 at 02:11:46PM -0400, Andy Adamson wrote:
>
> On Jul 23, 2009, at 11:10 AM, Rick Macklem wrote:
>
>>>> I really don't want to enable write delegations until we figure
>>>> out how
>>>> to enforce them correctly against local (non-nfs) users of the
>>>> exported
>>>> filesystem as well. In addition to breaking delegations on read
>>>> opens,
>>>> that means breaking delegations or doing a cb_getattr on
>>>> operations like
>>>> stat.
>>>
>>> do you know whether there are local FS where the maintainers at
>>> least plan
>>> to incorporate delegations?
>>
>> I'm not a Linux guy, so I'm not familiar with the internal structure,
>> but...
>> in general, I don't think the problem is with local file systems.
>> Usually
>> the problem is with local system call access. For example, if a
>> process running locally on the server opens a file, the delegation
>> should
>> be recalled, so that changes done locally on the client get flushed
>> back
>> to the server. Also, a write delegation allows a client to do byte
>> range
>> locking locally in the client, so the write delegation needs to be
>> recalled before anything gets a byte range lock locally in the server.
>
> The delegation implementation on the Linux server uses the vfs lease
> subsystem, and so is integrated with local access - conflicting opens
> done locally do recall delegations. But the last time I looked, the
> lease subsystem is not complete as it doesn't recall leases (nor
> delegations) on remove, rename, etc. Another problem is that while write
> delegations improve performance for certain workloads, they kill
> performance for others.
Are there any published results yet with real workloads?
--b.