All,
We have a Quad-core server running gentoo 2.6.27-r7 that is used as an nfs server for a xfs partition.
We are using project quota's on directories within this partition and exporting this.
When a client tries to untar in the dir with project quotas set and reaches the quota limit the load on the NFS server goes up dramatically and stays like this for a few minutes after the tar has gone through trying to write all of it files and failing due to quota limit exceeded.
Is this a known issue and if so is there anything that can be done to resolve this.
We do not see this issue if on the locally mounted partition. This only happens over NFS.
Please let me know if you need more information or if this should be addressed on a different list.
TIA,
Brian
On Fri, Mar 13, 2009 at 09:59:26AM -0700, Brian Wince wrote:
> All,
> We have a Quad-core server running gentoo 2.6.27-r7 that is used as an nfs server for a xfs partition.
> We are using project quota's on directories within this partition and exporting this.
> When a client tries to untar in the dir with project quotas set and reaches the quota limit the load on the NFS server goes up dramatically and stays like this for a few minutes after the tar has gone through trying to write all of it files and failing due to quota limit exceeded.
>
> Is this a known issue and if so is there anything that can be done to resolve this.
>
> We do not see this issue if on the locally mounted partition. This only happens over NFS.
>
> Please let me know if you need more information or if this should be addressed on a different list.
It doesn't sounds familiar to me. Might also be worth cc:'ing xfs
people.
--b.
J. Bruce Fields wrote:
> On Fri, Mar 13, 2009 at 09:59:26AM -0700, Brian Wince wrote:
>
>> All,
>> We have a Quad-core server running gentoo 2.6.27-r7 that is used as an nfs server for a xfs partition.
>> We are using project quota's on directories within this partition and exporting this.
>> When a client tries to untar in the dir with project quotas set and reaches the quota limit the load on the NFS server goes up dramatically and stays like this for a few minutes after the tar has gone through trying to write all of it files and failing due to quota limit exceeded.
>>
>> Is this a known issue and if so is there anything that can be done to resolve this.
>>
>> We do not see this issue if on the locally mounted partition. This only happens over NFS.
>>
>> Please let me know if you need more information or if this should be addressed on a different list.
>>
>
> It doesn't sounds familiar to me. Might also be worth cc:'ing xfs
> people.
>
>
>
Oh dear. Brian asked it there frst, and I redirected him here. In the
absence of network traces, it smelt to me like the old client problem
where errors like ENOSPC or EDQUOT during writeback were only returned
to the app on close() and not the next write(), thus causing clients
which exceed quota during long streaming writes pointlessly to send a
lot of data to the server all of which is dropped on the server.
--
Greg Banks, P.Engineer, SGI Australian Software Group.
the brightly coloured sporks of revolution.
I don't speak for SGI.
On Mon, 2009-03-16 at 14:03 +1100, Greg Banks wrote:
> J. Bruce Fields wrote:
> > On Fri, Mar 13, 2009 at 09:59:26AM -0700, Brian Wince wrote:
> >
> >> All,
> >> We have a Quad-core server running gentoo 2.6.27-r7 that is used as an nfs server for a xfs partition.
> >> We are using project quota's on directories within this partition and exporting this.
> >> When a client tries to untar in the dir with project quotas set and reaches the quota limit the load on the NFS server goes up dramatically and stays like this for a few minutes after the tar has gone through trying to write all of it files and failing due to quota limit exceeded.
> >>
> >> Is this a known issue and if so is there anything that can be done to resolve this.
> >>
> >> We do not see this issue if on the locally mounted partition. This only happens over NFS.
> >>
> >> Please let me know if you need more information or if this should be addressed on a different list.
> >>
> >
> > It doesn't sounds familiar to me. Might also be worth cc:'ing xfs
> > people.
> >
> >
> >
> Oh dear. Brian asked it there frst, and I redirected him here. In the
> absence of network traces, it smelt to me like the old client problem
> where errors like ENOSPC or EDQUOT during writeback were only returned
> to the app on close() and not the next write(), thus causing clients
> which exceed quota during long streaming writes pointlessly to send a
> lot of data to the server all of which is dropped on the server.
>
The patch "NFS: Fall back to synchronous writes when a background write
errors..."
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git&a=commitdiff&h=7b159fc18d417980f57aef64cab3417ee6af70f8
was merged prior to 2.6.27.
That helps catch the problem earlier on the client, but IIRC XFS had
(has?) some kind of advanced error recovery routines that exacerbated
the problem on the server side by taking forever to figure out that this
was an irrecoverable error.
Cheers
Trond