2007-08-23 18:46:55

by Zoot

[permalink] [raw]
Subject: umount fails with device busy

Hi guys,

I get the device busy error when I try to umount an NFS file system althoug=
h there is no process with an open file on it (at least fuser -m /mnt/point=
and lsof +f -- /mnt/point don't find anything).

To be more precise this only happens, when I run the umount right after sto=
pping an application which has its executables and work directory on this f=
ile system. Running umount again some seconds later works fine. =


I do not have any submounts under this mount point. There are no packet tra=
nsmit or receive errors nor any NFS/RPC errors only a very small number of =
RPC retransmitts. So it seems that I have a stable environment. Client and =
server is SLES9 with 2.6.5-7.202.7-smp/X86_64. =


I enabled NFS debug output and the only noticeable thing is, that some NFS =
operations are still ongoing after the application processes have already t=
erminated:

Aug 23 08:25:23 marvin kernel: NFS reply getattr
Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296 ct=3D1 i=
nfo=3D0x6)
Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation complete
Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
Aug 23 08:25:23 marvin kernel: NFS call getattr
Aug 23 08:25:23 marvin kernel: NFS reply getattr
Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=3D1 info=
=3D0x6)
Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
Aug 23 08:25:25 marvin kernel: NFS call fsstat
Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
Aug 23 08:25:35 marvin kernel: NFS call fsstat
Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS: dentr=
y_delete(log/mux_marvin.trc, 0)
Aug 23 08:25:38 marvin kernel: OK
Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS: dentr=
y_delete(log/pw_marvin_1.trc, 0)
Aug 23 08:25:38 marvin kernel: OK
Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS: dentr=
y_delete(log/pw_marvin_0.trc, 0)
Aug 23 08:25:38 marvin kernel: OK
Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS: dentry=
_delete(net.trc, 0)
Aug 23 08:25:38 marvin kernel: OK
Aug 23 08:25:45 marvin kernel: NFS call fsstat
Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
Aug 23 08:25:55 marvin kernel: NFS call fsstat

When I run umount during these ops I get an device busy. Any help would be =
very appreciated!


-- =

Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! =

Ideal f=FCr Modem und ISDN: http://www.gmx.net/de/go/smartsurfer

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs


2007-08-23 20:14:22

by Zoot

[permalink] [raw]
Subject: Re: umount fails with device busy

On Thu, Aug 23, 2007 at 04:03:17PM -0400, Chuck Lever wrote:
> [email protected] wrote:
> >Hi guys,
> >
> >I get the device busy error when I try to umount an NFS file system
> >although there is no process with an open file on it (at least fuser -m
> >/mnt/point and lsof +f -- /mnt/point don't find anything).
> >
> >To be more precise this only happens, when I run the umount right after
> >stopping an application which has its executables and work directory on
> >this file system. Running umount again some seconds later works fine.
> >I do not have any submounts under this mount point. There are no packet
> >transmit or receive errors nor any NFS/RPC errors only a very small number
> >of RPC retransmitts. So it seems that I have a stable environment. Client
> >and server is SLES9 with 2.6.5-7.202.7-smp/X86_64.
> >I enabled NFS debug output and the only noticeable thing is, that some NFS
> >operations are still ongoing after the application processes have already
> >terminated:
> >
> > Aug 23 08:25:23 marvin kernel: NFS reply getattr
> > Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296 ct=1
> > info=0x6)
> > Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation complete
> > Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
> > Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
> > Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
> > Aug 23 08:25:23 marvin kernel: NFS call getattr
> > Aug 23 08:25:23 marvin kernel: NFS reply getattr
> > Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=1
> > info=0x6)
> > Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
> > Aug 23 08:25:25 marvin kernel: NFS call fsstat
> > Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
> > Aug 23 08:25:35 marvin kernel: NFS call fsstat
> > Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
> > Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
> > Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
> > Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
> > Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
> > Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
> > Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS:
> > dentry_delete(log/mux_marvin.trc, 0)
> > Aug 23 08:25:38 marvin kernel: OK
> > Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
> > Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS:
> > dentry_delete(log/pw_marvin_1.trc, 0)
> > Aug 23 08:25:38 marvin kernel: OK
> > Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
> > Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS:
> > dentry_delete(log/pw_marvin_0.trc, 0)
> > Aug 23 08:25:38 marvin kernel: OK
> > Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
> > Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS:
> > dentry_delete(net.trc, 0)
> > Aug 23 08:25:38 marvin kernel: OK
> > Aug 23 08:25:45 marvin kernel: NFS call fsstat
> > Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
> > Aug 23 08:25:55 marvin kernel: NFS call fsstat
> >
> >When I run umount during these ops I get an device busy. Any help would be
> >very appreciated!
>
> This is normal and expected behavior. One problem may be that your
> server is slow, and thus there are RPCs left outstanding for a bit on
> your client after your application exits. The COMMIT calls from that
> trace suggest that there is dirty data the client is writing back to the
> server.
But I had expected that umount blocks until the commit calls are finished as it is with local file systems?

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-23 20:29:46

by Peter Staubach

[permalink] [raw]
Subject: Re: umount fails with device busy

Chuck Lever wrote:
> [email protected] wrote:
>> Hi guys,
>>
>> I get the device busy error when I try to umount an NFS file system
>> although there is no process with an open file on it (at least fuser
>> -m /mnt/point and lsof +f -- /mnt/point don't find anything).
>>
>> To be more precise this only happens, when I run the umount right
>> after stopping an application which has its executables and work
>> directory on this file system. Running umount again some seconds
>> later works fine.
>> I do not have any submounts under this mount point. There are no
>> packet transmit or receive errors nor any NFS/RPC errors only a very
>> small number of RPC retransmitts. So it seems that I have a stable
>> environment. Client and server is SLES9 with 2.6.5-7.202.7-smp/X86_64.
>> I enabled NFS debug output and the only noticeable thing is, that
>> some NFS operations are still ongoing after the application processes
>> have already terminated:
>>
>> Aug 23 08:25:23 marvin kernel: NFS reply getattr
>> Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296
>> ct=1 info=0x6)
>> Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation
>> complete
>> Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
>> Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
>> Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
>> Aug 23 08:25:23 marvin kernel: NFS call getattr
>> Aug 23 08:25:23 marvin kernel: NFS reply getattr
>> Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=1
>> info=0x6)
>> Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
>> Aug 23 08:25:25 marvin kernel: NFS call fsstat
>> Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
>> Aug 23 08:25:35 marvin kernel: NFS call fsstat
>> Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
>> Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
>> Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
>> Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
>> Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
>> Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS:
>> dentry_delete(log/mux_marvin.trc, 0)
>> Aug 23 08:25:38 marvin kernel: OK
>> Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS:
>> dentry_delete(log/pw_marvin_1.trc, 0)
>> Aug 23 08:25:38 marvin kernel: OK
>> Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS:
>> dentry_delete(log/pw_marvin_0.trc, 0)
>> Aug 23 08:25:38 marvin kernel: OK
>> Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS:
>> dentry_delete(net.trc, 0)
>> Aug 23 08:25:38 marvin kernel: OK
>> Aug 23 08:25:45 marvin kernel: NFS call fsstat
>> Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
>> Aug 23 08:25:55 marvin kernel: NFS call fsstat
>>
>> When I run umount during these ops I get an device busy. Any help
>> would be very appreciated!
>
> This is normal and expected behavior. One problem may be that your
> server is slow, and thus there are RPCs left outstanding for a bit on
> your client after your application exits. The COMMIT calls from that
> trace suggest that there is dirty data the client is writing back to
> the server.

It seems to me that this should not be the expected behavior unless
the file system is mounted "nocto". Is it?

Thanx...

ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-23 20:39:22

by Zoot

[permalink] [raw]
Subject: Re: umount fails with device busy

On Thu, Aug 23, 2007 at 04:29:46PM -0400, Peter Staubach wrote:
> Chuck Lever wrote:
> >[email protected] wrote:
> >>Hi guys,
> >>
> >>I get the device busy error when I try to umount an NFS file system
> >>although there is no process with an open file on it (at least fuser
> >>-m /mnt/point and lsof +f -- /mnt/point don't find anything).
> >>
> >>To be more precise this only happens, when I run the umount right
> >>after stopping an application which has its executables and work
> >>directory on this file system. Running umount again some seconds
> >>later works fine.
> >>I do not have any submounts under this mount point. There are no
> >>packet transmit or receive errors nor any NFS/RPC errors only a very
> >>small number of RPC retransmitts. So it seems that I have a stable
> >>environment. Client and server is SLES9 with 2.6.5-7.202.7-smp/X86_64.
> >>I enabled NFS debug output and the only noticeable thing is, that
> >>some NFS operations are still ongoing after the application processes
> >>have already terminated:
> >>
> >> Aug 23 08:25:23 marvin kernel: NFS reply getattr
> >> Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296
> >>ct=1 info=0x6)
> >> Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation
> >>complete
> >> Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
> >> Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
> >> Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
> >> Aug 23 08:25:23 marvin kernel: NFS call getattr
> >> Aug 23 08:25:23 marvin kernel: NFS reply getattr
> >> Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=1
> >>info=0x6)
> >> Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
> >> Aug 23 08:25:25 marvin kernel: NFS call fsstat
> >> Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
> >> Aug 23 08:25:35 marvin kernel: NFS call fsstat
> >> Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
> >> Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
> >> Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
> >> Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
> >> Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
> >> Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
> >> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS:
> >>dentry_delete(log/mux_marvin.trc, 0)
> >> Aug 23 08:25:38 marvin kernel: OK
> >> Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
> >> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS:
> >>dentry_delete(log/pw_marvin_1.trc, 0)
> >> Aug 23 08:25:38 marvin kernel: OK
> >> Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
> >> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS:
> >>dentry_delete(log/pw_marvin_0.trc, 0)
> >> Aug 23 08:25:38 marvin kernel: OK
> >> Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
> >> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS:
> >>dentry_delete(net.trc, 0)
> >> Aug 23 08:25:38 marvin kernel: OK
> >> Aug 23 08:25:45 marvin kernel: NFS call fsstat
> >> Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
> >> Aug 23 08:25:55 marvin kernel: NFS call fsstat
> >>
> >>When I run umount during these ops I get an device busy. Any help
> >>would be very appreciated!
> >
> >This is normal and expected behavior. One problem may be that your
> >server is slow, and thus there are RPCs left outstanding for a bit on
> >your client after your application exits. The COMMIT calls from that
> >trace suggest that there is dirty data the client is writing back to
> >the server.
>
> It seems to me that this should not be the expected behavior unless
> the file system is mounted "nocto". Is it?
>
> Thanx...
>
> ps
No, mount options are: nfsvers=3,soft,tcp,rsize=32768,wsize=32768

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-23 20:53:19

by Peter Staubach

[permalink] [raw]
Subject: Re: umount fails with device busy

Chuck Lever wrote:
>>>> When I run umount during these ops I get an device busy. Any help
>>>> would be very appreciated!
>>>
>>> This is normal and expected behavior. One problem may be that your
>>> server is slow, and thus there are RPCs left outstanding for a bit
>>> on your client after your application exits. The COMMIT calls from
>>> that trace suggest that there is dirty data the client is writing
>>> back to the server.
>>
>> It seems to me that this should not be the expected behavior unless
>> the file system is mounted "nocto". Is it?
>
> I'm a little puzzled myself about what dirty data there might be left
> after the application quits. However, I'll be there is an mmap()
> lurking somewhere in the background...
>
> Data that was dirtied via a mapped file is not subject to the
> writeback part of close-to-open.

I don't think that I agree with this last statement. Although
it can not be implemented using the normal close system call,
something should trigger the flush of any dirty pages and wait
for them to complete.

I view close-to-open as a semantic and not just as a syntax.
I view it as "check on first reference" and "flush when last
reference is released".

Thanx...

ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-23 20:53:59

by Peter Staubach

[permalink] [raw]
Subject: Re: umount fails with device busy

Zoot wrote:
> On Thu, Aug 23, 2007 at 04:29:46PM -0400, Peter Staubach wrote:
>
>> Chuck Lever wrote:
>>
>>> [email protected] wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I get the device busy error when I try to umount an NFS file system
>>>> although there is no process with an open file on it (at least fuser
>>>> -m /mnt/point and lsof +f -- /mnt/point don't find anything).
>>>>
>>>> To be more precise this only happens, when I run the umount right
>>>> after stopping an application which has its executables and work
>>>> directory on this file system. Running umount again some seconds
>>>> later works fine.
>>>> I do not have any submounts under this mount point. There are no
>>>> packet transmit or receive errors nor any NFS/RPC errors only a very
>>>> small number of RPC retransmitts. So it seems that I have a stable
>>>> environment. Client and server is SLES9 with 2.6.5-7.202.7-smp/X86_64.
>>>> I enabled NFS debug output and the only noticeable thing is, that
>>>> some NFS operations are still ongoing after the application processes
>>>> have already terminated:
>>>>
>>>> Aug 23 08:25:23 marvin kernel: NFS reply getattr
>>>> Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296
>>>> ct=1 info=0x6)
>>>> Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation
>>>> complete
>>>> Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
>>>> Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
>>>> Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
>>>> Aug 23 08:25:23 marvin kernel: NFS call getattr
>>>> Aug 23 08:25:23 marvin kernel: NFS reply getattr
>>>> Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=1
>>>> info=0x6)
>>>> Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
>>>> Aug 23 08:25:25 marvin kernel: NFS call fsstat
>>>> Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
>>>> Aug 23 08:25:35 marvin kernel: NFS call fsstat
>>>> Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
>>>> Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
>>>> Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
>>>> Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
>>>> Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
>>>> Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
>>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS:
>>>> dentry_delete(log/mux_marvin.trc, 0)
>>>> Aug 23 08:25:38 marvin kernel: OK
>>>> Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
>>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS:
>>>> dentry_delete(log/pw_marvin_1.trc, 0)
>>>> Aug 23 08:25:38 marvin kernel: OK
>>>> Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
>>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS:
>>>> dentry_delete(log/pw_marvin_0.trc, 0)
>>>> Aug 23 08:25:38 marvin kernel: OK
>>>> Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
>>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS:
>>>> dentry_delete(net.trc, 0)
>>>> Aug 23 08:25:38 marvin kernel: OK
>>>> Aug 23 08:25:45 marvin kernel: NFS call fsstat
>>>> Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
>>>> Aug 23 08:25:55 marvin kernel: NFS call fsstat
>>>>
>>>> When I run umount during these ops I get an device busy. Any help
>>>> would be very appreciated!
>>>>
>>> This is normal and expected behavior. One problem may be that your
>>> server is slow, and thus there are RPCs left outstanding for a bit on
>>> your client after your application exits. The COMMIT calls from that
>>> trace suggest that there is dirty data the client is writing back to
>>> the server.
>>>
>> It seems to me that this should not be the expected behavior unless
>> the file system is mounted "nocto". Is it?
>>
>> Thanx...
>>
>> ps
>>
> No, mount options are: nfsvers=3,soft,tcp,rsize=32768,wsize=32768
>

"soft"?

Dangerous.

ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-23 21:05:47

by Zoot

[permalink] [raw]
Subject: Re: umount fails with device busy

On Thu, Aug 23, 2007 at 04:53:58PM -0400, Peter Staubach wrote:
> Zoot wrote:
> >On Thu, Aug 23, 2007 at 04:29:46PM -0400, Peter Staubach wrote:
> >
> >>Chuck Lever wrote:
> >>
> >>>[email protected] wrote:
> >>>
> >>>>Hi guys,
> >>>>
> >>>>I get the device busy error when I try to umount an NFS file system
> >>>>although there is no process with an open file on it (at least fuser
> >>>>-m /mnt/point and lsof +f -- /mnt/point don't find anything).
> >>>>
> >>>>To be more precise this only happens, when I run the umount right
> >>>>after stopping an application which has its executables and work
> >>>>directory on this file system. Running umount again some seconds
> >>>>later works fine.
> >>>>I do not have any submounts under this mount point. There are no
> >>>>packet transmit or receive errors nor any NFS/RPC errors only a very
> >>>>small number of RPC retransmitts. So it seems that I have a stable
> >>>>environment. Client and server is SLES9 with 2.6.5-7.202.7-smp/X86_64.
> >>>>I enabled NFS debug output and the only noticeable thing is, that
> >>>>some NFS operations are still ongoing after the application processes
> >>>>have already terminated:
> >>>>
> >>>> Aug 23 08:25:23 marvin kernel: NFS reply getattr
> >>>> Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296
> >>>>ct=1 info=0x6)
> >>>> Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation
> >>>>complete
> >>>> Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
> >>>> Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
> >>>> Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
> >>>> Aug 23 08:25:23 marvin kernel: NFS call getattr
> >>>> Aug 23 08:25:23 marvin kernel: NFS reply getattr
> >>>> Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=1
> >>>>info=0x6)
> >>>> Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
> >>>> Aug 23 08:25:25 marvin kernel: NFS call fsstat
> >>>> Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
> >>>> Aug 23 08:25:35 marvin kernel: NFS call fsstat
> >>>> Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
> >>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS:
> >>>>dentry_delete(log/mux_marvin.trc, 0)
> >>>> Aug 23 08:25:38 marvin kernel: OK
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
> >>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS:
> >>>>dentry_delete(log/pw_marvin_1.trc, 0)
> >>>> Aug 23 08:25:38 marvin kernel: OK
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
> >>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS:
> >>>>dentry_delete(log/pw_marvin_0.trc, 0)
> >>>> Aug 23 08:25:38 marvin kernel: OK
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
> >>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS:
> >>>>dentry_delete(net.trc, 0)
> >>>> Aug 23 08:25:38 marvin kernel: OK
> >>>> Aug 23 08:25:45 marvin kernel: NFS call fsstat
> >>>> Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
> >>>> Aug 23 08:25:55 marvin kernel: NFS call fsstat
> >>>>
> >>>>When I run umount during these ops I get an device busy. Any help
> >>>>would be very appreciated!
> >>>>
> >>>This is normal and expected behavior. One problem may be that your
> >>>server is slow, and thus there are RPCs left outstanding for a bit on
> >>>your client after your application exits. The COMMIT calls from that
> >>>trace suggest that there is dirty data the client is writing back to
> >>>the server.
> >>>
> >>It seems to me that this should not be the expected behavior unless
> >>the file system is mounted "nocto". Is it?
> >>
> >> Thanx...
> >>
> >> ps
> >>
> >No, mount options are: nfsvers=3,soft,tcp,rsize=32768,wsize=32768
> >
>
> "soft"?
>
> Dangerous.
>
> ps
Yes, I know I know. But it's the same with "hard".

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-23 21:15:27

by Zoot

[permalink] [raw]
Subject: Re: umount fails with device busy

On Thu, Aug 23, 2007 at 04:35:28PM -0400, Chuck Lever wrote:
> >>>When I run umount during these ops I get an device busy. Any help
> >>>would be very appreciated!
> >>
> >>This is normal and expected behavior. One problem may be that your
> >>server is slow, and thus there are RPCs left outstanding for a bit on
> >>your client after your application exits. The COMMIT calls from that
> >>trace suggest that there is dirty data the client is writing back to
> >>the server.
> >
> >It seems to me that this should not be the expected behavior unless
> >the file system is mounted "nocto". Is it?
>
> I'm a little puzzled myself about what dirty data there might be left
> after the application quits. However, I'll be there is an mmap()
> lurking somewhere in the background...
>
> Data that was dirtied via a mapped file is not subject to the writeback
> part of close-to-open.
At least according to the NFS debug output are the commits only related
to .trc files which are used by the app to store it's debug output. Not
used for mmap.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-24 13:43:50

by Peter Staubach

[permalink] [raw]
Subject: Re: umount fails with device busy

Chuck Lever wrote:
>
>
> Peter Staubach wrote:
>> Chuck Lever wrote:
>>>>>> When I run umount during these ops I get an device busy. Any help
>>>>>> would be very appreciated!
>>>>>
>>>>> This is normal and expected behavior. One problem may be that
>>>>> your server is slow, and thus there are RPCs left outstanding for
>>>>> a bit on your client after your application exits. The COMMIT
>>>>> calls from that trace suggest that there is dirty data the client
>>>>> is writing back to the server.
>>>>
>>>> It seems to me that this should not be the expected behavior unless
>>>> the file system is mounted "nocto". Is it?
>>>
>>> I'm a little puzzled myself about what dirty data there might be
>>> left after the application quits. However, I'll be there is an
>>> mmap() lurking somewhere in the background...
>>>
>>> Data that was dirtied via a mapped file is not subject to the
>>> writeback part of close-to-open.
>>
>> I don't think that I agree with this last statement. Although
>> it can not be implemented using the normal close system call,
>> something should trigger the flush of any dirty pages and wait
>> for them to complete.
>
> This is the way Linux implements dirty mmaps on NFS because that's the
> way Linus wants it. I didn't mean to imply that's the way it *should*
> work, merely the way it *does* work in Linux.

Yes, I recognize that. That's the way that it worked in Solaris
too, until I decided that it should work differently and implemented
it.

That said, I don't know how hard it would be to sell the new
semantic to Linus. This may tie in with some other work that
I am doing, so perhaps I will give it a shot.

Thanx...

ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-24 18:52:56

by Zoot

[permalink] [raw]
Subject: Re: umount fails with device busy

On Thu, Aug 23, 2007 at 04:32:50PM -0400, Chuck Lever wrote:
> Zoot wrote:
> >>This is normal and expected behavior. One problem may be that your
> >>server is slow, and thus there are RPCs left outstanding for a bit on
> >>your client after your application exits. The COMMIT calls from that
> >>trace suggest that there is dirty data the client is writing back to the
> >>server.
> >But I had expected that umount blocks until the commit calls are
> > finished as it is with local file systems?
>
> Sorry, it doesn't. It has no way of knowing how long those RPCs will take.
>
> There is a "lazy unmount" facility that removes the NFS share from your
> client's name space and handles the outstanding RPCs in the background.
> I'm not sure if it's working, but you could try "umount -l".
What's with sync() would it wait for the nfs commits?

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs