Hi Bruce, Trond.
As you know, Currently umount results in busy on NFS server although
user tried to succeed to umount on NFS client.
I suggest to add umount procedure to avoid umount busy issue.
When calling umount on NFS client, The resources(exportfs entries
cache) of mount point will be flushed on NFS server. and umount will
be succeed without busy issue.
how do you think about this suggestion ?
Thanks.
2012/5/11, AMIT SAHRAWAT <[email protected]>:
> Hi Bruce,
> Sorry it was the mail composer - I did not check mark the plain text
> option.
>
> We need an opinion.
> We will be using around 15 NFS client mounted - with one NFS Server. We are
> try to make out how many NFSD should we start at the NFS Server.
> It all depends upon the load - these 15 NFS clients are going to generate.
> But is there any default number - we should use to start with? like 1 per
> NFS client mount?
> Please share your advise on this.
>
> Thanks & Regards,
> Amit Sahrawat
>
> ------- Original Message -------
> Sender : J. Bruce Fields<[email protected]>
> Date : May 10, 2012 17:46 (GMT+05:30)
> Title : Re: Re: [Problem]NFS Server – Um?ount results in Device Busy.
>
> On Thu, May 10, 2012 at 11:54:43AM +0000, AMIT SAHRAWAT wrote:
>>
>>
> Hi Bruce,
>
>>
>
>
>>
> Thanks for instant reply.
>
>
> Could you turn off html formatting on your mail?
>
>>
> The option you mentioned is working. 'exportfs -f'
>
>>
> Actually it is not mentioned anywhere - even when checked for options:
>
>>
> #> exportfs -h
> exportfs: invalid option -- 'h'
> usage: exportfs [-aruv] [host:/path]
>
>
>
> Sorry, it looks like the help test could be improved! Patches
> welcomed....
>
> It is documented in the exportfs man page.
>
>>
> But - when tried to give - '-f' option it is working.
>
>>
> Then on checking the code for exportfs - got to know the working of '-f'
> option.
>
>>
> I think it should work. I will check with a number of scenario - to check
> there is no blockage for other mount points - at the time 'exportfs -f' is
> being run. Please reply if it wont :)
>
>
> If no clients are using the export, it should work.
>
> --b.
>
>
>
> Thanks & Regards,
> Amit Sahrawat
On 06/07/2012 12:43 PM, Namjae Jeon wrote:
> 2012/6/7, Boaz Harrosh <[email protected]>:
>> On 06/07/2012 10:47 AM, Namjae Jeon wrote:
>>
>>> Hi Bruce, Trond.
>>>
>>> As you know, Currently umount results in busy on NFS server although
>>> user tried to succeed to umount on NFS client.
>>> I suggest to add umount procedure to avoid umount busy issue.
>>> When calling umount on NFS client, The resources(exportfs entries
>>> cache) of mount point will be flushed on NFS server. and umount will
>>> be succeed without busy issue.
>>>
>>> how do you think about this suggestion ?
>>>
>>
>>
>> I second this request, what is needed so when all clients unmounted,
>> the system comes back to the sate it was before any clients have
>> mounted. i.e filesystems is not referenced and may unmount cleanly.
>> (This also happens with >= 4.0 clients only, so no excuses)
>>
>> This is a real problem for me, on Fedora machines. Because I export
>> iscsi devices which are network devices, and in the shutdown procedure
>> for some reason the "service nfs stop" of the server is much much to
>> late. the original umount of exofs (-o _netdev) fails because it's
>> held by NFSD, the iscsi devices go away regardless, and when nfsd
>> finally releases exofs, it gets deadlocked on some error handling.
>> OK I know I must fix the stuck-ness, but the problem will remain.
>> The FS will not unmount cleanly because it will only attempt
>> an unmount after its devices are gone. This will be solved if
>> nfsd would release its hold on the FS when all clients are gone.
>>
>> It was on my TODO to fix this for a long time, but I seem to be
>> too busy with more urgent matters. (What's the point of fixing the
>> shutdown if the steady state doesn't work yet)
>>
>> If someone has investigated the matter and knows what to do I would
>> appreciate any insights, and/or patches would be wonderful ;-)
> Hi Boaz.
> Oh.. You also are facing same problem.
> Actually I almost finish to imprement the patch about umount busy
> issue on NFS server. But I want to know Maintainer's opinion before
> posting.
> Thanks.
Please post it (maybe as RFC) I want to see it
Thanks
Boaz
>>
>>> Thanks.
>>>
>>
>> <>
>>
>> Thanks
>> Boaz
>>
>>
On 06/07/2012 10:47 AM, Namjae Jeon wrote:
> Hi Bruce, Trond.
>
> As you know, Currently umount results in busy on NFS server although
> user tried to succeed to umount on NFS client.
> I suggest to add umount procedure to avoid umount busy issue.
> When calling umount on NFS client, The resources(exportfs entries
> cache) of mount point will be flushed on NFS server. and umount will
> be succeed without busy issue.
>
> how do you think about this suggestion ?
>
I second this request, what is needed so when all clients unmounted,
the system comes back to the sate it was before any clients have
mounted. i.e filesystems is not referenced and may unmount cleanly.
(This also happens with >= 4.0 clients only, so no excuses)
This is a real problem for me, on Fedora machines. Because I export
iscsi devices which are network devices, and in the shutdown procedure
for some reason the "service nfs stop" of the server is much much to
late. the original umount of exofs (-o _netdev) fails because it's
held by NFSD, the iscsi devices go away regardless, and when nfsd
finally releases exofs, it gets deadlocked on some error handling.
OK I know I must fix the stuck-ness, but the problem will remain.
The FS will not unmount cleanly because it will only attempt
an unmount after its devices are gone. This will be solved if
nfsd would release its hold on the FS when all clients are gone.
It was on my TODO to fix this for a long time, but I seem to be
too busy with more urgent matters. (What's the point of fixing the
shutdown if the steady state doesn't work yet)
If someone has investigated the matter and knows what to do I would
appreciate any insights, and/or patches would be wonderful ;-)
> Thanks.
>
<>
Thanks
Boaz
2012/6/7, Boaz Harrosh <[email protected]>:
> On 06/07/2012 12:43 PM, Namjae Jeon wrote:
>
>> 2012/6/7, Boaz Harrosh <[email protected]>:
>>> On 06/07/2012 10:47 AM, Namjae Jeon wrote:
>>>
>>>> Hi Bruce, Trond.
>>>>
>>>> As you know, Currently umount results in busy on NFS server although
>>>> user tried to succeed to umount on NFS client.
>>>> I suggest to add umount procedure to avoid umount busy issue.
>>>> When calling umount on NFS client, The resources(exportfs entries
>>>> cache) of mount point will be flushed on NFS server. and umount will
>>>> be succeed without busy issue.
>>>>
>>>> how do you think about this suggestion ?
>>>>
>>>
>>>
>>> I second this request, what is needed so when all clients unmounted,
>>> the system comes back to the sate it was before any clients have
>>> mounted. i.e filesystems is not referenced and may unmount cleanly.
>>> (This also happens with >= 4.0 clients only, so no excuses)
>>>
>>> This is a real problem for me, on Fedora machines. Because I export
>>> iscsi devices which are network devices, and in the shutdown procedure
>>> for some reason the "service nfs stop" of the server is much much to
>>> late. the original umount of exofs (-o _netdev) fails because it's
>>> held by NFSD, the iscsi devices go away regardless, and when nfsd
>>> finally releases exofs, it gets deadlocked on some error handling.
>>> OK I know I must fix the stuck-ness, but the problem will remain.
>>> The FS will not unmount cleanly because it will only attempt
>>> an unmount after its devices are gone. This will be solved if
>>> nfsd would release its hold on the FS when all clients are gone.
>>>
>>> It was on my TODO to fix this for a long time, but I seem to be
>>> too busy with more urgent matters. (What's the point of fixing the
>>> shutdown if the steady state doesn't work yet)
>>>
>>> If someone has investigated the matter and knows what to do I would
>>> appreciate any insights, and/or patches would be wonderful ;-)
>> Hi Boaz.
>> Oh.. You also are facing same problem.
>> Actually I almost finish to imprement the patch about umount busy
>> issue on NFS server. But I want to know Maintainer's opinion before
>> posting.
>> Thanks.
>
>
> Please post it (maybe as RFC) I want to see it
Okay, I will soon.
>
> Thanks
> Boaz
>
>>>
>>>> Thanks.
>>>>
>>>
>>> <>
>>>
>>> Thanks
>>> Boaz
>>>
>>>
>
>
>
On Fri, Jun 08, 2012 at 12:59:10AM +0300, Boaz Harrosh wrote:
> On 06/07/2012 02:33 PM, J. Bruce Fields wrote:
>
> > On Thu, Jun 07, 2012 at 11:13:04AM +0300, Boaz Harrosh wrote:
>
>
> >
> > I'm not sure how adding an "unmount" rpc to the protocol would really
> > help here, if that's what you're asking for.
> >
>
>
> Please elaborate a bit. I'm not familiar with the details of this
> discussion.
The mount protocol used with NFSv2/v3, for example, has mount/unmount
calls, and I seem to recall problems with keeping the list of mounted
clients accurate in the face of unreliable clients and networks.
I guess you could add the same but make them subject to the lease, so a
client would be assumed "unmounted" if it lost its lease. I don't know.
But the reported problems here sound like just simple bugs, so I'm not
interested in such discussion till the bugs are investigated and fixed
and someone explains what problem then remains to be solved.
--b.
On Thu, Jun 07, 2012 at 11:13:04AM +0300, Boaz Harrosh wrote:
> On 06/07/2012 10:47 AM, Namjae Jeon wrote:
> > As you know, Currently umount results in busy on NFS server although
> > user tried to succeed to umount on NFS client.
> > I suggest to add umount procedure to avoid umount busy issue.
> > When calling umount on NFS client, The resources(exportfs entries
> > cache) of mount point will be flushed on NFS server. and umount will
> > be succeed without busy issue.
> >
> > how do you think about this suggestion ?
I'm not sure how adding an "unmount" rpc to the protocol would really
help here, if that's what you're asking for.
If you're just looking for a command you could run on the server after
all the clients have unmounted--"exportfs -f" or "service stop nfsd" (or
equivalent) should do the job.
> I second this request, what is needed so when all clients unmounted,
> the system comes back to the sate it was before any clients have
> mounted. i.e filesystems is not referenced and may unmount cleanly.
> (This also happens with >= 4.0 clients only, so no excuses)
>
> This is a real problem for me, on Fedora machines. Because I export
> iscsi devices which are network devices, and in the shutdown procedure
> for some reason the "service nfs stop" of the server is much much to
> late.
Sounds like a bug in the Fedora systemd configuration?
> the original umount of exofs (-o _netdev) fails because it's
> held by NFSD, the iscsi devices go away regardless, and when nfsd
> finally releases exofs, it gets deadlocked on some error handling.
> OK I know I must fix the stuck-ness, but the problem will remain.
> The FS will not unmount cleanly because it will only attempt
> an unmount after its devices are gone. This will be solved if
> nfsd would release its hold on the FS when all clients are gone.
Note nfsd doesn't really know when that is. Even with NFSv4, processes
can be using the filesystem without holding state on the server: they
might just have a current working directory in the filesystem, or have a
device special file open.
> It was on my TODO to fix this for a long time, but I seem to be
> too busy with more urgent matters. (What's the point of fixing the
> shutdown if the steady state doesn't work yet)
>
> If someone has investigated the matter and knows what to do I would
> appreciate any insights, and/or patches would be wonderful ;-)
My first concern would be to fix any ordering bugs in the systemd
configuration, or any reference count leaks.
--b.
2012/6/7, Boaz Harrosh <[email protected]>:
> On 06/07/2012 10:47 AM, Namjae Jeon wrote:
>
>> Hi Bruce, Trond.
>>
>> As you know, Currently umount results in busy on NFS server although
>> user tried to succeed to umount on NFS client.
>> I suggest to add umount procedure to avoid umount busy issue.
>> When calling umount on NFS client, The resources(exportfs entries
>> cache) of mount point will be flushed on NFS server. and umount will
>> be succeed without busy issue.
>>
>> how do you think about this suggestion ?
>>
>
>
> I second this request, what is needed so when all clients unmounted,
> the system comes back to the sate it was before any clients have
> mounted. i.e filesystems is not referenced and may unmount cleanly.
> (This also happens with >= 4.0 clients only, so no excuses)
>
> This is a real problem for me, on Fedora machines. Because I export
> iscsi devices which are network devices, and in the shutdown procedure
> for some reason the "service nfs stop" of the server is much much to
> late. the original umount of exofs (-o _netdev) fails because it's
> held by NFSD, the iscsi devices go away regardless, and when nfsd
> finally releases exofs, it gets deadlocked on some error handling.
> OK I know I must fix the stuck-ness, but the problem will remain.
> The FS will not unmount cleanly because it will only attempt
> an unmount after its devices are gone. This will be solved if
> nfsd would release its hold on the FS when all clients are gone.
>
> It was on my TODO to fix this for a long time, but I seem to be
> too busy with more urgent matters. (What's the point of fixing the
> shutdown if the steady state doesn't work yet)
>
> If someone has investigated the matter and knows what to do I would
> appreciate any insights, and/or patches would be wonderful ;-)
Hi Boaz.
Oh.. You also are facing same problem.
Actually I almost finish to imprement the patch about umount busy
issue on NFS server. But I want to know Maintainer's opinion before
posting.
Thanks.
>
>> Thanks.
>>
>
> <>
>
> Thanks
> Boaz
>
>
On 06/07/2012 02:33 PM, J. Bruce Fields wrote:
> On Thu, Jun 07, 2012 at 11:13:04AM +0300, Boaz Harrosh wrote:
>
> I'm not sure how adding an "unmount" rpc to the protocol would really
> help here, if that's what you're asking for.
>
Please elaborate a bit. I'm not familiar with the details of this
discussion.
> If you're just looking for a command you could run on the server after
> all the clients have unmounted--"exportfs -f" or "service stop nfsd" (or
> equivalent) should do the job.
>
man exportfs
-f In 'new' mode, flush everything out of the kernels export table.
Any clients that are active will get new entries added by mountd
when they make their next request.
What if: (Just fantasizing here, if I understand the above text correctly)
* NFS4: A bit like expire client, as long as client has state on the server
he needs to refresh his leases so not to expire with some kind of
activity or renewals.
If a client has no state and 2*expiry timeout passed then it is
individually "exportfs -f". This is safe because on renewed activity
it will get it's mountd entries renewed safely (did I understand
this correctly?
* NFS3: Same as above But with say X*expiry timeout after last activity
Same safeness as above.
Any "unmount rpc" will expedite the above without waiting for timeout
<>
>
> Note nfsd doesn't really know when that is. Even with NFSv4, processes
> can be using the filesystem without holding state on the server: they
> might just have a current working directory in the filesystem, or have a
> device special file open.
>
How does the magic of above statement works? quote:
"will get new entries added by mountd when they make their next request"
<>
>
> My first concern would be to fix any ordering bugs in the systemd
> configuration, or any reference count leaks.
>
I'm currently shooting in the dark, I have not investigated this at all,
do to lack of time. It should be the same with an ext4 over iscsi device
re-exported by NFS. You must mount the ext4 as "-o _netdev" otherwise
ext4 mount itself will have a problem.
> --b.
Thanks Bruce, we'll keep this on a back burner until we can come back to
it. A TODO.
But Jeon, please do send in your proposal code, I'd like to see it.
Thanks
Boaz
On 06/08/2012 12:59 AM, Boaz Harrosh wrote:
<>
>
>> If you're just looking for a command you could run on the server after
>> all the clients have unmounted--"exportfs -f"
BTW: just that one does the trick, thanks learn something new
>> or "service stop nfsd" (or equivalent) should do the job.
>>
That's the one I used to do manually because of the wrong shut down
order in Fedora.
Thanks
Boaz