Signed-off-by: Trond Myklebust <[email protected]>
---
fs/nfs/dir.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c
index aaf7bd0cbae2..a924d66b5608 100644
--- a/fs/nfs/dir.c
+++ b/fs/nfs/dir.c
@@ -424,12 +424,17 @@ static int xdr_decode(nfs_readdir_descriptor_t *desc,
static
int nfs_same_file(struct dentry *dentry, struct nfs_entry *entry)
{
+ struct inode *inode;
struct nfs_inode *nfsi;
if (d_really_is_negative(dentry))
return 0;
- nfsi = NFS_I(d_inode(dentry));
+ inode = d_inode(dentry);
+ if (is_bad_inode(inode) || NFS_STALE(inode))
+ return 0;
+
+ nfsi = NFS_I(inode);
if (entry->fattr->fileid == nfsi->fileid)
return 1;
if (nfs_compare_fh(entry->fh, &nfsi->fh) == 0)
--
2.5.5
Hi Bruce,
I see that setting the number of nfsd threads to 0 (echo 0 >
/proc/fs/nfsd/threads) is not releasing the locks and putting the server
in grace mode. What is the best way to go into grace period, in new
version of the kernel, without restarting the nfs server?
Thanks, Marc.
On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> I see that setting the number of nfsd threads to 0 (echo 0 >
> /proc/fs/nfsd/threads) is not releasing the locks and putting the server
> in grace mode.
Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
certainly drop locks. If that's not happening, there's a bug, but we'd
need to know more details (version numbers, etc.) to help.
That alone has never been enough to start a grace period--you'd have to
start knfsd again to do that.
> What is the best way to go into grace period, in new version of the
> kernel, without restarting the nfs server?
Restarting the nfs server is the only way. That's true on older kernels
true, as far as I know. (OK, you can apparently make lockd do something
like this with a signal, I don't know if that's used much, and I doubt
it works outside an NFSv3-only environment.)
So if you want locks dropped and a new grace period, then you should run
"systemctl restart nfs-server", or your distro's equivalent.
But you're probably doing something more complicated than that. I'm not
sure I understand the question....
--b.
It used to be that sending KILL signal to lockd would free locks and start
Grace period, and when setting nfsd threads to zero, nfsd_last_thread()
calls nfsd_shutdown that called lockd_down that I believe was causing both
freeing of locks and starting grace period or maybe it was setting it back
to a value > 0 that started the grace period.
Any way starting with the kernels that are in RHEL7.1 and up echo 0 >
/proc/fs/nfsd/threads doesn't do it anymore, I assume going to common
grace period for NLM and NFSv4 changed things.
The question is how to do IP fail-over, so when a node fails and the IP is
moving to another node, we need to go into grace period on all the nodes
in the cluster so the locks of the failed node are not given to anyone
other than the client that is reclaiming his locks. Restarting NFS server
is to distractive. For NFSv3 KILL signal to lockd still works but for
NFSv4 have no way to do it for v4.
Marc.
From: Bruce Fields <[email protected]>
To: Marc Eshel/Almaden/IBM@IBMUS
Cc: [email protected]
Date: 07/01/2016 09:09 AM
Subject: Re: grace period
On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> I see that setting the number of nfsd threads to 0 (echo 0 >
> /proc/fs/nfsd/threads) is not releasing the locks and putting the server
> in grace mode.
Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
certainly drop locks. If that's not happening, there's a bug, but we'd
need to know more details (version numbers, etc.) to help.
That alone has never been enough to start a grace period--you'd have to
start knfsd again to do that.
> What is the best way to go into grace period, in new version of the
> kernel, without restarting the nfs server?
Restarting the nfs server is the only way. That's true on older kernels
true, as far as I know. (OK, you can apparently make lockd do something
like this with a signal, I don't know if that's used much, and I doubt
it works outside an NFSv3-only environment.)
So if you want locks dropped and a new grace period, then you should run
"systemctl restart nfs-server", or your distro's equivalent.
But you're probably doing something more complicated than that. I'm not
sure I understand the question....
--b.
On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> It used to be that sending KILL signal to lockd would free locks and start
> Grace period, and when setting nfsd threads to zero, nfsd_last_thread()
> calls nfsd_shutdown that called lockd_down that I believe was causing both
> freeing of locks and starting grace period or maybe it was setting it back
> to a value > 0 that started the grace period.
OK, apologies, I didn't know (or forgot) that.
> Any way starting with the kernels that are in RHEL7.1 and up echo 0 >
> /proc/fs/nfsd/threads doesn't do it anymore, I assume going to common
> grace period for NLM and NFSv4 changed things.
> The question is how to do IP fail-over, so when a node fails and the IP is
> moving to another node, we need to go into grace period on all the nodes
> in the cluster so the locks of the failed node are not given to anyone
> other than the client that is reclaiming his locks. Restarting NFS server
> is to distractive.
What's the difference? Just that clients don't have to reestablish tcp
connections?
--b.
> For NFSv3 KILL signal to lockd still works but for
> NFSv4 have no way to do it for v4.
> Marc.
>
>
>
> From: Bruce Fields <[email protected]>
> To: Marc Eshel/Almaden/IBM@IBMUS
> Cc: [email protected]
> Date: 07/01/2016 09:09 AM
> Subject: Re: grace period
>
>
>
> On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > I see that setting the number of nfsd threads to 0 (echo 0 >
> > /proc/fs/nfsd/threads) is not releasing the locks and putting the server
>
> > in grace mode.
>
> Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> certainly drop locks. If that's not happening, there's a bug, but we'd
> need to know more details (version numbers, etc.) to help.
>
> That alone has never been enough to start a grace period--you'd have to
> start knfsd again to do that.
>
> > What is the best way to go into grace period, in new version of the
> > kernel, without restarting the nfs server?
>
> Restarting the nfs server is the only way. That's true on older kernels
> true, as far as I know. (OK, you can apparently make lockd do something
> like this with a signal, I don't know if that's used much, and I doubt
> it works outside an NFSv3-only environment.)
>
> So if you want locks dropped and a new grace period, then you should run
> "systemctl restart nfs-server", or your distro's equivalent.
>
> But you're probably doing something more complicated than that. I'm not
> sure I understand the question....
>
> --b.
>
>
>
>
[email protected] wrote on 07/01/2016 01:07:42 PM:
> From: Bruce Fields <[email protected]>
> To: Marc Eshel/Almaden/IBM@IBMUS
> Cc: [email protected]
> Date: 07/01/2016 01:07 PM
> Subject: Re: grace period
> Sent by: [email protected]
>
> On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> > It used to be that sending KILL signal to lockd would free locks and
start
> > Grace period, and when setting nfsd threads to zero,
nfsd_last_thread()
> > calls nfsd_shutdown that called lockd_down that I believe was causing
both
> > freeing of locks and starting grace period or maybe it was setting it
back
> > to a value > 0 that started the grace period.
>
> OK, apologies, I didn't know (or forgot) that.
>
> > Any way starting with the kernels that are in RHEL7.1 and up echo 0 >
> > /proc/fs/nfsd/threads doesn't do it anymore, I assume going to common
> > grace period for NLM and NFSv4 changed things.
> > The question is how to do IP fail-over, so when a node fails and the
IP is
> > moving to another node, we need to go into grace period on all the
nodes
> > in the cluster so the locks of the failed node are not given to anyone
> > other than the client that is reclaiming his locks. Restarting NFS
server
> > is to distractive.
>
> What's the difference? Just that clients don't have to reestablish tcp
> connections?
I am not sure what else systemctl will do but I need to control the order
of the restart so the client will not see any errors.
I don't think that echo 0 > /proc/fs/nfsd/threads is freeing the lock, at
least not the v3 locks, I will try again with v4.
The question is what is the most basic operation that can be done to start
grace, will echo 8 > /proc/fs/nfsd/threads following echo 0 do it?
or is there any other primitive that will do it?
Marc.
>
> --b.
>
> > For NFSv3 KILL signal to lockd still works but for
> > NFSv4 have no way to do it for v4.
> > Marc.
> >
> >
> >
> > From: Bruce Fields <[email protected]>
> > To: Marc Eshel/Almaden/IBM@IBMUS
> > Cc: [email protected]
> > Date: 07/01/2016 09:09 AM
> > Subject: Re: grace period
> >
> >
> >
> > On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > > I see that setting the number of nfsd threads to 0 (echo 0 >
> > > /proc/fs/nfsd/threads) is not releasing the locks and putting the
server
> >
> > > in grace mode.
> >
> > Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> > certainly drop locks. If that's not happening, there's a bug, but
we'd
> > need to know more details (version numbers, etc.) to help.
> >
> > That alone has never been enough to start a grace period--you'd have
to
> > start knfsd again to do that.
> >
> > > What is the best way to go into grace period, in new version of the
> > > kernel, without restarting the nfs server?
> >
> > Restarting the nfs server is the only way. That's true on older
kernels
> > true, as far as I know. (OK, you can apparently make lockd do
something
> > like this with a signal, I don't know if that's used much, and I doubt
> > it works outside an NFSv3-only environment.)
> >
> > So if you want locks dropped and a new grace period, then you should
run
> > "systemctl restart nfs-server", or your distro's equivalent.
> >
> > But you're probably doing something more complicated than that. I'm
not
> > sure I understand the question....
> >
> > --b.
> >
> >
> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
This is my v3 test that show the lock still there after echo 0 >
/proc/fs/nfsd/threads
[root@sonascl21 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@sonascl21 ~]# uname -a
Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP Thu
Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@sonascl21 ~]# cat /proc/locks | grep 999
3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
[root@sonascl21 ~]# echo 0 > /proc/fs/nfsd/threads
[root@sonascl21 ~]# cat /proc/fs/nfsd/threads
0
[root@sonascl21 ~]# cat /proc/locks | grep 999
3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
From: Bruce Fields <[email protected]>
To: Marc Eshel/Almaden/IBM@IBMUS
Cc: [email protected]
Date: 07/01/2016 01:07 PM
Subject: Re: grace period
On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> It used to be that sending KILL signal to lockd would free locks and
start
> Grace period, and when setting nfsd threads to zero, nfsd_last_thread()
> calls nfsd_shutdown that called lockd_down that I believe was causing
both
> freeing of locks and starting grace period or maybe it was setting it
back
> to a value > 0 that started the grace period.
OK, apologies, I didn't know (or forgot) that.
> Any way starting with the kernels that are in RHEL7.1 and up echo 0 >
> /proc/fs/nfsd/threads doesn't do it anymore, I assume going to common
> grace period for NLM and NFSv4 changed things.
> The question is how to do IP fail-over, so when a node fails and the IP
is
> moving to another node, we need to go into grace period on all the nodes
> in the cluster so the locks of the failed node are not given to anyone
> other than the client that is reclaiming his locks. Restarting NFS
server
> is to distractive.
What's the difference? Just that clients don't have to reestablish tcp
connections?
--b.
> For NFSv3 KILL signal to lockd still works but for
> NFSv4 have no way to do it for v4.
> Marc.
>
>
>
> From: Bruce Fields <[email protected]>
> To: Marc Eshel/Almaden/IBM@IBMUS
> Cc: [email protected]
> Date: 07/01/2016 09:09 AM
> Subject: Re: grace period
>
>
>
> On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > I see that setting the number of nfsd threads to 0 (echo 0 >
> > /proc/fs/nfsd/threads) is not releasing the locks and putting the
server
>
> > in grace mode.
>
> Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> certainly drop locks. If that's not happening, there's a bug, but we'd
> need to know more details (version numbers, etc.) to help.
>
> That alone has never been enough to start a grace period--you'd have to
> start knfsd again to do that.
>
> > What is the best way to go into grace period, in new version of the
> > kernel, without restarting the nfs server?
>
> Restarting the nfs server is the only way. That's true on older kernels
> true, as far as I know. (OK, you can apparently make lockd do something
> like this with a signal, I don't know if that's used much, and I doubt
> it works outside an NFSv3-only environment.)
>
> So if you want locks dropped and a new grace period, then you should run
> "systemctl restart nfs-server", or your distro's equivalent.
>
> But you're probably doing something more complicated than that. I'm not
> sure I understand the question....
>
> --b.
>
>
>
>
On Fri, Jul 01, 2016 at 01:24:48PM -0700, Marc Eshel wrote:
> [email protected] wrote on 07/01/2016 01:07:42 PM:
>
> > From: Bruce Fields <[email protected]>
> > To: Marc Eshel/Almaden/IBM@IBMUS
> > Cc: [email protected]
> > Date: 07/01/2016 01:07 PM
> > Subject: Re: grace period
> > Sent by: [email protected]
> >
> > On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> > > It used to be that sending KILL signal to lockd would free locks and
> start
> > > Grace period, and when setting nfsd threads to zero,
> nfsd_last_thread()
> > > calls nfsd_shutdown that called lockd_down that I believe was causing
> both
> > > freeing of locks and starting grace period or maybe it was setting it
> back
> > > to a value > 0 that started the grace period.
> >
> > OK, apologies, I didn't know (or forgot) that.
> >
> > > Any way starting with the kernels that are in RHEL7.1 and up echo 0 >
> > > /proc/fs/nfsd/threads doesn't do it anymore, I assume going to common
> > > grace period for NLM and NFSv4 changed things.
> > > The question is how to do IP fail-over, so when a node fails and the
> IP is
> > > moving to another node, we need to go into grace period on all the
> nodes
> > > in the cluster so the locks of the failed node are not given to anyone
>
> > > other than the client that is reclaiming his locks. Restarting NFS
> server
> > > is to distractive.
> >
> > What's the difference? Just that clients don't have to reestablish tcp
> > connections?
>
> I am not sure what else systemctl will do but I need to control the order
> of the restart so the client will not see any errors.
> I don't think that echo 0 > /proc/fs/nfsd/threads is freeing the lock, at
> least not the v3 locks, I will try again with v4.
> The question is what is the most basic operation that can be done to start
> grace, will echo 8 > /proc/fs/nfsd/threads following echo 0 do it?
> or is there any other primitive that will do it?
That should do it, though really so should just "systemctl restart
nfs-server"--if that causes errors then there's a bug somewhere.
--b.
> Marc.
>
> >
> > --b.
> >
> > > For NFSv3 KILL signal to lockd still works but for
> > > NFSv4 have no way to do it for v4.
> > > Marc.
> > >
> > >
> > >
> > > From: Bruce Fields <[email protected]>
> > > To: Marc Eshel/Almaden/IBM@IBMUS
> > > Cc: [email protected]
> > > Date: 07/01/2016 09:09 AM
> > > Subject: Re: grace period
> > >
> > >
> > >
> > > On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > > > I see that setting the number of nfsd threads to 0 (echo 0 >
> > > > /proc/fs/nfsd/threads) is not releasing the locks and putting the
> server
> > >
> > > > in grace mode.
> > >
> > > Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> > > certainly drop locks. If that's not happening, there's a bug, but
> we'd
> > > need to know more details (version numbers, etc.) to help.
> > >
> > > That alone has never been enough to start a grace period--you'd have
> to
> > > start knfsd again to do that.
> > >
> > > > What is the best way to go into grace period, in new version of the
> > > > kernel, without restarting the nfs server?
> > >
> > > Restarting the nfs server is the only way. That's true on older
> kernels
> > > true, as far as I know. (OK, you can apparently make lockd do
> something
> > > like this with a signal, I don't know if that's used much, and I doubt
> > > it works outside an NFSv3-only environment.)
> > >
> > > So if you want locks dropped and a new grace period, then you should
> run
> > > "systemctl restart nfs-server", or your distro's equivalent.
> > >
> > > But you're probably doing something more complicated than that. I'm
> not
> > > sure I understand the question....
> > >
> > > --b.
> > >
> > >
> > >
> > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
On Fri, Jul 01, 2016 at 01:46:42PM -0700, Marc Eshel wrote:
> This is my v3 test that show the lock still there after echo 0 >
> /proc/fs/nfsd/threads
>
> [root@sonascl21 ~]# cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
>
> [root@sonascl21 ~]# uname -a
> Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP Thu
> Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> [root@sonascl21 ~]# cat /proc/locks | grep 999
> 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
>
> [root@sonascl21 ~]# echo 0 > /proc/fs/nfsd/threads
> [root@sonascl21 ~]# cat /proc/fs/nfsd/threads
> 0
>
> [root@sonascl21 ~]# cat /proc/locks | grep 999
> 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
Huh, that's not what I see. Are you positive that's the lock on the
backend filesystem and not the client-side lock (in case you're doing a
loopback mount?)
--b.
>
>
>
>
> From: Bruce Fields <[email protected]>
> To: Marc Eshel/Almaden/IBM@IBMUS
> Cc: [email protected]
> Date: 07/01/2016 01:07 PM
> Subject: Re: grace period
>
>
>
> On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> > It used to be that sending KILL signal to lockd would free locks and
> start
> > Grace period, and when setting nfsd threads to zero, nfsd_last_thread()
> > calls nfsd_shutdown that called lockd_down that I believe was causing
> both
> > freeing of locks and starting grace period or maybe it was setting it
> back
> > to a value > 0 that started the grace period.
>
> OK, apologies, I didn't know (or forgot) that.
>
> > Any way starting with the kernels that are in RHEL7.1 and up echo 0 >
> > /proc/fs/nfsd/threads doesn't do it anymore, I assume going to common
> > grace period for NLM and NFSv4 changed things.
> > The question is how to do IP fail-over, so when a node fails and the IP
> is
> > moving to another node, we need to go into grace period on all the nodes
>
> > in the cluster so the locks of the failed node are not given to anyone
> > other than the client that is reclaiming his locks. Restarting NFS
> server
> > is to distractive.
>
> What's the difference? Just that clients don't have to reestablish tcp
> connections?
>
> --b.
>
> > For NFSv3 KILL signal to lockd still works but for
> > NFSv4 have no way to do it for v4.
> > Marc.
> >
> >
> >
> > From: Bruce Fields <[email protected]>
> > To: Marc Eshel/Almaden/IBM@IBMUS
> > Cc: [email protected]
> > Date: 07/01/2016 09:09 AM
> > Subject: Re: grace period
> >
> >
> >
> > On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > > I see that setting the number of nfsd threads to 0 (echo 0 >
> > > /proc/fs/nfsd/threads) is not releasing the locks and putting the
> server
> >
> > > in grace mode.
> >
> > Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> > certainly drop locks. If that's not happening, there's a bug, but we'd
> > need to know more details (version numbers, etc.) to help.
> >
> > That alone has never been enough to start a grace period--you'd have to
> > start knfsd again to do that.
> >
> > > What is the best way to go into grace period, in new version of the
> > > kernel, without restarting the nfs server?
> >
> > Restarting the nfs server is the only way. That's true on older kernels
> > true, as far as I know. (OK, you can apparently make lockd do something
> > like this with a signal, I don't know if that's used much, and I doubt
> > it works outside an NFSv3-only environment.)
> >
> > So if you want locks dropped and a new grace period, then you should run
> > "systemctl restart nfs-server", or your distro's equivalent.
> >
> > But you're probably doing something more complicated than that. I'm not
> > sure I understand the question....
> >
> > --b.
> >
> >
> >
> >
>
>
>
>
Yes, the locks are requested from another node, what fs are you using, I
don't think it should make any difference, but I can try it with the same
fs.
Make sure you are using v3, it does work for v4.
Marc.
From: Bruce Fields <[email protected]>
To: Marc Eshel/Almaden/IBM@IBMUS
Cc: [email protected], Tomer Perry <[email protected]>
Date: 07/01/2016 02:01 PM
Subject: Re: grace period
On Fri, Jul 01, 2016 at 01:46:42PM -0700, Marc Eshel wrote:
> This is my v3 test that show the lock still there after echo 0 >
> /proc/fs/nfsd/threads
>
> [root@sonascl21 ~]# cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
>
> [root@sonascl21 ~]# uname -a
> Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP Thu
> Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> [root@sonascl21 ~]# cat /proc/locks | grep 999
> 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
>
> [root@sonascl21 ~]# echo 0 > /proc/fs/nfsd/threads
> [root@sonascl21 ~]# cat /proc/fs/nfsd/threads
> 0
>
> [root@sonascl21 ~]# cat /proc/locks | grep 999
> 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
Huh, that's not what I see. Are you positive that's the lock on the
backend filesystem and not the client-side lock (in case you're doing a
loopback mount?)
--b.
>
>
>
>
> From: Bruce Fields <[email protected]>
> To: Marc Eshel/Almaden/IBM@IBMUS
> Cc: [email protected]
> Date: 07/01/2016 01:07 PM
> Subject: Re: grace period
>
>
>
> On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> > It used to be that sending KILL signal to lockd would free locks and
> start
> > Grace period, and when setting nfsd threads to zero,
nfsd_last_thread()
> > calls nfsd_shutdown that called lockd_down that I believe was causing
> both
> > freeing of locks and starting grace period or maybe it was setting it
> back
> > to a value > 0 that started the grace period.
>
> OK, apologies, I didn't know (or forgot) that.
>
> > Any way starting with the kernels that are in RHEL7.1 and up echo 0 >
> > /proc/fs/nfsd/threads doesn't do it anymore, I assume going to common
> > grace period for NLM and NFSv4 changed things.
> > The question is how to do IP fail-over, so when a node fails and the
IP
> is
> > moving to another node, we need to go into grace period on all the
nodes
>
> > in the cluster so the locks of the failed node are not given to anyone
> > other than the client that is reclaiming his locks. Restarting NFS
> server
> > is to distractive.
>
> What's the difference? Just that clients don't have to reestablish tcp
> connections?
>
> --b.
>
> > For NFSv3 KILL signal to lockd still works but for
> > NFSv4 have no way to do it for v4.
> > Marc.
> >
> >
> >
> > From: Bruce Fields <[email protected]>
> > To: Marc Eshel/Almaden/IBM@IBMUS
> > Cc: [email protected]
> > Date: 07/01/2016 09:09 AM
> > Subject: Re: grace period
> >
> >
> >
> > On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > > I see that setting the number of nfsd threads to 0 (echo 0 >
> > > /proc/fs/nfsd/threads) is not releasing the locks and putting the
> server
> >
> > > in grace mode.
> >
> > Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> > certainly drop locks. If that's not happening, there's a bug, but
we'd
> > need to know more details (version numbers, etc.) to help.
> >
> > That alone has never been enough to start a grace period--you'd have
to
> > start knfsd again to do that.
> >
> > > What is the best way to go into grace period, in new version of the
> > > kernel, without restarting the nfs server?
> >
> > Restarting the nfs server is the only way. That's true on older
kernels
> > true, as far as I know. (OK, you can apparently make lockd do
something
> > like this with a signal, I don't know if that's used much, and I doubt
> > it works outside an NFSv3-only environment.)
> >
> > So if you want locks dropped and a new grace period, then you should
run
> > "systemctl restart nfs-server", or your distro's equivalent.
> >
> > But you're probably doing something more complicated than that. I'm
not
> > sure I understand the question....
> >
> > --b.
> >
> >
> >
> >
>
>
>
>
On Fri, Jul 01, 2016 at 03:42:43PM -0700, Marc Eshel wrote:
> Yes, the locks are requested from another node, what fs are you using, I
> don't think it should make any difference, but I can try it with the same
> fs.
> Make sure you are using v3, it does work for v4.
I tested v3 on upstream.--b.
> Marc.
>
>
>
> From: Bruce Fields <[email protected]>
> To: Marc Eshel/Almaden/IBM@IBMUS
> Cc: [email protected], Tomer Perry <[email protected]>
> Date: 07/01/2016 02:01 PM
> Subject: Re: grace period
>
>
>
> On Fri, Jul 01, 2016 at 01:46:42PM -0700, Marc Eshel wrote:
> > This is my v3 test that show the lock still there after echo 0 >
> > /proc/fs/nfsd/threads
> >
> > [root@sonascl21 ~]# cat /etc/redhat-release
> > Red Hat Enterprise Linux Server release 7.2 (Maipo)
> >
> > [root@sonascl21 ~]# uname -a
> > Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP Thu
>
> > Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> >
> > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
> >
> > [root@sonascl21 ~]# echo 0 > /proc/fs/nfsd/threads
> > [root@sonascl21 ~]# cat /proc/fs/nfsd/threads
> > 0
> >
> > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
>
> Huh, that's not what I see. Are you positive that's the lock on the
> backend filesystem and not the client-side lock (in case you're doing a
> loopback mount?)
>
> --b.
>
> >
> >
> >
> >
> > From: Bruce Fields <[email protected]>
> > To: Marc Eshel/Almaden/IBM@IBMUS
> > Cc: [email protected]
> > Date: 07/01/2016 01:07 PM
> > Subject: Re: grace period
> >
> >
> >
> > On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> > > It used to be that sending KILL signal to lockd would free locks and
> > start
> > > Grace period, and when setting nfsd threads to zero,
> nfsd_last_thread()
> > > calls nfsd_shutdown that called lockd_down that I believe was causing
> > both
> > > freeing of locks and starting grace period or maybe it was setting it
> > back
> > > to a value > 0 that started the grace period.
> >
> > OK, apologies, I didn't know (or forgot) that.
> >
> > > Any way starting with the kernels that are in RHEL7.1 and up echo 0 >
> > > /proc/fs/nfsd/threads doesn't do it anymore, I assume going to common
> > > grace period for NLM and NFSv4 changed things.
> > > The question is how to do IP fail-over, so when a node fails and the
> IP
> > is
> > > moving to another node, we need to go into grace period on all the
> nodes
> >
> > > in the cluster so the locks of the failed node are not given to anyone
>
> > > other than the client that is reclaiming his locks. Restarting NFS
> > server
> > > is to distractive.
> >
> > What's the difference? Just that clients don't have to reestablish tcp
> > connections?
> >
> > --b.
> >
> > > For NFSv3 KILL signal to lockd still works but for
> > > NFSv4 have no way to do it for v4.
> > > Marc.
> > >
> > >
> > >
> > > From: Bruce Fields <[email protected]>
> > > To: Marc Eshel/Almaden/IBM@IBMUS
> > > Cc: [email protected]
> > > Date: 07/01/2016 09:09 AM
> > > Subject: Re: grace period
> > >
> > >
> > >
> > > On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > > > I see that setting the number of nfsd threads to 0 (echo 0 >
> > > > /proc/fs/nfsd/threads) is not releasing the locks and putting the
> > server
> > >
> > > > in grace mode.
> > >
> > > Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> > > certainly drop locks. If that's not happening, there's a bug, but
> we'd
> > > need to know more details (version numbers, etc.) to help.
> > >
> > > That alone has never been enough to start a grace period--you'd have
> to
> > > start knfsd again to do that.
> > >
> > > > What is the best way to go into grace period, in new version of the
> > > > kernel, without restarting the nfs server?
> > >
> > > Restarting the nfs server is the only way. That's true on older
> kernels
> > > true, as far as I know. (OK, you can apparently make lockd do
> something
> > > like this with a signal, I don't know if that's used much, and I doubt
> > > it works outside an NFSv3-only environment.)
> > >
> > > So if you want locks dropped and a new grace period, then you should
> run
> > > "systemctl restart nfs-server", or your distro's equivalent.
> > >
> > > But you're probably doing something more complicated than that. I'm
> not
> > > sure I understand the question....
> > >
> > > --b.
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
echo 0 > /proc/fs/nfsd/threads does delete the locks for v4 but not for v3
Marc.
From: Marc Eshel/Almaden/IBM
To: Bruce Fields <[email protected]>
Cc: [email protected], Tomer Perry/Israel/IBM@IBMIL
Date: 07/01/2016 01:46 PM
Subject: Re: grace period
This is my v3 test that show the lock still there after echo 0 >
/proc/fs/nfsd/threads
[root@sonascl21 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@sonascl21 ~]# uname -a
Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP Thu
Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@sonascl21 ~]# cat /proc/locks | grep 999
3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
[root@sonascl21 ~]# echo 0 > /proc/fs/nfsd/threads
[root@sonascl21 ~]# cat /proc/fs/nfsd/threads
0
[root@sonascl21 ~]# cat /proc/locks | grep 999
3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
From: Bruce Fields <[email protected]>
To: Marc Eshel/Almaden/IBM@IBMUS
Cc: [email protected]
Date: 07/01/2016 01:07 PM
Subject: Re: grace period
On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> It used to be that sending KILL signal to lockd would free locks and
start
> Grace period, and when setting nfsd threads to zero, nfsd_last_thread()
> calls nfsd_shutdown that called lockd_down that I believe was causing
both
> freeing of locks and starting grace period or maybe it was setting it
back
> to a value > 0 that started the grace period.
OK, apologies, I didn't know (or forgot) that.
> Any way starting with the kernels that are in RHEL7.1 and up echo 0 >
> /proc/fs/nfsd/threads doesn't do it anymore, I assume going to common
> grace period for NLM and NFSv4 changed things.
> The question is how to do IP fail-over, so when a node fails and the IP
is
> moving to another node, we need to go into grace period on all the nodes
> in the cluster so the locks of the failed node are not given to anyone
> other than the client that is reclaiming his locks. Restarting NFS
server
> is to distractive.
What's the difference? Just that clients don't have to reestablish tcp
connections?
--b.
> For NFSv3 KILL signal to lockd still works but for
> NFSv4 have no way to do it for v4.
> Marc.
>
>
>
> From: Bruce Fields <[email protected]>
> To: Marc Eshel/Almaden/IBM@IBMUS
> Cc: [email protected]
> Date: 07/01/2016 09:09 AM
> Subject: Re: grace period
>
>
>
> On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > I see that setting the number of nfsd threads to 0 (echo 0 >
> > /proc/fs/nfsd/threads) is not releasing the locks and putting the
server
>
> > in grace mode.
>
> Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> certainly drop locks. If that's not happening, there's a bug, but we'd
> need to know more details (version numbers, etc.) to help.
>
> That alone has never been enough to start a grace period--you'd have to
> start knfsd again to do that.
>
> > What is the best way to go into grace period, in new version of the
> > kernel, without restarting the nfs server?
>
> Restarting the nfs server is the only way. That's true on older kernels
> true, as far as I know. (OK, you can apparently make lockd do something
> like this with a signal, I don't know if that's used much, and I doubt
> it works outside an NFSv3-only environment.)
>
> So if you want locks dropped and a new grace period, then you should run
> "systemctl restart nfs-server", or your distro's equivalent.
>
> But you're probably doing something more complicated than that. I'm not
> sure I understand the question....
>
> --b.
>
>
>
>
I tried again NFSv3 locks with xfs export. "echo 0 >
/proc/fs/nfsd/threads" releases locks on rhel7.0 but not on rhel7.2
What else can I show you to find the problem?
Marc.
works:
[root@boar11 ~]# uname -a
Linux boar11 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014
x86_64 x86_64 x86_64 GNU/Linux
[root@boar11 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.0 (Maipo)
not working:
[root@sonascl21 ~]# uname -a
Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP Thu
Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@sonascl21 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@sonascl21 ~]# cat /proc/fs/nfsd/threads
0
[root@sonascl21 ~]# cat /proc/locks
1: POSIX ADVISORY WRITE 2346 fd:00:1612092569 0 9999
From: Bruce Fields <[email protected]>
To: Marc Eshel/Almaden/IBM@IBMUS
Cc: [email protected], Tomer Perry <[email protected]>
Date: 07/01/2016 05:58 PM
Subject: Re: grace period
On Fri, Jul 01, 2016 at 03:42:43PM -0700, Marc Eshel wrote:
> Yes, the locks are requested from another node, what fs are you using, I
> don't think it should make any difference, but I can try it with the
same
> fs.
> Make sure you are using v3, it does work for v4.
I tested v3 on upstream.--b.
> Marc.
>
>
>
> From: Bruce Fields <[email protected]>
> To: Marc Eshel/Almaden/IBM@IBMUS
> Cc: [email protected], Tomer Perry <[email protected]>
> Date: 07/01/2016 02:01 PM
> Subject: Re: grace period
>
>
>
> On Fri, Jul 01, 2016 at 01:46:42PM -0700, Marc Eshel wrote:
> > This is my v3 test that show the lock still there after echo 0 >
> > /proc/fs/nfsd/threads
> >
> > [root@sonascl21 ~]# cat /etc/redhat-release
> > Red Hat Enterprise Linux Server release 7.2 (Maipo)
> >
> > [root@sonascl21 ~]# uname -a
> > Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP
Thu
>
> > Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> >
> > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
> >
> > [root@sonascl21 ~]# echo 0 > /proc/fs/nfsd/threads
> > [root@sonascl21 ~]# cat /proc/fs/nfsd/threads
> > 0
> >
> > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
>
> Huh, that's not what I see. Are you positive that's the lock on the
> backend filesystem and not the client-side lock (in case you're doing a
> loopback mount?)
>
> --b.
>
> >
> >
> >
> >
> > From: Bruce Fields <[email protected]>
> > To: Marc Eshel/Almaden/IBM@IBMUS
> > Cc: [email protected]
> > Date: 07/01/2016 01:07 PM
> > Subject: Re: grace period
> >
> >
> >
> > On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> > > It used to be that sending KILL signal to lockd would free locks and
> > start
> > > Grace period, and when setting nfsd threads to zero,
> nfsd_last_thread()
> > > calls nfsd_shutdown that called lockd_down that I believe was
causing
> > both
> > > freeing of locks and starting grace period or maybe it was setting
it
> > back
> > > to a value > 0 that started the grace period.
> >
> > OK, apologies, I didn't know (or forgot) that.
> >
> > > Any way starting with the kernels that are in RHEL7.1 and up echo 0
>
> > > /proc/fs/nfsd/threads doesn't do it anymore, I assume going to
common
> > > grace period for NLM and NFSv4 changed things.
> > > The question is how to do IP fail-over, so when a node fails and the
> IP
> > is
> > > moving to another node, we need to go into grace period on all the
> nodes
> >
> > > in the cluster so the locks of the failed node are not given to
anyone
>
> > > other than the client that is reclaiming his locks. Restarting NFS
> > server
> > > is to distractive.
> >
> > What's the difference? Just that clients don't have to reestablish
tcp
> > connections?
> >
> > --b.
> >
> > > For NFSv3 KILL signal to lockd still works but for
> > > NFSv4 have no way to do it for v4.
> > > Marc.
> > >
> > >
> > >
> > > From: Bruce Fields <[email protected]>
> > > To: Marc Eshel/Almaden/IBM@IBMUS
> > > Cc: [email protected]
> > > Date: 07/01/2016 09:09 AM
> > > Subject: Re: grace period
> > >
> > >
> > >
> > > On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > > > I see that setting the number of nfsd threads to 0 (echo 0 >
> > > > /proc/fs/nfsd/threads) is not releasing the locks and putting the
> > server
> > >
> > > > in grace mode.
> > >
> > > Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> > > certainly drop locks. If that's not happening, there's a bug, but
> we'd
> > > need to know more details (version numbers, etc.) to help.
> > >
> > > That alone has never been enough to start a grace period--you'd have
> to
> > > start knfsd again to do that.
> > >
> > > > What is the best way to go into grace period, in new version of
the
> > > > kernel, without restarting the nfs server?
> > >
> > > Restarting the nfs server is the only way. That's true on older
> kernels
> > > true, as far as I know. (OK, you can apparently make lockd do
> something
> > > like this with a signal, I don't know if that's used much, and I
doubt
> > > it works outside an NFSv3-only environment.)
> > >
> > > So if you want locks dropped and a new grace period, then you should
> run
> > > "systemctl restart nfs-server", or your distro's equivalent.
> > >
> > > But you're probably doing something more complicated than that. I'm
> not
> > > sure I understand the question....
> > >
> > > --b.
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
On Sat, Jul 02, 2016 at 10:30:11PM -0700, Marc Eshel wrote:
> I tried again NFSv3 locks with xfs export. "echo 0 >
> /proc/fs/nfsd/threads" releases locks on rhel7.0 but not on rhel7.2
> What else can I show you to find the problem?
Sorry, I can't reproduce, though I've only tried a slightly later kernel
than that. Could you submit a RHEL bug?
--b.
> Marc.
>
> works:
> [root@boar11 ~]# uname -a
> Linux boar11 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014
> x86_64 x86_64 x86_64 GNU/Linux
> [root@boar11 ~]# cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.0 (Maipo)
>
> not working:
> [root@sonascl21 ~]# uname -a
> Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP Thu
> Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> [root@sonascl21 ~]# cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
> [root@sonascl21 ~]# cat /proc/fs/nfsd/threads
> 0
> [root@sonascl21 ~]# cat /proc/locks
> 1: POSIX ADVISORY WRITE 2346 fd:00:1612092569 0 9999
>
>
>
> From: Bruce Fields <[email protected]>
> To: Marc Eshel/Almaden/IBM@IBMUS
> Cc: [email protected], Tomer Perry <[email protected]>
> Date: 07/01/2016 05:58 PM
> Subject: Re: grace period
>
>
>
> On Fri, Jul 01, 2016 at 03:42:43PM -0700, Marc Eshel wrote:
> > Yes, the locks are requested from another node, what fs are you using, I
>
> > don't think it should make any difference, but I can try it with the
> same
> > fs.
> > Make sure you are using v3, it does work for v4.
>
> I tested v3 on upstream.--b.
>
> > Marc.
> >
> >
> >
> > From: Bruce Fields <[email protected]>
> > To: Marc Eshel/Almaden/IBM@IBMUS
> > Cc: [email protected], Tomer Perry <[email protected]>
> > Date: 07/01/2016 02:01 PM
> > Subject: Re: grace period
> >
> >
> >
> > On Fri, Jul 01, 2016 at 01:46:42PM -0700, Marc Eshel wrote:
> > > This is my v3 test that show the lock still there after echo 0 >
> > > /proc/fs/nfsd/threads
> > >
> > > [root@sonascl21 ~]# cat /etc/redhat-release
> > > Red Hat Enterprise Linux Server release 7.2 (Maipo)
> > >
> > > [root@sonascl21 ~]# uname -a
> > > Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP
> Thu
> >
> > > Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > > 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
> > >
> > > [root@sonascl21 ~]# echo 0 > /proc/fs/nfsd/threads
> > > [root@sonascl21 ~]# cat /proc/fs/nfsd/threads
> > > 0
> > >
> > > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > > 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
> >
> > Huh, that's not what I see. Are you positive that's the lock on the
> > backend filesystem and not the client-side lock (in case you're doing a
> > loopback mount?)
> >
> > --b.
> >
> > >
> > >
> > >
> > >
> > > From: Bruce Fields <[email protected]>
> > > To: Marc Eshel/Almaden/IBM@IBMUS
> > > Cc: [email protected]
> > > Date: 07/01/2016 01:07 PM
> > > Subject: Re: grace period
> > >
> > >
> > >
> > > On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> > > > It used to be that sending KILL signal to lockd would free locks and
>
> > > start
> > > > Grace period, and when setting nfsd threads to zero,
> > nfsd_last_thread()
> > > > calls nfsd_shutdown that called lockd_down that I believe was
> causing
> > > both
> > > > freeing of locks and starting grace period or maybe it was setting
> it
> > > back
> > > > to a value > 0 that started the grace period.
> > >
> > > OK, apologies, I didn't know (or forgot) that.
> > >
> > > > Any way starting with the kernels that are in RHEL7.1 and up echo 0
> >
> > > > /proc/fs/nfsd/threads doesn't do it anymore, I assume going to
> common
> > > > grace period for NLM and NFSv4 changed things.
> > > > The question is how to do IP fail-over, so when a node fails and the
>
> > IP
> > > is
> > > > moving to another node, we need to go into grace period on all the
> > nodes
> > >
> > > > in the cluster so the locks of the failed node are not given to
> anyone
> >
> > > > other than the client that is reclaiming his locks. Restarting NFS
> > > server
> > > > is to distractive.
> > >
> > > What's the difference? Just that clients don't have to reestablish
> tcp
> > > connections?
> > >
> > > --b.
> > >
> > > > For NFSv3 KILL signal to lockd still works but for
> > > > NFSv4 have no way to do it for v4.
> > > > Marc.
> > > >
> > > >
> > > >
> > > > From: Bruce Fields <[email protected]>
> > > > To: Marc Eshel/Almaden/IBM@IBMUS
> > > > Cc: [email protected]
> > > > Date: 07/01/2016 09:09 AM
> > > > Subject: Re: grace period
> > > >
> > > >
> > > >
> > > > On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > > > > I see that setting the number of nfsd threads to 0 (echo 0 >
> > > > > /proc/fs/nfsd/threads) is not releasing the locks and putting the
> > > server
> > > >
> > > > > in grace mode.
> > > >
> > > > Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> > > > certainly drop locks. If that's not happening, there's a bug, but
> > we'd
> > > > need to know more details (version numbers, etc.) to help.
> > > >
> > > > That alone has never been enough to start a grace period--you'd have
>
> > to
> > > > start knfsd again to do that.
> > > >
> > > > > What is the best way to go into grace period, in new version of
> the
> > > > > kernel, without restarting the nfs server?
> > > >
> > > > Restarting the nfs server is the only way. That's true on older
> > kernels
> > > > true, as far as I know. (OK, you can apparently make lockd do
> > something
> > > > like this with a signal, I don't know if that's used much, and I
> doubt
> > > > it works outside an NFSv3-only environment.)
> > > >
> > > > So if you want locks dropped and a new grace period, then you should
>
> > run
> > > > "systemctl restart nfs-server", or your distro's equivalent.
> > > >
> > > > But you're probably doing something more complicated than that. I'm
>
> > not
> > > > sure I understand the question....
> > > >
> > > > --b.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
Can you please point me to the kernel that you are using so I can check if
it is an obvious problem before I open an RHEL bug?
Thanks, Marc.
From: Bruce Fields <[email protected]>
To: Marc Eshel/Almaden/IBM@IBMUS
Cc: [email protected], Tomer Perry <[email protected]>
Date: 07/05/2016 01:52 PM
Subject: Re: grace period
Sent by: [email protected]
On Sat, Jul 02, 2016 at 10:30:11PM -0700, Marc Eshel wrote:
> I tried again NFSv3 locks with xfs export. "echo 0 >
> /proc/fs/nfsd/threads" releases locks on rhel7.0 but not on rhel7.2
> What else can I show you to find the problem?
Sorry, I can't reproduce, though I've only tried a slightly later kernel
than that. Could you submit a RHEL bug?
--b.
> Marc.
>
> works:
> [root@boar11 ~]# uname -a
> Linux boar11 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014
> x86_64 x86_64 x86_64 GNU/Linux
> [root@boar11 ~]# cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.0 (Maipo)
>
> not working:
> [root@sonascl21 ~]# uname -a
> Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP Thu
> Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> [root@sonascl21 ~]# cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
> [root@sonascl21 ~]# cat /proc/fs/nfsd/threads
> 0
> [root@sonascl21 ~]# cat /proc/locks
> 1: POSIX ADVISORY WRITE 2346 fd:00:1612092569 0 9999
>
>
>
> From: Bruce Fields <[email protected]>
> To: Marc Eshel/Almaden/IBM@IBMUS
> Cc: [email protected], Tomer Perry <[email protected]>
> Date: 07/01/2016 05:58 PM
> Subject: Re: grace period
>
>
>
> On Fri, Jul 01, 2016 at 03:42:43PM -0700, Marc Eshel wrote:
> > Yes, the locks are requested from another node, what fs are you using,
I
>
> > don't think it should make any difference, but I can try it with the
> same
> > fs.
> > Make sure you are using v3, it does work for v4.
>
> I tested v3 on upstream.--b.
>
> > Marc.
> >
> >
> >
> > From: Bruce Fields <[email protected]>
> > To: Marc Eshel/Almaden/IBM@IBMUS
> > Cc: [email protected], Tomer Perry <[email protected]>
> > Date: 07/01/2016 02:01 PM
> > Subject: Re: grace period
> >
> >
> >
> > On Fri, Jul 01, 2016 at 01:46:42PM -0700, Marc Eshel wrote:
> > > This is my v3 test that show the lock still there after echo 0 >
> > > /proc/fs/nfsd/threads
> > >
> > > [root@sonascl21 ~]# cat /etc/redhat-release
> > > Red Hat Enterprise Linux Server release 7.2 (Maipo)
> > >
> > > [root@sonascl21 ~]# uname -a
> > > Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP
> Thu
> >
> > > Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > > 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
> > >
> > > [root@sonascl21 ~]# echo 0 > /proc/fs/nfsd/threads
> > > [root@sonascl21 ~]# cat /proc/fs/nfsd/threads
> > > 0
> > >
> > > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > > 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
> >
> > Huh, that's not what I see. Are you positive that's the lock on the
> > backend filesystem and not the client-side lock (in case you're doing
a
> > loopback mount?)
> >
> > --b.
> >
> > >
> > >
> > >
> > >
> > > From: Bruce Fields <[email protected]>
> > > To: Marc Eshel/Almaden/IBM@IBMUS
> > > Cc: [email protected]
> > > Date: 07/01/2016 01:07 PM
> > > Subject: Re: grace period
> > >
> > >
> > >
> > > On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> > > > It used to be that sending KILL signal to lockd would free locks
and
>
> > > start
> > > > Grace period, and when setting nfsd threads to zero,
> > nfsd_last_thread()
> > > > calls nfsd_shutdown that called lockd_down that I believe was
> causing
> > > both
> > > > freeing of locks and starting grace period or maybe it was setting
> it
> > > back
> > > > to a value > 0 that started the grace period.
> > >
> > > OK, apologies, I didn't know (or forgot) that.
> > >
> > > > Any way starting with the kernels that are in RHEL7.1 and up echo
0
> >
> > > > /proc/fs/nfsd/threads doesn't do it anymore, I assume going to
> common
> > > > grace period for NLM and NFSv4 changed things.
> > > > The question is how to do IP fail-over, so when a node fails and
the
>
> > IP
> > > is
> > > > moving to another node, we need to go into grace period on all the
> > nodes
> > >
> > > > in the cluster so the locks of the failed node are not given to
> anyone
> >
> > > > other than the client that is reclaiming his locks. Restarting NFS
> > > server
> > > > is to distractive.
> > >
> > > What's the difference? Just that clients don't have to reestablish
> tcp
> > > connections?
> > >
> > > --b.
> > >
> > > > For NFSv3 KILL signal to lockd still works but for
> > > > NFSv4 have no way to do it for v4.
> > > > Marc.
> > > >
> > > >
> > > >
> > > > From: Bruce Fields <[email protected]>
> > > > To: Marc Eshel/Almaden/IBM@IBMUS
> > > > Cc: [email protected]
> > > > Date: 07/01/2016 09:09 AM
> > > > Subject: Re: grace period
> > > >
> > > >
> > > >
> > > > On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > > > > I see that setting the number of nfsd threads to 0 (echo 0 >
> > > > > /proc/fs/nfsd/threads) is not releasing the locks and putting
the
> > > server
> > > >
> > > > > in grace mode.
> > > >
> > > > Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> > > > certainly drop locks. If that's not happening, there's a bug, but
> > we'd
> > > > need to know more details (version numbers, etc.) to help.
> > > >
> > > > That alone has never been enough to start a grace period--you'd
have
>
> > to
> > > > start knfsd again to do that.
> > > >
> > > > > What is the best way to go into grace period, in new version of
> the
> > > > > kernel, without restarting the nfs server?
> > > >
> > > > Restarting the nfs server is the only way. That's true on older
> > kernels
> > > > true, as far as I know. (OK, you can apparently make lockd do
> > something
> > > > like this with a signal, I don't know if that's used much, and I
> doubt
> > > > it works outside an NFSv3-only environment.)
> > > >
> > > > So if you want locks dropped and a new grace period, then you
should
>
> > run
> > > > "systemctl restart nfs-server", or your distro's equivalent.
> > > >
> > > > But you're probably doing something more complicated than that.
I'm
>
> > not
> > > > sure I understand the question....
> > > >
> > > > --b.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
On Tue, Jul 05, 2016 at 04:05:56PM -0700, Marc Eshel wrote:
> Can you please point me to the kernel that you are using so I can check if
> it is an obvious problem before I open an RHEL bug?
I've tried it on the latest upstream and on rhel 3.10.0-327.13.1.el7.
--b.
> Thanks, Marc.
>
>
>
> From: Bruce Fields <[email protected]>
> To: Marc Eshel/Almaden/IBM@IBMUS
> Cc: [email protected], Tomer Perry <[email protected]>
> Date: 07/05/2016 01:52 PM
> Subject: Re: grace period
> Sent by: [email protected]
>
>
>
> On Sat, Jul 02, 2016 at 10:30:11PM -0700, Marc Eshel wrote:
> > I tried again NFSv3 locks with xfs export. "echo 0 >
> > /proc/fs/nfsd/threads" releases locks on rhel7.0 but not on rhel7.2
> > What else can I show you to find the problem?
>
> Sorry, I can't reproduce, though I've only tried a slightly later kernel
> than that. Could you submit a RHEL bug?
>
> --b.
>
> > Marc.
> >
> > works:
> > [root@boar11 ~]# uname -a
> > Linux boar11 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014
> > x86_64 x86_64 x86_64 GNU/Linux
> > [root@boar11 ~]# cat /etc/redhat-release
> > Red Hat Enterprise Linux Server release 7.0 (Maipo)
> >
> > not working:
> > [root@sonascl21 ~]# uname -a
> > Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP Thu
>
> > Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> > [root@sonascl21 ~]# cat /etc/redhat-release
> > Red Hat Enterprise Linux Server release 7.2 (Maipo)
> > [root@sonascl21 ~]# cat /proc/fs/nfsd/threads
> > 0
> > [root@sonascl21 ~]# cat /proc/locks
> > 1: POSIX ADVISORY WRITE 2346 fd:00:1612092569 0 9999
> >
> >
> >
> > From: Bruce Fields <[email protected]>
> > To: Marc Eshel/Almaden/IBM@IBMUS
> > Cc: [email protected], Tomer Perry <[email protected]>
> > Date: 07/01/2016 05:58 PM
> > Subject: Re: grace period
> >
> >
> >
> > On Fri, Jul 01, 2016 at 03:42:43PM -0700, Marc Eshel wrote:
> > > Yes, the locks are requested from another node, what fs are you using,
> I
> >
> > > don't think it should make any difference, but I can try it with the
> > same
> > > fs.
> > > Make sure you are using v3, it does work for v4.
> >
> > I tested v3 on upstream.--b.
> >
> > > Marc.
> > >
> > >
> > >
> > > From: Bruce Fields <[email protected]>
> > > To: Marc Eshel/Almaden/IBM@IBMUS
> > > Cc: [email protected], Tomer Perry <[email protected]>
> > > Date: 07/01/2016 02:01 PM
> > > Subject: Re: grace period
> > >
> > >
> > >
> > > On Fri, Jul 01, 2016 at 01:46:42PM -0700, Marc Eshel wrote:
> > > > This is my v3 test that show the lock still there after echo 0 >
> > > > /proc/fs/nfsd/threads
> > > >
> > > > [root@sonascl21 ~]# cat /etc/redhat-release
> > > > Red Hat Enterprise Linux Server release 7.2 (Maipo)
> > > >
> > > > [root@sonascl21 ~]# uname -a
> > > > Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP
>
> > Thu
> > >
> > > > Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> > > >
> > > > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > > > 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
> > > >
> > > > [root@sonascl21 ~]# echo 0 > /proc/fs/nfsd/threads
> > > > [root@sonascl21 ~]# cat /proc/fs/nfsd/threads
> > > > 0
> > > >
> > > > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > > > 3: POSIX ADVISORY WRITE 2349 00:2a:489486 0 999
> > >
> > > Huh, that's not what I see. Are you positive that's the lock on the
> > > backend filesystem and not the client-side lock (in case you're doing
> a
> > > loopback mount?)
> > >
> > > --b.
> > >
> > > >
> > > >
> > > >
> > > >
> > > > From: Bruce Fields <[email protected]>
> > > > To: Marc Eshel/Almaden/IBM@IBMUS
> > > > Cc: [email protected]
> > > > Date: 07/01/2016 01:07 PM
> > > > Subject: Re: grace period
> > > >
> > > >
> > > >
> > > > On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> > > > > It used to be that sending KILL signal to lockd would free locks
> and
> >
> > > > start
> > > > > Grace period, and when setting nfsd threads to zero,
> > > nfsd_last_thread()
> > > > > calls nfsd_shutdown that called lockd_down that I believe was
> > causing
> > > > both
> > > > > freeing of locks and starting grace period or maybe it was setting
>
> > it
> > > > back
> > > > > to a value > 0 that started the grace period.
> > > >
> > > > OK, apologies, I didn't know (or forgot) that.
> > > >
> > > > > Any way starting with the kernels that are in RHEL7.1 and up echo
> 0
> > >
> > > > > /proc/fs/nfsd/threads doesn't do it anymore, I assume going to
> > common
> > > > > grace period for NLM and NFSv4 changed things.
> > > > > The question is how to do IP fail-over, so when a node fails and
> the
> >
> > > IP
> > > > is
> > > > > moving to another node, we need to go into grace period on all the
>
> > > nodes
> > > >
> > > > > in the cluster so the locks of the failed node are not given to
> > anyone
> > >
> > > > > other than the client that is reclaiming his locks. Restarting NFS
>
> > > > server
> > > > > is to distractive.
> > > >
> > > > What's the difference? Just that clients don't have to reestablish
> > tcp
> > > > connections?
> > > >
> > > > --b.
> > > >
> > > > > For NFSv3 KILL signal to lockd still works but for
> > > > > NFSv4 have no way to do it for v4.
> > > > > Marc.
> > > > >
> > > > >
> > > > >
> > > > > From: Bruce Fields <[email protected]>
> > > > > To: Marc Eshel/Almaden/IBM@IBMUS
> > > > > Cc: [email protected]
> > > > > Date: 07/01/2016 09:09 AM
> > > > > Subject: Re: grace period
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > > > > > I see that setting the number of nfsd threads to 0 (echo 0 >
> > > > > > /proc/fs/nfsd/threads) is not releasing the locks and putting
> the
> > > > server
> > > > >
> > > > > > in grace mode.
> > > > >
> > > > > Writing 0 to /proc/fs/nfsd/threads shuts down knfsd. So it should
> > > > > certainly drop locks. If that's not happening, there's a bug, but
>
> > > we'd
> > > > > need to know more details (version numbers, etc.) to help.
> > > > >
> > > > > That alone has never been enough to start a grace period--you'd
> have
> >
> > > to
> > > > > start knfsd again to do that.
> > > > >
> > > > > > What is the best way to go into grace period, in new version of
> > the
> > > > > > kernel, without restarting the nfs server?
> > > > >
> > > > > Restarting the nfs server is the only way. That's true on older
> > > kernels
> > > > > true, as far as I know. (OK, you can apparently make lockd do
> > > something
> > > > > like this with a signal, I don't know if that's used much, and I
> > doubt
> > > > > it works outside an NFSv3-only environment.)
> > > > >
> > > > > So if you want locks dropped and a new grace period, then you
> should
> >
> > > run
> > > > > "systemctl restart nfs-server", or your distro's equivalent.
> > > > >
> > > > > But you're probably doing something more complicated than that.
> I'm
> >
> > > not
> > > > > sure I understand the question....
> > > > >
> > > > > --b.
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
>