Hello there,
I run a diskless network where every user NFS mounts pretty much
everything including /home and /var/mail. It's often the case that a
misbehaved client will leave a locked file stuck on the server -- today
it is a file in a user's mail/ directory.
Is there a way to query what files are being held locked by clients? I'm
sure the kernel knows, as it is able to enforce the lock, but it isn't
obvious how to extract that information -- lsof is documented to and
indeed does not return any information pertaining NFS client locks, and
I'm not clear whether /proc/locks (on the server side obviously) does or
not.
A related question is whether it is possible to break a client lock
without rebooting the server (or restarting the NFS services).
Does anyone have any insight to share? Thanks,
--
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko
| [+55 16] 991 126 430 | http://launchpad.net/~kiko
On Mon, 21 Mar 2016 11:39:14 -0300
Christian Robottom Reis <[email protected]> wrote:
> Hello there,
>
> I run a diskless network where every user NFS mounts pretty much
> everything including /home and /var/mail. It's often the case that a
> misbehaved client will leave a locked file stuck on the server -- today
> it is a file in a user's mail/ directory.
>
> Is there a way to query what files are being held locked by clients? I'm
> sure the kernel knows, as it is able to enforce the lock, but it isn't
> obvious how to extract that information -- lsof is documented to and
> indeed does not return any information pertaining NFS client locks, and
> I'm not clear whether /proc/locks (on the server side obviously) does or
> not.
>
/proc/locks will generally show you all of the locks being held
(assuming the filesystem's ->lock routine records the locks). It's not
really possible to match those up with a particular client though.
> A related question is whether it is possible to break a client lock
> without rebooting the server (or restarting the NFS services).
>
> Does anyone have any insight to share? Thanks,
I assume you're using NFSv3? What happens when the client "misbehaves"?
Are the clients dropping offline or do you have applications that are
just sitting on the lock and not releasing it?
If the clients are just going away without unlocking first, then you
could consider using NFSv4, which has a lease-based locking. If the
client goes away for a while (90s or so), then it'll lose its lock.
Alternately, there is the /proc/fs/nfsd/unlock_ip interface. Supposedly
you can echo an address into there and it'll forcibly drop all of the
locks that that that client holds. I've not used that so YMMV there.
--
Jeff Layton <[email protected]>
On Mon, Mar 21, 2016 at 01:19:06PM -0400, Jeff Layton wrote:
> On Mon, 21 Mar 2016 11:39:14 -0300
> Christian Robottom Reis <[email protected]> wrote:
>
> > Hello there,
> >
> > I run a diskless network where every user NFS mounts pretty much
> > everything including /home and /var/mail. It's often the case that a
> > misbehaved client will leave a locked file stuck on the server -- today
> > it is a file in a user's mail/ directory.
> >
> > Is there a way to query what files are being held locked by clients? I'm
> > sure the kernel knows, as it is able to enforce the lock, but it isn't
> > obvious how to extract that information -- lsof is documented to and
> > indeed does not return any information pertaining NFS client locks, and
> > I'm not clear whether /proc/locks (on the server side obviously) does or
> > not.
>
> /proc/locks will generally show you all of the locks being held
> (assuming the filesystem's ->lock routine records the locks).
Great -- I couldn't find that documented elsewhere; perhaps that's
obvious if that's how locking in the filesystem layer always works (i.e.
for the locking operation to fail on an inode it must be recorded in
/proc/locks).
I wonder, is there a tool which parses /proc/locks and grabs the
filenames from the inode and device information? That would already be
quite interesting to help preemptively debug locking problems.
> It's not really possible to match those up with a particular client
> though.
Because the NFS server doesn't track which client requested a lock, and
instead just passes the request through to the filesystem, I assume?
Perhaps there is nowhere scalable to record that today?
> > A related question is whether it is possible to break a client lock
> > without rebooting the server (or restarting the NFS services).
> >
> > Does anyone have any insight to share? Thanks,
>
> I assume you're using NFSv3? What happens when the client "misbehaves"?
> Are the clients dropping offline or do you have applications that are
> just sitting on the lock and not releasing it?
In the situation which happened today my guess (because it's a mbox
file) is that a client ran something like mutt and the machine died
somewhere during shutdown. It's my guess because AIUI the lock doesn't
get stuck if the process is simply KILLed or crashes.
> If the clients are just going away without unlocking first, then you
> could consider using NFSv4, which has a lease-based locking. If the
> client goes away for a while (90s or so), then it'll lose its lock.
That's quite interesting -- and by client do you actually mean "client
application" or client in the sense of the machine itself?
> Alternately, there is the /proc/fs/nfsd/unlock_ip interface. Supposedly
> you can echo an address into there and it'll forcibly drop all of the
> locks that that that client holds. I've not used that so YMMV there.
Oh! That's a very interesting, and I now see it documented here:
http://people.redhat.com/rpeterso/Patches/NFS/NLM/004.txt
I've never seen it mentioned elsewhere. It doesn't seem to work:
kiko@chorus:~$ grep 9178074 /proc/locks
2: POSIX ADVISORY READ 2491 09:04:9178074 0 EOF
kiko@chorus:~$ echo "192.168.99.14" | sudo tee /proc/fs/nfsd/unlock_ip
192.168.99.14
kiko@chorus:~$ grep 9178074 /proc/locks
2: POSIX ADVISORY READ 2491 09:04:9178074 0 EOF
(Where .14 is holding the lock -- I have tested and releasing it does
make the lock entry disappear)
Also, if unlock_ip does work, that seems to not jive with the assertion
above that the kernel doesn't track what locks are held by a given
client. Surely if unlock_ip works for a given IP, somewhere that is
tracked?
Thanks,
--
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko
CEO, Async Open Source | [+55 16] 9 9112 6430 | http://launchpad.net/~kiko
On Mon, Mar 21, 2016 at 02:55:00PM -0300, Christian Robottom Reis wrote:
> > Alternately, there is the /proc/fs/nfsd/unlock_ip interface. Supposedly
> > you can echo an address into there and it'll forcibly drop all of the
> > locks that that that client holds. I've not used that so YMMV there.
>
> Oh! That's a very interesting, and I now see it documented here:
>
> http://people.redhat.com/rpeterso/Patches/NFS/NLM/004.txt
On second look, I don't think that interface is meant to take a client
IP, but rather a server IP:
"They are intended to allow admin or user mode script to release NLM
locks based on either a path name or a server in-bound ip address[...]"
That's why echoing the client IP makes no difference.
I'm surprised -- so far I've found no facility for lock management
server-side other than restarting the server.
--
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko
| [+55 16] 991 126 430 | http://launchpad.net/~kiko
On Mon, 21 Mar 2016 17:56:38 -0300
Christian Robottom Reis <[email protected]> wrote:
> On Mon, Mar 21, 2016 at 02:55:00PM -0300, Christian Robottom Reis wrote:
> > > Alternately, there is the /proc/fs/nfsd/unlock_ip interface. Supposedly
> > > you can echo an address into there and it'll forcibly drop all of the
> > > locks that that that client holds. I've not used that so YMMV there.
> >
> > Oh! That's a very interesting, and I now see it documented here:
> >
> > http://people.redhat.com/rpeterso/Patches/NFS/NLM/004.txt
>
> On second look, I don't think that interface is meant to take a client
> IP, but rather a server IP:
>
> "They are intended to allow admin or user mode script to release NLM
> locks based on either a path name or a server in-bound ip address[...]"
>
> That's why echoing the client IP makes no difference.
>
> I'm surprised -- so far I've found no facility for lock management
> server-side other than restarting the server.
Ahh that's exactly right -- my bad. I had forgotten that the idea there
was to use that for clustering.
And you're also correct that there is currently no facility for
administratively revoking locks. That's something that would be a nice
to have, if someone wanted to propose a sane interface and mechanism
for it. Solaris had such a thing, IIRC, but I don't know how it was
implemented.
There is one other option too -- you can send a SIGKILL to the lockd
kernel thread and it will drop _all_ of its locks. That sort of sucks
for all of the other clients, but it can unwedge things without
restarting NFS.
That said, your earlier email said:
> In the situation which happened today my guess (because it's a mbox
> file) is that a client ran something like mutt and the machine died
> somewhere during shutdown. It's my guess because AIUI the lock doesn't
> get stuck if the process is simply KILLed or crashes.
What should happen there is that the client notify the server when it
comes back up, so it can release its locks. That can fail to occur for
all sorts of reasons, and that leads exactly to the problem you have
now. It's also possible for the client to just drop off the net
indefinitely while holding locks in which case you're just out of luck.
It really is better to use NFSv4 if you can at all get away with it.
Lease-based locking puts the onus on the client to stay in contact with
the server if it wants to maintain its state.
--
Jeff Layton <[email protected]>
On Mon, Mar 21, 2016 at 05:27:35PM -0400, Jeff Layton wrote:
> And you're also correct that there is currently no facility for
> administratively revoking locks. That's something that would be a nice
> to have, if someone wanted to propose a sane interface and mechanism
> for it. Solaris had such a thing, IIRC, but I don't know how it was
> implemented.
I might look into that -- I think the right thing to do is (as you had
originally alluded to) dropping all locks pertaining to a specific
client, as the only failure scenario that can't be worked around that
I'm thinking about is the client disappearing.
I would also like to understand whether the data structure behind
/proc/locks could be extended to provide additional metadata which
the nfs kernel client could annotate to indicate client information.
That would allow one to figure out who the actual culprit machine was.
> There is one other option too -- you can send a SIGKILL to the lockd
> kernel thread and it will drop _all_ of its locks. That sort of sucks
> for all of the other clients, but it can unwedge things without
> restarting NFS.
That's quite useful to know, thanks -- I knew that messing with the
initscripts responsible for the nfs kernel services "fixed" the problem,
but killing lockd is much more convenient.
I wonder, is it normal application behaviour that any locks dropped
would be detected and reestablished on the client side?
> > In the situation which happened today my guess (because it's a mbox
> > file) is that a client ran something like mutt and the machine died
> > somewhere during shutdown. It's my guess because AIUI the lock doesn't
> > get stuck if the process is simply KILLed or crashes.
>
> What should happen there is that the client notify the server when it
> comes back up, so it can release its locks. That can fail to occur for
> all sorts of reasons, and that leads exactly to the problem you have
> now. It's also possible for the client to just drop off the net
> indefinitely while holding locks in which case you're just out of luck.
That's quite interesting. I had initially thought that a misbehaved
application could die while holding a lock, but it seems as though the
client kernel tracks any remote locks held and releases them regardless
of how the process dies. It seems like the actual problem scenarios are:
- Client disappears off the net while holding a lock
- Client kernel fails to clear NFS locks (likely a bug)
- Rogue or misbehaved client holds a lock indefinitely
In any of these cases, the useful thing to know is which client actually
holds the lock.
> It really is better to use NFSv4 if you can at all get away with it.
> Lease-based locking puts the onus on the client to stay in contact with
> the server if it wants to maintain its state.
I considered moving a few times, but the setup here is a bit fragile and
AIUI NFSv4 isn't a straight drop-in for v3. Beyond modifying nfsvers,
IIRC at least idmapd needed to be set up and perhaps there was more.
--
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko
| [+55 16] 991 126 430 | http://launchpad.net/~kiko
On Mon, Mar 21, 2016 at 09:09:11PM -0300, Christian Robottom Reis wrote:
> On Mon, Mar 21, 2016 at 05:27:35PM -0400, Jeff Layton wrote:
> > And you're also correct that there is currently no facility for
> > administratively revoking locks. That's something that would be a nice
> > to have, if someone wanted to propose a sane interface and mechanism
> > for it. Solaris had such a thing, IIRC, but I don't know how it was
> > implemented.
>
> I might look into that -- I think the right thing to do is (as you had
> originally alluded to) dropping all locks pertaining to a specific
> client, as the only failure scenario that can't be worked around that
> I'm thinking about is the client disappearing.
>
> I would also like to understand whether the data structure behind
> /proc/locks could be extended to provide additional metadata which
> the nfs kernel client could annotate to indicate client information.
> That would allow one to figure out who the actual culprit machine was.
>
> > There is one other option too -- you can send a SIGKILL to the lockd
> > kernel thread and it will drop _all_ of its locks. That sort of sucks
> > for all of the other clients, but it can unwedge things without
> > restarting NFS.
>
> That's quite useful to know, thanks -- I knew that messing with the
> initscripts responsible for the nfs kernel services "fixed" the problem,
> but killing lockd is much more convenient.
>
> I wonder, is it normal application behaviour that any locks dropped
> would be detected and reestablished on the client side?
No, you generally don't want that--you don't want an application to
believe it's held a lock continuously when it reality it's been dropped
(and conflicting locks possibly granted and dropped) and then acquired
again.
Client behavior varies. I believe recent linux clients should return
-EIO on subsequent attempts to use associated file descriptors after a
lock is lost. Other OS's apparently signal the process.
--b.
On Mon, Mar 21, 2016 at 11:39:14AM -0300, Christian Robottom Reis wrote:
> indeed does not return any information pertaining NFS client locks, and
> I'm not clear whether /proc/locks (on the server side obviously) does or
> not.
Somewhat OT, but I find it a PITA that /proc/locks gives inode numbers
that then need to be looked up individually. I have often been surprised
no tool exists to parse that and give you back a report of filenames, so
I just put together a small tool that just offloads the work to debugfs.
I've attached it in case others might find it useful.
(Interestingly, in a network of Ubuntu desktops, the long-term remote lock
holders are basically Spotify, Firefox and Zeitgeist, of which the
latter two are essentially sqlite.)
--
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko
| [+55 16] 991 126 430 | http://launchpad.net/~kiko
On Tue, Mar 22 2016, Christian Robottom Reis wrote:
> On Mon, Mar 21, 2016 at 11:39:14AM -0300, Christian Robottom Reis wrote:
>> indeed does not return any information pertaining NFS client locks, and
>> I'm not clear whether /proc/locks (on the server side obviously) does or
>> not.
>
> Somewhat OT, but I find it a PITA that /proc/locks gives inode numbers
> that then need to be looked up individually. I have often been surprised
> no tool exists to parse that and give you back a report of filenames, so
> I just put together a small tool that just offloads the work to debugfs.
>
> I've attached it in case others might find it useful.
Nice idea, though it only works with extXfs - better than nothing
though.
It would be really easy to do this in the kernel.
I would be very much in favour of a /proc/locks-extended (or whatever)
that provides the same information as /proc/locks, but in a format
which includes full path name, process name, and - for nfs locks -
client name etc.
NeilBrown
On Tue, Mar 22 2016, Jeff Layton wrote:
> On Mon, 21 Mar 2016 17:56:38 -0300
> Christian Robottom Reis <[email protected]> wrote:
>
>> On Mon, Mar 21, 2016 at 02:55:00PM -0300, Christian Robottom Reis wrote:
>> > > Alternately, there is the /proc/fs/nfsd/unlock_ip interface. Supposedly
>> > > you can echo an address into there and it'll forcibly drop all of the
>> > > locks that that that client holds. I've not used that so YMMV there.
>> >
>> > Oh! That's a very interesting, and I now see it documented here:
>> >
>> > http://people.redhat.com/rpeterso/Patches/NFS/NLM/004.txt
>>
>> On second look, I don't think that interface is meant to take a client
>> IP, but rather a server IP:
>>
>> "They are intended to allow admin or user mode script to release NLM
>> locks based on either a path name or a server in-bound ip address[...]"
>>
>> That's why echoing the client IP makes no difference.
>>
>> I'm surprised -- so far I've found no facility for lock management
>> server-side other than restarting the server.
>
> Ahh that's exactly right -- my bad. I had forgotten that the idea there
> was to use that for clustering.
>
> And you're also correct that there is currently no facility for
> administratively revoking locks. That's something that would be a nice
> to have, if someone wanted to propose a sane interface and mechanism
> for it.
I was all set to give you an answer until I saw that word
"sane"... nearly scared me off, but I chose to persist.
You know this, but let me remind you and inform Christian.
When an NFS client ("bob") asks the NFS server ("jane") to lock a
file (the first time), the kernel says to statd "Hey, Bob wants a lock.
Can you keep and eye on him and let me know when he reboots - when he
does I want to discard his locks".
So statd on Jane talks to statd on Bob saying "Hey Bob, tell me if you
ever reboot - OK"? Bob takes note of this request by writing "Jane" in
/var/lib/nfs/sm.
When Bob reboots, sm-notify sees "Jane" in /var/lib/nfs/sm and sends a
message to statd on Jane "Hey Jane, Bob just rebooted. You're welcome".
statd on Jane then tells the kernel "Bob rebooted" and the kernel drops
all those locks.
And there, in that last step, we see the key. It is already possible to
tell the kernel "drop all the locks held by Bob", you just have to say
"Hey, I'm statd - Bob rebooted". Or maybe we could stay to statd "Hi
Jane, this is Bob, I just rebooted" - even though we aren't really Bob.
(Or we could just reboot Bob and let it do the talking).
I'd have to hunt through the statd code to figure out what is possible
and what is best. It can't be too hard though.
Christian: If the problem client actually comes back up (instead of
staying down) do the locks get drops as they should?
NeilBrown
On Thu, Mar 31, 2016 at 1:07 AM, NeilBrown <[email protected]> wrote:
>
> On Tue, Mar 22 2016, Christian Robottom Reis wrote:
>
> > On Mon, Mar 21, 2016 at 11:39:14AM -0300, Christian Robottom Reis wrote:
> >> indeed does not return any information pertaining NFS client locks, and
> >> I'm not clear whether /proc/locks (on the server side obviously) does or
> >> not.
> >
> > Somewhat OT, but I find it a PITA that /proc/locks gives inode numbers
> > that then need to be looked up individually. I have often been surprised
> > no tool exists to parse that and give you back a report of filenames, so
> > I just put together a small tool that just offloads the work to debugfs.
> >
> > I've attached it in case others might find it useful.
>
> Nice idea, though it only works with extXfs - better than nothing
> though.
>
> It would be really easy to do this in the kernel.
> I would be very much in favour of a /proc/locks-extended (or whatever)
> that provides the same information as /proc/locks, but in a format
> which includes full path name, process name, and - for nfs locks -
> client name etc.
Why isn't the current interface sufficient? The 'lslocks' utility
appears quite capable of extracting the full path info.
Cheers
Trond
> On Tue, Mar 22 2016, Jeff Layton wrote:
>
> > On Mon, 21 Mar 2016 17:56:38 -0300
> > Christian Robottom Reis <[email protected]> wrote:
> >
> >> On Mon, Mar 21, 2016 at 02:55:00PM -0300, Christian Robottom Reis
> wrote:
> >> > > Alternately, there is the /proc/fs/nfsd/unlock_ip interface.
> >> > > Supposedly you can echo an address into there and it'll forcibly
> >> > > drop all of the locks that that that client holds. I've not used
that so
> YMMV there.
> >> >
> >> > Oh! That's a very interesting, and I now see it documented here:
> >> >
> >> > http://people.redhat.com/rpeterso/Patches/NFS/NLM/004.txt
> >>
> >> On second look, I don't think that interface is meant to take a
> >> client IP, but rather a server IP:
> >>
> >> "They are intended to allow admin or user mode script to release NLM
> >> locks based on either a path name or a server in-bound ip
address[...]"
> >>
> >> That's why echoing the client IP makes no difference.
> >>
> >> I'm surprised -- so far I've found no facility for lock management
> >> server-side other than restarting the server.
> >
> > Ahh that's exactly right -- my bad. I had forgotten that the idea
> > there was to use that for clustering.
> >
> > And you're also correct that there is currently no facility for
> > administratively revoking locks. That's something that would be a nice
> > to have, if someone wanted to propose a sane interface and mechanism
> > for it.
>
> I was all set to give you an answer until I saw that word "sane"... nearly
> scared me off, but I chose to persist.
>
> You know this, but let me remind you and inform Christian.
>
> When an NFS client ("bob") asks the NFS server ("jane") to lock a file
(the
> first time), the kernel says to statd "Hey, Bob wants a lock.
> Can you keep and eye on him and let me know when he reboots - when he
> does I want to discard his locks".
>
> So statd on Jane talks to statd on Bob saying "Hey Bob, tell me if you
ever
> reboot - OK"? Bob takes note of this request by writing "Jane" in
> /var/lib/nfs/sm.
Actually, Jane's statd and Bob's statd don't talk until one of them comes
back up after reboot...
The actual process is:
1. application on Bob makes a lock request on a mount from jane
2. Bob's lockd asks Bob's statd to record jane as a party of interest after
reboot
3. Bob's lockd requests lock from Jane's lockd
4. Jane's lockd asks Jane's statd to record Bob as a party of interest after
reboot
5. Jane's lockd completes lock request and responds to Bob's lockd
6. Bob crashes
7. Jane is oblivious to this crash for now...
8. Bob restarts and Bob's statd NOW sends a message to Jane's statd that it
rebooted
9 Jane's statd notes that Jane's lockd took and interest in Bob, and passes
on the fact that Bob rebooted
10. Jane's lockd cleans up the lock
> When Bob reboots, sm-notify sees "Jane" in /var/lib/nfs/sm and sends a
> message to statd on Jane "Hey Jane, Bob just rebooted. You're welcome".
>
> statd on Jane then tells the kernel "Bob rebooted" and the kernel drops
all
> those locks.
>
> And there, in that last step, we see the key. It is already possible to
tell the
> kernel "drop all the locks held by Bob", you just have to say "Hey, I'm
statd -
> Bob rebooted". Or maybe we could stay to statd "Hi Jane, this is Bob, I
just
> rebooted" - even though we aren't really Bob.
Yes, either of those would work. I'm pretty sure there are tools out there
that do this.
What is sometimes more interesting to sysadmins is being able to free up a
specific lock rather than ALL locks held by that client.
> (Or we could just reboot Bob and let it do the talking).
>
> I'd have to hunt through the statd code to figure out what is possible and
> what is best. It can't be too hard though.
>
> Christian: If the problem client actually comes back up (instead of
staying
> down) do the locks get drops as they should?
>
> NeilBrown
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
On Fri, Apr 01 2016, Trond Myklebust wrote:
> On Thu, Mar 31, 2016 at 1:07 AM, NeilBrown <[email protected]> wrote:
>>
>> On Tue, Mar 22 2016, Christian Robottom Reis wrote:
>>
>> > On Mon, Mar 21, 2016 at 11:39:14AM -0300, Christian Robottom Reis wrote:
>> >> indeed does not return any information pertaining NFS client locks, and
>> >> I'm not clear whether /proc/locks (on the server side obviously) does or
>> >> not.
>> >
>> > Somewhat OT, but I find it a PITA that /proc/locks gives inode numbers
>> > that then need to be looked up individually. I have often been surprised
>> > no tool exists to parse that and give you back a report of filenames, so
>> > I just put together a small tool that just offloads the work to debugfs.
>> >
>> > I've attached it in case others might find it useful.
>>
>> Nice idea, though it only works with extXfs - better than nothing
>> though.
>>
>> It would be really easy to do this in the kernel.
>> I would be very much in favour of a /proc/locks-extended (or whatever)
>> that provides the same information as /proc/locks, but in a format
>> which includes full path name, process name, and - for nfs locks -
>> client name etc.
>
> Why isn't the current interface sufficient? The 'lslocks' utility
> appears quite capable of extracting the full path info.
>
I don't think I've come across 'lslocks' before - thanks.
It does help a bit but when I run it, most of the "PATH" column is
empty.
Which is odd because I can fill in some of the blanks by cross
referencing pid/inode from /proc/locks with inodes numbers from
ls -liL /proc/$PID/fd
to get full names.
Any way, you are right that more information is available if you are
willing to hunt around, but not everything. In particular, for locks
held by lockd (or NFSv4 locks held by nfsd) it would be nice to know
which client host it was held on behalf of.
NeilBrown