Hello. I am with the institutional hosting service at NASA's Jet
Propulsion Laboratory and we have been trying to find the answer to an
apparently simple question.
We are running an NFSv4 server on Centos 7 and we need to get the server
to tell us which files it thinks are locked by which clients. This is
because we have observed failure modes where something apparently has a
lock (because attempting to lock the file again blocks) but we can't
find a client that has the lock. Finding out what the server believes
would be critical to troubleshooting this.
We tracked down Trond Myklebust and Neil Brown and conversation suggests
that this is a function that is (a) not currently available and (b)
reasonable to ask for. So this is me suggesting that an interface be
provided to have knfsd output its list of locked files and clients.
Regards,
Peter Scott
Office of the CIO
On Mon, Jul 09, 2018 at 07:28:53PM -0700, Peter Scott wrote:
> Hello. I am with the institutional hosting service at NASA's Jet
> Propulsion Laboratory and we have been trying to find the answer to
> an apparently simple question.
>
> We are running an NFSv4 server on Centos 7 and we need to get the
> server to tell us which files it thinks are locked by which
> clients. This is because we have observed failure modes where
> something apparently has a lock (because attempting to lock the file
> again blocks) but we can't find a client that has the lock. Finding
> out what the server believes would be critical to troubleshooting
> this.
>
> We tracked down Trond Myklebust and Neil Brown and conversation
> suggests that this is a function that is (a) not currently available
> and (b) reasonable to ask for. So this is me suggesting that an
> interface be provided to have knfsd output its list of locked files
> and clients.
Yes, we really need to do that. If anyone has the cycles to take it on
I'm happy to talk about it. Otherwise, I'll get to it myself
eventually....
--b.
Solaris and FreeBSD dtrace facility supports this functionality.
Unfortunately, Linux rejected dtrace for license issues, and instead
did... nothing.
Ced
On 10 July 2018 at 04:28, Peter Scott <[email protected]> wrote:
> Hello. I am with the institutional hosting service at NASA's Jet Propulsion
> Laboratory and we have been trying to find the answer to an apparently
> simple question.
>
> We are running an NFSv4 server on Centos 7 and we need to get the server to
> tell us which files it thinks are locked by which clients. This is because
> we have observed failure modes where something apparently has a lock
> (because attempting to lock the file again blocks) but we can't find a
> client that has the lock. Finding out what the server believes would be
> critical to troubleshooting this.
>
> We tracked down Trond Myklebust and Neil Brown and conversation suggests
> that this is a function that is (a) not currently available and (b)
> reasonable to ask for. So this is me suggesting that an interface be
> provided to have knfsd output its list of locked files and clients.
>
> Regards,
> Peter Scott
> Office of the CIO
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Cedric Blancher <[email protected]>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur
On Thu, Aug 02, 2018 at 08:50:20PM +0200, Cedric Blancher wrote:
> Solaris and FreeBSD dtrace facility supports this functionality.
> Unfortunately, Linux rejected dtrace for license issues, and instead
> did... nothing.
I didn't think was this sort of question tracing was designed to
answer--it's a question about the system state at a given time (who
holds what locks), rather than a list of events. But I know nothing
about dtrace.
--b.
>
> Ced
>
> On 10 July 2018 at 04:28, Peter Scott <[email protected]> wrote:
> > Hello. I am with the institutional hosting service at NASA's Jet Propulsion
> > Laboratory and we have been trying to find the answer to an apparently
> > simple question.
> >
> > We are running an NFSv4 server on Centos 7 and we need to get the server to
> > tell us which files it thinks are locked by which clients. This is because
> > we have observed failure modes where something apparently has a lock
> > (because attempting to lock the file again blocks) but we can't find a
> > client that has the lock. Finding out what the server believes would be
> > critical to troubleshooting this.
> >
> > We tracked down Trond Myklebust and Neil Brown and conversation suggests
> > that this is a function that is (a) not currently available and (b)
> > reasonable to ask for. So this is me suggesting that an interface be
> > provided to have knfsd output its list of locked files and clients.
> >
> > Regards,
> > Peter Scott
> > Office of the CIO
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Cedric Blancher <[email protected]>
> [https://plus.google.com/u/0/+CedricBlancher/]
> Institute Pasteur
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 07 Aug 2018, J. Bruce Fields wrote:
> On Thu, Aug 02, 2018 at 08:50:20PM +0200, Cedric Blancher wrote:
> > Solaris and FreeBSD dtrace facility supports this functionality.
> > Unfortunately, Linux rejected dtrace for license issues, and instead
> > did... nothing.
>
> I didn't think was this sort of question tracing was designed to
> answer--it's a question about the system state at a given time (who
> holds what locks), rather than a list of events. But I know nothing
> about dtrace.
FWIW a couple of years ago I wrote a systemtap script (attached) that
dumps out some NFSv4 state info (mostly lock info) from nfsd. I ran
it on a recent RHEL 7 kernel and it still works, so it should work on
Centos 7.
-Scott
>
> --b.
>
> >
> > Ced
> >
> > On 10 July 2018 at 04:28, Peter Scott <[email protected]> wrote:
> > > Hello. I am with the institutional hosting service at NASA's Jet Propulsion
> > > Laboratory and we have been trying to find the answer to an apparently
> > > simple question.
> > >
> > > We are running an NFSv4 server on Centos 7 and we need to get the server to
> > > tell us which files it thinks are locked by which clients. This is because
> > > we have observed failure modes where something apparently has a lock
> > > (because attempting to lock the file again blocks) but we can't find a
> > > client that has the lock. Finding out what the server believes would be
> > > critical to troubleshooting this.
> > >
> > > We tracked down Trond Myklebust and Neil Brown and conversation suggests
> > > that this is a function that is (a) not currently available and (b)
> > > reasonable to ask for. So this is me suggesting that an interface be
> > > provided to have knfsd output its list of locked files and clients.
> > >
> > > Regards,
> > > Peter Scott
> > > Office of the CIO
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > > the body of a message to [email protected]
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
> >
> > --
> > Cedric Blancher <[email protected]>
> > [https://plus.google.com/u/0/+CedricBlancher/]
> > Institute Pasteur
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Bruce,
FWIW, I don't think dtrace does/did what NASA is asking for, but that's
really irrelevant. The locks in NFSv4 are application specific locks,
not generic kernel locks, so a service looking for spinlocks, mutexes,
etc... wouldn't really be helpful I don't think. However implementing a
/sys/class, debugfs thingy that dumped the nfsv4 locks held is fairly
simple, although I would think there are security implications.
Tom
On 8/7/2018 3:34 PM, J. Bruce Fields wrote:
> On Thu, Aug 02, 2018 at 08:50:20PM +0200, Cedric Blancher wrote:
>> Solaris and FreeBSD dtrace facility supports this functionality.
>> Unfortunately, Linux rejected dtrace for license issues, and instead
>> did... nothing.
> I didn't think was this sort of question tracing was designed to
> answer--it's a question about the system state at a given time (who
> holds what locks), rather than a list of events. But I know nothing
> about dtrace.
>
> --b.
>
>> Ced
>>
>> On 10 July 2018 at 04:28, Peter Scott <[email protected]> wrote:
>>> Hello. I am with the institutional hosting service at NASA's Jet Propulsion
>>> Laboratory and we have been trying to find the answer to an apparently
>>> simple question.
>>>
>>> We are running an NFSv4 server on Centos 7 and we need to get the server to
>>> tell us which files it thinks are locked by which clients. This is because
>>> we have observed failure modes where something apparently has a lock
>>> (because attempting to lock the file again blocks) but we can't find a
>>> client that has the lock. Finding out what the server believes would be
>>> critical to troubleshooting this.
>>>
>>> We tracked down Trond Myklebust and Neil Brown and conversation suggests
>>> that this is a function that is (a) not currently available and (b)
>>> reasonable to ask for. So this is me suggesting that an interface be
>>> provided to have knfsd output its list of locked files and clients.
>>>
>>> Regards,
>>> Peter Scott
>>> Office of the CIO
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to [email protected]
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
>> --
>> Cedric Blancher <[email protected]>
>> [https://plus.google.com/u/0/+CedricBlancher/]
>> Institute Pasteur
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Thomas Tucker
President
Open Grid Computing, Inc.
(m) 512-826-7622
(o) 512-220-3434
On Tue, Aug 07, 2018 at 04:05:42PM -0500, Tom Tucker wrote:
> FWIW, I don't think dtrace does/did what NASA is asking for, but
> that's really irrelevant. The locks in NFSv4 are application
> specific locks, not generic kernel locks, so a service looking for
> spinlocks, mutexes, etc... wouldn't really be helpful I don't
> think. However implementing a /sys/class, debugfs thingy that
> dumped the nfsv4 locks held is fairly simple, although I would think
> there are security implications.
We already have /proc/locks for locks taken by regular applications, it
just misses locks from nfsd.
We could extend that. But /proc/locks has some other problems and best
is probably to make a list of those and make sure we address them all at
once. It's a potential woodshedding exercise. I haven't done any work
on it.
Alternatively I guess we could make a separate interface just for file
locks held by knfsd.
--b.
On Tue, Aug 07, 2018 at 05:16:24PM -0400, J. Bruce Fields wrote:
> On Tue, Aug 07, 2018 at 04:05:42PM -0500, Tom Tucker wrote:
> > FWIW, I don't think dtrace does/did what NASA is asking for, but
> > that's really irrelevant. The locks in NFSv4 are application
> > specific locks, not generic kernel locks, so a service looking for
> > spinlocks, mutexes, etc... wouldn't really be helpful I don't
> > think. However implementing a /sys/class, debugfs thingy that
> > dumped the nfsv4 locks held is fairly simple, although I would think
> > there are security implications.
>
> We already have /proc/locks for locks taken by regular applications, it
> just misses locks from nfsd.
>
> We could extend that. But /proc/locks has some other problems and best
> is probably to make a list of those and make sure we address them all at
> once. It's a potential woodshedding exercise.
(Um, bikeshedding, bikeshedding. Woodshedding is an entirely different
thing....)
--b.
> I haven't done any work
> on it.
>
> Alternatively I guess we could make a separate interface just for file
> locks held by knfsd.
>
> --b.