Today, apparently at random, we had a locking problem on our LAN. Client
applications hung, restarting them led to hangs, and the client dmesgs
showed a familiar:
[443619.682118] lockd: server anthem not responding, still trying
So the server lockd apparently stopped responding to clients, and
restarting clients got us nowhere. Eventually we cycled the server and
everything's back to normal, but I'm pretty confused as to what
happened. I couldn't scrape any evidence on the server that would point
to why this happened -- no OOPS, error or even warning output.
I was reading through the thread at
http://groups.google.com.br/group/fa.linux.kernel/browse_thread/thread/6c7b5e49a46aef75/91adbb9f298db509?lnk=st&q=nfs+locking+server#91adbb9f298db509
and figured that it might be a similar problem I'm facing, but I'm not
entirely sure as it's hard to say if somebody interrupted a client
program or not (it's a large diskless network).
Clients run 2.6.24-16-generic (stock Ubuntu Hardy) and server is
2.6.22-14-generic (stock Ubuntu Gutsy).
If the problem happens again, what can I do on server and client to
further debug the problem? And is there a utility that clears locks that
we could use to avoid having to restart the server (acking the risks in
cleared locks)?
--
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3376 0125
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Christian Robottom Reis wrote:
>
> This time I did a ps auxww locking for the lockd process. And guess
> what?
>
> root 6323 0.0 0.0 0 0 ? D Jun01 0:50 [lockd]
>
> I wonder why it's in the D state. I also wonder if there's a way to get
> it back once it's in this state -- without reloading the kernel module
> or rebooting, I guess.
>
> I've collected a trace, at any rate, but lockd isn't even listed in it --
> I can send it in if it makes sense.
>
What kind of "trace" data you've collected ? As a rule of thumb, when a
process is stuck inside the kernel, the best approach is to:
shell> cd /proc
shell> echo w > sysrq-trigger // do this a couple of times
shell> echo t > sysrq-trigger
The "w" will force kernel to print out threads' backtrace that are
currently on the active CPUs. The "t" will print out all the thread
backtraces on this machine (but sometime skip the ones spinning on the
CPUs). These traces will give people a much better idea what went on in
the kernel at that particular time. All the backtraces should show up in
/var/log/messages file and/or system console.
*Warning* ... the "t" will pause system for a noticeable amount of time
(few seconds to few minutes, depending on thread counts) since it has to
walk thru every thread's stack in that running system. If you have
cluster configured, it could make the node missing its heartbeat
processing (so you need to increase the heartbeat interval before doing
this).
> What sort of debugging can I do to figure out what's wrong here?
>
> (This is a dual-Xeon running:
>
> Linux anthem 2.6.22-14-generic #1 SMP Tue Feb 12 07:42:25 UTC 2008 i686 GNU/Linux)
>
Another approach is to make a debug kernel and run "crash" to poke the
live kernel. Dave Anderson from Red Hat has an excellent tutorial in his
people's page: http://people.redhat.com/anderson . It is also very helpful.
-- Wendy
Christian Robottom Reis wrote:
> On Fri, May 09, 2008 at 11:43:05AM -0400, J. Bruce Fields wrote:
>
>
>> Perhaps a sysrq-T dump of lockd would show where (and whether) it's
>> blocked? (So once lockd stops responding, log into the server, run
>> "echo t >/proc/sysrq-trigger", and collect the output from the logs,
>> especially the stacktrace for the lockd process).
>>
Sorry, I was debugging some other stuffs on my screen while mindlessly
reading the mailing list - somehow missed Bruce's original "sysrq-t"
paragraph. However, look like you've missed the "w" part. My guess is
that the lockd was spinning on the CPU when you hit the sysrq-t key or
your sysrq-t didn't run to its completion at all.
-- Wendy
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Thu, 12 Jun 2008 18:43:40 -0300
Christian Robottom Reis <[email protected]> wrote:
> On Fri, May 09, 2008 at 11:43:05AM -0400, J. Bruce Fields wrote:
> > I don't think the server stopped responding to clients in the case
> > Miklos described.
>
> Okay. Well, one month later, it happened again to me.
>
> > Perhaps a sysrq-T dump of lockd would show where (and whether) it's
> > blocked? (So once lockd stops responding, log into the server, run
> > "echo t >/proc/sysrq-trigger", and collect the output from the logs,
> > especially the stacktrace for the lockd process).
>
> This time I did a ps auxww locking for the lockd process. And guess
> what?
>
> root 6323 0.0 0.0 0 0 ? D Jun01 0:50 [lockd]
>
> I wonder why it's in the D state. I also wonder if there's a way to get
> it back once it's in this state -- without reloading the kernel module
> or rebooting, I guess.
>
> I've collected a trace, at any rate, but lockd isn't even listed in it --
> I can send it in if it makes sense.
>
That's not atypical at all. syslog uses unreliable transport. When you
send it a flood of data (say, with a sysrq-t) some of it can be lost.
Usually I recommend dumping the data straight out of the ring buffer
from a sysrq-t:
# dmesg > /tmp/sysrq-t.out
...or something. You might still lose stuff that got pushed out of the
ring buffer, but the stuff that is there will at least be complete.
> What sort of debugging can I do to figure out what's wrong here?
>
You'll really need that sysrq-t info...or a core dump, or to run a
debugger on the running kernel (like Wendy recommended).
> (This is a dual-Xeon running:
>
> Linux anthem 2.6.22-14-generic #1 SMP Tue Feb 12 07:42:25 UTC 2008 i686 GNU/Linux)
There were some patches that went into 2.6.25 (I think) that fix
problems that could cause lockd to hang in some cases. This patch,
in particular, may be of interest:
Subject: [PATCH 1/4] NLM: set RPC_CLNT_CREATE_NOPING for NLM RPC clients
Cheers,
--
Jeff Layton <[email protected]>
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs