2008-11-04 01:24:48

by Brian J. Murrell

[permalink] [raw]
Subject: Re: stuck/hung nfsv4 mounts

On Mon, 2008-11-03 at 17:33 -0500, Kevin Coffman wrote:
> If you can get
> a packet trace between the client and server during the mount /
> context negotiation, check the NULL reply from the server for this
> error code.

Here is the NULL Call from client:
No. Time Time Delta Source Destination Protocol Info
8868 20:07:43.933888 0.043671 10.75.22.1 10.75.22.3 NFS V4 NULL Call (Reply In 8870)

Frame 8868 (606 bytes on wire, 606 bytes captured)
Ethernet II, Src: AsustekC_e1:65:cc (00:18:f3:e1:65:cc), Dst: Intel_3f:8b:74 (00:d0:b7:3f:8b:74)
Internet Protocol, Src: 10.75.22.1 (10.75.22.1), Dst: 10.75.22.3 (10.75.22.3)
Transmission Control Protocol, Src Port: imaps (993), Dst Port: nfs (2049), Seq: 1, Ack: 1, Len: 552
Remote Procedure Call, Type:Call XID:0x48023eeb
Network File System
[Program Version: 4]
[V4 Procedure: NULL (0)]
GSS Token: ...
GSS Token Length: 483
GSS-API Generic Security Service Application Program Interface
OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5)
krb5_blob: ...
krb5_tok_id: KRB5_AP_REQ (0x0001)
Kerberos AP-REQ
Pvno: 5
MSG Type: AP-REQ (14)
Padding: 0
APOptions: 20000000 (Mutual required)
.0.. .... .... .... .... .... .... .... = Use Session Key: Do NOT use the session key to encrypt the ticket
..1. .... .... .... .... .... .... .... = Mutual required: MUTUAL authentication is REQUIRED
Ticket
Tkt-vno: 5
Realm: ILINX
Server Name (Service and Host): nfs/linux.interlinx.bc.ca
enc-part des-cbc-crc
Authenticator des-cbc-crc

Here is the NULL reply from the server:

No. Time Time Delta Source Destination Protocol Info
8870 20:07:44.062206 0.128081 10.75.22.3 10.75.22.1 NFS V4 NULL Reply (Call In 8868)

Frame 8870 (262 bytes on wire, 262 bytes captured)
Ethernet II, Src: Intel_3f:8b:74 (00:d0:b7:3f:8b:74), Dst: AsustekC_e1:65:cc (00:18:f3:e1:65:cc)
Internet Protocol, Src: 10.75.22.3 (10.75.22.3), Dst: 10.75.22.1 (10.75.22.1)
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: imaps (993), Seq: 1, Ack: 553, Len: 208
Remote Procedure Call, Type:Reply XID:0x48023eeb
Network File System
[Program Version: 4]
[V4 Procedure: NULL (0)]
GSS Context: <DATA>
GSS Major Status: 0
GSS Minor Status: 0
GSS Sequence Window: 128
GSS Token: ...
GSS Token Length: 114
GSS-API Generic Security Service Application Program Interface
OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5)
krb5_blob: ...
krb5_tok_id: KRB5_AP_REP (0x0002)
Kerberos AP-REP
Pvno: 5
MSG Type: AP-REP (15)
enc-part des-cbc-crc
Encryption type: des-cbc-crc (1)
enc-part: ...

b.




2008-11-04 15:14:39

by Brian J. Murrell

[permalink] [raw]
Subject: Re: stuck/hung nfsv4 mounts

Hi All,

Is there any more debugging I can/you want me to do here or can/shall I
just reboot this machine?

Thanx,
b.



2008-11-04 17:22:27

by Kevin Coffman

[permalink] [raw]
Subject: Re: stuck/hung nfsv4 mounts

On Mon, Nov 3, 2008 at 8:24 PM, Brian J. Murrell <[email protected]> wrote:
> On Mon, 2008-11-03 at 17:33 -0500, Kevin Coffman wrote:
>> If you can get
>> a packet trace between the client and server during the mount /
>> context negotiation, check the NULL reply from the server for this
>> error code.
>
> Here is the NULL Call from client:
> No. Time Time Delta Source Destination Protocol Info
> 8868 20:07:43.933888 0.043671 10.75.22.1 10.75.22.3 NFS V4 NULL Call (Reply In 8870)
>
> Frame 8868 (606 bytes on wire, 606 bytes captured)
> Ethernet II, Src: AsustekC_e1:65:cc (00:18:f3:e1:65:cc), Dst: Intel_3f:8b:74 (00:d0:b7:3f:8b:74)
> Internet Protocol, Src: 10.75.22.1 (10.75.22.1), Dst: 10.75.22.3 (10.75.22.3)
> Transmission Control Protocol, Src Port: imaps (993), Dst Port: nfs (2049), Seq: 1, Ack: 1, Len: 552
> Remote Procedure Call, Type:Call XID:0x48023eeb
> Network File System
> [Program Version: 4]
> [V4 Procedure: NULL (0)]
> GSS Token: ...
> GSS Token Length: 483
> GSS-API Generic Security Service Application Program Interface
> OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5)
> krb5_blob: ...
> krb5_tok_id: KRB5_AP_REQ (0x0001)
> Kerberos AP-REQ
> Pvno: 5
> MSG Type: AP-REQ (14)
> Padding: 0
> APOptions: 20000000 (Mutual required)
> .0.. .... .... .... .... .... .... .... = Use Session Key: Do NOT use the session key to encrypt the ticket
> ..1. .... .... .... .... .... .... .... = Mutual required: MUTUAL authentication is REQUIRED
> Ticket
> Tkt-vno: 5
> Realm: ILINX
> Server Name (Service and Host): nfs/linux.interlinx.bc.ca
> enc-part des-cbc-crc
> Authenticator des-cbc-crc
>
> Here is the NULL reply from the server:
>
> No. Time Time Delta Source Destination Protocol Info
> 8870 20:07:44.062206 0.128081 10.75.22.3 10.75.22.1 NFS V4 NULL Reply (Call In 8868)
>
> Frame 8870 (262 bytes on wire, 262 bytes captured)
> Ethernet II, Src: Intel_3f:8b:74 (00:d0:b7:3f:8b:74), Dst: AsustekC_e1:65:cc (00:18:f3:e1:65:cc)
> Internet Protocol, Src: 10.75.22.3 (10.75.22.3), Dst: 10.75.22.1 (10.75.22.1)
> Transmission Control Protocol, Src Port: nfs (2049), Dst Port: imaps (993), Seq: 1, Ack: 553, Len: 208
> Remote Procedure Call, Type:Reply XID:0x48023eeb
> Network File System
> [Program Version: 4]
> [V4 Procedure: NULL (0)]
> GSS Context: <DATA>
> GSS Major Status: 0
> GSS Minor Status: 0
> GSS Sequence Window: 128
> GSS Token: ...
> GSS Token Length: 114
> GSS-API Generic Security Service Application Program Interface
> OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5)
> krb5_blob: ...
> krb5_tok_id: KRB5_AP_REP (0x0002)
> Kerberos AP-REP
> Pvno: 5
> MSG Type: AP-REP (15)
> enc-part des-cbc-crc
> Encryption type: des-cbc-crc (1)
> enc-part: ...
>
> b.

>From this output, it looks like the response back from the server was
a success. So it appears to be some other issue.

Perhaps, back to the processes stuck in the kernel? Unless Trond has
suggestions, I would reboot the client and see if it happens again
(w/o soft mounts).

K.C.