I have seen some weird NFS issues in my days, but this one may top them
all. I have one EL9 client connecting to an EL7 server where I'm seeing
random access issues:
$ wc -l scipy39-intel-*
454 scipy39-intel-qcomp1.1
wc: scipy39-intel-qcomp2: Operation not permitted
363 scipy39-intel-smmic1
365 scipy39-intel-smmic1.1
1182 total
$ wc -l scipy39-intel-*
454 scipy39-intel-qcomp1.1
435 scipy39-intel-qcomp2
363 scipy39-intel-smmic1
365 scipy39-intel-smmic1.1
Client:
nfs-utils-2.5.4-18.el9.x86_64
kernel-5.14.0-284.30.1.el9_2.x86_64
earthg.cora.nwra.com:/export/home/orion /home/orion nfs4
rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=1,acregmax=1,acdirmin=1,acdirmax=1,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.133,lookupcache=pos,local_lock=none,addr=192.168.1.5
0 0
Server:
nfs-utils-1.3.0-0.68.el7.2.x86_64
kernel-3.10.0-1160.99.1.el7.x86_64
/ -ro,async,fsid=0 192.168.1.0/24(sec=sys)
192.168.2.0/24(sec=sys) 10.0.0.0/8(sec=krb5) *.nwra.com(sec=krb5)
/export/home -rw,async,nohide 192.168.1.0/24(sec=sys)
192.168.2.0/24(sec=sys) 10.0.0.0/8(sec=krb5) *.nwra.com(sec=krb5)
tshark reports the server returning NFS4ERR_NOENT:
Frame 41: 318 bytes on wire (2544 bits), 318 bytes captured (2544 bits)
on interface 0
Ethernet II, Src: RealtekU_a9:d8:34 (52:54:00:a9:d8:34), Dst:
Dell_4d:82:59 (00:24:e8:4d:82:59)
Internet Protocol Version 4, Src: 192.168.1.133 (192.168.1.133), Dst:
192.168.1.5 (192.168.1.5)
Transmission Control Protocol, Src Port: gdoi (848), Dst Port: nfs
(2049), Seq: 5329, Ack: 4625, Len: 264
Remote Procedure Call, Type:Call XID:0xef4cfdaf
Network File System, Ops(3): SEQUENCE, PUTFH, CLOSE
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Tag: <EMPTY>
length: 0
contents: <EMPTY>
minorversion: 2
Operations (count: 3): SEQUENCE, PUTFH, CLOSE
Opcode: SEQUENCE (53)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f7
slot id: 0
high slot id: 0
cache this?: Yes
Opcode: PUTFH (22)
filehandle
length: 32
[hash (CRC-32): 0xefafdd27]
decode type as: unknown
filehandle:
01000681c370d9f027664f269d1025f5132996603f233624...
Opcode: CLOSE (4)
seqid: 0x00000000
stateid
[StateID Hash: 0x6ae3]
seqid: 0x00000001
Data: 1f100365e983952638000000
[Main Opcode: CLOSE (4)]
Frame 42: 170 bytes on wire (1360 bits), 170 bytes captured (1360 bits)
on interface 0
Ethernet II, Src: Dell_4d:82:59 (00:24:e8:4d:82:59), Dst:
RealtekU_a9:d8:34 (52:54:00:a9:d8:34)
Internet Protocol Version 4, Src: 192.168.1.5 (192.168.1.5), Dst:
192.168.1.133 (192.168.1.133)
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: gdoi
(848), Seq: 4625, Ack: 5593, Len: 116
Remote Procedure Call, Type:Reply XID:0xef4cfdaf
Network File System, Ops(3): SEQUENCE PUTFH CLOSE
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Status: NFS4_OK (0)
Tag: <EMPTY>
length: 0
contents: <EMPTY>
Operations (count: 3)
Opcode: SEQUENCE (53)
Status: NFS4_OK (0)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f7
slot id: 0
high slot id: 9
target high slot id: 9
status flags: 0x00000000
.... .... .... .... .... .... .... ...0 =
SEQ4_STATUS_CB_PATH_DOWN: Not set
.... .... .... .... .... .... .... ..0. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING: Not set
.... .... .... .... .... .... .... .0.. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED: Not set
.... .... .... .... .... .... .... 0... =
SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED: Not set
.... .... .... .... .... .... ...0 .... =
SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED: Not set
.... .... .... .... .... .... ..0. .... =
SEQ4_STATUS_ADMIN_STATE_REVOKED: Not set
.... .... .... .... .... .... .0.. .... =
SEQ4_STATUS_RECALLABLE_STATE_REVOKED: Not set
.... .... .... .... .... .... 0... .... =
SEQ4_STATUS_LEASE_MOVED: Not set
.... .... .... .... .... ...0 .... .... =
SEQ4_STATUS_RESTART_RECLAIM_NEEDED: Not set
.... .... .... .... .... ..0. .... .... =
SEQ4_STATUS_CB_PATH_DOWN_SESSION: Not set
.... .... .... .... .... .0.. .... .... =
SEQ4_STATUS_BACKCHANNEL_FAULT: Not set
.... .... .... .... .... 0... .... .... =
SEQ4_STATUS_DEVID_CHANGED: Not set
.... .... .... .... ...0 .... .... .... =
SEQ4_STATUS_DEVID_DELETED: Not set
Opcode: PUTFH (22)
Status: NFS4_OK (0)
Opcode: CLOSE (4)
Status: NFS4_OK (0)
stateid
[StateID Hash: 0x6414]
seqid: 0x00000002
Data: 1f100365e983952638000000
[Main Opcode: CLOSE (4)]
Frame 43: 326 bytes on wire (2608 bits), 326 bytes captured (2608 bits)
on interface 0
Ethernet II, Src: RealtekU_a9:d8:34 (52:54:00:a9:d8:34), Dst:
Dell_4d:82:59 (00:24:e8:4d:82:59)
Internet Protocol Version 4, Src: 192.168.1.133 (192.168.1.133), Dst:
192.168.1.5 (192.168.1.5)
Transmission Control Protocol, Src Port: gdoi (848), Dst Port: nfs
(2049), Seq: 5593, Ack: 4741, Len: 272
Remote Procedure Call, Type:Call XID:0xf04cfdaf
Network File System, Ops(5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Tag: <EMPTY>
length: 0
contents: <EMPTY>
minorversion: 2
Operations (count: 5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
Opcode: SEQUENCE (53)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f8
slot id: 0
high slot id: 0
cache this?: No
Opcode: PUTFH (22)
filehandle
length: 32
[hash (CRC-32): 0x6fd1a280]
decode type as: unknown
filehandle:
01000681c370d9f027664f269d1025f51329966086b50420...
Opcode: LOOKUP (15)
Name: .git
length: 4
contents: .git
Opcode: GETFH (10)
Opcode: GETATTR (9)
Attr mask[0]: 0x0010011a (TYPE, CHANGE, SIZE, FSID, FILEID)
reqd_attr: TYPE (1)
reqd_attr: CHANGE (3)
reqd_attr: SIZE (4)
reqd_attr: FSID (8)
reco_attr: FILEID (20)
Attr mask[1]: 0x00b0a23a (MODE, NUMLINKS, OWNER,
OWNER_GROUP, RAWDEV, SPACE_USED, TIME_ACCESS, TIME_METADATA,
TIME_MODIFY, MOUNTED_ON_FILEID)
reco_attr: MODE (33)
reco_attr: NUMLINKS (35)
reco_attr: OWNER (36)
reco_attr: OWNER_GROUP (37)
reco_attr: RAWDEV (41)
reco_attr: SPACE_USED (45)
reco_attr: TIME_ACCESS (47)
reco_attr: TIME_METADATA (52)
reco_attr: TIME_MODIFY (53)
reco_attr: MOUNTED_ON_FILEID (55)
[Main Opcode: LOOKUP (15)]
Frame 44: 154 bytes on wire (1232 bits), 154 bytes captured (1232 bits)
on interface 0
Ethernet II, Src: Dell_4d:82:59 (00:24:e8:4d:82:59), Dst:
RealtekU_a9:d8:34 (52:54:00:a9:d8:34)
Internet Protocol Version 4, Src: 192.168.1.5 (192.168.1.5), Dst:
192.168.1.133 (192.168.1.133)
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: gdoi
(848), Seq: 4741, Ack: 5865, Len: 100
Remote Procedure Call, Type:Reply XID:0xf04cfdaf
Network File System, Ops(3): SEQUENCE PUTFH LOOKUP(NFS4ERR_NOENT)
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Status: NFS4ERR_NOENT (2)
Tag: <EMPTY>
length: 0
contents: <EMPTY>
Operations (count: 3)
Opcode: SEQUENCE (53)
Status: NFS4_OK (0)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f8
slot id: 0
high slot id: 9
target high slot id: 9
status flags: 0x00000000
.... .... .... .... .... .... .... ...0 =
SEQ4_STATUS_CB_PATH_DOWN: Not set
.... .... .... .... .... .... .... ..0. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING: Not set
.... .... .... .... .... .... .... .0.. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED: Not set
.... .... .... .... .... .... .... 0... =
SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED: Not set
.... .... .... .... .... .... ...0 .... =
SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED: Not set
.... .... .... .... .... .... ..0. .... =
SEQ4_STATUS_ADMIN_STATE_REVOKED: Not set
.... .... .... .... .... .... .0.. .... =
SEQ4_STATUS_RECALLABLE_STATE_REVOKED: Not set
.... .... .... .... .... .... 0... .... =
SEQ4_STATUS_LEASE_MOVED: Not set
.... .... .... .... .... ...0 .... .... =
SEQ4_STATUS_RESTART_RECLAIM_NEEDED: Not set
.... .... .... .... .... ..0. .... .... =
SEQ4_STATUS_CB_PATH_DOWN_SESSION: Not set
.... .... .... .... .... .0.. .... .... =
SEQ4_STATUS_BACKCHANNEL_FAULT: Not set
.... .... .... .... .... 0... .... .... =
SEQ4_STATUS_DEVID_CHANGED: Not set
.... .... .... .... ...0 .... .... .... =
SEQ4_STATUS_DEVID_DELETED: Not set
Opcode: PUTFH (22)
Status: NFS4_OK (0)
Opcode: LOOKUP (15)
Status: NFS4ERR_NOENT (2)
[Main Opcode: LOOKUP (15)]
Frame 45: 326 bytes on wire (2608 bits), 326 bytes captured (2608 bits)
on interface 0
Ethernet II, Src: RealtekU_a9:d8:34 (52:54:00:a9:d8:34), Dst:
Dell_4d:82:59 (00:24:e8:4d:82:59)
Internet Protocol Version 4, Src: 192.168.1.133 (192.168.1.133), Dst:
192.168.1.5 (192.168.1.5)
Transmission Control Protocol, Src Port: gdoi (848), Dst Port: nfs
(2049), Seq: 5865, Ack: 4841, Len: 272
Remote Procedure Call, Type:Call XID:0xf14cfdaf
Network File System, Ops(5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Tag: <EMPTY>
length: 0
contents: <EMPTY>
minorversion: 2
Operations (count: 5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
Opcode: SEQUENCE (53)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f9
slot id: 0
high slot id: 0
cache this?: No
Opcode: PUTFH (22)
filehandle
length: 32
[hash (CRC-32): 0x6fd1a280]
decode type as: unknown
filehandle:
01000681c370d9f027664f269d1025f51329966086b50420...
Opcode: LOOKUP (15)
Name: .git
length: 4
contents: .git
Opcode: GETFH (10)
Opcode: GETATTR (9)
Attr mask[0]: 0x0010011a (TYPE, CHANGE, SIZE, FSID, FILEID)
reqd_attr: TYPE (1)
reqd_attr: CHANGE (3)
reqd_attr: SIZE (4)
reqd_attr: FSID (8)
reco_attr: FILEID (20)
Attr mask[1]: 0x00b0a23a (MODE, NUMLINKS, OWNER,
OWNER_GROUP, RAWDEV, SPACE_USED, TIME_ACCESS, TIME_METADATA,
TIME_MODIFY, MOUNTED_ON_FILEID)
reco_attr: MODE (33)
reco_attr: NUMLINKS (35)
reco_attr: OWNER (36)
reco_attr: OWNER_GROUP (37)
reco_attr: RAWDEV (41)
reco_attr: SPACE_USED (45)
reco_attr: TIME_ACCESS (47)
reco_attr: TIME_METADATA (52)
reco_attr: TIME_MODIFY (53)
reco_attr: MOUNTED_ON_FILEID (55)
[Main Opcode: LOOKUP (15)]
Frame 46: 154 bytes on wire (1232 bits), 154 bytes captured (1232 bits)
on interface 0
Ethernet II, Src: Dell_4d:82:59 (00:24:e8:4d:82:59), Dst:
RealtekU_a9:d8:34 (52:54:00:a9:d8:34)
Internet Protocol Version 4, Src: 192.168.1.5 (192.168.1.5), Dst:
192.168.1.133 (192.168.1.133)
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: gdoi
(848), Seq: 4841, Ack: 6137, Len: 100
Remote Procedure Call, Type:Reply XID:0xf14cfdaf
Network File System, Ops(3): SEQUENCE PUTFH LOOKUP(NFS4ERR_NOENT)
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Status: NFS4ERR_NOENT (2)
Tag: <EMPTY>
length: 0
contents: <EMPTY>
Operations (count: 3)
Opcode: SEQUENCE (53)
Status: NFS4_OK (0)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f9
slot id: 0
high slot id: 9
target high slot id: 9
status flags: 0x00000000
.... .... .... .... .... .... .... ...0 =
SEQ4_STATUS_CB_PATH_DOWN: Not set
.... .... .... .... .... .... .... ..0. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING: Not set
.... .... .... .... .... .... .... .0.. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED: Not set
.... .... .... .... .... .... .... 0... =
SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED: Not set
.... .... .... .... .... .... ...0 .... =
SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED: Not set
.... .... .... .... .... .... ..0. .... =
SEQ4_STATUS_ADMIN_STATE_REVOKED: Not set
.... .... .... .... .... .... .0.. .... =
SEQ4_STATUS_RECALLABLE_STATE_REVOKED: Not set
.... .... .... .... .... .... 0... .... =
SEQ4_STATUS_LEASE_MOVED: Not set
.... .... .... .... .... ...0 .... .... =
SEQ4_STATUS_RESTART_RECLAIM_NEEDED: Not set
.... .... .... .... .... ..0. .... .... =
SEQ4_STATUS_CB_PATH_DOWN_SESSION: Not set
.... .... .... .... .... .0.. .... .... =
SEQ4_STATUS_BACKCHANNEL_FAULT: Not set
.... .... .... .... .... 0... .... .... =
SEQ4_STATUS_DEVID_CHANGED: Not set
.... .... .... .... ...0 .... .... .... =
SEQ4_STATUS_DEVID_DELETED: Not set
Opcode: PUTFH (22)
Status: NFS4_OK (0)
Opcode: LOOKUP (15)
Status: NFS4ERR_NOENT (2)
[Main Opcode: LOOKUP (15)]
Frame 47: 326 bytes on wire (2608 bits), 326 bytes captured (2608 bits)
on interface 0
Ethernet II, Src: RealtekU_a9:d8:34 (52:54:00:a9:d8:34), Dst:
Dell_4d:82:59 (00:24:e8:4d:82:59)
Internet Protocol Version 4, Src: 192.168.1.133 (192.168.1.133), Dst:
192.168.1.5 (192.168.1.5)
Transmission Control Protocol, Src Port: gdoi (848), Dst Port: nfs
(2049), Seq: 6137, Ack: 4941, Len: 272
Remote Procedure Call, Type:Call XID:0xf24cfdaf
Network File System, Ops(5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Tag: <EMPTY>
length: 0
contents: <EMPTY>
minorversion: 2
Operations (count: 5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
Opcode: SEQUENCE (53)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002fa
slot id: 0
high slot id: 0
cache this?: No
Opcode: PUTFH (22)
filehandle
length: 32
[hash (CRC-32): 0x6fd1a280]
decode type as: unknown
filehandle:
01000681c370d9f027664f269d1025f51329966086b50420...
Opcode: LOOKUP (15)
Name: HEAD
length: 4
contents: HEAD
Opcode: GETFH (10)
Opcode: GETATTR (9)
Attr mask[0]: 0x0010011a (TYPE, CHANGE, SIZE, FSID, FILEID)
reqd_attr: TYPE (1)
reqd_attr: CHANGE (3)
reqd_attr: SIZE (4)
reqd_attr: FSID (8)
reco_attr: FILEID (20)
Attr mask[1]: 0x00b0a23a (MODE, NUMLINKS, OWNER,
OWNER_GROUP, RAWDEV, SPACE_USED, TIME_ACCESS, TIME_METADATA,
TIME_MODIFY, MOUNTED_ON_FILEID)
reco_attr: MODE (33)
reco_attr: NUMLINKS (35)
reco_attr: OWNER (36)
reco_attr: OWNER_GROUP (37)
reco_attr: RAWDEV (41)
reco_attr: SPACE_USED (45)
reco_attr: TIME_ACCESS (47)
reco_attr: TIME_METADATA (52)
reco_attr: TIME_MODIFY (53)
reco_attr: MOUNTED_ON_FILEID (55)
[Main Opcode: LOOKUP (15)]
Frame 48: 154 bytes on wire (1232 bits), 154 bytes captured (1232 bits)
on interface 0
Ethernet II, Src: Dell_4d:82:59 (00:24:e8:4d:82:59), Dst:
RealtekU_a9:d8:34 (52:54:00:a9:d8:34)
Internet Protocol Version 4, Src: 192.168.1.5 (192.168.1.5), Dst:
192.168.1.133 (192.168.1.133)
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: gdoi
(848), Seq: 4941, Ack: 6409, Len: 100
Remote Procedure Call, Type:Reply XID:0xf24cfdaf
Network File System, Ops(3): SEQUENCE PUTFH LOOKUP(NFS4ERR_NOENT)
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Status: NFS4ERR_NOENT (2)
Tag: <EMPTY>
length: 0
contents: <EMPTY>
Operations (count: 3)
Opcode: SEQUENCE (53)
Status: NFS4_OK (0)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002fa
slot id: 0
high slot id: 9
target high slot id: 9
status flags: 0x00000000
.... .... .... .... .... .... .... ...0 =
SEQ4_STATUS_CB_PATH_DOWN: Not set
.... .... .... .... .... .... .... ..0. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING: Not set
.... .... .... .... .... .... .... .0.. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED: Not set
.... .... .... .... .... .... .... 0... =
SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED: Not set
.... .... .... .... .... .... ...0 .... =
SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED: Not set
.... .... .... .... .... .... ..0. .... =
SEQ4_STATUS_ADMIN_STATE_REVOKED: Not set
.... .... .... .... .... .... .0.. .... =
SEQ4_STATUS_RECALLABLE_STATE_REVOKED: Not set
.... .... .... .... .... .... 0... .... =
SEQ4_STATUS_LEASE_MOVED: Not set
.... .... .... .... .... ...0 .... .... =
SEQ4_STATUS_RESTART_RECLAIM_NEEDED: Not set
.... .... .... .... .... ..0. .... .... =
SEQ4_STATUS_CB_PATH_DOWN_SESSION: Not set
.... .... .... .... .... .0.. .... .... =
SEQ4_STATUS_BACKCHANNEL_FAULT: Not set
.... .... .... .... .... 0... .... .... =
SEQ4_STATUS_DEVID_CHANGED: Not set
.... .... .... .... ...0 .... .... .... =
SEQ4_STATUS_DEVID_DELETED: Not set
Opcode: PUTFH (22)
Status: NFS4_OK (0)
Opcode: LOOKUP (15)
Status: NFS4ERR_NOENT (2)
[Main Opcode: LOOKUP (15)]
I did switch the client from mounting with sec=krb5 to sec=sys. Perhaps
this has left the server in a bad state?
Any ideas what could be going wrong? I've rebooted the client many
times, but have not yet had the opportunity to reboot the server.
I've not yet been able to reproduce the issue on any other EL9 (or
other) client.
--
Orion Poplawski
he/him/his - surely the least important thing about me
IT Systems Manager 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane [email protected]
Boulder, CO 80301 https://www.nwra.com/
On 9/26/23 11:48, Orion Poplawski wrote:
> I have seen some weird NFS issues in my days, but this one may top them all.
> I have one EL9 client connecting to an EL7 server where I'm seeing random
> access issues:
>
> $ wc -l scipy39-intel-*
> 454 scipy39-intel-qcomp1.1
> wc: scipy39-intel-qcomp2: Operation not permitted
> 363 scipy39-intel-smmic1
> 365 scipy39-intel-smmic1.1
> 1182 total
> $ wc -l scipy39-intel-*
> 454 scipy39-intel-qcomp1.1
> 435 scipy39-intel-qcomp2
> 363 scipy39-intel-smmic1
> 365 scipy39-intel-smmic1.1
I (re-)discovered the source of these errors - BitDefender onaccess scanning.
Sorry for the noise.
--
Orion Poplawski
IT Systems Manager 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane [email protected]
Boulder, CO 80301 https://www.nwra.com/