Running SLES 10 on multiple compute nodes, with an OpenSuSE 10.2 NFS
server, and I am receiving the above log message on a semi-regular basis
in the NFS client system-logs. When this occurs, one of the users
receives an error (although there seems to be no way to easily collate
the users experience with the system log, short of using the timestamp).
Sometimes it's "read/write error", sometimes it's more specific
about something that was a file now being a directory (SVN tends to be
rather more verbose). Most of the time this is just a nuisance, but at
times (such as last week or so) it occurs so often that it blocks any
work getting done.
The NFS Server is running OpenSuSE 10.2 on a Dell PowerEdge 2900
with the PERC5/i controller. The NFS Clients are all SLES10 with the
standard mountoptions, and running in TCP mode (something about UDP +
NFS + GbE leads to subtle data corruption).
I have been able to find some references to this problem in Google,
but no solutions, and no discussion about what the problem stems from,
nor if any fixes have been attempted.
On Wed, 2008-03-05 at 12:52 -0800, Adam Schrotenboer wrote:
> Running SLES 10 on multiple compute nodes, with an OpenSuSE 10.2 NFS
> server, and I am receiving the above log message on a semi-regular basis
> in the NFS client system-logs. When this occurs, one of the users
> receives an error (although there seems to be no way to easily collate
> the users experience with the system log, short of using the timestamp).
> Sometimes it's "read/write error", sometimes it's more specific
> about something that was a file now being a directory (SVN tends to be
> rather more verbose). Most of the time this is just a nuisance, but at
> times (such as last week or so) it occurs so often that it blocks any
> work getting done.
>
> The NFS Server is running OpenSuSE 10.2 on a Dell PowerEdge 2900
> with the PERC5/i controller. The NFS Clients are all SLES10 with the
> standard mountoptions, and running in TCP mode (something about UDP +
> NFS + GbE leads to subtle data corruption).
Could you be a little bit more specific about the nature of the NFS
server: are you using knfsd or is this the legacy userland server?
Please also specify what kind of filesystem you are exporting, and the
export options you are using.
As for the client side, when do these errors occur? Have you seen them
occur after a prolonged period of use, or are they more frequent
immediately after a reboot or mount operation?
Cheers
Trond
Trond Myklebust wrote:
> On Wed, 2008-03-05 at 12:52 -0800, Adam Schrotenboer wrote:
>
>> Running SLES 10 on multiple compute nodes, with an OpenSuSE 10.2 NFS
>> server, and I am receiving the above log message on a semi-regular basis
>> in the NFS client system-logs. When this occurs, one of the users
>> receives an error (although there seems to be no way to easily collate
>> the users experience with the system log, short of using the timestamp).
>> Sometimes it's "read/write error", sometimes it's more specific
>> about something that was a file now being a directory (SVN tends to be
>> rather more verbose). Most of the time this is just a nuisance, but at
>> times (such as last week or so) it occurs so often that it blocks any
>> work getting done.
>>
>> The NFS Server is running OpenSuSE 10.2 on a Dell PowerEdge 2900
>> with the PERC5/i controller. The NFS Clients are all SLES10 with the
>> standard mountoptions, and running in TCP mode (something about UDP +
>> NFS + GbE leads to subtle data corruption).
>>
>
> Could you be a little bit more specific about the nature of the NFS
> server: are you using knfsd or is this the legacy userland server?
> Please also specify what kind of filesystem you are exporting, and the
> export options you are using.
>
We're using the kernel NFSd. The exported FS is XFS.
/mnt/storage0/users
192.168.1.6(rw,sync,insecure,no_subtree_check,no_root_squash)
192.168.1.0/24(rw,sync,insecure,no_subtree_check)
> As for the client side, when do these errors occur? Have you seen them
> occur after a prolonged period of use, or are they more frequent
> immediately after a reboot or mount operation?
>
They occur mostly randomly, we have not noticed any particular pattern
to their occurrence, we have not noticed any correlation with mount-time.
Here is an excerpt from one of our servers. Note that all servers were
shutdown on Feb 29th and restarted on March 1st due to a planned power
outage. All systems are backed up by UPS.
> Cheers
> Trond
>
On Wednesday March 5, [email protected] wrote:
> They occur mostly randomly, we have not noticed any particular pattern
> to their occurrence, we have not noticed any correlation with mount-time.
> Here is an excerpt from one of our servers. Note that all servers were
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Where :-)
A excerpt from the logs to show actual error messages could be
helpful.
Thanks,
NeilBrown
> shutdown on Feb 29th and restarted on March 1st due to a planned power
> outage. All systems are backed up by UPS.
> > Cheers
> > Trond
> >
>
>
Neil Brown wrote:
> On Wednesday March 5, [email protected] wrote:
>
> Where :-)
>
> A excerpt from the logs to show actual error messages could be
> helpful.
>
> Thanks,
> NeilBrown
sorry, must have missed the paste
Feb 28 17:27:58 koi kernel: nfs_update_inode: inode 3221350147 mode
changed, 0040755 to 0100644
Feb 28 21:57:22 koi kernel: nfs_update_inode: inode 2149056279 mode
changed, 0040755 to 0100755
Mar 5 01:43:21 koi kernel: nfs_update_inode: inode 3222473680 mode
changed, 0040755 to 0100644
Mar 5 08:52:19 koi kernel: nfs_update_inode: inode 3222473568 mode
changed, 0120777 to 0100644
Mar 5 15:00:01 koi kernel: nfs_update_inode: inode 3222473569 mode
changed, 0120777 to 0100644
Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473674 mode
changed, 0120777 to 0100644
Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473675 mode
changed, 0120777 to 0100644
Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473672 mode
changed, 0040755 to 0100644
Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473673 mode
changed, 0120777 to 0100644