2010-11-28 14:23:26

by Jan Engelhardt

[permalink] [raw]
Subject: NFS: server error: fileid changed

Hi,


Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really
unreliable, having clients start to emit messages like

(nfsv3)
[42306.521225] NFS: server nova error: fileid changed
[42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0

On nfsv4, it is even worse, as updates on the server are not immediately
reflected on the client like it was the case on nfsv3.
Unfortunately, that is all the information I currently have.
(Server is on 2.6.36-rc8, client on 2.6.37-rc1.)

thanks,
Jan


2010-11-28 16:05:45

by J. Bruce Fields

[permalink] [raw]
Subject: Re: NFS: server error: fileid changed

On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really
> unreliable, having clients start to emit messages like
>
> (nfsv3)
> [42306.521225] NFS: server nova error: fileid changed
> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
>
> On nfsv4, it is even worse, as updates on the server are not immediately
> reflected on the client like it was the case on nfsv3.
> Unfortunately, that is all the information I currently have.
> (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)

Have you really being changing only the server, or have you been
changing the client at the same time?

See e.g.

http://marc.info/?l=linux-nfs&m=129088225122788&w=2

--b.

2010-11-28 18:19:46

by J. Bruce Fields

[permalink] [raw]
Subject: Re: NFS: server error: fileid changed

On Sun, Nov 28, 2010 at 06:56:37PM +0100, Jan Engelhardt wrote:
>
> On Sunday 2010-11-28 17:05, J. Bruce Fields wrote:
> >On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
> >> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really
> >> unreliable, having clients start to emit messages like
> >>
> >> (nfsv3)
> >> [42306.521225] NFS: server nova error: fileid changed
> >> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
> >>
> >> On nfsv4, it is even worse, as updates on the server are not immediately
> >> reflected on the client like it was the case on nfsv3.
> >> Unfortunately, that is all the information I currently have.
> >> (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
> >
> >Have you really being changing only the server, or have you been
> >changing the client at the same time?
> >
> >See e.g.
> > http://marc.info/?l=linux-nfs&m=129088225122788&w=2
>
> For my definition of not touching the server yes. (There may be
> atime updates, but that's about it.)

I was asking about kernel versions; so:

client 2.6.37-rc1, server 2.6.33

worked, but

client 2.6.37-rc1, server 2.6.36-rc8

didn't?

It would be worth trying the most recent client patches anyway, as your
symptoms look suspiciously similar to those reported recently, and we'd
like to rule out the known 2.6.37-rc1 client bugs.

What filesystem are you exporting?

--b.

2010-11-28 17:56:38

by Jan Engelhardt

[permalink] [raw]
Subject: Re: NFS: server error: fileid changed


On Sunday 2010-11-28 17:05, J. Bruce Fields wrote:
>On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
>> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really
>> unreliable, having clients start to emit messages like
>>
>> (nfsv3)
>> [42306.521225] NFS: server nova error: fileid changed
>> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
>>
>> On nfsv4, it is even worse, as updates on the server are not immediately
>> reflected on the client like it was the case on nfsv3.
>> Unfortunately, that is all the information I currently have.
>> (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
>
>Have you really being changing only the server, or have you been
>changing the client at the same time?
>
>See e.g.
> http://marc.info/?l=linux-nfs&m=129088225122788&w=2

For my definition of not touching the server yes. (There may be
atime updates, but that's about it.)

2010-12-05 14:41:58

by Jan Engelhardt

[permalink] [raw]
Subject: Re: NFS: server error: fileid changed


On Sunday 2010-11-28 19:19, J. Bruce Fields wrote:
>> >On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
>> >> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really
>> >> unreliable, having clients start to emit messages like
>> >>
>> >> (nfsv3)
>> >> [42306.521225] NFS: server nova error: fileid changed
>> >> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
>> >>
>> >> On nfsv4, it is even worse, as updates on the server are not immediately
>> >> reflected on the client like it was the case on nfsv3.
>> >> Unfortunately, that is all the information I currently have.
>> >> (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
>> >
>> >Have you really being changing only the server, or have you been
>> >changing the client at the same time?
>> >
>> >See e.g.
>> > http://marc.info/?l=linux-nfs&m=129088225122788&w=2
>>
>> For my definition of not touching the server yes. (There may be
>> atime updates, but that's about it.)
>
>I was asking about kernel versions; so:
>
> client 2.6.37-rc1, server 2.6.33
>
>worked, but
>
> client 2.6.37-rc1, server 2.6.36-rc8
>
>didn't?
>
>It would be worth trying the most recent client patches anyway, as your
>symptoms look suspiciously similar to those reported recently, and we'd
>like to rule out the known 2.6.37-rc1 client bugs.

So me doing what exactly?

>What filesystem are you exporting?

ext4.

2010-12-06 16:17:03

by J. Bruce Fields

[permalink] [raw]
Subject: Re: NFS: server error: fileid changed

On Sun, Dec 05, 2010 at 03:41:54PM +0100, Jan Engelhardt wrote:
>
> On Sunday 2010-11-28 19:19, J. Bruce Fields wrote:
> >> >On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
> >> >> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really
> >> >> unreliable, having clients start to emit messages like
> >> >>
> >> >> (nfsv3)
> >> >> [42306.521225] NFS: server nova error: fileid changed
> >> >> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
> >> >>
> >> >> On nfsv4, it is even worse, as updates on the server are not immediately
> >> >> reflected on the client like it was the case on nfsv3.
> >> >> Unfortunately, that is all the information I currently have.
> >> >> (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
> >> >
> >> >Have you really being changing only the server, or have you been
> >> >changing the client at the same time?
> >> >
> >> >See e.g.
> >> > http://marc.info/?l=linux-nfs&m=129088225122788&w=2
> >>
> >> For my definition of not touching the server yes. (There may be
> >> atime updates, but that's about it.)
> >
> >I was asking about kernel versions; so:
> >
> > client 2.6.37-rc1, server 2.6.33
> >
> >worked, but
> >
> > client 2.6.37-rc1, server 2.6.36-rc8
> >
> >didn't?
> >
> >It would be worth trying the most recent client patches anyway, as your
> >symptoms look suspiciously similar to those reported recently, and we'd
> >like to rule out the known 2.6.37-rc1 client bugs.
>
> So me doing what exactly?

I don't understand the question. You may actually want to wait till the
discussion of the client changes dies down and then try the next -rc.

(But it would still be helpful to have an answer to my question about
kernel versions above.)

> >What filesystem are you exporting?
>
> ext4.

OK, thanks.--b.

2011-01-31 15:40:51

by J. Bruce Fields

[permalink] [raw]
Subject: Re: NFS: server error: fileid changed

On Sun, Jan 30, 2011 at 02:27:34PM +0100, Jan Engelhardt wrote:
>
> On Sunday 2010-11-28 19:19, J. Bruce Fields wrote:
> >>>On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
> >>>> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really
> >>>> unreliable, having clients start to emit messages like
> >>>>
> >>>> (nfsv3)
> >>>> [42306.521225] NFS: server nova error: fileid changed
> >>>> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
> >>>>
> >>>> On nfsv4, it is even worse, as updates on the server are not
> >>>> immediately reflected on the client like it was the case on
> >>>> nfsv3. Unfortunately, that is all the information I currently
> >>>> have. (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
> >>>
> >>>Have you really being changing only the server, or have you been
> >>>changing the client at the same time?
>
> Fast forward. I have now a 2.6.38-rc1+ client with a 2.6.37 server,
> using nfs4 with ext4. The "fileid changes" messages seem to be gone.

That leaves in the dark about that particular problem.

> I change a file on the server, compile it there, and then would expect
> the client to pick up the change, but some awkward internal caching
> mechanism pins the old copy and I don't get, say, new debug messages I
> have added to my program that was just recompiled.

Remind me what the backend filesystem is? Also, does

echo 0 >/proc/sys/fs/leases-enable

on startup fix the problem?

--b.

2011-01-30 13:27:36

by Jan Engelhardt

[permalink] [raw]
Subject: Re: NFS: server error: fileid changed


On Sunday 2010-11-28 19:19, J. Bruce Fields wrote:
>>>On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
>>>> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really
>>>> unreliable, having clients start to emit messages like
>>>>
>>>> (nfsv3)
>>>> [42306.521225] NFS: server nova error: fileid changed
>>>> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
>>>>
>>>> On nfsv4, it is even worse, as updates on the server are not
>>>> immediately reflected on the client like it was the case on
>>>> nfsv3. Unfortunately, that is all the information I currently
>>>> have. (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
>>>
>>>Have you really being changing only the server, or have you been
>>>changing the client at the same time?

Fast forward. I have now a 2.6.38-rc1+ client with a 2.6.37 server,
using nfs4 with ext4. The "fileid changes" messages seem to be gone.

I change a file on the server, compile it there, and then would expect
the client to pick up the change, but some awkward internal caching
mechanism pins the old copy and I don't get, say, new debug messages I
have added to my program that was just recompiled.